AI workflow construction method, system and electronic equipment based on image recognition

Abstract

The present invention relates to the technical field of workflow construction, specifically to an AI workflow construction method, system, and electronic device based on image recognition. First, a video recording tool is used to record the workflow and collect operation data. Subsequently, image recognition and natural language processing technologies are used to analyze the video content, extract key frames and operation steps, and then, combined with user operation event data, the operation behavior is divided into time series and analyzed. The actual operation steps are determined and a detailed step list with alliance label information is generated. Then, based on the analysis results, the operation steps are combined with video segments to form a structured workflow, and detailed descriptions and annotation information are added to each step. Finally, an index structure is established so that users can quickly retrieve the required business process videos or documents by keywords. The present invention helps to improve the visualization and comprehensibility of business processes, reduce learning costs, and improve work efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of workflow construction, and in particular to an AI workflow construction method, system, and electronic device based on image recognition. Background Art

[0002] In modern business operations, optimizing and standardizing workflows are key to improving efficiency, reducing errors, and enhancing service quality. With the rapid development of artificial intelligence (AI) technology, particularly advancements in computer vision, natural language processing (NLP), and machine learning, AI-based workflow construction systems have become possible. These systems can automatically analyze existing workflows, identify key operational steps, generate detailed operational guides, and provide interactive learning materials, thereby helping companies train employees more effectively, improve process design, and enhance business execution efficiency.

[0003] Chinese invention patent publication number CN113805885B discloses a front-end construction system and method for a workflow-based management system. The system comprises a technical construction subsystem for constructing a front-end architecture, including a workflow front-end engine for implementing front-end process-based flows; a business component library for providing business components; a tool module for providing tool classes; a data persistence module for persisting user and business information in the system and providing a public data acquisition interface; a business construction subsystem for constructing a front-end business processing architecture, including a system foundation construction module for providing basic front-end business processing functions; a workflow design module for workflow configuration management and business process management; and an actual business design module for developing actual business functions. The front-end construction subsystem integrates the front-end architecture and business processing architecture to form the system front-end. This provides a standardized front-end and reduces costs.

[0004] However, as enterprise workflows continue to become more complex, the demand for employee training and process learning is increasing. Traditional learning methods, such as operation manuals or long video tutorials, are often inefficient and difficult to maintain. Summary of the Invention

[0005] The purpose of the present invention is to address the problems existing in the background technology and propose an AI workflow construction method, system and electronic equipment based on image recognition.

[0006] The technical solution of the present invention is a method for constructing an AI workflow based on image recognition, which includes the following specific implementation steps:

[0007] S1. Use a screen recording tool to record the user's operation process in the business system to generate video data. At the same time, capture the user's operation data and system feedback information to generate operation event data.

[0008] S2. Extract video frames from the video data at fixed time intervals. By comparing the pixel differences between adjacent frames, detect scene changes or interface changes to extract key frames. Use a pre-trained image recognition model to analyze the important operation steps, interface changes, and text content in the key frames. Combine the user's operation event data with the video data, analyze each operation behavior and divide it into time series. Decompose the video into multiple step segments, divide the operation steps based on the operation event data, filter out unnecessary intermediate steps, and generate a detailed step list with alliance label information;

[0009] S3. Verify the completeness of the extracted detailed step list based on the alliance tag information. Then extract the steps in the detailed step list, sort all steps by timestamp, remove redundant steps, and optimize the logical structure. Then, reorganize the workflow based on the optimized step set, including splicing step descriptions, intercepting corresponding video segments, adding subtitles and visual annotations. Finally, construct standardized operation instructions and combine them with video segments to generate interactive content, output them to a document, and generate the final workflow list.

[0010] S4. Label the generated workflow videos and text content and create index keywords to facilitate user retrieval. Users can quickly find the corresponding business process videos or documents through keyword search for learning, and output the business process videos or documents in multiple formats, including visual video tutorials, process documents, and interactive process guides.

[0011] Preferably, the key frame extraction method is as follows:

[0012] S21. Use a video processing tool to load video data and transcode the video to a fixed frame rate to evenly extract frames and reduce computational complexity:

[0013] F rate =min(F original ,F target );

[0014] Where, F rate Indicates the actual frame rate used, which is used to uniformly sample video frames; F original Indicates the original frame rate of the video data; F target Indicates the target frame rate, which is used to reduce redundant information and improve analysis efficiency;

[0015] S22. For each pair of adjacent frames, calculate:

[0016]

[0017] Where D(i,j) represents the average pixel difference between the i-th frame and the j-th frame; F i (x,y) and Fj (x, y) represents the grayscale value of the pixel (x, y) in the i-th frame and the grayscale value of the pixel (x, y) in the j-th frame respectively; N and M represent the width and height of the frame respectively;

[0018] S23, setting pixel difference threshold T d , if the pixel difference between any frame i and its adjacent frames is greater than the set pixel difference threshold T d , that is, D(i,i+1)>T d , then mark frame i+1 as the key frame, that is, extract the key frame.

[0019] Preferably, the generation process of alliance tag information is as follows:

[0020] S31, randomly select a positive integer O as the group number, select a random number r∈Z q * ={1,2,...,q-1}, calculate the alliance audit code CJ = r×P;

[0021] Where q is a prime number, thereby defining two cyclic groups G1 and G2 of order q, P is a generator of the cyclic group G1; based on this, a bilinear map e: G1×G1→G2 is defined;

[0022] S32, for any step s i Randomly pick a number x i ∈Z q * , calculate its audit parameter PA i =x i ×P;

[0023] S33, calculate any of the following steps s i Auxiliary information generation factor: FS 1i =H1(D' i , k i , C.J., P.A. i , O), FS2 = H2 (O, CJ);

[0024] Among them, D' i Indicates any step s i Step description information D i The binary string form of k i Indicates any step s i Step identifier; H1 represents the hash function, H1: {0, 1} * →G1; H2 represents the hash function, H2: {0, 1} ** →G1;

[0025] S34. Calculate any step s i Tag information for ILablei =x i ×FS2+r×FS 1i ;

[0026] S35, generate detailed step list List' Step The steps included in generating alliance tag information IJ:

[0027] Among them, n represents the detailed step list List' Step The number of steps included.

[0028] Preferably, the integrity verification process of the detailed step list of the detailed step list extracted based on the alliance tag information verification is as follows:

[0029] S41. Calculate any of the following steps s i Auxiliary check factor: FA 1i =H1(D' i , k i , C.J., P.A. i , O), FA2 = H2 (O, CJ);

[0030] Among them, D' i Indicates any step s i Step description information D i The binary string form of k i Indicates any step s i Step identifier; H1 represents the hash function, H1: {0, 1} * →G1; H2 represents the hash function, H2: {0, 1} ** →G1; O represents any step s i Group number; CJ represents the alliance review code of group number O; PA i Indicates any step s i Audit parameter; G1 and G2 represent two cyclic groups of order q;

[0031] S42. Calculate a first check factor FV1 = e(IJ, P);

[0032] Where e represents a bilinear map, e:G1×G1→G2; P is a generator of the cyclic group G1;

[0033] S43. Calculate the second check factor FV2:

[0034] S44. If FV1 = FV2, the steps of the work to be reorganized have been completely extracted, that is, the integrity check of the extracted detailed step list has passed.

[0035] Preferably, the process of generating the detailed step list is as follows:

[0036] S51. Read the user's operation event data and analyze the timestamp and event type of the operation event data:

[0037] E={[t1,e1,p1],[t2,e2,p2],...,[t j ,e j ,p j ],...,[t n ,e n ,p n ]};

[0038] Where E represents the event sequence in the operation event data; t j Indicates the time stamp of any operation event in the operation event data; j Represents any event type in the operation event data, that is, identifies the specific operation; p i Represents any event e j Identifier of

[0039] S52, according to the time interval T of the set event gap , divide the event sequence into different operation steps, if any two consecutive events t k and t k+1 The timestamps between them exceed T gap , then the event t k+1 It is a new step;

[0040] S53, the event sequence E and the key frame data set F are combined according to the timestamp t j and frame timestamp t i Alignment, calculate the difference Δt between the event timestamp and the keyframe timestamp i,j , for any e j , select the key frame f that is closest to its timestamp i As context for this event:

[0041] Δt i,j =|t i -t j |;

[0042] Where Δt i,j Indicates event e j The timestamp and key frame f i The time difference between the timestamps;

[0043] S54. Match the event sequence E with the key frame dataset F to find the associated time window:

[0044] W k =[tstart ,t end ];

[0045] Where W k represents the kth time window, i.e., the time range of a possible operation step, which contains several key frames and constitutes a video segment; t start With t end Respectively represent the start and end time of the window, which are determined by the transformation time of events and keyframes;

[0046] S55. Further inspect the key frames within the time window to ensure that they contain obvious interface changes, generate a list of steps, score and optimize the importance of each step, and screen out the key steps that contribute to the workflow;

[0047] S56, further mark the selected key steps in detail, generate step description information, and i Provide additional explanation and then generate a detailed step list List' Step ={s1,s2,...,s i ,..,s m};

[0048] Among them, s i Indicates any operation step, s i ={t start,i ,t end,i ,e i ,Elements i ,Scoer(i),D i};D i Indicates s i Step description information; start,i and t end,i Respectively represent the operation steps i The start and end time of e i Indicates step s i Operation type; Elements i Indicates operation e i The interface elements involved.

[0049] Preferably, the screening process of key steps is as follows:

[0050] S61, in order to determine whether there is a significant change between two frames, calculate any frame f i and adjacent frames f i+1 Pixel difference:

[0051]

[0052] Where, f i With fi+1 Represent adjacent key frames respectively; N and M represent the width and height of the frame respectively; D(f i ,f i+1 ) represents the normalized value of the pixel difference between frames; ||f i -f i+1 ||1 represents the L1 norm of the pixel matrix, that is, the sum of the absolute values ​​of the grayscale value differences of all pixels;

[0053] S62、If D(f i ,f i+1 )>T d , it is considered that an interface change has occurred;

[0054] Among them, T d Indicates the change threshold of the preset pixel;

[0055] S63, according to the time window W k and interface changes, preliminarily classify the operation events into corresponding steps;

[0056] S64. Set the event type of any operation event k Operation type weight W type (e k ) and set the operation change weight W frame (W k ):

[0057]

[0058] S65. Score each step based on the operation type and interface change weight:

[0059] Score(k)=α×W type (e k )+β×W frame (W k );

[0060] Where Score(k) represents the importance score of step k; α and β represent weight coefficients, which reflect the importance ratio of the operation event and the interface change.

[0061] S66: Remove irrelevant steps based on the scoring results, i.e., remove steps with scores below the minimum scoring threshold T. score Steps, filter out the key steps, and get the step list List Step ={s1,s2,...,s i ,..,s m};

[0062] Among them, s i Indicates any operation step, s i ={t start,i,t end,i ,e i ,Elements i , Scoer(i)}, set its step identifier to k i ;t start,i and t end,i Respectively represent the operation steps i The start and end time of e i Indicates step s i Operation type; Elements i Indicates operation e i Involved interface elements; Scoer(i) represents step s i 's rating.

[0063] Preferably, the process of generating the workflow list is as follows:

[0064] S71. Arrange all steps in ascending order of timestamps:

[0065] S ordered =Sort({s1,s2,...,s i ,...,s m},key=t si );

[0066] Where S ordered Represents a set of steps after preliminary sorting; Sort() represents a sorting function; key represents the sorting criterion; s i Indicates any step; t si Indicates any step s i The timestamp recorded when the data was extracted;

[0067] S72, determine whether the two steps are repeated steps, if sim(s i ,s j )>T sim , then it is considered that s j To repeat the steps, delete them;

[0068] sim(s i ,s j )=w1·sim e (s i ,s j )+w2·sim Elements (s i ,s j );

[0069] Among them, sim(s i ,s j ) represents step s i and steps jSimilarity; sim e (s i ,s j ) represents step s i and steps j The similarity of the operation type, including but not limited to clicking the same button twice; Elements (s i ,s j ) represents step s i and steps j The similarity of the interface elements, including but not limited to the operation design of the same interface elements; w1 and w2 represent step s i and steps j The weight coefficient reflects the priority of similarity calculation; T sim Indicates the set step similarity threshold;

[0070] S73. By detecting the operation type and interface elements, determine the logical dependencies between steps, reorder the step sequence according to the dependencies, and obtain the optimized step set S' ordered , and from the optimized step set S' ordered Extract the description information of each step after optimization D i 、Operation type e i and the interface elements involved i , describe the information of all steps D i Sequentially stitch together a complete process description, based on the time interval of each step [t start,i , t end,i ], extract the corresponding video segments from the original video, and splice all the segments into a complete video process in step order;

[0071] S74. Add subtitles and visual annotations to each video segment. The subtitle content is taken from the text description of each step. i , the annotation information comes from the interface elements in the step i , and construct any step s accordingly i Instruction template: i ={s i ,e i ,Elements i ,D i}, combining the operation instructions with the key frames in the video segment to generate interactive content and output interactive instructions;

[0072] S75. Compile the interactive guidance of each step into the output document in the form of a diagram to generate a workflow list. work ={s'1,s'2,...,s'i ,...};

[0073] Among them, s' i Represents any optimized operation step, Represents any optimized operation step s' i The corresponding video segment includes the video content of the original video segment, the added subtitles, and the visual annotations.

[0074] Preferably, the image recognition model is based on the network structure design of the YOLOv8s model, combined with multi-scale sliding window attention.

[0075] The technical solution of the present invention is an AI workflow construction system based on image recognition, which is used to execute the above-mentioned AI workflow construction method based on image recognition, including:

[0076] The data acquisition module is used to record the user's operation process in the business system through a screen recording tool to generate high-quality video files. It also captures the user's operation event data and system feedback information for subsequent video analysis and disassembly;

[0077] The video analysis and step decomposition module extracts video frames at fixed time intervals from the video. By comparing the pixel differences between adjacent frames, it detects key frames indicating scene or interface changes. It then uses image recognition and natural language processing technologies to analyze the important content in the video. It then decomposes the video into multiple step segments, divides the operation steps based on the operation event data, and filters out unnecessary intermediate steps.

[0078] The workflow reorganization module is used to automatically reorganize the video based on the disassembled segments, remove irrelevant or duplicate content, and generate videos of each workflow. At the same time, it uses computer vision, image segmentation, and object recognition technologies to extract text content from the video and generate text descriptions of the workflow, generating operational instructions corresponding to the business process.

[0079] The content annotation and retrieval module is used to annotate the generated workflow videos and text content, establish index keywords, and provide a search interface for users. Users can enter keywords through the search interface to quickly find the corresponding business process videos or documents for learning, and it also supports multi-format output;

[0080] Database, used to store data generated during system operation.

[0081] The technical solution of the present invention: An electronic device includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the above-mentioned method for constructing an AI workflow based on image recognition is implemented.

[0082] Compared with the prior art, the above technical solution of the present invention has the following beneficial technical effects:

[0083] This invention proposes an AI workflow construction method, system, and electronic device based on image recognition. By recording a customer's workflow video and using artificial intelligence technology to analyze, disassemble, and reassemble the video, it can efficiently generate easy-to-learn process steps and provide support for rapid retrieval and automated learning. This approach not only improves the efficiency of workflow learning but also enables employees to easily master complex workflows:

[0084] (1) Automated process generation: The present invention can automatically record, analyze, and generate video and text content of the workflow, greatly reducing the time and cost of manually writing process documents;

[0085] (2) Process optimization: Through video disassembly and reorganization, irrelevant steps are removed, making the learning content more concise and clear, and users can quickly grasp the core process;

[0086] (3) Intelligent retrieval: The system establishes detailed indexes and annotations for the generated learning content. Users can quickly find relevant processes through simple keyword searches, improving learning efficiency.

[0087] (4) Multiple learning methods: The present invention supports process display in multiple forms such as video, text, and charts, and users can choose the most suitable learning method according to their needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 This is a system flow chart of an AI workflow construction system based on image recognition proposed by the present invention;

[0089] Figure 2 This is a principle block diagram of the AI ​​workflow construction method based on image recognition proposed in this invention. DETAILED DESCRIPTION

[0090] Example 1, as Figure 1 As shown, the present invention proposes an AI workflow construction system based on image recognition, including: a data acquisition module, a video analysis and step decomposition module, a workflow reorganization module, a content annotation and retrieval module and a database.

[0091] The data acquisition module is used to record the user's operation process in the business system through a screen recording tool to generate high-quality video files. It also captures the user's operation event data (including but not limited to keyboard input and mouse clicks) and system feedback (including but not limited to window changes and pop-up boxes) for subsequent video analysis and disassembly;

[0092] The video analysis and step decomposition module is used to extract video frames from the video at fixed time intervals. By comparing the pixel differences between adjacent frames, it detects key frames that indicate scene changes or interface changes for subsequent analysis. It also uses image recognition and natural language processing technologies to analyze important content in the video, including but not limited to operation steps, interface changes, and text content. It then decomposes the video into multiple step segments, divides the operation steps based on operation event data, and filters out unnecessary intermediate steps (including but not limited to waiting time and loading pages).

[0093] The workflow reorganization module is used to automatically reorganize the video based on the disassembled segments, remove irrelevant or duplicate content, and generate workflow videos. It also uses computer vision, image segmentation, and object recognition technologies to extract text content from the video and generate a text description of the workflow, generating operational instructions corresponding to the business process, including but not limited to specific operational steps, diagrams, or screenshots.

[0094] The content annotation and retrieval module is used to annotate the generated workflow videos and text content, establish keywords to facilitate user retrieval, and provide users with a search interface. Users can enter keywords to search through the search interface to quickly find the corresponding business process videos or documents for learning. It also supports multi-format output, including but not limited to visual video tutorials, process documents, and interactive process guides, so that users can use them according to their needs.

[0095] The database is used to store data generated during system operation.

[0096] In the second embodiment, the present invention proposes an AI workflow construction method based on image recognition, which is applied to the AI ​​workflow construction system based on image recognition proposed in the first embodiment, such as Figure 2 As shown, the specific implementation steps include the following:

[0097] S1. The data acquisition module uses a video recording tool to record the customer's workflow to generate high-quality video data. It also captures the customer's operation data (including but not limited to keyboard input and mouse clicks) and system feedback information (including but not limited to window changes and pop-up boxes), structures the operation data and system feedback information, generates operation event data, and then stores the video data and operation event data in the database.

[0098] S2, the video analysis and step disassembly module extracts video data and operation event data from the database and implements the following operation process:

[0099] S21. Extract video frames from the extracted video data at fixed time intervals, and detect key frames indicating scene changes or interface changes by comparing pixel differences between adjacent frames. The specific implementation process is as follows:

[0100] S2101. Use a video processing tool to load video data and transcode the video to a fixed frame rate to evenly extract frames and reduce computational complexity.

[0101] F rate =min(F original ,F target );

[0102] Where, F rate Indicates the actual frame rate used, which is used to uniformly sample video frames; F original Indicates the original frame rate of the video data; F target Indicates the target frame rate, which is used to reduce redundant information and improve analysis efficiency;

[0103] S2102. For each pair of adjacent frames, calculate:

[0104]

[0105] Where D(i,j) represents the average pixel difference between the i-th frame and the j-th frame; F i (x,y) and F j (x, y) represents the grayscale value of the pixel (x, y) in the i-th frame and the grayscale value of the pixel (x, y) in the j-th frame respectively; N and M represent the width and height of the frame respectively;

[0106] S2103, setting pixel difference threshold T d , if the pixel difference between any frame i and its adjacent frames is greater than the set pixel difference threshold T d , that is, D(i,i+1)>T d , then mark frame i+1 as the key frame;

[0107] S2104. Save the timestamps {t1, t2, ...} corresponding to the key frames and construct a key frame dataset:

[0108] F={[t1,f1],[t2,f2],...,[t i ,f i ],...,[t n ,f n ]};

[0109] Where, F represents the key frame dataset; f i Represents any key frame; t i Represents any key frame f i The corresponding timestamp;

[0110] S22. Use image recognition and natural language processing technology to analyze important operation steps, interface changes, and text content in the video. The specific implementation process is as follows:

[0111] S2201, extract any key frame image f i Perform grayscale processing or scaling processing to adapt to the input of the image recognition model to obtain the preprocessed key frame image f i ;

[0112] S2202, compare the processed key frame image f i Input into the pre-trained image recognition model, perform target detection on the keyframe, and output the confidence, category label and target position coordinates of the interface element in the keyframe;

[0113] It should be noted that the image recognition model is based on the network structure design of the YOLOv8s model, combined with a multi-scale sliding window attention (Multi-Scale Sliding Window Attention), using a multi-scale convolutional neural network to extract features from keyframes, understand the details and global information of keyframes, and improve recognition accuracy. The input keyframe is divided into multiple small blocks, and an attention weight is calculated for each small block. These weights represent the importance of each small block to the entire keyframe, that is, the image recognition model focuses on the parts of the keyframe that are most relevant to the interface elements and assigns higher weights to these parts. The weighted feature vectors are obtained by multiplying the attention weights by the feature vector of each small block. These weighted feature vectors are then used to detect interface elements.

[0114] S2203: Using OCR (Optical Character Recognition) technology to recognize the output of the image recognition model and identify text information in the image frame;

[0115] S2204, output the information group C (f i ):

[0116] C(f i )={Confidence i ,Lable i ,Loc i ,text i};

[0117] In the formula, C(f i ) represents any key frame f i Information group of interface elements; Confidence i Represents any key frame f iConfidence of interface elements in Lable i Represents any key frame f i Category label of the interface element in Loc i Represents any key frame f i Text information in;

[0118] When the confidence level is greater than the set confidence threshold (T C ), it is considered that there are interface elements in the key frame;

[0119] S23. Combine the user's operation event data and video data to analyze each operation behavior and divide it into time series. The specific implementation process is as follows:

[0120] S2301. Read the extracted user operation event data and analyze the timestamp and event type of the operation event data:

[0121] E={[t1,e1,p1],[t2,e2,p2],...,[t j ,e j ,p j ],...,[t n ,e n ,p n ]};

[0122] Where E represents the event sequence in the operation event data; t j Indicates the time stamp of any operation event in the operation event data; j Represents any event type in the operation event data, that is, identifies a specific operation, including but not limited to "clicking a button" and "entering text"; i Represents any event e j Identifiers, including but not limited to:

[0123] Click operation: mouse click;

[0124] Text input: keyboard input;

[0125] Form submission: Submit button click;

[0126] Drag operation: mouse drag;

[0127] S2302, according to the time interval T of the set event gap , divide the event sequence into different operation steps, if any two consecutive events t k and t k+1 The timestamps between them exceed T gap , then the event t k+1 It is a new step;

[0128] S2303, each operation event has a timestamp, and the key frame extraction of the video frame also has a timestamp, and the event sequence E and the key frame dataset F are combined according to the timestamp t j and frame timestamp t i Alignment, calculate the difference Δt between the event timestamp and the keyframe timestamp i,j , for any e j , select the key frame f that is closest to its timestamp i As the context of this event, minΔt i,j , ensure that the key frame f when the event occurs i Can be used as the event j Context:

[0129] Δt i,j =|t i -t j |;

[0130] Where Δt i,j Indicates event e j The timestamp and key frame f i The time difference between the timestamps;

[0131] S24, after completing the event sequence E and the key frame data set F according to the timestamp t j and frame timestamp t i After alignment, the video analysis and step decomposition module extracts the actual operation steps and adds annotation information for each step. The specific implementation process is as follows:

[0132] S2401. Match the event sequence E with the key frame dataset F to find the associated time window:

[0133] W k =[t start ,t end ];

[0134] Where W k represents the kth time window, i.e., the time range of a possible operation step, which contains several key frames and constitutes a video segment; t start With t end Respectively represent the start and end time of the window, which are determined by the transformation time of events and keyframes;

[0135] S2402: Further check the key frames within the time window to ensure that they contain obvious interface changes (including but not limited to jumping to a new page after a button is pressed). The specific implementation process is as follows:

[0136] (1) To determine whether there is a significant change between two frames, calculate f i and adjacent frames fi+1 Pixel difference:

[0137]

[0138] Where, f i With f i+1 Represent adjacent key frames respectively; N and M represent the width and height of the frame respectively; D(f i ,f i+1 ) represents the normalized value of the pixel difference between frames; ||f i -f i+1 ||1 represents the L1 norm of the pixel matrix, that is, the sum of the absolute values ​​of the grayscale value differences of all pixels;

[0139] (2) If D(f i ,f i+1 )>T d (preset pixel change threshold), it is considered that an interface change has occurred, and the subsequent step S2403 is continued;

[0140] (3) According to the time window W k and interface changes, preliminarily classify the operation events into corresponding steps;

[0141] S2403. After initially extracting the steps, each step is scored and optimized for importance, and the key steps that contribute to the workflow are screened out. The specific implementation process is as follows:

[0142] (1) Different types of operation events have different importance. For example, "submitting a form" is more important than "mouse hovering". Set the event type of any operation event to k Operation type weight W type (e k );

[0143] It should be noted that W type (e k ) determined by prior knowledge;

[0144] (2) The greater the interface change, the more important the step is, and the weight of the operation change is set as W. frame (W k ):

[0145]

[0146] (3) Score each step based on the operation type and interface change weight:

[0147] Score(k)=α×W type (e k )+β×W frame (W k );

[0148] Where Score(k) represents the importance score of step k; α and β represent weight coefficients, which reflect the importance ratio of the operation event and the interface change.

[0149] (4) Remove irrelevant steps (scores below the minimum scoring threshold T) based on the scoring results. score Steps), including but not limited to waiting for page loading and idle time without operation, to filter out key steps;

[0150] (5) Get the step list List Step ={s1,s2,...,s i ,..,s m};

[0151] Among them, s i Indicates any operation step, s i ={t start,i ,t end,i ,e i ,Elements i , Scoer(i)}, set its step identifier to k i ;

[0152] Where, t start,i and t end,i Respectively represent the operation steps i The start and end time of e i Indicates step s i Action types, including but not limited to mouse clicks or text input; Elements i Indicates operation e i Involved interface elements; Scoer(i) represents step s i 's rating;

[0153] S2404: further annotate the selected key steps in detail and generate step description information. The process of generating the step description information is as follows:

[0154] (1) Extract video data, use the pre-trained video understanding model to classify the video scenes, identify different operation contexts, and output video content understanding, including but not limited to: form filling scenes, button clicking scenes, or data uploading scenes;

[0155] (2) Based on the operational event data and combined with video content understanding, the semantic information of the event is supplemented;

[0156] For example, if a "mouse click" is detected, the classification results are combined to determine whether it is "clicking the login button" or "clicking the download link"; when text input is recognized, the field title detected on the interface is combined to supplement it with "enter user name" or "enter search keywords";

[0157] (3) Generate intelligent prompts by combining interface text extracted by OCR and objects detected by vision;

[0158] For example: in a form scenario, the prompt is "Possible action is to fill in the field"; in a confirmation page scenario, the prompt is "Possible action is to submit the form";

[0159] (4) Based on intelligent prompts, accurately label the steps:

[0160] Based on the initially extracted operation events, the operation type is adjusted using the results of video content understanding, and the detection results of video understanding and the information group C (f i ) and operation events, mark the interface elements and their positions of each key step, and use intelligent prompts to supplement the actions and content, integrate scene classification and visual-text analysis results, generate natural language descriptions, and obtain the s of any step. i Step description information D i ;

[0161] It should be noted that the video understanding model is built on the SlowFast model, which is an efficient video content element recognition model. It uses two branches: the Slow branch (used to process low-frame-rate videos and capture slow features and long-term contextual information) and the Fast branch (used to process high-frame-rate videos and capture rapidly changing details). These branches process the slow and fast features of the video respectively, thereby better capturing dynamic changes in the video.

[0162] S2405, for any step s i Provide additional explanation and then generate a detailed step list List' Step :

[0163] List' Step ={s1,s2,...,s i ,..,s m};

[0164] Among them, s i Indicates any operation step, s i ={t start,i ,t end,i ,e i ,Elements i ,Scoer(i),D i};

[0165] Where D i Indicates s i Step description information;

[0166] S2406, List' for detailed steps Step The steps included in the code are to generate the alliance tag information. The generation process is as follows:

[0167] (1) Randomly select a positive integer O as the group number and select a random number r∈Z q * ={1,2,...,q-1}, calculate the alliance audit code CJ = r×P;

[0168] Where q is a prime number, thereby defining two cyclic groups G1 and G2 of order q, and P is a generator of the cyclic group G1; based on this, a bilinear map e: G1×G1→G2 is defined and transmitted to the workflow reorganization module;

[0169] (2) for any step s i Randomly pick a number x i ∈Z q * , calculate its audit parameter PA i =x i ×P;

[0170] (3) Calculate any of the following steps s i Auxiliary information generation factor: FS 1i =H1(D' i , k i , C.J., P.A. i , O), FS2 = H2 (O, CJ);

[0171] Among them, D' i Indicates any step s i Step description information D i The binary string form of k i Indicates any step s i Step identifier; H1 represents the hash function, H1: {0, 1} * →G1; H2 represents the hash function, H2: {0, 1} ** →G1;

[0172] (4) Calculate any step s i Tag information for ILable i =x i ×FS2+r×FS 1i ;

[0173] (5) Generate a detailed step list List'Step The steps included in generating alliance tag information IJ:

[0174] Among them, n represents the detailed step list List' Step The number of steps included in

[0175] S2407, {O, CJ, List' Step , IJ} and {k i , P.A. i , s i}Store in a structured format in the database;

[0176] S3. The workflow reorganization module combines the video segments and generated workflows according to the workflow steps obtained through analysis and decomposition to generate workflow videos. It also uses computer vision, image segmentation, and object recognition technologies to extract text content from the videos and generate a text description of the workflow. It also decomposes the operations of each step to generate operation instructions corresponding to the workflow. The specific implementation steps are as follows:

[0177] S31, extract {O, CJ, List' from the database Step , IJ} and {k i , P.A. i , s i}, verify whether the steps of the work to be reorganized are completely extracted. The verification process is as follows:

[0178] (1) Calculate any of the following steps s i Auxiliary check factor: FA 1i =H1(D' i , k i , C.J., P.A. i , O), FA2 = H2 (O, CJ);

[0179] Among them, D' i Indicates any step s i Step description information D i The binary string form of k i Indicates any step s i Step identifier; H1 represents the hash function, H1: {0, 1} * →G1; H2 represents the hash function, H2: {0, 1} ** →G1; O represents any step s i Group number; CJ represents the alliance review code of group number O; PA i Indicates any step s i Audit parameter; G1 and G2 represent two cyclic groups of order q;

[0180] (2) Calculate the first check factor FV1 = e(IJ, P);

[0181] Where e represents a bilinear map, e:G1×G1→G2; P is a generator of the cyclic group G1;

[0182] (3) Calculate the second check factor FV2:

[0183] (4) If FV1 = FV2, the steps required for the reorganization have been completely extracted;

[0184] S32. Arrange all steps in ascending order of timestamps:

[0185]

[0186] Where S ordered Represents a set of steps after preliminary sorting; Sort() represents a sorting function; key represents the sorting criterion; s i Indicates any step; Indicates any step s i The timestamp recorded when the data was extracted;

[0187] S33, the set of steps S after preliminary sorting ordered Some steps may be redundant. Remove the redundant steps. The specific implementation process is as follows:

[0188] S3301: Determine whether two steps are repeated steps:

[0189] sim(s i ,s j )=w1·sim e (s i ,s j )+w2·sim Elements (s i ,s j );

[0190] Where, sim(s i ,s j ) represents step s i and steps j Similarity; sim e (s i ,s j ) represents step s i and steps j The similarity of the operation type, including but not limited to clicking the same button twice; Elements (s i ,s j ) represents step si and steps j The similarity of the interface elements, including but not limited to the operation design of the same interface elements; w1 and w2 represent step s i and steps j The weight coefficient reflects the priority of similarity calculation;

[0191] If sim(s i ,s j )>T sim (set step similarity threshold), then it is considered that s j To repeat the steps, delete them;

[0192] S3302. Optimize the logical structure between steps to form a coherent operation process, thereby avoiding the problem of incomplete logic or disordered sequence in the steps after cleaning. The specific implementation process is as follows:

[0193] (1) By detecting the operation type and interface elements, the logical dependencies between steps are determined. The dependencies are analyzed and determined based on the following conditions:

[0194] Time relationship: t end,i <t start,j ;

[0195] Among them, t end,i Indicates step s i End time; t start,j Indicates step s j The start time of

[0196] Operation type: including but not limited to "opening a page" relying on "login";

[0197] Interface elements: including but not limited to clicked buttons belonging to pre-loaded pages;

[0198] (2) Reorder the step sequence according to the dependency relationship to obtain the optimized step set S' ordered ;

[0199] S34, according to the optimized step set S' ordered , reorganize the steps according to the operation type and interface elements, and combine them into a workflow. The specific implementation process is as follows:

[0200] S3401, from the optimized step set S' ordered Extract the description information of each step after optimization D i 、Operation type e i and the interface elements involved i , describe the information of all steps D i Splice them together in sequence to form a complete text description of the process;

[0201] S3402, according to the time interval of each step [t start,i , t end,i ], extract the corresponding video segments from the original video, and splice all the segments into a complete video process in step order;

[0202] S3403, add subtitles to each video segment (step description information D i ) and visual annotations (including but not limited to click area highlights and operation instructions). The subtitles are taken from the text description of each step. i , the annotation information comes from the interface elements in the step i ;

[0203] S35. Operation type based on each step i , Interface elements involved i and step description information D i , build standardized operation guidelines, the specific implementation steps are as follows:

[0204] S3501, construct any step s i Operation guide template:

[0205] Instruction i ={s i ,e i ,Elements i ,D i};

[0206] For example: {step: "Step 2";

[0207] Action type: "Click";

[0208] Interface element: "Save button";

[0209] Guidance: "Click on the 'Save' button in the top-right corner to save the current progress;"};

[0210] S3502: Combine the operation instructions with the key frames in the video segment to generate interactive content and output the interactive instructions;

[0211] S3503, compiling the interactive guidance of each step into the output document in the form of a graphic;

[0212] S36. Generate workflow list List work ={s'1,s'2,...,s'i ,...};

[0213] Among them, s' i Represents any optimized operation step,

[0214] Where, Represents any optimized operation step s' i The corresponding video segment includes the video content of the original video segment, the added subtitles, and the visual annotations;

[0215] S4. Content annotation and retrieval module receives workflow list List work , and annotate the generated workflow videos and text content, create indexes and metadata tags, so users can quickly find the corresponding business process videos or documents through keyword searches for learning. The specific implementation process is as follows:

[0216] S41. Automatically perform preliminary annotation of videos, text, and interface elements based on image recognition technology, OCR technology, and natural language processing (NLP) algorithms:

[0217] Add tags to each video clip to indicate the operation steps, operation types, and interface elements involved in the video;

[0218] Annotate the text description of each operation step and generate keywords and tags to facilitate retrieval and indexing;

[0219] Label the interface elements in the operation steps, including but not limited to: element type, location, and function information;

[0220] S42. By structuring content annotation and metadata, an index structure is constructed to facilitate users to quickly search for required workflow steps, video clips, and operation instructions by keyword. The specific implementation process is as follows:

[0221] S4201. Convert the annotated content into an index structure so that each annotated information can be quickly searched using specific keywords, and construct index keywords:

[0222] In the formula, keyword i Indicates step s i Keywords;

[0223] Users can enter index keywords based on Query the relevant steps, and then follow Obtain step description information, video clips, and operation instructions. The system quickly returns all workflow content containing the keyword based on the inverted index.

[0224] For example, if a user enters "save" as a query keyword, the system returns all steps, video clips, and text descriptions related to "save";

[0225] S43. Output workflows in a variety of formats, including but not limited to text, video, and interactive content:

[0226] Text process documentation: Generate structured documents (including but not limited to Word or PDF) containing step-by-step instructions, relevant screenshots, and interactive guides for users to reference offline;

[0227] Video Tutorial: Export video segments of each step of the visualization with subtitles and visual annotations for intuitive learning and demonstration;

[0228] Interactive display: Generates web-based interactive instructions so that users can experience the operation process directly in the browser.

[0229] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A method for constructing an AI workflow based on image recognition, characterized in that: The specific implementation steps include the following: S1. Use a screen recording tool to record the user's operation process in the business system to generate video data. At the same time, capture the user's operation data and system feedback information to generate operation event data. S2. Extract video frames from the video data at fixed time intervals. By comparing the pixel differences between adjacent frames, detect scene changes or interface changes to extract key frames. Use a pre-trained image recognition model to analyze the important operation steps, interface changes, and text content in the key frames. Combine the user's operation event data with the video data, analyze each operation behavior and divide it into time series. Decompose the video into multiple step segments, divide the operation steps based on the operation event data, filter out unnecessary intermediate steps, and generate a detailed step list with alliance label information; The generation process of alliance tag information is as follows: A1. Randomly select a positive integer O as the group number and select a random number r∈Z q * ={1,2,...,q-1}, calculate the alliance audit code CJ=r×P; Where q is a prime number, thereby defining two cyclic groups G1 and G2 of order q, P is a generator of the cyclic group G1; based on this, a bilinear map e: G1×G1→G2 is defined; A2, for any step s i Randomly pick a number x i ∈Z q * , calculate its audit parameter PA i =x i ×P; A3. Calculate any of the following steps s i Auxiliary information generation factor: FS 1i =H1(D' i , k i , C.J., P.A. i , O), FS2=H2(O, CJ); Among them, D' i Indicates any step s i Step description information D i The binary string form of k i Indicates any step s i Step identifier; H1 represents the hash function, H1: {0, 1} * →G1; H2 represents the hash function, H2: {0, 1} ** →G1; A4. Calculate any step s i Tag information for ILable i =x i ×FS2+r×FS 1i ; A5. Generate a detailed step list Step The steps included in generating alliance tag information IJ: ; Among them, n represents the detailed step list List' Step The number of steps included in S3. Verify the completeness of the extracted detailed step list based on the alliance tag information. Then extract the steps in the detailed step list, sort all steps by timestamp, remove redundant steps, and optimize the logical structure. Then, reorganize the workflow based on the optimized step set, including splicing step descriptions, intercepting corresponding video segments, adding subtitles and visual annotations. Finally, construct standardized operation instructions and combine them with video segments to generate interactive content, output them to a document, and generate the final workflow list. The integrity verification process for the detailed steps list based on the alliance tag information verification is as follows: B1. Calculate any of the following steps s i Auxiliary check factor: FA 1i =H1(D' i , k i , C.J., P.A. i , O), FA2=H2(O, CJ); Among them, D' i Indicates any step s i Step description information D i The binary string form of k i Indicates any step s i Step identifier; H1 represents the hash function, H1: {0, 1} * →G1; H2 represents the hash function, H2: {0, 1} ** →G1; O represents any step s i Group number; CJ represents the alliance review code of group number O; PA i Indicates any step s i Audit parameter; G1 and G2 represent two cyclic groups of order q; B2. Calculate the first check factor FV1 = e(IJ, P); Where e represents a bilinear map, e:G1×G1→G2; P is a generator of the cyclic group G1; B3. Calculate the second verification factor FV2: ; B4. If FV1 = FV2, the steps of the work to be reorganized have been completely extracted, that is, the integrity check of the extracted detailed step list has passed; S4. Label the generated workflow videos and text content and create index keywords to facilitate user retrieval. Users can quickly find the corresponding business process videos or documents through keyword search for learning, and output the business process videos or documents in multiple formats, including visual video tutorials, process documents, and interactive process guides.

2. The method for constructing an AI workflow based on image recognition according to claim 1, characterized in that: The key frame extraction method is as follows: S21. Use a video processing tool to load video data and transcode the video to a fixed frame rate to evenly extract frames and reduce computational complexity: ； Where, Indicates the actual frame rate used, which is used to uniformly sample video frames; Indicates the original frame rate of the video data; Indicates the target frame rate, which is used to reduce redundant information and improve analysis efficiency; S22. For each pair of adjacent frames, calculate: ； Where D(i,j) represents the average pixel difference between the i-th frame and the j-th frame; F i (x,y) and F j (x, y) represents the grayscale value of the pixel (x, y) in the i-th frame and the grayscale value of the pixel (x, y) in the j-th frame respectively; N and M represent the width and height of the frame respectively; S23, setting pixel difference threshold T d , if the pixel difference between any frame i and its adjacent frames is greater than the set pixel difference threshold T d , that is, D(i,i+1)>T d , then mark frame i+1 as the key frame, that is, extract the key frame.

3. The method for constructing an AI workflow based on image recognition according to claim 1, characterized in that: The detailed steps list is generated as follows: S51. Read the user's operation event data and analyze the timestamp and event type of the operation event data: ； Where E represents the event sequence in the operation event data; t j Indicates the time stamp of any operation event in the operation event data; j Represents any event type in the operation event data, that is, identifies the specific operation; p i Represents any event e j Identifier of S52, according to the time interval T of the set event gap , divide the event sequence into different operation steps, if any two consecutive events t k and t k+1 The timestamps between them exceed T gap , then the event t k+1 It is a new step; S53, the event sequence E and the key frame data set F are combined according to the timestamp t j and frame timestamp t i Alignment, calculate the difference between event timestamps and keyframe timestamps , for any e j , select the keyframe closest to its timestamp As context for this event: ； Where, Indicates event e j Timestamps and keyframes The time difference between the timestamps; S54. Match the event sequence E with the key frame dataset F to find the associated time window: ； Where, represents the kth time window, i.e., the time range of a possible operation step, which contains several key frames and constitutes a video segment; and Respectively represent the start and end time of the window, which are determined by the transformation time of events and keyframes; S55. Further inspect the key frames within the time window to ensure that they contain obvious interface changes, generate a list of steps, score and optimize the importance of each step, and screen out the key steps that contribute to the workflow; S56, further mark the selected key steps in detail, generate step description information, and i Provide additional explanation and then generate a detailed step list List' Step ={s1,s2,...,s i ,..,s m }; Among them, s i Indicates any operation step. ;D i Indicates s i Step description information; and Respectively represent the operation steps i The start and end time of Indicates step s i The type of operation; Indicates an operation The interface elements involved.

4. The method for constructing an AI workflow based on image recognition according to claim 3, characterized in that: The key steps of the screening process are as follows: S61, in order to determine whether there is a significant change between two frames, calculate any frame and adjacent frames Pixel difference: ； Where, and Represent adjacent key frames respectively; N and M represent the width and height of the frame respectively; Represents the normalized value of the pixel difference between frames; Represents the L1 norm of the pixel matrix, that is, the sum of the absolute values ​​of the grayscale value differences of all pixels; S62, if >T d , it is considered that an interface change has occurred; Among them, T d Indicates the change threshold of the preset pixel; S63, according to the time window and interface changes, preliminarily classify the operation events into corresponding steps; S64. Set the event type of any operation event k Operation type weight W type (e k ) and set the operation change weight W frame (W k ): ； S65. Score each step based on the operation type and interface change weight: ； Where, represents the importance score of step k; α and β represent weight coefficients, which reflect the importance ratio of the operation event and the interface change; S66: Remove irrelevant steps based on the scoring results, i.e., remove steps with scores below the minimum scoring threshold T. score Steps, filter out the key steps, and get the step list List Step ={s1,s2,...,s i ,..,s m }; Among them, s i Indicates any operation step. , set its step identifier to k i ; and Respectively represent the operation steps i The start and end time of Indicates step s i The type of operation; Indicates an operation The interface elements involved; Indicates step s i 's rating.

5. The method for constructing an AI workflow based on image recognition according to claim 1, wherein: The process of generating a workflow list is as follows: S71. Arrange all steps in ascending order of timestamps: ； Where, Represents a set of steps after preliminary sorting; Sort() represents a sorting function; key represents the sorting criterion; s i Indicates any step; Indicates any step s i The timestamp recorded when the data was extracted; S72, determine whether the two steps are repeated steps, if >T sim , then it is considered that s j To repeat the steps, delete them; ； in, Indicates step s i and steps j similarity; Indicates step s i and steps j Similarity of operation types; Indicates step s i and steps j Similarity of interface elements; w1 and w2 represent step s i and steps j The weight coefficient reflects the priority of similarity calculation; T sim Indicates the set step similarity threshold; S73. By detecting the operation type and interface elements, determine the logical dependencies between steps, reorder the step sequence according to the dependencies, and obtain an optimized step set. , and from the optimized step set Extract the description information of each step after optimization D i 、Operation type e i and the interface elements involved , describe the information of all steps D i Sequentially stitch together a complete process description, based on the time interval of each step [ , ], extract the corresponding video segments from the original video, and splice all the segments into a complete video process in step order; S74. Add subtitles and visual annotations to each video segment. The subtitle content is taken from the text description of each step. i , the annotation information comes from the interface elements in the step , and construct any step s accordingly i Operation guide template: ,combine the operation instructions with the key frames in the video segment, generate interactive content, and output interactive instructions; S75. Compile the interactive guidance of each step into the output document in the form of a diagram to generate a workflow list: List work ={s'1,s'2,...,s' i ,...}; Among them, s' i Represents any optimized operation step, , Represents any optimized operation step s' i The corresponding video segment includes the video content of the original video segment, the added subtitles, and the visual annotations.

6. The method for constructing an AI workflow based on image recognition according to claim 1, characterized in that: The image recognition model is based on the network structure design of the YOLOv8s model, combined with multi-scale sliding window attention.

7. An AI workflow construction system based on image recognition, which is used to execute the AI ​​workflow construction method based on image recognition according to any one of claims 1 to 6, characterized in that: include: The data acquisition module is used to record the user's operation process in the business system through a screen recording tool to generate high-quality video files. It also captures the user's operation event data and system feedback information for subsequent video analysis and disassembly; The video analysis and step decomposition module extracts video frames at fixed time intervals from the video. By comparing the pixel differences between adjacent frames, it detects key frames indicating scene or interface changes. It then uses image recognition and natural language processing technologies to analyze the important content in the video. It then decomposes the video into multiple step segments, divides the operation steps based on the operation event data, and filters out unnecessary intermediate steps. The workflow reorganization module is used to automatically reorganize the video based on the disassembled segments, remove irrelevant or duplicate content, and generate videos of each workflow. At the same time, it uses computer vision, image segmentation, and object recognition technologies to extract text content from the video and generate text descriptions of the workflow, generating operational instructions corresponding to the business process. The content annotation and retrieval module is used to annotate the generated workflow videos and text content, establish index keywords, and provide a search interface for users. Users can enter keywords through the search interface to quickly find the corresponding business process videos or documents for learning, and it also supports multi-format output; Database, used to store data generated during system operation.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the method for constructing an AI workflow based on image recognition as described in any one of claims 1 to 6.

Images