AI workflow construction method and system based on image recognition and electronic equipment

Abstract

The invention relates to the technical field of workflow construction, in particular to an AI workflow construction method and system based on image recognition and electronic equipment. The method comprises the following steps: recording a working process by using a video recording tool, collecting operation data, analyzing video content and extracting key frames and operation steps by adopting image recognition and natural language processing technologies, and performing time sequence division and behavior analysis on operation behaviors in combination with user operation event data so as to determine actual operation steps. The method comprises the following steps: generating a detailed step list with alliance tag information, then combining operation steps with video segments according to an analysis result to form a structured work flow, adding detailed description and annotation information for each step, and finally establishing an index structure, so that a user can quickly retrieve a required business flow video or document through keywords. According to the method, the visualization degree and the understandability of the business process can be improved, the learning cost is reduced, and the working efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of workflow construction, and in particular to an AI workflow construction method, system and electronic device based on image recognition. Background Art

[0002] In modern business operations, optimizing and standardizing workflows are the key to improving efficiency, reducing errors, and enhancing service quality. With the rapid development of artificial intelligence (AI) technology, especially advances in computer vision, natural language processing (NLP), and machine learning, AI-based workflow construction systems have become possible. Such systems can automatically analyze existing workflows, identify key operating steps, generate detailed operating guides, and provide interactive learning materials, thereby helping companies to more effectively train employees, improve process design, and improve business execution efficiency.

[0003] The Chinese invention patent with the announcement number CN113805885B discloses a front-end construction system and method for a process management system based on a workflow engine. The system includes: a technical construction subsystem for constructing a front-end architecture, including: a workflow front-end engine for realizing the front-end process flow; a business component library for providing business components; a tool class module for providing tool classes; a data persistence module for persisting user information and business information of the system and providing a public data acquisition interface; a business construction subsystem for constructing a front-end business processing architecture, including: a system basic construction module for providing the basic functions of the front-end business processing; a workflow design module for performing workflow configuration management and business process management; an actual business design module for developing actual business functions; and a front-end construction subsystem for integrating the front-end architecture and the business processing architecture to obtain the front-end of the system. Provide a standardized front-end to reduce costs.

[0004] However, as corporate workflows continue to become more complex, the demand for employee training and process learning is increasing, and traditional learning methods, such as operation manuals or long video tutorials, are often inefficient and difficult to maintain. Summary of the invention

[0005] The purpose of the present invention is to address the problems existing in the background technology and to propose an AI workflow construction method, system and electronic device based on image recognition.

[0006] The technical solution of the present invention is a method for constructing an AI workflow based on image recognition, comprising the following specific implementation steps:

[0007] S1. Record the user's operation process in the business system through a screen recording tool to generate video data, and at the same time capture the user's operation data and system feedback information to generate operation event data;

[0008] S2. Extract video frames from the video data at fixed time intervals, detect scene changes or interface changes to extract key frames by comparing pixel differences between adjacent frames, use pre-trained image recognition models to analyze important operation steps, interface changes, and text content in key frames, combine user operation event data with video data, analyze each operation behavior and divide it into time series, break the video into multiple step segments, divide the operation steps based on the operation event data, filter out unnecessary intermediate steps, and generate a detailed step list with alliance label information;

[0009] S3. Verify the integrity of the detailed step list based on the alliance tag information, then extract the steps in the detailed step list, sort all the steps by timestamp, remove redundant steps, and optimize the logical structure. Then reorganize the workflow according to the optimized step set, including splicing step descriptions, intercepting corresponding video segments and adding subtitles and visual annotations. Finally, build standardized operation instructions and combine them with video segments to generate interactive content, output to documents, and generate the final workflow list.

[0010] S4. Annotate the generated workflow videos and text content, and create index keywords to facilitate user retrieval. Users can quickly find the corresponding business process videos or documents through keyword search for learning, and output the business process videos or documents in multiple formats, including visual video tutorials, process documents, and interactive process guides.

[0011] Preferably, the key frame extraction method is as follows:

[0012] S21. Use video processing tools to load video data and transcode the video to a fixed frame rate to evenly extract frames and reduce computational complexity:

[0013] F rate =min(F original ,F target );

[0014] In the formula, F rate Indicates the actual frame rate used, which is used to uniformly sample video frames; F original Indicates the original frame rate of the video data; F target Indicates the target frame rate, which is used to reduce redundant information and improve analysis efficiency;

[0015] S22. For each pair of adjacent frames, calculate:

[0016]

[0017] Where D(i,j) represents the average pixel difference between the i-th frame and the j-th frame; F i (x,y) and Fj (x, y) represents the grayscale value of the pixel (x, y) in the i-th frame and the grayscale value of the pixel (x, y) in the j-th frame; N and M represent the width and height of the frame respectively;

[0018] S23, setting pixel difference threshold T d , if the pixel difference between any frame i and its adjacent frames is greater than the set pixel difference threshold T d , that is, D(i,i+1)>T d , then mark frame i+1 as a key frame, that is, extract the key frame.

[0019] Preferably, the generation process of the alliance tag information is as follows:

[0020] S31, randomly select a positive integer O as the group number, select a random number r∈Z q * ={1,2,...,q-1}, calculate the alliance audit code CJ=r×P;

[0021] Where q is a prime number, thereby defining two cyclic groups G1 and G2 of order q, and P is a generator of the cyclic group G1; based on this, a bilinear map e: G1×G1→G2 is defined;

[0022] S32, for any step s i Randomly pick a number x i ∈Z q * , calculate its audit parameter PA i =x i ×P;

[0023] S33, calculate any of the following steps s i Auxiliary information generation factor: FS 1i =H1(D' i , k i , C.J., P.A. i , O), FS2 = H2 (O, CJ);

[0024] Among them, D' i Indicates any step s i Step description information D i The binary string form of k i Indicates any step s i Step identifier; H1 represents the hash function, H1: {0, 1} * →G1; H2 represents the hash function, H2: {0, 1} ** →G1;

[0025] S34. Calculate any step s i Tag information for ILablei =x i ×FS2+r×FS 1i ;

[0026] S35, generate detailed step list List' Step The steps included in generating alliance tag information IJ:

[0027] Among them, n represents the detailed step list List' Step The number of steps contained in .

[0028] Preferably, the integrity verification process of the detailed step list of the detailed step list extracted based on the alliance tag information verification is as follows:

[0029] S41. Calculate any of the following steps s i Auxiliary check factor: FA 1i =H1(D' i , k i , C.J., P.A. i , O), FA2 = H2 (O, CJ);

[0030] Among them, D' i Indicates any step s i Step description information D i The binary string form of k i Indicates any step s i Step identifier; H1 represents the hash function, H1: {0, 1} * →G1; H2 represents the hash function, H2: {0, 1} ** →G1; O represents any step s i The group number; CJ represents the alliance audit code of group number O; PA i Indicates any step s i Audit parameter; G1 and G2 represent two cyclic groups of order prime number q respectively;

[0031] S42, calculate the first check factor FV1=e(IJ, P);

[0032] Where, e represents a bilinear map, e: G1×G1→G2; P is a generator of the cyclic group G1;

[0033] S43, calculate the second verification factor FV2:

[0034] S44. If FV1=FV2, the steps of the work to be reorganized have been completely extracted, that is, the integrity check of the extracted detailed step list has passed.

[0035] Preferably, the detailed steps list is generated as follows:

[0036] S51. Read the user's operation event data, and analyze the timestamp and event type of the operation event data:

[0037] E={[t1,e1,p1],[t2,e2,p2],...,[t j ,e j ,p j ],...,[t n ,e n ,p n ]};

[0038] Where E represents the event sequence in the operation event data; t j Indicates the time stamp of any operation event in the operation event data; j Represents any event type in the operation event data, that is, identifies the specific operation; p i Represents any event e j Identifier of

[0039] S52, according to the time interval T of the set event gap , divide the event sequence into different operation steps, if any two consecutive events t k and t k+1 The timestamps between them exceed T gap , then the event t k+1 It is a new step;

[0040] S53, the event sequence E and the key frame data set F are combined according to the timestamp t j and frame timestamp t i Alignment, calculate the difference Δt between the event timestamp and the keyframe timestamp i,j , for any e j , select the key frame f that is closest to its timestamp i As context for this event:

[0041] Δt i,j =|t i -t j |;

[0042] In the formula, Δt i,j Indicates event e j The timestamp and key frame f i The time difference between the timestamps of

[0043] S54, match the event sequence E with the key frame data set F to find the associated time window:

[0044] W k =[tstart ,t end ];

[0045] Where W k represents the kth time window, i.e., the time range of a possible operation step, which contains several key frames and constitutes a video segment; t start With t end Respectively represent the start and end time of the window, which is determined by the transformation time of events and keyframes;

[0046] S55. Further detect the key frames in the time window to ensure that they contain obvious interface changes, generate a step list, score and optimize the importance of each step, and screen out the key steps that contribute to the workflow;

[0047] S56, further annotate the selected key steps in detail, generate step description information, and i Provide additional explanation and then generate a detailed step list List' Step ={s1,s2,...,s i ,..,s m};

[0048] Among them, s i Indicates any operation step, s i ={t start,i ,t end,i ,e i ,Elements i ,Scoer(i),D i}; D i Indicates i Step description information; start,i and t end,i Respectively represent the operation steps i The start and end time of i Indicates step s i Operation type; Elements i Indicates operation e i The interface elements involved.

[0049] Preferably, the screening process of the key steps is as follows:

[0050] S61, in order to determine whether there is a significant change between two frames, calculate any frame f i and adjacent frames f i+1 Pixel difference:

[0051]

[0052] In the formula, f i With fi+1 represent adjacent key frames respectively; N and M represent the width and height of the frame respectively; D(f i ,f i+1 ) represents the normalized value of the pixel difference between frames; ||f i -f i+1 ||1 represents the L1 norm of the pixel matrix, that is, the sum of the absolute values ​​of the grayscale value differences of all pixels;

[0053] S62, if D(f i ,f i+1 )>T d , it is considered that an interface change has occurred;

[0054] Among them, T d Indicates the change threshold of the preset pixel;

[0055] S63, according to the time window W k and interface changes, preliminarily classify the operation events into corresponding steps;

[0056] S64. Set the event type of any operation event k The operation type weight W type (e k ) and set the operation change weight W frame (W k ):

[0057]

[0058] S65. Score each step based on the operation type and interface change weight:

[0059] Score(k)=α×W type (e k )+β×W frame (W k );

[0060] In the formula, Score(k) represents the importance score of step k; α and β represent weight coefficients, which reflect the importance ratio of the operation event and the interface change.

[0061] S66: Remove irrelevant steps based on the scoring results, that is, remove steps with scores lower than the minimum scoring threshold T. score Steps, filter out the key steps, and get the step list List Step ={s1,s2,...,s i ,..,s m};

[0062] Among them, s i Indicates any operation step, s i ={t start,i,t end,i ,e i ,Elements i , Scoer(i)}, set its step identifier to k i ;t start,i and t end,i Respectively represent the operation steps i The start and end time of i Indicates step s i Operation type: Elements i Indicates operation e i Involved interface elements; Scoer(i) represents step s i 's rating.

[0063] Preferably, the process of generating the workflow list is as follows:

[0064] S71. Arrange all steps in ascending order according to timestamps:

[0065] S ordered =Sort({s1,s2,...,s i ,...,s m},key=t si );

[0066] In the formula, S ordered Represents a set of steps after preliminary sorting; Sort() represents a sorting function; key represents a sorting criterion; s i Indicates any step; t si Indicates any step s i The timestamp recorded when the data was extracted;

[0067] S72, determine whether the two steps are repeated steps, if sim(s i ,s j )>T sim , then we think that s j To repeat the steps, delete them;

[0068] sim(s i ,s j )=w1·sim e (s i ,s j )+w2·sim Elements (s i ,s j );

[0069] Among them, sim(s i ,s j ) indicates step s i and steps jSimilarity; sim e (s i ,s j ) indicates step s i and steps j The similarity of the operation type, including but not limited to clicking the same button twice; Elements (s i ,s j ) indicates step s i and steps j The similarity of the interface elements, including but not limited to the operation design of the same interface elements; w1 and w2 represent step s i and steps j The weight coefficient reflects the priority of similarity calculation; T sim Indicates the set step similarity threshold;

[0070] S73, by detecting the operation type and interface elements, determine the logical dependencies between the steps, reorder the step sequence according to the dependencies, and obtain the optimized step set S' ordered , and from the optimized step set S' ordered Extract the description information of each step after optimization D i 、Operation type e i and the interface elements involved i , the description information of all steps D i Sequentially stitch together to form a complete text description of the process, according to the time interval of each step [t start,i , t end,i ], extract the corresponding video segments from the original video, and splice all the segments into a complete video process in step order;

[0071] S74. Add subtitles and visual annotations to each video segment. The subtitle content is taken from the text description of each step. i , the annotation information comes from the interface elements in the step i , and construct any step s accordingly i Instruction template: i ={s i ,e i ,Elements i ,D i}, combining the operation instructions with the key frames in the video segment, generating interactive content, and outputting interactive instructions;

[0072] S75. The interactive instructions for each step are compiled into the output document in the form of diagrams to generate a workflow list. work ={s'1,s'2,...,s'i ,...};

[0073] Among them, s' i represents any optimized operation step, Represents any optimized operation step s' i The corresponding video segment includes the video content of the original video segment, the added subtitles and the visual annotations.

[0074] Preferably, the image recognition model is designed based on the network structure of the YOLOv8s model, combined with multi-scale sliding window attention.

[0075] The technical solution of the present invention is: an AI workflow construction system based on image recognition, which is used to execute the above-mentioned AI workflow construction method based on image recognition, including:

[0076] The data collection module is used to record the user's operation process in the business system through a screen recording tool to generate high-quality video files, while capturing the user's operation event data and system feedback information for subsequent video analysis and disassembly;

[0077] The video analysis and step decomposition module is used to extract video frames from the video at fixed time intervals, detect key frames of scene changes or interface changes by comparing the pixel differences between adjacent frames, and use image recognition and natural language processing technology to analyze the important content in the video. After that, the video is decomposed into multiple step segments, and the operation steps are divided based on the operation event data, filtering out unnecessary intermediate steps;

[0078] The workflow reorganization module is used to automatically reorganize the video according to the disassembled fragments, remove irrelevant or repeated content, and generate each workflow video. At the same time, it uses computer vision, image segmentation and object recognition technology to extract the text content in the video and generate a text description of the workflow, and generate operating instructions corresponding to the business process;

[0079] Content annotation and retrieval module, which is used to annotate the generated workflow videos and text content, establish index keywords, and provide a retrieval interface for users. Users can enter keywords to search through the retrieval interface to quickly find the corresponding business process videos or documents for learning, and it also supports multi-format output;

[0080] Database is used to store data generated during system operation.

[0081] The technical solution of the present invention is: an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for constructing an AI workflow based on image recognition when executing the computer program.

[0082] Compared with the prior art, the above technical solution of the present invention has the following beneficial technical effects:

[0083] The present invention proposes an AI workflow construction method, system and electronic device based on image recognition. By recording the customer's workflow video and using artificial intelligence technology to analyze, disassemble and reorganize the video, it can efficiently generate easy-to-learn process steps and provide support for rapid retrieval and automated learning. This method not only improves the efficiency of workflow learning, but also enables employees to easily master complex workflows:

[0084] (1) Automated process generation: The present invention can automatically record, analyze and generate video and text content of the workflow, greatly reducing the time and cost of manually writing process documents;

[0085] (2) Process optimization: Through video disassembly and reorganization, irrelevant steps are removed, making the learning content more concise and clear, and users can quickly grasp the core process;

[0086] (3) Intelligent retrieval: The system establishes detailed indexes and annotations for the generated learning content. Users can quickly find relevant processes through simple keyword searches, which improves learning efficiency.

[0087] (4) Multiple learning methods: The present invention supports process display in multiple forms such as video, text, and charts, and users can choose the most suitable learning method according to their needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 A system flow chart of an AI workflow construction system based on image recognition proposed by the present invention;

[0089] Figure 2 This is a principle block diagram of an AI workflow construction method based on image recognition proposed in the present invention. DETAILED DESCRIPTION

[0090] Embodiment 1, as Figure 1 As shown, the present invention proposes an AI workflow construction system based on image recognition, including: a data acquisition module, a video analysis and step decomposition module, a workflow reorganization module, a content annotation and retrieval module and a database.

[0091] The data acquisition module is used to record the user's operation process in the business system through a screen recording tool to generate high-quality video files, and at the same time capture the user's operation event data (including but not limited to keyboard input, mouse clicks) and system feedback (including but not limited to window changes, pop-up boxes) information for subsequent video analysis and disassembly;

[0092] The video analysis and step decomposition module is used to extract video frames from the video at fixed time intervals, detect key frames of scene changes or interface changes by comparing the pixel differences between adjacent frames for subsequent analysis, and use image recognition and natural language processing technology to analyze important content in the video, including but not limited to operation steps, interface changes and text content, and then decompose the video into multiple step segments, divide the operation steps based on the operation event data, and filter out unnecessary intermediate steps (including but not limited to waiting time and loading pages);

[0093] The workflow reorganization module is used to automatically reorganize the video according to the disassembled fragments, remove irrelevant or repeated content, generate each workflow video, and use computer vision, image segmentation and object recognition technology to extract the text content in the video and generate a text description of the workflow, and generate operation instructions corresponding to the business process, including but not limited to specific operation steps, charts or screenshots;

[0094] The content annotation and retrieval module is used to annotate the generated workflow videos and text content, establish keywords, and facilitate user retrieval. It also provides a retrieval interface for users, through which users can quickly find the corresponding business process videos or documents for learning by entering keywords. It also supports multi-format output, including but not limited to visual video tutorials, process documents, and interactive process guides, so that users can use them according to their needs.

[0095] The database is used to store data generated during system operation.

[0096] In the second embodiment, the present invention proposes an AI workflow construction method based on image recognition, which is applied to the AI ​​workflow construction system based on image recognition proposed in the first embodiment, such as Figure 2 As shown, the specific implementation steps include the following:

[0097] S1. The data acquisition module records the customer's workflow through a video recording tool to generate high-quality video data. On the other hand, it captures the customer's operation data (including but not limited to keyboard input, mouse clicks) and system feedback information (including but not limited to window changes, pop-up boxes), and performs structured processing on the operation data and system feedback information to generate operation event data, and then stores the video data and operation event data in the database;

[0098] S2, the video analysis and step disassembly module extracts video data and operation event data from the database and implements the following operation process:

[0099] S21, extracting video frames from the extracted video data at fixed time intervals, and detecting key frames of scene changes or interface changes by comparing pixel differences between adjacent frames. The specific implementation process is as follows:

[0100] S2101. Use a video processing tool to load video data and transcode the video to a fixed frame rate to evenly extract frames and reduce computational complexity:

[0101] F rate =min(F original ,F target );

[0102] In the formula, F rate Indicates the actual frame rate used, which is used to uniformly sample video frames; F original Indicates the original frame rate of the video data; F target Indicates the target frame rate, which is used to reduce redundant information and improve analysis efficiency;

[0103] S2102. For each pair of adjacent frames, calculate:

[0104]

[0105] Where D(i,j) represents the average pixel difference between the i-th frame and the j-th frame; F i (x,y) and F j (x, y) represents the grayscale value of the pixel (x, y) in the i-th frame and the grayscale value of the pixel (x, y) in the j-th frame; N and M represent the width and height of the frame respectively;

[0106] S2103, setting pixel difference threshold T d , if the pixel difference between any frame i and its adjacent frames is greater than the set pixel difference threshold T d , that is, D(i,i+1)>T d , then mark frame i+1 as a key frame;

[0107] S2104. Save the timestamps {t1, t2, ...} corresponding to the key frames and construct a key frame data set:

[0108] F={[t1,f1],[t2,f2],...,[t i ,f i ],...,[t n ,f n ]};

[0109] Where, F represents the key frame dataset; f i represents any key frame; t i Denotes any key frame f i The corresponding timestamp;

[0110] S22. Use image recognition and natural language processing technology to analyze important operation steps, interface changes, and text content in the video. The specific implementation process is as follows:

[0111] S2201, extract any key frame image f i Grayscale processing or scaling processing is performed to adapt to the input of the image recognition model to obtain the preprocessed key frame image f i ;

[0112] S2202, the processed key frame image f i Input into the pre-trained image recognition model, perform target detection on the keyframe, and output the confidence, category label and target position coordinates of the interface elements in the keyframe;

[0113] It should be noted that the image recognition model is based on the network structure design of the YOLOv8s model, combined with a multi-scale sliding window attention (Multi-Scale Sliding WindowAttention), using a multi-scale convolutional neural network to extract features in key frames, understand the details and global information of key frames, and improve recognition accuracy. The input key frame is divided into multiple small blocks, and an attention weight is calculated for each small block. These weights represent the importance of each small block to the entire key frame, that is, the image recognition model pays attention to the parts of the key frame that are most relevant to the interface elements and assigns higher weights to these parts. The weighted feature vector is obtained by multiplying the attention weight with the feature vector of each small block. These weighted feature vectors are then used to detect interface elements.

[0114] S2203, using OCR (Optical Character Recognition) technology to recognize the output of the image recognition model, and recognize text information in the image frame;

[0115] S2204, output the information group C (f i ):

[0116] C(f i )={Confidence i ,Label i ,Loc i ,text i};

[0117] In the formula, C(f i ) represents any key frame f i Information group of interface elements; Confidence i Denotes any key frame f iConfidence of interface elements in Lable i Denotes any key frame f i Category label of the interface element in Loc i Denotes any key frame f i Text information in ;

[0118] When the confidence level is greater than the set confidence threshold (T C ), it is considered that there are interface elements in the key frame;

[0119] S23. Combine the user's operation event data and video data to analyze each operation behavior and divide it into time series. The specific implementation process is as follows:

[0120] S2301. Read the extracted user's operation event data, and analyze the timestamp and event type of the operation event data:

[0121] E={[t1,e1,p1],[t2,e2,p2],...,[t j ,e j ,p j ],...,[t n ,e n ,p n ]};

[0122] Where E represents the event sequence in the operation event data; t j Indicates the time stamp of any operation event in the operation event data; j Represents any event type in the operation event data, that is, identifies a specific operation, including but not limited to "clicking a button" and "entering text"; p i Represents any event e j Identifiers, including but not limited to:

[0123] Click operation: mouse click;

[0124] Text input: keyboard input;

[0125] Form submission: click the submit button;

[0126] Drag operation: mouse drag;

[0127] S2302, according to the time interval T of the set event gap , divide the event sequence into different operation steps. If any two consecutive events t k and t k+1 The timestamps between them exceed T gap , then the event t k+1 It is a new step;

[0128] S2303, each operation event has a timestamp, and the key frame extraction of the video frame is also accompanied by a timestamp, and the event sequence E and the key frame data set F are combined according to the timestamp t j and frame timestamp t i Alignment, calculate the difference Δt between the event timestamp and the keyframe timestamp i,j , for any e j , select the key frame f that is closest to its timestamp i As the context of this event, minΔt i,j , to ensure that the key frame f when the event occurs i Can be used as the event j The context:

[0129] Δt i,j =|t i -t j |;

[0130] In the formula, Δt i,j Indicates event e j The timestamp and key frame f i The time difference between the timestamps of

[0131] S24, after completing the event sequence E and the key frame data set F according to the timestamp t j and frame timestamp t i After alignment, the video analysis and step decomposition module will extract the actual operation steps and add annotation information for each step. The specific implementation process is as follows:

[0132] S2401, match the event sequence E with the key frame data set F to find the associated time window:

[0133] W k =[t start ,t end ];

[0134] Where W k represents the kth time window, i.e., the time range of a possible operation step, which contains several key frames and constitutes a video segment; t start With t end Respectively represent the start and end time of the window, which is determined by the transformation time of events and keyframes;

[0135] S2402: further detect the key frames in the time window to ensure that they contain obvious interface changes (including but not limited to jumping to a new page after a button is pressed). The specific implementation process is as follows:

[0136] (1) In order to determine whether there is a significant change between two frames, calculate any frame f i and adjacent frames fi+1 Pixel difference:

[0137]

[0138] In the formula, f i With f i+1 represent adjacent key frames respectively; N and M represent the width and height of the frame respectively; D(f i ,f i+1 ) represents the normalized value of the pixel difference between frames; ||f i -f i+1 ||1 represents the L1 norm of the pixel matrix, that is, the sum of the absolute values ​​of the grayscale value differences of all pixels;

[0139] (2) If D(f i ,f i+1 )>T d (preset pixel change threshold), it is considered that an interface change has occurred, and the subsequent step S2403 is continued;

[0140] (3) According to the time window W k and interface changes, preliminarily classify the operation events into corresponding steps;

[0141] S2403. After initially extracting the steps, each step is scored and optimized for importance, and the key steps that contribute to the workflow are screened out. The specific implementation process is as follows:

[0142] (1) Different types of operation events have different importance. For example, "submitting a form" is more important than "mouse hovering". Set the event type of any operation event to k The operation type weight W type (e k );

[0143] It should be noted that W type (e k ) is determined by prior knowledge;

[0144] (2) The greater the interface change, the more important the step is, and the weight of the operation change is set as W. frame (W k ):

[0145]

[0146] (3) Score each step based on the operation type and interface change weight:

[0147] Score(k)=α×W type (e k )+β×W frame (W k );

[0148] In the formula, Score(k) represents the importance score of step k; α and β represent weight coefficients, which reflect the importance ratio of the operation event and the interface change.

[0149] (4) Remove irrelevant steps (those with scores lower than the minimum score threshold T) based on the scoring results. score Steps), including but not limited to waiting for page loading and idle time without operation, to filter out key steps;

[0150] (5) Get the step list List Step ={s1,s2,...,s i ,..,s m};

[0151] Among them, s i Indicates any operation step, s i ={t start,i ,t end,i ,e i ,Elements i , Scoer(i)}, set its step identifier to k i ;

[0152] Where, t start,i and t end,i Respectively represent the operation steps i The start and end time of i Indicates step s i The type of operation, including but not limited to mouse click or text input; Elements i Indicates operation e i Involved interface elements; Scoer(i) represents step s i Ratings;

[0153] S2404. Further annotate the selected key steps in detail to generate step description information. The process of generating the step description information is as follows:

[0154] (1) Extract video data, use the pre-trained video understanding model to classify the video scenes, identify different operation contexts, and output video content understanding, including but not limited to: form filling scenes, button clicking scenes, or data uploading scenes;

[0155] (2) Based on the operation event data and combined with the video content understanding, the semantic information of the event is supplemented;

[0156] For example: when a "mouse click" is detected, it is determined whether it is "clicking the login button" or "clicking the download link" based on the classification results; when text input is recognized, it is supplemented with "enter user name" or "enter search keywords" based on the field title detected by the interface;

[0157] (3) Generate intelligent prompts by combining interface text extracted by OCR and objects detected by vision;

[0158] For example: in a form scenario, the prompt is "Possible action is to fill in the field"; in a confirmation page scenario, the prompt is "Possible action is to submit the form";

[0159] (4) Based on intelligent prompts, accurately label the steps:

[0160] Based on the initially extracted operation events, the operation type is adjusted using the results of video content understanding, and the detection results of video understanding and the information group C(f i ) and operation events, annotate the interface elements and their locations of each key step, and use intelligent prompts to supplement the actions and content. Combine scene classification and visual-text analysis results to generate natural language descriptions to obtain the s i Step description information D i ;

[0161] It should be noted that the video understanding model is built on the basis of the SlowFast model. The SlowFast model is an efficient video content element recognition model. It processes the slow features and fast features of the video respectively through two branches: the Slow branch (used to process low frame rate videos to capture slow features and long-term context information) and the Fast branch (used to process high frame rate videos to capture fast-changing details), so as to better capture the dynamic changes in the video.

[0162] S2405, for any step s i Provide additional explanation and then generate a detailed step list List' Step :

[0163] List' Step ={s1,s2,...,s i ,..,s m};

[0164] Among them, s i Indicates any operation step, s i ={t start,i ,t end,i ,e i ,Elements i ,Scoer(i),D i};

[0165] Where D i Indicates i Step description information;

[0166] S2406, List' for detailed steps Step The steps included in the above code generate alliance tag information. The generation process is as follows:

[0167] (1) Randomly select a positive integer O as the group number and select a random number r∈Z q * ={1,2,...,q-1}, calculate the alliance audit code CJ=r×P;

[0168] Wherein, q is a prime number, thereby defining two cyclic groups G1 and G2 of order q, and P is a generator of the cyclic group G1; accordingly, a bilinear mapping e: G1×G1→G2 is defined and transmitted to the workflow reorganization module;

[0169] (2) for any step s i Randomly pick a number x i ∈Z q * , calculate its audit parameter PA i =x i ×P;

[0170] (3) Calculate any of the following steps s i Auxiliary information generation factor: FS 1i =H1(D' i , k i , C.J., P.A. i , O), FS2 = H2 (O, CJ);

[0171] Among them, D' i Indicates any step s i Step description information D i The binary string form of k i Indicates any step s i Step identifier; H1 represents the hash function, H1: {0, 1} * →G1; H2 represents the hash function, H2: {0, 1} ** →G1;

[0172] (4) Calculate any step s i Tag information for ILable i =x i ×FS2+r×FS 1i ;

[0173] (5) Generate a detailed step list List'Step The steps included in generating alliance tag information IJ:

[0174] Among them, n represents the detailed step list List' Step The number of steps contained in

[0175] S2407, {O, CJ, List' Step , IJ} and {k i , P.A. i ,s i}Store in a structured format in a database;

[0176] S3, the workflow reorganization module combines the video segments and the generated workflows according to the workflow steps obtained through analysis and disassembly, generates videos of each workflow, and uses computer vision, image segmentation, and object recognition technology to extract the text content in the video and generate a text description of the workflow, and disassembles the operation of each step to generate the operation instructions corresponding to the workflow. The specific implementation steps are as follows:

[0177] S31. Extract {O, CJ, List' from the database Step , IJ} and {k i , P.A. i ,s i}, verify whether the steps of the work to be reorganized are completely extracted, and the verification process is as follows:

[0178] (1) Calculate any of the following steps s i Auxiliary check factor: FA 1i =H1(D' i , k i , C.J., P.A. i , O), FA2 = H2 (O, CJ);

[0179] Among them, D' i Indicates any step s i Step description information D i The binary string form of k i Indicates any step s i Step identifier; H1 represents the hash function, H1: {0, 1} * →G1; H2 represents the hash function, H2: {0, 1} ** →G1; O represents any step s i The group number; CJ represents the alliance audit code of group number O; PA i Indicates any step s i Audit parameter; G1 and G2 represent two cyclic groups of order prime number q respectively;

[0180] (2) Calculate the first check factor FV1 = e(IJ, P);

[0181] Where, e represents a bilinear map, e: G1×G1→G2; P is a generator of the cyclic group G1;

[0182] (3) Calculate the second check factor FV2:

[0183] (4) If FV1 = FV2, the steps of the required reorganization work have been completely extracted;

[0184] S32. Arrange all steps in ascending order according to timestamps:

[0185]

[0186] In the formula, S ordered Represents a set of steps after preliminary sorting; Sort() represents a sorting function; key represents a sorting criterion; s i Indicates any step; Indicates any step s i The timestamp recorded when the data was extracted;

[0187] S33, the set of steps S after preliminary sorting ordered In the process, some steps may be redundant. To remove the redundant steps, the specific implementation process is as follows:

[0188] S3301, determine whether two steps are repeated steps:

[0189] sim(s i ,s j )=w1·sim e (s i ,s j )+w2·sim Elements (s i ,s j );

[0190] In the formula, sim(s i ,s j ) indicates step s i and steps j Similarity; sim e (s i ,s j ) indicates step s i and steps j The similarity of the operation type, including but not limited to clicking the same button twice; Elements (s i ,s j ) indicates step si and steps j The similarity of the interface elements, including but not limited to the operation design of the same interface elements; w1 and w2 represent step s i and steps j The weight coefficient reflects the priority of similarity calculation;

[0191] If sim(s i ,s j )>T sim (set step similarity threshold), then it is considered that s j To repeat the steps, delete them;

[0192] S3302. Optimize the logical structure between steps to form a coherent operation process, thereby avoiding the problem of incomplete logic or disordered sequence in the steps after cleaning. The specific implementation process is as follows:

[0193] (1) By detecting the operation type and interface elements, the logical dependencies between steps are determined. The dependencies are analyzed and determined based on the following conditions:

[0194] Time relationship: t end,i <t start,j ;

[0195] Among them, t end,i Indicates step s i End time of start,j Indicates step s j The start time of

[0196] Operation type: including but not limited to "opening a page" relying on "login";

[0197] Interface elements: including but not limited to the buttons clicked belong to the preloaded pages;

[0198] (2) Reorder the step sequence according to the dependency relationship to obtain the optimized step set S' ordered ;

[0199] S34, according to the optimized step set S' ordered , reorganize the steps according to the operation type and interface elements, and combine them into a workflow. The specific implementation process is as follows:

[0200] S3401, from the optimized step set S' ordered Extract the description information of each step after optimization D i 、Operation type e i and the interface elements involved i , the description information of all steps D i Splice them together in order to form a complete text description of the process;

[0201] S3402, according to the time interval of each step [t start,i , t end,i ], extract the corresponding video segments from the original video, and splice all the segments into a complete video process in step order;

[0202] S3403, add subtitles to each video segment (step description information D i ) and visual annotations (including but not limited to click area highlights and operation instructions), and the subtitles are taken from the text description of each step. i , the annotation information comes from the interface elements in the step i ;

[0203] S35, operation type based on each step i , Interface elements involved i and step description information D i , build standardized operation guidelines, the specific implementation steps are as follows:

[0204] S3501, construct any step s i Operation guide template:

[0205] Instruction i ={s i ,e i ,Elements i ,D i};

[0206] For example: {step: "Step 2";

[0207] Action type: "Click";

[0208] Interface element: "Savebutton";

[0209] Directions: "Click on the 'Save' button in the top-right corner to save the current progress;"};

[0210] S3502, combining the operation guide with the key frames in the video segment to generate interactive content and output the interactive guide;

[0211] S3503, compiling the interactive guidance of each step into the output document in the form of a diagram;

[0212] S36. Generate workflow list List work ={s'1,s'2,...,s'i ,...};

[0213] Among them, s' i represents any optimized operation step,

[0214] In the formula, Represents any optimized operation step s' i A corresponding video segment, which includes the video content of the original video segment, the added subtitles and the visual annotations;

[0215] S4. Content annotation and retrieval module receives workflow list List work , and annotate the generated workflow videos and text content, create indexes and metadata tags, so that users can quickly find the corresponding business process videos or documents through keyword searches for learning. The specific implementation process is as follows:

[0216] S41, based on image recognition technology, OCR technology and natural language processing (NLP) algorithm, automatically perform preliminary annotation on videos, texts and interface elements:

[0217] Add tags to each video clip to indicate the operation steps, operation types, and interface elements involved in the video;

[0218] Annotate the text description of each operation step and generate keywords and tags to help with retrieval and indexing;

[0219] Label the interface elements in the operation steps, including but not limited to: element type, location, and function information;

[0220] S42. By structuring content annotation and metadata, an index structure is constructed to facilitate users to quickly search for required workflow steps, video clips and operation instructions by keywords. The specific implementation process is as follows:

[0221] S4201, converting the annotated content into an index structure so that each annotated information can be quickly searched by a specific keyword, and constructing index keywords:

[0222] In the formula, keyword i Indicates step s i Keywords:

[0223] Users can enter index keywords based on Query the relevant steps, and then follow Obtain step description information, video clips, and operation instructions. The system quickly returns all workflow content containing the keyword based on the inverted index.

[0224] For example, if a user enters "save" as a query keyword, the system returns all steps, video clips, and text descriptions related to "save";

[0225] S43. Output workflows in a variety of formats, including but not limited to text, video, and interactive content:

[0226] Text process documentation: Generate structured documents (including but not limited to Word or PDF) containing step-by-step instructions, relevant screenshots, and interactive guides for users to refer to offline;

[0227] Video tutorials: Export video segments of each step of the visualization, with subtitles and visualization annotations for intuitive learning and demonstration;

[0228] Interactive display: Generates web-based interactive instructions so that users can experience the operation process directly in the browser.

[0229] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A method for constructing an AI workflow based on image recognition, characterized in that: The specific implementation steps include the following: S1. Record the user's operation process in the business system through a screen recording tool to generate video data, and at the same time capture the user's operation data and system feedback information to generate operation event data; S2. Extract video frames from the video data at fixed time intervals, detect scene changes or interface changes to extract key frames by comparing pixel differences between adjacent frames, use pre-trained image recognition models to analyze important operation steps, interface changes, and text content in key frames, combine user operation event data with video data, analyze each operation behavior and divide it into time series, break the video into multiple step segments, divide the operation steps based on the operation event data, filter out unnecessary intermediate steps, and generate a detailed step list with alliance label information; S3. Verify the integrity of the detailed step list based on the alliance tag information, then extract the steps in the detailed step list, sort all the steps by timestamp, remove redundant steps, and optimize the logical structure. Then reorganize the workflow according to the optimized step set, including splicing step descriptions, intercepting corresponding video segments and adding subtitles and visual annotations. Finally, build standardized operation instructions and combine them with video segments to generate interactive content, output to documents, and generate the final workflow list. S4. Annotate the generated workflow videos and text content, and create index keywords to facilitate user retrieval. Users can quickly find the corresponding business process videos or documents through keyword search for learning, and output the business process videos or documents in multiple formats, including visual video tutorials, process documents, and interactive process guides.

2. The method for constructing an AI workflow based on image recognition according to claim 1, characterized in that: The key frame extraction method is as follows: S21. Use video processing tools to load video data and transcode the video to a fixed frame rate to evenly extract frames and reduce computational complexity: F rate =min(F original ,F target ); In the formula, F rate Indicates the actual frame rate used, which is used to uniformly sample video frames; F original Indicates the original frame rate of the video data; F target Indicates the target frame rate, which is used to reduce redundant information and improve analysis efficiency; S22. For each pair of adjacent frames, calculate: Where D(i,j) represents the average pixel difference between the i-th frame and the j-th frame; F i (x,y) and F j (x, y) represents the grayscale value of the pixel (x, y) in the i-th frame and the grayscale value of the pixel (x, y) in the j-th frame; N and M represent the width and height of the frame respectively; S23, setting pixel difference threshold T d , if the pixel difference between any frame i and its adjacent frames is greater than the set pixel difference threshold T d , that is, D(i,i+1)>T d , then mark frame i+1 as a key frame, that is, extract the key frame.

3. The method for constructing an AI workflow based on image recognition according to claim 1, characterized in that: The generation process of alliance tag information is as follows: S31, randomly select a positive integer O as the group number, select a random number r∈Z q * ={1,2,...,q-1}, calculate the alliance audit code CJ=r×P; Where q is a prime number, thereby defining two cyclic groups G1 and G2 of order q, and P is a generator of the cyclic group G1; based on this, a bilinear map e: G1×G1→G2 is defined; S32, for any step s i Randomly pick a number x i ∈Z q * , calculate its audit parameter PA i =x i ×P; S33, calculate any of the following steps s i Auxiliary information generation factor: FS 1i =H1(D' i , k i , C.J., P.A. i , O), FS2 = H2 (O, CJ); Among them, D' i Indicates any step s i Step description information D i The binary string form of k i Indicates any step s i Step identifier; H1 represents the hash function, H1: {0, 1} * →G1; H2 represents the hash function, H2: {0, 1} ** →G1; S34. Calculate any step s i Tag information for ILable i =x i ×FS2+r×FS 1i ; S35, generate detailed step list List' Step The steps included in generating alliance tag information IJ: Among them, n represents the detailed step list List' Step The number of steps contained in .

4. The method for constructing an AI workflow based on image recognition according to claim 1, characterized in that: The integrity verification process of the detailed step list based on the alliance tag information verification extraction detailed step list is as follows: S41. Calculate any of the following steps s i Auxiliary check factor: FA 1i =H1(D' i , k i , C.J., P.A. i , O), FA2 = H2 (O, CJ); Among them, D' i Indicates any step s i Step description information D i The binary string form of k i Indicates any step s i Step identifier; H1 represents the hash function, H1: {0, 1} * →G1; H2 represents the hash function, H2: {0, 1} ** →G1; O represents any step s i The group number; CJ represents the alliance audit code of group number O; PA i Indicates any step s i Audit parameter; G1 and G2 represent two cyclic groups of order prime number q respectively; S42, calculate the first check factor FV1=e(IJ, P); Where, e represents a bilinear map, e: G1×G1→G2; P is a generator of the cyclic group G1; S43, calculate the second verification factor FV2: S44. If FV1=FV2, the steps of the work to be reorganized have been completely extracted, that is, the integrity check of the extracted detailed step list has passed.

5. The method for constructing an AI workflow based on image recognition according to claim 1, characterized in that: The detailed step list is generated as follows: S51. Read the user's operation event data, and analyze the timestamp and event type of the operation event data: E={[t1,e1,p1],[t2,e2,p2],…,[t j ,and j ,p j ],…,[t n ,and n ,p n ]}; Where E represents the event sequence in the operation event data; t j Indicates the time stamp of any operation event in the operation event data; j Represents any event type in the operation event data, that is, identifies the specific operation; p i Represents any event e j Identifier of S52, according to the time interval T of the set event gap , divide the event sequence into different operation steps. If any two consecutive events t k and t k+1 The timestamps between them exceed T gap , then the event t k+1 It is a new step; S53, the event sequence E and the key frame data set F are combined according to the timestamp t j and frame timestamp t i Alignment, calculate the difference Δt between the event timestamp and the keyframe timestamp i,j , for any e j , select the key frame f that is closest to its timestamp i As context for this event: Δt i,j =|t i -t j |; In the formula, Δt i,j Indicates event e j The timestamp and key frame f i The time difference between the timestamps of S54, match the event sequence E with the key frame data set F to find the associated time window: W k =[t start ,t end ]; Where W k represents the kth time window, i.e., the time range of a possible operation step, which contains several key frames and constitutes a video segment; t start With t end Respectively represent the start and end time of the window, which is determined by the transformation time of events and keyframes; S55. Further detect the key frames in the time window to ensure that they contain obvious interface changes, generate a step list, score and optimize the importance of each step, and screen out the key steps that contribute to the workflow; S56, further annotate the selected key steps in detail, generate step description information, and i Provide additional explanation and then generate a detailed step list List' Step ={s1,s2,...,s i ,..,s m }; Among them, s i Indicates any operation step, s i ={t start,i ,t end,i ,e i ,Elements i ,Scoer(i),D i }; D i Indicates i Step description information; start,i and t end,i Respectively represent the operation steps i The start and end time of i Indicates step s i Operation type; Elements i Indicates operation e i The interface elements involved.

6. The method for constructing an AI workflow based on image recognition according to claim 5, characterized in that: The key steps of the screening process are as follows: S61, in order to determine whether there is a significant change between two frames, calculate any frame f i and adjacent frames f i+1 Pixel difference: In the formula, f i With f i+1 represent adjacent key frames respectively; N and M represent the width and height of the frame respectively; D(f i ,f i+1 ) represents the normalized value of the pixel difference between frames; ||f i -f i+1 ||1 represents the L1 norm of the pixel matrix, that is, the sum of the absolute values ​​of the grayscale value differences of all pixels; S62, if D(f i ,f i+1 )>T d , it is considered that an interface change has occurred; Among them, T d Indicates the change threshold of the preset pixel; S63, according to the time window W k and interface changes, preliminarily classify the operation events into corresponding steps; S64. Set the event type of any operation event k The operation type weight W type (e k ) and set the operation change weight W frame (W k ): S65. Score each step based on the operation type and interface change weight: Score(k)=α×W type (e k )+β×W frame (W k ); In the formula, Score(k) represents the importance score of step k; α and β represent weight coefficients, which reflect the importance ratio of the operation event and the interface change. S66: Remove irrelevant steps based on the scoring results, that is, remove steps with scores lower than the minimum scoring threshold T. score Steps, filter out the key steps, and get the step list List Step ={s1,s2,...,s i ,..,s m }; Among them, s i Indicates any operation step, s i ={t start,i ,t end,i ,e i ,Elements i , Scoer(i)}, set its step identifier to k i ;t start,i and t end,i Respectively represent the operation steps i The start and end time of i Indicates step s i Operation type; Elements i Indicates operation e i Involved interface elements; Scoer(i) represents step s i 's rating.

7. The method for constructing an AI workflow based on image recognition according to claim 1, characterized in that: The workflow list is generated as follows: S71. Arrange all steps in ascending order according to the timestamp: S ordered =Sort({s1,s2,...,s i ,...,s m },key=t si ); In the formula, S ordered Represents a set of steps after preliminary sorting; Sort() represents a sorting function; key represents a sorting criterion; s i Indicates any step; t si Indicates any step s i The timestamp recorded when the data was extracted; S72, determine whether the two steps are repeated steps, if sim(s i ,s j )>T sim , then we think that s j To repeat the steps, delete them; sim(s i ,s j )=w1·sim e (s i ,s j )+w2·sim Elements (s i ,s j ); Among them, sim(s i ,s j ) indicates step s i and steps j Similarity; sim e (s i ,s j ) indicates step s i and steps j The similarity of the operation type; sim Elements (s i ,s j ) indicates step s i and steps j similarity of interface elements; w1 and w2 represent step s i and steps j The weight coefficient reflects the priority of similarity calculation; T sim Indicates the set step similarity threshold; S73, by detecting the operation type and interface elements, determine the logical dependencies between the steps, reorder the step sequence according to the dependencies, and obtain the optimized step set S' ordered , and from the optimized step set S' ordered Extract the description information of each step after optimization D i 、Operation type e i and the interface elements involved i , the description information of all steps D i Sequentially stitch together to form a complete text description of the process, according to the time interval of each step [t start,i , t end,i ], extract the corresponding video segments from the original video, and splice all the segments into a complete video process in step order; S74. Add subtitles and visual annotations to each video segment. The subtitle content is taken from the text description of each step. i , the annotation information comes from the interface elements in the step i , and construct any step s accordingly i Instruction template: i ={s i ,e i ,Elements i ,D i }, combining the operation instructions with the key frames in the video segment, generating interactive content, and outputting interactive instructions; S75. The interactive instructions for each step are compiled into the output document in the form of diagrams to generate a workflow list. work ={s'1,s'2,...,s' i ,...}; Among them, s' i represents any optimized operation step, Represents any optimized operation step s' i The corresponding video segment includes the video content of the original video segment, the added subtitles and the visual annotations.

8. The method for constructing an AI workflow based on image recognition according to claim 1, characterized in that: The image recognition model is based on the network structure design of the YOLOv8s model, combined with multi-scale sliding window attention.

9. An AI workflow construction system based on image recognition, which is used to execute an AI workflow construction method based on image recognition according to any one of claims 1 to 8, characterized in that: include: The data collection module is used to record the user's operation process in the business system through a screen recording tool to generate high-quality video files, while capturing the user's operation event data and system feedback information for subsequent video analysis and disassembly; The video analysis and step decomposition module is used to extract video frames from the video at fixed time intervals, detect key frames of scene changes or interface changes by comparing the pixel differences between adjacent frames, and use image recognition and natural language processing technology to analyze the important content in the video. After that, the video is decomposed into multiple step segments, and the operation steps are divided based on the operation event data, filtering out unnecessary intermediate steps; The workflow reorganization module is used to automatically reorganize the video according to the disassembled fragments, remove irrelevant or repeated content, and generate each workflow video. At the same time, it uses computer vision, image segmentation and object recognition technology to extract the text content in the video and generate a text description of the workflow, and generate operating instructions corresponding to the business process; Content annotation and retrieval module, which is used to annotate the generated workflow videos and text content, establish index keywords, and provide a retrieval interface for users. Users can enter keywords to search through the retrieval interface to quickly find the corresponding business process videos or documents for learning, and it also supports multi-format output; Database is used to store data generated during system operation.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for constructing an AI workflow based on image recognition as described in claims 1 to 8 is implemented.

Images