Codeless rpa process automatic generation method based on screen semantic recognition

Abstract

The application relates to the technical field of robot process automation, and discloses a code-free RPA process automatic generation method based on screen semantic recognition, which comprises the following steps: constructing a pixel matrix and a discrete operation event into a time sequence operation data set; extracting visual features of key frames to generate discrete visual nodes, calculating the distance and error between the nodes to construct a directed edge, generating an elastic topological semantic subgraph and a full-screen directed acyclic graph; calculating a time sequence variance to generate a binary mask matrix and extracting a region perception steady-state hash value; performing differential subtraction calculation on the directed acyclic graphs before and after operation to generate a semantic incremental expectation matrix; encapsulating the above data into a closed loop state verification node; addressing by using an approximate subgraph matching algorithm, and when the hash value reaches a convergence threshold, an operation instruction is issued, and the actual semantic satisfies a post-state condition and is stored in the node. The application does not depend on underlying codes, solves the positioning failure problem caused by interface structure change, and realizes closed loop state verification of an automatic process.

Description

Technical Field

[0001] This invention relates to the field of robotic process automation (RPA) technology, specifically to a no-code RPA process automatic generation method based on screen semantic recognition. Background Technology

[0002] Currently, robotic process automation (RPA) technology is often used to replace manual execution of repetitive cross-system interaction tasks. In existing no-code process generation solutions, the system relies on parsing the underlying source code structure attributes of the target application or directly recording the absolute physical coordinates of the screen at the moment of mouse and keyboard triggering to locate interactive elements.

[0003] The above addressing mechanism has limitations in practical applications. When the target software undergoes a version upgrade that leads to the reconstruction of the underlying code tags, or when the operating device experiences resolution scaling and window size adjustment, the pre-recorded static attributes and fixed coordinates will fail to match. When using basic image template matching technology as an alternative, the system will experience image feature extraction offset when facing dynamic pixel changes such as loading animations or blinking cursors in the target interface, resulting in the final element positioning failure.

[0004] Furthermore, the existing process recording mechanism lacks verification logic for the causal relationship between the interface state before and after the action execution. When the system replays operation instructions, it generally sends events in sequence according to a set fixed time interval. It cannot determine whether the target program is in an unready state due to network latency or slow server processing. Even if a single instruction does not trigger the expected interface change, the system will still continue to execute subsequent instructions. This open-loop execution mode will cause the interruption of the entire process. Since this mode lacks a suspension waiting and polling retry mechanism for abnormal states, the success rate of automatically generated process scripts in dynamic software environments is limited. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a no-code RPA process automatic generation method based on screen semantic recognition. This method solves the problems of existing process automation technologies failing to locate target elements due to reliance on underlying code or fixed coordinates when the interface structure changes, and the automation process being interrupted due to the lack of pre- and post-operation state verification logic.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a no-code RPA process automatic generation method based on screen semantic recognition, the method comprising the following steps:

[0007] A time-series computation dataset is constructed by using a two-dimensional pixel matrix, discrete manual operation events, and corresponding timestamps. The time-series computation dataset includes a keyframe pixel matrix and a sequence of consecutive video keyframes divided according to adjacent operation events.

[0008] Visual features are extracted from the keyframe pixel matrix in the time-series operation dataset to generate discrete visual nodes, and macro nodes are constructed through local grid pooling. The Manhattan distance and collinearity error between discrete visual nodes and macro nodes are calculated to construct directed edges. Combined with the location of discrete manual operation events, an elastic topological semantic subgraph and a full-screen directed acyclic graph are generated.

[0009] The pixel-level temporal variance of a continuous video keyframe sequence is calculated to generate a binary mask matrix. The region-aware steady-state hash value of the target region is extracted by combining the binary mask matrix. The full-screen directed acyclic graph before and after the operation is obtained. Graph theory difference subtraction is performed on the full-screen directed acyclic graph before and after the operation to generate a semantic increment expectation matrix.

[0010] The region-aware steady-state hash value, the binarized mask matrix, the semantic increment expectation matrix, and the elastic topological semantic subgraph are encapsulated into a closed-loop state verification node.

[0011] The approximate subgraph matching algorithm is used to locate candidate subgraphs that match the elastic topological semantic subgraph in the current interface, and the real-time feature hash value of the corresponding region is extracted by combining the binarized mask matrix. When the Hamming distance between the real-time feature hash value and the region-aware steady-state hash value reaches the convergence threshold, the physical coordinates of the central target node are output and the operation command is sent. After the command is executed, if the current actual semantics meet the post-state conditions of the semantic increment expectation matrix, the closed-loop state verification node is stored to complete the automatic generation of the no-code RPA process.

[0012] By extracting discrete visual nodes to construct a full-screen directed acyclic graph, the spatial physical structure of the interface is transformed into graph theory feature data. The semantic increment expectation matrix is ​​extracted using the graph difference before and after the operation, and a closed-loop state verification node is formed by combining the region-aware steady-state hash value. During the execution process replay, the approximate addressing of local spatial adjacency relationships is performed based on the elastic topological semantic subgraph, and the multimodal difference between the actual state of the interface before and after the operation and the expectation matrix is ​​compared. Through the above processing, the interface state verification and automatic process generation are completed without relying on the application's underlying code.

[0013] Preferably, the step of constructing a time-series computation dataset from two-dimensional pixel matrices, discrete manual operation events, and corresponding timestamps includes: calculating the difference between the system timestamp of the discrete manual operation event and the system timestamp of each of the two-dimensional pixel matrices; extracting the two-dimensional pixel matrix whose system timestamp is closest to the trigger time of the discrete manual operation event from the two-dimensional pixel matrices whose difference is within a preset time synchronization tolerance, and defining it as the keyframe pixel matrix; performing structured data fusion on the keyframe pixel matrix and the associated discrete manual operation events and system timestamps according to discrete time-series labels, and caching the two-dimensional pixel matrices collected between two adjacent operation events as the continuous video keyframe sequence.

[0014] The technical principle of this step is as follows: by setting a time synchronization tolerance value, the asynchronously generated discrete operation event stream of the operating system and the interface rendering video stream are time-aligned to eliminate errors caused by hardware input delays, establish a temporal causal mapping relationship between operation intentions and visual interface states, and provide structured data with correlation for subsequent graph calculations.

[0015] Preferably, the step of extracting visual features from the keyframe pixel matrix in the time-series computation dataset to generate discrete visual nodes, and constructing macro nodes through local grid pooling includes: inputting the keyframe pixel matrix into the visual feature extraction model, and outputting the discrete visual nodes with bounding box center coordinates and semantic labels; when it is recognized that the local discrete visual nodes present a single-axis or dual-axis equidistant distribution, extracting the global extreme value boundary formed by the discrete visual nodes presenting a single-axis or dual-axis equidistant distribution to generate an outer bounding box as the macro node; establishing a two-dimensional index matrix mapping absolute physical coordinates inside the macro node, and mapping and filling the discrete visual nodes into the corresponding coordinate positions to construct the macro node.

[0016] The technical principle of this step is as follows: For interface areas that present a grid-like distribution pattern, directly calculating the topological connections of all nodes in the graph would increase the computational cost. By extracting the global extreme value boundary, discrete visual nodes that present a regular distribution in the local area are encapsulated into a macro node, thereby reducing the computational complexity of the topological connectivity of the entire screen. At the same time, a two-dimensional index matrix is ​​used to preserve the relative spatial positions of the elements contained within the macro node, thereby reducing the computational resource overhead of subsequent graph construction.

[0017] Preferably, the step of constructing directed edges by calculating the Manhattan distance and collinearity error between discrete visual nodes and macro nodes, and generating an elastic topological semantic subgraph and a full-screen directed acyclic graph by combining the location of discrete human operation events includes: establishing the start and end points of directed edges between node pairs according to a preset scanning order from left to right and from top to bottom; calculating the physical topological weight of each directed edge by combining the Manhattan distance and the collinearity error, and retaining a preset number of directed edges in ascending order of the physical topological weight to generate the full-screen directed acyclic graph; taking the target node corresponding to the location of the discrete human operation event as the center starting point, performing a breadth-first search algorithm according to the connection direction of the directed edges to extract all associated nodes and connecting edges within a set connection order, and defining the extracted local network topology as the elastic topological semantic subgraph.

[0018] The technical principle of this step is as follows: using a computer-generated standard that combines Manhattan distance and collinearity error as the standard for calculating edge weights, the distribution of visual elements in the two-dimensional screen is transformed into a directed network structure. Starting from the operation target, the local network is extracted according to the connectivity order. The positioning reference of the core element is transformed from the absolute physical coordinates of the screen to the local spatial topological adjacency relationship, thus avoiding element positioning failure caused by resolution scaling and element displacement.

[0019] Preferably, the step of calculating the pixel-level temporal variance of the continuous video keyframe sequence to generate a binary mask matrix, and extracting the region-aware steady-state hash value of the target region by combining the binary mask matrix includes: traversing each numerical element in the matrix composed of the pixel-level temporal variance, mapping matrix elements greater than the dynamic noise threshold to 0, and mapping matrix elements less than or equal to the dynamic noise threshold to 1 to output the binary mask matrix.

[0020] The target region image block is extracted outward from the center coordinates of the target element, and the corresponding local mask slice is extracted from the binary mask matrix. The Hadamard product operation is performed on the target region image block and the local mask slice to change the pixel value of the dynamically changing state to 0. The low frequency feature coefficients are extracted from the result matrix using the discrete cosine transform algorithm, the mean of the sequence formed by each feature coefficient is calculated, and binary encoding quantization is performed to output the region-aware steady-state hash value.

[0021] The technical principle of this step is as follows: high-frequency jump features such as blinking cursors or loading animations in continuous video sequences can cause hash comparison to fail. The pixel-level temporal variance is used to identify the jump regions and generate the corresponding binary mask matrix. Combined with the Hadamard product operation, the dynamically changing pixels in the target region are zeroed. This process enables the discrete cosine transform to extract low-frequency feature coefficients for static low-frequency texture features, maintaining the numerical stability of the system output region when sensing the steady-state hash value.

[0022] Preferably, the step of obtaining the full-screen directed acyclic graph before and after the operation, and performing graph theory difference subtraction calculation on the full-screen directed acyclic graph before and after the operation to generate a semantic increment expectation matrix includes: performing a bidirectional traversal matching search across the graph based on the structured semantic labels and absolute physical coordinates of the nodes in the full-screen directed acyclic graph; using graph theory difference operation to cluster the set of disappeared nodes that existed before the operation but are missing after the operation, and the set of newly added nodes that were missing before the operation but are newly generated after the operation; and merging and encapsulating the set of newly added nodes and the set of disappeared nodes, along with their associated physical attribute parameters, at the memory level to generate the semantic increment expectation matrix.

[0023] The technical principle of this step is as follows: Based on the extracted graph data before and after the operation, node difference operation is performed to cluster the graph structure changes after a single discrete event is triggered. This calculation logic filters out unrelated background area data updates and separates the local change dataset generated by the single discrete event, providing a post-verification benchmark for subsequent execution processes.

[0024] Preferably, the step of encapsulating the region-aware steady-state hash value, the binarized mask matrix, the semantic increment expectation matrix, and the elastic topological semantic subgraph into a closed-loop state verification node includes: merging the elastic topological semantic subgraph, the binarized mask matrix, and the region-aware steady-state hash value into a data structure to generate an independent structure that serves as a precondition assertion library;

[0025] The macro node containing the core title text of the page is defined as the global state verification benchmark node, and is converted into a post-result assertion library along with the semantic increment expectation matrix. A contiguous memory block is allocated in the memory heap area to instantiate the autonomous closed-loop state machine node object. The precondition assertion library, action function, post-result assertion library, and ring retry logic control block containing the preset maximum number of retries and the waiting delay timer are written in the order of time sequence and cause and effect to generate the closed-loop state verification node.

[0026] The technical principle of this step is as follows: generate a precondition assertion library for the parameters corresponding to the operation before the operation, and generate a postcondition assertion library for the change results corresponding to the operation after the operation. In the memory heap area, the precondition assertion library, action function, postcondition assertion library and ring retry logic control block are encapsulated and combined. This structure makes each generated closed-loop state verification node independent of each other. A single node can independently call the logic data to complete the precondition state verification, addressing and instruction issuance actions.

[0027] Preferably, the step of using an approximate subgraph matching algorithm to locate candidate subgraphs that match the elastic topological semantic subgraph on the current interface, and extracting the real-time feature hash value of the corresponding region by combining a binarized mask matrix; when the Hamming distance between the real-time feature hash value and the region-aware steady-state hash value reaches the convergence threshold, outputting the physical coordinates of the central target node and sending an operation command includes: extracting multiple real-time local candidate subgraphs with the same connectivity order as the elastic topological semantic subgraph in the real-time global directed acyclic graph; calculating the Levenstein edit distance based on the structured semantic tag string to obtain the text semantic similarity, and simultaneously extracting the physical topological weights of each directed edge of the two and calculating the absolute difference, then dividing the absolute difference by the global maximum weight value to perform extreme value normalization mapping, and outputting the topological structure difference of the edge set accordingly;

[0028] The semantic similarity of node text and the topological difference of normalized edge weights are calculated by positive and negative weighted subtraction, with semantic similarity as the positive gain and topological difference as the negative penalty. Candidate subgraphs with a comprehensive approximate topological matching similarity score greater than a preset matching threshold are addressed. The coordinates of the central target node in each candidate subgraph are extracted as a reference point to expand outward and capture real-time image blocks. The Hadamard product operation is performed on the real-time image block in combination with the binarized mask matrix to remove dynamic interference pixels, generating real-time feature hash values ​​of the physical regions corresponding to each candidate subgraph. The candidate subgraph with the smallest Hamming distance value that is less than the preset hash convergence threshold is selected, and the absolute physical coordinate parameters of the central target node in the subgraph are extracted as the physical coordinates. The application programming interface at the operating system level is called to send the corresponding artificial simulated hardware interrupt event to the peripheral driver as the operation instruction.

[0029] The technical principle of this step is as follows: introduce a hybrid calculation logic of text attributes and topological weights. When the text content of interface elements is modified or the relative spacing changes, a weighted difference mechanism with positive gain for text similarity and negative penalty for topological structure difference is used to determine the matching score and output candidate subgraphs. At the same time, operation instructions are issued in conjunction with the hardware interrupt mechanism of the underlying operating system to achieve non-intrusive automated process execution.

[0030] Preferably, after the instruction is executed, if the current actual semantics satisfy the post-state conditions of the semantic increment expectation matrix, the step of storing the closed-loop state verification node to complete the automatic generation of the no-code RPA process includes: extracting the current interface actual semantic nodes by combining the real-time frame data of the interface after the action is triggered, and cross-comparing them with the semantic increment expectation matrix, performing multimodal state difference comparison calculation to obtain the state difference calculation result; when the state difference calculation result and the semantic increment expectation matrix reach a preset post-verification matching threshold at the numerical feature level, the current closed-loop state verification node flows into the success state and the closed-loop state verification node is stored.

[0031] When the state difference calculation result does not reach the preset post-verification matching threshold, the system triggers polling suspension based on the waiting delay parameter configured in the closed-loop state verification node. If the polling comparison count reaches the maximum retry count limit and the post-verification is still not passed, the system calls the precondition assertion library again. Only when the current interface still meets the precondition will the system re-execute the target addressing and action playback process.

[0032] The technical principle of this step is as follows: In the case of interface change response delay caused by network latency or system processing lag, when the initial state differential calculation result does not reach the preset threshold, a polling suspension operation is performed based on the node's built-in waiting delay parameter to wait for the state data to change. When the number of retries reaches the upper limit, the previous state is rolled back for verification. This two-way verification and polling mechanism can avoid execution errors caused by slow interface feedback. After the actual feedback differential calculation result on the interface reaches the preset post-verification matching threshold, it is then transferred and stored in the closed-loop state verification node.

[0033] Preferably, the step of extracting the actual semantic nodes of the current interface and performing multimodal state difference comparison calculation with the semantic increment expectation matrix includes: before performing the state difference comparison calculation, if the post-result assertion library contains a global state verification benchmark node, the system first traverses and addresses anchor objects in the real-time frame data that match the semantic label and physical bounding box features of the global state verification benchmark node; when the anchor object is successfully addressed and the physical coordinate offset is within the tolerance range, the subsequent difference calculation process continues; if the addressing fails or the physical coordinate offset exceeds the tolerance range, the system directly determines that the current post-state verification has failed and triggers the subsequent polling suspension mechanism.

[0034] The technical principle of this step is as follows: Before extracting the state difference calculation results, a priori displacement judgment is performed on the verification benchmark node object. When the overall page scrolling or window dragging causes the coordinate system benchmark to shift, the coordinate offset of the anchor point object corresponding to the benchmark node is compared first. If the coordinate offset exceeds the tolerance range, the verification is intercepted and determined to have failed in advance, thereby avoiding the issuance of invalid graph theory difference operation instructions in abnormal states and reducing the computational burden.

[0035] In summary, the present invention has at least one of the following beneficial technical effects:

[0036] 1. This invention extracts discrete visual nodes from keyframes of the interface and constructs a full-screen directed acyclic graph and a flexible topological semantic subgraph by combining the Manhattan distance and collinearity error between nodes. This transforms the traditional absolute physical coordinate positioning mechanism of the screen into a graph theory feature addressing logic based on local spatial adjacency. When the target interface undergoes resolution scaling, relative displacement of internal elements, or page structure reorganization, this scheme can maintain the effectiveness of the positioning reference based on the physical topological weights and connectivity directions between nodes, avoiding the failure of automated process execution caused by the solidification of a single coordinate.

[0037] 2. This invention generates a semantic increment expectation matrix containing graph structure change data by comparing the full-screen directed acyclic graph before and after the action is executed. This matrix, along with a pre-assertion library and action functions, is encapsulated to construct a closed-loop state verification node with polling and retry logic. After the operation command is issued, the system performs multimodal state differential verification and timeout suspension polling by cross-comparing the actual frame state of the interface with the numerical characteristics of the semantic increment expectation matrix. This establishes a data verification mechanism for the state correlation before and after a single discrete operation, solving the problem of incorrect state transition judgment caused by network latency or front-end rendering lag.

[0038] 3. This invention generates a binary mask matrix by calculating the pixel-level temporal variance of a continuous video keyframe sequence, and performs a Hadamard product operation on the binary mask matrix after capturing real-time image blocks. This processing step forces the pixels containing dynamic jump features such as blinking cursors and loading animations to zero in the numerical dimension of the underlying matrix. Subsequently, the discrete cosine transform algorithm is used to extract low-frequency feature coefficients to generate region-aware steady-state hash values, eliminating the feature interference of high-frequency dynamic noise on the image hash comparison process and improving the region addressing matching accuracy of the interface in dynamic rendering scenarios. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the system timing interaction of the present invention;

[0040] Figure 2 This is a schematic diagram of the method flow of the present invention;

[0041] Figure 3 This is a schematic diagram of the multimodal data alignment and timing synchronization tolerance curve of the present invention;

[0042] Figure 4 This is a schematic diagram of the steps of topology modeling and process synthesis in this invention;

[0043] Figure 5 This is a schematic diagram of the confidence decay and steady-state evaluation curves of the closed-loop verification link of the present invention;

[0044] Figure 6 This is a schematic diagram of the steps for automated closed-loop execution and status verification of the present invention;

[0045] Figure 7 This is a schematic diagram comparing the success rate of automated addressing under multi-dimensional variation interference scenarios according to the present invention.

[0046] Among them, 100 is the data acquisition module; 200 is the topology modeling module; 300 is the process synthesis module; and 400 is the closed-loop execution module. Detailed Implementation

[0047] See attached document Figure 1The present invention provides a no-code RPA process automatic generation system based on screen semantic recognition. The system includes: a data acquisition module 100, a topology modeling module 200, a process synthesis module 300, and a closed-loop execution module 400.

[0048] The data acquisition module 100 listens to the display output and peripheral input ports, acquires the two-dimensional pixel matrix of the screen at a fixed frame rate, and synchronously reads discrete manual operation events carrying event types and physical coordinates. The data acquisition module 100 merges the two-dimensional pixel matrix, discrete manual operation events and corresponding timestamps to construct a time-series calculation dataset.

[0049] The topology modeling module 200 is connected to the data acquisition module 100 and is used to receive the time-series operation dataset. It performs visual feature extraction on the keyframe pixel matrix containing effective operations and generates discrete visual nodes with center coordinates and semantic labels. The topology modeling module 200 performs local grid pooling on the discrete visual nodes that meet the spatial equidistant distribution conditions to merge them into macro nodes. The topology modeling module 200 combines the Manhattan distance and collinearity error between nodes to construct directed edges and generates locally connected elastic topological semantic subgraphs and full-screen directed acyclic graphs based on the location of operation events.

[0050] The process synthesis module 300 is connected to the topology modeling module 200 and is used to extract the continuous video key frame sequence between adjacent operations recorded by the data acquisition module 100. The process synthesis module 300 calculates the temporal variance within the time window of the continuous sequence to generate a binary mask matrix, and extracts the region-aware steady-state hash value of the target area by combining the mask matrix. The process synthesis module 300 generates a semantic increment expectation matrix by performing difference calculation on the full-screen directed acyclic graph before and after the operation, and encapsulates and outputs the binary mask matrix, the region-aware steady-state hash value, the semantic increment expectation matrix and the elastic topology semantic subgraph as a state verification node.

[0051] The closed-loop execution module 400 is connected to the process synthesis module 300 and is used to receive the state verification node. The closed-loop execution module 400 extracts the real-time feature hash value of the real-time video frame stream by combining the preset binarized mask matrix. When the Hamming distance between the real-time feature hash value and the region-aware steady-state hash value is within the convergence threshold, the closed-loop execution module 400 uses the elastic topological semantic subgraph to perform pre-state verification on the current interface. After the pre-state verification is passed, the approximate subgraph matching algorithm is used to perform mapping cost calculation and addressing in the full directed acyclic graph of the current interface, outputs the physical coordinates of the mapping node with the minimum cost, and calls the underlying interface to send the corresponding operation command to the physical coordinate. After the operation command is executed, the closed-loop execution module 400 extracts the semantics of the current interface and cross-compares it with the semantic increment expectation matrix. When the actual semantics meet the post-state conditions limited by the semantic increment expectation matrix, the current state verification node is determined to have executed successfully.

[0052] See attached document Figure 2 This invention provides a no-code RPA process automatic generation method based on screen semantic recognition, the method comprising the following steps:

[0053] S100 calls the operating system's graphics display interface or desktop composite image interface to read the current frame buffer image data of the display, and listens to the peripheral input port to obtain the two-dimensional pixel matrix of the screen at a fixed frame rate. Simultaneously, it reads discrete human operation events carrying event types and physical coordinates, and fuses the two-dimensional pixel matrix, discrete human operation events and corresponding timestamps to construct a time-series operation dataset. The time-series operation dataset includes a keyframe pixel matrix aligned with the timestamps of discrete human operation events, and a continuous video keyframe sequence divided according to adjacent discrete human operation events.

[0054] S200 extracts keyframe pixel matrices containing valid operations based on a time-series computation dataset. It performs visual feature extraction on the keyframe pixel matrices to generate discrete visual nodes with bounding box center coordinates and semantic labels. It traverses the spatial coordinate positions of discrete visual nodes. When it identifies local nodes that are equidistant from a single axis or two axes, it performs local grid pooling to merge discrete visual nodes in the corresponding region into macro nodes. It calculates the Manhattan distance and collinearity error between nodes to construct directed edges between nodes. It combines the location of discrete human operation events to generate locally connected elastic topological semantic subgraphs and full-screen directed acyclic graphs.

[0055] S300 extracts the continuous video keyframe sequence between adjacent discrete manual operation events, calculates the pixel-level temporal variance of the continuous video keyframe sequence within the time window to generate a binarized mask matrix, and extracts the region-aware steady-state hash value of the target area by combining the binarized mask matrix. Then, it performs graph theory difference subtraction calculation on the full-screen directed acyclic graph before and after the operation to generate a semantic increment expectation matrix containing the states of newly added nodes and the states of disappeared nodes.

[0056] S400 extracts the region-aware steady-state hash value, binarized mask matrix, semantic increment expectation matrix, and elastic topology semantic subgraph. It merges and encapsulates the above data structures according to the set protocol, generates and outputs a closed-loop state verification node that does not depend on the target business software's underlying control tree, document object model, or business system open interface, and completes the structured synthesis of the automated process document. The closed-loop state verification node includes node number, preceding elastic topology semantic subgraph, binarized mask matrix, region-aware steady-state hash value, action function type, action parameters, dynamic parameter variable mapping table, subsequent semantic increment expectation matrix, global state verification benchmark node, maximum number of retries, and waiting delay parameters.

[0057] S500 parses the closed-loop state verification node and extracts the real-time video frame stream, using the binarized mask matrix within it to calculate the real-time feature hash value. When the Hamming distance between this hash value and the region-aware steady-state hash value is within the convergence threshold, it uses the elastic topological semantic subgraph to perform pre-state verification on the actual semantic node of the current interface. After the pre-state condition verification is passed, it uses an approximate subgraph matching algorithm to address in the real-time global directed acyclic graph, outputs the physical coordinates of the minimum cost mapping node, and sends the corresponding operation command. After the command is executed, it extracts the actual semantic node and cross-compares it with the semantic increment expectation matrix. If the post-state condition is met, the execution is considered successful; otherwise, it is suspended and retried according to the delay parameter. If the retry limit is reached and the pre-state condition is met, the addressing and command sending process is re-executed.

[0058] See attached document Figure 3 The specific implementation of step S100 can be handled through the following sub-steps:

[0059] S101, the data acquisition module 100 starts as an independent background process in the computer terminal operating system, calls the system's underlying graphics device interface, desktop composite image interface, or frame buffer reading interface, and combines with peripheral global hook functions to listen for changes in the screen display image and peripheral input events. During operation, the data acquisition module 100 does not inject dynamic link libraries into the target business software and does not obtain the system's built-in document object model parsing component. The actual operating environment of the data acquisition module 100 constitutes a seamless bypass listening mode that does not depend on the underlying control tree structure of the target business software. The data acquisition module 100 uses the underlying graphics infrastructure interface to trigger the video memory reading mechanism according to the fixed frame rate set internally by the control system and captures the full-screen image buffer data in the display's video memory. The value range of this fixed frame rate is configured to be 10 to 30 frames per second.

[0060] The data acquisition module 100 converts the captured full-screen image buffer data into a two-dimensional pixel matrix containing multi-channel color values ​​and width and height parameters. The color space conversion and image matrix reshaping operations of the two-dimensional pixel matrix can be implemented using existing computer graphics processing technology. During the synchronous process of capturing the two-dimensional pixel matrix, the data acquisition module 100 intercepts discrete manual operation events issued by external input devices through a peripheral global hook function. The discrete manual operation events include mouse click status, keyboard key codes, and the absolute physical coordinates on the screen at the moment the command is issued. The module internally presets a circular buffer to temporarily store the above discrete events to avoid concurrent conflicts.

[0061] For continuous input events formed by keyboard key encoding, the data acquisition module 100 aggregates multiple discrete keyboard events into a string input event according to the time interval threshold between adjacent key events. The specific value of the time interval threshold is configured to be between 200 milliseconds and 500 milliseconds, and the string input event is bound to the input box node hit by the most recent mouse click.

[0062] The data acquisition module 100 then calls the high-precision hardware counter inside the operating system to assign the corresponding system timestamps for the generation time of the intercepted and temporarily stored discrete manual operation events and the captured two-dimensional pixel matrices of each frame. With the help of continuous timed triggering and global event interception actions, a discrete time sampling sequence of the underlying multimodal data is established. The multimodal data covers the visual pixel information and discrete control logic information obtained based on different physical channels.

[0063] S102, after acquiring the discrete-time sampling sequence, the data acquisition module 100 uses its internal high-speed cache to read the allocated system timestamps of each item, and performs a time axis alignment mapping operation based on timestamp comparison on the discrete manual operation events and the continuously cached two-dimensional pixel matrix. In specific processing, the data acquisition module 100 calculates the difference between the system timestamp of the discrete manual operation event and the system timestamp of each two-dimensional pixel matrix, and filters out candidates whose absolute value of the time difference is within the preset time synchronization tolerance. The specific value of the preset time synchronization tolerance is configured to be between 50 milliseconds and 100 milliseconds.

[0064] The data acquisition module 100 extracts the two-dimensional pixel matrix whose system timestamp is closest to the trigger time of the discrete manual operation event from the candidate options, and defines the two-dimensional pixel matrix whose system timestamp is closest to the trigger time of the discrete manual operation event as the key frame pixel matrix associated with the operation instruction. At the same time, the data acquisition module 100 caches the continuous two-dimensional pixel matrix acquired between two adjacent discrete manual operation events as a continuous video key frame sequence according to the timestamp boundary of two adjacent operation events, so as to be used for subsequent temporal variance calculation, steady-state hash extraction and interface stability judgment.

[0065] The aforementioned timestamp filtering mechanism is used to establish the alignment relationship between operation events and keyframes without deleting the continuous frame cache data used for timing analysis between adjacent operation events. The data acquisition module 100 further establishes a continuous memory data structure by addressing the data bus inside the controller, and performs structured data fusion with the extracted keyframe pixel matrix and the associated discrete manual operation events and system timestamps according to discrete timing marks, thereby constructing a timing operation dataset with independent address pointers.

[0066] The memory allocation and underlying serialization encapsulation of the timing operation dataset can be performed according to the existing multimodal data serialization encapsulation method. The generated timing operation dataset is stored in memory as an independent data carrier and published on the internal storage bus. This publication action allows subsequent software logic units to perform direct addressing calls across modules. After the data publication is completed, the system's internal storage management unit clears the underlying video memory data register resources occupied in the early stage and enters the listening and waiting state of the next discrete event acquisition cycle.

[0067] The aforementioned time-series computation dataset is essentially a physical mapping model that binds screen visual states and human operation logic in a discrete-time dimension. This physical mapping model exists in the computer's volatile memory space in the form of a discrete-time series set, and simultaneously includes a set of operation event alignment keyframes and a set of consecutive video frame buffers within adjacent operation event intervals. The basic set data structure follows the following expression:

[0068] ;

[0069] In the formula, For constructing a time-series computation dataset; To capture the extracted keyframe 2D pixel matrix under the matching system timestamp; For discrete manual operation events that carry event type identifiers and physical screen absolute coordinate parameters; A unified system timestamp allocated by a high-precision hardware counter after being aligned with the operation action; For each positive integer index item in a discrete time series; This is the total number of operation events recorded by the system during the global data acquisition phase, and it is a positive integer.

[0070] See attached document Figure 4 The specific implementation of step S200 can be handled through the following sub-steps:

[0071] S201, the topology modeling module 200 receives the time-series computation dataset through the memory bus and extracts the encapsulated keyframe pixel matrix. The topology modeling module 200 performs image preprocessing operations on the keyframe pixel matrix, uses bilinear interpolation algorithm to scale and reshape the keyframe pixel matrix into a fixed-dimensional standard tensor of 640×640×3, and performs normalization processing by dividing the pixel value by 255. The topology modeling module 200 loads the preprocessed standard tensor into the internally deployed visual feature extraction model, which is configured as a detection architecture including a feature extraction backbone network, a feature pyramid fusion network, and a prediction head.

[0072] The visual feature extraction model employs a deep residual network structure with 50 convolutional layers as its feature extraction backbone. This backbone extracts multi-scale feature maps containing shallow edges and deep semantics through cascaded residual blocks. The feature pyramid fusion network receives these multi-scale feature maps and performs top-down upsampling and lateral concatenation operations. The prediction head outputs prediction data containing bounding box coordinate offsets, component confidence, and class probabilities based on the fused feature tensor. The visual feature extraction model is trained under supervision using an offline screenshot dataset. The training sample set consists of screenshots of business software interfaces with manually labeled bounding boxes and classification labels. The loss function during training is a weighted combination of cross-entropy loss and intersection-union bounding box regression loss.

[0073] The connection-series classification optical character recognition branch within the visual feature extraction model is cascaded at the detection box output of the above detection architecture. It receives the image slice tensor of the corresponding bounding box region and outputs the text content string.

[0074] The specific creation and training methods for the aforementioned visual feature extraction model include the following steps:

[0075] First, real-world screenshots covering various operating systems, web applications, and client-side business software were collected as raw image data. Data augmentation operations were performed on the raw image data, including random scaling, local cropping, Gaussian noise addition, and color space perturbation, to improve the model's robustness and generalization ability. Then, data annotation tools were used to manually annotate all interactive elements and text regions within the augmented images. Annotations included the bounding box coordinates of the corresponding elements, interface component classification labels from a pre-defined business dictionary (e.g., buttons, input boxes, dropdown menus, checkboxes), and the actual string content of the text regions. The images and their corresponding annotation files were serialized and combined to construct a standardized multimodal screenshot training dataset.

[0076] Secondly, a deep residual network with 50 convolutional layers is initialized in the deep learning framework as the feature extraction backbone network, and pre-trained weight parameters on a general image dataset are selectively loaded. The feature pyramid fusion network is skip-connected and spliced ​​with the backbone network. At the multi-scale output end of the feature pyramid fusion network, detection prediction heads for outputting bounding box coordinate offsets and component class probabilities, and optical character recognition branches for outputting text strings are respectively attached to complete the creation and construction of the overall forward computation graph of the model.

[0077] Finally, the training method for the visual feature extraction model is implemented:

[0078] The completed multimodal screenshot training dataset is divided into training and validation sets according to a set ratio; the training hyperparameters of the model are set, including using the Adam optimizer, configuring the initial learning rate to 0.0001 and using a cosine annealing learning rate decay strategy, and configuring the batch size of a single training session to an integer between 16 and 32.

[0079] In each training iteration, the training set images are scaled and normalized in sequence, and then input into the model that has not fully converged in batches in tensor form. Forward propagation is then performed to calculate the corresponding bounding boxes, classification and text prediction data.

[0080] The joint loss function is called to calculate the error between the model's predicted data and the real labeled data. The joint loss function is a weighted combination of the cross-entropy loss function, the intersection-union bounding box regression loss function, and the linked temporal classification loss function to calculate the overall loss value.

[0081] Based on the calculated overall loss value, the backpropagation algorithm is executed to calculate the network gradient, and the weight parameters of each feature layer of the network are updated using the set optimizer.

[0082] After each iteration cycle, the model is evaluated and validated using the validation set for mean accuracy and character error rate. When the comprehensive index on the validation set no longer shows significant improvement within the continuously set evaluation cycle, the early stopping mechanism is triggered, the parameter training process is terminated, and the visual feature extraction model with the optimal weight parameters is exported and saved, and deployed to solid-state storage or memory space for subsequent inference calls.

[0083] The topology modeling module 200 obtains the physical bounding box coordinates of each target element output by the visual feature extraction model, as well as the corresponding interface component category and text content string. Based on the physical bounding box coordinates, it calculates the absolute physical coordinates of the center point of the corresponding target element, and performs a string concatenation operation on the interface component category and text content string to generate the structured semantic label of the target element. The topology modeling module 200 encapsulates and fuses the absolute physical coordinates of the center point and the corresponding structured semantic label to generate discrete visual nodes.

[0084] The topology modeling module 200 traverses all target elements in the pixel matrix of the current keyframe and repeatedly performs encapsulation operations to build a discrete visual node set in memory that covers the effective elements of the current screen.

[0085] S202, the topology modeling module 200 traverses the spatial coordinates and corresponding structured semantic labels of each element in the discrete visual node set. First, it divides local candidate regions based on the spatial adjacency distance between nodes, the boundary of connected regions, or the density clustering results. Then, it extracts discrete visual nodes of the same type with the same interface component classification category in each local candidate region to form multiple independent subsets. The topology modeling module 200 detects the total number of nodes contained in each independent subset. When it is determined that the total number of nodes in the subset is greater than the preset effective comparison number threshold, it triggers the subsequent equidistant calculation action. The effective comparison number threshold is configured as a positive integer 3. When the total number of nodes does not reach the threshold, the system determines that the subset does not have periodic arrangement characteristics.

[0086] The topology modeling module 200 calculates the coordinate differences of the center points of adjacent discrete visual nodes in the horizontal and vertical directions within the effective subset, and defines the set of calculated coordinate differences as the spatial spacing sequence. When the variance of each element in the spatial spacing sequence is less than the preset spatial equidistant tolerance threshold, the system determines that the discrete visual nodes in the subset exhibit single-axis or dual-axis equidistant distribution characteristics. The specific value of the spatial equidistant tolerance threshold is configured to be 2 to 8 pixels. For the subset of discrete visual nodes determined to be equidistantly distributed, the topology modeling module 200 performs a local mesh pooling operation on it, extracts the global extreme boundary formed by all nodes in the subset to generate an outer bounding box, and configures it as a macro node.

[0087] The topology modeling module 200 establishes a two-dimensional index matrix inside the macro node, mapping each element in the original discrete visual node subset into the corresponding coordinate position of the two-dimensional index matrix according to the row and column order of its absolute physical coordinates; the system transforms the densely arranged table or list data nodes in the screen interface into a single macro node object in terms of data structure through local grid pooling; the dimensionality-reduced macro node is scheduled as an independent unit entity in the graph theory operation memory space; the regular discrete visual nodes that have not triggered the equidistant distribution condition and the generated macro nodes are jointly reorganized in the system memory to form a dimensionality-reduced operation node pool; the above-mentioned dimensionality-reduced operation node pool construction action controls the total number of physical nodes in the global topology graph model.

[0088] S203, the topology modeling module 200 traverses each node in the dimensionality reduction operation node pool and performs pairing between node pairs that meet preset spatial neighborhood conditions, same visual row and column conditions, or topological proximity conditions. Based on the left-to-right and top-to-bottom scanning order set according to the screen reading rules, the starting and ending points of the directed edges between the node pairs are established. The topology modeling module 200 calculates the Manhattan distance between each node pair and the collinearity error that represents the physical alignment deviation, and calculates the physical topological weight of each directed edge by combining the Manhattan distance and the collinearity error.

[0089] The physical topology weight calculation logic uses the spatial straight line span and the degree of alignment of the interface layout axis to comprehensively quantify the connectivity cost at the physical visual level. The system assigns directed edges with physical topology weights to the corresponding node pairs, and sorts the successor nodes of each node in ascending order of physical topology weights. Only the directed edges corresponding to the first preset number of successor nodes are retained. The preset number is configured to be 3 to 8, thereby generating a full-screen directed acyclic graph covering the current interface.

[0090] The calculation logic for the physical topological weights of each directed edge in a full-screen directed acyclic graph follows the mathematical model expression below:

[0091] ;

[0092] In the formula, For a full-screen directed acyclic graph, from node Pointing to node The physical topological weight value of a directed edge; The preset distance attenuation penalty factor is configured to range from 0.5 to 1.5. For nodes Center point coordinates and nodes The Manhattan distance quantized from the center point coordinates on a two-dimensional plane; The preset collinearity deviation penalty factor is configured to have a value range of 1.0 to 2.0. For nodes With nodes The collinearity penalty function is calculated based on the nodes. With nodes The coordinates of the center point in the horizontal or vertical direction are obtained by linear mapping to the ratio of the absolute value of the coordinate difference to the average side length of the corresponding bounding boxes of the two points. When constructing topological edges, the positive integer index of the discrete visual node or macro node that serves as the starting point for connectivity; When constructing topological edges, the positive integer index of the discrete visual node or macro node that serves as the endpoint of the connection.

[0093] S204, the topology modeling module 200 extracts the discrete manual operation events associated with the time-series operation dataset and parses the absolute physical coordinates carried by the event. The topology modeling module 200 performs traversal comparison in the full-screen directed acyclic graph and addresses the target node whose bounding box boundary covers the absolute physical coordinate. When the traversal comparison determines that the absolute physical coordinate does not fall inside any existing node bounding box, the topology modeling module 200 checks whether the current full-screen directed acyclic graph is an empty set. In a non-empty graph, the system extracts and confirms the entity node with the smallest Euclidean distance from the center point coordinate of the current full-screen directed acyclic graph to the absolute physical coordinate as the target node. When the graph is an empty set, the system directly generates a virtual isolated node with a globally unique identifier centered on the absolute physical coordinate as the target node, ensuring the continuous execution of the subsequent topology addressing closed-loop logic.

[0094] The topology modeling module 200 takes the target node as the starting point and performs a breadth-first search algorithm based on the connection direction of the directed edges to extract all associated nodes and connecting edges within a set connection order. The extracted local network topology is defined as a flexible topological semantic subgraph. The set connection order is configured to be from order 1 to order 3. During this search and addressing process, the topology modeling module 200 detects the type attribute of the target node. When the target node is determined to be a macro node, the system synchronously extracts and caches the two-dimensional index matrix bound inside the macro node and the exact position of the two-dimensional index coordinate system hit by the corresponding operation event.

[0095] The specific implementation of step S300 can be carried out through the following sub-steps:

[0096] S301, the process synthesis module 300 accesses the time-series operation dataset through the internal data bus and extracts the continuous video keyframe sequence recorded between two adjacent discrete manual operation events. The process synthesis module 300 detects the total number of image frames contained in the continuous video keyframe sequence. When it determines that the total number of image frames is greater than the preset time-series comparison lower limit threshold, it triggers the variance operation logic. The time-series comparison lower limit threshold is configured as a positive integer 3. When the total number of image frames does not reach the time-series comparison lower limit threshold, the process synthesis module 300 directly generates a global steady-state matrix with all elements set to 1. This pre-processing quantity classification and determination mechanism enables the system to be compatible with processing data recording intervals with short operation intervals.

[0097] The process synthesis module 300 places the continuous video keyframe sequence in timestamp order within a sliding time window, calculates the pixel-level temporal variance of the time dimension for pixels at the same coordinate position in each image frame, and constructs a spatiotemporal variance matrix covering the global resolution size in memory based on the calculation results of each pixel. The process synthesis module 300 traverses each numerical element in the spatiotemporal variance matrix and compares it with a preset dynamic noise threshold. Matrix elements greater than the dynamic noise threshold are mapped to 0, while matrix elements less than or equal to the dynamic noise threshold are mapped to 1. Based on this, a binary mask matrix is ​​output. This matrix maps the set of dynamic pixel regions that change frequently within the current interaction cycle at the data structure level.

[0098] In this embodiment, the generation logic of each element of the binarized mask matrix follows the following mathematical mapping expression:

[0099] ;

[0100] In the formula, For the binary mask matrix in spatial coordinates The mask element value at the location; This is the horizontal coordinate index of a pixel within the image matrix; This is the vertical ordinate index of a pixel within the image matrix; This is the parameter representing the total number of frames within a continuous video keyframe sequence. It is a positive integer index of the corresponding image frame within a continuous video keyframe sequence; For the first Frame image frame at coordinate position The gray intensity value at that location; Coordinate position within the sliding time window The temporal average gray value of the pixel; The preset dynamic noise threshold is configured as an integer between 10 and 30.

[0101] S302, the process synthesis module 300 obtains the center coordinates of the target element hit by the current discrete manual operation event, and uses the center coordinates as a reference to expand outward to extract the target area image block at the corresponding position with a set pixel radius. The set pixel radius is configured to be 64px to 128px. If the boundary of the target area exceeds the physical resolution boundary of the screen during the extraction process, the process synthesis module 300 uses a zero-value filling mechanism to fill in the missing pixels to maintain the constant dimension of the image block matrix.

[0102] The process synthesis module 300 extracts the local mask slices that coincide with the physical location of the image block in the binarized mask matrix and simultaneously uses a zero-value filling mechanism to fill them to the same matrix dimension as the target region image block. Then, it performs a Hadamard product operation on the target region image block and the local mask slices to change the pixel values ​​of the dynamically changing state to 0.

[0103] The process synthesis module 300 performs grayscale dimensionality reduction and size reshaping on the result matrix, extracts low-frequency feature coefficients using the existing discrete cosine transform algorithm, calculates the mean of the sequence composed of each feature coefficient, and performs binary encoding quantization to output the region-aware steady-state hash value. The length of the region-aware steady-state hash value is configured to be 64 bits, 128 bits, or 256 bits. The low-frequency feature coefficients are selected from the upper left corner of the discrete cosine transform coefficient matrix of a preset size region. The Hamming distance convergence threshold is configured to be 5% to 20% of the hash length.

[0104] S303, the process synthesis module 300 extracts the full-screen directed acyclic graphs before and after the operation, namely the full-screen directed acyclic graph at the moment before the current operation event is triggered and the subsequent full-screen directed acyclic graph at the moment when the interface rendering is stable after the trigger, and extracts all discrete visual nodes and macro nodes in these two graphs; wherein, the moment when the interface rendering is stable is determined by the global perceptual hash Hamming distance between several consecutive frames being less than a preset stability threshold, which is configured to be 2% to 5% of the hash length, or the pixel-level temporal variance of several consecutive frames being less than a preset stability threshold, which is configured to be 5 to 15.

[0105] The process synthesis module 300 performs a bidirectional cross-graph traversal matching search based on the structured semantic labels of each node and the absolute physical coordinates of the center. During the search and matching, it calls the configured coordinate offset tolerance threshold. When the semantic labels of the nodes being compared in two graphs are consistent and the Euclidean distance between their center coordinates is less than the coordinate offset tolerance threshold, they are determined to be the same equivalent node. The coordinate offset tolerance threshold is configured to be between 5px and 10px. The system clusters nodes that existed before the operation but are missing after the operation into a set of missing nodes, and clusters nodes that were missing before the operation but are newly generated after the operation into a set of new nodes. Based on graph theory difference subtraction, the above two sets of nodes are calculated independently.

[0106] The process synthesis module 300 counts the total number of nodes contained in the set of disappeared nodes and the set of newly added nodes. When the total number of nodes exceeds the preset redirection refresh threshold, the system recognizes that the current operation triggers a full page jump state. At this time, the process synthesis module 300 extracts the macro node containing the core title text of the page and defines it as the global state verification benchmark node.

[0107] The core title text of the page uses the screen vertical position, bounding box area, character height, semantic category confidence, and the number of times it appears in consecutive stable frames as feature parameters. These parameters are combined with the corresponding feature weight factors preset by the system to perform a weighted summation calculation to obtain the title score. The text node with the highest title score or its associated macro node is determined as the global state verification benchmark node. The redirect refresh threshold is configured to be 50% of the total number of nodes contained in the benchmark full-screen directed acyclic graph before the operation is executed.

[0108] When the total number of nodes is within the redirection refresh threshold, the process synthesis module 300 performs memory-level data merging and encapsulation of the newly added node set, the disappearing node set, and the physical attribute parameters attached to both, generating an independent data structure block and defining its output as a semantic increment expectation matrix.

[0109] Each row in the semantic increment expectation matrix corresponds to an interface state change item, which includes at least the change type, node semantic label, node category, center coordinates, bounding box size, index of the macro node to which it belongs, node confidence, and change direction. The change type includes addition, disappearance, and attribute change. The output semantic increment expectation matrix is ​​saved as a discrete feature carrier in the controller's non-volatile storage area for subsequent execution engine calls.

[0110] See attached document Figure 5 The specific implementation of step S400 can be processed through the following sub-steps:

[0111] S401, the process synthesis module 300 accesses the memory space through the system's internal data bus, extracts the discrete manual operation events recorded in the time-series operation dataset, as well as the elastic topological semantic subgraph and region-aware steady-state hash value generated in the previous operation; the process synthesis module 300 parses the event type identifier and absolute physical coordinates in the discrete manual operation events, and combines the structured semantic tags of the corresponding target elements to instantiate the manual operation into an executable specific device action function. The underlying hardware interrupt simulation and system application programming interface call encapsulation operation of the specific device action function can be implemented according to existing computer automation control technology.

[0112] The process synthesis module 300 detects the type attribute of the instantiated action function. When the action function contains a keyboard string input instruction, the system calls the internally preset regular expression engine to match the business data dictionary. It performs rule matching on the string formed by continuous keyboard input events, clipboard pasted strings, or text content identified in the input box. The string that matches the rules of mobile phone number, date, amount, order number, name, address, or other business fields is replaced with the global parameter variable name with an independent memory address address pointer, forming a dynamic parameterized binding mapping mechanism for external environment input.

[0113] The process synthesis module 300 merges the elastic topological semantic subgraph with the region-aware steady-state hash value into a data structure to generate an independent structure as a precondition assertion library. This precondition assertion library is mapped in the model to the premise steady-state feature space in which the closed-loop state verification node is located before the transition is triggered. The constraint action function of the precondition assertion library is activated and executed only after it has been verified in the decision space.

[0114] S402, the process synthesis module 300 extracts the semantic increment expectation matrix and global state verification benchmark node stored in the non-volatile storage area, and transforms the semantic increment expectation matrix and global state verification benchmark node into a post-result assertion library triggered after the action is executed; the process synthesis module 300 allocates a contiguous memory block in the memory heap area to instantiate the closed-loop state verification node, and writes the precondition assertion library, action function and post-result assertion library into the data segment of the closed-loop state verification node in the order of temporal causality.

[0115] Among them, the precondition assertion library is used to verify whether the current interface is in an operable steady state before the action is executed, and the postcondition assertion library is used to verify whether the interface change meets the state conditions defined by the semantic increment expectation matrix after the action is executed; the process synthesis module 300 loads a ring retry logic control block inside the method segment of the closed-loop state verification node. The ring retry logic control block includes a preset maximum retry number variable and a waiting delay timer. The specific value of the maximum retry number variable is configured to be 3 to 5 times, and the delay period parameter of the waiting delay timer is configured to be 500 milliseconds to 2000 milliseconds.

[0116] The process synthesis module 300 mounts the circular retry logic control block as an independent attribute in the memory of the closed-loop state verification node. This lays the control foundation for the cyclic scheduling process after the action verification fails at the data structure level, enabling the system to build a local closed-loop structure within a single node.

[0117] S403, the process synthesis module 300 pushes the multiple currently generated closed-loop state verification nodes into the memory queue as vertices of a directed graph in sequence, and establishes directed connected edges between adjacent vertices according to the physical operation order in the discrete time dimension, forming a global verification link that maps external business logic.

[0118] The process synthesis module 300 calculates the confidence evaluation index of the transition execution of each node in the link, and evaluates the execution robustness of the overall automated link in memory based on the average confidence of each node. When the system determines that the link confidence meets the preset security standard threshold, it sends the global verification link to the internal script code generator, generates an automated execution script file for a specific high-level language system according to the preset script template, the closed-loop state verification node field, and the directed connection edges between nodes, and persists it to the underlying hard disk sector. The preset security standard threshold is configured to be between 0.75 and 0.90. When the link confidence is lower than the threshold, the system triggers an exception interception interruption, packages the current link data structure and throws it to the application layer to wait for manual review instructions, and completes the control flow closed loop of the exception evaluation branch.

[0119] In this embodiment, the calculation logic for the confidence evaluation index of node transition execution follows the following mathematical mapping expression:

[0120] ;

[0121] In the formula, For the first in the global verification chain The execution confidence calculation value corresponding to each closed-loop state verification node; The preset time delay attenuation weighting factor is configured with a value range of 0.3 to 0.6; For the first The absolute physical time difference between the precondition assertion library and the postcondition assertion library of each node during the original recording and acquisition stage. The maximum allowable service response tolerance value is preset within the system, specifically configured to be between 5000 milliseconds and 15000 milliseconds; The preset topology stability weight factor is configured with a value range of 0.4 to 0.7; For the first The total number of dynamic interference nodes contained in the elastic topological semantic subgraph corresponding to each node. The statistical rule for this total number is that when the proportion of the overlap area between the absolute physical coordinate region of an entity node and the dynamically changing pixel region in the binary mask matrix is ​​greater than a preset overlap threshold, it is marked as a dynamic interference node. The value of the preset overlap threshold is configured to be between 30% and 50%. For the first The total number of all entity nodes covered in the elastic topology semantic subgraph corresponding to each node; This is a positive integer index item for the closed-loop state verification node in the global verification link.

[0122] See attached document Figure 6 The specific implementation of step S500 can be handled through the following sub-steps:

[0123] S501, during the automated playback phase, the closed-loop execution module 400 parses the automated execution script file persistently written to the underlying hard disk sector and extracts the data segment of the currently pending closed-loop state verification node; the closed-loop execution module 400 reads the precondition assertion library contained in the closed-loop state verification node, loads the elastic topological semantic subgraph encapsulated in the precondition assertion library as the graph matching benchmark into the stack area of ​​the system memory, and at the same time reads the binarized mask matrix and the region-aware steady-state hash value in the precondition assertion library as the precondition verification basis for the current interface steady-state judgment and candidate subgraph selection;

[0124] The closed-loop execution module 400 calls the underlying graphics infrastructure interface to capture the real-time full-screen image buffer data of the current target terminal display. It uses a pre-deployed visual feature extraction model to perform feature tensor extraction and semantic label splicing operations on the real-time image buffer data. Based on the preset Manhattan distance and collinearity error calculation rules, it constructs a real-time global directed acyclic graph in memory that maps the current real-time interface.

[0125] S502, the closed-loop execution module 400 uses an approximate subgraph matching algorithm to perform a breadth-first search outward from each entity node in the real-time global directed acyclic graph, extracting multiple real-time local candidate subgraphs with the same connectivity order as the baseline elastic topological semantic subgraph; the closed-loop execution module 400 extracts the corresponding nodes of the baseline subgraph and each candidate subgraph, calculates the Levenstein edit distance based on the structured semantic label strings of the two and divides it by the maximum length of the two strings for normalization to obtain the text semantic similarity, and at the same time extracts the physical topological weights of each directed edge of the two and calculates the absolute difference, and then divides the absolute difference by the global maximum weight value to perform extreme value normalization mapping, thereby outputting the topological structure difference of the edge set.

[0126] The closed-loop execution module 400 performs a weighted difference calculation on the semantic similarity of node text and the normalized edge weight topological difference, with text similarity as the positive gain and topological difference as the negative penalty, to obtain the comprehensive approximate topological matching similarity score between the baseline subgraph and each real-time local candidate subgraph, and generates a one-dimensional mapping array that records the score parameters of each candidate subgraph.

[0127] In this embodiment, the mathematical calculation logic for the topological isomorphic mapping similarity score utilizes the string editing cost and the graph edge weight deviation to construct an isomorphic mapping model, specifically following the following mathematical mapping expression:

[0128] ;

[0129] In the formula, For the baseline elastic topological semantic subgraph With real-time local candidate subgraphs Topological isomorphic mapping similarity score between them; For the extracted baseline elastic topological semantic subgraph; For the extracted real-time local candidate sub-images; The preset semantic feature tendency adjustment factor is configured with a value range of 0.6 to 0.8; This represents the total number of entity nodes contained within the comparison subgraph; For comparing the positive integer indexes of entity nodes within the subgraph; For the baseline subgraph The node label and the candidate subgraph The normalized Lewinstein edit distance value calculated from the node labels; For the baseline elastic topological semantic subgraph The Middle Structured semantic tags for each node; Real-time local candidate subgraphs The Middle Structured semantic tags for each node; This represents the total number of directed connected edges contained within the comparison subgraph; For comparison, use positive integer indices of directed connected edges within the subgraph; For the baseline elastic topological semantic subgraph The Middle The physical topological weight parameters of a directed edge; Real-time local candidate subgraphs The Middle Physical topology weight parameters for a directed edge.

[0130] S503, the closed-loop execution module 400 addresses candidate subgraphs in the one-dimensional mapping array whose similarity scores to topological isomorphic mappings are greater than a preset matching threshold and pushes them into the candidate memory set. The preset matching threshold is configured to be between 0.80 and 0.95. When the candidate memory set is empty, the system suspends the current execution thread and triggers the interface retry frame capture action until the maximum number of retryes set by the node is exceeded, at which point an addressing exception interrupt signal is thrown. This mechanism constitutes a fault-tolerant closed loop for operation in dynamic interface loading scenarios.

[0131] When the candidate memory set contains multiple independent candidate subgraph objects, the closed-loop execution module 400 extracts the region-aware steady-state hash value encapsulated in the precondition assertion library, and extracts the coordinates of the center target node in each candidate subgraph as a reference point to expand outward and capture real-time image blocks. At the same time, it performs a Hadamard product operation on the real-time image block in combination with the binarized mask matrix in the precondition assertion library to remove dynamic interference pixels, thereby generating the real-time feature hash value of the physical region corresponding to each candidate subgraph. Then, it calculates the Hamming distance between the region-aware steady-state hash value and the real-time feature hash value of the physical region corresponding to each candidate subgraph. The system selects the candidate subgraph with the smallest Hamming distance value that is less than the preset hash convergence threshold and confirms it as the approximate topological matching target subgraph, relying on multi-dimensional feature dimensions to eliminate spatial positioning interference of redundant similar components in the interface.

[0132] S504, the closed-loop execution module 400 extracts the central target node in the above isomorphic mapping target subgraph and parses the absolute physical coordinate parameters of the node in the current real-time screen physical resolution coordinate system; the closed-loop execution module 400 overwrites and replaces the newly obtained absolute physical coordinate parameters in the memory parameter list of the original instantiated specific device action function, and calls the application programming interface at the operating system level to send the corresponding artificial simulated hardware interrupt event to the peripheral driver.

[0133] After the hardware interrupt event is completed, the closed-loop execution module 400 extracts the post-result assertion library from the closed-loop state verification node data segment, combines it with the real-time frame data of the interface after the action is triggered to extract the actual semantic node of the current interface, and performs cross-comparison with the semantic increment expectation matrix to perform multimodal state differential comparison calculation. Before performing state differential comparison calculation, if the post-result assertion library contains a global state verification benchmark node, the system first traverses and addresses anchor point objects in the real-time frame data that match the semantic label and physical bounding box features of the global state verification benchmark node. If the addressing is successful and the physical coordinate offset is within the tolerance range, the subsequent differential calculation process continues.

[0134] If addressing fails or physical coordinate offset exceeds the tolerance range, the system directly determines that the current post-state verification has failed and triggers the subsequent polling suspension mechanism. When the state difference calculation result and the semantic increment expectation matrix reach the preset post-verification matching threshold at the numerical feature level, the current closed-loop state verification node flows into the success state and stores the closed-loop state verification node to complete the automatic generation of the no-code RPA process. The value range of the preset post-verification matching threshold is configured to be 0.85 to 0.95. When the state difference calculation result does not reach the preset post-verification matching threshold, the closed-loop execution module 400 triggers polling suspension according to the waiting delay parameter configured in the closed-loop state verification node, and re-extracts the cross-comparison of the current interface's actual semantic node execution post-state.

[0135] If the polling comparison reaches the maximum retry limit and the post-verification still fails, the system will call the precondition assertion library again. Only if the current interface still meets the precondition will the system re-execute the target addressing and action playback process; otherwise, a business flow abnormal interruption signal will be thrown.

[0136] To aid in understanding the technical solution of this invention, the following provides a specific application example and test case for automatic generation of no-code RPA processes based on screen semantic recognition.

[0137] The specific application example is set as a scenario of automated extraction of financial statements and automatic filling of online forms across business systems. In the target system operating environment, the front-end software includes an old enterprise resource planning client without an open application programming interface and a financial shared service center web interface inside a remote desktop container.

[0138] During the data acquisition phase, the data acquisition module 100 monitors the display output and external input devices in an independent background process. When an operator clicks the login input box of the Enterprise Resource Planning (ERP) client with a mouse, keyboard key events form a continuous string, and the operator enters the password information. The data acquisition module 100 obtains a two-dimensional pixel matrix of the screen at a fixed frame rate. Combining this with the global system timestamp, the module performs a time-axis alignment mapping operation on the two-dimensional pixel matrix with the mouse click events and keyboard input events, thereby constructing a time-series computation dataset in a discrete time dimension. The operation of the data acquisition module 100 does not depend on the underlying control tree of the ERP client.

[0139] In the topology modeling stage, the topology modeling module 200 performs feature tensor extraction on the keyframe pixel matrix recorded in the time-series computation dataset. The visual feature extraction model outputs the absolute physical coordinates and semantic labels of the username text component, input box component, and login button component included in the login interface. For the historical bill list presented by the enterprise resource planning client interface with dual-axis equidistant distribution, the topology modeling module 200 performs local grid pooling to construct independent macro nodes. The topology modeling module 200 calculates the Manhattan distance and collinearity error between each discrete visual node, generating a full-screen directed acyclic graph containing directed edges between nodes and a locally connected elastic topological semantic subgraph, converting the spatial relative layout relationship of the components into a graph theory data structure.

[0140] The process synthesis module 300 processes the continuous video keyframe sequence between discrete manual operation events. If the login interface has a high-frequency flashing announcement marquee component, the process synthesis module 300 calculates the pixel-level temporal variance in the time dimension. If the temporal variance corresponding to the marquee area is greater than the set dynamic noise threshold, the system maps the matrix elements of the marquee area to 0 when constructing the binary mask matrix.

[0141] The process synthesis module 300 performs Hadamard product operation on the target region image block and the corresponding local mask slice to generate a region-aware steady-state hash value to eliminate optical jump interference from the dynamic marquee component. After the operator clicks the login button to trigger the page jump action, the process synthesis module 300 compares the state difference of the full-screen directed acyclic graph node set before and after the operation, and outputs the state variables containing the attributes of newly added nodes and the attributes of disappeared nodes as a semantic increment expectation matrix.

[0142] During the process replay phase, the closed-loop execution module 400 extracts the automatically generated script file from the process, captures real-time image data, calculates the region-aware steady-state hash value of the current interface based on the binarized mask matrix encapsulated in the precondition assertion library, and performs an approximate subgraph matching algorithm in the full topology map of the real-time interface using the benchmark elastic topological semantic subgraph.

[0143] Based on the node text features, the Levenstein edit distance is calculated and the edge weight difference is combined for addressing. The system determines the coordinates of the matching node with the minimum mapping cost and sends a simulated mouse click event to the matching node coordinates. After the instruction is executed, the closed-loop execution module 400 extracts the actual semantic nodes of the current interface and cross-compares them with the semantic increment expectation matrix to determine whether the interface has completed the corresponding state transition action.

[0144] The system test cases are designed to quantitatively verify the fault-tolerant addressing performance of the system under multiple nondeterministic interface variations. The test baseline process includes the operation of clicking multiple input boxes and submit command components in a specified data entry form. In the playback execution environment, periodically switching dynamic image interference sources are actively injected into the corresponding coordinate area of ​​the target interface, and the front-end layout cascading style sheet of the target interface is modified to cause the overall structure of the target form to be reconstructed from a two-column layout to a single-column layout. The single-column layout change causes a large amount of offset in the vertical physical absolute coordinates of the original target components.

[0145] During the testing and verification, the closed-loop execution module 400 loaded the previously constructed binarized mask matrix to mask the pixel values ​​of the injected dynamic image interference source regions. The Hamming distance comparison logic of the region-aware steady-state hash value was not affected by the interference source, and the system successfully passed the pre-state condition verification. For the vertical absolute coordinate offset, the closed-loop execution module 400 abandoned the traditional physical coordinate alignment method and traversed the real-time global directed acyclic graph to extract real-time local candidate subgraphs. The change of the absolute physical coordinates of the components did not destroy the logical reading order and relative connectivity between nodes. The Manhattan distance ratio and physical topology weight parameter corresponding to the real-time local candidate subgraph remained constant. The closed-loop execution module 400 completed the addressing and positioning based on the topological isomorphic mapping similarity score and refreshed the execution coordinate list. The system did not throw an addressing exception interruption signal.

[0146] The specific test results are shown in the table below.

[0147] Statistical test data table of addressing success rate of automated addressing model under multi-dimensional variation interference scenarios

[0148] Categories of Automated Modeling Methods Addressing success rate in static environments with no change in resolution Interface layout reconstruction and overall component coordinate offset scene addressing success rate Remote Desktop Protocol and Documentless Object Model Environment Addressing Success Rate Success rate of scene addressing in implanted periodic dynamic optical interference areas Based on physical absolute coordinates and static image template matching model 91.3% 14.8% 89.2% 38.6% Based on the document object model parsing model of the business system 97.5% 93.1% 0.0% 88.4% Screen semantic recognition and flexible topology modeling matching system 98.7% 96.4% 95.8% 97.1%

[0149] According to the table above and appendix Figure 7 The data shows that the addressing success rate of the physical absolute coordinate and static image template matching model decreased to 14.8% in the scenario of interface layout reconstruction and overall component coordinate offset. The addressing success rate of the business system underlying document object model parsing model was 0.0% in the scenario of remote desktop protocol and no document object model. The addressing success rate of the screen semantic recognition and elastic topology modeling matching system was maintained at 96.4% in the scenario of interface layout reconstruction, 95.8% in the scenario of no document object model, and 97.1% in the scenario of implanted dynamic interference area.

[0150] The above data demonstrates that the data acquisition module 100 and the visual feature extraction model extract component categories and string values ​​by directly calculating the two-dimensional pixel matrix, eliminating the dependency of the automated process on the target software's underlying interface. The topology modeling module 200 constructs an elastic topological semantic subgraph, converting absolute physical coordinates into spatial connectivity weight features. The system introduces structural and text editing distance constraints in the approximate subgraph matching algorithm to overcome the positioning failure problem caused by resolution scaling and component displacement. For dynamic image interference, the process synthesis module 300 uses temporal variance to generate a binary mask matrix to isolate high-frequency pixel jump regions, and combines it with the region-aware steady-state hash value for initial screening and filtering to reduce the optical disturbance amplitude of the interface environment.

[0151] Meanwhile, the closed-loop execution module 400 encapsulates the precondition assertion library, semantic increment expectation matrix and circular retry logic within the autonomous closed-loop state machine node object, providing a recovery mechanism for asynchronous rendering with network latency. This invention relies on visual topological semantics and time dimension variance to establish a closed-loop state machine link, and completes the automatic generation and addressing control of anti-interference processes across business platforms.

[0152] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A no-code RPA process automatic generation method based on screen semantic recognition, characterized in that, The method includes the following steps: A time-series computation dataset is constructed by using a two-dimensional pixel matrix, discrete manual operation events, and corresponding timestamps. The time-series computation dataset includes a keyframe pixel matrix and a sequence of consecutive video keyframes divided according to adjacent operation events. Visual features are extracted from the keyframe pixel matrix in the time-series computation dataset to generate discrete visual nodes, and macro nodes are constructed through local grid pooling. The Manhattan distance and collinearity error between discrete visual nodes and macro nodes are calculated to construct directed edges. Combined with the location of discrete manual operation events, an elastic topological semantic subgraph and a full-screen directed acyclic graph are generated. The pixel-level temporal variance of a continuous video keyframe sequence is calculated to generate a binary mask matrix. The region-aware steady-state hash value of the target region is extracted by combining the binary mask matrix. The full-screen directed acyclic graph before and after the operation is obtained. Graph theory difference subtraction is performed on the full-screen directed acyclic graph before and after the operation to generate a semantic increment expectation matrix. The region-aware steady-state hash value, the binarized mask matrix, the semantic increment expectation matrix, and the elastic topological semantic subgraph are encapsulated into a closed-loop state verification node. The approximate subgraph matching algorithm is used to locate candidate subgraphs that match the elastic topological semantic subgraph in the current interface, and the real-time feature hash value of the corresponding region is extracted by combining the binarized mask matrix. When the Hamming distance between the real-time feature hash value and the region-aware steady-state hash value reaches the convergence threshold, the physical coordinates of the central target node are output and the operation command is sent. After the command is executed, if the current actual semantics meet the post-state conditions of the semantic increment expectation matrix, the closed-loop state verification node is stored to complete the automatic generation of the no-code RPA process.

2. The no-code RPA process automatic generation method based on screen semantic recognition according to claim 1, characterized in that, The step of constructing a time-series computation dataset from a two-dimensional pixel matrix, discrete manual operation events, and corresponding timestamps includes: Calculate the difference between the system timestamp of the discrete manual operation event and the system timestamp of each of the two-dimensional pixel matrices; In the two-dimensional pixel matrix where the difference is within the preset time synchronization tolerance, the two-dimensional pixel matrix whose system timestamp is closest to the trigger time of the discrete manual operation event is extracted and defined as the keyframe pixel matrix; The keyframe pixel matrix is ​​fused with the associated discrete manual operation events and system timestamps according to discrete time sequence markings, and the two-dimensional pixel matrix collected between two adjacent operation events is cached as the continuous video keyframe sequence.

3. The no-code RPA process automatic generation method based on screen semantic recognition according to claim 1, characterized in that, The step of extracting visual features from the keyframe pixel matrix in the time-series computation dataset to generate discrete visual nodes, and constructing macro nodes through local grid pooling includes: The keyframe pixel matrix is ​​input into the visual feature extraction model, which outputs the discrete visual nodes with bounding box center coordinates and semantic labels. When a local discrete visual node is identified to be equidistant from a single axis or a dual axis, the global extremum boundary formed by the discrete visual nodes that are equidistant from a single axis or a dual axis is extracted to generate an outer bounding box as the macro node. A two-dimensional index matrix mapping absolute physical coordinates is established inside the macro node, and the discrete visual nodes are mapped and filled into the corresponding coordinate positions to construct the macro node.

4. The no-code RPA process automatic generation method based on screen semantic recognition according to claim 1, characterized in that, The steps of calculating the Manhattan distance and collinearity error between discrete visual nodes and macro nodes to construct directed edges, and combining the location of discrete manual operation events to generate an elastic topological semantic subgraph and a full-screen directed acyclic graph include: The starting and ending points of directed edges between node pairs are determined according to the preset scanning order from left to right and from top to bottom. The physical topological weight of each directed edge is calculated by combining the Manhattan distance and the collinearity error, and a preset number of directed edges are retained in order of increasing physical topological weight to generate the full-screen directed acyclic graph. Taking the target node corresponding to the location of the discrete manual operation event as the center starting point, a breadth-first search algorithm is executed according to the connection direction of the directed edges to extract all associated nodes and connecting edges within a set connection order, and the extracted local network topology is defined as the elastic topological semantic subgraph.

5. The no-code RPA process automatic generation method based on screen semantic recognition according to claim 1, characterized in that, The step of calculating the pixel-level temporal variance of a continuous video keyframe sequence to generate a binary mask matrix, and then combining the binary mask matrix to extract the region-aware steady-state hash value of the target region includes: Traverse each numerical element in the matrix formed by the pixel-level temporal variance, map matrix elements greater than the dynamic noise threshold to 0, and map matrix elements less than or equal to the dynamic noise threshold to 1 to output the binarized mask matrix. Using the center coordinates of the target element as a reference, the target region image block is cropped outward, and the corresponding local mask slice in the binarized mask matrix is ​​extracted. Perform a Hadamard product operation on the target region image block and the local mask slice to change the pixel value of the dynamically changing state to 0; The low-frequency feature coefficients are extracted from the result matrix using the discrete cosine transform algorithm. The mean of the sequence formed by each feature coefficient is calculated and binary encoded and quantized to output the region-aware steady-state hash value.

6. The no-code RPA process automatic generation method based on screen semantic recognition according to claim 1, characterized in that, The steps of obtaining the full-screen directed acyclic graph before and after the operation, and performing graph theory difference subtraction calculation on the full-screen directed acyclic graph before and after the operation to generate the semantic increment expectation matrix include: Based on the structured semantic labels of the nodes in the full-screen directed acyclic graph and their center absolute physical coordinates, a bidirectional cross-graph traversal matching search is performed. Using graph theory difference operations, we can cluster the set of missing nodes that existed before the operation and are missing after the operation, as well as the set of newly generated nodes that were missing before the operation and are newly generated after the operation. The newly added node set, the disappeared node set, and their associated physical attribute parameters are merged and encapsulated at the memory level to generate the semantic increment expectation matrix.

7. The no-code RPA process automatic generation method based on screen semantic recognition according to claim 1, characterized in that, The step of encapsulating the region-aware steady-state hash value, binarized mask matrix, semantic increment expectation matrix, and elastic topological semantic subgraph into a closed-loop state verification node includes: The elastic topological semantic subgraph, the binarized mask matrix, and the region-aware steady-state hash value are merged into a data structure to generate an independent structure that serves as a precondition assertion library. The macro node containing the core title text of the page is defined as the global state verification benchmark node, and is converted into a post-result assertion library along with the semantic increment expectation matrix. A contiguous memory block is allocated in the memory heap area to instantiate the autonomous closed-loop state machine node object. The precondition assertion library, action function, postcondition assertion library, and ring retry logic control block containing the preset maximum number of retries and the waiting delay timer are written in the order of time causality to generate the closed-loop state verification node.

8. The method for automatic generation of no-code RPA processes based on screen semantic recognition according to claim 1, characterized in that, The approximate subgraph matching algorithm is used to address candidate subgraphs that match the elastic topological semantic subgraphs on the current interface, and the real-time feature hash values ​​of the corresponding regions are extracted by combining the binarized mask matrix. When the Hamming distance between the real-time feature hash value and the region-aware steady-state hash value reaches the convergence threshold, the steps of outputting the physical coordinates of the central target node and sending operation instructions include: Extract multiple real-time local candidate subgraphs with the same connectivity order as the elastic topological semantic subgraph from the real-time global directed acyclic graph. The Levenstein edit distance is calculated based on the structured semantic tag strings to obtain the text semantic similarity. At the same time, the physical topological weights of each directed edge are extracted and the absolute difference is calculated. Then, the absolute difference is divided by the global maximum weight value to perform extreme value normalization mapping, and the topological structure difference of the edge set is output accordingly. The semantic similarity of node text and the topological difference of normalized edge weights are calculated by positive and negative weighting. The semantic similarity of text is used as a positive gain and the topological difference is used as a negative penalty to address candidate subgraphs whose comprehensive approximate topological matching similarity scores are greater than the preset matching threshold. The coordinates of the central target node in each candidate sub-image are extracted as the reference point to expand outward and capture real-time image blocks. The binarized mask matrix is ​​combined with the Hadamard product operation to remove dynamic interference pixels and generate real-time feature hash values ​​of the physical regions corresponding to each candidate sub-image. The candidate subgraph with the smallest Hamming distance value that is less than the preset hash convergence threshold is selected. The absolute physical coordinate parameters of the central target node in the subgraph are extracted as the physical coordinates. The application programming interface at the bottom layer of the operating system is called to send the corresponding artificial simulated hardware interrupt event to the peripheral driver as the operation instruction.

9. The no-code RPA process automatic generation method based on screen semantic recognition according to claim 1, characterized in that, After the instruction is executed, if the current actual semantics satisfy the post-state conditions of the semantic increment expectation matrix, then storing the closed-loop state verification node to complete the steps of automatically generating the no-code RPA process includes: By combining real-time frame data of the interface after the action is triggered, the actual semantic nodes of the current interface are extracted and cross-compared with the semantic increment expectation matrix. Multimodal state difference comparison calculation is performed to obtain the state difference calculation result. When the state difference calculation result and the semantic increment expectation matrix reach a preset post-verification matching threshold at the numerical feature level, the current closed-loop state verification node transitions to the success state and stores the closed-loop state verification node. When the state difference calculation result does not reach the preset post-verification matching threshold, the system triggers polling suspension based on the waiting delay parameter configured in the closed-loop state verification node. If the polling comparison count reaches the maximum retry count limit and the post-verification is still not passed, the system calls the precondition assertion library again. Only when the current interface still meets the precondition will the system re-execute the target addressing and action playback process.

10. The no-code RPA process automatic generation method based on screen semantic recognition according to claim 9, characterized in that, The step of extracting the actual semantic nodes of the current interface and performing multimodal state difference comparison calculation with the semantic increment expectation matrix includes: Before performing the state difference comparison calculation, if the post-result assertion library contains a global state verification benchmark node, the system will preferentially traverse and address the anchor object that matches the semantic label and physical bounding box feature of the global state verification benchmark node in the real-time frame data. Once the anchor point object is successfully addressed and the physical coordinate offset is within the tolerance range, the subsequent differential calculation process will continue. If addressing fails or physical coordinate offset exceeds the tolerance range, the system directly determines that the current post-state verification has failed and triggers the subsequent polling suspension mechanism.

Images