Desktop Element Localization Method and System Based on Dynamic Skip-Level Matching and CV Technology

Abstract

The present invention belongs to the technical field of RPA, and particularly relates to a desktop element positioning method and system based on dynamic skip-level matching and CV technology. The method includes: S1, when using RPA technology to locate and pick up the structural information of elements, simultaneously capture the interface image of the application at the current time point, and use computer vision technology to extract the visual data information for positioning the element position; S2, during the process execution, combine RPA technology and computer vision technology to perform element positioning and picking, and extract the structural information of the elements; S3, introduce a skip-level matching mechanism, and automatically identify and set skip-level nodes by dynamically analyzing the changes in the element structural information to achieve the positioning of the target element. The present invention can solve the limitations of traditional RPA technology in a dynamic environment, can also significantly improve the robustness and accuracy of element positioning, and has the characteristics of high efficiency and flexibility.

Description

Technical Field

[0001] The present invention belongs to the technical field of RPA, and particularly relates to a desktop element positioning method and system based on dynamic skip-level matching and CV technology. Background Art

[0002] RPA (Robotic Process Automation) is a way of using "digital employees" to replace humans in business operations and its related technologies. Essentially, RPA is a software automation technology that simulates human operations on objects such as systems, software, web pages, and documents on a computer to achieve unmanned operation, obtain business information, execute business actions, and ultimately achieve automated process processing, save labor costs, and improve processing efficiency.

[0003] One of the core technologies of RPA is the positioning and picking of operation elements. For example, when it is necessary to simulate a human button click action, the position of the button element must be located first. Traditional element positioning technologies usually extract and save the underlying page structure information to a computer by means of API interfaces or page source code parsing. When executing the process, the system will locate the element again according to the saved structure information and perform simulation operations.

[0004] In the field of RPA, CV (Computer Vision) technology is widely used in element positioning and picking. Common methods include two technologies: object detection and OCR (Optical Character Recognition).

[0005] In addition, Windows UI Automation (UIA) is a technology for desktop application automation, aiming to provide a standardized way for automation tools to access and operate the user interface elements of desktop applications.

[0006] The advantage of UIA technology is that it can directly interact with desktop applications compatible with the Windows operating system and supports the parsing and operation of complex interfaces.

[0007] However, the above existing RPA robot element positioning and picking methods and related technologies have the following disadvantages:

[0008] 1. After the element structure changes slightly, the element cannot be located

[0009] Traditional RPA technology extracts and saves the structural information of elements by parsing the interface layout and code, so that the process can re-locate the elements according to the saved structural information for simulation operations during execution. The element structure information usually includes all element levels from the window to the target element, as well as the corresponding feature information for each level. However, when there are minor changes in the structure of the picked-up element, such as changes in the number of element levels, changes in the element features of a certain level, or changes in the features of the element itself, etc., the saved element structure information will become invalid, resulting in the failure of the process execution. In this case, even if there are no major visual changes in the interface, manual re-location of the picked-up element is required, increasing the maintenance cost and labor input.

[0010] 2. Limitations of element operations based on computer vision

[0011] The element location and picking technology based on computer vision is essentially achieved by combining the recognition of the application interface image with the simulation of keyboard and mouse operations. However, when the functions to be operated cannot be completed through the keyboard and mouse mode, the computer vision-based method cannot meet the usage requirements. For example, functions such as obtaining or setting the attribute information, style format of elements, or triggering the events of elements cannot be achieved through computer vision. In addition, computer vision technology has high requirements for conditions such as image clarity, resolution, and ambient light, and depends on the accuracy of the model and the quality of the training data, which further limits its application in complex scenarios.

[0012] 3. Limitations of Windows UI Automation (UIA) technology

[0013] Although Windows UI Automation (UIA) technology can provide a standardized way of element access and operation for desktop application automation, there are also significant limitations in practical applications. UIA parses the structure of desktop applications, generates a DOM tree and extracts the attribute information of the target element and its parent link to achieve element location and operation. However, when the interface layout of the desktop application changes, or the hierarchical structure or attributes of the target element are adjusted, the original element structure information may become invalid, resulting in the abnormal execution of the process. In addition, when dealing with dynamically changing interfaces, UIA technology needs to frequently update the element structure information, which not only increases the development and maintenance costs, but also places high requirements on the user's ability to respond in a timely manner.

[0014] 4. Limitations of the skip-level matching function in desktop element matching

[0015] Existing RPA element location technologies usually rely on strict hierarchical matching and lack the function of skipping levels for matching. Although Windows UI Automation (UIA) provides the ability to traverse and match descendant nodes in the Control View, this ability is limited to the Control View and does not apply to the Raw View. The Raw View is a representation in UIA that is closer to the actual interface structure. It closely follows the native programming structure of the application and is therefore the most detailed available view and the most commonly used view in the RPA process. However, the Raw View does not support skipping-level matching, which limits the flexibility and robustness of RPA technology in desktop applications. For example, when the interface layout of a desktop application changes, or the properties or hierarchical structure of a certain parent element are adjusted, the traditional hierarchical matching method will not be able to adapt to this dynamic change, resulting in process interruption. To solve this problem, users often need to add a large amount of judgment logic to the RPA process. For example, use two structures to respectively find any method that meets the requirements, or even introduce computer vision technology to replace the traditional RPA element location technology. Although these methods can solve the problem to a certain extent, they will undoubtedly significantly increase the development cost and maintenance cost.

[0016] 5. Limitations of Existing Intelligent Repair Solutions

[0017] Although existing intelligent repair solutions combine traditional RPA technology and computer vision technology, there are obvious deficiencies in their repair mechanisms during the process execution. The specific manifestations are as follows:

[0018] Single repair method: After the computer vision module in the existing solution repairs the position, it directly saves the XPath structure of the new element, lacking the comparative analysis of the information and structure information of the old and new elements. This results in a low accuracy rate in the repair process and is difficult to adapt to complex and changing interface structure changes.

[0019] Limited applicability: Due to the lack of in-depth analysis of element structure changes, the existing solutions have poor adaptability in the face of different application scenarios. Users cannot participate in modifying the skipping-level structure, lacking flexibility.

[0020] Therefore, it is very important to design a desktop element location method and system based on dynamic skipping-level matching and CV technology that can solve the limitations of traditional RPA technology in a dynamic environment, significantly improve the robustness and accuracy of element location, and have the characteristics of high efficiency and flexibility. Summary of the Invention

[0021] The present invention aims to overcome the problem in the prior art that the existing RPA robot element positioning and picking method relies on fixed element structure information for element positioning, and once the element hierarchy or attributes change, the positioning fails. A desktop element positioning method and system based on dynamic skip-level matching and CV technology are provided, which can solve the limitations of traditional RPA technology in a dynamic environment, significantly improve the robustness and accuracy of element positioning, and have the characteristics of high efficiency and flexibility.

[0022] To achieve the above invention purpose, the present invention adopts the following technical solutions:

[0023] A desktop element positioning method based on dynamic skip-level matching and CV technology includes the following steps:

[0024] S1. When using RPA technology to position and pick up element structure information, capture the interface image of the application at the current time point, and use computer vision technology to extract the visual data information for positioning the element position.

[0025] S2. During the process execution, combine RPA technology and computer vision technology for element positioning and picking, and extract the element structure information.

[0026] S3. Introduce a skip-level matching mechanism, automatically identify and set skip-level nodes by dynamically analyzing the change of element structure information, and achieve target element positioning.

[0027] Preferably, in step S1, associate the extracted visual data information with the element structure information picked up by RPA technology for data, and store it in the computer.

[0028] Preferably, step S2 includes the following steps:

[0029] S21. First use RPA technology for element positioning.

[0030] S22. If the element cannot be positioned using RPA technology, the system automatically switches to computer vision technology, analyzes the interface content picked up by computer vision technology according to the stored visual data information, and at the same time converts the analysis result into a target element recognizable by RPA technology, and finally obtains the position information of the target element.

[0031] S23. According to the position information of the target element, combine RPA technology to position and pick up the element at the corresponding position in real time, and extract the element structure information.

[0032] Preferably, step S3 includes the following steps:

[0033] S31. Compare the extracted element structure information with the original element structure information, and judge the nodes with differences by analyzing the node hierarchy and node attribute information; the nodes with differences include the changed nodes, redundant nodes, and missing nodes.

[0034] S32. For the nodes with differences, set the nodes that meet the preset judgment logic as skip-level nodes, and at the same time generate new skip-level element structure information; the nodes that do not meet the requirements remain unchanged.

[0035] Preferably, step S3 further includes the following steps:

[0036] S33. When the element hierarchy structure changes, the system skips the skip-level nodes and directly locates to the target element.

[0037] Preferably, in step S3, the user can actively set skip-level nodes according to actual needs when editing the element structure.

[0038] Preferably, the skip-level element structure information includes non-skip-level structure information and skip-level structure information;

[0039] The non-skip-level structure information refers to the set of all continuously enabled hierarchical structures starting from the first-level node of the element structure; the non-skip-level structure information is used to locate all non-skip-level elements and serve as the starting point for skip-level matching.

[0040] The skip-level structure information refers to the set of hierarchical structures starting from the first set skip-level hierarchical structure to the last enabled hierarchical structure; the skip-level structure information is used to guide the skip-level matching process; the skip-level matching process includes that when encountering dynamically changing or unavailable nodes, the system will skip the corresponding nodes according to the skip-level structure information and directly locate to the target element.

[0041] The present invention also provides a desktop element positioning system based on dynamic skip-level matching and CV technology, including:

[0042] An element structure information acquisition module, which is responsible for capturing and extracting the element structure information in the desktop application;

[0043] A computer vision calculation module, which is responsible for processing the interface image captured by the element structure information acquisition module and extracting the visual data information for element positioning;

[0044] An element structure information update module, which is responsible for dynamically updating the element structure information during the process execution, that is, automatically setting skip-level nodes;

[0045] An element full-link positioning module, which is responsible for using RPA technology for element positioning and operation during the process execution;

[0046] The element skip-level positioning module is used to automatically identify and set skip-level nodes by analyzing the differences in the structure information of new and old elements, and skip the changed nodes during the layer-by-layer matching process to directly locate the target element.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention proposes an integrated solution combining traditional RPA technology, computer vision technology, and a dynamic skip-level matching mechanism; by dynamically analyzing the changes in element structure information, the present invention can automatically identify and set skip-level nodes, so that the target element can still be accurately matched when the element hierarchical structure changes; (2) The present invention also supports users to manually set skip-level matching nodes according to actual needs, further enhancing the flexibility and adaptability of the solution of the present invention; (3) The present invention not only solves the limitations of traditional RPA technology in a dynamic environment, but also significantly improves the robustness and accuracy of element positioning, providing an efficient and flexible solution for the field of desktop automation. Brief Description of the Drawings

[0048] Figure 1 It is a schematic diagram of the principle framework of the desktop element positioning method and system based on dynamic skip-level matching and CV technology of the present invention;

[0049] Figure 2 It is a flowchart of traditional RPA element skip-level positioning;

[0050] Figure 3 It is a schematic diagram of the classification of skip-level element structures in the present invention. Detailed Embodiments

[0051] In order to more clearly illustrate the embodiments of the present invention, the specific embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts, and other embodiments can also be obtained.

[0052] As Figure 1 shown, the present invention provides a desktop element positioning method based on dynamic skip-level matching and CV technology, and the specific process is as follows:

[0053] First, when using traditional RPA technology to locate and pick up element structure information, capture the interface image of the application at the current time point, and use the element picking technology based on computer vision to extract the computer vision data information that can accurately locate the element position. Associate the extracted computer vision data information with the element structure information picked up by traditional RPA technology and store it in the computer.

[0054] Secondly, during the process execution, the traditional RPA technology is preferentially used to locate the element based on the captured element structure information. When the traditional RPA technology fails to locate the element using the captured element structure information, the associated computer vision data information is used in combination with computer vision technology to calculate the position of the element in the interface. If the element only undergoes changes in structure, text, or style, but the overall interface layout remains unchanged, the position information of the target element can surely be obtained.

[0055] Finally, according to the position information calculated by computer vision technology, combined with the traditional RPA technology, the element at this position is located and captured in real time, and the structure information of the element is extracted. Subsequently, the extracted element structure information is compared with the original element structure information. By analyzing information such as node hierarchy and node attributes, it is determined which nodes have changed, which are redundant, and which are missing. For the nodes with differences, the nodes that meet the preset judgment logic will be set as skip-level nodes, and the nodes that do not meet the requirements will remain unchanged. During the layer-by-layer matching process, when a skip-level node is encountered, the skip-level matching will be performed according to the method of the present invention.

[0056] In addition, the user can also actively set skip-level nodes when editing the element structure according to their own observations. During the layer-by-layer matching process, when a skip-level node is encountered, the skip-level matching is also performed according to the method of the present invention. This function not only improves the flexibility and robustness of the RPA process, but also reduces the maintenance cost and development complexity caused by interface changes. Aiming at the limitations of the existing intelligent repair solutions, the present invention not only improves the repair accuracy by comparing the old and new element structure information, but also enhances the applicability and user participation of the solution. The user can actively set skip-level nodes to further optimize the matching process.

[0057] In summary, the present invention proposes a new solution for the deficiencies of the existing RPA technology in desktop element automation. By dynamically analyzing the changes in element structure information, the present invention can automatically identify and set skip-level nodes, so that when the element hierarchy structure changes, the target element can still be accurately matched. In addition, the user can also actively set skip-level nodes when editing the element structure according to actual needs, further enhancing the flexibility and robustness of the process.

[0058] During the process execution, traditional RPA technology relies on fixed element structure information for element positioning. Once the element hierarchy or attributes change, it will lead to positioning failure. However, the present invention introduces a skip-level matching mechanism, which can identify and skip the changed nodes in advance during the element structure positioning process, and directly locate the target element. This dynamic skip-level matching method not only improves the positioning accuracy but also reduces the risk of process interruption caused by interface changes. At the same time, users can actively set the skip-level nodes according to actual needs to further optimize the matching process.

[0059] In addition, for another example Figure 1 As shown, the present invention also provides a desktop element positioning system based on dynamic skip-level matching and CV technology, which mainly includes the following five parts:

[0060] 1. Element Structure Information Acquisition Module

[0061] The element structure information acquisition module is the basic module of the present invention, responsible for capturing and extracting the element structure information in the desktop application. This module uses traditional RPA technology to parse the DOM tree of the desktop application, extracts the attribute information of the target element and its parent link, and generates serializable element structure information. At the same time, this module is also responsible for capturing the application interface image at the current time point and associating it with the element structure information for storage. This process ensures that in the subsequent process execution, the system can use both the structure information and visual data for element positioning.

[0062] 2. Computer Vision Calculation Module

[0063] The computer vision calculation module is one of the core modules of the present invention, responsible for processing the interface image captured by the element structure information acquisition module and extracting the visual data that can be used for element positioning. This module uses advanced computer vision technologies, such as object detection and OCR recognition, to analyze the interface image and extract the visual feature information of the target element. When the traditional RPA technology fails to locate the target element due to element structure changes, the computer vision calculation module will calculate the position information of the target element in the interface according to the extracted visual data, providing a basis for the automatic skip-level setting of the element structure information update module.

[0064] 3. Element Structure Information Update Module

[0065] The element structure information update module is responsible for dynamically updating the element structure information during the process execution, that is, automatically setting skip nodes. When the computer vision calculation module successfully locates the target element, this module will extract the latest element structure information, compare it with the original structure information, analyze the changes in node levels and attributes, and complete the operations of automatically identifying, setting, and updating skip nodes. This process ensures that the system can reflect the interface changes in real time, improving the stability and accuracy of the RPA process.

[0066] 4. Element full-link positioning module

[0067] The element full-link positioning module (traditional RPA element matching module) is the execution module of the present invention, responsible for using traditional RPA technology to perform element positioning and operations during the process execution. This module preferentially uses the element structure information extracted by the element structure information acquisition module for layer-by-layer matching to achieve efficient element positioning. When it detects a matching failure caused by element structure changes, the module will automatically trigger the computer vision calculation module to obtain the visual position information of the target element, and update the element structure information according to this information, so as to ensure that the process can continue to execute.

[0068] 5. Element skip positioning module

[0069] The element skip positioning module (dynamic skip matching module) is the core innovative module of the present invention, responsible for handling the challenges brought by changes in the element hierarchical structure. This module analyzes the differences between the old and new element structure information, automatically identifies and sets skip nodes, so as to skip the changed nodes during the layer-by-layer matching process and directly locate the target element. This mechanism not only improves the robustness and flexibility of the process, but also significantly reduces the maintenance cost caused by interface changes.

[0070] Based on the technical solution of the present invention, the following case scenario is used to illustrate the implementation process of the present invention in actual applications. The specific application implementation plan is as follows:

[0071] This embodiment takes a complex desktop application - the client of a management system with an embedded web page as an example. The interface element structure of this client may change during operation, resulting in the inability of traditional RPA technology to accurately match the target element.

[0072] Implementation scenario description:

[0073] This management system client contains multiple levels of element structures, such as: the top-level window, the navigation bar in the middle layer, the specific operation buttons at the bottom layer, etc.

[0074] During actual operation, due to page updates or A / B testing, the hierarchical structure of the navigation bar may change. For example, some elements in the navigation bar may be rearranged or replaced, resulting in the inability of the traditional layer-by-layer matching method to find the target operation button. The specific implementation process is as follows:

[0075] Element Structure Information Acquisition Module:

[0076] Parse the DOM tree of the desktop application through traditional RPA technology, extract the attribute information of the target element and its parent link, and generate serializable element structure information.

[0077] At the same time, capture information such as the application interface image, element screenshot, and element size at the current time point, and associate and store them with the element structure information.

[0078] Send the collected element structure information and interface image to the computer vision calculation module. The computer vision module receives the current image of the interface, extracts visual data through object detection and OCR technology, and associates and stores the extracted data with the element structure information.

[0079] Element Full-Link Positioning Module (Traditional RPA Element Matching Module):

[0080] During the process execution, preferentially use the collected element structure information for layer-by-layer matching.

[0081] Due to the change in the hierarchical structure of the navigation bar, the traditional layer-by-layer matching method cannot find the target operation button.

[0082] At this time, the traditional RPA module triggers a failure signal and notifies the computer vision calculation module to intervene.

[0083] Computer Vision Calculation Module:

[0084] The computer vision module receives the current image of the interface, combines the visual data extracted during the picking process, extracts the visual data of the current interface through object detection and OCR technology, and calculates the position information of the target operation button in the interface.

[0085] Transfer the calculated position information to the Element Structure Information Update Module.

[0086] Element Structure Information Update Module:

[0087] After receiving the position information passed by the computer vision module, the Element Structure Information Update Module obtains a traditional RPA element in real time according to the position information provided by the computer vision calculation module, extracts the attribute information of the target element and its parent link, and generates a new, serializable element structure information.

[0088] Compare the structural information of the new and old elements, judge the differences in the structure comparison, automatically set the skip nodes, and generate new element structure information.

[0089] Pass the new element structure information with automatically set skip nodes to the element skip positioning module (dynamic skip matching module). If the elements can be normally matched and the matched elements meet the threshold of element feature similarity, it is considered that the skip setting is available.

[0090] Store the element structure information after automatically setting the skip, and feedback the latest matched elements to the traditional RPA module to ensure that the subsequent process can be executed smoothly. On the contrary, if the automatic skip fails, the process execution will terminate, and the user can manually adjust the element structure or re-collect the elements.

[0091] Element skip positioning module (dynamic skip matching module):

[0092] As Figure 2 shown, after the element skip positioning module receives the skip element structure information from the element structure information update module, it divides the skip element structure information into two parts, as Figure 3 shown, including non-skip structure information and skip structure information.

[0093] Non-skip structure information: Starting from the first-level node of the element structure, it includes all continuously enabled hierarchical structure sets. This part of the structure information is used to locate all non-skip elements as the starting point for skip matching.

[0094] Skip structure information: Starting from the first hierarchical structure where the skip is set to the last enabled hierarchical structure set. This part of the structure information is used to guide the skip matching process. Especially when encountering dynamically changing or unavailable nodes, the system will skip these nodes according to the skip structure information and directly locate the target element.

[0095] As Figure 2 shown, using the non-skip structure information, starting from the first-level node, layer by layer match and locate all continuously enabled elements. These elements serve as the starting point for skip matching. Traverse the DOM tree structure of these starting elements, extract the unique identifier (path key) of each leaf node, which is generated by combining information such as the type of the element and its position in the parent. At the same time, record the set of ancestor elements of each leaf node.

[0096] According to the skip structure information, extract key information such as the skip hierarchical position, element type, and position in the parent to generate a skip path (jump path key). Use regular expressions to match the unique identifiers (path key) of all leaf nodes and filter out the leaf nodes that match the skip path.

[0097] For each leaf node that conforms to the skip path, match its set of ancestor elements with the skip structure information. During the matching process, when a skip node is encountered, directly skip that node and continue the matching with the next valid node. If the matching is successful, use the matched node as the starting node for the next match and continue the matching process until the entire skip path is matched.

[0098] Finally, return the matching result to the traditional RPA module to ensure the smooth execution of subsequent processes. If the matching fails, record the reason for the failure and provide corresponding debugging information to facilitate manual adjustment by the user or re-acquisition of element structure information.

[0099] Implementation effect Through the above steps, the present invention successfully solves the problem of matching failure caused by changes in the hierarchical structure of the navigation bar. Even if the hierarchical structure of the navigation bar changes, the system can still accurately find the target operation button, ensuring the stability and accuracy of the automation process. At the same time, through the dynamic skip matching mechanism and the real-time update of element structure information, the maintenance cost and manpower investment caused by interface changes are reduced.

[0100] The innovation points of the present invention are as follows:

[0101] 1. The present invention proposes a dynamic skip matching mechanism that can automatically identify and set skip nodes to solve the problem that traditional RPA technology cannot accurately locate target elements when facing changes in the element hierarchical structure. This mechanism automatically sets skip nodes by analyzing the differences between the old and new element structure information, ensuring efficient and accurate matching of target elements in a dynamic environment. This mechanism is essentially different from the method based on XPath path repair in the prior art.

[0102] 2. The present invention proposes a method of automatically setting skip nodes by combining visual data extracted by computer vision technology during the dynamic skip matching process. The computer vision module extracts visual data of the current interface through object detection and OCR technology and calculates the position information of the target element to support the dynamic skip matching. This way of completing automatic skip setting by combining computer vision data is different from the method of only relying on computer vision technology for element repair in the prior art. The present invention pays more attention to the realization of dynamic skip matching.

[0103] 3. The present invention obtains the latest structure information of the target element in real time through the position information calculated by computer vision technology, and compares and updates it with the original structure information. This mechanism not only improves the robustness and accuracy of element positioning, but also reduces the maintenance cost caused by interface changes, ensuring the stability of the RPA process. Different from the method of only updating the XPath path in the prior art, the present invention directly updates the element structure information through the dynamic skip matching mechanism, thus more comprehensively solving the element matching problem.

[0104] The above is only a detailed description of the preferred embodiments and principles of the present invention. For those of ordinary skill in the art, according to the idea provided by the present invention, there will be changes in the specific implementation manners, and these changes should also be regarded as the protection scope of the present invention.

Claims

1. A desktop element positioning method based on dynamic skip-level matching and CV technology, characterized in that, It includes the following steps: S1. When using RPA technology to locate and pick up element structure information, capture the interface image of the application at the current time point, and use computer vision technology to extract visual data information for locating the element position; S2. During the process of process execution, combine RPA technology and computer vision technology to locate and pick up elements, and extract the structure information of the elements; S3. Introduce a jump-level matching mechanism, automatically identify and set jump-level nodes by dynamically analyzing the changes in element structure information, and achieve the positioning of target elements; Step S2 includes the following steps: S21. First use RPA technology to locate and pick up elements; S22. If the element cannot be located using RPA technology, the system automatically switches to computer vision technology, parses the interface content picked up by computer vision technology according to the stored visual data information, and at the same time converts the parsing result into a target element recognizable by RPA technology, and finally obtains the position information of the target element; S23. According to the position information of the target element, combine RPA technology to locate and pick up the element at the corresponding position in real time, and extract the structure information of the element; Step S3 includes the following steps: S31. Compare the extracted element structure information with the original element structure information, and judge the nodes with differences by analyzing the node hierarchy and node attribute information; the nodes with differences include the changed nodes, redundant nodes and missing nodes; S32. For the nodes with differences, set the nodes that meet the preset judgment logic as jump-level nodes, and at the same time generate new jump-level element structure information; the nodes that do not meet the requirements remain unchanged; S33. When the element hierarchy structure changes, the system skips the jump-level nodes and directly locates the target element.

2. The desktop element positioning method based on dynamic skip-level matching and CV technology according to claim 1, wherein In step S1, associate the extracted visual data information with the element structure information picked up by RPA technology for positioning, and store it in the computer.

3. The desktop element positioning method based on dynamic skip-level matching and CV technology according to claim 1, wherein In step S3, the user can actively set jump-level nodes according to actual needs when editing the element structure.

4. The desktop element positioning method based on dynamic skip-level matching and CV technology according to claim 1, wherein The jump-level element structure information includes non-jump-level structure information and jump-level structure information; The non-jump-level structure information refers to the set of all continuously enabled hierarchical structures starting from the first-level node of the element structure; the non-jump-level structure information is used to locate all non-jump-level elements and serve as the starting point for jump-level matching; The jump-level structure information refers to the set of hierarchical structures starting from the first hierarchical structure where jump-level is set to the last enabled hierarchical structure; The jump-level structure information is used to guide the jump-level matching process; the jump-level matching process includes that when encountering dynamically changing or unavailable nodes, the system will skip the corresponding nodes according to the jump-level structure information and directly locate the target element.

5. A desktop element positioning system based on dynamic skip-level matching and CV technology, which is used to implement the desktop element positioning method based on dynamic skip-level matching and CV technology according to any one of claims 1-4, characterized in that, The desktop element positioning system based on dynamic jump-level matching and CV technology includes: An element structure information acquisition module, which is responsible for capturing and extracting the element structure information in the desktop application; A computer vision calculation module, which is responsible for processing the interface image captured by the element structure information acquisition module and extracting the visual data information for locating the element; The element structure information update module is responsible for dynamically updating the structure information of elements during the process execution, that is, automatically setting skip nodes; The element full-link positioning module is responsible for using RPA technology to perform element positioning and operations during the process execution; The element skip positioning module is used to automatically identify and set skip nodes by analyzing the differences between the old and new element structure information, and skip the changed nodes during the layer-by-layer matching process to directly locate the target element.

Images