Detection method of interface elements for Windows system
Patent Information
- Application Number
- DE112023000236
- Authority / Receiving Office
- DE · DE
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-03-31
- Filing Date
- 2023-07-21
- Publication Date
- 2025-09-04
AI Technical Summary
Existing interface element recognition methods usually only support a few interface element frames. Recognizing different frames requires different methods and technologies. The flexibility is poor, and image matching and positioning is easily affected by occlusion, making it difficult to achieve flexible and efficient recognition.
Using the method of combining YOLO target detection model and UIA technology, by obtaining system interface screenshots and information text, identifying the positions of interface elements, overcoming the differences in the frameworks of different interface elements, and realizing the detection of different interfaces without downloading additional plug-ins. For the identification of elements within the element frame, cosine similarity and IOU matching are used to ensure identification accuracy.
It realizes the flexible identification of elements within the framework of different interface elements without the need to download additional plug-ins, overcomes the limitations of image matching and positioning, improves the recognition rate and user experience, and provides a flexible operation method.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
A method for identifying interface elements in Windows systems Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method for identifying interface elements in a Windows system. Background Art
[0002] RPA is a technology that uses software to automatically process business according to specified rules and processes. It aims to simulate the interaction between humans and computers, replacing manual processing of complex, tedious and large-scale tasks, so as to assist or replace manual operations, thereby reducing the company's labor costs and improving overall work efficiency.
[0003] Interface elements refer to all graphical user interface components used to build systems or applications, such as windows, input boxes, and buttons. Using computer peripherals like monitors, keyboards, and mice, humans can interact with operating systems and business systems by performing operations such as entering text in designated input boxes and clicking designated buttons. Similarly, for RPA to perform these operations on behalf of humans, it needs to be able to recognize these interface elements.
[0004] There are many methods for identifying interface elements, but they typically only support a few frameworks. Identifying different frameworks requires different methods and techniques, making it more difficult to use. For example, UiBot, the most influential domestic application, requires a dedicated browser plug-in when using a browser, which is very inflexible. Therefore, a flexible interface element recognition method for Windows systems is needed.
[0005] Summary of the Invention
[0006] To address the above technical issues, the present invention provides a method for identifying interface elements in Windows systems. This method overcomes the limitations of GUI methods, ignores the differences between different interface element frames, and can use the same method to identify elements within different interface element frames without the need to download additional plug-ins. The method is flexible and easy to use. Furthermore, it overcomes the limitations of image matching positioning, and the recognition of interface elements is not affected by occlusion and other conditions.
[0007] To achieve the above object, the technical solution of the present invention is as follows:
[0008] A method for identifying interface elements in Windows systems, comprising the following steps:
[0009] Step 1: Obtain a screenshot of the system interface of the current window, an image of the interface element to be identified, and information text of the interface element to be identified;
[0010] Step 2: Use the YOLO target detection model to perform target detection on the windows system interface screenshot to obtain the position P of the interface element to be identified. YOLO ;
[0011] Step 3: Based on the information text of the interface element to be identified, the position P of the interface element to be identified is obtained through UIA technology. UIA , and get the image corresponding to the position
[0012] Step 4: Change the two positions P YOLO and P UIA Perform matching and obtain the interface element to be identified based on the matching result.
[0013] In the above scheme, step 1 is as follows:
[0014] (1.1) Use the createScreenCapture function of the robot class to take a screenshot of the current system interface of Windows;
[0015] (1.2) Read the local file of the Windows system to obtain the image of the interface element to be identified and the information text of the interface element to be identified, wherein the information text includes the parent window type of the interface element to be identified, the parent window name of the interface element to be identified, the type of the interface element to be identified, the name of the interface element to be identified, and the path of the interface element to be identified in the parent window.
[0016] In a further technical solution, the path refers to the path from the parent window element to the interface element to be identified in the interface element tree with the parent window as the root node and the window layout as the tree arrangement.
[0017] In the above scheme, step 2 is as follows:
[0018] (2.1) Obtain a pre-trained YOLO object detection model, and based on the type of interface element to be identified in the information text, detect the set of all matching interface elements in the Windows system interface screenshot E = {E1, E2, ..., Ei, ..., En}, where Ei represents the i-th matching interface element and n represents the number of all matching interface elements;
[0019] (2.2) The image of the interface element to be identified is compared with the interface element found in (2.1) by using the cosine similarity to calculate the similarity between the two images. The calculation formula is: Cosθ=(a·b) / (|a|×|b|)
[0020] Where a is the image vector obtained from the image of the interface element to be identified, and b is the image vector obtained from an interface element Ei in the interface element set E detected by the YOLO target detection model; |a| is the modulus of the image vector a, and |b| is the modulus of the image vector b. The closer the cosθ value is to 1, the higher the image similarity.
[0021] Get the interface element position area P with the highest similarity and meeting the threshold 1 YOLO , and save the similarity calculated this time.
[0022] In a further technical solution, the training set of the YOLO target detection model is labeled interface elements in the Windows system, and the labels are interface element types.
[0023] In the above solution, step 3 is as follows:
[0024] (3.1) Using UIA technology, obtain the first-level child elements under the root node of the windows system, that is, the window elements, and obtain the window list listW = [W1, W2, ..., Wi, ..., Wm], where Wi is the i-th window element and m is the number of all found window elements; traverse the window list listW and find the matching window based on the parent window type and parent window name of the interface element to be identified in the information text;
[0025] (3.2) Using UIA technology, obtain the window interface element tree of the matching window;
[0026] (3.3) According to the interface element type and name to be identified provided in the information text, and the path of the interface element to be identified in the parent window, the interface element to be identified is searched in the window interface element tree to obtain the location area P of the interface element to be identified. UIA At the same time, use the createScreenCapture function of the robot class to capture the area P UIA Take a screenshot to get an image of the area.
[0027] In a further technical solution, step (3.3) is specifically as follows:
[0028] According to the path of the interface element to be identified in the parent window, search the window interface element tree, and save the interface element to be identified as E path ;
[0029] If the name of the interface element to be identified is not empty, traverse the window interface element tree, match the interface element to be identified according to the name and type of the interface element to be identified, find the interface element to be identified and save it as E name ;
[0030] If Ename and E path If both are not empty and are the same, then get the position of the interface element area to be identified. If they are not the same, then get E name The regional position of P UIA ;
[0031] If E name and E path If one of them is not empty, get the region position of the element that is not empty and get P UIA ;
[0032] If E name and E path If all are empty, update the element position P obtained by the UIA method UIA Empty.
[0033] In the above solution, step 4 is as follows:
[0034] (4.1) Make a basic judgment based on the two positions obtained above. If P YOLO and P UIA If both are empty, it means that the interface element does not exist. YOLO and P UIA If only one is empty, the position of the interface element is considered to be in the non-empty position. YOLO and P UIA If neither is empty, position matching is performed;
[0035] (4.2) According to the above judgment results, YOLO and P UIA Perform position matching: IOU=(P UIA ∩P YOLO ) / (P UIA ∪P YOLO )
[0036] When the overlap of the two positions reaches the threshold value 2, the element search is considered successful, otherwise P UIA The similarity between the captured image of the corresponding element and the image of the interface element to be identified is calculated. If the calculated similarity is higher than the similarity calculated in (2.2), the element position P found by UIA is confirmed. UIA is the interface element to be identified, otherwise confirm the element P found by the target detection YOLO For interface elements.
[0037] Through the above technical solution, the present invention provides a method for identifying interface elements for Windows systems, which has the following beneficial effects:
[0038] (1) The present invention overcomes the limitations of the GUI method, ignores the differences between different interface element frames, and can use the same method to identify elements in different interface element frames without the need to download additional plug-ins. It is simple to use and easy to operate.
[0039] (2) The present invention overcomes the limitations of image matching positioning, and the recognition of interface elements will not be affected by occlusion and other situations.
[0040] (3) The present invention can independently choose whether to obtain interface elements according to the GUI method, the image matching positioning method, or a combination of the two methods based on user preferences. The element recognition rate is high and the user experience is good. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.
[0042] FIG1 is a flow chart of a method for identifying interface elements for Windows systems disclosed in an embodiment of the present invention. DETAILED DESCRIPTION
[0043] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.
[0044] The present invention provides a method for identifying interface elements in a Windows system, as shown in FIG1 , comprising the following steps:
[0045] Step 1: Obtain a screenshot of the system interface of the current windows, an image of the interface element to be identified, and information text of the interface element to be identified.
[0046] The details are as follows:
[0047] (1.1) Use the createScreenCapture function of the robot class to take a screenshot of the current system interface of Windows;
[0048] (1.2) Read the local file of the Windows system to obtain the image of the interface element to be identified and the information text of the interface element to be identified, the information text includes the parent window type of the interface element to be identified, the parent window name of the interface element to be identified, the type of the interface element to be identified, the name of the interface element to be identified, and the path of the interface element to be identified in the parent window.
[0049] In a further technical solution, the path refers to the path from the parent window element to the interface element to be identified in the interface element tree with the parent window as the root node and the window layout as the tree arrangement.
[0050] Step 2: Use the YOLO target detection model to perform target detection on the windows system interface screenshot to obtain the position P of the interface element to be identified. YOLO .
[0051] The details are as follows:
[0052] (2.1) Obtain a pre-trained YOLO object detection model, and based on the type of interface element to be identified in the information text, detect the set of all matching interface elements in the Windows system interface screenshot E = {E1, E2, ..., Ei, ..., En}, where Ei represents the i-th matching interface element and n represents the number of all matching interface elements;
[0053] The training set of the YOLO target detection model is the interface elements in the Windows system with labels, and the labels are the interface element types.
[0054] (2.2) The image of the interface element to be identified is compared with the interface element found in (2.1) by using the cosine similarity to calculate the similarity between the two images. The calculation formula is: Cosθ=(a·b) / (|a|×|b|)
[0055] Where a is the image vector obtained from the image of the interface element to be identified, and b is the image vector obtained from an interface element Ei in the interface element set E detected by the YOLO target detection model; |a| is the modulus of the image vector a, and |b| is the modulus of the image vector b. The closer the cosθ value is to 1, the higher the image similarity.
[0056] Get the interface element position area P with the highest similarity and meeting the threshold 1 YOLO , and save the similarity calculated this time.
[0057] Step 3: Based on the information text of the interface element to be identified, the position P of the interface element to be identified is obtained through UIA technology. UIA , and get the image corresponding to the position.
[0058] The details are as follows:
[0059] (3.1) Using UIA technology, obtain the first-level child elements under the root node of the windows system, that is, the window elements, and obtain the window list listW = [W1, W2, ..., Wi, ..., Wm], where Wi is the i-th window element and m is the number of all found window elements; traverse the window list listW and find the matching window based on the parent window type and parent window name of the interface element to be identified in the information text;
[0060] (3.2) Using UIA technology, obtain the window interface element tree of the matching window;
[0061] (3.3) According to the interface element type and name to be identified provided in the information text, and the path of the interface element to be identified in the parent window, the interface element to be identified is searched in the window interface element tree to obtain the location area P of the interface element to be identified. UIA At the same time, use the createScreenCapture() function of the robot class to capture the area P UIA Take a screenshot to get an image of the area.
[0062] Furthermore, step (3.3) is as follows:
[0063] According to the path of the interface element to be identified in the parent window, search the window interface element tree, and save the interface element to be identified as E path ;
[0064] If the name of the interface element to be identified is not empty, traverse the window interface element tree, match the interface element to be identified according to the name and type of the interface element to be identified, find the interface element to be identified and save it as E name ;
[0065] If E name and E path If both are not empty and are the same, then get the position of the interface element area to be identified. If they are not the same, then get E name The regional position of P UIA ;
[0066] If E name and E path If one of them is not empty, get the region position of the element that is not empty and get P UIA ;
[0067] If E name and E path If all are empty, update the element position P obtained by the UIA method UIA Empty.
[0068] Step 4: Change the two positions P YOLO and P UIA Perform matching and obtain the interface element to be identified based on the matching result.
[0069] The details are as follows:
[0070] (4.1) Make a basic judgment based on the two positions obtained above. If P YOLO and P UIA If both are empty, it means that the interface element does not exist. YOLO and P UIAIf only one is empty, the position of the interface element is considered to be in the non-empty position. YOLO and P UIA If neither is empty, position matching is performed;
[0071] (4.2) According to the above judgment results, P YOLO and P UIA Perform position matching: IOU=(P UIA ∩P YOLO ) / (P UIA ∪P YOLO )
[0072] When the overlap of the two positions reaches the threshold value 2, the element search is considered successful, otherwise P UIA The similarity between the captured image of the corresponding element and the image of the interface element to be identified is calculated. If the calculated similarity is higher than the similarity calculated in (2.2), the element position P found by UIA is confirmed. UIA is the interface element to be identified, otherwise confirm the element P found by the target detection YOLO For interface elements.
[0073] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for identifying interface elements in Windows systems, characterized in that: The following steps are involved: Step 1, obtaining a screenshot of the system interface of the current windows, an image of the interface element to be identified, and information text of the interface element to be identified; Step 2: Use the YOLO target detection model to perform target detection on the windows system interface screenshot to obtain the position P of the interface element to be identified. YOLO ; Step 3: Based on the information text of the interface element to be identified, the position P of the interface element to be identified is obtained through UIA technology. UIA , and get the image corresponding to the position; Step 4: Set the above two positions P YOLO and P UIA Perform matching and obtain the interface element to be identified according to the matching result.
2. The method for identifying interface elements for Windows system according to claim 1, characterized in that: Step 1 is as follows: (1.1) Use the createScreenCapture function of the robot class to take a screenshot of the current system interface of Windows; (1.2) Read the local file of the Windows system to obtain the image of the interface element to be identified and the information text of the interface element to be identified, wherein the information text includes the parent window type of the interface element to be identified, the parent window name of the interface element to be identified, the type of the interface element to be identified, the name of the interface element to be identified, and the path of the interface element to be identified in the parent window.
3. The method for identifying interface elements for Windows system according to claim 2, characterized in that: The path refers to the path from the parent window element to the interface element to be identified in the interface element tree with the parent window as the root node and the window layout as the tree arrangement.
4. The method for identifying interface elements for Windows system according to claim 2, characterized in that: Step 2 is as follows: (2.1) Obtain a pre-trained YOLO target detection model, and detect the set of all matching interface elements in the windows system interface screenshot E = {E1, E2, ..., Ei, ..., En} according to the type of interface element to be identified in the information text, where Ei represents the i-th matching interface element and n represents the number of all matching interface elements; (2.2) The image of the interface element to be identified and the interface element found in (2.1) are used to calculate the similarity of the two images through cosine similarity. The calculation formula is: Cosθ=(a·b) / (|a|×|b|) Wherein, a is the image vector obtained by the image of the interface element to be identified, b is the image vector obtained by a certain interface element Ei in the interface element set E detected by the YOLO target detection model; |a| is the modulus of the image vector a, |b| is the modulus of the image vector b, and the closer the cosθ value is to 1, the higher the image similarity; Get the interface element position area P with the highest similarity and meeting the threshold 1 YOLO , and save the similarity calculated this time.
5. The method for identifying interface elements for Windows system according to claim 4, characterized in that: The training set of the YOLO target detection model is the interface elements in the Windows system with labels, and the labels are the interface element types.
6. The method for identifying interface elements for Windows system according to claim 2, characterized in that: Step 3 is as follows: (3.1) Using UIA technology to obtain the first-level child elements under the root node of the windows system, that is, the window elements, and obtain the window list listW = [W1, W2, ..., Wi, ..., Wm], where Wi is the i-th window element and m is the number of all found window elements; traverse the window list listW and find the matching window according to the parent window type and parent window name of the interface element to be identified in the information text; (3.2) Using UIA technology, obtain the window interface element tree of the matching window; (3.3) Search the interface element to be identified in the window interface element tree according to the type of interface element to be identified, the name of the interface element to be identified, and the path of the interface element to be identified in the parent window, and obtain the position area P of the interface element to be identified UIA At the same time, use the createScreenCapture function of the robot class to capture the area P UIA Take a screenshot to get an image of the area.
7. The method for identifying interface elements for Windows system according to claim 6, characterized in that: Step (3.3) is as follows: According to the path of the interface element to be identified in the parent window, search the window interface element tree, and save the interface element to be identified as E if it is found. path ; If the name of the interface element to be identified is not empty, traverse the window interface element tree, match the interface element to be identified according to the name of the interface element to be identified and the type of the interface element to be identified, find the interface element to be identified and save it as E name ; If E name and E path If both are not empty and are the same, then get the position of the interface element area to be identified; if they are not the same, then get E name The regional position of P UIA ; If E name and E path If one of them is not empty, get the region position of the element that is not empty and get P UIA ; If E name and E path If all are empty, update the element position P obtained by the UIA method UIA Is empty.
8. The method for identifying interface elements for Windows systems according to claim 2, characterized in that: Step 4 is as follows: (4.1) Make a basic judgment based on the two positions obtained above. If P YOLO and P UIA If both are empty, it means that the interface element does not exist. YOLO and P UIA If only one is empty, the interface element is considered to be located at the position that is not empty. YOLO and P UIA If neither is empty, position matching is performed; (4.2) According to the above judgment results, YOLO and P UIA To perform position matching: IOU=(P UIA ∩P YOLO ) / (P UIA ∪P YOLO ) When the overlap between the two positions reaches the threshold value 2, the element search is considered successful, otherwise P UIA The similarity between the captured image of the corresponding element and the image of the interface element to be identified is calculated. If the calculated similarity is higher than the similarity calculated in (2.2), the element position P that UIA is looking for is confirmed. UIA is the interface element to be identified, otherwise confirm the element P found by the target detection YOLO For interface elements.