Desktop control picking and positioning method for RPA system

By combining the construction of the websocket interface and the use of win32gui library and UIAutomation library, real-time highlighted rectangle box drawing and information collection of desktop controls is realized, solving the integration problem of desktop control picking and positioning in RPA system, simplifying the operation process and improving the Positioning accuracy provides efficient control positioning capabilities for RPA systems.

CN118916001AActive Publication Date: 2024-11-08XINWEI TECHNOLOGY (HARBIN) CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410970155.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-08
Estimated Expiration
2044-07-19

AI Technical Summary

Technical Problem

In the prior art, traditional desktop control picking and positioning methods cannot be integrated into the RPA system, the operation process is complex and the positioning capability is single, so it cannot be used as a component of a large-scale RPA system.

Method used

By building a websocket interface for pickups and locators, using win32gui library and UIAutomation library, real-time highlighted rectangle frame drawing and information collection of desktop controls, using desktop control positioning methods, control screenshot positioning methods and coordinate positioning methods during picking , ensure accurate acquisition and matching of control information.

Benefits of technology

It realizes the integration of desktop control picking and positioning capabilities of the RPA system, simplifies the operation process, improves positioning accuracy and credibility, and supports the use of large-scale RPA systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916001A_ABST
    Figure CN118916001A_ABST
Patent Text Reader

Abstract

The invention discloses a desktop control picking and positioning method for an RPA system, and belongs to the technical field of automatic software testing. The problem that in the prior art, a traditional desktop control picking and positioning method cannot be integrated in an RPA system is solved. According to the method, a picker websocket interface and a locator websocket interface are constructed, a foreground process designer calls a picker, user operation is collected by monitoring a mouse and a keyboard, and when a desktop control under the position of the mouse changes, a highlight rectangular frame is refreshed to obtain control information and display the control information back to the foreground process designer; and the foreground process designer calls the locator, matches the control information acquired by the picker, calculates the center coordinate of the desktop control according to the successfully matched control information, and echoes the center coordinate to the foreground process designer. According to the method, the picking and positioning operation process is effectively simplified, the accuracy of the obtained control information corresponding to the desktop control is verified, and the method can be applied to a large-scale RPA system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a desktop control picking and positioning method, and in particular to a desktop control picking and positioning method for an RPA system, and belongs to the technical field of automated software testing. Background Art

[0002] At present, the creation process of process automation software requires picking up and locating the desktop controls of the Windows operating system, locating the controls after picking up the controls, and performing a series of automated operations on the corresponding controls. The traditional method of locating and picking up desktop controls is generally implemented through UISpy (window capture tool). UISpy is a Windows operating system application provided by Microsoft. UISpy can locate desktop controls and obtain control information. The underlying logic is to call the corresponding interface through the Win32gui library (graphical user interface gateway) to obtain and highlight the user interface control, that is, first obtain the desktop control operation handle, then capture the control information, and then draw a rectangular box around the desktop control through the Rectangle drawing rectangular box instruction. Finally, the control information is output to the foreground process designer to realize the operation of visually highlighting the desktop control; however, the window capture tool in the prior art is generally an independent program, does not provide an access interface, is difficult to integrate into a large-scale automation system, and has a complex operation process, unclear process, and single positioning capability. It can only be used as a simple test software, and cannot be used as a component of a large-scale RPA (robotic process automation) system.

[0003] In summary, a desktop control picking and positioning method is needed which actively provides an interface, has a simple operation process, and has a high accuracy of picking results. Summary of the invention

[0004] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0005] In view of this, in order to solve the problem that the traditional desktop control picking and positioning method in the prior art cannot be integrated into the RPA system, the present invention provides a desktop control picking and positioning method for the RPA system.

[0006] The technical solution is as follows: A desktop control picking and positioning method for an RPA system includes the following steps:

[0007] S1. Build a picker websocket interface and a locator websocket interface. The front-end process designer of the RPA system calls the picker and the locator by connecting with the picker websocket interface and the locator websocket interface.

[0008] S2. The front-end process designer calls the picker, collects user operations by monitoring the mouse and keyboard, and draws a highlighted rectangular box of the desktop control in real time through the win32gui library. When the desktop control under the mouse position changes, the highlighted rectangular box is refreshed to obtain the control information and echo it to the front-end process designer;

[0009] S3. The foreground process designer calls the locator and uses the desktop control positioning method, the control screenshot positioning method and the coordinate positioning method during picking to obtain the desktop control and its corresponding control information in turn, until the control information obtained by the picker is successfully matched, and the center coordinates of the desktop control are calculated and echoed back to the foreground process designer.

[0010] Furthermore, in the S1, the specific steps of constructing the picker websocket interface are as follows: define the publicly accessible server PickerSocketServer through the class class, establish a private connection session with the client through the Session function, when the connection status openFlag is true, judge the connection is successful, when the connection status openFlag is false, judge the connection failed, the connection status openFlag is output to the server PickerSocketServer through the log.info function, annotate the message message sent by the client to the server through the @param annotation, use the onMessage instruction to listen to the message message, when the client and the server send a message, the message message will be called back through the log.info function to verify the parameters in the message message, and when the parameter starts with chromepickerstart:, it means starting the picker;

[0011] The specific steps to build the locator websocket interface are as follows: define the publicly accessible server LocatorSocketServer through the class class, establish a private connection session with the client through the Session function, when the connection status openFlag is true, the connection is judged to be successful, when the connection status openFlag is false, the connection is judged to be failed, the connection status openFlag is output to the server LocatorSocketServer through the log.info function, annotate the message message sent by the client to the server through the @param annotation, use the onMessage instruction to listen to the message message, when the client and the server send messages, the message message will be called back through the log.info function to verify the parameters in the message message, and when the parameters start with chromelocatorstart:, it means that the locator is started.

[0012] Furthermore, in S2, the keyboard listener keyboard_listener and the mouse listener mouse_listener are started through the start function. When the esc key of the keyboard is pressed, it indicates that the pickup is interrupted, the mouse listener mouse_listener is stopped, the highlighted rectangular frame currently drawn by the win32gui library is refreshed, that is, the highlighted rectangular frame is removed, and the interrupt pickup information is returned to the caller;

[0013] When the left ctrl key is pressed, the current desktop control is picked up and the coordinates of the picking are recorded. The mouse listener mouse_listener is stopped. The highlighted rectangle frame drawn by the win32gui library is refreshed, that is, the highlighted rectangle frame is removed. The screenshot flag screen_flag is True. A screenshot is taken and the coordinates of the top, bottom, left, and right of the screenshot area are recorded as the screenshot position. The picking is terminated.

[0014] The specific process of the keyboard listener keyboard_listener is as follows: when taking a screenshot, construct a screenshot name through the user identity uid, construct a screenshot area for taking a screenshot, construct a return parameter, that is, the screenshot location, and output the return parameter to the foreground process designer;

[0015] The specific process of the mouse listening mouse_listener is as follows: obtain the global COM object of the UIAutomation library under the current mouse coordinates, obtain the UIAElement class according to the global COM object, obtain the basic packaging wrapper class of the UI automation element, if the cached object is changed compared to the current wrapper class, that is, the desktop control under the mouse position has changed, based on the win32gui library, the Redraw redraw instruction refreshes the highlight rectangle box, and obtains the refreshed current highlight rectangle box through the wrapper class, and updates the corresponding desktop control at the same time, draws the highlight rectangle box around the desktop control, and obtains the coordinates of the highlight rectangle box.

[0016] Furthermore, in said S3, the specific process of the desktop control positioning method is as follows: obtain the control information obtained by the picker, format it, obtain the coordinates of the desktop control, obtain the desktop control through the classname class, that is, generate the boundary of the highlighted rectangular box by the app class to obtain the desktop control, if it is successfully obtained and the control information corresponding to the above desktop control matches the control information obtained by the picker, then calculate the center coordinates of the desktop control according to the coordinates of the highlighted rectangular box and output it to the foreground process designer, if it is not successfully matched, execute the control screenshot positioning method;

[0017] The specific process of the control screenshot positioning method is as follows: call the pyautogui library in the current desktop to find the screenshot position through the locateOn instruction to obtain the desktop control. If it is successfully obtained and the control information corresponding to the above desktop control matches the control information obtained by the picker, the center coordinates of the desktop control are calculated according to the screenshot position and output to the foreground process designer. If it is not successfully matched, the coordinate positioning method at the time of picking is executed;

[0018] The specific process of the coordinate positioning method during picking is as follows: the center coordinates of the desktop control are calculated according to the coordinates during picking and output to the foreground process designer.

[0019] The beneficial effects of the present invention are as follows: the present invention provides support for the integration of large-scale RPA systems, and integrates the picker and the locator through the front-end process designer of the RPA system by connecting the picker websocket interface and the locator websocket interface, so as to realize the invocation of the picker and the locator and data interaction; the present invention only needs to import the package docking interface to enable the RPA system to obtain the ability to pick up and locate desktop controls. Compared with the abstract operation process of the prior art, the present invention optimizes the process, and the user can move the mouse at will to select the relevant information of the currently highlighted desktop control, so as to locate the desktop control in the subsequent automated process. The locator of the present invention verifies the accuracy of the control information through the desktop control positioning method, the control screenshot positioning method and the coordinate positioning method during picking, and provides high-reliability and available support for the picker acquisition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0021] Figure 1 A flowchart of a method for picking and locating desktop controls for an RPA system is provided;

[0022] Figure 2 A schematic diagram of an embodiment of a method for picking and locating desktop controls for an RPA system;

[0023] Figure 3 A schematic diagram of the code for the first part of the implementation of the pickup websocket interface;

[0024] Figure 4 A schematic diagram of the code for the second embodiment of the websocket interface for building a pickup;

[0025] Figure 5 A schematic diagram of the code of the third embodiment for constructing the websocket interface of the picker;

[0026] Figure 6 A schematic diagram of the code for the first part of the implementation example for constructing a locator websocket interface;

[0027] Figure 7 A schematic diagram of the code for the second embodiment of the construction of the locator websocket interface;

[0028] Figure 8 A schematic diagram of the code of the third embodiment for constructing the locator websocket interface;

[0029] Fig. 9 A schematic diagram of the code of the first embodiment of the connection between the foreground process designer and the picker websocket interface and the locator websocket interface;

[0030] Fig.10 A schematic diagram of the code of the second embodiment of the connection between the foreground process designer and the picker websocket interface and the locator websocket interface;

[0031] Fig.11 A schematic diagram of the embodiment code of the first part of picking up by a picker;

[0032] Fig.12 A schematic diagram of the embodiment code of the second part of picking up by a picker;

[0033] Fig.13 A schematic diagram of the code of the third embodiment of the method of picking up with a picker;

[0034] Fig.14 A schematic diagram of the embodiment code of the fourth part for picking up with a picker;

[0035] Fig.15 A schematic diagram of code for an embodiment of positioning using a locator. DETAILED DESCRIPTION

[0036] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0037] refer to Figure 1-Figure 15 The present embodiment is described in detail, and the method for picking and locating desktop controls for the RPA system specifically includes the following steps:

[0038] S1. Build and connect the picker websocket interface and the locator websocket interface. The front-end process designer of the RPA system calls the picker and the locator by connecting with the picker websocket interface and the locator websocket interface.

[0039] S2. The front-end process designer calls the picker, collects user operations by monitoring the mouse and keyboard, and draws a highlighted rectangular box of the desktop control in real time through the win32gui library. When the desktop control under the mouse position changes, the highlighted rectangular box is refreshed to obtain the control information and echo it to the front-end process designer;

[0040] S3. The foreground process designer calls the locator and uses the desktop control positioning method, the control screenshot positioning method and the coordinate positioning method during picking to obtain the desktop control and its corresponding control information in turn, until the control information obtained by the picker is successfully matched, and the center coordinates of the desktop control are calculated and echoed back to the foreground process designer.

[0041] Furthermore, in the S1, the specific steps of constructing the picker websocket interface are as follows: define the publicly accessible server PickerSocketServer through the class class, establish a private connection session with the client through the Session function, when the connection status openFlag is true, judge the connection is successful, when the connection status openFlag is false, judge the connection failed, the connection status openFlag is output to the server PickerSocketServer through the log.info function, annotate the message message sent by the client to the server through the @param annotation, use the onMessage instruction to listen to the message message, when the client and the server send a message, the message message will be called back through the log.info function to verify the parameters in the message message, and when the parameter starts with chromepickerstart:, it means starting the picker;

[0042] The specific steps to build the locator websocket interface are as follows: define the publicly accessible server LocatorSocketServer through the class class, establish a private connection session with the client through the Session function, when the connection status openFlag is true, the connection is judged to be successful, when the connection status openFlag is false, the connection is judged to be failed, the connection status openFlag is output to the server LocatorSocketServer through the log.info function, annotate the message message sent by the client to the server through the @param annotation, use the onMessage instruction to listen to the message message, when the client and the server send a message, the message message will be called back through the log.info function to verify the parameters in the message message, and when the parameter starts with chromelocatorstart:, it means that the locator is started;

[0043] Specifically, refer to Figure 3-Figure 8 After the WebSocket communication connection is established, the client and the server can communicate bidirectionally through the connection. The present invention uses WebSocket communication as the server to realize long-term bidirectional communication between the client, that is, the foreground process designer, and the picker and locator. Fig. 9 and Fig.10 , as the calling code of the picker and locator, there is no need to pay attention to the language, it only needs to support websocket communication, and this embodiment uses front-end technology to implement;

[0044] Furthermore, in S2, the keyboard listener keyboard_listener and the mouse listener mouse_listener are started through the start function. When the esc key of the keyboard is pressed, it indicates that the pickup is interrupted, and the mouse listener mouse_listener is stopped. The highlighted rectangular frame currently drawn by the win32gui library is refreshed through "win32gui.RedrawWindow(None,None,None,win32con.RDW_FRAME|win32con.RDW_INVA LIDATE|win32con.RDW_UPDATENOW|win32con.RDW_ALLCHILDREN)", that is, the highlighted rectangular frame is removed, and the interrupt pickup information is returned to the caller;

[0045] When the left ctrl key is pressed, the current desktop control is picked up and the coordinates of the picking are recorded. The mouse listener mouse_listener is stopped. The highlighted rectangle frame drawn by the win32gui library is refreshed, that is, the highlighted rectangle frame is removed. The screenshot flag screen_flag is True. A screenshot is taken and the coordinates of the top, bottom, left, and right of the screenshot area are recorded as the screenshot position. The picking is terminated.

[0046] The specific process of the keyboard listener keyboard_listener is as follows: when taking a screenshot, construct a screenshot name through the user identity uid, construct a screenshot area for taking a screenshot, construct a return parameter, that is, the screenshot location, and output the return parameter to the foreground process designer;

[0047] The specific process of the mouse listening mouse_listener is as follows: obtain the global COM object of the UIAutomation library under the current mouse coordinates, obtain the UIAElement class according to the global COM object, obtain the basic wrapper class of the UI automation element, if the cache object is changed compared with the current wrapper class, that is, the desktop control under the mouse position has changed, based on the win32gui library, the Redraw redraw instruction is used to refresh the highlight rectangle, and the current highlight rectangle after the refresh is obtained through the wrapper class, and the corresponding desktop control is updated at the same time, and the highlight rectangle around the desktop control is drawn to obtain the coordinates of the highlight rectangle;

[0048] Specifically, refer to Figure 11-Figure 15, pickers and locators are written in python scripts and integrated into the websocket server written in Java. The pickers and locators need to call the underlying methods of desktop controls. The python language provides a wealth of related libraries such as the pyautogui library and the win32gui library. After calling the picker, the mouse monitor collects the controls under the user's mouse coordinates in real time, and uses UIA (user automation design) technology to obtain desktop control information, such as the desktop control classname and the coordinates around the desktop control, that is, the upper, lower, left and right coordinates of the screenshot area. The UIA technology is mainly used to store global COM objects related to the UIAutomation library. In the use of pywinauto (computer-side automation tool), UIA technology is a technical framework provided for the Windows operating system for realizing automated testing of user interfaces and accessibility to auxiliary functions.

[0049] Furthermore, in said S3, the specific process of the desktop control positioning method is as follows: obtain the control information obtained by the picker, format it, obtain the coordinates of the desktop control, obtain the desktop control through the classname class, that is, generate the boundary of the highlighted rectangular box by the app class to obtain the desktop control, if it is successfully obtained and the control information corresponding to the above desktop control matches the control information obtained by the picker, then calculate the center coordinates of the desktop control according to the coordinates of the highlighted rectangular box and output it to the foreground process designer, if it is not successfully matched, execute the control screenshot positioning method;

[0050] The specific process of the control screenshot positioning method is as follows: call the pyautogui library in the current desktop to find the screenshot position through the locateOn instruction to obtain the desktop control. If it is successfully obtained and the control information corresponding to the above desktop control matches the control information obtained by the picker, the center coordinates of the desktop control are calculated according to the screenshot position and output to the foreground process designer. If it is not successfully matched, the coordinate positioning method at the time of picking is executed;

[0051] The specific process of the coordinate positioning method during picking is as follows: the center coordinates of the desktop control are calculated according to the coordinates during picking and output to the foreground process designer;

[0052] Specifically, refer to Fig.15The locator uses three methods to locate the desktop control. Its core intention is to determine the center coordinates of the control for subsequent automated process operations. The desktop control positioning method mainly uses UIA technology to find the desktop control on the current desktop. The control screenshot positioning method uses the pyautogui library. The underlying logic uses the opencv library to extract features first. A variety of feature extraction algorithms can be used, such as SIFT (feature extraction algorithm based on scale-invariant feature transform), SURF (accelerated robust feature extraction algorithm), HOG (directional gradient histogram feature extraction algorithm), etc. The above feature extraction algorithms can find local key points in the image and calculate their The corresponding feature vector is then used for target recognition. After feature extraction, the OpenCV library provides the function of target recognition. Target recognition refers to identifying the target in the image based on the extracted feature vector, and the OpenCV library supports a variety of target recognition methods including machine learning methods and template matching-based methods. Among them, machine learning methods such as SVM (support vector machine) and KNN (K nearest neighbor algorithm) can determine the category of the target in the input image based on the similarity of the feature vector. If the desktop control positioning method and the control screenshot positioning method cannot obtain the control coordinates, the coordinates at the time of picking are directly selected for subsequent calculations and output to the foreground process designer.

[0053] refer to Figure 2 The locator is used to verify the control information obtained by the picker. If the control information matches successfully, the corresponding desktop control will be highlighted and flash three times.

[0054] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A desktop control picking and positioning method for an RPA system, characterized in that: The following steps are involved: S1. Build the picker websocket interface and the locator websocket interface. The RPA system's front-end process designer calls the picker and locator by connecting to the picker websocket interface and the locator websocket interface. S2. The front-end process designer calls the picker, which monitors the mouse and keyboard for user operations. It then uses the win32gui library to draw a highlighted rectangle around the desktop control in real time. When the desktop control under the mouse position changes, the highlighted rectangle is refreshed to obtain control information and echo it back to the front-end process designer. S3. The foreground process designer calls the locator and uses the desktop control positioning method, the control screenshot positioning method, and the coordinate positioning method during picking to obtain the desktop control and its corresponding control information in sequence until the control information obtained by the picker is successfully matched. The center coordinates of the desktop control are calculated and echoed back to the foreground process designer.

2. The desktop control picking and positioning method for an RPA system according to claim 1, characterized in that: In the S1, the specific steps of constructing the picker websocket interface are as follows: define the publicly accessible server PickerSocketServer through the class class, establish a private connection session with the client through the Session function, when the connection status openFlag is true, it is judged that the connection is successful, when the connection status openFlag is false, it is judged that the connection fails, the connection status openFlag is output to the server PickerSocketServer through the log.info function, annotate the message sent by the client to the server through the @param annotation, use the onMessage instruction to monitor the message, when the client and the server send a message, the message will be called back through the log.info function to verify the parameters in the message. When the parameter starts with chromepickerstart:, it means that the picker is started; The specific steps to build the locator websocket interface are as follows: define the publicly accessible server LocatorSocketServer through the class class, establish a private connection session with the client through the Session function, when the connection status openFlag is true, the connection is judged to be successful, when the connection status openFlag is false, the connection is judged to be failed, the connection status openFlag is output to the server LocatorSocketServer through the log.info function, annotate the message sent by the client to the server through the @param annotation, use the onMessage instruction to listen to the message, when the client and the server send a message, the message will be called back through the log.info function to verify the parameters in the message. When the parameter starts with chromelocatorstart:, it means that the locator is started.

3. The desktop control picking and positioning method for an RPA system according to claim 2, characterized in that: In the S2, the keyboard listener keyboard_listener and the mouse listener mouse_listener are started through the start function. When the esc key of the keyboard is pressed, it indicates that the picking is interrupted, the mouse listener mouse_listener is stopped, the highlighted rectangle currently drawn by the win32gui library is refreshed, that is, the highlighted rectangle is removed, and the interrupt picking information is returned to the caller; When the left ctrl key is pressed, the current desktop control is picked up and the coordinates at the time of picking are recorded. The mouse listener mouse_listener is stopped, the highlighted rectangle currently drawn by the win32gui library is refreshed, that is, the highlighted rectangle is removed, the screenshot flag screen_flag is True, a screenshot is taken, and the coordinates of the top, bottom, left, and right of the screenshot area are recorded as the screenshot position, and the picking is terminated. The specific process of keyboard listening keyboard_listener is as follows: when taking a screenshot, construct a screenshot name through the user identity uid, construct a screenshot area for taking a screenshot, construct a return parameter, that is, the screenshot location, and output the return parameter to the foreground process designer; The specific process of the mouse listening mouse_listener is as follows: obtain the global COM object of the UIAutomation library under the current mouse coordinates, obtain the UIAElement class according to the global COM object, obtain the basic packaging wrapper class of the UI automation element, if the cached object is changed compared with the current wrapper class, that is, the desktop control under the mouse position has changed, based on the win32gui library, the Redraw redraw instruction is used to refresh the highlight rectangle, and the refreshed current highlight rectangle is obtained through the wrapper class, and the corresponding desktop control is updated at the same time, and the highlight rectangle around the desktop control is drawn to obtain the coordinates of the highlight rectangle.

4. The desktop control picking and positioning method for an RPA system according to claim 3, characterized in that: In said S3, the specific process of the desktop control positioning method is as follows: obtain the control information obtained by the picker, format it, obtain the coordinates of the desktop control, obtain the desktop control through the classname class, that is, generate the boundary of the highlighted rectangular box by the app class to obtain the desktop control, if the acquisition is successful and the control information corresponding to the above desktop control matches the control information obtained by the picker, then calculate the center coordinates of the desktop control according to the coordinates of the highlighted rectangular box and output it to the foreground process designer, if the match is not successful, execute the control screenshot positioning method; The specific process of the control screenshot positioning method is as follows: call the pyautogui library in the current desktop to find the screenshot location through the locateOn instruction to obtain the desktop control. If the acquisition is successful and the control information corresponding to the above desktop control matches the control information obtained by the picker, the center coordinates of the desktop control are calculated according to the screenshot position and output to the foreground process designer. If the match is not successful, the coordinate positioning method at the time of picking is executed; The specific process of the coordinate positioning method during picking is as follows: the center coordinates of the desktop control are calculated according to the coordinates during picking and output to the foreground process designer.

Citation Information

Patent Citations

  • Method for carrying out automatic operation in opened Chrome browser

    CN112445994A

  • Method and system for intelligent fusion pickup of RPA elements

    CN113391871A

  • RPA interface determination method and device, computer equipment and storage medium

    CN116185411A

  • Element positioning method and system for HTMLayout application program in RPA scene

    CN117130717A