Desktop control picking and positioning method for RPA system

Abstract

The present invention discloses a desktop control picking and positioning method for an RPA system, which belongs to the field of automated software testing technology. It solves the problem that the traditional desktop control picking and positioning method in the prior art cannot be integrated into the RPA system; the present invention constructs a picker websocket interface and a locator websocket interface, and the foreground process designer calls the picker, collects user operations by monitoring the mouse and keyboard, and when the desktop control under the mouse position changes, refreshes the highlighted rectangular box to obtain control information and echoes it to the foreground process designer; the foreground process designer calls the locator, matches the control information obtained by the picker, calculates the center coordinates of the desktop control based on the successfully matched control information, and echoes it to the foreground process designer. The present invention effectively simplifies the picking and positioning operation process, verifies the accuracy of the control information corresponding to the obtained desktop control, and can be applied to large-scale RPA systems.

Description

Technical Field

[0001] The present invention relates to a desktop control picking and positioning method, and in particular to a desktop control picking and positioning method for an RPA system, belonging to the technical field of automated software testing. Background Art

[0002] Currently, the creation process of process automation software requires picking and locating desktop controls in the Windows operating system. After picking and locating the controls, a series of automated operations are performed on the corresponding controls. Traditional methods for locating and picking desktop controls are generally implemented using UISpy (a window capture tool). UISpy is a Windows operating system application provided by Microsoft. It can locate desktop controls and obtain control information. The underlying logic is to call the corresponding interface through the Win32GUI library (graphical user interface gateway) to obtain and highlight user interface controls. Specifically, the process first obtains the desktop control operation handle, then captures the control information. Finally, a rectangular box is drawn around the desktop control using the Rectangle command. Finally, the control information is output to the front-end process designer to achieve visual highlighting of the desktop control. However, existing window capture tools are generally standalone programs without an access interface, making them difficult to integrate into large-scale automation systems. Furthermore, their operation is complex, the process is unclear, and their positioning capabilities are limited. Therefore, they can only be used as simple testing software and cannot be used as components of large-scale RPA (robotic process automation) systems.

[0003] In summary, a desktop control picking and positioning method is needed that actively provides an interface, has a simple operation process, and has high picking result accuracy. Summary of the Invention

[0004] A brief overview of the present invention is provided below to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important aspects of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is simply to present certain concepts in a simplified form as a prelude to the more detailed description discussed later.

[0005] In view of this, in order to solve the problem that the traditional desktop control picking and positioning method in the prior art cannot be integrated into the RPA system, the present invention provides a desktop control picking and positioning method for the RPA system.

[0006] The technical solution is as follows: A method for picking and locating desktop controls for an RPA system includes the following steps:

[0007] S1. Build the picker websocket interface and the locator websocket interface. The RPA system's front-end process designer calls the picker and locator by connecting to the picker websocket interface and the locator websocket interface.

[0008] S2. The front-end process designer calls the picker, which monitors the mouse and keyboard for user operations. It then uses the win32gui library to draw a highlighted rectangle around the desktop control in real time. When the desktop control under the mouse position changes, the highlighted rectangle is refreshed to obtain control information and echo it back to the front-end process designer.

[0009] S3. The foreground process designer calls the locator and uses the desktop control positioning method, the control screenshot positioning method, and the coordinate positioning method during picking to obtain the desktop control and its corresponding control information in sequence until the control information obtained by the picker is successfully matched. The center coordinates of the desktop control are calculated and echoed back to the foreground process designer.

[0010] Furthermore, in the S1, the specific steps of constructing the picker websocket interface are as follows: define the publicly accessible server PickerSocketServer through the class class, establish a private connection session with the client through the Session function, when the connection status openFlag is true, it is judged that the connection is successful, when the connection status openFlag is false, it is judged that the connection fails, the connection status openFlag is output to the server PickerSocketServer through the log.info function, annotate the message sent by the client to the server through the @param annotation, use the onMessage instruction to listen to the message, when the client and the server send a message, the message will be called back through the log.info function to verify the parameters in the message, and when the parameter starts with chromepickerstart:, it means that the picker is started;

[0011] The specific steps to build the locator websocket interface are as follows: define the publicly accessible server LocatorSocketServer through the class class, establish a private connection session with the client through the Session function, when the connection status openFlag is true, the connection is judged to be successful, when the connection status openFlag is false, the connection is judged to be failed, the connection status openFlag is output to the server LocatorSocketServer through the log.info function, annotate the message sent by the client to the server through the @param annotation, use the onMessage instruction to listen to the message, when the client and the server send a message, the message will be called back through the log.info function to verify the parameters in the message. When the parameter starts with chromelocatorstart:, it means that the locator is started.

[0012] Furthermore, in the S2, the keyboard listener keyboard_listener and the mouse listener mouse_listener are started through the start function. When the esc key of the keyboard is pressed, it indicates that the pickup is interrupted, the mouse listener mouse_listener is stopped, the highlighted rectangle currently drawn by the win32gui library is refreshed, that is, the highlighted rectangle is removed, and the interrupt pickup information is returned to the caller;

[0013] When the left ctrl key is pressed, the current desktop control is picked up and the coordinates at the time of picking are recorded. The mouse listener mouse_listener is stopped, the highlighted rectangle currently drawn by the win32gui library is refreshed, that is, the highlighted rectangle is removed, the screenshot flag screen_flag is True, a screenshot is taken, and the coordinates of the top, bottom, left, and right of the screenshot area are recorded as the screenshot position, and the picking is terminated.

[0014] The specific process of keyboard listening keyboard_listener is as follows: when taking a screenshot, construct a screenshot name through the user identity uid, construct a screenshot area for taking a screenshot, construct a return parameter, that is, the screenshot location, and output the return parameter to the foreground process designer;

[0015] The specific process of the mouse listening mouse_listener is as follows: obtain the global COM object of the UIAutomation library under the current mouse coordinates, obtain the UIAElement class according to the global COM object, obtain the basic packaging wrapper class of the UI automation element, if the cached object is changed compared with the current wrapper class, that is, the desktop control under the mouse position has changed, based on the win32gui library, the Redraw redraw instruction is used to refresh the highlight rectangle, and the refreshed current highlight rectangle is obtained through the wrapper class, and the corresponding desktop control is updated at the same time, and the highlight rectangle around the desktop control is drawn to obtain the coordinates of the highlight rectangle.

[0016] Furthermore, in said S3, the specific process of the desktop control positioning method is as follows: obtaining the control information obtained by the picker, formatting it, obtaining the coordinates of the desktop control, obtaining the desktop control through the classname class, that is, generating the boundary of the highlighted rectangular box by the app class to obtain the desktop control, if the acquisition is successful and the control information corresponding to the above desktop control matches the control information obtained by the picker, then calculating the center coordinates of the desktop control according to the coordinates of the highlighted rectangular box and outputting it to the foreground process designer, if the match is not successful, executing the control screenshot positioning method;

[0017] The specific process of the control screenshot positioning method is as follows: call the pyautogui library in the current desktop to find the screenshot location through the locateOn instruction to obtain the desktop control. If the acquisition is successful and the control information corresponding to the above desktop control matches the control information obtained by the picker, the center coordinates of the desktop control are calculated according to the screenshot position and output to the foreground process designer. If the match is not successful, the coordinate positioning method at the time of picking is executed;

[0018] The specific process of the coordinate positioning method during picking is as follows: the center coordinates of the desktop control are calculated according to the coordinates during picking and output to the foreground process designer.

[0019] The beneficial effects of the present invention are as follows: the present invention provides support for the integration of large-scale RPA systems, and integrates the picker and locator by connecting the picker websocket interface and the locator websocket interface of the RPA system's front-end process designer to realize the invocation of the picker and locator and data interaction; the present invention only needs to import the program package docking interface to enable the RPA system to obtain the ability to pick up and locate desktop controls. Compared with the abstract operation process of the existing technology, the present invention optimizes the process. The user can move the mouse at will to select the relevant information of the currently highlighted desktop control, so as to facilitate the positioning of the desktop control in the subsequent automated process. The locator of the present invention verifies the accuracy of the control information through the desktop control positioning method, the control screenshot positioning method and the coordinate positioning method during picking, providing highly reliable and available support for the picker acquisition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0021] Figure 1 A flowchart of a method for picking and locating desktop controls for an RPA system;

[0022] Figure 2 Schematic diagram of an embodiment of a method for picking and locating desktop controls in an RPA system;

[0023] Figure 3 A schematic diagram of the embodiment code for constructing the first part of the pickup websocket interface;

[0024] Figure 4 A schematic diagram of the embodiment code for constructing the second part of the pickup websocket interface;

[0025] Figure 5 A schematic diagram of the embodiment code for the third part of building the pickup websocket interface;

[0026] Figure 6 A schematic diagram of the embodiment code for constructing the first part of the locator websocket interface;

[0027] Figure 7 A schematic diagram of the embodiment code for constructing the second part of the locator websocket interface;

[0028] Figure 8 A schematic diagram of the code for the third embodiment of building a locator websocket interface;

[0029] Figure 9 A schematic diagram of the embodiment code of the first part of the connection between the foreground process designer and the picker websocket interface and the locator websocket interface;

[0030] Figure 10 A schematic diagram of the embodiment code of the second part of the connection between the foreground process designer and the picker websocket interface and the locator websocket interface;

[0031] Figure 11 A schematic diagram of the embodiment code of the first part of picking up by a picker;

[0032] Figure 12 A schematic diagram of the embodiment code of the second part of picking up by a picker;

[0033] Figure 13 A schematic diagram of the embodiment code of the third part for picking up with a picker;

[0034] Figure 14 A schematic diagram of the embodiment code of the fourth part for picking up with a picker;

[0035] Figure 15 A schematic diagram of code for an embodiment of positioning using a locator. DETAILED DESCRIPTION

[0036] To make the technical solutions and advantages of the embodiments of the present invention more clearly understood, exemplary embodiments of the present invention are further described in detail below with reference to the accompanying drawings. It should be noted that the embodiments described are only a portion of the embodiments of the present invention, and are not an exhaustive list of all embodiments. It should be noted that the embodiments of the present invention and the features thereof may be combined with each other unless they conflict.

[0037] refer to Figures 1-15 The present embodiment describes in detail a method for picking and locating desktop controls in an RPA system, specifically including the following steps:

[0038] S1. Build and connect the picker websocket interface and the locator websocket interface. The RPA system's front-end process designer calls the picker and locator by connecting to the picker websocket interface and the locator websocket interface.

[0039] S2. The front-end process designer calls the picker, which monitors the mouse and keyboard for user operations. It then uses the win32gui library to draw a highlighted rectangle around the desktop control in real time. When the desktop control under the mouse position changes, the highlighted rectangle is refreshed to obtain control information and echo it back to the front-end process designer.

[0040] S3. The foreground process designer calls the locator and uses the desktop control positioning method, the control screenshot positioning method, and the coordinate positioning method during picking to obtain the desktop control and its corresponding control information in sequence until the control information obtained by the picker is successfully matched. The center coordinates of the desktop control are calculated and echoed back to the foreground process designer.

[0041] Furthermore, in the S1, the specific steps of constructing the picker websocket interface are as follows: define the publicly accessible server PickerSocketServer through the class class, establish a private connection session with the client through the Session function, when the connection status openFlag is true, it is judged that the connection is successful, when the connection status openFlag is false, it is judged that the connection fails, the connection status openFlag is output to the server PickerSocketServer through the log.info function, annotate the message sent by the client to the server through the @param annotation, use the onMessage instruction to listen to the message, when the client and the server send a message, the message will be called back through the log.info function to verify the parameters in the message, and when the parameter starts with chromepickerstart:, it means that the picker is started;

[0042] The specific steps to build the locator websocket interface are as follows: define the publicly accessible server LocatorSocketServer through the class class, establish a private connection session with the client through the Session function, when the connection status openFlag is true, the connection is judged to be successful, when the connection status openFlag is false, the connection is judged to be failed, the connection status openFlag is output to the server LocatorSocketServer through the log.info function, annotate the message sent by the client to the server through the @param annotation, use the onMessage instruction to listen for the message, when the client and the server send a message, the message will be called back through the log.info function to verify the parameters in the message. When the parameter starts with chromelocatorstart:, it means that the locator is started;

[0043] Specifically, refer to Figure 3-Figure 8 After the WebSocket communication connection is established, the client and server can communicate bidirectionally through the connection. The present invention uses WebSocket communication as the server to realize the long-term bidirectional communication between the client, that is, the foreground process designer, the picker and the locator. Figure 9 and Figure 10 , as the calling code of the picker and locator, there is no need to pay attention to the language, it only needs to support websocket communication. In this embodiment, the front-end technology is used to implement it;

[0044] Furthermore, in S2, the keyboard listener keyboard_listener and the mouse listener mouse_listener are started through the start function. When the esc key of the keyboard is pressed, it indicates that the pickup is interrupted, and the mouse listener mouse_listener is stopped. The highlighted rectangular frame currently drawn by the win32gui library is refreshed through "win32gui.RedrawWindow(None,None,None,win32con.RDW_FRAME|win32con.RDW_INVA LIDATE|win32con.RDW_UPDATENOW|win32con.RDW_ALLCHILDREN)", that is, the highlighted rectangular frame is removed, and the interrupt pickup information is returned to the caller;

[0045] When the left ctrl key is pressed, the current desktop control is picked up and the coordinates at the time of picking are recorded. The mouse listener mouse_listener is stopped, the highlighted rectangle currently drawn by the win32gui library is refreshed, that is, the highlighted rectangle is removed, the screenshot flag screen_flag is True, a screenshot is taken, and the coordinates of the top, bottom, left, and right of the screenshot area are recorded as the screenshot position, and the picking is terminated.

[0046] The specific process of keyboard listening keyboard_listener is as follows: when taking a screenshot, construct a screenshot name through the user identity uid, construct a screenshot area for taking a screenshot, construct a return parameter, that is, the screenshot location, and output the return parameter to the foreground process designer;

[0047] The specific process of the mouse listening mouse_listener is as follows: obtain the global COM object of the UIAutomation library under the current mouse coordinates, obtain the UIAElement class according to the global COM object, obtain the basic wrapper class of the UI automation element, if the cache object is changed compared with the current wrapper class, that is, the desktop control under the mouse position has changed, refresh the highlight rectangle based on the win32gui library through the Redraw redraw instruction, and obtain the refreshed current highlight rectangle through the wrapper class, and at the same time update the corresponding desktop control, draw the highlight rectangle around the desktop control, and obtain the coordinates of the highlight rectangle;

[0048] Specifically, refer to Figure 11-Figure 15, use python script to write pickers and locators and integrate them into the websocket server written in Java. The pickers and locators need to call the underlying methods of desktop controls. The python language provides a wealth of related libraries such as the pyautogui library and the win32gui library. After calling the picker, the mouse monitor collects the controls under the user's mouse coordinates in real time, and uses UIA (user automation design) technology to obtain desktop control information, such as the desktop control classname and the coordinates around the desktop control, that is, the upper, lower, left and right coordinates of the screenshot area. UIA technology is mainly used to store global COM objects related to the UIAutomation library. In the use of pywinauto (computer-side automation tool), UIA technology is a technical framework provided for the Windows operating system for realizing automated testing of user interfaces and accessibility.

[0049] Furthermore, in said S3, the specific process of the desktop control positioning method is as follows: obtaining the control information obtained by the picker, formatting it, obtaining the coordinates of the desktop control, obtaining the desktop control through the classname class, that is, generating the boundary of the highlighted rectangular box by the app class to obtain the desktop control, if the acquisition is successful and the control information corresponding to the above desktop control matches the control information obtained by the picker, then calculating the center coordinates of the desktop control according to the coordinates of the highlighted rectangular box and outputting it to the foreground process designer, if the match is not successful, executing the control screenshot positioning method;

[0050] The specific process of the control screenshot positioning method is as follows: call the pyautogui library in the current desktop to find the screenshot location through the locateOn instruction to obtain the desktop control. If the acquisition is successful and the control information corresponding to the above desktop control matches the control information obtained by the picker, the center coordinates of the desktop control are calculated according to the screenshot position and output to the foreground process designer. If the match is not successful, the coordinate positioning method at the time of picking is executed;

[0051] The specific process of the coordinate positioning method when picking is as follows: the center coordinates of the desktop control are calculated according to the coordinates when picking and output to the foreground process designer;

[0052] Specifically, refer to Figure 15The locator uses three methods to realize the positioning of desktop controls. Its core intention is to determine the center coordinates of the control for subsequent automated process operations. The desktop control positioning method mainly uses UIA technology to realize the search of desktop controls on the current desktop. The control screenshot positioning method is implemented using the pyautogui library. The underlying logic is to use the opencv library to perform feature extraction first. A variety of feature extraction algorithms can be used, such as SIFT (feature extraction algorithm based on scale-invariant feature transform), SURF (accelerated robust feature extraction algorithm), HOG (histogram of directional gradients feature extraction algorithm), etc. The above feature extraction algorithms can find local key points in the image and calculate their The corresponding feature vector is then used for target recognition. After feature extraction, the OpenCV library provides the target recognition function. Target recognition refers to identifying the target in the image based on the extracted feature vector, and the OpenCV library supports a variety of target recognition methods including machine learning methods and template matching-based methods. Among them, machine learning methods such as SVM (support vector machine) and KNN (K nearest neighbor algorithm) can determine the category of the target in the input image based on the similarity of the feature vector. If the desktop control positioning method and the control screenshot positioning method cannot obtain the control coordinates, the coordinates at the time of picking are directly selected for subsequent calculation and output to the front-end process designer.

[0053] refer to Figure 2 The locator is used to verify the control information obtained by the picker. If the control information matches successfully, the corresponding desktop control will be highlighted and flash three times.

[0054] Although the present invention has been described with respect to a limited number of embodiments, it will be apparent to those skilled in the art, having benefit of the foregoing description, that other embodiments are contemplated within the scope of the invention thus described. Furthermore, it should be noted that the language used in this specification has been selected primarily for readability and didactic purposes, rather than for the purpose of explaining or limiting the subject matter of the present invention. Consequently, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. A desktop control picking and positioning method for an RPA system, characterized in that: The following steps are involved: S1. Build the picker websocket interface and the locator websocket interface. The RPA system's front-end process designer calls the picker and locator by connecting to the picker websocket interface and the locator websocket interface. S2. The front-end process designer calls the picker, which monitors the mouse and keyboard for user operations. It then uses the win32gui library to draw a highlighted rectangle around the desktop control in real time. When the desktop control under the mouse position changes, the highlighted rectangle is refreshed to obtain control information and echo it back to the front-end process designer. S3. The front-end process designer calls the locator and sequentially obtains the desktop control and its corresponding control information using the desktop control positioning method, the control screenshot positioning method, and the coordinate positioning method at the time of picking. It then successfully matches the control information obtained by the picker. The center coordinates of the desktop control are calculated and returned to the front-end process designer. In the S1, the specific steps of constructing the picker websocket interface are as follows: define the publicly accessible server PickerSocketServer through the class class, establish a private connection session with the client through the Session function, when the connection status openFlag is true, it is judged that the connection is successful, when the connection status openFlag is false, it is judged that the connection fails, the connection status openFlag is output to the server PickerSocketServer through the log.info function, annotate the message sent by the client to the server through the @param annotation, use the onMessage instruction to monitor the message, when the client and the server send a message, the message will be called back through the log.info function to verify the parameters in the message. When the parameter starts with chromepickerstart:, it means that the picker is started; The specific steps to build the locator websocket interface are as follows: define the publicly accessible server LocatorSocketServer through the class class, establish a private connection session with the client through the Session function, when the connection status openFlag is true, the connection is judged to be successful, when the connection status openFlag is false, the connection is judged to be failed, the connection status openFlag is output to the server LocatorSocketServer through the log.info function, annotate the message sent by the client to the server through the @param annotation, use the onMessage instruction to listen for the message, when the client and the server send a message, the message will be called back through the log.info function to verify the parameters in the message. When the parameter starts with chromelocatorstart:, it means that the locator is started; In said S3, the specific process of the desktop control positioning method is as follows: obtain the control information obtained by the picker, format it, obtain the coordinates of the desktop control, obtain the desktop control through the classname class, that is, generate the boundary of the highlighted rectangular box by the app class to obtain the desktop control, if the acquisition is successful and the control information corresponding to the above desktop control matches the control information obtained by the picker, then calculate the center coordinates of the desktop control according to the coordinates of the highlighted rectangular box and output it to the foreground process designer, if the match is not successful, execute the control screenshot positioning method; The specific process of the control screenshot positioning method is as follows: call the pyautogui library in the current desktop to find the screenshot location through the locateOn instruction to obtain the desktop control. If the acquisition is successful and the control information corresponding to the above desktop control matches the control information obtained by the picker, the center coordinates of the desktop control are calculated according to the screenshot position and output to the foreground process designer. If the match is not successful, the coordinate positioning method at the time of picking is executed; The specific process of the coordinate positioning method during picking is as follows: the center coordinates of the desktop control are calculated according to the coordinates during picking and output to the foreground process designer.

2. The desktop control picking and positioning method for an RPA system according to claim 1, characterized in that: In the S2, the keyboard listener keyboard_listener and the mouse listener mouse_listener are started through the start function. When the esc key of the keyboard is pressed, it indicates that the picking is interrupted, the mouse listener mouse_listener is stopped, the highlighted rectangle currently drawn by the win32gui library is refreshed, that is, the highlighted rectangle is removed, and the interrupt picking information is returned to the caller; When the left ctrl key is pressed, the current desktop control is picked up and the coordinates at the time of picking are recorded. The mouse listener mouse_listener is stopped, the highlighted rectangle currently drawn by the win32gui library is refreshed, that is, the highlighted rectangle is removed, the screenshot flag screen_flag is True, a screenshot is taken, and the coordinates of the top, bottom, left, and right of the screenshot area are recorded as the screenshot position, and the picking is terminated. The specific process of keyboard listening keyboard_listener is as follows: when taking a screenshot, construct a screenshot name through the user identity uid, construct a screenshot area for taking a screenshot, construct a return parameter, that is, the screenshot location, and output the return parameter to the foreground process designer; The specific process of the mouse listening mouse_listener is as follows: obtain the global COM object of the UIAutomation library under the current mouse coordinates, obtain the UIAElement class according to the global COM object, obtain the basic packaging wrapper class of the UI automation element, if the cached object is changed compared with the current wrapper class, that is, the desktop control under the mouse position has changed, based on the win32gui library, the Redraw redraw instruction is used to refresh the highlight rectangle, and the refreshed current highlight rectangle is obtained through the wrapper class, and the corresponding desktop control is updated at the same time, and the highlight rectangle around the desktop control is drawn to obtain the coordinates of the highlight rectangle.

Images