A numbering method, positioning method, device, equipment and medium for UI elements

Abstract

The present invention discloses a numbering method, a positioning method, a device, a device and a medium for UI elements. The method includes identifying UI elements in a received UI image; extracting a first region contour in the UI image; determining a second region contour therein based on the first region contour, wherein the second region contours do not overlap; performing numbering on the second region contours to obtain region numbers of the second region contours; and performing numbering on the UI elements within the second region contours based on the region numbers. The present invention takes into account the invariance of the relative positions of UI elements within a certain region, is not easily affected by the screen size, resolution and zoom ratio, has good stability, and can improve the accuracy of UI element positioning.

Description

Technical Field

[0001] The present invention relates to a method for numbering UI elements, a positioning method, a device, a device and a medium, belonging to the technical field of RPA process design. Background Art

[0002] RPA (Robotic Process Automation) technology is a technology based on process automation robots to replace humans to perform some repetitive tasks. The positioning of UI elements is the key to the successful operation of RPA.

[0003] In scenarios involving remote desktop applications, JAVA applications, and custom graphical interface applications, etc., the general element identification method is no longer effective. A feasible way in the prior art is to number UI elements to achieve the positioning of UI elements. However, this method is easily affected by screen size, resolution, and zoom ratio, and is unstable. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method for numbering UI elements, a positioning method, a device, a device and a medium, which are used to solve the problem of poor stability of UI element numbering.

[0005] To achieve the above purpose, the present invention is implemented by the following technical solutions:

[0006] In the first aspect, the present invention provides a method for region division, including:

[0007] Identifying UI elements in the received UI image;

[0008] Extracting the first region contour from the UI image;

[0009] Based on the first region contour, determining the second region contours therein, where the second region contours do not overlap;

[0010] Numbering the second region contours to obtain the region numbers of the second region contours;

[0011] Based on the region numbers, numbering the UI elements within the second region contours.

[0012] Further, the step of identifying UI elements in the received UI image includes:

[0013] Based on a pre-trained object detection model, identifying UI elements in the received UI image.

[0014] Furthermore, in the case where the first area outlines are overlapped, the step of extracting the first area outlines from the UI image includes:

[0015] Processing the UI image using a Canny-based edge detection algorithm to obtain a UI image with a region outline;

[0016] Contour extraction is performed on the UI image having the region contour to obtain the first region contour.

[0017] Furthermore, before the step of determining the second area outline based on the first area outline, the method further includes:

[0018] For the first region outlines, first region outlines that do not meet a preset size threshold are eliminated.

[0019] Furthermore, the step of determining the second area outline based on the first area outline includes:

[0020] The UI image and the first area outline are used as input and processed in a preset dynamic programming algorithm to obtain the second area outline; wherein, the constraint condition of the dynamic programming algorithm is that the area outlines do not overlap in the UI image, and the objective function is to minimize the total interval between the area outlines.

[0021] Furthermore, the step of determining the second area outline based on the first area outline includes:

[0022] Based on the first area outline and the UI image, starting from any side of the UI image and searching for the outline of the first area outline within a certain step length;

[0023] Keep the longest contour line in each step size and remove the rest of the contour lines;

[0024] Both ends of the longest contour line are extended to the edge of the UI image to obtain the second contour area.

[0025] In a second aspect, the present invention provides a UI element positioning method, which is applied to the RPA process execution stage and includes:

[0026] For the UI image to be located, apply any of the aforementioned UI element numbering methods to obtain the numbers of the UI elements in the UI image to be located;

[0027] Based on the numbers of the UI elements pre-configured in the RPA process, the same numbers are searched from the numbers of the UI elements in the UI image to be located to achieve the location of the UI elements.

[0028] In a third aspect, the present invention provides a numbering device for UI elements, including:

[0029] An identification module, which is used to identify UI elements in the received UI image;

[0030] An extraction module, which is used to extract the contour of the first region in the UI image;

[0031] A determination module, which is used to determine the contour of the second region therein based on the contour of the first region, wherein the contours of the second regions do not overlap;

[0032] A region numbering module, which is used to perform numbering on the contour of the second region to obtain the region number of the contour of the second region;

[0033] A UI element numbering module, which is used to perform numbering on the UI elements within the contour of the second region based on the region number.

[0034] In a fourth aspect, the present invention provides an electronic device, including a processor and a storage medium;

[0035] The storage medium is used to store instructions;

[0036] The processor is used to operate according to the instructions to execute the steps of the method according to any one of the foregoing.

[0037] In a fifth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method according to any one of the foregoing are implemented.

[0038] Compared with the prior art, the beneficial effects achieved by the present invention:

[0039] The present invention provides a numbering method, a positioning method, a device, a device and a medium for UI elements. First, the contour of the first region in the UI image is detected by an edge detection algorithm, then the non-overlapping contour of the second region is determined from the contour of the first region, and then numbering is performed based on the contour of the second region to obtain the region number. Finally, based on the region number, numbering is performed on the UI elements within each contour of the second region. This method takes into account the invariance of the relative positions of UI elements within a certain region, is not easily affected by the screen size, resolution and zoom ratio, has good stability, and can improve the accuracy of UI element positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flowchart of a numbering method for UI elements provided by an embodiment of the present invention.

[0041] Figure 2 is a block diagram of a numbering device for UI elements provided by an embodiment of the present invention.

[0042] Figure 3 This is a block diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0043] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and cannot be used to limit the protection scope of the present invention.

[0044] In the related art, the implementation of RPA technology can be divided into the design stage of the RPA process and the execution stage of the RPA process.

[0045] In the design stage, it is necessary to determine the UI elements clicked on the UI interface in the subsequent automated process. These UI elements can be a certain button, dropdown box or other components.

[0046] In the execution stage, the RPA robot can locate the target UI element according to the conventional element location method, so that the automated process can run normally. This conventional element location method, for example, is to use general element identification methods such as UIA on the Windows system and xpath (XML Path Language) of web pages to locate UI elements.

[0047] When the application scenarios of RPA involve remote desktop applications, JAVA applications, and custom graphical interface applications, the above conventional element location methods are no longer applicable. In the prior art, a feasible method is to use CV (computer vision) computer vision recognition technology, such as deep learning technology, to identify and locate elements. When performing CV recognition, the entire application is first saved as an image, and then elements such as icons, text blocks, buttons, and edit boxes on the image are recognized. After obtaining the position information and content of these UI elements (such as the text information of text-based UI elements), they are stored in the RPA process as the identifiers of these elements. If the RPA robot needs to find them, or compare whether two UI elements are the same, it can directly use the position information and content of the UI elements for location or comparison, but the position information is difficult to remember and not intuitive, and cannot be directly used. To solve this problem, another feasible way is to number all the UI elements in the UI interface in the design stage, select the numbers of the UI elements to be operated and write them into the RPA script. In the execution stage, the RPA robot first obtains each UI element of the UI image through object detection and then numbers them, and locates the target UI elements in the execution stage based on the numbers written in the RPA script in the design stage.

[0048] However, this method of numbering UI elements to achieve UI element positioning has some defects. For example, during the actual operation of the RPA robot, the UI interface that requires UI element positioning often changes. For example, the screen size changes, the resolution changes, the UI interface is scaled, and so on. Once the above changes occur, it will affect the numbering of UI elements in the UI interface during the execution stage, making the numbering of UI elements in the design stage not correspond to the numbering of UI elements in the execution stage. Therefore, accurate UI element positioning cannot be achieved.

[0049] The present invention provides a method and device for numbering UI elements, a positioning method, a device, a device, and a storage medium. First, the edge detection algorithm is used to detect the first area contour in the UI image, then the non-overlapping second area contour is determined from the first area contour, and then the area number is obtained by numbering based on the second area contour. Finally, based on the area number, the UI elements in each second area contour are numbered. The UI element numbers obtained by the method of the present invention consider both the absolute position of the UI elements in the UI interface and the relative position of the UI elements in the area. Therefore, it has good stability. Correspondingly, when performing UI element positioning, it also has high accuracy.

[0050] Embodiment 1

[0051] As Figure 1 shown, this embodiment introduces a method for numbering UI elements, including:

[0052] Step S100, identify the UI elements in the received UI image.

[0053] In this embodiment, the above identification action can be represented as identifying the UI elements existing in the UI image based on a pre-trained object detection model. The UI elements may include icon-based elements or text-based elements. The text-based elements may include single-line text elements or multi-line text elements.

[0054] In this embodiment, the pre-trained object detection model can be pre-trained by using the object detection algorithm of the YOLO series. For example, a large number of pre-annotated UI images are used as training data to train the YOLOv5 object detection model, so as to obtain the above pre-trained object detection model. The output result of the model is the position information and category information of the UI elements existing in the UI image.

[0055] In this embodiment, the above UI image may be a complete screenshot of the display device, or a screenshot of a certain area of the display device, such as a screenshot of a certain application window, a screenshot of a certain page, and so on. The display device may be a PC screen, or a screen of a smart phone, a tablet computer, a digital assistant, and a shopping guide terminal.

[0056] Step S200: Extract the contour of the first region from the UI image.

[0057] In this embodiment, the above extraction operation can be represented as using an edge detection algorithm to extract the region contour existing in the UI image. The region contour existing in the UI image is also the above-mentioned first region contour.

[0058] This step may specifically include the following sub-steps:

[0059] Step S201: Process the UI image using an edge detection algorithm based on Canny to obtain a UI image with a region contour.

[0060] Specifically, consider a specific application scenario. For example, it is required to automatically log in to the email application every day and enter the "Unread Messages" module of the email application, and execute the RPA process of intelligent document extraction for unread messages. In this process, after automatically logging in to the email application, it is necessary to accurately locate the UI element of "Unread Messages" in the main interface of the email application so that the subsequent intelligent document extraction operations can be normally executed and the RPA process can run smoothly. The main interface of this email application can be defined as the above-mentioned UI image. After converting the UI image into a corresponding grayscale image, the Canny detection algorithm with a kernel of 5*5 can be used to perform the detection of the boundary contour.

[0061] In this implementation method, using the Canny detection algorithm with a kernel of 5*5 can better distinguish the boundary from the background compared to the 3*3 in the prior art. It can better reflect the region contour than the binaryzation in the traditional method.

[0062] In some embodiments, a morphological transformation method based on closing operation can also be used to process the UI image with a region contour to strengthen the continuity of the region contour.

[0063] The region contour obtained in step S201 may not be able to optimally represent the region boundary information in the UI image. Therefore, some optimization operations are required. Therefore, in some embodiments, the closing operation in morphological transformation, that is, first image dilation and then image erosion, can be used to strengthen the edge detection result, making the UI boundary contour more continuous. The closing operation uses a 3*3 kernel, a cross-shaped kernel. The transformed contour information has stronger adhesion and better boundary sorting.

[0064] Step S202: Perform contour extraction on the UI image with a region contour to obtain the first region contour.

[0065] In this embodiment, the operation of contour extraction is to extract the contours of each region in the UI image, and convert them into boundary information through numerical calculation. For example, calculate the maximum rectangular boundary of the contour, etc. This boundary information is also the position information of the region contour in the image, which can be understood as a set of multiple position coordinates. Specifically, the (cv2.findcontours) in the opencv library can be used to perform contour extraction.

[0066] Step S300: Based on the first region contours, determine the second region contours therein, where the second region contours do not overlap with each other.

[0067] In this embodiment, the first region contours extracted for the UI image often overlap. For example, the region contour of the pop-up window often overlaps with the region contour of the toolbar, and the region contour of the pop-up window often covers multiple region contours.

[0068] In this embodiment, the above-mentioned determination action is expressed as performing corresponding processing on the above-mentioned first region contours to obtain the second region contours. It should be noted that these second region contours do not overlap with each other.

[0069] Specifically, the above-mentioned corresponding processing can be expressed as using some optimization algorithms to find the contours that meet the calculation results of the optimization algorithms from the first region contours output in step S202, and these contours do not overlap with each other. These contours are also the above-mentioned second region contours. In this embodiment, the optimization algorithm can use the dynamic programming algorithm or the greedy algorithm to find the region contours that meet the target. It should be noted that this processing will not make any modifications to the first region contours, but only search for the first region contours that meet the target from multiple first region contours, and define this type of contour as the second region contour. The target can be set through the objective function and constraint conditions of the above-mentioned optimization algorithm.

[0070] In this embodiment, the above-mentioned corresponding processing can also be expressed as based on the first region contours output in step S202, removing the edges of some of the contours and extending the edges of some other contours to form new contour regions, and the new contour regions do not overlap with each other. It should be noted that this processing method will modify the first region contours.

[0071] In some embodiments, considering some uncommon situations, for example, the first region arguments obtained in step S202 do not overlap with each other. This may be because in the UI image in step S100, there are too few UI elements, or the layout is simple, etc. For other reasons. In this case, the above-mentioned corresponding processing can be expressed as directly defining the first region contours as the second region contours.

[0072] This implementation method takes into account that there are often overlapping areas in the UI interface, that is, there are overlapping situations in the above-mentioned first contour area, and it is difficult to determine the attribution of UI elements in the overlapping area. Therefore, this implementation method provides a specific method to determine the second contour area from the first area contour, so that each UI element corresponds to only one area.

[0073] Step S400: Number the second contour area to obtain the area number of the second contour area.

[0074] In this implementation manner, the above-mentioned numbering operation can be to number each second contour area in the UI image in a certain order using Arabic numerals, English, and / or Greek letters. For example, it can be numbered in the order from left to right and from top to bottom. In short, it is necessary to ensure that each second contour area has an independent area number, the area numbers do not repeat, and the same numbering rule is adopted for each second contour area.

[0075] Specifically, taking the above-mentioned specific application scenario as an example, the main interface of the above email application adopts the method in this implementation manner, and two second contour areas can be extracted. One of the second contour areas is the left area of the main interface, which has one icon-type UI element, the search box, and four text-type UI elements, "Inbox", "Sent", "Unread Messages", and "Trash", from top to bottom, as the classification labels of emails; the rightmost area of the main interface is another second contour area, which is the attached area for displaying corresponding information after operating on the above four text-type UI elements, and if no operation is performed, it will be displayed as blank. Therefore, the second contour area on the left can be numbered as: 1, or it can be numbered as A1, or it can also be A1α. Correspondingly, the second contour area on the rightmost can be numbered as: 2, or it can be numbered as B2, or it can also be B2β.

[0076] More specifically, if one of the second contour areas uses Arabic numerals for numbering, then the remaining second contour areas must also use Arabic numerals for numbering. Correspondingly, if one of the second contour areas uses a numbering method of English + Arabic numerals or English + Arabic numerals + Greek letters, then the remaining second contour areas must also use this numbering method. In other words, if the area numbers are regarded as strings, each string has the same number of digits, and the same language is used for the same digit positions among the strings.

[0077] Step S500: Number the UI elements within the second contour area based on the area number.

[0078] In this embodiment, the numbers in this step are represented as follows: based on the region numbers in step S400, numbers are respectively assigned to the UI elements inside the corresponding regions on this basis, so as to obtain the UI element numbers of each UI element in the UI interface, which can be defined as the global UI element numbers. In other words, the global UI element numbers include two parts. The first part, that is, the head of the number, is the region number of the contour of the second region corresponding to the UI element, and the part after the region number is the number of the UI element inside the region contour, which can be defined as the local UI element number.

[0079] Specifically, the region number can be directly placed at the head of the local UI element number to form the global UI element number, or a separator can be added between the region number and the local UI element number for distinction, such as @, -, / , etc.

[0080] In this embodiment, when assigning local UI element numbers, the numbering rules in step S400 also need to be satisfied.

[0081] In some embodiments, considering the computational complexity, before executing step S300, that is, before determining the second region contour from the first region contour, some preprocessing can be done. For example, for the first region contour, the first region contours that do not meet the preset size threshold can be removed. The preset size threshold can refer to UI elements such as buttons, that is, the region contours of buttons detected by canny can be removed. Such overly small region contours often have too few UI elements and are not valuable for forming the second region contour.

[0082] In some embodiments, a specific method is provided for determining the second region contour therein based on the first region contour. The method may include: taking the UI image and the first region contour as inputs and processing them in a preset dynamic programming algorithm to obtain the second region contour; wherein, the constraint condition of the dynamic programming algorithm is that in the UI image, the region contours do not overlap, and the objective function is that the total interval between the region contours is the smallest.

[0083] Specifically, the preset dynamic programming algorithm may include:

[0084] 101. Define the state: Define dp[i][j] as the minimum value of the total interval between the region contours when selecting the first i region contours and the last region contour is the jth one.

[0085] 102. Define the state transition equation: According to the constraint conditions, a state transition equation can be designed to update the dp array. Assume that the upper left coordinate of the contour of the i-th region is (x_i, y_i), and the lower right coordinate is (x_i', y_i'), then the state transition equation can be defined as:

[0086] dp[i][j] = min(dp[i - 1][k] + interval(k, j))

[0087] In the above formula, interval(k, j) represents the interval between the contour of region k and the contour of region j, which can be defined according to specific requirements. For example, the Euclidean distance between the boundaries of the two region contours can be used as the interval. k satisfies the following conditions:

[0088] The contour of region i does not overlap with the contour of region k, that is, (x_i', y_i') is not inside the contour of region k, and (x_i, y_i) is not inside the contour of region k.

[0089] There is no other region contour between the contour of region i and the contour of region k, that is, there are no points of other region contours between (x_i, y_i) and (x_i', y_i').

[0090] 103. Initialization: Initialize the dp array to a relatively large value, indicating that no region contour is selected.

[0091] 104. Recursive calculation: Calculate the values of the dp array from left to right and from top to bottom according to the state transition equation. Finally, the value of dp[m][n] is the minimum total interval of the selected region contours, which is the objective function F, and its expression can be: F = Σinterval(k, j).

[0092] 105. Backtracking solution: According to the values of the dp array, the selected region contours can be found by backtracking. Starting from dp[m][n], gradually backtrack forward according to the state transition equation to find the region contours that meet the constraint conditions.

[0093] In some embodiments, a graph-based optimization method is provided for determining a second region contour among the first region contours. The method may include:

[0094] 201. Construct a graph: Use each first region contour as a node of the graph, and create an edge between the first region contours with overlap, thereby obtaining the graph. The overlap may include: partial overlap of two first region contours, that is, there is an intersection, and complete overlap, that is, one first region contour is completely inside another first region contour.

[0095] 202. Assign node weights: For nodes corresponding to larger outlines of the first region, the weights of the nodes are greater.

[0096] 203. Construct an optimization problem: Perform segmentation on the graph with assigned node weights to obtain multiple subgraphs, and there are no interconnected nodes between each subgraph, that is, there are no overlapping outlines of the first region, and the sum of the weights of the subgraphs, that is, the sum of the weights of all nodes in the subgraph, is the largest, thereby constructing an optimization problem. In this embodiment, a simulated annealing algorithm can be used to solve this optimization problem.

[0097] 204. Final verification: Verify whether the subgraphs obtained in 203 meet the coverage condition, that is, whether all regions in the UI interface are covered without omission. If there are omissions, adjust the weight parameters in the simulated annealing algorithm in 203.

[0098] In some embodiments, a specific method is also provided for determining the second region outline therein based on the first region outline. The method may include: based on the first region outline and the UI image, starting from any side of the UI image, and searching for the outline lines of the first region outline within a certain step size; retaining the longest outline line within each step size and removing the remaining outline lines; extending both ends of the longest outline line until the side of the UI image to obtain the second contour region.

[0099] Specifically, for a UI image, starting from any one of the top, bottom, left, and right sides, for example, starting from the left side, the step size can be 150 pixels, and search for the outline lines within 150 pixels starting from the left side. There may be vertical and horizontal outline lines. Compare the lengths of the outline lines. In many cases, the horizontal outline line often extends beyond the area where the 150 pixels are located. For this situation, it can be considered that the horizontal outline line has a length of only 150 pixels and participates in the comparison of the outline line lengths. If it is not the longest outline line, the length of 150 pixels can be removed, or the original outline line corresponding to the 150 - pixel outline line can be removed entirely.

[0100] In the above - mentioned embodiment, it is also possible to only compare the outline lines in the corresponding search direction. For example, when searching from the left side or the right side, only compare the vertical outline lines, retain the longest vertical outline line, remove the remaining vertical outline lines, and then extend the longest vertical outline line. Correspondingly, after implementing the segmentation in the vertical direction, it is possible to start searching from the upper side or the lower side and execute the above - mentioned method to achieve the segmentation in the horizontal direction.

[0101] In summary, the present invention proposes a method for numbering UI elements. This method first detects the contour of the first region in the UI image through an edge detection algorithm, then determines non-overlapping second region contours from the first region contour, then performs numbering based on the second region contour to obtain a region number, and finally, based on the region number, numbers the UI elements within each second region contour. The UI element numbers obtained by this method take into account the region numbers of the regions where the UI elements are located, that is, the region numbers. The region numbers are not affected by the screen size, window size, or screen scaling ratio, nor are they affected by changes in resolution. Therefore, in actual use, the region numbers are stable, and the UI elements within a certain region are also stable. Therefore, a stable UI element number can be obtained through the method of the present invention, enabling the RPA robot to accurately locate the UI elements. And in the method of the present invention, considering that there are often overlapping regions in the UI interface in most cases, a specific method is given to determine non-overlapping second region contours from the first region contour, so that each UI element corresponds to only one region, and each UI element has only one region number.

[0102] Embodiment 2

[0103] This embodiment provides a method for locating UI elements. The locating method is applied to the RPA process execution stage, and the locating method includes:

[0104] Step S600: For the UI image to be located, apply the method for numbering UI elements described in Embodiment 1 to obtain the numbers of the UI elements in the UI image to be located.

[0105] Step S700: Based on the numbers of the UI elements pre-configured in the RPA process, search for the same numbers from the numbers of the UI elements in the UI image to be located to achieve the location of the UI elements.

[0106] Specifically, when designing the RPA process, first use the method in steps S100 - S500 in Embodiment 1 above for the UI image, such as the screenshot of the main interface of the above mailbox application, to generate the above-mentioned global UI element numbers for each UI element in the main interface. When the user selects a certain UI element, the picture of the UI element and the global UI element number are recorded in the RPA process. In other words, the picture of the UI element and the global UI element number are configured in the RPA process.

[0107] When the RPA robot executes this process, the RPA robot first obtains a UI image in the same way, such as by taking a screenshot, to obtain the main interface of the above-mentioned mailbox application. For the UI image of the main interface of the mailbox application, the UI element global numbers of each UI element can also be obtained by using the method in steps S100 - S500. Therefore, according to the UI element global numbers already configured in the RPA process, the UI element with the same number can be found from the current UI image, and the position of the UI element can be obtained, thus realizing the positioning of the UI element. After realizing the positioning of the UI element, subsequent operations can be executed.

[0108] Embodiment 3

[0109] As Figure 3 shown, this embodiment provides a numbering device for UI elements, including:

[0110] An identification module, which is used to identify the UI elements in the received UI image;

[0111] An extraction module, which is used to extract the first area contour in the UI image;

[0112] A determination module, which is used to determine the second area contours therein based on the first area contour, wherein the second area contours do not overlap;

[0113] An area numbering module, which is used to perform numbering on the second area contours to obtain the area numbers of the second area contours;

[0114] A UI element numbering module, which is used to perform numbering on the UI elements within the second area contours based on the area numbers.

[0115] Embodiment 4

[0116] As Figure 3 shown, this embodiment provides an electronic device, including a processor and a storage medium;

[0117] The storage medium is used to store instructions;

[0118] The processor is used to operate according to the instructions to execute the steps of the method according to any one of Embodiment 1.

[0119] Embodiment 5

[0120] This embodiment provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method according to any one of Embodiment 1 are implemented.

[0121] Embodiment 6:

[0122] The present invention also proposes a hardware system applying the UI element numbering method in Embodiment 1. This method can be applied to the terminal side or the server side.

[0123] The terminal can be an electronic device with network access capabilities. For example, the client can be a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a shopping guide terminal, a television set, etc. Alternatively, the client can also be software that can run on the electronic device.

[0124] The server can be an electronic device with certain computing and processing capabilities. For example, the server can be a server of a distributed system, which can be a system with multiple processors, memories, network communication modules, etc. operating in coordination. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server can also be a new technical means capable of implementing the corresponding functions of the embodiments of the specification. For example, it can be a new form of "server" based on quantum computing.

[0125] Embodiment 7:

[0126] The present invention also provides a computer program product containing instructions, which when executed by a computer cause the computer to execute the UI element numbering method in Embodiment 1.

[0127] It can be understood that the specific examples herein are only for helping those skilled in the art to better understand the embodiments of the present specification, rather than limiting the scope of the present invention.

[0128] It can be understood that in various embodiments of the present specification, the magnitudes of the sequence numbers of the processes do not mean the order of execution is prior or posterior. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present specification.

[0129] It can be understood that the various embodiments described in the present specification can be implemented alone or in combination, and the embodiments of the present specification do not limit this.

[0130] Unless otherwise specified, all technical and scientific terms used in the embodiments of the present specification have the same meaning as commonly understood by those skilled in the technical field of the present specification. The terms used in the present specification are only for the purpose of describing specific embodiments, and are not intended to limit the scope of the present specification. The term "and / or" used in the embodiments of the present specification and the appended claims includes any and all combinations of one or more of the related listed items. The singular forms "a", "above-mentioned", and "the" used in the embodiments of the present specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0131] It can be understood that the processor in the embodiments of this specification can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiments can be completed by the integrated logic circuit in the hardware of the processor or instructions in the form of software. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of this specification. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of this specification can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0132] It can be understood that the memory in the embodiments of this specification can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0133] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this specification.

[0134] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0135] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0136] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, in each embodiment of this specification, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0138] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, in essence, or the part that contributes to the prior art or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of this specification. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0139] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and deformations can be made, and these improvements and deformations should also be regarded as the protection scope of the present invention.

Claims

1. A numbering method for UI elements, characterized in that, Including: Identifying UI elements in the received UI image; For the UI image, extracting the first region contour therein; including: Processing the UI image using a canny-based edge detection algorithm to obtain a UI image with region contours; Performing contour extraction on the UI image with region contours to obtain the first region contour; Based on the first region contour, determining the second region contour therein, including: Taking the UI image and the first region contour as inputs and processing them in a preset dynamic programming algorithm to obtain the second region contour; wherein, the second region contours do not overlap; the constraint condition of the dynamic programming algorithm is that among the UI images, the region contours do not overlap, and the objective function is that the total interval between the region contours is the smallest; The dynamic programming algorithm includes:

101. Define dp[i][j] as the minimum value of the total interval between the region contours when selecting the first i region contours and the last region contour is the jth; 102. Assume that the upper left coordinate of the ith region contour is (x_i, y_i) and the lower right coordinate is (x_i', y_i'), then define the state transition equation as: dp[i][j] = min(dp[i - 1][k] + interval(k, j)); wherein, interval(k, j) represents the interval between region contour k and region contour j, and k satisfies the following conditions: The ith region contour does not overlap with the kth region contour, that is, (x_i', y_i') is not inside the kth region contour, and (x_i, y_i) is not inside the kth region contour; There are no other region contours between the ith region contour and the kth region contour, that is, there are no points of other region contours between (x_i, y_i) and (x_i', y_i'); 103. Initialize the dp array to a set large value, indicating that no region contours are selected; 104. Calculate the values of the dp array from left to right and from top to bottom according to the state transition equation. Finally, the value of dp[m][n] is the minimum total interval of the selected region contours, and the expression of the objective function F is: F = Σinterval(k, j); 105. According to the values of the dp array, find the selected region contours by backtracking. Starting from dp[m][n], gradually backtrack forward according to the state transition equation to find the region contours that meet the constraint conditions; Alternatively, based on the first region contour, determining the second region contour therein, including: Based on the first region contour and the UI image, starting from any side of the UI image and searching for the contour line of the first region contour within a certain step size; Retaining the contour line with the longest size within each step size and removing the remaining contour lines; Extending both ends of the longest contour line until the side of the UI image to obtain the second region contour; Performing numbering on the second region contour to obtain the region number of the second region contour; Based on the region number, performing numbering on the UI elements within the second region contour.

2. The numbering method of the UI element according to claim 1, characterized in that, Steps for identifying UI elements in a received UI image, including: Based on a pre-trained object detection model, identify UI elements in the received UI image.

3. The numbering method of the UI element according to claim 1, characterized in that, Before the step of determining the second region contour therein based on the first region contour, further include: For the first region contour, eliminate the first region contours that do not meet the preset size threshold.

4. A method for positioning UI elements, characterized in that, The positioning method is applied in the RPA process execution stage, and the positioning method includes: For the UI image to be positioned, apply the UI element numbering method described in any one of claims 1-3 to obtain the numbers of UI elements in the UI image to be positioned; Based on the numbers of UI elements pre-configured in the RPA process, find the same numbers from the numbers of UI elements in the UI image to be positioned to achieve the positioning of UI elements.

5. A numbering device for UI elements, adopting the numbering method for UI elements described in claim 1, characterized in that, Include: An identification module for identifying UI elements in a received UI image; An extraction module for extracting the first region contour from the UI image; A determination module for determining the second region contour therein based on the first region contour, wherein the second region contours do not overlap; A region numbering module for numbering the second region contours to obtain the region numbers of the second region contours; A UI element numbering module for numbering the UI elements within the second region contour based on the region numbers.

6. An electronic device, characterized in that: Include a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method described in any one of claims 1 to 3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 3.

Images