A UI interface element numbering method, system, device and storage medium

Abstract

The application discloses a UI interface element numbering method, system and device and a storage medium, and belongs to the technical field of RPA. The method comprises the following steps: acquiring a picture containing a UI interface; identifying all elements in the picture; identifying horizontal and vertical split lines in the picture, and performing multi-level splitting on the picture according to the horizontal and vertical split lines; after each splitting, assigning a hierarchical number to each new region obtained by splitting; after the splitting is completed, merging all hierarchical numbers corresponding to the regions in the order of splitting to obtain a region number; assigning a region internal number to each element in the region; adding the region number of the region where the element is located to the region internal number of the element to obtain a global number of the element; and setting the global number of the element as an absolute position defined by the split line and a relative position in the region, so that the global number of the element is not easily affected by other elements and a screen scaling ratio and the like, and has good stability and is convenient for practical application.

Description

Technical Field

[0001] This invention relates to a method, system, device, and storage medium for numbering UI elements, belonging to the field of RPA technology. Background Technology

[0002] When designing an RPA (Robotic Process Automation) workflow, it's necessary to locate elements on the UI (User Interface), such as "Search" or "Submit" buttons, to implement the RPA workflow design. During the execution phase, the RPA robot automatically finds the corresponding elements and executes the workflow, achieving automation. Common element location methods use generic element identification methods like UIA on Windows systems and XPath (XML Path Language) for web pages. However, these methods are ineffective for Java applications, remote desktop applications, or custom graphical interface applications. Computer vision (CV) recognition technology, especially deep learning, is required to identify and locate elements. CV recognition first saves the entire application as an image, then identifies elements such as icons, text blocks, buttons, and edit boxes within the image. After obtaining the position and content of these elements, they are stored in the RPA workflow as identifiers. If the RPA robot needs to find them or compare whether two elements are the same, it can directly use the element's position and content for location or comparison. However, position information is difficult to remember and not intuitive, making it unusable directly. In practical business applications, a simple and easy-to-understand method for numbering UI elements is needed to locate elements. Moreover, this numbering method should not change with screen size, resolution, or scaling ratio, and its stability and accuracy must be guaranteed. Summary of the Invention

[0003] The purpose of this invention is to provide a method, system, device, and storage medium for numbering UI elements, thereby solving the problems of poor usability, stability, and accuracy of element numbering in the prior art.

[0004] To achieve the above objectives, the present invention employs the following technical solution:

[0005] In a first aspect, the present invention provides a method for numbering UI elements, including:

[0006] Get the image containing the UI interface;

[0007] Identify all elements in an image;

[0008] Identify the horizontal and vertical dividing lines in the image, and perform multi-level segmentation of the image based on the horizontal and vertical dividing lines. After each segmentation, assign a level number to each new region obtained by segmentation. After the segmentation is completed, merge all the level numbers corresponding to the region according to the order of segmentation to obtain the region number.

[0009] Assign an internal number to each element within the region, and add the internal number of the element to the region number of the element's region to obtain the global number of the element.

[0010] In conjunction with the first aspect, the image further includes a complete screenshot of the display content, a screenshot of a certain area of ​​the display, and a screenshot of a certain application window of the display.

[0011] In conjunction with the first aspect, the following steps are also included:

[0012] Get the global ID of the element to be located;

[0013] Obtain the location image, identify all elements in the image, and generate a global number for each element;

[0014] The global ID of the element to be located is compared with the global IDs of all elements in the image. Elements with the same ID are the elements to be located. The goal of locating the element from the image is achieved by matching the position of the corresponding element in the image.

[0015] In conjunction with the first aspect, further, all elements in the image include icons, text blocks, buttons, and edit boxes;

[0016] The icons are identified and located using a deep learning-based object recognition model;

[0017] The text blocks are identified and located using optical character recognition;

[0018] The buttons and edit boxes are identified and located by combining the recognition results of icons and text blocks with edge detection.

[0019] In conjunction with the first aspect, furthermore, when elements in an image overlap, the overlapping elements are selected according to preset rules to ensure that each pixel belongs to only one element.

[0020] In conjunction with the first aspect, further methods for segmenting images include:

[0021] The image is identified by an edge detection algorithm. The horizontal cutting line is at least 90% of the width of the area and no less than 100 pixels, and the vertical cutting line is at least 90% of the width of the area and no less than 80 pixels.

[0022] When dividing the region, first divide it horizontally to obtain multiple sub-regions, assign a hierarchical number to each sub-region, then divide each sub-region vertically and number it, and then further divide each new region horizontally and vertically until the preset division end condition is met.

[0023] If no horizontal or vertical dividing line is found during a certain segmentation, it is still considered a segmentation, and all content is treated as the same region;

[0024] The number of each final region is formed by concatenating the numbers of all the previous levels of this region. The nth digit in the region number always corresponds to the nth level sub-region.

[0025] In conjunction with the first aspect, the further condition for ending the image segmentation is: the area of ​​each segmented region meets the preset requirements or the segmentation level reaches the preset number of levels.

[0026] Secondly, the present invention also provides a UI element numbering system, comprising:

[0027] Data acquisition module: Used to acquire images containing the UI interface;

[0028] Element recognition module: used to identify all elements in an image;

[0029] Region numbering module: It is used to identify the horizontal and vertical dividing lines in the image, and to perform multi-level segmentation of the image based on the horizontal and vertical dividing lines. After each segmentation, each new region is assigned a level number. After the segmentation is completed, all level numbers corresponding to the region are merged in the order of segmentation to obtain the region number.

[0030] Global Number Generation Module: Used to assign an internal number to each element within a region. The global number of an element is obtained by adding the region number of the region containing the element to the element's internal number.

[0031] Thirdly, the present invention also provides a UI element numbering device, including a processor and a storage medium;

[0032] The storage medium is used to store instructions;

[0033] The processor is configured to operate according to the instructions to perform the steps of the method according to any one of the first aspects.

[0034] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects.

[0035] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0036] This invention provides a method, system, device, and storage medium for numbering UI elements. It involves multi-level segmentation of an image to obtain multi-level regions, assigning a region number to each region, and then assigning an internal region number to all elements within each region. The element's region number plus its internal region number equals its global number. This global number is unique, easy to remember, and improves usability. Furthermore, it is relative to the position of obvious lines in the image, making it less susceptible to interference from screen scaling and other elements, thus exhibiting good stability and improving accuracy. When locating an element, the global number of the element to be located is compared one by one with the global numbers of all elements in the image. A successful match correctly locates the corresponding element, making it very convenient to use and possessing high value. Attached Figure Description

[0037] Figure 1 This is a flowchart of a UI element numbering method provided in an embodiment of the present invention;

[0038] Figure 2 These are images of the original application provided in the embodiments of the present invention;

[0039] Figure 3 This is the text recognition result using an OCR algorithm based on a CRNN model provided in this embodiment of the invention;

[0040] Figure 4 This is the recognition result of images and icons using a UI element detection model based on the YOLOv5 model provided in the embodiments of the present invention;

[0041] Figure 5 This is a schematic diagram of line detection provided in an embodiment of the present invention;

[0042] Figure 6 This is a diagram illustrating region segmentation provided in an embodiment of the present invention;

[0043] Figure 7 This is a diagram illustrating the global numbering provided in an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to illustrate the technical solution of the present invention more clearly, and should not be used to limit the scope of protection of the present invention.

[0045] Example 1

[0046] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for numbering UI elements, which includes the following steps:

[0047] S1. Get the image containing the UI interface.

[0048] Images can include full screenshots of the monitor's content, screenshots of a specific area of ​​the monitor, screenshots of a specific application window on the monitor, and can be images of PC screens, mobile phone or tablet screens, or other images showing display settings. For example... Figure 2 As shown, it is a screenshot of an application window.

[0049] Images are typically saved in JPEG or PNG format. In this embodiment, JPEG format is used.

[0050] S2. Identify all elements in the image.

[0051] Elements include icons, text blocks, buttons, and edit boxes; icons can be identified and located using deep learning-based object recognition models, edit boxes can be identified and located using edge detection (such as border line detection methods), and text boxes can be identified and located using optical character recognition (OCR).

[0052] If different types of elements overlap, some post-processing steps can be added. For example, if two edit boxes overlap, the box with the largest area can be selected to ensure that each pixel belongs to only one element.

[0053] In this embodiment, an OCR algorithm based on a CRNN model is first used to locate and recognize text blocks in the original image. The recognition result is as follows: Figure 3 As shown; then, an element detection model based on the YOLOv5 model is used to identify icons and text blocks, and the recognition results are as follows. Figure 4 As shown; finally, a border detection algorithm is used to identify elements with obvious borders, such as edit boxes and buttons; after obtaining all elements, overlapping elements are merged. In this embodiment, the element with the highest priority is used, and all elements that overlap with it are ignored. The priority order is text blocks, icons, buttons, and edit boxes.

[0054] S3. Identify the horizontal and vertical dividing lines in the image, and perform multi-level segmentation of the image based on the horizontal and vertical dividing lines. After each segmentation, assign a level number to each new region obtained. After the segmentation is completed, merge all the level numbers corresponding to the regions according to the order of segmentation to obtain the region number.

[0055] A dividing line is a line that can horizontally or vertically divide a region. The length of the line should be equal to or slightly less than the width or height of the region. In this embodiment, the dividing lines include horizontal dividing lines and vertical dividing lines. The length of the horizontal dividing line is more than 90% of the region width and not less than 100 pixels, and the length of the vertical dividing line is more than 90% of the region width and not less than 80 pixels.

[0056] There are several methods for finding horizontal and vertical dividing lines. For images with obvious lines, edge detection algorithms can be used to identify them. If their length is greater than 90% of the region's width and the color difference between adjacent points is less than 10, they are considered dividing lines. For images without obvious lines or with gradient colors, the flood algorithm can be used to calculate connected regions. If the connected region covers 90% of the width or 90% of the height of the region, its corresponding horizontal or vertical boundary line can also be considered a dividing line. In this embodiment, multiple methods were used to obtain various lines, with similar results. Figure 5 From these lines, obtain the dividing lines, such as Figure 6 As shown.

[0057] During segmentation, the image is divided into multiple levels, alternating between horizontal and vertical slices. First, a horizontal dividing line is found in the image, which divides it into upper and lower sections from left to right. Figure 6 In the process, two horizontal dividing lines are obtained, dividing the entire area into three sub-regions, each numbered sequentially from top to bottom. If no horizontal line is found, no area division is performed, and the original area remains unchanged and is also assigned a number. In this embodiment, the areas are numbered 0-9. If there is only one area, it is numbered 0; if there are two areas, they are numbered 0 and 1 from top to bottom. Next, each primary region is vertically divided. Within each primary region, a vertical dividing line is found, dividing the region into multiple blocks from top to bottom to obtain secondary regions. After the division is completed, each sub-region is numbered sequentially from left to right. In this embodiment, if there is only one sub-region, it is numbered 0; if there are two sub-regions, they are numbered 0 and 1 from left to right.

[0058] Each secondary region is then horizontally subdivided to obtain tertiary regions, which are then numbered. In this embodiment, the subdivision is fixed at three levels, such as... Figure 6 As shown, three primary regions are obtained during the first-level segmentation, numbered 0, 1, and 2 respectively; among them, primary region 3 includes three secondary sub-regions, and its third sub-region can be horizontally segmented into two tertiary sub-regions.

[0059] The image is segmented multiple times until the area of ​​each segmented region meets a preset requirement or the number of segmentation levels is reached. Each segmentation is treated as a level and assigned a level number, thus giving each region a unique number. These numbers are typically represented by digits 0-9, but can also be a combination of numbers and letters. Each segment can be divided into 36 parts, sufficient for practical applications. Since the segmentation method is fixed—always horizontal, vertical, horizontal, vertical—each number in the region numbering represents the region number of the corresponding level of segmentation. For example... Figure 6As shown, the area number of the address bar is 000, which represents area 0 in the first level of segmentation, area 0 in the second level of segmentation, and area 0 in the third level of segmentation; the area number of the table is 221, which represents area 2 in the first level of segmentation, area 2 in the second level of segmentation, and area 1 in the third level of segmentation.

[0060] S4. Assign an internal region number to each element within the region. Add the internal region number of the element to the region number of the element to obtain the global number of the element.

[0061] For each element in the region obtained after the segmentation in step S3, assign an internal region number. The principle is to number them sequentially from left to right and top to bottom according to the element type. Since the region area is small enough and the elements do not overlap, the internal region numbers will also not overlap. Add the region number of the element's region to the element's internal region number to obtain the global number (e.g., ...). Figure 7 (As shown).

[0062] Each element's global ID includes its absolute position defined by the dividing line and its relative position within the region. Therefore, when the window containing the image moves, expands, or the screen is resized, its global ID remains unchanged, exhibiting good stability, reducing the difficulty of element positioning, greatly reducing the error frequency of element positioning, and improving accuracy.

[0063] Once the global ID of each element in the image is obtained, it can be used for element location in the RPA process.

[0064] When designing an RPA process, a screenshot of the application is first taken, and a global identifier (GID) is generated for each element in the screenshot. When a user selects an element, the RPA process records the image and the GID for that element. When the RPA robot executes this process, it first obtains an application screenshot in the same way, identifies all elements within it, and numbers them. Based on the element GIDs saved in the RPA process, it finds the element with the same GID in the current screenshot, obtains the element's position, and can then perform the corresponding operation.

[0065] Since the global number is derived from the region number plus the relative number within that region, and the region number does not change with window size or screen scaling, the region number is stable. The elements within a region generally do not change either, so the internal numbering within that region is also relatively stable. This results in a relatively stable global number, ensuring the accurate operation of the RPA robot.

[0066] Example 2

[0067] This invention provides a UI element numbering system, comprising:

[0068] Data acquisition module: Used to acquire images containing the UI interface;

[0069] Element recognition module: used to identify all elements in an image;

[0070] Region numbering module: It is used to identify the horizontal and vertical dividing lines in the image, and to perform multi-level segmentation of the image based on the horizontal and vertical dividing lines. After each segmentation, each new region is assigned a level number. After the segmentation is completed, all level numbers corresponding to the region are merged in the order of segmentation to obtain the region number.

[0071] Global Number Generation Module: Used to assign an internal number to each element within a region. The global number of an element is obtained by adding the region number of the region containing the element to the element's internal number.

[0072] Example 3

[0073] This invention provides a UI element numbering device, which includes a processor and a storage medium;

[0074] The storage medium is used to store instructions;

[0075] The processor is configured to operate according to the instructions to perform the steps of the following method:

[0076] Get the image containing the UI interface;

[0077] Identify all elements in an image;

[0078] Identify the horizontal and vertical dividing lines in the image, and perform multi-level segmentation of the image based on the horizontal and vertical dividing lines. After each segmentation, assign a level number to each new region obtained by segmentation. After the segmentation is completed, merge all the level numbers corresponding to the region according to the order of segmentation to obtain the region number.

[0079] Assign an internal number to each element within the region, and add the internal number of the element to the region number of the element's region to obtain the global number of the element.

[0080] Example 4

[0081] This invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the following method:

[0082] Get the image containing the UI interface;

[0083] Identify all elements in an image;

[0084] Identify the horizontal and vertical dividing lines in the image, and perform multi-level segmentation of the image based on the horizontal and vertical dividing lines. After each segmentation, assign a level number to each new region obtained by segmentation. After the segmentation is completed, merge all the level numbers corresponding to the region according to the order of segmentation to obtain the region number.

[0085] Assign an internal number to each element within the region, and add the internal number of the element to the region number of the element's region to obtain the global number of the element.

[0086] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for numbering UI elements, characterized in that, include: Get the image containing the UI interface; Identify all elements in an image; Identify the horizontal and vertical dividing lines in the image, and perform multi-level segmentation of the image based on the horizontal and vertical dividing lines. After each segmentation, assign a level number to each new region obtained by segmentation. After the segmentation is completed, merge all the level numbers corresponding to the region according to the order of segmentation to obtain the region number. Assign an internal number to each element within the region, and add the internal number of the element to the region number of the element's region to obtain the global number of the element. Methods for segmenting images include: The image is divided into horizontal and vertical cutting lines by edge detection algorithm. The length of the horizontal cutting line is at least 90% of the width of the area and not less than 100 pixels, and the length of the vertical cutting line is at least 90% of the width of the area and not less than 80 pixels. When dividing the region, first divide it horizontally to obtain multiple sub-regions, assign a hierarchical number to each sub-region, then divide each sub-region vertically and number it, and then further divide each new region horizontally and vertically until the preset division end condition is met. If no horizontal or vertical dividing line is found during a certain segmentation, it is still considered a segmentation, and all content is treated as the same region; The number of each final region is formed by concatenating the numbers of all the previous levels of this region. The nth digit in the region number always corresponds to the nth level sub-region. The termination condition for segmenting an image is: the area of ​​each segmented region meets the preset requirements or the number of segmented layers reaches the preset number of layers.

2. The UI element numbering method according to claim 1, characterized in that, The images include full screenshots of the display content, screenshots of a specific area of ​​the display, and screenshots of a specific application window on the display.

3. The UI element numbering method according to claim 1, characterized in that, It also includes the following steps: Get the global ID of the element to be located; Obtain the location image, identify all elements in the image, and generate a global number for each element; The global ID of the element to be located is compared with the global IDs of all elements in the image. Elements with the same ID are the elements to be located. The goal of locating the element from the image is achieved by matching the position of the corresponding element in the image.

4. The UI element numbering method according to claim 1, characterized in that, All elements in the image include icons, text blocks, buttons, and edit boxes; The icons are identified and located using a deep learning-based object recognition model; The text blocks are identified and located using optical character recognition; The buttons and edit boxes are identified and located by combining the recognition results of icons and text blocks with edge detection.

5. A method for numbering UI elements according to claim 1, characterized in that, When elements in an image overlap, the overlapping elements are selected according to preset rules to ensure that each pixel belongs to only one element.

6. A UI element numbering system based on the method of any one of claims 1 to 5, characterized in that, include: Data acquisition module: Used to acquire images containing the UI interface; Element recognition module: used to identify all elements in an image; Region numbering module: It is used to identify the horizontal and vertical dividing lines in the image, and to perform multi-level segmentation of the image based on the horizontal and vertical dividing lines. After each segmentation, each new region is assigned a level number. After the segmentation is completed, all level numbers corresponding to the region are merged in the order of segmentation to obtain the region number. Global Number Generation Module: Used to assign an internal number to each element within a region. The global number of an element is obtained by adding the region number of the region containing the element to the element's internal number.

7. A UI element numbering device, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 5.

Images