A method, medium, and electronic device for positioning an element on an interface

Abstract

Embodiments of the present application provide a method, medium and electronic device for positioning the position of an element on an interface, the method comprising: inputting an image of a to-be-operated interface into a target detection model, and obtaining all elements on the to-be-operated interface and an element aggregation result through the target detection model, wherein the element aggregation result is represented by an aggregation box; constructing an element structure tree corresponding to the to-be-operated interface according to at least attribute information of the aggregation box, so that a robot completes positioning of the elements on the to-be-operated interface according to the element structure tree; wherein the element structure tree comprises leaf nodes at a bottom layer and parent nodes at each level, the leaf nodes at the bottom layer are the all elements, and one parent node in the element structure tree corresponds to one aggregation box. Embodiments of the present application detect all contents required for building the structure tree in an end-to-end manner, thereby improving the effect of positioning the elements on the interface according to the element structure tree.

Description

Technical Field

[0001] This application relates to the field of Robotic Process Automation (RAP), and more specifically, embodiments of this application relate to a method, medium, and electronic device for locating the position of elements on an interface. Background Technology

[0002] In the implementation of Robotic Process Automation (RPA), for a typical interface (such as a webpage or app interface), before a software robot clicks a button (as an example of an element on the interface), it needs to accurately identify the position and semantics of the button. The accuracy of this technology depends on the combined accuracy of multiple modules, such as the target element detection module, the template matching module, and the OCR (Optical Character Recognition) module.

[0003] Because multiple modules of the related technology (e.g., object detection module, template matching module, and OCR module) work in series, each module depends on the accuracy of the upstream module, and the error rate will accumulate. This leads to a low success rate for the software robot in locating the position of elements on the interface to be operated. At the same time, the large number of modules in series also leads to a decrease in the implementation speed of the software robot. In addition, the software robot relies too much on the semantic information provided by OCR to find certain elements in the interface, so it has poor robustness to language versions or color and shape changes. Summary of the Invention

[0004] The purpose of this application is to provide a method, medium, and electronic device for locating the position of elements on an interface. Some embodiments of this application will use a "target detection" method to detect all the contents required to build the structure tree from end to end, and use simple post-processing to realize the construction of the element structure tree of the interface, thereby improving the effect of locating elements on the interface according to the element structure tree.

[0005] In a first aspect, embodiments of this application provide a method for locating the position of elements on an interface. The method includes: inputting an image of the interface to be operated into a target detection model, and obtaining all elements on the interface to be operated and element aggregation results through the target detection model, wherein the element aggregation results are represented by aggregation boxes; constructing an element structure tree corresponding to the interface to be operated based at least on the attribute information of the aggregation boxes, so that a robot can complete the location of elements on the interface to be operated based on the element structure tree; wherein the element structure tree includes leaf nodes at the bottom layer and parent nodes at each level, the leaf nodes at the bottom layer are all elements, and one parent node in the element structure tree corresponds to one aggregation box.

[0006] Some embodiments of this application employ a target detection model that can find hierarchical relationships between elements in an end-to-end manner. The steps are simplified, which improves the speed of obtaining the element structure tree and the speed and accuracy of locating elements on the interface.

[0007] In some embodiments, identifying all elements on the interface to be operated and the element aggregation result through the target detection model includes: obtaining all boxes on the image, the type of all boxes, and the attribute information of all boxes through the target detection model, wherein the type of all boxes includes element detection boxes and aggregation boxes, an element detection box includes an element identified from the interface, and an aggregation box includes multiple elements aggregated according to element position and logical relationship, and the attribute information includes at least one of the size and position of the corresponding box; elements located in all the element detection boxes are regarded as elements identified from the interface to be operated, and multiple elements located in the aggregation box are regarded as a group of aggregateable elements.

[0008] Some embodiments of this application can use the trained target detection model to mine all elements on the interface image to be operated and the set of aggregateable elements, and then build an element structure tree. For example, the identified elements are used as the bottom leaf nodes of the element structure tree, and the aggregateable elements are used as a parent node on the element structure tree.

[0009] In some embodiments, constructing an element structure tree corresponding to the interface to be operated based at least on the attribute information of the aggregation box includes: obtaining the parent nodes at each level based on the attribute information of the aggregation box; and determining the parent node to which the bottom leaf node belongs based on the position information of the element detection box.

[0010] Some embodiments of this application determine the parent nodes at each level of the element structure tree by using the properties of the aggregation box, and determine the parent node to which each leaf node at the bottom layer belongs by using the position of the element detection box, thereby improving the accuracy of the obtained element structure tree.

[0011] In some embodiments, the attribute information includes the position information and size information of each aggregation box.

[0012] Some embodiments of this application determine whether the two parent nodes corresponding to the two aggregation boxes have a subordinate relationship in the element structure tree by using the position and size of the aggregation boxes, thereby improving the accuracy of the obtained element structure tree.

[0013] In some embodiments, the plurality of aggregation boxes includes a first aggregation box and a second aggregation box, wherein obtaining the parent nodes at each level based on the attribute information of the aggregation boxes includes: if it is determined based on the attribute information of the first aggregation box and the attribute information of the second aggregation box that the first aggregation box contains the second aggregation box on the image, then confirm that the level of the parent node corresponding to the first aggregation box in the element structure tree is higher than the level of the parent node corresponding to the second aggregation box.

[0014] Some embodiments of this application determine the relative hierarchy of the parent node corresponding to two aggregation boxes by the position and size of the aggregation boxes, thereby improving the accuracy of the obtained element result tree.

[0015] In some embodiments, obtaining the parent nodes at each level based on the attribute information of the aggregation box includes: obtaining multiple aggregation boxes with inclusion relationships; sorting the multiple aggregation boxes with inclusion relationships according to their area to obtain a queue; in the queue, the two parent nodes corresponding to two adjacent aggregation boxes are at adjacent levels in the element structure tree.

[0016] Some embodiments of this application obtain parent nodes at each level by sorting the aggregation boxes according to their size relationship, thereby improving the accuracy and efficiency of the obtained element structure tree.

[0017] Secondly, some embodiments of this application provide a method for training a detection model, the method comprising: marking an element detection box in the area covered by any element on a training interface image, and marking at least one aggregation box on the image marked with the element detection box to obtain training data, wherein multiple elements are aggregateable within an aggregation box, the multiple elements being determined by the positional distribution and function of the elements on the corresponding training interface; and training the detection model at least based on the training data to obtain a target detection model.

[0018] Some embodiments of this application train the detection model by differentially annotating the location of elements and the location of multiple aggregated elements on the training image. This enables the target detection model after training to recognize two types of boxes and to identify which type of box each box belongs to. In turn, the element structure tree can be obtained from the data output by the target detection model, thus realizing end-to-end element detection.

[0019] In some embodiments, the at least one aggregation box includes a first aggregation box, a second aggregation box, ..., an (N-1)th aggregation box and an Nth aggregation box in ascending order of size. The step of annotating at least one aggregation box on the image containing the element detection boxes includes: annotating the first aggregation box in the region containing m aggregateable elements, annotating the second aggregation box in the region containing n aggregateable elements, where the n elements include the m elements, and so on, until the Nth aggregation box is annotated in the region containing all elements on the training interface image. The Nth aggregation box corresponds to the root node of the tree, and the Nth aggregation box includes the (N-1)th aggregation box. The value of N is an integer greater than 1.

[0020] Some embodiments of this application use multiple aggregated boxes of different sizes labeled on an image as training data, which enables the trained model to recognize various aggregated boxes on the input image.

[0021] In some embodiments, training the detection model to obtain the target detection model based at least on the training data includes: determining whether the training of the detection model is feasible based on a loss function, wherein the loss function is related to the area of ​​the element detection box and the aggregation box.

[0022] Some embodiments of this application can effectively avoid the problem of the model focusing on learning elements while ignoring the aggregation box by introducing the area of ​​the box into the aggregation function.

[0023] In some embodiments, the formula for calculating the loss function is:

[0024]

[0025] Where n is the total number of the element detection boxes and the aggregation boxes, i is the sequence number of any box in the element detection boxes and the aggregation boxes, Ai represents the area of ​​the i-th box, and li is the initial loss function value of the i-th box.

[0026] Some embodiments of this application improve the technical problem of unsatisfactory training results caused by the small number of aggregation boxes relative to the number of element detection boxes by adding area weights to the loss function.

[0027] Secondly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method for locating the position of an element on an interface as described in any of the embodiments of the first aspect above.

[0028] Thirdly, some embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it can implement the method for locating the position of elements on an interface as described in any of the embodiments of the first aspect above.

[0029] Fourthly, some embodiments of this application provide an apparatus for locating the position of elements on an interface. The apparatus includes: an image acquisition module configured to acquire an image of the interface to be operated; an element recognition and element aggregation result recognition module configured to input the image into a target detection model and obtain all elements on the interface to be operated and element aggregation results through the target detection model, wherein the element aggregation results are represented by aggregation boxes; and an element structure tree construction module configured to construct an element structure tree corresponding to the interface to be operated based at least on the attribute information of the aggregation boxes, so that a robot can locate the elements on the interface to be operated based on the element structure tree; wherein the element structure tree includes leaf nodes at the bottom layer and parent nodes at each level, the leaf nodes at the bottom layer are all the elements, and one parent node in the element structure tree corresponds to one aggregation box. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A schematic diagram illustrating the composition of a system for locating the position of elements on an interface, provided in an embodiment of this application.

[0032] Figure 2 One of the flowcharts for a method of locating the position of an element on an interface provided in an embodiment of this application;

[0033] Figure 3 An end-to-end architecture diagram provided for embodiments of this application;

[0034] Figure 4 This is a schematic diagram illustrating the aggregation of multiple elements provided in an embodiment of this application;

[0035] Figure 5 This is a flowchart illustrating the process of training a detection model to obtain a model weight file, provided in an embodiment of this application.

[0036] Figure 6 This is one of the flowcharts for a method of constructing an element structure tree provided in an embodiment of this application;

[0037] Figure 7 A second flowchart illustrating the method for constructing an element structure tree provided in this application embodiment;

[0038] Figure 8 This is a schematic diagram showing all the boxes marked on any interface image provided in the embodiments of this application;

[0039] Figure 9 The hierarchy number of the parent node corresponding to each aggregation box in the element structure tree provided in the embodiments of this application;

[0040] Figure 10 According to the embodiments provided in this application Figure 8 The element structure tree constructed from the hierarchical relationships;

[0041] Figure 11 A block diagram of a device for locating the position of elements on an interface, provided in an embodiment of this application;

[0042] Figure 12 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0043] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0044] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0045] Robotic process automation (RPA) technology can simulate how employees operate computers using keyboards and mice in their daily work. It can replace humans in performing tasks such as logging into systems, operating software, reading and writing data, downloading files, and reading emails. By using automated robots as virtual labor for enterprises, employees can be freed from repetitive, low-value tasks, allowing them to focus their energy on high-value-added work. This enables enterprises to reduce costs and increase efficiency while undergoing digital and intelligent transformation.

[0046] RPA (Robotic Process Automation) is a software robot that replaces manual tasks in business processes and interacts with computer front-end systems like a human. Therefore, RPA can be seen as a software-based program robot running on a personal PC or server, mimicking user actions on a computer to automate repetitive tasks such as retrieving emails, downloading attachments, logging into systems, and data processing and analysis—faster, more accurate, and more reliable. While both RPA and traditional physical robots address the speed and accuracy issues in human work through specific rules, traditional physical robots are hardware-software hybrids requiring specific hardware support and software to perform tasks. RPA robots, on the other hand, are purely software-based; once the appropriate software is installed, they can be deployed to any PC or server to complete the assigned tasks.

[0047] In other words, RPA is a method and related technologies that utilize "digital employees" to perform business operations in place of humans. Essentially, RPA uses software automation technology to simulate human operation of computer systems, software, web pages, and documents, acquiring business information, executing business actions, and ultimately achieving automated process processing, labor cost savings, and improved processing efficiency. As described, one of the core technologies of RPA is locating and picking the elements to be operated on on the interface. For example, when simulating a human clicking a button, the prerequisite is to locate the button element. Some embodiments of this application, through the rapid construction of a precise element structure tree, can significantly improve the speed and accuracy of locating the position of elements on the interface based on this element structure tree.

[0048] To achieve element localization on the interface, some embodiments of this application take a structuring approach, performing structured parsing on all elements detected by the object detection model to construct an element structure tree. This allows the software robot to avoid relying on cumbersome OCR results or image semantic information to determine the selection of buttons (as examples of elements on the interface), but instead to map the structural relationships between elements onto the actual image and find the corresponding button (as an example of an element on the interface) location. It should be noted that while some methods for element structuring utilize graph convolutional networks, this method relies heavily on graph structure generation, thus placing high demands on the model.

[0049] Unlike the methods described above for constructing an element structure tree, some embodiments of this application employ an "object detection model" to detect all the content required to build the structure tree (including parent nodes at all levels and bottom leaf nodes) end-to-end, and utilize simple post-processing (i.e., the method executed in the post-processing stage below) to build the element structure tree of the interface.

[0050] It is understandable that some embodiments of this application do not employ the method of "first detecting elements, then finding the structural relationships between elements through their features and other information, and thus constructing an element structure tree," but instead use an object detection model to directly mine the elements on the interface and aggregation boxes that can represent the hierarchical relationships of elements, thereby constructing an element structure tree. Therefore, compared with related element structure tree construction methods, some embodiments of this application have at least the following technical advantages: 1. Reducing the cumulative effect of error rates caused by multi-model concatenation; 2. Improving the construction speed of the element structure tree. In addition, compared with related technologies based on the interface element localization mode combining interface element detection and OCR technology, some embodiments of this application have the following advantages: 1. Reducing the cumulative effect of error rates caused by multi-model concatenation; 2. Improving the overall working speed. Furthermore, since the present invention does not rely on the semantic information given by OCR to find certain elements in the interface, but rather on the structural relationships of the elements, it can: 3. Have high robustness to language versions or color and shape changes, while reducing the training cost of the model.

[0051] In other words, some embodiments of this application directly predict elements and aggregation boxes that can represent the hierarchical relationship of elements using object detection models, and construct an element structure tree through simple post-processing. In RPA implementation, some embodiments of this application can use the structural position of elements instead of semantic plus feature matching to locate elements on the interface, thus improving the capture speed of elements on the interface.

[0052] Please refer to Figure 1 , Figure 1 The system for locating the position of elements on an interface provided in this application includes an operation terminal 200 and a server 100. The robot can simulate human operation on the target elements of the interface through the operation terminal 200, and the server 100 can assist the robot in locating the position of the target elements on the interface to be operated.

[0053] For example, in Figure 1 In the process, the operation terminal 200 provides the server 100 with an image 110 of the interface to be operated. Then, the server 100 locates the position of the element based on the image of the interface to be operated and feeds back the element location result 121 on the interface to be operated to the operation terminal 200. Then, the robot completes the corresponding operation on the operation terminal 200 based on the element location result.

[0054] For example, in Figure 1 The terminal displays the Baidu search interface, that is, on the operating terminal. Figure 1Image 110, which uses the Baidu search interface as the interface to be operated, includes multiple elements, namely, first element 101, second element 102, third element 103, fourth element 104, fifth element 105, sixth element 106, seventh element 107, eighth element 108, ninth element 109, and tenth element. Among them, the first to seventh elements are all hyperlinks, the eighth element is an editable input box, the ninth element is a button, and the tenth element 190 is a bounded image.

[0055] It is easy to understand that the element location result returned by server 100 to operation terminal 200 can be location information used to determine the position of any element on the Baidu search image.

[0056] It should be noted that during the design phase of the RPA process, a certain interface (for example, an interface of an application app or a webpage) is selected as the baseline interface (or standard interface). During the subsequent execution of RPA (i.e., when the robot imitates the human user interface), the robot will repeatedly access this interface. In the embodiments of this application, the interface that the robot accesses and operates on is called the interface to be operated, and the interface that the human actually operates on (i.e., the interface for image recording during the design phase of the RPA process) is called the baseline interface or standard interface.

[0057] The following example illustrates how server 100 obtains the location results.

[0058] For example, A is a standard interface containing elements, and B is a standard element structure tree constructed based on A (this standard element structure tree is pre-stored in server 100, and the construction method of this element structure tree can also adopt the specific implementation process for constructing the element structure tree corresponding to the interface to be operated provided in the following embodiments of this application). Each corresponding node in B is manually labeled according to the actual semantics of each element in A, thereby obtaining a standard element semantic tree B+; C is a certain interface in the implementation process (i.e., an interface to be operated), and D is an element structure tree constructed based on C; if A and C are consistent in element structure distribution, then the tree structures of D and B+ are also consistent; if you want to find a certain "button" (as an example of an element on the interface) in C, you only need to find a node with the same semantics in B+, then find the position of that node in D, thereby obtaining the position information of that node in C, and finally you can click the "button".

[0059] It should be noted that the above process of locating elements on the interface by the server based on the constructed element structure tree of the interface to be operated is only an implementation example. Those skilled in the art can use a different method than the previous paragraph to locate elements on the interface to be operated based on the element structure tree constructed by some embodiments of this application.

[0060] The following is combined with Figure 2 This paper exemplifies a method for locating the position of elements on an interface, executed by server 100. It should be noted that in some embodiments of this application, the program for locating the position of elements on the interface can also be integrated into the operating terminal 200, in which case it can be done without relying on... Figure 1 Instead of a server, the method for locating elements on the interface is implemented by the operating terminal.

[0061] like Figure 2 As shown in the figure, this application provides a method for locating the position of an element on an interface, the method comprising:

[0062] S101, Obtain the image of the interface to be operated.

[0063] As described above, the image of the interface to be operated in S101 is the interface accessed by the robot during the execution of RPA (i.e., when the robot imitates the human operating interface).

[0064] For example, S101 may include reading an interface image to be operated from memory or receiving an interface image to be operated from another device.

[0065] S102, the image is input into the target detection model, and all elements on the interface to be operated and the element aggregation result are obtained through the target detection model, wherein the element aggregation result is represented by an aggregation box.

[0066] For example, in some embodiments of this application, S102 includes: obtaining all bounding boxes on the image, the types of all bounding boxes, and attribute information of all bounding boxes through the target detection model, wherein the types of all bounding boxes include element detection boxes and aggregation boxes, an element detection box includes an element identified from the interface, and an aggregation box includes multiple elements aggregated according to element position and logical relationship, and the attribute information includes at least one of the size and position of the corresponding box; elements located in all the element detection boxes are taken as elements identified from the interface to be operated, and multiple elements located in the aggregation box are taken as a group of aggregateable elements.

[0067] In other words, some embodiments of this application can use the trained target detection model to mine all elements on the interface image to be operated and the set of aggregateable elements, and then build an element structure tree. For example, the identified elements are used as the bottom leaf nodes of the element structure tree, and the aggregateable elements are used as a parent node on the element structure tree.

[0068] Understandably, in order for the S102 object detection model to be able to identify all boxes, all box categories, and all box attributes (e.g., position and size information of all boxes) in an image, the detection model needs to be trained in advance and the object detection model needs to be obtained based on the weight file after training.

[0069] The following example illustrates the process of training a detection model to obtain a target detection model.

[0070] like Figure 3 As shown, in some embodiments of this application, the obtained training data needs to be input into the detection model 111 to train the model, obtain prediction results for each iteration, adjust the parameters of the detection model 111 based on the prediction results, and obtain the target detection model 120 when it is confirmed that the training of the detection model can end (e.g., by confirming that the training of the model can end through the loss function). Figure 3 As shown, the object detection model 120 has the capability to perform information mining on any interface image, such as the input image of the interface to be operated or a standard interface image, to obtain all bounding boxes on these images, the category of each bounding box (e.g., whether each bounding box belongs to an element detection box or an aggregate box), and the attributes of each bounding box (e.g., the position information and size information of each bounding box). Then, it is further processed... Figure 3 The element structure tree construction model 130 (i.e., the post-processing module) determines the parent nodes of each layer and the bottom leaf nodes of the element structure tree based on the attributes of each box, thus completing the structure tree construction and obtaining the corresponding element structure tree.

[0071] It is understood that, in some embodiments of this application, the method further includes, before inputting the image into the target detection model:

[0072] The first step involves labeling the area covered by any element on the training interface image with an element detection box, and then labeling at least one aggregation box (i.e., labeling the total area covered by any number of aggregateable elements on the training interface image with an aggregation box) on the image with the element detection boxes. This yields the training data, where elements located within any aggregation box are aggregated elements. These multiple elements are determined by their positional distribution and function on the corresponding training interface. For example, these multiple elements are adjacent elements on the training interface with the same or similar functions, such as... Figure 4As shown, the two elements within the first frame 411 on interface 400 are aggregateable elements. These two elements are adjacent and have similar functions, both used for login. The three elements within the second frame 412 on interface 400 are also aggregateable elements. These three elements are adjacent and all belong to third-party login methods other than the currently displayed login method. It is understandable that for... Figure 4 When annotating interface images, you can annotate aggregation boxes separately for multiple aggregateable elements, for example, Figure 4 The first box 411 and the second box 412.

[0073] The second step is to train the detection model based on at least the training data to obtain the target detection model.

[0074] It should be noted that, since there are more element detection boxes than aggregate boxes in the labeled data, some embodiments of this application propose an improved loss function to effectively avoid the problem that the detection model focuses on learning elements and ignores aggregate boxes. This loss function is positively correlated with the size of each box obtained by the detection model.

[0075] In some embodiments of this application, the second step of training the detection model to obtain the target detection model based at least on the training data includes, for example, determining whether the training of the detection model is binding based on a loss function, wherein the loss function is related to the area of ​​the element detection boxes and the aggregation boxes. For example, the size of the loss function is positively correlated with the area of ​​each box (i.e., any box in all element detection boxes and aggregation boxes). Some embodiments of this application, by introducing the area of ​​the boxes into the aggregation function, can effectively avoid the problem of the model focusing on learning elements while ignoring aggregation boxes.

[0076] For example, in some embodiments of this application, the formula for calculating the loss function is:

[0077]

[0078] Where n is the total number of the element detection boxes and the aggregation boxes, i is the index of any box in the element detection boxes and the aggregation boxes, Ai represents the area of ​​the i-th box, l i This is the initial loss function value for the i-th box. Some embodiments of this application improve the technical problem of unsatisfactory training results caused by the small number of aggregated boxes relative to the number of element detection boxes by adding area weights to the loss function.

[0079] It should be noted that if the object detection model or the detection model is YOLOv5 or another model, then l in the above calculation formula... i The value of l can be calculated by referring to the loss function published by the corresponding model. To avoid repeating the calculation here, l is not specified.i The calculation process for the value will not be elaborated in detail.

[0080] It is understood that some embodiments of this application train the detection model by differentially annotating the location of elements and the location of multiple aggregated elements on the training image, so that the target detection model obtained after training has the ability to recognize two types of boxes and to identify which type of box each box belongs to. Then, the element structure tree can be obtained from the data output by the target detection model, thus realizing end-to-end element detection.

[0081] S103, at least based on the attribute information of the aggregation box, construct an element structure tree corresponding to the interface to be operated, so that the robot can locate the elements on the interface to be operated based on the element structure tree; wherein, the element structure tree includes leaf nodes at the bottom layer and parent nodes at each level, the leaf nodes at the bottom layer are all the elements, and one parent node in the element structure tree corresponds to one aggregation box.

[0082] To determine the parent nodes at each level and the bottom leaf nodes in the element structure tree, in some embodiments of this application, step S103 includes: obtaining the parent nodes at each level based on the attribute information of the aggregation box; and determining the parent node to which the bottom leaf node belongs based on the position information of the element detection box. In other words, some embodiments of this application determine the parent nodes at each level in the element structure tree through the attributes of the aggregation box and determine the parent node to which each bottom leaf node belongs through the position of the element detection box, thereby improving the accuracy of the obtained element structure tree.

[0083] It should be noted that, in some embodiments of this application, the attribute information of the aggregation boxes includes the position information and size information of each aggregation box. Some embodiments of this application use the position and size of the aggregation boxes to determine whether the two parent nodes corresponding to two aggregation boxes have a subordinate relationship in the element structure tree, thereby improving the accuracy of the obtained element structure tree.

[0084] In some embodiments of this application, the plurality of aggregation boxes includes a first aggregation box and a second aggregation box. S103 exemplarily includes: if it is determined, based on the attribute information of the first aggregation box and the attribute information of the second aggregation box, that the first aggregation box contains the second aggregation box on the image, then it is confirmed that the level of the parent node corresponding to the first aggregation box in the element structure tree is higher than the level of the parent node corresponding to the second aggregation box. Some embodiments of this application determine the relative level of the parent nodes corresponding to two aggregation boxes by using the position and size of the aggregation boxes, thereby improving the accuracy of the obtained element result tree.

[0085] In some embodiments of this application, the process of obtaining the parent nodes at each level based on the attribute information of the aggregate boxes described in S103 above includes, for example,: obtaining multiple aggregate boxes with inclusion relationships; sorting the multiple aggregate boxes with inclusion relationships according to their area to obtain a queue; in the queue, the two parent nodes corresponding to two adjacent aggregate boxes are at adjacent levels in the element structure tree. Some embodiments of this application improve the accuracy and efficiency of the obtained element structure tree by sorting the aggregate boxes according to their size relationship to obtain the parent nodes at each level.

[0086] It should be noted that in some embodiments of this application, the above-mentioned target detection model or detection model can be Yolov7 or YOLOx.

[0087] It is easy to understand that some embodiments of this application employ a target detection model to find the hierarchical relationship between elements in an end-to-end manner. The steps are simplified, which improves the speed of obtaining the element structure tree and the speed and accuracy of locating elements on the interface.

[0088] It should be noted that in some embodiments of this application, after S103, in order to locate the elements on the interface to be operated based on the constructed element structure tree, it is also necessary to pre-store multiple element result trees on the server 100. These stored element structure trees need to include the element structure tree corresponding to the interface to be operated. That is, as mentioned above, in the RPA process designer stage (i.e., the stage of manual interface operation), an interface identical to the interface to be operated, i.e., the baseline interface (or standard interface), is selected, and the element structure tree of that interface is obtained. For the specific process of locating the elements of the interface to be operated based on the pre-stored element structure tree, please refer to the above-described technical solution on how the server 100 obtains the location results. To avoid repetition, it will not be elaborated further here.

[0089] The following is combined with Figures 5-9 A specific example (which includes the model training stage, the object detection model inference stage, and the post-processing stage) illustrates the process of training the detection model and constructing an element structure tree based on the output of the trained object detection model.

[0090] Phase 1: Model Training Phase

[0091] like Figure 5 As shown, the training process for the detection model includes:

[0092] The collected data yields the acquired training images.

[0093] S201, The size of the acquired training images is transformed to obtain the data to be predicted.

[0094] S202, annotate the element detection boxes and aggregation boxes on the acquired training images, and then perform size transformation on the annotated images to obtain the supervision data during training.

[0095] For example, in some embodiments of this application, the above-mentioned at least one aggregation box includes a first aggregation box, a second aggregation box, ..., an (N-1)th aggregation box and an Nth aggregation box in ascending order of size. The above-mentioned annotation of at least one aggregation box on the image annotated with the element detection boxes exemplarily includes: annotating the first aggregation box in the region containing m aggregateable elements (e.g., the m elements are adjacent in position and have the same function); annotating the second aggregation box in the region containing n aggregateable elements (e.g., the n elements are adjacent in position and have similar function); wherein the n elements include the m elements; and so on, until the Nth aggregation box is annotated in the region containing all elements on the training interface image. The Nth aggregation box corresponds to the root node of the tree, and the Nth aggregation box includes the (N-1)th aggregation box. The value of N is an integer greater than 1.

[0096] In other words, the collected webpage or software interface images are labeled to form a corresponding label set. The labeling method can be found by referring to... Figure 8 In the figure, the dashed boxes are the labeled element detection boxes, and the solid boxes are the labeled aggregation boxes (i.e. aggregation boxes that can represent the hierarchical relationship of elements).

[0097] For example, Figure 8 In the context of elements A, B, C, D, E, F, G, H, I, J, K, L, and M (these elements are in...), Figure 8 In the array (abbreviated as the corresponding letters), each element's region is marked with an element detection box. An aggregation box is marked in the region shared by aggregateable elements A and B, C and D, E and F, G and H, I, J, and K, and L and M. Figure 8 The text also includes annotations for larger aggregation boxes that further aggregate the existing aggregation boxes; to avoid repetition, these will not be described in detail here. Figure 8 The aggregation boxes in the text.

[0098] Because the images acquired from different interfaces may vary in size, and excessively large images would hinder model training and inference, some embodiments of this application require image size adjustment. For example, all images are resized to the same size, such as by adjusting the image to 1920*1088 pixels, and the corresponding bounding boxes are scaled. In other words, the size of all acquired images is unified (i.e., the number of pixels in all images is modified to be the same), and the bounding boxes on the corresponding images are scaled accordingly.

[0099] S204. Input the data to be predicted and the supervised data into the detection model at the same time to train the model and obtain the model weight file. The weight file is the specific value of each parameter of the target detection model.

[0100] For example, the image processed by S201 and S203 is fed into the detection model YOLOv5 large (or YOLOv7, YOLOvX, etc.) for supervised training. The corresponding labeled set serves as the supervision data, resulting in the trained model weight file pkg. It should be noted that in some embodiments of this application, YOLOv5 large requires the following adjustments: First, some embodiments of this application add downsampling factor categories, thereby increasing the corresponding downsampling feature maps. This is because there are many small targets smaller than 8*8 pixels in the interface parsing, so new downsampling feature maps are added to the model head part. Originally, they were 1 / 8, 1 / 16, and 1 / 32; now they are 1 / 4, 1 / 8, 1 / 16, and 1 / 32. Secondly, because the order of magnitude of the larger and smaller bounding boxes in the annotation boxes of some embodiments of this application differs significantly—that is, the number of element detection boxes (usually smaller) is much larger than the number of aggregation boxes (usually larger)—some embodiments of this application add weights to the loss function by calculating the area of ​​the boxes to avoid the model focusing on learning elements while ignoring aggregation boxes. This allows the model to balance the learning of elements and aggregation boxes. The loss function is designed as follows:

[0101]

[0102] Where n is the total number of objects, i is the index, Ai represents the area of ​​the i-th object, and li is the loss of the i-th object.

[0103] The second stage is the object detection model inference stage.

[0104] like Figure 6 As shown, the inference result flow of the object detection model is as follows:

[0105] S301, adjust the size of the image of the interface to be operated to the preset size.

[0106] For example, Figure 5 The obtained model weight file pkg is used as the model parameters. The object detection model Yolov5 large is obtained during inference. The size of the interface image to be detected (e.g., the image of the interface to be operated or the standard interface image) is adjusted to an image with a width of 1920 (pixels) and a height of 1088 (pixels) before being fed into the model.

[0107] S302, the output result is obtained through the object detection model.

[0108] In other words, the object detection model in some embodiments of this application will discover all element detection boxes and aggregate boxes on the input image, and these boxes will contain location and category information.

[0109] It should be noted that in some embodiments of this application, the aggregated boxes are not hierarchical during the annotation and object detection stages. Subsequent post-processing algorithms will determine the hierarchy of the aggregated boxes based on their size and position. The aforementioned category information refers to the category of the object (i.e., all detected boxes), that is, whether each box belongs to an element detection box or an aggregated box. The purpose is that subsequent post-processing algorithms will determine the tree structure based on the object category. For example, elements may be treated as leaf nodes at the bottom of the element tree structure, and aggregated boxes may be treated as parent nodes.

[0110] S303, construct an element structure tree based on the aggregation box and the element detection box.

[0111] In other words, the element detection box and aggregation box are sent to the post-processing step to obtain the element structure tree.

[0112] The third stage, post-processing stage.

[0113] like Figure 7 As shown, the post-processing steps include, for example:

[0114] S401, based on the inclusion relationship of all aggregate boxes from largest to smallest, gradually build a tree structure, and take the entire interface as the root node (i.e. the top-level parent node).

[0115] By constructing a tree structure from largest to smallest based on the inclusion relationship of the aggregate boxes, and treating the entire interface as the root node, a tree structure without leaf nodes is obtained.

[0116] For example, based on the position and size relationship between any two aggregate boxes, determine whether they have an inclusion relationship. Specifically, if the first aggregate box is larger than the second aggregate box, and the second aggregate box is located inside the first aggregate box, then the first aggregate box includes the second aggregate box. Repeating this process determines the inclusion relationship of all aggregate boxes; an element result tree is then constructed based on all inclusion relationships.

[0117] S402: Based on the position of all elements on the interface, set the corresponding leaf nodes in the tree structure to obtain the element structure tree.

[0118] Based on the position information of the element detection box, set the corresponding leaf nodes in the structure tree obtained in S401 to obtain the final element structure tree.

[0119] The demonstration process of S401 and S402 is as follows: Figure 9 and Figure 10 As shown, where, Figure 9 This represents the detection results obtained by the object detection model from the input image. The lighter colors represent aggregated boxes, and the darker colors represent element detection boxes.

[0120] This diagram shows that the steps involved in generating the structure tree are:

[0121] Step 1: Generating a tree structure that does not contain leaf nodes: Figure 9 Structure box 0 contains boxes 1 and 4, corresponding to Figure 10 In the diagram, nodes 1 and 4 are child nodes of node 0; Figure 9 Box 1 contains boxes 2 and 3, corresponding to Figure 10 In the diagram, nodes 2 and 3 are child nodes of node 1; and so on. Figure 10 The overall interface contains boxes 0 and 9, so the interface is the root node, and nodes 0 and 9 are the child nodes of the root node.

[0122] The second step is to generate leaf nodes: if the first element A and the second element B within the element detection box belong to aggregation box 2, then... Figure 10 Nodes A and B are leaf nodes and are children of node 2.

[0123] Please refer to Figure 11 , Figure 11 The present application illustrates a device for locating the position of elements on an interface, as provided in an embodiment. It should be understood that this device is similar to the one described above. Figure 2Corresponding to the method embodiments, it can execute the various steps involved in the above method embodiments. The specific functions of the device can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here. The device includes at least one software function module that can be stored in the memory or embedded in the device's operating system in the form of software or firmware. The device for locating the position of elements on the interface includes: an image acquisition module 401, an element recognition and element aggregation result recognition module 402, and an element structure tree construction module 403.

[0124] The image acquisition module is configured to acquire images of the interface to be operated.

[0125] The element recognition and element aggregation result recognition module is configured to input the image into the target detection model and obtain all elements and element aggregation results on the interface to be operated through the target detection model, wherein the element aggregation results are represented by aggregation boxes.

[0126] The element structure tree construction module is configured to construct an element structure tree corresponding to the interface to be operated based at least on the attribute information of the aggregation box, so that the robot can locate the elements on the interface to be operated based on the element structure tree; wherein, the element structure tree includes leaf nodes at the bottom layer and parent nodes at each level, the leaf nodes at the bottom layer are all the elements, and one parent node in the element structure tree corresponds to one aggregation box.

[0127] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0128] Some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the technical solutions of any of the embodiments of the methods for locating the position of elements on an interface described above.

[0129] like Figure 12 As shown, some embodiments of this application provide an electronic device 500, including a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520. When the processor 520 reads the program from the memory 510 via a bus 530 and executes the program, it can implement the technical solutions corresponding to any of the embodiments included in the above-described method for locating the position of elements on an interface.

[0130] Processor 520 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 520 can be a microprocessor.

[0131] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code used to implement some or all of the functions of one or more modules described in the embodiments of this application. The processor 520 of the embodiments of this disclosure can be used to execute the instructions in the memory 510 to implement… Figure 2 The method shown. Memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory well known to those skilled in the art.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0133] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0134] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0136] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0137] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for positioning an element on an interface, characterized in that, The method includes: The image of the interface to be operated is input into the target detection model, and all elements on the interface to be operated and the element aggregation result are obtained through the target detection model, wherein the element aggregation result is represented by an aggregation box. At least an element structure tree corresponding to the interface to be operated is constructed based on the attribute information of the aggregation box, so that the robot can locate the elements on the interface to be operated based on the element structure tree. The element structure tree includes leaf nodes at the bottom level and parent nodes at each level. The leaf nodes at the bottom level are all the elements. One parent node in the element structure tree corresponds to one aggregation box. The step of identifying all elements on the interface to be operated and the element aggregation results through the target detection model includes: The target detection model obtains all bounding boxes on the image, the types of all bounding boxes, and the attribute information of all bounding boxes. The types of all bounding boxes include element detection boxes and aggregate boxes. An element detection box contains an element identified from the interface, and an aggregate box contains multiple elements aggregated according to element positions and logical relationships. The attribute information includes at least one of the size and position of the corresponding box. Elements located within all the element detection boxes are considered as elements identified from the interface to be operated, and multiple elements located within the aggregation box are considered as a group of aggregateable elements.

2. The method as described in claim 1, characterized in that, The step of constructing an element structure tree corresponding to the interface to be operated, based at least on the attribute information of the aggregation box, includes: The parent nodes at each level are obtained based on the attribute information of the aggregation box; The parent node to which the leaf node at the bottom layer belongs is determined based on the position information of the element detection box.

3. The method as described in claim 2, characterized in that, The attribute information includes the position information and size information of each aggregation box.

4. The method according to any one of claims 2-3, characterized in that, The plurality of aggregation boxes includes a first aggregation box and a second aggregation box. in, The step of obtaining the parent nodes at each level based on the attribute information of the aggregation box includes: If it is determined, based on the attribute information of the first aggregation box and the attribute information of the second aggregation box, that the first aggregation box contains the second aggregation box on the image, then it is confirmed that the level of the parent node corresponding to the first aggregation box in the element structure tree is higher than the level of the parent node corresponding to the second aggregation box.

5. The method according to any one of claims 2-3, characterized in that, The step of obtaining the parent nodes at each level based on the attribute information of the aggregation box includes: Retrieve multiple aggregate boxes that have a containment relationship; Sort the multiple aggregated boxes with containment relationships according to their area to obtain a queue; In the queue, the two parent nodes corresponding to two adjacent aggregation boxes are at adjacent levels in the element structure tree.

6. A method for training a detection model, characterized in that, The method includes: An element detection box is marked on the area covered by any element on the training interface image, and at least one aggregation box is marked on the image marked with the element detection box to obtain training data. Among them, multiple elements can be aggregated in an aggregation box. The multiple elements are determined by the position distribution and function of the elements on the corresponding training interface. The target detection model is obtained by training the detection model based on at least the training data; The target detection model is used to obtain all elements on the interface to be operated and the element aggregation results; The step of identifying all elements on the interface to be operated and the element aggregation results through the target detection model includes: The target detection model obtains all bounding boxes on the image, the types of all bounding boxes, and the attribute information of all bounding boxes. The types of all bounding boxes include element detection boxes and aggregate boxes. An element detection box contains an element identified from the interface, and an aggregate box contains multiple elements aggregated according to element positions and logical relationships. The attribute information includes at least one of the size and position of the corresponding box. Elements located within all the element detection boxes are considered as elements identified from the interface to be operated, and multiple elements located within the aggregation box are considered as a group of aggregateable elements.

7. The method of claim 6, wherein, The at least one aggregation box includes a first aggregation box, a second aggregation box, ..., an (N-1)th aggregation box, and an Nth aggregation box, in ascending order of size. The step of annotating at least one aggregated box on the image labeled with the element detection boxes includes: The first aggregation box is labeled in the region containing m aggregateable elements, and the second aggregation box is labeled in the region containing n aggregateable elements, wherein the n elements include the m elements, and so on, until the Nth aggregation box is labeled in the region containing all elements on the training interface image, wherein the Nth aggregation box corresponds to the root node of the tree, the Nth aggregation box includes the (N-1)th aggregation box, and the value of N is an integer greater than 1.

8. The method as described in claim 6, characterized in that, The step of training the detection model based at least on the training data to obtain the target detection model includes: The training of the detection model is determined based on a loss function, wherein the loss function is related to the area of ​​the element detection box and the aggregation box.

9. The method of claim 8, wherein, The formula for calculating the loss function is as follows: Where n is the total number of the element detection boxes and the aggregation boxes, i is the index of any box in the element detection boxes and the aggregation boxes, and Ai represents the area of ​​the i-th box. It is the initial loss function value of the i-th box.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it can implement the method described in any one of claims 1-9.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the program, it can implement the method as described in any one of claims 1-9.

Images