A method and system for locating web page elements in an RPA system based on a DOM tree

Abstract

The present invention proposes a method and system for locating web page elements in an RPA system based on a DOM tree, which belongs to the field of robotic process automation (RPA) technology. The method includes: obtaining a target element and a DOM structure tree by using an RPA system; traversing the DOM structure tree from bottom to top, using a custom element structure to store the attributes of the current element and adding them to a node list; using a breadth-first search algorithm to find out whether the current element has a brother element with the same element attribute, and if so, setting the "use" attribute value of the "index" object of the element to true; finally, returning the node list that stores the path position of the target element in the DOM structure tree to the background, and the target element positioning task can be achieved according to the path position.

Description

Technical Field

[0001] The present invention relates to the technical field of Robotic Process Automation (RPA), and in particular to a method and system for locating web page elements of an RPA system based on a DOM tree. Background Art

[0002] Since large factories have introduced assembly line robots, the automation rate of factory assembly lines has increased significantly. However, in sharp contrast, the office automation rate is still extremely low, and almost all office work needs to be done manually. At present, companies have chosen to introduce software robots in offices to reduce costs and increase efficiency, just like factories introduced assembly line hardware robots, and handed over some repetitive and simple work to software robots. Implementers simplify business processes according to business needs, use RPA to simulate human mouse and keyboard operations, such as clicking on hyperlinks in web pages, entering a paragraph of text, etc., to replace people to interact with business systems, complete a series of actions to process business, and realize the full automation of business processes through RPA robots deployed on computers or servers.

[0003] A large part of RPA system operations is web interface operations. Therefore, how to accurately locate web page elements is a key technology of RPA. The DOM structure tree is a standard programming interface for processing extensible markup language recommended by the W3C organization. It accurately describes the interrelationships between tags in HTML documents. JavaScript programming language can be used to implement various operations on web page elements, such as obtaining the attributes of the target element, obtaining the parent element of the target element, etc.

[0004] Existing web page element positioning methods mainly use web page parsers and AI technology to detect and identify target elements. The former cannot accurately locate web page elements in some special scenarios, and the latter needs to rely on a large amount of manually annotated data to train the model, which is labor-intensive, and the positioning accuracy is heavily dependent on the model and data set. Summary of the invention

[0005] In view of the above problems, the present invention provides a method and system for locating web page elements in an RPA system based on a DOM tree, and the technical solution adopted is as follows:

[0006] A method for locating web page elements of an RPA system based on a DOM tree includes the following steps:

[0007] S1: Use the RPA system to obtain the target element of the web page interface and the DOM structure tree; during initialization, use the target element as the current element;

[0008] S2: Determine whether the current element is a or tag element. If the current element is a or element, go to S6, otherwise go to S3;

[0009] S3: Get the attributes of the current element in the DOM structure tree, and store them through the custom element structure. Set the "use" attribute of its "tagName" object to true, and set the "use" attributes of the "id" object, "class" object, and "index" object to false, that is, only enable the "tagName" object of the current element; add the current element to the node list;

[0010] S4: Determine whether the current element has a sibling element with the same element attributes. If so, set the "use" attribute value of the "index" object of the element to true, that is, enable the "index" object of the current element;

[0011] S5: Update the current element to its parent element and loop into S2;

[0012] S6: traverse the node list in reverse order, and set the "use" attribute of the "id" object of the remaining elements to false, except for the first element with the "id" object, that is, only enable the last element with the "id" object;

[0013] S7: Return a node list, in which the path position of the target element in the DOM structure tree is stored. The target element positioning task can be achieved according to the path position.

[0014] Furthermore, each custom element structure contains four objects: "tagName" object, "id" object, "class" object and "index" object; each object contains "value" attribute, "use" attribute and "matchMode" attribute; among them, the "value" attribute is used to mark the value of the object, called the attribute value; the "use" attribute is used to mark whether the object is enabled; the "matchMode" attribute is used to mark the "value" attribute value matching method;

[0015] Among the four objects described, the "tagName" object is used to store the tag name information of the element, the "id" object is used to store the id attribute information of the element, the "class" object is used to store the class attribute information of the element, and the "index" object is used to store the sequence number information of the element from left to right among sibling elements, and the sequence number starts from 0.

[0016] Furthermore, the step S4 is specifically as follows:

[0017] S4-1: According to the DOM structure tree, determine whether the parent element of the current element is empty. If so, define the tag element as the parent element of the current element;

[0018] S4-2: Get the child elements of the parent element of the current element and add them to the element list;

[0019] S4-3: Match the elements in the node list with each element in the element list in turn. If the match is successful, store the elements in the node list in the matching element list at the same level, update the element in the current node list to the next element in the node list, and update the element list to the child elements of each element in the matching element list at the same level; if the match is unsuccessful, update the element list to the child elements of each element in the original element list;

[0020] S4-4: Determine whether the number of elements in the matching element list at the same level is greater than 1. If so, there are sibling elements with the same element attributes, and set the "use" attribute of the "index" object of the current element to true.

[0021] Furthermore, the matching method in step S4-3 is:

[0022] Adopt the breadth-first search algorithm, loop through the node list, loop through the elements in the element list for a certain element in the current node list, and compare the attributes of the two elements one by one, such as whether the "value" attribute value of the "tagName" object or the "value" attribute value of the "class" object is equal. If they are equal, the match is successful, and the elements in the node list are stored in the list of matching elements at the same level; after the elements in the element list complete a loop, determine whether the list of matching elements at the same level is empty;

[0023] If the matching element list at the same level is empty, the element list is updated to the child elements of each element in the original element list, and the elements in the updated element list are traversed in a loop;

[0024] If the same-level matching element list is not empty, the element list is updated to the child elements of each element in the same-level matching element list, the element in the current node list is updated to the next element in the node list, and matching continues until all elements in the node list are matched.

[0025] The present invention also provides a DOM tree-based RPA system web page element positioning system, which is used to implement the above-mentioned DOM tree-based RPA system web page element positioning method.

[0026] The present invention can realize the accurate positioning of web page elements by the RPA system in common or special web page scenarios, without the need to use web page parsers and AI technology for element identification, and without relying on machine learning models and annotated data sets. It has low computational complexity and high positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0028] Figure 1 It is a flow chart of a method for locating web page elements of an RPA system based on a DOM tree according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present application is described clearly and completely below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0030] like Figure 1 As shown, the present invention proposes a method for locating web page elements in an RPA system based on a DOM tree, comprising the following steps:

[0031] S1: Use the RPA system to obtain the target element of the web page interface and the DOM structure tree. During initialization, the target element is used as the current element.

[0032] In this embodiment, the RPA system can obtain the target element of the web page interface through a browser plug-in, such as a Chrome plug-in, and grab the DOM structure tree file to the background for subsequent positioning tasks.

[0033] S2: Determine whether the current element is a or tag element. If the current element is a or element, enter S6, otherwise enter S3.

[0034] In this embodiment, the current element is traversed from bottom to top based on the DOM structure tree. Specifically, the parent element of the current element is obtained in a continuous loop until the parent element is a or tag element.

[0035] S3: Get the attributes of the current element in the DOM structure tree, and store them through the custom element structure, set the "use" attribute of its "tagName" object to true, and set the "use" attributes of the "id" object, "class" object, and "index" object to false, that is, only enable the "tagName" object of the current element; add the current element to the node list; in this embodiment, after obtaining the object and attributes of the current element, save them in the custom element structure in json format, as shown in Table 1:

[0036] Table 1 Objects and attributes of an element

[0037]

[0038] Each custom element structure contains four objects: "tagName" object, "id" object, "class" object and "index" object; each object contains "value" attribute, "use" attribute and "matchMode" attribute. In this embodiment, the "value" attribute is used to mark the value of the object, which is called the attribute value; the "use" attribute is used to mark whether the object is enabled; the "matchMode" attribute is used to mark the "value" attribute value matching method, including the string comparison method, which is recorded as "equal", the regular expression matching method, which is recorded as "regexp", and the wildcard matching method, which is recorded as "wildcard", and the default value in Table 1 is "equal".

[0039] Among the four objects, the "tagName" object is used to store the tag name information of the element, the "id" object is used to store the id attribute information of the element, the "class" object is used to store the class attribute information of the element, and the "index" object is used to store the sequence information of the element from left to right among the sibling elements, and the sequence number starts from 0. S4: Determine whether the current element has a sibling element with the same element attribute. If so, set the "use" attribute value of the "index" object of the element to true, that is, enable the "index" object of the current element;

[0040] The advantage of this is that even if there are sibling elements with the same element attributes, the "index" object can be used to accurately locate the position of the current element in the same level of elements, ensuring the uniqueness of the path position of the target element in the DOM structure tree.

[0041] S4-1: According to the DOM structure tree, determine whether the parent element of the current element is empty. If so, define the tag element as the parent element of the current element. In this embodiment, the determination is made through a ternary operator.

[0042] S4-2: Get the child elements of the parent element of the current element and add them to the element list;

[0043] S4-3: Match the elements in the node list with each element in the element list in turn. If the match is successful, store the elements in the node list in the matching element list at the same level, update the element in the current node list to the next element in the node list, and update the element list to the child elements of each element in the matching element list at the same level; if the match is unsuccessful, update the element list to the child elements of each element in the original element list;

[0044] In this embodiment, the matching method is:

[0045] Adopt breadth-first search algorithm, loop through node list, for a certain element in current node list, loop through elements in element list, compare attributes of two elements one by one, such as "tagName", "class" and other objects and corresponding "value" attribute values, if they are equal, the match is successful, and elements in node list are stored in matching element list at the same level; after elements in element list complete one loop, determine whether matching element list at the same level is empty; technicians in this field can modify matching rules according to customized element attributes.

[0046] If the matching element list at the same level is empty, the element list is updated to the child elements of each element in the original element list, and the elements in the updated element list are traversed in a loop.

[0047] If the same-level matching element list is not empty, the element list is updated to the child elements of each element in the same-level matching element list, the element in the current node list is updated to the next element in the node list, and matching continues until all elements in the node list are matched.

[0048] S4-4: Determine whether the number of elements in the matching element list at the same level is greater than 1. If so, there are sibling elements with the same element attributes, and set the "use" attribute of the "index" object of the current element to true;

[0049] S5: Update the current element to its parent element and loop into S2;

[0050] S6: traverse the node list in reverse order, and set the "use" attribute of the "id" object of the remaining elements except the first element with the "id" object to false, that is, only enable the last element with the "id" object;

[0051] In this embodiment, since the "id" object is an element identifier in the HTML language, there is no same id attribute information in the same web page, and it is unique. Therefore, only the last element with the "id" object needs to be selected to complete the element identification task.

[0052] S7: Return the node list. The final node list stores the path position of the current element in the DOM structure tree. The target element positioning task can be achieved based on the path position.

[0053] In this embodiment, the path position of the target element is obtained as shown in Table 2.

[0054] Table 2 Path location of target element

[0055]

[0056]

[0057]

[0058] Corresponding to the aforementioned embodiment of a method for locating web page elements in an RPA system based on a DOM tree, the present application also provides an embodiment of a system for locating web page elements in an RPA system based on a DOM tree, which includes:

[0059] The web page element acquisition module is used to acquire the target element of the web page interface; during initialization, the target element is used as the current element;

[0060] DOM structure tree generation module, which captures the DOM structure tree of the web page background;

[0061] A judgment module is used to judge whether the current element is a or tag element. If the current element is a or element, the loop module is executed; otherwise, the node list processing module is executed;

[0062] The loop module is used to obtain the attributes of the current element in the DOM structure tree, and store them through the custom element structure, setting the "use" attribute of its "tagName" object to true, and setting the "use" attributes of the "id" object, "class" object, and "index" object to false, that is, only enabling the "tagName" object of the current element; adding the current element to the node list;

[0063] And, it is used to determine whether the current element has a sibling element with the same element attributes. If so, the "use" attribute value of the "index" object of the element is set to true, that is, the "index" object of the current element is enabled;

[0064] And, it is used to update the current element to its parent element and return to the judgment module;

[0065] A node list processing module is used to traverse the node list in reverse order, and set the "use" attribute of the "id" object of the remaining elements to false, except for the first element with the "id" object, that is, only the last element with the "id" object is enabled; and return a node list, which stores the path position of the target element in the DOM structure tree;

[0066] The element positioning module is used to execute the path position returned by the node list processing module to achieve the positioning task of the target element.

[0067] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0068] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts refer to the partial description of the method embodiment. The system embodiment described above is merely illustrative, wherein the circulation module may or may not be physically separated. In addition, each functional module in the present invention may be integrated in a processing unit, or each module may exist physically separately, or two or more modules may be integrated in one unit. The above-mentioned integrated modules or units may be implemented in the form of hardware or in the form of software functional units, so as to select some or all of the modules according to actual needs to achieve the purpose of the present application scheme.

[0069] The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or associated with the contents disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. A method for locating web page elements in an RPA system based on a DOM tree, characterized in that: The steps include: S1: Use the RPA system to obtain the target element of the web page interface and the DOM structure tree; during initialization, use the target element as the current element; S2: Determine whether the current element is a or tag element. If the current element is a or element, go to S6, otherwise go to S3; S3: Get the attributes of the current element in the DOM structure tree, and store them through the custom element structure. Set the "use" attribute of its "tagName" object to true, and set the "use" attributes of the "id" object, "class" object, and "index" object to false, that is, only enable the "tagName" object of the current element; add the current element to the node list; S4: Determine whether the current element has a sibling element with the same element attributes. If so, set the "use" attribute value of the "index" object of the element to true, that is, enable the "index" object of the current element; The step S4 is specifically as follows: S4-1: According to the DOM structure tree, determine whether the parent element of the current element is empty. If so, define the tag element as the parent element of the current element; S4-2: Get the child elements of the parent element of the current element and add them to the element list; S4-3: Match the elements in the node list with each element in the element list in turn. If the match is successful, store the elements in the node list in the matching element list at the same level, update the element in the current node list to the next element in the node list, and update the element list to the child elements of each element in the matching element list at the same level. If the match is unsuccessful, update the element list to the sub-elements of each element in the original element list; S4-4: Determine whether the number of elements in the matching element list at the same level is greater than 1. If so, there are sibling elements with the same element attributes, and set the "use" attribute of the "index" object of the current element to true; S5: Update the current element to its parent element and loop into S2; S6: traverse the node list in reverse order, and set the "use" attribute of the "id" object of the remaining elements to false, except for the first element with the "id" object, that is, only enable the last element with the "id" object; S7: Return a node list, in which the path position of the target element in the DOM structure tree is stored. The target element positioning task can be achieved according to the path position.

2. According to the DOM tree-based RPA system web page element positioning method according to claim 1, it is characterized in that: Each custom element structure consists of four objects: "tagName" object, "id" object, "class" object and "index" object; each object contains "value" attribute, "use" attribute and "matchMode" attribute; the "value" attribute is used to mark the value of the object, called the attribute value; the "use" attribute is used to mark whether the object is enabled; the "matchMode" attribute is used to mark the "value" attribute value matching method; Among the four objects, the "tagName" object is used to store the tag name information of the element, the "id" object is used to store the id attribute information of the element, and the "class" object is used to store the class attribute information of the element. The "index" object is used to store the sequence information of an element from left to right among sibling elements, and the sequence starts from 0.

3. According to the method for locating web page elements of an RPA system based on a DOM tree according to claim 1, the matching method in step S4-3 is: Adopt the breadth-first search algorithm, loop through the node list, loop through the elements in the element list for a certain element in the current node list, and compare the attributes of the two elements one by one, such as whether the "value" attribute value of the "tagName" object or the "value" attribute value of the "class" object is equal. If they are equal, the match is successful, and the elements in the node list are stored in the list of matching elements at the same level; after the elements in the element list complete a loop, determine whether the list of matching elements at the same level is empty; If the matching element list at the same level is empty, the element list is updated to the child elements of each element in the original element list, and the elements in the updated element list are traversed in a loop; If the same-level matching element list is not empty, the element list is updated to the child elements of each element in the same-level matching element list, the element in the current node list is updated to the next element in the node list, and matching continues until all elements in the node list are matched.

4. A DOM tree-based RPA system web page element positioning system, characterized in that: Used to implement the RPA system web page element positioning method based on DOM tree as described in claim 1.

Images