A method for identifying similar elements in RPA based on view relationship graph

Abstract

The present invention discloses a method for identifying similar elements in RPA based on a view relationship graph, comprising the following steps: step 1, obtaining visible elements and element frames corresponding to the visible elements in a web page, constructing a view relationship tree using the visible elements and the element frames, and then converting the view relationship tree into a view relationship graph; step 2, when capturing a target large frame of a target element in a target page, treating an element frame in the view relationship graph that is the same or similar in size to the target large frame as a similar frame; step 3, traversing all similar frames, identifying a target small frame inside the similar frame that is fixed in a relative position to the similar frame, and treating the visible element corresponding to the target small frame as a high-confidence element; step 4, treating the visible elements in the similar frame that meet the positional relationship as similar elements using the high-confidence elements. The present invention can enable the RPA working process to accurately find the range of similar elements, without multiple searches or omissions of similar elements, and has the advantage of high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of robotic process automation technology, and specifically to a method for identifying similar elements in RPA based on a view relationship graph. Background Art

[0002] RPA stands for Robotic Process Automation. Its core capability is to automatically execute the user's business process through robots to improve human work efficiency. In the RPA system, batch processing of similar tasks is a type of frequently called function, which includes but is not limited to specific instructions such as "batch data capture" and "loop similar elements". The core of batch processing of similar tasks is to find "similar elements". The currently known solution for finding similar elements is to find similar nodes of tree nodes in the accessibility tree structure by analyzing the node level and node number of the "barrier-free tree". The disadvantages of this solution are: 1. The "barrier-free tree" structure often changes with the revision of websites and software, resulting in the failure of rules or algorithm systems that only use the "barrier-free tree" structure as input. 2. The "barrier-free tree" structure is a data structure for website and software developers, which is different from the information that users see on the screen after the browser rendering is completed, resulting in RPA applications developed for the "barrier-free tree" structure cannot perfectly reproduce the user's intention. For example, since there are multiple similar standards for tree nodes in the barrier-free tree, it is easy to find or miss similar elements without knowing the (unrendered) interface layout. Summary of the invention

[0003] The purpose of the present invention is to provide a method for identifying similar elements in RPA based on a view relationship graph. The present invention can enable the RPA working process to accurately find the range of similar elements without over-finding or missing similar elements, and has the advantage of high accuracy.

[0004] The technical solution of the present invention is a method for identifying similar elements in RPA based on a view relationship graph, comprising the following steps:

[0005] Step 1: Obtain visible elements in the web page and element frames corresponding to the visible elements, use the visible elements and element frames to build a view relationship tree, and then convert the view relationship tree into a view relationship graph;

[0006] Step 2: When using the RPA software to capture the target large frame of the target element in the target page, the element frames in the view relationship graph that are the same or similar in size to the target large frame are regarded as similar frames;

[0007] Step 3: Within the target page, traverse all similar boxes, identify the target small box inside the similar box with a fixed relative position to the similar box, and take the visible element corresponding to the target small box as a high-confidence element;

[0008] Step 4: Traverse all similar boxes and use high-confidence elements to regard visible elements in similar boxes that satisfy the positional relationship as similar elements.

[0009] In the above-mentioned method for identifying similar elements in RPA based on the view relationship graph, in step 1, the process of constructing a view relationship tree using visible elements and element frames is: sorting the element frames of the visible elements from large to small according to the area, and taking the visible element corresponding to the element frame with the largest area as the root node of the view relationship tree, and taking the corresponding visible elements of the remaining element frames as the descendant nodes of the view relationship tree according to the inclusion relationship of the area size, thereby forming a complete view relationship tree.

[0010] In the aforementioned method for identifying similar elements in RPA based on the view relationship graph, the process of the remaining element frames taking the corresponding visible elements as descendant nodes of the view relationship tree according to the inclusion relationship of the area size is as follows: starting from the root node of the view relationship tree, if the element frame of the visible element C to be inserted is completely contained by the element frame of the root node visible element, then the current visible element C is a descendant node of the root node visible element, and then recursively observing the relationship between all child node visible elements of the root node visible element and the visible element C. If the element frame of a child node visible element A of the root node visible element still completely contains the element frame of the visible element C to be inserted, then the visible element C to be inserted is a descendant node of the child node visible element A, and then repeating the process until a child node visible element B of a certain layer is encountered. At this time, the visible element C is not a descendant node of any other child node visible element of the child node visible element B, then the visible element C is a direct child node of the child node visible element B, and is in a parallel relationship with other child node visible elements of the child node visible element B.

[0011] The aforementioned method for identifying similar elements in RPA based on the view relationship graph converts the view relationship tree into a view relationship graph in the following process:

[0012] Step 1.1, based on the multimodal model, taking the coordinates of the element box and the webpage screenshot as input, output the style category of the visible element;

[0013] Step 1.2, based on a predefined semantic model, taking the text of the visible element as input, outputting the semantic category of the visible element;

[0014] Step 1.3, storing the style category and semantic category of the visible element as node attributes in the node of the view relationship tree;

[0015] Step 1.4: Based on spatial location, group visible elements with the same function, close location and / or similar pattern into the same group of elements;

[0016] Step 1.5: Based on the semantics and spatial location of visible elements, consider the elements that are functionally and conceptually generalized to other elements as conceptually related elements;

[0017] Step 1.6: construct bidirectional edges between nodes of the same group of elements, and construct unidirectional edges or semantically relative bidirectional edges between nodes of conceptually related elements, so as to expand the view relationship tree structure into a view relationship graph structure.

[0018] In the aforementioned method for identifying similar elements in RPA based on a view relationship graph, the pattern similarity includes similarity in element box layout and semantics of visible elements;

[0019] Whether the element frame layouts are similar is determined based on the coordinates and sizes of the element frames;

[0020] Whether the semantics of the visible elements are similar is determined by the texts of the visible elements.

[0021] In the aforementioned method for identifying similar elements in RPA based on the view relationship graph, the process of determining the element frame with a size similar to the target large frame in the view relationship graph is as follows:

[0022] When the length or width of an element box in the view relationship graph is equal to the length or width of the target large box captured by the RPA software, the distance between the four sides of the target small boxes corresponding to all visible elements in all candidate element boxes and the target large box is calculated. At this time, it is determined whether the candidate element box and the target small box of the target element captured by the user in the target large box meet a specific combination of the following conditions:

[0023] i. The distances of n ≥ 3 of the 4 edges are equal;

[0024] ii. The distances of n ≥ 2 of the 4 edges are equal, and the length and width of the target box of the visible element are exactly equal:

[0025] If so, the target frame and the element frame are considered similar frames.

[0026] In the above-mentioned method for identifying similar elements in RPA based on view relationship graph, in step 3, the judgment process of the high-confidence element is as follows:

[0027] Step 3.1. Calculate the distances between the four edges of the target small box and the similar box of all visible elements inside all similar boxes;

[0028] Step 3.2. When one of the following conditions is met:

[0029] a. At least 3 margins are equal, and at least one of the width or height is equal;

[0030] b. At least 2 margins are equal, width and height are equal, and the text of the visible elements is the same;

[0031] The visible elements inside the similarity box are considered high-confidence elements, and the others that do not meet the conditions are considered non-high-confidence elements;

[0032] Step 3.3. Assign the same group_id to high confidence elements in different similarity boxes.

[0033] In the aforementioned method for identifying similar elements in RPA based on the view relationship graph, in step 4, using high-confidence elements to regard visible elements in similar boxes that satisfy the positional relationship as similar elements includes the following conditions:

[0034] a. All visible elements in similar boxes with the same group_id are directly regarded as similar elements;

[0035] b. For target elements whose target small boxes do not overlap with the target small boxes of other high-confidence elements in the target large box to which they belong, obtain the relative position y1 and distance d1 of the target element and the high-confidence element; then search for the visible element N and the corresponding high-confidence element in other similar boxes, obtain the relative position y1 and distance d2 of the visible element N and the high-confidence element, and when y1=y2, d1=d2, the target element and element N are similar elements;

[0036] c. If there are target elements contained in the target small box and the target small boxes of other high-confidence elements in the target large box to which it belongs, calculate the distance between the four edges of the target element and the high-confidence element; then find the visible element N and the corresponding high-confidence element in other similar boxes, and obtain the distance between the four edges of the visible element N and the high-confidence element; if there are at least three edges with equal margins, and at least one of the width or height is equal, then the target element and element N are similar elements;

[0037] d. For a target element whose length and width are exactly the same as those of the visible element N in the target small box and the similar box, and whose texts are the same or similar to those of the visible element N, the target element and element N are similar elements.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. The method designed by the present invention for finding similar elements based on the view relationship graph includes the position layout information and style information of the visible elements in the interface. Compared with the "barrier-free DOM tree" structure, it retains the interface information consistent with the user's perception (such as: interface element layout, style, text semantics), and includes the necessary information for fully understanding user needs, so that the scope of similar elements can be accurately found. By combining with the barrier-free tree method, the effect of not finding too many similar elements and not missing them can be achieved.

[0040] 2. The method designed by the present invention for finding similar elements based on the view relationship graph has a cross-platform feature compared to the "barrier-free DOM tree" structure. The present invention converts the "barrier-free trees" of different platforms into a unified view relationship graph representation, so that the subsequent cross-platform similar element identification can use the same set of solutions, thereby reducing development costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Represents the element boxes corresponding to different visible elements in a web page;

[0042] Figure 2 A schematic diagram of the present invention for sorting visible elements from large to small in terms of area;

[0043] Figure 3 A schematic diagram of the present invention using a visible element as a descendant node of a root node visible element;

[0044] Figure 4 A schematic diagram of a child node of a visible element as a child node of the present invention;

[0045] Figure 5 It is a data structure diagram of the view relationship graph;

[0046] Figure 6 This is a schematic diagram of the target frame;

[0047] Figure 7 It is a schematic diagram of condition b for similar element determination;

[0048] Figure 8 It is a schematic diagram of condition c for similar element determination;

[0049] Fig. 9 It is a schematic diagram of condition d for similar element determination. DETAILED DESCRIPTION

[0050] The present invention is further described below in conjunction with the accompanying drawings and embodiments, but they are not intended to limit the present invention.

[0051] Embodiment: A method for identifying similar elements in RPA based on a view relationship graph comprises the following steps:

[0052] Step 1: Obtain visible elements in the web page and element frames corresponding to the visible elements, use the visible elements and element frames to build a view relationship tree, and then convert the view relationship tree into a view relationship graph;

[0053] In this step, web pages include computer web pages, software clients or mobile phone clients; visible elements refer to buttons, icons, input boxes, text and other controls or content on the web pages that can be perceived by users. Due to the presence of multiple layers of panels or areas on the page, the elements edited in advance in the accessibility tree (dom code) are not always visible. There are special functions in js to distinguish whether the elements in the current web page or client interface are in a visible state. After obtaining the visible element, a unique-id will be assigned to the visible element for subsequent access to the element. In addition, all visible elements can call js functions to calculate the coordinate position and size (x, y, width width, height height). These four values ​​can be used to draw a rectangular box, called an element box, which can just cover the visible elements in the screenshot of the target page. For example Figure 1 As shown, Figure 1 Indicates the element boxes corresponding to different visible elements in a web page. The coordinates of the element boxes can be obtained through Selenium and Microsoft Inspect tools. Selenium and Microsoft Inspect tools can also be used to obtain screenshots of the front-end code of the target page after rendering.

[0054] In this step, the basic data of the visible element is also obtained, that is, the text of the visible element. The text is generally used to describe the function of the visible element. When obtaining the text (innerText) attribute of the visible element in the accessibility tree, attributes that are invisible to the user are filtered out, such as category, id, image hyperlink, etc.

[0055] When the element frame is obtained, such as Figure 2 As shown, the element frames of the visible elements are sorted from large to small by area, and the visible element corresponding to the element frame with the largest area is used as the root node of the view relationship tree. The remaining element frames use the corresponding visible elements as descendant nodes of the view relationship tree according to the inclusion relationship of the area size, thus forming a complete view relationship tree. Among them, the specific process of constructing the view relationship tree is to start from the root node of the view relationship tree. If the element frame of the currently inserted visible element C is completely contained by the element frame of the root node visible element, then the current visible element C is a descendant node of the root node visible element. Next, recursively observe the relationship between all the child node visible elements of the root node visible element and the visible element C. If the element frame of a child node visible element A of the root node visible element still completely contains the element frame of the currently inserted visible element C (corresponding Figure 3), then the visible element C to be inserted is the descendant node of the child node visible element A, and the process is repeated until a child node visible element B of a certain layer is encountered. At this time, the visible element C is not the descendant node of any other child node visible element of the child node visible element B. Then the visible element C is the direct child node of the child node visible element B, and is in a parallel relationship with other child node visible elements of the child node visible element B (corresponding to Figure 4 ).

[0056] After the view relationship tree is constructed, the view relationship graph is constructed using the view relationship tree. The specific process is as follows:

[0057] Step 1.1, based on the multimodal model, take the element frame coordinates and the interface screenshot as input, and output the category of the visible element;

[0058] In this step, the multimodal model can use local CV models predefined based on RPA business, such as classification (ViT) and object detection (Yolo). In these local CV models, the element box coordinates and interface screenshots obtained in step 1 are used as input, and the style category of the element is output, such as "input box", "button", "icon", "date panel", etc.

[0059] Step 1.2, based on a predefined semantic model, taking the text and style of the visible element as input, outputting the semantic category of the visible element;

[0060] In this step, you can use regular expressions, large language models (LLMs), and other semantic models predefined based on RPA services to take element text and style as input and output the semantic category of the element, such as title, navigation bar, number, time, and location.

[0061] Step 1.3, storing the style category and semantic category of the visible element as node attributes in the node of the view relationship tree;

[0062] Step 1.4: Based on spatial location, group visible elements with the same function, close location and / or similar pattern into the same group of elements;

[0063] In this step, the similarity pattern includes the element box layout and the semantics of the visible elements;

[0064] Whether the element frame layout is similar is determined based on the coordinates and size of the element frame; for example, if the x-coordinate and width of the element frame are completely equal, it means that the corresponding elements are similar, which can be vertical list content, including Weibo, Baidu search results, etc. in an example;

[0065] Whether the semantics of the visible elements are similar is determined by the text attributes of the visible elements. For example, if the text of a group of elements are all place names (such as Hangzhou, Shanghai, Ningbo), then this group of text elements can be regarded as a "place name" element group, and in subsequent applications, this element group can be used as an input variable for a screening action.

[0066] Step 1.5: Based on the semantics and spatial positions of visible elements, elements that are functionally and conceptually generalized to other elements are considered as conceptually related elements;

[0067] For example, if the element "Remember password" is visible on the target page, there is usually a "check box" on the left side of it, so "Remember password" and "check box" are conceptually related elements;

[0068] Step 1.6: Build bidirectional edges between nodes of the same group of elements, and build unidirectional edges or semantically related bidirectional edges between nodes of conceptually related elements to expand the view relationship tree structure into a view relationship graph structure. The final data structure is as follows: Figure 5 shown.

[0069] For example, take "number of visitors" as a visible element on a page. When it is a node in the view relationship graph, "payment amount", "number of paid sub-orders", ..., "number of views", etc. are elements in the same group as "number of visitors". At this time, there is a bidirectional edge named "same group element" from the "number of visitors" node to the "number of views" node. "0" is a conceptual association element of "number of visitors", where the "number of visitors" node is the key and the "0" node is the value. At this time, the "number of visitors" node has an edge pointing to the "0" node, named "key", and the "0" node has an edge pointing to the "number of visitors" node, named "value".

[0070] Step 2: When using the RPA software to capture the target large frame of the target element in the target page, the element frames in the view relationship graph that are the same or similar in size (length, width) to the target large frame are regarded as similar frames;

[0071] In this step, the target frame contains the content required for one cycle of RPA software batch operation. Figure 6 As shown, for example, when using RPA software to obtain multiple products displayed on an e-commerce page, each product will correspond to product pictures, product introductions, product prices, sales volume and other contents. The target large frame captured by the RPA software will include product pictures, product introductions, product prices and sales volume and other contents. These product pictures, product introductions, product prices and sales volume are visible elements, and the element frame corresponding to the visible elements is the target small frame, which is contained in the target large frame.

[0072] In this step, the process of determining the element frame with a similar size to the target large frame in the view relationship graph is as follows:

[0073] When the length or width of an element box in the view relationship graph is equal to the length or width of the target large box captured by the RPA software, the distance between the four sides (up, down, left, and right) of the target small boxes corresponding to all visible elements in all candidate element boxes and the target large box is calculated. At this time, it is determined whether the candidate element box and the target small box of the target element captured by the user in the target large box meet a specific combination of the following conditions:

[0074] i. The distances of n ≥ 3 of the 4 edges are equal;

[0075] ii. The distances of n ≥ 2 of the 4 edges are equal, and the length and width of the target box of the visible element are exactly equal:

[0076] If so, the target frame and the element frame are considered similar frames.

[0077] Step 3: Within the target page range, traverse all similar frames, identify the target small frame inside the similar frame whose relative position is fixed to the similar frame, and use the visible element corresponding to the target small frame as a high-confidence element. The high-confidence element is used to assist in identifying similar elements of any visible element inside the target large frame.

[0078] In this step, the judgment process of the high confidence element is as follows:

[0079] Step 3.1. Calculate the distances between all visible elements inside all similar boxes and the four edges between the similar boxes;

[0080] Step 3.2. When one of the following conditions is met:

[0081] a. At least 3 margins are equal, and at least one of the width or height is equal;

[0082] b. At least 2 margins are equal, width and height are equal, and the text of the visible elements is the same;

[0083] The visible elements inside the similarity box are considered high-confidence elements, and the others that do not meet the conditions are considered non-high-confidence elements;

[0084] Step 3.3. Assign the same group_id to high confidence elements in different similarity boxes.

[0085] Step 4: traverse all similar boxes, and use high-confidence elements to regard visible elements in similar boxes that satisfy the position relationship as similar elements;

[0086] In this step, using high-confidence elements to regard visible elements in similar boxes that satisfy the positional relationship as similar elements includes the following conditions:

[0087] a. All visible elements in similar boxes with the same group_id are directly regarded as similar elements;

[0088] b. For target elements whose target small boxes do not overlap with the target small boxes of other high-confidence elements in the target large box to which they belong, obtain the relative position y1 and distance d1 of the target element and the high-confidence element; then find the visible element N and the corresponding high-confidence element in other similar boxes, obtain the relative position y1 and distance d2 of the visible element N and the high-confidence element, and when y1=y2, d1=d2, the target element and element N are similar elements, such as Figure 7 As shown;

[0089] c. For the target small box and other high-confidence elements in the target large box to which it belongs, there are target elements that are mutually contained, such as Figure 8 As shown, the distances between the four edges of the target element and the high-confidence element are calculated; then the visible element N and the corresponding high-confidence element are searched in other similar boxes. If the visible element N and the high-confidence element have at least three edges with equal margins (such as d1=d4, d2=d5, d3=d6), and at least one of the width or height is equal (x1=x2), then the target element and element N are similar elements;

[0090] d. For target elements whose length and width are exactly the same as those of visible element N in the target small box and similar box (x1=x2, y1=y2), and whose texts are the same or similar to those of visible element N (the text contents are both place names, such as Hangzhou and Shanghai), then the target element and element N are similar elements. Fig. 9 shown.

[0091] Finally, when the RPA software captures any product introduction, it can obtain other product introductions as its similar elements. When the RPA software captures any product picture, it can obtain other product pictures as its similar elements. When the RPA software captures any product price, it can obtain other product prices as its similar elements. When the RPA software captures any product sales, it can obtain other product sales as other elements.

[0092] In summary, the method designed by the present invention for finding similar elements based on the view relationship graph includes the position layout information and style information of the visible elements in the interface. Compared with the "barrier-free DOM tree" structure, it retains the interface information consistent with the user's perception (such as: interface element layout, style, text semantics), and includes the necessary information for a complete understanding of user needs, so that the scope of similar elements can be accurately found. By combining with the barrier-free tree method, the effect of not finding too many or missing similar elements can be achieved.

Claims

1. A method for identifying similar elements in RPA based on a view relationship graph, characterized in that: The steps include: Step 1: Obtain visible elements in the web page and element frames corresponding to the visible elements, use the visible elements and element frames to build a view relationship tree, and then convert the view relationship tree into a view relationship graph; Step 2: When using the RPA software to capture the target large frame of the target element in the target page, the element frames in the view relationship graph that are the same or similar in size to the target large frame are regarded as similar frames; Step 3: Within the target page range, traverse all similar boxes in the view relationship graph, identify the target small box inside the similar box with a fixed relative position to the similar box, and take the visible element corresponding to the target small box as a high-confidence element; Step 4: traverse all similar boxes, and use high-confidence elements to regard visible elements in similar boxes that satisfy the position relationship as similar elements; The process of converting the view relationship tree into a view relationship graph is: Step 1.1, based on the multimodal model, taking the coordinates of the element box and the webpage screenshot as input, output the style category of the visible element; Step 1.2, based on a predefined semantic model, taking the text of the visible element as input, outputting the semantic category of the visible element; Step 1.3, storing the style category and semantic category of the visible element as node attributes in the node of the view relationship tree; Step 1.4: Based on spatial location, group visible elements with the same function, close location and / or similar pattern into the same group of elements; Step 1.5: Based on the semantics and spatial location of visible elements, consider the elements that are functionally and conceptually generalized to other elements as conceptually related elements; Step 1.6: construct bidirectional edges between nodes of the same group of elements, and construct unidirectional edges or semantically relative bidirectional edges between nodes of conceptually related elements, so as to expand the view relationship tree structure into a view relationship graph structure.

2. The method for identifying similar elements in RPA based on a view relationship graph according to claim 1 is characterized in that: In step 1, the process of constructing a view relationship tree using visible elements and element frames is: sort the element frames of the visible elements from large to small according to area, and use the visible element corresponding to the element frame with the largest area as the root node of the view relationship tree. The remaining element frames use the corresponding visible elements as descendant nodes of the view relationship tree according to the inclusion relationship of area size, thereby forming a complete view relationship tree.

3. The method for identifying similar elements in RPA based on a view relationship graph according to claim 2 is characterized in that: The process of the remaining element frames treating the corresponding visible elements as descendant nodes of the view relationship tree according to the inclusion relationship of area size is as follows: starting from the root node of the view relationship tree, if the element frame of the currently to-be-inserted visible element C is completely contained by the element frame of the root node visible element, then the currently visible element C is a descendant node of the root node visible element, and then recursively observing the relationship between all child node visible elements of the root node visible element and the visible element C, if the element frame of a child node visible element A of the root node visible element still completely contains the element frame of the currently to-be-inserted visible element C, then the to-be-inserted visible element C is treated as a descendant node of the child node visible element A, and the process is repeated until a child node visible element B of a certain layer is encountered, and at this time the visible element C is not a descendant node of any other child node visible element of the child node visible element B, then the visible element C is treated as a direct child node of the child node visible element B, and is in a parallel relationship with other child node visible elements of the child node visible element B.

4. The method for identifying similar elements in RPA based on a view relationship graph according to claim 1, characterized in that: The pattern similarity includes similarity of element box layout and semantics of visible elements; Whether the element frame layouts are similar is determined based on the coordinates and sizes of the element frames; Whether the semantics of the visible elements are similar is determined by the texts of the visible elements.

5. The method for identifying similar elements in RPA based on a view relationship graph according to claim 1, characterized in that: The process of determining the element frame with a similar size to the target large frame in the view relationship graph is as follows: When the length or width of an element box in the view relationship graph is equal to one of the length or width of the target large box captured by the RPA software, the distance between the four sides of the target small boxes corresponding to all visible elements in all candidate element boxes and the target large box is calculated. At this time, it is determined whether the candidate element box and the target small box of the target element captured by the user in the target large box meet the following conditions: i. The distances of n ≥ 3 of the 4 edges are equal; ii. The distances of n ≥ 2 of the 4 edges are equal, and the length and width of the target box of the visible element are exactly equal: If so, the element box is treated as a similar box.

6. The method for identifying similar elements in RPA based on a view relationship graph according to claim 1, characterized in that: In step 3, the judgment process of the high confidence element is as follows: Step 3.

1. Calculate the distances between the four edges of the target small box and the similar box of all visible elements inside all similar boxes; Step 3.

2. When one of the following conditions is met: a. At least 3 margins are equal, and at least one of the width or height is equal; b. At least 2 margins are equal, width and height are equal, and the text of the visible elements is the same; The visible elements inside the similarity box are considered high-confidence elements, and the others that do not meet the conditions are considered non-high-confidence elements; Step 3.

3. Assign the same group_id to high confidence elements in different similarity boxes.

7. The method for identifying similar elements in RPA based on a view relationship graph according to claim 6 is characterized in that: In step 4, using high confidence elements to regard visible elements in similar boxes that satisfy the positional relationship as similar elements includes the following conditions: a. All visible elements in similar boxes that have the same group_id are directly considered as similar elements; b. For target elements whose target small boxes do not overlap with the target small boxes of other high-confidence elements in the target large box to which they belong, obtain the relative position y1 and distance d1 of the target element and the high-confidence element; then search for the visible element N and the corresponding high-confidence element in other similar boxes, obtain the relative position y1 and distance d2 of the visible element N and the high-confidence element, and when y1=y2, d1=d2, the target element and element N are similar elements; c. If there are target elements contained in the target small box and the target small boxes of other high-confidence elements in the target large box to which it belongs, calculate the distance between the four edges of the target element and the high-confidence element; then find the visible element N and the corresponding high-confidence element in other similar boxes, and obtain the distance between the four edges of the visible element N and the high-confidence element; if there are at least three edges with equal margins, and at least one of the width or height is equal, then the target element and element N are similar elements; d. For a target element whose length and width are exactly the same as those of the visible element N in the target small box and the similar box, and whose texts are the same or similar to those of the visible element N, the target element and element N are similar elements.

Images