RPA element position relationship modeling method and system based on normalized vector representation

Abstract

The present invention relates to the field of RPA technology, and in particular to an RPA element position relationship modeling method and system based on normalized vector representation. The method comprises: S1, in the picking stage, setting D as the vertex coordinate matrix of the target element, A as the vertex coordinate matrix of the anchor element, establishing the relative position relationship between D and A and expressing it with a matrix; S2, normalizing the scale of the matrix to obtain a scale-independent relative position relationship between the two elements; S3, in the execution stage, according to the scale-independent relative position relationship between the two elements, using the anchor element position to locate the target element in the execution interface. The present invention has the characteristics of being able to simplify the construction method of the relative position relationship between the anchor element and the target element in the RPA element matching, and being able to solve the problem that the current construction scheme is sensitive to the system scale factor.

Description

Technical Field

[0001] The present invention relates to the field of RPA technology, and in particular to a method and system for modeling RPA element position relationships based on normalized vector representation. Background Art

[0002] Robotic Process Automation (RPA) is an effective technology for replacing regular, repetitive tasks in enterprises. It is currently being widely adopted across various industries, including finance, taxation, approvals, risk control, operations, logistics, and advertising. RPA technology provides a visual process editor, allowing developers to design and configure process scripts that can be automatically executed within enterprise business systems. During script execution, RPA simulates human actions such as button clicks, keyboard input, file creation, web page navigation, and logical reasoning by simulating keyboard and mouse operations, calling operating system interfaces, and analyzing web page structures.

[0003] The core of RPA technology is the precise location and manipulation of various elements on computer software interfaces, such as buttons, text, search boxes, input boxes, and actionable areas, to complete the clicks, jumps, input, copy, and paste actions required for process execution. Traditional picking and matching techniques can be implemented through operating system instructions, web page structure analysis, and software code parsing. For software that cannot be picked and matched using traditional techniques, the current mainstream approach is to use computer vision technology for recognition.

[0004] RPA element positioning based on computer vision technology can be divided into two types of elements based on "style uniqueness": 1. Unique elements, whose style is unique across the entire interface; 2. Repeating elements, which are elements with the same style but different functions within the entire interface. For the positioning of unique elements, current industry solutions are relatively mature, primarily using template matching or deep learning technology to extract element features for precise positioning. For the positioning of repeating elements, the current mainstream approach in the industry is to introduce anchor elements to assist in positioning.

[0005] When using anchor points for assisted positioning, the relative positional relationship between the anchor element and the target element must be recorded during the picking phase. This allows the target element to be located at its corresponding element on the execution interface during execution, using the anchor element's position on the execution interface. Currently, the industry's representation of relative positional relationships primarily includes orientation relationships such as "upper left" and "lower right," as well as the relative distance and angle between two elements. This representation of relative positional relationships is not only cumbersome and complex, but also has significant limitations. For example, if the system resolution or DPI changes, the size of the two elements will change, and the relative distance between them will also change. In this case, using the relative distance information recorded during picking to assist positioning will result in positioning failure, leading to the failure of the entire RPA process.

[0006] The current mainstream representation method for expressing the relative position relationship between the anchor element and the target element is "direction + distance + angle".

[0007] The method of expressing the relative position relationship between the anchor element and the target element in the form of "direction + distance + angle" has its shortcomings. First, the construction of the relative position relationship is cumbersome; second, it cannot cope with changes in system scale factors (resolution, dpi, etc.). Specifically:

[0008] 1. The relative position relationship representation of "direction + distance + angle" has limitations

[0009] First, this relative position relationship representation method is cumbersome to construct. It requires not only obtaining the coordinates of the two elements but also calculating and recording the three relative position attributes: "direction + distance + angle." Inferring the target element's position based on the anchor element's position is also complex, requiring first inferring the direction, then the angle, and finally the distance to locate the target element, involving numerous tedious calculation steps. Second, these cumbersome calculations consume computing and memory resources (the recorded direction, distance, and angle information all require memory), resulting in lower operational efficiency. Finally, this representation method strictly requires consistency between the system scale factor during element picking and execution. If the system scale factor or interface shape changes during execution, the distance information recorded during the picking phase cannot be used to locate the target element's position on the execution interface based on the corresponding element's position on the execution interface. For example, during the picking phase, the target element and its anchor element are picked, and the target element's relative position relative to the anchor element is recorded as "top left, 100 pixels, 45 degrees." During the execution phase, the anchor element is first located on the execution interface. Then, based on coordinate offsets, the target element is located at a position 45 degrees and 100 pixels above and to the left of the anchor element. However, if the system DPI of the execution interface is 1.5 times that of the initial position, the target element should be located 45 degrees and 150 pixels above and to the left of the anchor element. Using the relative position relationship of 45 degrees and 100 pixels above and to the left to assist in positioning will result in incorrect positioning of the target element, causing the entire RPA process to fail. Therefore, this relative position relationship representation method of "direction + distance + angle" has certain limitations.

[0010] Therefore, it is very important to design an RPA element position relationship modeling method and system based on normalized vector representation, which can simplify the construction method of the relative position relationship between anchor elements and target elements in RPA element matching and solve the problem that the current construction scheme is sensitive to system scale factors. Summary of the Invention

[0011] The present invention is intended to overcome the problems in the prior art of expressing the relative position relationship between anchor elements and target elements in the form of "orientation + distance + angle", which are cumbersome to construct and unable to cope with changes in system scale factors. It provides an RPA element position relationship modeling method and system based on normalized vector representation, which can simplify the construction method of the relative position relationship between anchor elements and target elements in RPA element matching, and can solve the problem that the current construction scheme is sensitive to system scale factors.

[0012] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0013] The RPA element position relationship modeling method based on normalized vector representation includes the following steps:

[0014] S1, in the picking stage, set D as the vertex coordinate matrix of the target element, A as the vertex coordinate matrix of the anchor element, establish the relative position relationship between D and A and express it with matrix θ;

[0015] S2, scale normalize the matrix θ to obtain the scale-independent relative position relationship between the two elements;

[0016] S3, in the execution stage, according to the scale-independent relative position relationship between the two elements, the target element is located using the position of the anchor element in the execution interface.

[0017] Preferably, the vertex coordinate matrix of the target element and the vertex coordinate matrix of the anchor point element are both N rows and 2 columns.

[0018] Preferably, the number of rows N of the matrix θ represents the number of vertices involved in constructing the relative position relationship, and the columns represent the relative coordinate differences of the corresponding vertices.

[0019] Preferably, in step S1, the relative position relationship between D and A is established and represented by a matrix θ, as shown below:

[0020]

[0021] Among them, P A1 、P A2 Respectively represent the two-dimensional coordinates of the lower right and upper left vertices of the anchor element.

[0022] Preferably, the relative position relationship between the two scale-independent elements is specifically as follows:

[0023] D′=A′+θ·(P A′1 -P A′2 );

[0024] Where D′ represents the vertex coordinate matrix of the corresponding element of the target element on the execution interface; A′ represents the vertex coordinate matrix of the corresponding element of the anchor element on the execution interface; P A′1 、P A′2 They respectively represent the lower right and upper left vertex coordinates of the corresponding element of the anchor element on the execution interface.

[0025] Preferably, step S3 includes the following steps:

[0026] S31, when the system scale factor does not change, the formula D′=A′+θ·(P A′1 -P A′2) is established, and the vertex coordinates of the target element corresponding to the element on the execution interface are inferred based on the vertex coordinates of the anchor element corresponding to the element on the execution interface;

[0027] S32, when the system scale factor changes, set the degree of change to α, then:

[0028] A′=A·α

[0029] P A′1 =P A1 α

[0030] P A′2 =P A2 α

[0031] And the relative position relationship of the execution interface is expressed as:

[0032]

[0033] The representation of the relative position relationship is scale-independent;

[0034] The process of locating the target element using the anchor element position in the execution interface is as follows:

[0035] D′=A′+θ·(P A′1 -P A′2 )

[0036] =A·α+θ·((P A1 -P A2 )·α)

[0037] =(A+θ·(P A1 -P A2 ))·α

[0038] =D·α

[0039] Among them, α is calculated by the following formula:

[0040]

[0041] Finally, the process of locating the target element using the anchor element position in the execution interface is directly described as:

[0042]

[0043] Preferably, the system scale factors include resolution and dots per inch (DPI).

[0044] The present invention also provides an RPA element position relationship modeling system based on normalized vector representation, comprising:

[0045] The picking module is used to set D as the vertex coordinate matrix of the target element and A as the vertex coordinate matrix of the anchor element during the picking phase, establish the relative position relationship between D and A and express it with the matrix θ;

[0046] Normalization module, used to normalize the scale of matrix θ to obtain the relative position relationship between two elements that is independent of scale;

[0047] The execution module is used to locate the target element using the anchor element position in the execution interface according to the scale-independent relative position relationship between the two elements during the execution phase.

[0048] Compared with the prior art, the present invention has the following advantages: (1) in terms of relative position relationship representation, the present invention directly obtains the relative position relationship matrix of the two elements by subtracting the vertex coordinates of the anchor element and the target element, which greatly simplifies the steps of obtaining the relative position relationship representation; (2) in terms of scale adaptation, the present invention normalizes the scale of the obtained relative position relationship matrix and further obtains a scale-independent relative position relationship matrix of the two elements. The representation method of the present invention not only has simple construction steps, but also can cope with the situation of changes in various system scale factors, greatly improving the accuracy of RPA repeated element positioning and the stability of RPA process execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the distribution of elements of the picking interface provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the distribution of execution interface elements provided by an embodiment of the present invention;

[0051] Figure 3 A flowchart of the RPA element position relationship modeling method based on normalized vector representation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] To more clearly illustrate the embodiments of the present invention, specific embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings and other embodiments can be obtained based on these drawings without inventive efforts.

[0053] Example:

[0054] The present invention provides an RPA element position relationship modeling method based on normalized vector representation, comprising the following steps:

[0055] S1, in the picking stage, set D as the vertex coordinate matrix of the target element (N rows and 2 columns), and A as the vertex coordinate matrix of the anchor element (N rows and 2 columns). Establish the relative position relationship between D and A and express it with matrix θ; the number of rows N in matrix θ represents the number of vertices involved in building the relative position relationship, and the columns represent the relative coordinate difference of the corresponding vertices, which directly implies the distance, angle, and orientation information;

[0056] In step S1, the relative position relationship between D and A is established and represented by a matrix θ, as shown below:

[0057]

[0058] Among them, P A1 、P A2 Respectively represent the two-dimensional coordinates of the lower right and upper left vertices of the anchor element.

[0059] The relative position relationship between the two scale-independent elements is as follows:

[0060] D′=A′+θ·(P A′1 -P A′2 )

[0061] Where D′ represents the vertex coordinate matrix of the corresponding element of the target element on the execution interface; A′ represents the vertex coordinate matrix of the corresponding element of the anchor element on the execution interface; P A′1 、P A′2 They respectively represent the lower right and upper left vertex coordinates of the corresponding element of the anchor element on the execution interface.

[0062] S2, scale normalize the matrix θ to obtain the scale-independent relative position relationship between the two elements;

[0063] The relative position relationship between two scale-independent elements is as follows:

[0064] D′=A′+θ·(P A′1 -P A′2 )

[0065] Where D′ represents the vertex coordinate matrix of the corresponding element of the target element on the execution interface; A′ represents the vertex coordinate matrix of the corresponding element of the anchor element on the execution interface; P A′1 、P A′2 Respectively represent the lower right and upper left vertex coordinates of the corresponding element of the anchor element on the execution interface

[0066] S3, in the execution phase, based on the relative position relationship between the two scale-independent elements, locating the target element using the anchor element position in the execution interface;

[0067] The specific process of step S3 is as follows:

[0068] S31, when the system scale factor (resolution, dots per inch DPI) does not change, the formula D′=A′+θ·(P A′1 -P A′2 ) is established, and the vertex coordinates of the target element corresponding to the element on the execution interface are inferred based on the vertex coordinates of the anchor element corresponding to the element on the execution interface;

[0069] S32, when the system scale factor (resolution, dots per inch DPI) changes, the degree of change is set to α, then:

[0070] A′=A·ɑ

[0071] P A′1 =P A1 α

[0072] P A′2 =P A2 ·ɑ

[0073] And the relative position relationship of the execution interface is expressed as:

[0074]

[0075] The representation of the relative position relationship is scale-independent;

[0076] The process of locating the target element using the anchor element position in the execution interface is as follows:

[0077] D′=A′+θ·(P A′1 -P A′2 )

[0078] =A·α+θ·((P A1 -P A2 )·α)

[0079] =(A+θ·(P A1 -P A2 ))·α

[0080] =D·α

[0081] Among them, α is calculated by the following formula:

[0082]

[0083] Finally, the process of locating the target element using the anchor element position in the execution interface is directly described as:

[0084]

[0085] It can be seen that the process of using the coordinates of the corresponding element of the anchor element on the execution interface to locate the coordinates of the corresponding element of the target element on the execution interface is also scale-independent.

[0086] The present invention also provides an RPA element position relationship modeling system based on normalized vector representation, comprising:

[0087] The picking module is used to set D as the vertex coordinate matrix of the target element and A as the vertex coordinate matrix of the anchor element during the picking phase, establish the relative position relationship between D and A and express it with the matrix θ;

[0088] Normalization module, used to normalize the scale of matrix θ to obtain the relative position relationship between two elements that is independent of scale;

[0089] The execution module is used to locate the target element using the anchor element position in the execution interface according to the scale-independent relative position relationship between the two elements during the execution phase.

[0090] Based on the technical solution of the present invention, Figure 3 The process shown demonstrates the implementation scheme of the present invention and some of its beneficial effects through an actual use case:

[0091] 1. Such as Figure 1 The following figure shows the interface for picking duplicate elements in an RPA process, with a system DPI of 100%. The target element is 022. During the picking phase, the system automatically determines that 022, 021, and 023 are duplicate elements. Therefore, when picking element 022, the system automatically selects element 026 as the anchor element for target element 022. The coordinates of the top left, top right, bottom left, and bottom right vertices of target element 022 are D = {(Dx1, Dy1), (Dx2, Dy1), (Dx1, Dy2), (Dx2, Dy2)}. The coordinates of the top left, top right, bottom left, and bottom right vertices of anchor element 026 are A = {(Ax1, Ay1), (Ax2, Ay1), (Ax1.Ay2), (Ax2, Ay2)}.

[0092] 2. Based on the relative position relationship representation designed by the present invention, a normalized vector representation θ of the relative position relationship between the anchor element 026 and the target element 022 is established, specifically:

[0093]

[0094] α=[A x2 -A x1 A y2 -A y1 ]

[0095]

[0096] 3. Such as Figure 2The execution interface is shown, and the system DPI is 200%. In the execution phase, the position of the target element 022 to be located in the candidate elements 031, 032, and 033 of the execution interface is first determined based on the appearance features through template matching or deep learning model. Since the style of the anchor element is unique in the entire interface, the position of the anchor element 036 can be uniquely located in the execution interface first. Assume that the vertex coordinate set of the anchor element 036 is A′ and the vertex coordinate set of the target element is D′. According to the positioning model constructed by the present invention: D′=A′+θ·(P A′1 -P A′2 ), we can find out that the corresponding element of the target element in the execution interface is 032, where P A′1 、P A′2 They represent the coordinates of the lower right and upper left corner vertices of the anchor element 036 respectively.

[0097] The present invention greatly simplifies the steps of constructing the relative position relationship between anchor elements and target elements in repeated element matching by normalizing the relative position relationship between anchor elements and target elements into a vector representation. The normalized relative position relationship vector representation is scale-independent and can cope with the inconsistency of system scale factors in various picking and execution stages, greatly improving the accuracy of RPA repeated element matching and the stability of RPA process execution.

[0098] The innovative features of the present invention are as follows:

[0099] 1. The present invention creatively proposes a method for representing the relative position relationship between RPA elements, replacing the existing "direction + distance + angle" representation method with a vectorized representation of the subtraction of element vertex coordinates, greatly simplifying the relative position relationship construction steps.

[0100] 2. This invention creatively optimizes the relative position relationship between RPA elements into a scale-independent vectorized representation. By normalizing the difference in vertex coordinates between two elements, it eliminates the impact of scale factors on the relative position relationship representation, greatly improving the accuracy of RPA element positioning and thus greatly improving the stability of RPA process execution.

[0101] The above description is only a detailed description of the preferred embodiments and principles of the present invention. For ordinary technicians in this field, based on the ideas provided by the present invention, there may be changes in the specific implementation methods, and these changes should also be considered as the scope of protection of the present invention.

Claims

1. The RPA element position relationship modeling method based on normalized vector representation is characterized by: The method includes the following steps: S1, in the picking stage, set D as the vertex coordinate matrix of the target element, A as the vertex coordinate matrix of the anchor element, establish the relative position relationship between D and A and express it with matrix θ; S2, scale normalize the matrix θ to obtain the scale-independent relative position relationship between the two elements; S3, in the execution phase, based on the relative position relationship between the two scale-independent elements, locating the target element using the anchor element position in the execution interface; In step S1, the relative position relationship between D and A is established and represented by a matrix θ, as shown below: Among them, P A1 、P A2 Respectively represent the two-dimensional coordinates of the lower right and upper left vertices of the anchor element; The relative position relationship between the two scale-independent elements is as follows: D′(A′+θ·(P A′1 -P A′2 ) Where D′ represents the vertex coordinate matrix of the corresponding element of the target element on the execution interface; A′ represents the vertex coordinate matrix of the corresponding element of the anchor element on the execution interface; P A′1 、P A′2 Respectively represent the lower right and upper left vertex coordinates of the corresponding element of the anchor element on the execution interface; Step S3 includes the following steps: S31, when the system scale factor does not change, the formula D′=A′+θ·(P A′1 -P A′2 ) is established, and the vertex coordinates of the target element corresponding to the element on the execution interface are inferred based on the vertex coordinates of the anchor element corresponding to the element on the execution interface; S32, when the system scale factor changes, set the degree of change to α, then: A′=A·α P A′1 =P A1 ·α P A′2 =P A2 ·α And the relative position relationship of the execution interface is expressed as: The representation of the relative position relationship is scale-independent; The process of locating the target element using the anchor element position in the execution interface is as follows: D′(A′+θ·(P A′1 -P A′2 ) =A·α+θ·((P A1 -P A2 )·a) =(A+θ·(P A1 -P A2 ))·a ＝D·α Among them, α is calculated by the following formula: Finally, the process of locating the target element using the anchor element position in the execution interface is directly described as:

2. The RPA element position relationship modeling method based on normalized vector representation according to claim 1 is characterized in that: The vertex coordinate matrix of the target element and the vertex coordinate matrix of the anchor point element both have N rows and 2 columns.

3. The RPA element position relationship modeling method based on normalized vector representation according to claim 2 is characterized in that: The number of rows N of the matrix θ represents the number of vertices involved in constructing the relative position relationship, and the columns represent the relative coordinate differences of the corresponding vertices.

4. The RPA element position relationship modeling method based on normalized vector representation according to claim 1 is characterized in that: The system scale factors include resolution and dots per inch (DPI).

5. An RPA element position relationship modeling system based on normalized vector representation, used to implement the RPA element position relationship modeling method based on normalized vector representation according to any one of claims 1 to 4, characterized in that: The RPA element position relationship modeling system based on normalized vector representation includes: The picking module is used to set D as the vertex coordinate matrix of the target element and A as the vertex coordinate matrix of the anchor element during the picking phase, establish the relative position relationship between D and A and express it with the matrix θ; Normalization module, used to normalize the scale of matrix θ to obtain the relative position relationship between two elements that is independent of scale; The execution module is used to locate the target element using the anchor element position in the execution interface according to the scale-independent relative position relationship between the two elements during the execution phase.

Images