Anchor point-based RPA element anchor point automatic repair method

Abstract

The invention discloses an anchor point-based RPA element anchor point automatic restoration method, which comprises the following steps of: acquiring candidate elements through target detection and image feature matching, performing combination vector restoration on the candidate elements, and performing separation vector restoration if the combination vector restoration fails; the combined vector repairing process comprises the following steps: firstly, acquiring coordinates of an original target element, an original anchor point element and a current anchor point element, then calculating a scaling ratio according to the coordinates of the original anchor point element and the current anchor point element, performing scaling calculation on a vector from the current anchor point element to the original target element according to the scaling ratio, and restoring to obtain a region of the original target element; calculating the similarity between the original target element region and the candidate element region according to the original target element region and the candidate element region, and taking the candidate element with the similarity reaching a set threshold value and the highest value as a repaired target element; according to the method, multiple elements can be repaired by combining vector repair and separation vector repair, and the method has the advantage of wide coverage; meanwhile, the method also has the characteristic of low repair error rate.

Description

Technical Field

[0001] The present invention relates to the field of information detection, and particularly to an automatic repair method for RPA element anchor points based on anchor points. Background Art

[0002] At present, Robotic Process Automation (RPA) has become an important tool for many enterprises to optimize business processes and improve efficiency; in RPA, the stability and reliability of automated processes highly depend on the ability of the robot to accurately locate and interact with target interface elements. However, due to the dynamic changes in the target system interface (such as interface layout adjustment, element attribute update, or version upgrade), the robot may not be able to locate the originally set elements, resulting in the interruption or even failure of the automated process. This problem is particularly common in the production environment, posing challenges to the process efficiency and business continuity of enterprises. To solve this problem, element repair technology has gradually become a key technology in the RPA system; element repair technology can automatically adjust the robot's positioning logic in an intelligent way when the target element changes, so as to re-match and locate the correct target element, thus ensuring the continuous operation of the process. Currently, the main methods for automatic element repair are as follows: 1. Intelligent repair through element paths: Calculate a relatively stable element path through rules, and then when operating on similar pages, the corresponding element can also be found through the corresponding rules to achieve element repair; however, since many pages often modify the page path structure and naming to prevent crawling, there are often situations where the path changes while the page visual remains unchanged. Therefore, the element repair relying only on element paths has a narrow coverage. 2. Re-positioning through element images: Based on the screenshot of the target element, perform cv template matching on the current page to achieve element repair; however, for many elements (such as input boxes), when the internal content changes, it cannot be retrieved through template matching. In addition, for many similar elements, such as check boxes, template matching will return multiple elements and cannot achieve element repair either. 3. Re-positioning through anchor points and vectors: Starting from the anchor point of the current page through a normalized vector, re-position the target element, but it can only cover the situation where the relative position between the target element and the anchor point has not changed, and the applicable range is limited. Summary of the Invention

[0003] The purpose of the present invention is to provide an automatic repair method for RPA element anchor points based on anchor points. The present invention can repair various elements by combining vector repair and off-vector repair, and has the advantage of wide coverage; at the same time, the present invention also has the characteristic of low repair error rate.

[0004] Technical solution of the present invention: An automatic repair method for RPA element anchor points based on anchor points, which obtains candidate elements through object detection and image feature matching, first performs combined vector repair on the candidate elements, and if the combined vector repair fails, performs detached vector repair; The process of the combined vector repair is first to obtain the coordinates of the original target element, the original anchor element, and the current anchor element, then calculate the scaling ratio based on the coordinates of the original anchor element and the current anchor element, and scale the vector from the current anchor element to the original target element according to the scaling ratio to restore the area of the original target element; calculate the similarity between the area of the original target element and the area of the candidate element, and select the candidate element with the highest similarity reaching the set threshold as the repaired target element; The process of the detached vector repair is to perform rule repair or model repair on the candidate elements based on the trusted anchor elements; the rule repair is to calculate scores based on the distance and angle between the candidate element and the trusted anchor element, and regard the candidate element that meets the specified score threshold as the repaired target element; the model repair uses a multi-modal model to find a pending anchor element for each candidate element, and if the pending anchor element is the same as the trusted anchor element, the candidate element corresponding to the pending anchor element is the repaired target element.

[0005] In the above automatic repair method for RPA element anchor points based on anchor points, the candidate elements include input boxes, dropdown boxes, checkboxes, icons, and text elements; the rule repair is applied to input boxes, dropdown boxes, and checkboxes; the model repair is applied to icons and text elements.

[0006] In the foregoing automatic repair method for RPA element anchor points based on anchor points, for input boxes and dropdown boxes, the rule repair is to rank the candidate elements according to the relative distance and absolute distance from the trusted anchor element to obtain a comprehensive distance score, and obtain an angle score by calculating the angle between the candidate element and the trusted anchor element; the candidate element that satisfies both the comprehensive distance score and the angle score reaching the specified score threshold is regarded as the repaired target element; for checkboxes, the rule repair obtains the absolute distance score of the candidate element according to the absolute distance from the trusted anchor element, and regards the candidate element whose absolute distance score meets the specified score threshold as the repaired target element.

[0007] In the foregoing automatic repair method for RPA element anchor points based on anchor points, the comprehensive distance score is calculated as shown in the following formula: Comprehensive distance score = relative distance ranking score × absolute distance score; Among them, the relative distance ranking score = 1 - (relative distance ranking of the candidate element and the trusted anchor element / 10); Absolute distance score = min(1, absolute distance between the original target element and the trusted anchor element / absolute distance between the candidate element and the trusted anchor element); The angle score is calculated as shown in the following formula: Angle score = 1 - ((vector angle between the candidate element and the trusted anchor element - minimum angle directly below and directly to the right of the candidate element) / 90).

[0008] In the aforementioned anchor - based RPA element anchor automatic repair method, the image feature matching is performed according to the following steps: The image feature matching is performed according to the following steps: Step A1: Obtain the current interface image and the screenshot image of the original target element, and perform pre - processing of graying and denoising on the two images; Step A2: Detect points with high recognition and high stability in the two images through the Scale - Invariant Feature Transform (SIFT) method to obtain feature points; Step A3: Extract the feature descriptors of the feature points through the Scale - Invariant Feature Transform descriptor, and the feature descriptors are used to describe the regional features around the feature points; Step A4: Compare the feature descriptors of different feature points in the two images through the K - Nearest Neighbor (KNN) algorithm to obtain multiple similar feature points for each feature point in the interface image; Step A5: Calculate the distance ratio between the feature descriptor of each feature point and the feature descriptor of the corresponding similar feature point, compare the distance ratio with the set threshold, and the similar feature points exceeding the threshold are incorrect matches, and the rest are correct matches and are used for the next step; Step A6: Apply the geometric relationship between the correctly matched feature points and the similar feature points to the interface image by solving the transformation matrix to obtain multiple similar regions, and use the similar regions as candidate elements.

[0009] In the aforementioned anchor - based RPA element anchor automatic repair method, the scaling ratio is the ratio of the short - side length of the current anchor element to the short - side length of the original anchor element; the scaling calculation is as shown in the following formula: Current target element coordinates = current anchor element coordinates + scaling ratio × (original target element coordinates - original anchor element coordinates).

[0010] In the aforementioned anchor - based RPA element anchor automatic repair method, the similarity is calculated by first calculating the regional overlap and edge alignment degree between the region of the original target element and the region of the candidate element, and then obtaining the similarity through the product of the regional overlap and the edge alignment degree; The degree of regional overlap is obtained by calculating the intersection over union (IoU) of the regions of the candidate element and the original target element; the IoU is the ratio of the area of the intersection of the regions of the candidate element and the original target element to the area of the union of the regions of the candidate element and the original target element; The calculation of the degree of edge alignment is shown in the following formula: Degree of edge alignment = 1 - min (minimum margin on both sides of the left side of the candidate element / length of the region of the original target element, minimum margin on both sides of the right side of the candidate element / length of the region of the original target element, minimum margin on both sides of the upper side of the candidate element / width of the region of the original target element, minimum margin on both sides of the lower side of the candidate element / width of the region of the original target element).

[0011] In the foregoing anchor - based RPA element anchor automatic repair method, the process of the multi - modal model for finding the to - be - determined anchor element is carried out according to the following steps: Step S1: Obtain the coordinates, text information, and regional screenshots of the candidate element and the corresponding candidate anchor element; Step S2: According to the regional screenshots, obtain the element categories of the candidate element and the candidate anchor element through the object detection model; Step S3: Input the coordinates, text information, regional screenshots, and element categories of the candidate element and the candidate anchor element into the multi - modal model, and use the multi - modal model to detect and determine the to - be - determined anchor element.

[0012] In the foregoing anchor - based RPA element anchor automatic repair method, the detection process of the multi - modal model is carried out according to the following steps: Step S3.1: Form a text input with the text information of the candidate element, the text information of the candidate anchor element, the category of the candidate element, and the category of the candidate anchor element; use the regional screenshots of the candidate element and the candidate anchor element as the image input; extract coordinate features from the coordinates of the candidate element and the candidate anchor element as the coordinate input; Step S3.2: Use the embedding transformation model to transform the text input and the image input into a text vector and an image vector, and then align the text vector, the image vector, and the coordinate feature vector through the fully - connected layer; Step S3.3: Distinguish the candidate element and the candidate anchor element through rotational position encoding, and then enable the text vector, the image vector, and the coordinate feature vector of the candidate element to mutually attend to the text vector, the image vector, and the coordinate feature vector between the candidate element and the candidate anchor factor, obtain the weights of each element through similarity, and sum the vectors after weighting; Step S3.4: Output the candidate anchor element with the highest sum value through the fully - connected layer as the to - be - determined anchor element.

[0013] Compared with the prior art, the present invention first obtains candidate elements through target detection and image feature matching, and first performs combined vector repair on the candidate elements. If the combined vector repair fails, the detached vector repair is performed; the combined vector repair obtains the area of ​​the original target element through scaling calculation, and further obtains the similarity by combining the degree of regional overlap and the degree of edge alignment, and obtains the repaired target element by comparison with the threshold, and repairs the elements that produce scaling changes; the process of detached vector repair is to perform rule repair or model repair on the candidate elements based on trusted anchor elements. The rule repair uses the position law of the elements to formulate rules, and calculates its distance and angle scores according to the rules. When the score reaches the set threshold, the element repair is realized; the model repair searches for the pending anchor element through a multimodal model, and the candidate element corresponding to the pending anchor element and the trusted anchor element is the repaired target element. The detached vector repair realizes the repair of the target element whose position and content have changed, and the repair coverage is wide; at the same time, the present invention obtains candidate elements through target detection and image feature matching, narrows the range of candidate elements, and thereby reduces the error rate of element repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of the present invention; Figure 2 The positional relationship between the original anchor element and the original target element when the present invention was developed; Figure 3 A schematic diagram of repairing candidate elements in combination with vector repair of the present invention; Figure 4 A repair schematic diagram of the regular repair of the present invention; Figure 5 A repair schematic diagram of the model repair of the present invention; Figure 6 is a schematic diagram of the input of the multimodal model of the present invention; Figure 7 It is a schematic diagram of the detection process of step S3.-1 to step S3.2 of the present invention; Figure 8 It is a schematic diagram of the detection process of step S3.3-step S3.4 of the present invention. DETAILED DESCRIPTION

[0015] The present invention is further described below in conjunction with the accompanying drawings and embodiments, but they are not intended to limit the present invention.

[0016] Embodiment: An RPA element anchor point automatic repair method based on anchor points, as shown in the attached Figure 1As shown, candidate elements are obtained through target detection and image feature matching, and the candidate elements are first repaired by combining vectors. If the combined vector repair fails, the trusted anchor element is obtained. If a trusted anchor element exists, the detached vector repair is performed. If the group exists, the repair fails.

[0017] Object detection is a computer vision task that aims to identify and locate specific objects in images or videos. Unlike image classification, object detection not only needs to identify the categories of objects in the image, but also needs to accurately find the locations of these objects on the image. The types of target elements should be recorded in advance during the application development period. When repairing, all elements of this type on the page can be obtained as candidate elements through object detection.

[0018] The image feature matching is performed according to the following steps: Step A1: Obtain the current interface image and the original target element screenshot image, and perform grayscale and denoising preprocessing on the two images. Grayscale is to convert a color image into a grayscale image to reduce computational complexity, and denoising is to use filtering technology (such as Gaussian filtering) to remove noise in the image to avoid affecting the effect of feature extraction; Step A2: Feature points are points with high recognition and stability in the image, usually at the corners and edges of the image. The points with high recognition and high stability in the two images are detected by the scale-invariant feature transformation method to obtain feature points with scale and rotation invariance. Step A3: extract the feature descriptor of the feature point through the scale-invariant feature transform descriptor to describe the features of its surrounding area. It is based on the local image gradient and has rotation and scale invariance. The scale-invariant feature transform algorithm first constructs the scale space of the image through Gaussian blur to detect potential key points. In order to make the features rotation-invariant, the scale-invariant feature transform algorithm assigns a main direction to each key point. The main direction is determined by calculating the gradient direction histogram of the area around the key point. After determining the position and direction of the key point, the algorithm generates a feature descriptor to describe the local image area around the key point. After generating the feature descriptors of all key points, the feature points in different images can be matched by calculating the distance between the feature descriptors. Step A4: Compare the feature descriptors of different feature points in the two images by using the K nearest neighbor algorithm to obtain multiple similar feature points for each feature point in the interface image; that is, in the feature space, if most of the k nearest (i.e., the closest) samples near a sample belong to a certain category, then the sample also belongs to this category; Step A5: Calculate the distance ratio between the feature descriptor of each feature point and the feature descriptor of the corresponding similar feature point (such as the distance ratio between the nearest neighbor and the second nearest neighbor), compare the distance ratio with a set threshold, and the similar feature points exceeding the threshold are incorrect matches, and the rest are correct matches and are used for the next step; for example, there are and two images, and their scale-invariant feature transform feature points and feature descriptors are extracted respectively. The feature descriptor set of image is: , and the feature descriptor set of image is: , where the feature descriptor is a 128-dimensional vector; for the feature descriptor in image , find its nearest neighbor and the second nearest neighbor in image , and calculate the ratio of the nearest neighbor distance to the second nearest neighbor distance through . If the ratio is less than the threshold (0.7 - 0.8), it is a correct match; Step A6: Apply the geometric relationship between the correctly matched feature points and the similar feature points to the interface image by solving the transformation matrix (such as affine transformation, perspective transformation) to obtain multiple similar regions, and use the similar regions as candidate elements.

[0019] The process of solving the transformation matrix is as follows: Step A6.1: Prepare the matching point pairs. For example, there are N pairs of correctly matched feature points and similar feature points. The point set in image is , and the point set in image is ; Step A6.2: The matrix of the perspective transformation satisfies: ; After expansion, it gets: ; ; Further linearize the equation to get: ; ; After arrangement, the linear equation is obtained: ; For N pairs of correctly matched feature points and similar feature points, construct a 2N×8 matrix A and a 2N×1 vector , and the two satisfy: ; wherein, , (fixed as 1 to eliminate scale uncertainty); Step A6.3: Solve the transformation matrix, and use the least squares method to solve the linear equations: ; Step A6.4: Convert the obtained by solving into a perspective transformation matrix : .

[0020] As shown in Appendix Figure 2 and Appendix Figure 3 , the process of combining vector repair is to obtain the coordinates of the original target element, the original anchor element (obtained during the development period), and the current anchor element, and then calculate the scaling ratio based on the coordinates of the original anchor element and the current anchor element. The scaling ratio is the ratio of the short side length of the current anchor element to the short side length of the original anchor element. Scale the vector from the current anchor element to the original target element according to the scaling ratio to restore the original target element area. The scaling calculation is shown in the following formula: Coordinates of the current target element = Coordinates of the current anchor element + Scaling ratio × (Coordinates of the original target element - Coordinates of the original anchor element); Obtain the similarity between the candidate element and the original target element based on the product of the coincidence degree of the area of the candidate element and the original target element and the edge alignment degree. Select the candidate element with the similarity reaching the set threshold and the highest value as the repaired target element; wherein, the coincidence degree of the area is obtained by calculating the intersection over union of the areas of the candidate element and the original target element; the intersection over union is the ratio of the area of the intersection of the areas of the candidate element and the original target element to the area of the union of the areas of the candidate element and the original target element. The calculation of the edge alignment degree is shown in the following formula: Edge alignment degree = 1 - min (Minimum margin of the left two sides of the candidate element / Length of the original target element area, Minimum margin of the right two sides of the candidate element / Length of the original target element area, Minimum margin of the upper two sides of the candidate element / Width of the original target element area, Minimum margin of the lower two sides of the candidate element / Width of the original target element area).

[0021] The process of detached vector repair is to perform regular repair or model repair on the candidate element based on the trusted anchor element. The trusted anchor element is manually captured by the user or automatically found by the model; As shown in Appendix Figure 4As shown, for the input box and the dropdown box, the rule repair ranks the candidate elements according to the relative distance and absolute distance between the candidate elements and the trusted anchor elements to obtain a comprehensive distance score, and obtains an angle score by calculating the angle between the candidate elements and the trusted anchor elements. Since for the input box and the dropdown box, the target element is usually directly below, directly to the right, or to the lower right of the anchor element, the candidate elements that meet the specified score threshold are regarded as the repaired target elements. The calculation of the comprehensive distance score is shown in the following formula: Comprehensive distance score = relative distance ranking score × absolute distance score; Among them, the relative distance ranking score = 1 - (relative distance ranking of the candidate element and the trusted anchor element (ranked from 0 in ascending order / 10)); Absolute distance score = min(1, absolute distance between the original target element and the trusted anchor element / absolute distance between the candidate element and the trusted anchor element); The calculation of the angle score is shown in the following formula: Angle score = 1 - ((vector angle between the candidate element and the trusted anchor element - minimum angle directly below and directly to the right of the candidate element) / 90). When the angle exceeds this area (i.e., 90°), the score is zero; For the checkbox, the target element is usually the first element directly to the left of the anchor element. Therefore, only the absolute distance score of the first element on the left needs to be calculated, and the candidate elements that meet the specified score threshold are regarded as the repaired target elements.

[0022] As attached Figure 5 As shown, the model repair uses a multimodal model to find a pending anchor element for each candidate element. If the pending anchor element is the same as the trusted anchor element, the candidate element corresponding to the pending anchor element is the repaired target element; As attached Figure 6 As shown, the process of the multimodal model finding the pending anchor element is carried out according to the following steps: Step S1: Obtain several leaf nodes close to the target element from the DOM tree as candidate anchor elements. At the same time, obtain the coordinates and text information of the candidate elements and candidate anchor elements in the DOM. The coordinates of the elements can be obtained by using the getBoundingClientRect() method in the DOM interface. At the same time, obtain the area screenshot of them on the web page according to the coordinates of the candidate elements and candidate anchor elements; Step S2: Convert the web page screenshot into a feature map through a feature extraction network; the feature extraction network uses a convolutional neural network (CNN) model, such as ResNet, VGG, and MobileNet. This network converts the image data into a feature map, which carries spatial information and semantic information; generate detection boxes in the feature map through a region proposal network based on the coordinates of the target element and the candidate anchor element; the region proposal network (RPN) is a component for detecting specific objects and can generate potential candidate boxes (i.e., detection boxes). The candidate boxes contain the target or background. Subsequently, the RPN will screen and classify each candidate box. Region proposal methods include the sliding window-based framework, Anchor-based methods (such as YOLO and SSD), and Anchor-free methods (such as FCOS); perform bounding box regression on the detection boxes to adjust the boundary positions; the bounding box regression module calculates the exact boundary position of the object through regression and fine-tunes the position and size of the detection box; determine whether there is a target inside the detection box through the classification layer. If there is a target, distinguish the object category inside the detection box; remove duplicate and overlapping detection boxes in the detection boxes through non-maximum suppression. For example, when the IoU (Intersection over Union) of two detection boxes exceeds the threshold of 0.5, keep the detection box with a higher confidence level; keep the detection box that is most likely to be a real object and eliminate overlapping low-score detection boxes to improve the accuracy of the detection results; Step S3: Input the coordinates, text information, region screenshots, and element categories of the candidate elements and candidate anchor elements into the multi-modal model, and use the multi-modal model for vector transformation, vector alignment, element discrimination, element attention, and element judgment to detect and determine the target anchor element.

[0023] As shown in the Figure 7 and Figure 8 accompanying figures, the detection process of the multi-modal model is carried out according to the following steps: Step S3.1: Combine the text information of the candidate elements and candidate anchor elements with their corresponding categories to form a text input, use the region screenshots of the candidate elements and candidate anchor elements as the image input, and extract coordinate features from the coordinates of the candidate elements and candidate anchor elements as the coordinate input. The coordinate features include distance, angle, and margin, which are calculated based on the coordinate differences between each candidate anchor element and the candidate element; Step S3.2: Use the embedding transformation models (BERT model and ViT model) to transform the text input and image input into text vectors and image vectors, and then align the text vectors, image vectors, and coordinate feature vectors through a fully connected layer; Step S3.3: Convert the element coordinates into angles and radii in the polar coordinate system through rotational position encoding, and generate a position feature vector by combining sine function encoding to distinguish the spatial relationship between the candidate element and the candidate anchor element. Subsequently, through the self-attention module, the text vector, image vector, and coordinate feature vector of the candidate element are made to mutually attend to the text vector, image vector, and coordinate feature vector between the candidate anchor elements. The weights of each element are obtained through similarity, and the vectors are weighted and summed; The self-attention module is a mechanism for dynamically attending to different parts of the input sequence, used to capture the associations between elements in sequence data (such as text vectors, image vectors, and coordinate feature vectors), assign different weights to each element in the sequence, so that the model can automatically focus on important information and ignore irrelevant information; Step S3.4: The output of the self-attention module passes through a fully connected layer to generate a one-dimensional vector of the candidate anchor that carries an information vector representing the target element and a key vector representing the candidate anchor element. After passing through the Softmax function, the scores of the information vector and the key vector are obtained. If the score of the information vector is the highest, it is judged that there is no pending anchor element, otherwise the key vector with the highest score is used as the found pending anchor element.

[0024] In summary, the present invention first obtains candidate elements through object detection and image feature matching. Traditional element repair methods do not filter the candidate range of the target element. However, the initial element type and screenshot of the target element are attributes that well depict the portrait of the target element. These attributes are used to obtain candidate elements to effectively narrow the element range. Moreover, after filtering by type and image features, the probability of repairing to incorrect elements can be reduced; The present invention adds a scaling calculation on the basis of the original vector positioning, obtains the similarity by combining the degree of regional overlap and the degree of edge alignment, compares it with the threshold to obtain the repaired target element, and increases the applicable range; The present invention includes rule repair applied to input boxes, dropdown boxes, and checkbox boxes, and model repair applied to icons and text elements. Rule repair formulates rules using the position rules of elements, calculates their distance and angle scores according to the rules, and realizes element repair when the scores reach the set threshold; Model repair searches for pending anchor elements through a multimodal model. The candidate element corresponding to the found pending anchor element and the trustworthy anchor element is the repaired target element, realizing the repair of target elements with changed positions and contents without vector repair, and having a wide repair coverage.

Claims

1. An anchor-based RPA element anchor point automatic repair method, characterized in that: Obtain candidate elements through target detection and image feature matching, and first perform vector repair on the candidate elements. If the vector repair fails, perform vector repair. The process of combining vector repair is first to obtain the coordinates of the original target element, the original anchor point element and the current anchor point element, then calculate the scaling ratio according to the coordinates of the original anchor point element and the current anchor point element, and perform scaling calculation on the vector from the current anchor point element to the original target element according to the scaling ratio to restore the area of ​​the original target element; Calculate the similarity between the original target element area and the candidate element area, and take the candidate element with the highest similarity value that reaches the set threshold as the repaired target element; The process of the off-vector repair is to perform rule repair or model repair on the candidate elements based on the trusted anchor element; the rule repair is to calculate the score according to the distance and angle between the candidate element and the trusted anchor element, and regard the candidate element that meets the specified score threshold as the repaired target element; the model repair searches for the pending anchor element for each candidate element through the multimodal model. If the pending anchor element is the same as the trusted anchor element, the candidate element corresponding to the pending anchor element is the repaired target element.

2. The anchor-based RPA element anchor point automatic repair method according to claim 1 is characterized by: The candidate elements include input boxes, drop-down boxes, check boxes, icons and text elements; the rule repair is applied to input boxes, drop-down boxes and check boxes; and the model repair is applied to icons and text elements.

3. The anchor-based RPA element anchor point automatic repair method according to claim 2 is characterized in that: For input boxes and drop-down boxes, the rule repair is to rank the candidate elements according to the relative distance and absolute distance to the trusted anchor element to obtain a comprehensive distance score, and to obtain an angle score by calculating the angle between the candidate element and the trusted anchor element; the candidate elements whose comprehensive distance score and angle score both meet the specified score threshold are regarded as the target elements after repair; For the check box, the rule repair obtains the absolute distance score of the candidate element based on the absolute distance from the trusted anchor element, and the candidate element whose absolute distance score meets the specified score threshold is regarded as the repaired target element.

4. The anchor-based RPA element anchor point automatic repair method according to claim 3 is characterized by: The comprehensive distance score is calculated as follows: Comprehensive distance score = relative distance ranking score × absolute distance score; Among them, the relative distance ranking score = 1-(relative distance ranking between candidate element and trusted anchor element / 10); Absolute distance score = min (1, absolute distance between the original target element and the trusted anchor element / absolute distance between the candidate element and the trusted anchor element); The angle score is calculated as follows: Angle score = 1-((vector angle between the candidate element and the trusted anchor element-minimum angle below and to the right of the candidate element) / 90).

5. The anchor-based RPA element anchor point automatic repair method according to claim 1, characterized in that: The image feature matching is performed according to the following steps: Step A1: Obtain the current interface image and the original target element screenshot image, and perform grayscale and denoising preprocessing on the two images; Step A2: Detect points in the two images that meet high recognition and high stability by using a scale-invariant feature transformation method to obtain feature points; Step A3: extracting a feature descriptor of the feature point through a scale-invariant feature transform descriptor, where the feature descriptor is used to describe the features of the surrounding area of ​​the feature point; Step A4: comparing feature descriptors of different feature points in the two images by using a K-nearest neighbor algorithm to obtain multiple similar feature points for each feature point in the interface image; Step A5: Calculate the distance ratio between the feature descriptor of each feature point and the feature descriptor of the corresponding similar feature point, compare the distance ratio with the set threshold, and the similar feature points exceeding the threshold are considered as wrong matches, and the rest are considered as correct matches and used for the next step; Step A6: Apply the geometric relationship between the correctly matched feature points and similar feature points to the interface image by solving the transformation matrix, obtain multiple similar regions, and use the similar regions as candidate elements.

6. The anchor-based RPA element anchor point automatic repair method according to claim 1, characterized in that: The scaling ratio is the ratio of the short side length of the current anchor element to the short side length of the original anchor element; the scaling calculation is shown in the following formula: Current target element coordinates = current anchor point element coordinates + scaling ratio × (original target element coordinates - original anchor point element coordinates).

7. The anchor-based RPA element anchor point automatic repair method according to claim 1, characterized in that: The similarity is calculated by first calculating the region overlap and edge alignment of the region of the original target element and the region of the candidate element, and then obtaining the similarity by multiplying the region overlap and edge alignment. The degree of regional overlap is obtained by calculating the intersection-and-union ratio of the regions of the candidate element and the original target element; the intersection-and-union ratio is the ratio of the intersection area of ​​the candidate element and the original target element to the union area of ​​the candidate element and the original target element; The calculation of the edge alignment degree is shown in the following formula: Edge alignment degree = 1-min (minimum margins on the left side of the candidate element / length of the original target element area, minimum margins on the right side of the candidate element / length of the original target element area, minimum margins on the top side of the candidate element / width of the original target element area, minimum margins on the bottom side of the candidate element / width of the original target element area).

8. The anchor-based RPA element anchor point automatic repair method according to claim 1, characterized in that: The process of searching for the undetermined anchor element by the multimodal model is performed in the following steps: Step S1: Obtain the coordinates, text information and area screenshots of candidate elements and corresponding candidate anchor elements; Step S2: according to the region screenshot, the element categories of the candidate elements and the candidate anchor elements are obtained through the object detection model; Step S3: input the coordinates, text information, area screenshots and element categories of the candidate elements and candidate anchor elements into the multimodal model, and use the multimodal model to detect and determine the anchor element to be determined.

9. The anchor-based RPA element anchor point automatic repair method according to claim 8, characterized in that: The detection process of the multimodal model is carried out in the following steps: Step S3.1: forming a text input with the text information of the candidate element, the text information of the candidate anchor element, the category of the candidate element and the category of the candidate anchor element; taking a screenshot of the region of the candidate element and the candidate anchor element as an image input; extracting coordinate features from the coordinates of the candidate element and the candidate anchor element as a coordinate input; Step S3.2: Use the embedding conversion model to convert the text input and image input into text vectors and image vectors, and then align the text vectors, image vectors and coordinate feature vectors through a fully connected layer; Step S3.3: The candidate elements and the candidate anchor elements are distinguished by rotating the position encoding, and then the text vector, image vector and coordinate feature vector of the candidate elements are made to pay attention to the text vector, image vector and coordinate feature vector of the candidate anchor factors through the self-attention module, and the weight of each element is obtained by similarity and the vector is weighted summed; Step S3.4: Output the candidate anchor element with the highest sum value through the fully connected layer as the pending anchor element.

Images