RPA process processing analysis method and computer equipment

Abstract

The present application provides an RPA process processing and analysis method and computer device, which obtains multiple associated image elements of a first target image element in an RPA element page image to be picked up, then obtains a knowledge representation of the first target image element through the first target image element and its context image element, and then integrates the knowledge representation of the first target image element with the knowledge representation of the associated image elements through the commonality measurement results between the knowledge representation of the first target image element and the knowledge representations of the multiple associated image elements to obtain an integrated knowledge representation of the first target image element, and obtains a corresponding target element picking result based on the integrated knowledge representation. By improving the semantic information contained in the knowledge representation of the first target image element based on the associated image elements, the semantic information of the first target image element in the RPA element page image to be picked up can be more accurately represented, thereby increasing the accuracy of the target element picking result obtained by the integrated knowledge representation.

Description

Technical Field

[0001] This application relates to, but is not limited to, technical fields such as RPA, data processing, and machine learning, and in particular to an RPA process processing and analysis method and computer equipment. Background Art

[0002] RPA (Robotic Process Automation) refers to technology that uses software robots or automation tools to perform routine, repetitive tasks. It simulates human behavior, automating a series of prescribed tasks to improve productivity, reduce errors, and alleviate staff burdens. RPA can be applied across various industries and sectors, such as finance, insurance, human resources, and customer service. It processes structured data and repetitive tasks, freeing up manpower and improving productivity through automation. RPA can be integrated with existing applications and systems, interacting through user interfaces or APIs to automate processes and tasks. This saves time and costs, and provides greater accuracy and consistency. However, in certain scenarios, such as remote desktops, virtual machines, and custom software, API operation targets cannot be determined. Existing technologies use various object detection technologies to detect RPA elements in these scenarios. For example, some technologies use a combination of image search, optical character detection, and template matching to detect RPA elements. However, these methods are complex, and there is a need for an efficient and simple method for detecting elements in RPA processes that guarantees accuracy. Summary of the Invention

[0003] In view of this, the embodiments of the present application at least provide an RPA process processing and analysis method and computer equipment to improve the above technical problems.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In one aspect, an embodiment of the present application provides an RPA process processing and analysis method, applied to a computer device, comprising:

[0006] Acquire multiple associated image elements of a first target image element in the RPA element page image to be picked, wherein the multiple associated image elements are used to represent various element possibilities included in the first target image element;

[0007] Acquire a knowledge representation of the first target image element through the first target image element and a context image element of the first target image element in the to-be-picked RPA element page image;

[0008] integrating the knowledge representation of the first target image element with the knowledge representations of the multiple associated image elements based on a commonality measurement result between the knowledge representation of the first target image element and the knowledge representations of the multiple associated image elements to obtain an integrated knowledge representation of the first target image element;

[0009] The target element picking result corresponding to the to-be-picked RPA element page image is obtained through the integrated knowledge representation.

[0010] In some embodiments, acquiring the knowledge representation of the first target image element by using the first target image element and a context image element of the first target image element in the to-be-picked RPA element page image includes:

[0011] Inputting the first target image element and the context image element into a knowledge representation mining network;

[0012] The knowledge representation mining network is used to perform saliency feature embedding mapping on the first target image element and the context image element to obtain a knowledge representation of the first target image element.

[0013] In some embodiments, performing salient feature embedding mapping on the first target image element and the context image element to obtain a knowledge representation of the first target image element includes:

[0014] Obtaining a first search array, a first anchor array, and a first result array of the first target image element;

[0015] Obtain a second anchor array and a second result array of the context image element;

[0016] performing a normalization operation on a multiplication result of the first search array and the first anchor array and a multiplication result of the first search array and the second anchor array to obtain a first saliency eccentricity factor of the first target image element and a second saliency eccentricity factor of the context image element with respect to the first target image element;

[0017] Summing a multiplication result of the first saliency eccentricity factor and the first result array and a multiplication result of the second saliency eccentricity factor and the second result array to obtain a knowledge representation of the first target image element;

[0018] The step of integrating the knowledge representation of the first target image element with the knowledge representations of the multiple associated image elements based on the commonality measurement result between the knowledge representation of the first target image element and the knowledge representations of the multiple associated image elements to obtain the integrated knowledge representation of the first target image element includes:

[0019] The following operations are performed through the knowledge representation mining network:

[0020] determining, based on a commonality measurement result between the knowledge representation of the first target image element and the knowledge representations of the plurality of associated image elements, a plurality of first correlation eccentricity factors between the knowledge representation of the first target image element and the knowledge representations of the plurality of associated image elements, wherein the first correlation eccentricity factors are used to represent a degree of correlation between the corresponding associated image elements and the first target image element;

[0021] The knowledge representation of the to-be-picked RPA element page image and the knowledge representations of the multiple associated image elements are integrated through a plurality of first correlation eccentricity factors to obtain an integrated knowledge representation of the first target image element.

[0022] In some embodiments, integrating the knowledge representation of the to-be-picked RPA element page image with the knowledge representations of the multiple associated image elements using multiple first correlation eccentricity factors to obtain the integrated knowledge representation of the first target image element includes:

[0023] fusing the knowledge representation of the to-be-picked RPA element page image with the knowledge representations of the multiple associated image elements using a plurality of first correlation eccentricity factors to obtain a fused knowledge representation of the first target image element;

[0024] Performing multi-head salient feature embedding mapping on the fused knowledge representation to obtain a plurality of salient feature embedding mapping arrays of the first target image element;

[0025] Combining the plurality of salient feature embedding map arrays to obtain a salient feature embedding map tensor;

[0026] A downsampling operation is performed on the salient feature embedding map tensor to obtain an integrated knowledge representation of the first target image element.

[0027] In some embodiments, the debugging process of the knowledge representation mining network includes:

[0028] Obtaining a debugging learning sample, wherein the debugging learning sample includes an RPA element page image learning sample, a target element picking result sample, and a commonality measurement result sample between the RPA element page image learning sample and the target element picking result sample;

[0029] Inputting the RPA element page image learning sample and the target element picking result sample into the knowledge representation mining network;

[0030] Extracting, through the knowledge representation mining network, an integrated knowledge representation of the target image element sample in the RPA element page image learning sample and an integrated knowledge representation of the target element picking result sample;

[0031] The network internal configuration variables of the knowledge representation mining network are optimized by the commonality measurement results between the integrated knowledge representation of the target image element sample and the integrated knowledge representation of the target element picking result sample and the loss between the commonality measurement result samples.

[0032] In some embodiments, acquiring multiple associated image elements of a first target image element in the RPA element page image to be picked includes:

[0033] traversing a target image element associated with the first target image element in an associated image element set, wherein the associated image element set stores a plurality of image elements and a plurality of associated image elements corresponding to each of the image elements;

[0034] determining a plurality of associated image elements corresponding to the target image element as a plurality of associated image elements of the first target image element;

[0035] The process of acquiring the first target image element includes:

[0036] Performing image segmentation processing on the RPA element page image to be picked up to obtain a plurality of contrasting image elements of the RPA element page image to be picked up;

[0037] When any one of the multiple contrast image elements is consistent with any one of the image elements in the associated image element set, the any one of the contrast image elements is determined as the first target image element, and the associated image element set stores multiple image elements and multiple associated image elements corresponding to each of the image elements.

[0038] In some embodiments, performing image segmentation processing on the to-be-picked RPA element page image to obtain a plurality of comparison image elements of the to-be-picked RPA element page image includes:

[0039] performing image segmentation processing on the RPA element page image to be picked using different strategies, obtaining a plurality of comparison image element sets corresponding to the different strategies, each comparison image element set including a plurality of comparison image elements in the RPA element page image to be picked, wherein different comparison image elements in the same comparison image element set have the same number of image blocks, and different comparison image element sets have different numbers of image blocks;

[0040] When any one of the multiple compared image elements is consistent with any one of the associated image element set, determining the any one of the compared image elements as the first target image element includes:

[0041] When multiple contrast image elements belonging to different contrast image element sets are respectively consistent with multiple image elements in the associated image element set, the contrast image element with the largest number of image blocks among the multiple contrast image elements belonging to different contrast image element sets is determined as the first target image element.

[0042] In some embodiments, the method for acquiring the knowledge representation of the plurality of associated image elements includes:

[0043] For any associated image element, inputting the associated image element into a knowledge representation mining network;

[0044] The knowledge representation mining network is used to perform significant feature embedding mapping on multiple image elements in any associated image element to obtain a knowledge representation of the any associated image element.

[0045] In some embodiments, performing salient feature embedding mapping on multiple image elements in any associated image element to obtain a knowledge representation of any associated image element includes:

[0046] For any image element among the multiple image elements in any associated image element, obtaining a third search array, a third anchor array, and a third result array of the any image element;

[0047] Obtaining a fourth anchor array and a fourth result array of the remaining image elements other than the any one image element in the plurality of image elements in the any one associated image element;

[0048] performing a normalization operation on a multiplication result of the third search array and the third anchor array and a multiplication result of the third search array and the fourth anchor array to obtain a third saliency eccentricity factor of the arbitrary image element and a fourth saliency eccentricity factor of the remaining image elements with respect to the arbitrary image element;

[0049] Summing a multiplication result of the third saliency eccentricity factor and the third result array and a multiplication result of the fourth saliency eccentricity factor and the fourth result array to obtain a knowledge representation of the arbitrary image element;

[0050] The knowledge representations of multiple image elements in any associated image element are integrated to obtain the knowledge representation of any associated image element.

[0051] In another embodiment, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps in the above method when executing the program.

[0052] The present application has at least the following beneficial effects: The RPA process processing and analysis method and computer device provided herein obtain multiple associated image elements of a first target image element in an RPA element page image to be picked, wherein the multiple associated image elements are used to represent the various element possibilities contained in the first target image element. A knowledge representation of the first target image element is then obtained using the first target image element and its context image element in the RPA element page image to be picked. The knowledge representation of the first target image element is then integrated with the knowledge representations of the multiple associated image elements based on a commonality measurement result between the knowledge representation of the first target image element and the knowledge representations of the multiple associated image elements to obtain an integrated knowledge representation of the first target image element. Finally, the target element picking result corresponding to the RPA element page image to be picked is obtained using the integrated knowledge representation. When mining the knowledge representation of the first target image element, the associated image elements are added to the analysis process, and the semantic information contained in the knowledge representation of the first target image element is improved based on the associated image elements. The resulting integrated knowledge representation can more accurately represent the semantic information of the first target image element in the RPA element page image to be picked, thereby increasing the accuracy of the target element picking result obtained using the integrated knowledge representation.

[0053] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0055] Figure 1 A schematic diagram of the implementation flow of an RPA process processing and analysis method provided in an embodiment of the present application.

[0056] Figure 2 A schematic diagram of the composition structure of an RPA process processing and analysis device provided in an embodiment of the present application.

[0057] Figure 3 A hardware entity diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions of this application are further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0059] In the following description, references to "some embodiments" describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or sequence of "first / second / third" may be interchanged where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing this application only and are not intended to limit this application.

[0061] This application embodiment provides an RPA process analysis method that can be executed by a processor of a computer device. The computer device may include a server, laptop, tablet, desktop computer, smart TV, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or other device with data processing capabilities.

[0062] Figure 1 A schematic diagram of an implementation flow of an RPA process processing and analysis method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method includes the following steps S110 to S140:

[0063] Step S110 : Acquire multiple associated image elements of a first target image element in the RPA element page image to be picked up, where the multiple associated image elements are used to represent various element possibilities included in the first target image element.

[0064] The RPA element page image to be picked up can be the target page when the RPA robot performs a task, such as an operating interface in a remote desktop or virtual system. When the task is executed, the operating elements contained in the target page need to be picked up for subsequent automated operations. Operating elements include buttons, input boxes, links, etc. Buttons include submit buttons, start buttons, end buttons, jump buttons, etc. Each operating element corresponds to a corresponding operation. The first target image element is an image element of the element type to be identified determined in the RPA element page image to be picked up. It has recognition difficulty. In other words, the target image element may contain multiple recognition possibilities. For example, the image element contained in the page is button A, which can have different meanings in different scenarios, such as jump, submit, end, close, etc. An associated image element represents a possible meaning of the first target image element, and multiple associated image elements represent different meanings of the first target image element. The meaning of the associated image element is clear.

[0065] Step S120 : Acquire a knowledge representation of the first target image element through the first target image element and the context image element of the first target image element in the RPA element page image to be picked up.

[0066] Because the first target image element may have different meanings in different interface environments, and the interface environment is constructed by combining the context image element of the first target image element in the RPA element page image to be picked up with the first target image element, the knowledge representation of the first target image element obtained through the first target image element itself and the context image element can reflect the semantic information of the first target image element in the RPA element page image to be picked up. The knowledge representation of the image element is the feature information of the image element extracted by the machine learning model, which is the knowledge of the image element acquired by the model and can be represented by carriers such as feature vectors, matrices, or tensors.

[0067] Step S130 , integrating the knowledge representation of the first target image element with the knowledge representations of the multiple associated image elements based on the commonality measurement results between the knowledge representation of the first target image element and the knowledge representations of the multiple associated image elements to obtain an integrated knowledge representation of the first target image element.

[0068] Because multiple associated image elements can represent multiple different meanings of the first target image element, the knowledge representation of the first target image element and the knowledge representation of multiple associated image elements are integrated through the commonality measurement results between the knowledge representation of the first target image element and the knowledge representations of multiple associated image elements (that is, the similarity measurement value between the knowledge representations, which can be reflected by similarity or matching, and the calculation method can be to calculate the vector distance, the smaller the vector distance, the greater the commonality measurement result). Additional information can be added to the knowledge representation of the first target image element, and the integrated knowledge representation of the improved first target image element can more accurately represent the semantic information of the first target image element in the image of the RPA element page to be picked up. The process of knowledge representation integration is also the process of fusion, such as vector addition, splicing or connection, etc., which is not limited in this application.

[0069] Step S140 , obtaining a target element picking result corresponding to the RPA element page image to be picked by integrating the knowledge representation.

[0070] In this application, because the integrated knowledge representation can accurately and completely represent the semantic information of the first target image element in the RPA element page image to be picked, the target element picking result obtained by the integrated knowledge representation can accurately restore the true semantics of the target image element, increasing the accuracy of element picking. The target element picking result is, for example, the type of the identified target image element, that is, the element meaning represented, such as submit, input, jump, etc.

[0071] Based on the above method provided in the embodiment of the present application, when performing knowledge representation mining on the first target image element, the associated image elements are added to the analysis process, and different associated image elements can correspond to different meanings of the first target image element. Through the degree of correlation between the knowledge representation of the associated image element and the knowledge representation of the first target image element, the knowledge representation of the associated image element is integrated with the knowledge representation of the first target image element, and the semantic information contained in the knowledge representation of the first target image element is improved, so that the integrated knowledge representation can accurately represent the semantic information of the first target image element in the image of the RPA element page to be picked, thereby increasing the accuracy of the target element picking result obtained by the integrated knowledge representation.

[0072] As another embodiment, the RPA process analysis method provided in this application may include the following steps:

[0073] Step S210: Acquire a first target image element from the RPA element page image to be picked up.

[0074] Optionally, the image of the RPA element page to be picked is subjected to image segmentation processing to obtain a plurality of contrast image elements of the RPA element page image to be picked. The image segmentation process may be to cut the target page according to a preset pixel size to obtain a plurality of image blocks. The preset pixel size may be adaptively selected according to the actual pixels of the page, and this application does not limit this. When any contrast image element among the plurality of contrast image elements is consistent with any image element in the associated image element set, the contrast image element is used as the first target image element. The associated image element set stores a plurality of image elements and a plurality of associated image elements corresponding to each image element.

[0075] The associated image element set stores multiple image elements and multiple associated image elements corresponding to each image element. In other words, multiple image elements are used as anchors (keys), and the associated image elements corresponding to each image element are stored as results (values). The corresponding associated image elements can be searched through image elements. Optionally, the associated image element set can be generated through a graph network, in which the image elements in the graph network are represented by feature vectors, and the associated image elements corresponding to the target image elements are connected to the target image elements through edges. Based on the above embodiment, when the image of the RPA element page to be picked is subjected to image segmentation processing, multiple comparison image elements can be searched in the associated image element set, so as to determine the first target image element in the RPA element page image to be picked, and the recognition speed of the first target image element is fast.

[0076] Optionally, image segmentation processing is performed on the RPA element page image to be picked up using different strategies to obtain multiple comparison image element sets corresponding to different strategies. The different strategies mentioned here are to perform image segmentation according to different pixel scales. Each comparison image element set includes multiple comparison image elements in the RPA element page image to be picked up. A comparison image element is constructed by combining at least one image block. The number of image blocks of different comparison image elements in the same comparison image element set is the same, and the number of image blocks of comparison image elements in different comparison image element sets is different. When multiple comparison image elements belonging to different comparison image element sets are respectively consistent with multiple image elements in the associated image element set, the comparison image element with the largest number of image blocks among the multiple comparison image elements belonging to different comparison image element sets is determined as the first target image element.

[0077] Step S220 : Acquire multiple associated image elements of a first target image element in the RPA element page image to be picked up, where the multiple associated image elements are used to represent various element possibilities included in the first target image element.

[0078] Optionally, the target image elements associated with the first target image element are traversed in the associated image element set. The multiple associated image elements corresponding to the target image element are determined as multiple associated image elements of the first target image element. Based on the above embodiment, multiple associated image elements of the first target image element can be efficiently determined in the associated image element set, so that an integrated knowledge representation of the first target image element can be obtained through the multiple associated image elements. Furthermore, the semantic information of the first target image element in the RPA element page image to be picked up can be more accurately represented.

[0079] Step S230 : Acquire a knowledge representation of the first target image element through the first target image element and the context image element of the first target image element in the RPA element page image to be picked up.

[0080] Optionally, the first target image element and the context image element are input into a knowledge representation mining network. The knowledge representation mining network then performs saliency feature embedding mapping on the first target image element and the context image element to obtain a knowledge representation of the first target image element. Based on the above embodiment, because the semantic information of the first target image element in the image of the RPA element page to be picked up can be reflected by combining the first target image element and the context image element, saliency feature embedding mapping can be performed on the first target image element and the context image element. This knowledge representation of the first target image element can accurately and comprehensively represent the meaning of the first target image element. Saliency feature embedding mapping is a process of embedding the target element based on an attention mechanism (e.g., an internal attention mechanism). The embedding mapping can be based on encoding by an encoder. For example, the first target image element and the context image element are separately input into the knowledge representation mining network. The knowledge representation mining network can be any feasible neural network model, such as a transformer. Based on the knowledge representation mining network, a first search array, a first anchor array, and a first result array are obtained for a first target image element. The search array is the query in the attention mechanism, and its corresponding array can be a one-dimensional array, i.e., a vector. The anchor array is the key in the attention mechanism, and the result array is the value in the attention mechanism. A second anchor array and a second result array of context image elements are obtained through the knowledge representation mining network. The multiplication results of the first search array and the first anchor array and the multiplication results of the first search array and the second anchor array are normalized to obtain a first saliency eccentricity factor of the first target image element and a second saliency eccentricity factor of the context image element with respect to the first target image element. The multiplication results of the first saliency eccentricity factor and the first result array are summed with the multiplication results of the second saliency eccentricity factor and the second result array to obtain a knowledge representation of the first target image element. Optionally, the first search array and the first anchor array are used to obtain the first saliency eccentricity factor of the first target image element, the first result array is used to represent the first target image element, and the first saliency eccentricity factor and the first result array are used to obtain a knowledge representation of the first target image element. The process of normalization is to project the multiplication results of different values ​​into a specific numerical range, such as uniformly mapping them to [0, 1]. The saliency eccentricity factor of the target image element is its corresponding attention influence, which can be reflected by assigning corresponding weights. Then, the saliency eccentricity factor is a weight that can adjust the importance of the corresponding target image element.

[0081] Step S240 , integrating the knowledge representation of the first target image element with the knowledge representations of the multiple associated image elements based on the commonality measurement results between the knowledge representation of the first target image element and the knowledge representations of the multiple associated image elements to obtain an integrated knowledge representation of the first target image element.

[0082] Optionally, a knowledge representation mining network is employed to determine multiple first correlation eccentricity factors between the knowledge representation of the first target image element and the knowledge representations of multiple associated image elements based on commonality measurements between the knowledge representation of the first target image element and the knowledge representations of the multiple associated image elements. The first correlation eccentricity factors are used to characterize the degree of correlation between the corresponding associated image elements and the first target image element. The knowledge representation of the first target image element and the knowledge representations of the multiple associated image elements are integrated using the multiple first correlation eccentricity factors to obtain an integrated knowledge representation of the first target image element. Based on the above implementation scheme, a knowledge representation mining network can be used to determine the first correlation eccentricity factor through the commonality measurement result between the knowledge representation of the first target image element and the knowledge representations of multiple associated image elements. In other words, if the commonality measurement result between the knowledge representation of the first target image element and the knowledge representation of an associated image element is large, it means that the associated image element is closer to the semantic information of the first target image element in the RPA element page image to be picked up. The numerical value of the first correlation eccentricity factor between the knowledge representation of the first target image element and the knowledge representation of the associated image element can be configured to be larger to improve the semantic information contained in the knowledge representation of the first target image element. The obtained integrated knowledge representation can more accurately represent the semantic information of the first target image element in the RPA element page image to be picked up.

[0083] The following describes the process of obtaining the first correlation eccentricity factor and the process of obtaining the integrated knowledge representation. For obtaining the first correlation eccentricity factor, optionally, the knowledge representation of the first target image element and the knowledge representations of multiple associated image elements are input into the knowledge representation mining network. When the knowledge representation mining network obtains the commonality measurement result between the knowledge representation of the first target image element and the knowledge representations of multiple associated image elements, the following formula 1 can be used:

[0084] αn=2sigmoid(2×(M1×V1+M2×V n ))-1

[0085] When determining a plurality of first correlation eccentricity factors between the knowledge representation of the first target image element and the knowledge representations of the plurality of associated image elements, the following formula 2 may be used:

[0086] W n =exp(V n) / ∑exp(V m )

[0087] Among them, αn is the commonality measurement result, V1 is the knowledge representation of the first target image element, V2 is the knowledge representation of the associated image element, and n is the number of the associated image element. M1 and M2 are parameter arrays generated when debugging the knowledge representation mining network. n is the first correlation eccentricity factor. Because the knowledge representation V1 of the first target image element has the context information of the RPA element page image to be picked up, the meaning of each associated image element in the current page can be accurately obtained based on the above calculation method.

[0088] Optionally, the first correlation eccentricity factor is positively correlated with the commonality measurement result between the knowledge representation of the first target image element and the knowledge representation of the associated image element. In other words, the greater the commonality measurement result between the knowledge representation of the first target image element and the knowledge representation of an associated image element, the greater the first correlation eccentricity factor between the knowledge representation of the first target image element and the knowledge representation of the associated image element; the greater the commonality measurement result between the knowledge representation of the first target image element and the knowledge representation of another associated image element, the smaller the first correlation eccentricity factor between the knowledge representation of the first target image element and the knowledge representation of the associated image element.

[0089] In the process of obtaining an integrated knowledge representation, optionally, a knowledge representation mining network is used to fuse the knowledge representation of the first target image element with the knowledge representations of multiple associated image elements through multiple first correlation eccentricity factors (for example, weighting is first performed, the values ​​are adjusted, and then the sum is added to complete the fusion), thereby obtaining a fused knowledge representation of the first target image element. The fused knowledge representation is subjected to multi-headed saliency feature embedding mapping through the knowledge representation mining network to obtain multiple saliency feature embedding mapping arrays of the first target image element. The knowledge representation mining network is used to obtain an integrated knowledge representation of the first target image element through multiple saliency feature embedding mapping arrays. Multi-headed saliency feature embedding mapping uses different adjustment arrays to encode the integrated knowledge representation. Multi-headed saliency feature embedding mapping can complete deeper feature information mining of the integrated knowledge representation, thereby increasing the feature representation effect of the integrated knowledge representation.

[0090] The process of obtaining the integrated knowledge representation can be further refined into the following three levels for detailed introduction: first, the method of obtaining the integrated knowledge representation of the first target image element is introduced; then, the method of obtaining the multiple salient features of the first target image element by embedding the mapping array is introduced; finally, the method of obtaining the integrated knowledge representation of the first target image element is introduced.

[0091] First, as for the method of obtaining the integrated knowledge representation of the first target image element, optionally, the following formula 3 may be used:

[0092] V f =(V1+ ) / 2

[0093] Among them, V f It is the integrated knowledge representation of the first target image element.

[0094] When obtaining an integrated knowledge representation of the first target image element using the knowledge representation of the first target image element and the knowledge representations of multiple associated image elements, this can be done in the following manner. The following description briefly introduces the structure of the knowledge representation mining network. Optionally, the first target image element and the multiple associated image elements are input into an encoder of the knowledge representation mining network. The encoder performs embedding mapping encoding on the first target image element and the multiple associated image elements to obtain a knowledge representation V1 of the first target image element and the knowledge representations of the multiple associated image elements. The knowledge representation V1 of the first target image element and the knowledge representations of the multiple associated image elements are then input into a first correlation eccentricity factor operator of the knowledge representation mining network. The first correlation eccentricity factor operator calculates multiple first correlation eccentricity factors using the above formulas 1 and 2, where the first correlation eccentricity factor operator is an attention operator. Using the knowledge representation mining network, the first correlation eccentricity factors are multiplied by the corresponding knowledge representations. The resulting result (a vector) is then loaded into a summarization mapping operator of the knowledge representation mining network. The summarization mapping operator can be a fully connected network. The integrated knowledge representation of the first target image element is calculated using the above formula 3.

[0095] Regarding the method for obtaining multiple salient feature embedding mapping arrays for the first target image element, assume that the multi-head salient feature embedding mapping is a two-head salient feature embedding mapping, such as salient head 1 and salient head 2. For each salient head, three corresponding adjustment arrays (e.g., two-dimensional matrices) Mq, Mk, and Mv are matched. For salient head 1, the integrated knowledge representation of the first target image element is multiplied by the three adjustment arrays Mq, Mk, and Mv through the knowledge representation mining network to obtain the search array, anchor array, and result array of the integrated knowledge representation. The search array is multiplied by the transpose of the anchor array through the knowledge representation mining network to obtain the product figure1. This product figure1 is normalized to obtain the saliency eccentricity factor Wx of the first target image element. The saliency eccentricity factor Wx is multiplied by the result array of the first target image element to obtain a salient feature embedding mapping array 1 for the first target image element. For saliency head 2, the integrated knowledge representation of the first target image element is multiplied by the three adjustment arrays Mq, Mk, and Mv through the knowledge representation mining network, resulting in the search array, anchor array, and result array of the integrated knowledge representation. The search array is multiplied by the transpose of the anchor array through the knowledge representation mining network to obtain the product figure2. This product figure2 is normalized to obtain the saliency eccentricity factor Wy of the first target image element. This saliency eccentricity factor Wy is multiplied by the result array of the first target image element to obtain a saliency feature embedding mapping array 2 for the first target image element.

[0096] Regarding the method for obtaining an integrated knowledge representation of the first target image element, multiple salient feature embedding mapping arrays are combined through a knowledge representation mining network to obtain a salient feature embedding mapping tensor. The knowledge representation mining network is then used to downsample the salient feature embedding mapping tensor to obtain an integrated knowledge representation of the first target image element. Assuming the multi-headed salient feature embedding mapping is the two-headed salient feature embedding mapping at the second level above, the knowledge representation mining network is used to combine, i.e., concatenate, salient feature embedding mapping array 1 and salient feature embedding mapping array 2 of the first target image element to obtain a salient feature embedding mapping tensor. The knowledge representation mining network is then used to fully connect the salient feature embedding mapping tensor, i.e., the salient feature embedding mapping tensor is multiplied by the fully connected weight vector, and then summed with the fully connected bias vector to obtain the integrated knowledge representation of the first target image element.

[0097] When obtaining the knowledge representation of multiple related image elements, the following steps can be used:

[0098] Optionally, for any associated image element, any associated image element is input into a knowledge representation mining network, and the knowledge representation mining network performs significant feature embedding mapping on multiple image elements in the associated image element to obtain a knowledge representation of the associated image element. For example, for any image element among multiple image elements in any associated image element, a third search array, a third anchor array and a third result array of the image element are obtained through the knowledge representation mining network, and a fourth anchor array and a fourth result array of the remaining image elements other than the image element among the multiple image elements in the associated image element are obtained through the knowledge representation mining network; the multiplication results of the third search array and the third anchor array and the multiplication results of the third search array and the fourth anchor array are standardized through the knowledge representation mining network to obtain a third significant eccentricity factor of the image element and a fourth significant eccentricity factor of the remaining image elements with respect to the image element; the multiplication results of the third significant eccentricity factor and the third result array and the multiplication results of the fourth significant eccentricity factor and the fourth result array are summed through the knowledge representation mining network to obtain a knowledge representation of the image element; the knowledge representation mining network integrates the knowledge representations of multiple image elements in the associated image element to obtain a knowledge representation of the associated image element.

[0099] Step S250 , obtaining a target element picking result corresponding to the RPA element page image to be picked by integrating the knowledge representation.

[0100] Optionally, when the commonality measurement result between the integrated knowledge representation of any target element picking result and the integrated knowledge representation of the first target image element meets the target commonality measurement result requirement, the target element picking result is determined as the target element picking result corresponding to the RPA element page image to be picked. The commonality measurement result meets the target commonality measurement result requirement if the commonality measurement result is not less than a preset value of the commonality measurement result. Based on the above embodiment, the target element picking result corresponding to the RPA element page image to be picked can be determined by the commonality measurement result between the integrated knowledge representation of the first target image element and the integrated knowledge representation of the target element picking result, and the target element picking result can be determined faster.

[0101] For example, including N target element picking results 1, 2, 3...N, let the integrated knowledge representation of the first target image element be V f , the integrated knowledge representations of the N target element picking results are V1, V2, V3...V n , respectively determine the integrated knowledge representation V of the first target image element fThe Euclidean distance between the integrated knowledge representation of the N target element picking results is used to obtain the commonality measurement result. If the preset value of the commonality measurement result is S1, the target element picking result whose integrated knowledge representation is greater than S1 is determined as the target element picking result corresponding to the RPA element page image to be picked.

[0102] When obtaining the integrated knowledge representation of the target element picking result, optionally, image segmentation processing is performed on the target element picking result to obtain multiple contrast image elements of the target element picking result. When any contrast image element of the multiple contrast image elements of the target element picking result is consistent with any image element in the associated image element set, any contrast image element is determined as the second target image element. The associated image element set stores multiple image elements and multiple associated image elements corresponding to the multiple image elements. Multiple associated image elements corresponding to the second target image element are obtained from the associated image element set. The knowledge representation of the second target image element is obtained through the second target image element and the context image element of the second target image element in the target element picking result. The knowledge representation of the second target image element is integrated with the knowledge representation of the multiple associated image elements corresponding to the second target image element through the commonality measurement result between the knowledge representation of the second target image element and the knowledge representations of the multiple associated image elements corresponding to the second target image element to obtain the integrated knowledge representation of the target element picking result.

[0103] In the above embodiment of the present application, since different associated image elements can correspond to different meanings of the first target image element, the knowledge representation of the associated image element is integrated with the knowledge representation of the first target image element through the correlation between the knowledge representation of the associated image element and the knowledge representation of the first target image element, and the semantic information contained in the knowledge representation of the first target image element is improved, so that the integrated knowledge representation can accurately represent the semantic information of the first target image element in the image of the RPA element page to be picked up, thereby increasing the accuracy of the target element picking result obtained by the integrated knowledge representation.

[0104] Next, we will introduce the debugging process of the knowledge representation mining network described above, which can specifically include the following steps:

[0105] Step S310: Obtain debugging learning samples.

[0106] The debugging learning examples are data used to train the debugging network, including RPA element page image learning examples, target element picking result examples, and commonality measurement result examples between the RPA element page image learning examples and the target element picking result examples.

[0107] For example, Y and N are used to represent the commonality measurement result samples between the RPA element page image learning sample and the target element picking result sample, where Y represents that the commonality measurement result between the RPA element page image learning sample and the target element picking result sample is high, or that the RPA element page image learning sample and the target element picking result sample match, and N represents that the commonality measurement result between the RPA element page image learning sample and the target element picking result sample is low, or that the RPA element page image learning sample and the target element picking result sample do not match.

[0108] Step S320: Input the RPA element page image learning sample and the target element picking result sample into the knowledge representation mining network.

[0109] Step S330 , extracting the integrated knowledge representation of the target image element sample and the integrated knowledge representation of the target element picking result sample in the RPA element page image learning sample through the knowledge representation mining network.

[0110] Optionally, based on the knowledge representation mining network, a target image element sample in the RPA element page image learning sample is obtained, based on the knowledge representation mining network, the identified target image element of the target element picking result sample is obtained, based on the knowledge representation mining network, multiple associated image elements of the target image element sample and multiple associated image elements of the index target image element are obtained, and based on the knowledge representation mining network, the integrated knowledge representation of the target image element sample and the integrated knowledge representation of the target element picking result sample are obtained.

[0111] For example, the knowledge extraction operator of the knowledge representation mining network obtains a target image element sample from an RPA element page image learning sample, obtains multiple associated image elements corresponding to the target image element sample from the associated image element set, and extracts the knowledge representation Va of the target image element sample and the knowledge representation Vr of the multiple associated image elements. The knowledge extraction operator of the knowledge representation mining network obtains an identified target image element from a target element picking result sample, obtains multiple associated image elements corresponding to the identified target image element from the associated image element set, and extracts the knowledge representation Vb of the identified target image element and the knowledge representation Vs of the multiple associated image elements. The knowledge representation Va of the target image element sample and the knowledge representation Vr of multiple associated image elements are input into the internal focus feature analysis operator of the knowledge representation mining network (i.e., the network operator based on the self-attention mechanism). The knowledge representation Va of the target image element sample and the knowledge representation Vr of multiple associated image elements are integrated through the internal focus feature analysis operator to obtain the integrated knowledge representation of the target image element sample. The knowledge representation Vb of the identified target image element and the knowledge representation Vs of multiple associated image elements are input into the internal focus feature analysis operator of the knowledge representation mining network. The knowledge representation Vb of the identified target image element and the knowledge representation Vs of multiple associated image elements are integrated through the internal focus feature analysis operator to obtain the integrated knowledge representation of the identified target image element. The integrated knowledge representation of the target image element sample and the integrated knowledge representation of the identified target image element are input into the joint mapping integration operator of the knowledge representation mining network (i.e., a network operator based on a multi-head attention mechanism that jointly performs saliency mapping on the information of multiple heads). The integrated knowledge representation of the target image element sample is subjected to multi-head saliency feature embedding mapping through the joint mapping integration operator to obtain a plurality of saliency feature embedding mapping arrays corresponding to the integrated knowledge representation of the target image element sample; the integrated knowledge representation of the identified target image element is subjected to multi-head saliency feature embedding mapping based on the joint mapping integration operator to obtain a plurality of saliency feature embedding mapping arrays corresponding to the integrated knowledge representation of the identified target image element. The multiple salient features corresponding to the integrated knowledge representation of the target image element sample are embedded in the mapping array and input into the summary decision operator of the knowledge representation mining network (for example, a fully connected network operator). The summary decision operator makes a decision (fully connected mapping) on ​​the multiple salient features corresponding to the integrated knowledge representation of the target image element sample to obtain the integrated knowledge representation of the target image element sample. The multiple salient features corresponding to the integrated knowledge representation of the identified target image element are embedded in the mapping array and input into the summary decision operator of the knowledge representation mining network. The summary decision operator makes a decision on the multiple salient features corresponding to the integrated knowledge representation of the identified target image element to obtain the integrated knowledge representation of the identified target image element. The integrated knowledge representation of the target element picking result sample is obtained by identifying the integrated knowledge representation of the target image element.

[0112] Step S340 , optimizing the network internal configuration variables of the knowledge representation mining network through the commonality measurement results between the integrated knowledge representation of the target image element sample and the integrated knowledge representation of the target element picking result sample and the loss between the commonality measurement result samples.

[0113] For example, the network internal configuration variables (including various parameters, hyperparameters, etc.) of the knowledge representation mining network are optimized based on the contrast loss function, or, in other embodiments, the optimization can also be performed based on the cross entropy loss function.

[0114] When debugging the knowledge representation mining network as above, associated image elements are added to the analysis process. Different associated image elements represent different meanings of the first target image element. The knowledge representation of the target image element sample and the semantic information contained in the identified target image element are improved based on the associated image elements. The two integrated knowledge representations obtained can more accurately represent the semantic information of the target image element sample in the RPA element page image learning sample and identify the semantic information of the target image element in the target element picking result, thereby increasing the knowledge representation mining effect of the knowledge representation mining network.

[0115] Based on the foregoing embodiments, an embodiment of the present application provides an RPA process processing and analysis device. The various units included in the device, and the various modules included in each unit, can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; during implementation, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0116] Figure 2 A schematic diagram of the structure of an RPA process processing and analysis device provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the RPA process processing and analysis device 200 includes:

[0117] The associated element acquisition module 210 is configured to acquire a plurality of associated image elements of a first target image element in the RPA element page image to be picked up, wherein the plurality of associated image elements are used to represent various element possibilities included in the first target image element;

[0118] The knowledge representation extraction module 220 is configured to obtain a knowledge representation of the first target image element through the first target image element and a context image element of the first target image element in the RPA element page image to be picked up;

[0119] a knowledge representation integration module 230 configured to integrate the knowledge representation of the first target image element with the knowledge representations of the multiple associated image elements based on a commonality measurement result between the knowledge representation of the first target image element and the knowledge representations of the multiple associated image elements to obtain an integrated knowledge representation of the first target image element;

[0120] The RPA element picking module 240 is configured to obtain a target element picking result corresponding to the to-be-picked RPA element page image through the integrated knowledge representation.

[0121] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to perform the methods described in the above method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0122] It should be noted that in the embodiments of the present application, if the above-mentioned RPA process processing and analysis method is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk. Thus, the embodiments of the present application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0123] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0124] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above method. The computer-readable storage medium may be transient or non-transient.

[0125] An embodiment of the present application provides a computer program, including computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0126] An embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, implements some or all of the steps in the above method. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In some embodiments, the computer program product is embodied as a computer storage medium. In other embodiments, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0127] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between the various embodiments, and their similarities or similarities can be referenced to each other. The descriptions of the above device, storage medium, computer program, and computer program product embodiments are similar to the descriptions of the above method embodiments and have similar beneficial effects as the method embodiments. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the description of the method embodiments of this application for understanding.

[0128] Figure 3 A hardware entity diagram of a computer device provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the hardware entity of the computer device 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and when the processor 1001 executes the program, the steps in the method of any of the above embodiments are implemented.

[0129] The memory 1002 stores computer programs that can be run on the processor. The memory 1002 is configured to store instructions and applications executable by the processor 1001. It can also cache data to be processed or processed by the processor 1001 and various modules in the computer device 1000 (for example, image data, audio data, voice communication data, and video communication data). This can be achieved through flash memory (FLASH) or random access memory (RAM).

[0130] When the processor 1001 executes the program, it implements the steps of any of the above-mentioned RPA process processing and analysis methods. The processor 1001 generally controls the overall operation of the computer device 1000.

[0131] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the RPA process processing and analysis method as described in any of the above embodiments.

[0132] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding. The above-mentioned processor can be at least one of a target application integrated circuit (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), a programmable logic device (Programmable Logic Device, PLD), a field programmable gate array (Field Programmable Gate Array, FPGA), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that realizes the above-mentioned processor function can also be other, and the embodiments of the present application are not specifically limited.

[0133] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0134] It should be understood that the “one embodiment” or “an embodiment” mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the “in one embodiment” or “in an embodiment” appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution. The order of execution of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. It should be noted that, in this article, the terms “comprises”, “includes” or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. Without further constraints, an element defined by the phrase "comprises a..." does not exclude the existence of other identical elements in the process, method, article or apparatus that includes the element.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0136] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0137] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0138] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0139] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0140] The above is only an implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. An RPA process processing and analysis method, characterized in that: Applied to a computer device, the method comprises: Acquiring multiple associated image elements of a first target image element in an RPA element page image to be picked up specifically includes: traversing a target image element associated with the first target image element in an associated image element set, wherein the associated image element set stores multiple image elements and multiple associated image elements corresponding to each of the image elements; determining the multiple associated image elements corresponding to the target image element as multiple associated image elements of the first target image element; the acquisition process of the first target image element includes: performing image segmentation processing on the RPA element page image to be picked up to obtain multiple comparison image elements of the RPA element page image to be picked up; when any comparison image element in the multiple comparison image elements is consistent with any image element in the associated image element set, determining the any comparison image element as the first target image element, wherein the associated image element set stores multiple image elements and multiple associated image elements corresponding to each of the image elements, wherein the multiple associated image elements are used to represent various element possibilities contained in the first target image element; Obtaining a knowledge representation of the first target image element through the first target image element and the context image element of the first target image element in the RPA element page image to be picked, specifically including: inputting the first target image element and the context image element into a knowledge representation mining network; performing saliency feature embedding mapping on the first target image element and the context image element through the knowledge representation mining network to obtain a knowledge representation of the first target image element, specifically, obtaining a first search array, a first anchor array, and a first result array of the first target image element; obtaining a second anchor array and a second result array of the context image element; performing a normalization operation on the multiplication result of the first search array and the first anchor array and the multiplication result of the first search array and the second anchor array to obtain a first saliency eccentricity factor of the first target image element and a second saliency eccentricity factor of the context image element with respect to the first target image element; summing the multiplication result of the first saliency eccentricity factor and the first result array and the multiplication result of the second saliency eccentricity factor and the second result array to obtain the knowledge representation of the first target image element; The knowledge representation of the first target image element and the knowledge representation of the multiple associated image elements are integrated through the commonality measurement results between the knowledge representation of the first target image element and the knowledge representation of the multiple associated image elements to obtain the integrated knowledge representation of the first target image element. Specifically, the following operations are performed through the knowledge representation mining network: a plurality of first correlation eccentricity factors between the knowledge representation of the first target image element and the knowledge representation of the multiple associated image elements are determined through the commonality measurement results between the knowledge representation of the first target image element and the knowledge representation of the multiple associated image elements, the first correlation eccentricity factors being used to characterize the correlation between the corresponding associated image elements and the first target image element; a plurality of first correlation eccentricity factors are used to characterize the correlation between the corresponding associated image elements and the first target image element; The knowledge representation of the to-be-picked RPA element page image is integrated with the knowledge representations of the multiple associated image elements to obtain an integrated knowledge representation of the first target image element. Specifically, the knowledge representation of the to-be-picked RPA element page image is fused with the knowledge representations of the multiple associated image elements through multiple first correlation eccentricity factors to obtain a fused knowledge representation of the first target image element; a multi-headed saliency feature embedding mapping is performed on the fused knowledge representation to obtain multiple saliency feature embedding mapping arrays of the first target image element; the multiple saliency feature embedding mapping arrays are combined to obtain a saliency feature embedding mapping tensor; and a downsampling operation is performed on the saliency feature embedding mapping tensor to obtain an integrated knowledge representation of the first target image element. The target element picking result corresponding to the to-be-picked RPA element page image is obtained through the integrated knowledge representation.

2. The method according to claim 1, characterized in that The debugging process of the knowledge representation mining network includes: Obtaining a debugging learning sample, wherein the debugging learning sample includes an RPA element page image learning sample, a target element picking result sample, and a commonality measurement result sample between the RPA element page image learning sample and the target element picking result sample; Inputting the RPA element page image learning sample and the target element picking result sample into the knowledge representation mining network; Extracting, through the knowledge representation mining network, an integrated knowledge representation of the target image element sample in the RPA element page image learning sample and an integrated knowledge representation of the target element picking result sample; The network internal configuration variables of the knowledge representation mining network are optimized by the commonality measurement results between the integrated knowledge representation of the target image element sample and the integrated knowledge representation of the target element picking result sample and the loss between the commonality measurement result samples.

3. The method according to claim 1, characterized in that The performing image segmentation processing on the to-be-picked RPA element page image to obtain a plurality of comparison image elements of the to-be-picked RPA element page image includes: performing image segmentation processing on the RPA element page image to be picked using different strategies, obtaining a plurality of comparison image element sets corresponding to the different strategies, each comparison image element set including a plurality of comparison image elements in the RPA element page image to be picked, wherein different comparison image elements in the same comparison image element set have the same number of image blocks, and different comparison image element sets have different numbers of image blocks; When any one of the multiple compared image elements is consistent with any one of the associated image element set, determining the any one of the compared image elements as the first target image element includes: When multiple contrast image elements belonging to different contrast image element sets are respectively consistent with multiple image elements in the associated image element set, the contrast image element with the largest number of image blocks among the multiple contrast image elements belonging to different contrast image element sets is determined as the first target image element.

4. The method according to any one of claims 1 to 3, characterized in that The method for acquiring the knowledge representation of the multiple associated image elements includes: For any associated image element, inputting the associated image element into a knowledge representation mining network; The knowledge representation mining network is used to perform significant feature embedding mapping on multiple image elements in any associated image element to obtain a knowledge representation of the any associated image element.

5. The method according to claim 4, characterized in that The performing saliency feature embedding mapping on a plurality of image elements in any associated image element to obtain a knowledge representation of the any associated image element includes: For any image element among the multiple image elements in any associated image element, obtaining a third search array, a third anchor array, and a third result array of the any image element; Obtaining a fourth anchor array and a fourth result array of the remaining image elements other than the any one image element in the plurality of image elements in the any one associated image element; performing a normalization operation on a multiplication result of the third search array and the third anchor array and a multiplication result of the third search array and the fourth anchor array to obtain a third saliency eccentricity factor of the arbitrary image element and a fourth saliency eccentricity factor of the remaining image elements with respect to the arbitrary image element; Summing a multiplication result of the third saliency eccentricity factor and the third result array and a multiplication result of the fourth saliency eccentricity factor and the fourth result array to obtain a knowledge representation of the arbitrary image element; The knowledge representations of multiple image elements in any associated image element are integrated to obtain the knowledge representation of any associated image element.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images