Method and device for identifying rpa elements, electronic equipment and storage medium

Abstract

The application discloses a kind of identification method, device, electronic equipment and storage medium of RPA element, it is related to automation control field, the target element actually needed to be identified is compared and matched with each historical element in preset element library, when matching succeeds, the identification mode of element is automatically recommended in combination with the identification mode of historical element whether it is using traditional element exploration identification mode or model identification mode. By matching with preset element library, the identification mode of suitable target element can be quickly found, and in the present application, there are two identification modes, one is the traditional preset element exploration identification mode, which can ensure the accuracy of the target element identification result, and the other is the model identification mode, which can ensure that each target element can be identified, improving the element identification probability in RPA. The element identification method in the present application can be applied to different identification scenarios, improving the robustness of the element identification method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automation control, and particularly relates to a method and device for identifying RPA elements, an electronic device and a storage medium. BACKGROUND

[0002] Currently, there are mobile phone automation controls based on traditional element positioning methods, but the element positioning methods are too single, and the UI designs of different systems or software made by different manufacturers are different, so the forms of elements are diversified. The traditional element positioning methods cannot accurately position and identify each element, and cannot meet the automation control requirements of different mobile phones or systems. SUMMARY

[0003] The main purpose of the present application is to provide a method and device for identifying RPA elements, an electronic device and a storage medium, which aims to solve the technical problem that the traditional element positioning method cannot accurately position and identify each element.

[0004] To achieve the above purpose, the present application provides a method for identifying RPA elements, which comprises the following steps:

[0005] obtaining a page image of a smart device;

[0006] determining a target element selected in the page image, and matching a target historical element same as the target element from a preset element library;

[0007] if the target historical element is matched, identifying the target element based on an identification method of the target historical element to obtain a target identification result, wherein the identification method comprises a preset element exploration identification method and a model identification method.

[0008] Optionally, after the step of matching the target historical element same as the target element from the preset element library, the method comprises the following steps:

[0009] if the target historical element is not matched, identifying the target element based on the preset element exploration identification method to obtain a first identification result, and identifying the target element based on the model identification method to obtain a second identification result;

[0010] determining the target identification result based on the first identification result and the second identification result.

[0011] Optionally, the step of identifying the target element based on the preset element exploration identification method to obtain a first identification result comprises the following steps:

[0012] obtaining a selected target position in the page image and a current UI hierarchy in which the page image is located;

[0013] traversing the current UI hierarchy to determine a pre-partition region in which the target position is located in the current UI hierarchy, and taking an element in the pre-partition region as the target element;

[0014] obtaining a first recognition result of the target element by recognizing the pre-partition region, wherein the first recognition result includes first pointing information pointing to the target element and interface element information of the target element in the current UI hierarchy.

[0015] Optionally, the model in the model recognition manner is a preset element recognition model, and the step of obtaining a second recognition result of the target element based on the model recognition manner includes:

[0016] matching, by the preset element recognition model, a target local region from the page image, the target local region having a target image similar to the target element and having a similarity greater than a preset similarity threshold;

[0017] generating second pointing information pointing to the target element based on a relative position of the target local region in the page image, and determining the second recognition result according to the second pointing information.

[0018] Optionally, the first recognition result includes interface element information and first pointing information, the second recognition result includes second pointing information, and the step of determining the target recognition result based on the first recognition result and the second recognition result includes:

[0019] determining a credibility of the first recognition result based on the interface element information;

[0020] if the credibility of the first recognition result is greater than or equal to a preset credibility threshold, taking the first pointing information in the first recognition result as the target recognition result;

[0021] if the credibility of the first recognition result is less than the preset credibility threshold, taking the second pointing information in the second recognition result as the target recognition result.

[0022] Optionally, the step of obtaining a target recognition result of the target element based on the target historical element recognition manner includes:

[0023] determine the target recognition manner of the target historical element from the preset element exploration recognition manner and the model recognition manner by comparing a first recognition parameter of the target historical element and a second recognition parameter of the target historical element, wherein the first recognition parameter is a parameter of recognizing the target historical element by the preset element exploration recognition manner, and the second recognition parameter is a parameter of recognizing the target historical element by the model recognition manner;

[0024] recognize the target element by the target recognition manner to obtain a target recognition result.

[0025] Optionally, the RPA element recognition method comprises:

[0026] obtain an actual execution result after a target control step is executed based on the second recognition result in the RPA control flow, wherein the second recognition result is a recognition result of recognizing the target element by the model recognition manner;

[0027] determine whether the actual execution result is same as an expected execution result after the target control step is executed in the RPA control flow;

[0028] if not, optimize an element recognition model corresponding to the model recognition manner according to correction information indicating that the second recognition result is wrong.

[0029] In addition, to achieve the above object, the present application also provides an RPA element recognition device, which comprises:

[0030] an acquisition module, configured to acquire a page image of an intelligent device;

[0031] a matching module, configured to determine a target element selected in the page image, and match a target historical element same as the target element from a preset element library;

[0032] a historical recognition module, configured to, if the target historical element is matched, recognize the target element based on a recognition manner of the target historical element to obtain a target recognition result, wherein the recognition manner comprises a preset element exploration recognition manner and a model recognition manner.

[0033] In addition, to achieve the above object, the present application also provides an electronic device, which comprises a memory, a processor, and an RPA element recognition program stored in the memory and capable of running on the processor, and the RPA element recognition program realizes the steps of the RPA element recognition method when executed by the processor.

[0034] In addition, to achieve the above object, the application further provides a storage medium, wherein the storage medium stores an RPA element identification program, and the RPA element identification program is executed by a processor to implement the steps of the RPA element identification method.

[0035] In the embodiment, a page image of the intelligent device is acquired, a target element selected in the page image is determined, and a target historical element same as the target element is matched from a preset element library. If the target historical element is matched, a target recognition result is obtained by recognizing the target element based on a recognition mode of the target historical element, wherein the recognition mode includes a preset element exploration recognition mode and a model recognition mode. The application compares and matches the target element actually needed to be recognized with each historical element in the preset element library, and automatically recommends an element recognition mode by combining the recognition mode of the historical element when the matching is successful, and uses the traditional element exploration recognition mode or the model recognition mode. By matching with the preset element library, the recognition mode of the appropriate target element can be quickly found, and there are two recognition modes in the application, one is the traditional preset element exploration recognition mode, which can ensure the accuracy of the target element recognition result, and the other is the model recognition mode, which can ensure that each target element can be recognized, and the element recognition probability in RPA is improved. The element recognition method in the application scheme can be applied to different recognition scenes, and the robustness of the element recognition method is improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is an electronic device structure schematic diagram of a hardware running environment related to the embodiment scheme of the application;

[0037] Figure 2 is a flowchart of the first embodiment of the RPA element identification method of the application;

[0038] Figure 3 is a flowchart of the second embodiment of the RPA element identification method of the application;

[0039] Figure 4 is a flowchart of the third embodiment of the RPA element identification method of the application;

[0040] Figure 5 is a flowchart of the fourth embodiment of the RPA element identification method of the application;

[0041] Figure 6 is a framework structure schematic diagram of the RPA element identification method of the application;

[0042] Figure 7 is a schematic diagram of the RPA element identification device of the RPA element identification method of the application.

[0043] The implementation, functional features and advantages of the present application will be further illustrated with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are merely exemplary and do not limit the application.

[0045] As shown in Figure 1 , the electronic device structure of the hardware running environment involved in the embodiment scheme of the present application is shown. Figure 1

[0046] The electronic device of the embodiment of the present application can be a server, or a smart phone, a PC, a tablet computer, a portable computer and the like electronic terminal device.

[0047] As shown in Figure 1 , the electronic device can include a processor 1001 such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 can further include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 can be a high-speed RAM memory, or a stable memory (non-volatile memory) such as a magnetic disk memory. The memory 1005 can optionally be a storage device independent of the aforementioned processor 1001.

[0048] Optionally, the electronic device can further include a camera, an RF (Radio Frequency, radio frequency) circuit, a sensor, an audio circuit, a WiFi module and the like. The terminal can also be configured with a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor and other sensors, which are not described here. Those skilled in the art can understand that the electronic device structure shown in Figure 1 does not constitute a limitation on the electronic device, and can include more or fewer components than the diagram, or combine certain components, or different component arrangements.

[0049] Those skilled in the art can understand that the electronic device structure shown in Figure 1 does not constitute a limitation on the electronic device, and can include more or fewer components than the diagram, or combine certain components, or different component arrangements.

[0050] In addition, as Figure 1 ​As shown, the memory 1005 as a computer storage medium can include an operating system, a network communication module, a user interface module, and an RPA element recognition program.

[0051] In Figure 1 In the electronic device shown, the network interface 1004 is mainly used for connecting a background server and communicating data with the background server; the user interface 1003 is mainly used for connecting a client (user end) and communicating data with the client; and the processor 1001 can be used to call the RPA element recognition program stored in the memory 1005 and perform the following operations:

[0052] Obtain a page image of the intelligent device;

[0053] Determine a target element selected in the page image, and match a target historical element same as the target element from a preset element library;

[0054] If the target historical element is matched, identify the target element based on an identification mode of the target historical element to obtain a target identification result, wherein the identification mode includes a preset element exploration identification mode and a model identification mode.

[0055] In an implementable embodiment, the processor 1001 can call the RPA element recognition program stored in the memory 1005 and further perform the following operations:

[0056] After the step of matching the target historical element same as the target element from the preset element library, the method includes:

[0057] If the target historical element is not matched, identify the target element based on the preset element exploration identification mode to obtain a first identification result, and identify the target element based on the model identification mode to obtain a second identification result;

[0058] Determine the target identification result based on the first identification result and the second identification result.

[0059] In an implementable embodiment, the processor 1001 can call the RPA element recognition program stored in the memory 1005 and further perform the following operations:

[0060] The step of identifying the target element based on the preset element exploration identification mode to obtain a first identification result includes:

[0061] Obtain a target position selected in the page image and a current UI hierarchical structure where the page image is located;

[0062] traverse the current UI hierarchy to determine a pre-partitioned region in which the target position is located in the current UI hierarchy, and take an element in the pre-partitioned region as the target element;

[0063] The first identification result of the target element is obtained by identifying the pre-partitioned region, where the first identification result includes first pointing information pointing to the target element and interface element information of the target element in the current UI hierarchy.

[0064] In an implementation, the processor 1001 can call an RPA element identification program stored in the memory 1005, and further perform the following operations:

[0065] The model in the model identification manner is a preset element identification model, and the step of obtaining the second identification result based on the model identification manner includes:

[0066] matching, by the preset element identification model, a target local region from the page image, where a target image of the target local region has a similarity greater than a preset similarity threshold to the target element;

[0067] generating second pointing information pointing to the target element based on a relative position of the target local region in the page image, and determining the second identification result based on the second pointing information.

[0068] In an implementation, the processor 1001 can call an RPA element identification program stored in the memory 1005, and further perform the following operations:

[0069] The first identification result includes interface element information and first pointing information, the second identification result includes second pointing information, and the step of determining the target identification result based on the first identification result and the second identification result includes:

[0070] determining a credibility of the first identification result based on the interface element information;

[0071] if the credibility of the first identification result is greater than or equal to a preset credibility threshold, taking the first pointing information in the first identification result as the target identification result;

[0072] if the credibility of the first identification result is less than the preset credibility threshold, taking the second pointing information in the second identification result as the target identification result.

[0073] In an implementation, the processor 1001 can call an RPA element identification program stored in the memory 1005, and further perform the following operations:

[0074] The step of identifying the target element based on the identification manner of the target historical element comprises:

[0075] The target identification manner of the target historical element is determined from the preset element exploration identification manner and the model identification manner by comparing a first identification parameter of the target historical element and a second identification parameter of the target historical element, wherein the first identification parameter is a parameter for identifying the target historical element by the preset element exploration identification manner, and the second identification parameter is a parameter for identifying the target historical element by the model identification manner.

[0076] The target element is identified by the target identification manner to obtain a target identification result.

[0077] In an implementation, the processor 1001 can invoke the identification program of the RPA element stored in the memory 1005, and further perform the following operations:

[0078] An actual execution result after a target control step is executed based on a second identification result in an RPA control flow is obtained, wherein the second identification result is an identification result of the target element identified by the model identification manner;

[0079] It is judged whether the actual execution result is same as an expected execution result after the target control step is executed in the RPA control flow;

[0080] If not, an element identification model corresponding to the model identification manner is optimized according to correction information indicating that the second identification result is wrong.

[0081] For the purpose of clearly describing the present scheme, part of the traditional identification manner and the framework adopted by the present scheme are described.

[0082] For example, in a general RPA mobile phone automation design system, the RPA mobile phone automation design system includes a service background for simultaneously connecting multiple mobile communication devices and a cross-platform operation module, and the use of the service background for simultaneously connecting multiple mobile communication devices includes the following steps: Step 1: Prepare multiple mobile communication devices and connect them to the service end of the corresponding RPA product service background; Step 2: Start the device service end corresponding to the device system; Step 3: The mobile device service end processes multiple devices, records device unique identifiers and other information, and starts multiple threads. This scheme proposes the application of mobile phone automation in RPA. Users can design the control flow of mobile phone automation through the RPA platform to achieve the desired control effect. However, this scheme emphasizes the connection control of multiple mobile phone devices and cross-platform operation, describes the steps of connecting and controlling multiple mobile phone devices, and does not have a specific positioning method for mobile phone elements. The traditional positioning method is based on the traditional Dump Hierarchy method (an interface structure acquisition method), but due to the large number and variety of mobile phone software, the traditional positioning method cannot meet the actual needs. To solve this problem, the present application proposes a RPA element recognition method that combines multiple recognition methods to recognize elements to avoid inaccurate element recognition and positioning.

[0083] In addition, the framework used in the present application mainly includes a three-layer structure, the first layer and the second layer are deployed on the PC (Personal Computer) end, and the third layer is deployed on the mobile device end. The reason for using this three-layer architecture is to deploy the real interface operation on the second layer mobile device service end, the purpose is to ensure that the mobile device can normally perform screen projection and display, and also can execute the designed automation process file, which records the specific steps of the automation process. The specific framework diagram and level description of the system are as follows Figure 6 As follows:

[0084] 1. The first layer is the designer and executor layer. The designer completes the user's mobile phone automation design process through the free combination of mobile phone components (the mobile phone components include common modules of mobile phone operations, such as Figure 6 , as shown, connect mobile phone, click, slide, input,...), and saves it as a process file. The executor executes the process file designed in the designer. The executor and the designer both call the interface of the mobile phone service end through GRPC (global Remote Procedure Call).

[0085] 2. The second layer is the mobile server layer. The mobile server is launched as a separate EXE (executable) application, listens for requests from the designer and executor, and interacts with the mobile device. The mobile server is responsible for interface implementation, mobile phone control, and functions such as listening for mouse messages and picking elements (e.g., Figure 6 The system includes functions such as obtaining image streams, connecting to the phone, obtaining element information, etc. It mainly includes interfaces for calling the device acquisition interface, connecting / disconnecting the device interface, obtaining the image stream interface, and phone control interface. Calling the image stream acquisition interface enables screen mirroring from the phone, while calling the phone control interface allows for real-time control of the phone from the PC.

[0086] 3. The third layer is the mobile device layer, where services are deployed on the mobile device. Android and HarmonyOS phones deploy the same mobile server called AndroidServer, while iOS phones deploy the iOSServer service. The mobile device service interacts with the PC-side mobile service via RPC (Remote Procedure Call), HTTP (Hypertext Transfer Protocol), and ADB (Android Debug Bridge). This layer is where the phone actually performs the operations. When it receives a command from the PC mobile server, it first parses the command, then executes it, and finally returns the result to the PC mobile server.

[0087] Reference Figure 2 The first embodiment of the RPA element identification method of this application includes:

[0088] Step S10: Obtain the page image of the smart device;

[0089] It should be noted that in the embodiment, the implementation subject of the above-mentioned RPA element recognition method can be other electronic devices such as a PC, a tablet computer, a server, and the like, and is applied to element recognition when the RPA intelligent device automation control generates a control flow. The intelligent device can be a mobile phone, and the RPA intelligent device automation control is the automation control of the mobile phone. It can replace manual processing on the mobile phone, such as recognizing manual operations on the mobile phone, generating a control flow (a process file), and then completing operations on the mobile phone through the process file. For example, a user can perform an operation process that he wants to repeat, and the RPA intelligent device automation control generation program can recognize the operation content and operation object of each step of the user on the mobile phone, and then generate a complete control flow according to the operation content and operation object of each step. The recognition of the operation object is actually the recognition of the element selected by the user on the screen when generating the control flow, so the element is the operation object.

[0090] For example, the page image of the intelligent device is obtained. The page image can be obtained through the above-mentioned three-layer architecture. The service of the mobile device and the mobile phone service of the PC complete the transmission of the page image of the mobile phone to the PC through protocols.

[0091] In step S20, the target element selected in the page image is determined, and a target historical element identical to the target element is matched from a preset element library.

[0092] It should be noted that when generating the control flow, the user can select the target element on the page image displayed on the PC to perform related operations. The target element can be an icon, a button, an input box, or the like. For example, the operation of the user is to move the mouse to select the APP on the page image. The APP icon selected by the mouse arrow is the target element.

[0093] For example, the determination process of the target element can be through image recognition. For example, the image in the smallest rectangular frame where the mouse arrow is located can be taken as the target element. Alternatively, based on the interface corresponding to the page image and the structure information of the interface, the target element selected by the user is determined. For example, the tree structure information of the current interface, i.e., the element distributed on the current interface and the position information of each element, can be directly obtained from the mobile phone. Correspondingly, the element closest to the mouse arrow is obtained as the target element.

[0094] Then, a target historical element identical to or similar to the target element is matched from the preset element library. The similarity can usually be matched through the image shape. For example, the image shapes of different APPs are usually different, so the icon shape can be used to distinguish different elements or match identical (or similar) elements.

[0095] In an implementable embodiment, the step of matching the target historical element identical to the target element from the preset element library comprises:

[0096] In step S21, the target element is compared with the historical elements in the preset element library to obtain the similarity between the historical elements and the target element.

[0097] In step S22, the historical element with a similarity greater than another preset similarity threshold is taken as the target historical element.

[0098] For example, the target element is compared with the historical elements in the preset element library to obtain the similarity between the historical elements and the target element. The similarity between the two elements can be determined based on the element attribute and the element confidence. The element attribute can be shape, size, color, coordinate, etc., and the element confidence is the confidence when the target element is determined. For any historical element, the element attribute of the historical element is compared with the element attribute of the target element to obtain an initial similarity, and the element confidence is taken as a parameter to combine the initial similarity to obtain the similarity between the element attribute of the historical element and the target element. If the similarity is greater than a first preset similarity threshold, the historical element is taken as the target historical element.

[0099] In step S30, if the target historical element is matched, the target element is recognized based on the recognition mode of the target historical element to obtain a target recognition result. The recognition mode comprises a preset element exploration recognition mode and a model recognition mode.

[0100] For example, if the target historical element can be matched from the preset element library, the target element is recognized based on the recognition mode of the target historical element to obtain a target recognition result. It should be noted that the recognition mode comprises a preset element exploration recognition mode and a model recognition mode. The preset element exploration recognition mode is a traditional recognition mode (the traditional recognition mode is to recognize the element structure of the interface, but not every recognition scene of the element can be recognized by the traditional recognition mode, for example, the element structure of the interface cannot be parsed in some scenes), and the model recognition mode is to recognize by an AI (Artificial Intelligence) image recognition model.

[0101] In addition, it should be noted that, in addition to storing the target historical element, the use information of each historical element is also recorded in the preset element library. For example, for any historical element, the number of times that the result of using each recognition mode to recognize the historical element is recorded as the final target recognition result in the preset element library, or the recognition mode based on the user's preference. Accordingly, if the number of times that the result obtained by using the preset element exploration recognition mode is used as the target recognition result is large, the preset element exploration recognition mode is used for recognition to obtain the target recognition result. If the number of times that the result obtained by using the model recognition mode is used as the target recognition result is large, the model recognition mode is used for recognition to obtain the target recognition result.

[0102] In the embodiment, the page image of the intelligent device is acquired; the selected target element in the page image is determined, and the target historical element identical to the target element is matched from the preset element library; if the target historical element is matched, the target element is recognized based on the recognition mode of the target historical element to obtain the target recognition result, wherein the recognition mode includes the preset element exploration recognition mode and the model recognition mode. By matching with the preset element library, the recognition mode of the appropriate target element can be quickly found, and in the present application, there are two recognition modes, one is the traditional preset element exploration recognition mode, which can ensure the accuracy of the target element recognition result, and the other is the model recognition mode, which can ensure that each target element can be recognized, thereby improving the element recognition probability in RPA. The element recognition method in the present application scheme can be applicable to different recognition scenes, and the robustness of the element recognition method is improved.

[0103] Reference Figure 3 Based on the first embodiment of the present application, the second embodiment of the present application is proposed. In the embodiment, the same parts as the above embodiments can be referred to the above content, which will not be described herein. After the step of matching the target historical element identical to the target element from the preset element library, the method comprises:

[0104] In step S40, if the target historical element is not matched, the target element is recognized based on the preset element exploration recognition mode to obtain a first recognition result, and the target element is recognized based on the model recognition mode to obtain a second recognition result.

[0105] For example, if no target historical element is matched, the target element is recognized by using the above two recognition modes respectively, that is, the target element is recognized based on the preset element exploration recognition mode to obtain a first recognition result, and the target element is recognized based on the model recognition mode to obtain a second recognition result. The optimal result can be selected from the first recognition result and the second recognition result as the final target recognition result.

[0106] In an implementation, the step of identifying the target element based on the preset element identification manner to obtain a first identification result comprises:

[0107] In step S411, a selected target position in the page image and a current UI layer structure in which the page image is located are obtained.

[0108] In step S412, the current UI layer structure is traversed to determine a pre-divided region in which the target position is located in the current UI layer structure, and elements in the pre-divided region are taken as the target element.

[0109] In step S413, the pre-divided region is identified to obtain a first identification result of the target element, wherein the first identification result comprises first pointing information pointing to the target element and interface element information of the target element in the current UI layer structure.

[0110] For example, a UI element exploration function is started, a selected target position in a page image, such as a position of a mouse arrow, and a current UI layer structure in which the page image is located are obtained. For example, a UI layer structure in a tree form can be directly obtained from a smart device (a mobile phone), a minimum rectangular region (a pre-divided region) in which a mouse arrow is located is located, and related information, such as name, text, and resource ID, of the minimum rectangular region in the UI layer structure is determined, which is used to point to an element in the pre-divided region, that is, first pointing information. Interface element information of the target element in the current UI layer structure can comprise element granularity (a level of the current UI layer), element confidence, and element attributes.

[0111] In an implementation, the model in the model identification manner is a preset element identification model, and the step of identifying the target element based on the model identification manner to obtain a second identification result comprises:

[0112] In step S421, a target local region with a target image similar to the target element and a similarity greater than a preset similarity threshold is matched from the page image by using the preset element identification model.

[0113] In step S422, second pointing information pointing to the target element is generated based on a relative position of the target local region in the page image, and the second identification result is determined according to the second pointing information.

[0114] Exemplarily, the model in the model recognition mode is a preset element recognition model. The preset element recognition model is pre-trained to have an identification function. The preset element recognition model is used to match a target local region from the page image, which has a similarity greater than a preset similarity threshold with the target image of the target element. Similarly, in general, different elements can be distinguished by the line patterns of the elements. Therefore, the similarity can also be determined based on the line image of the element. For example, assuming that the target element is an icon of APP1, the icon of APP1 is matched from the icons of all APPs in the page image. In generating the second pointing information in the second recognition result based on the relative position of the target local region in the page image, it can be understood that the second pointing information can be the relative coordinates of the target local region in the page image. If the target local region is not found, an error message can be output, that is, the second recognition result cannot be obtained. In this case, the first recognition result can be directly used as the target recognition result.

[0115] In addition, it should be further explained that, before matching the target local region similar or identical to the target image from the page image, the target image and the page image can be normalized, scaled and gray-scaled. Since the display parameters such as the display ratio and resolution of the PC terminal and the mobile terminal are not consistent, the target image or the page image can be scaled before image recognition. In order to be compatible with the problem that the screen size and resolution of different devices are not consistent, the target image and the page image can also be normalized to adapt to different devices, reduce the deviation of element positioning and improve the success rate of matching. The gray-scaled processing is to avoid the influence of color difference on the similarity (usually, the different elements can be distinguished based on the line patterns, so the color features can be ignored to reduce the interference).

[0116] In step S50, the target recognition result is determined based on the first recognition result and the second recognition result.

[0117] Exemplarily, in a feasible embodiment, the first recognition result includes interface element information and first pointing information, the second recognition result includes second pointing information, and the step of determining the target recognition result based on the first recognition result and the second recognition result includes:

[0118] In step S510, the credibility of the first recognition result is determined based on the interface element information.

[0119] In step S520, if the credibility of the first recognition result is greater than or equal to a preset credibility threshold, the first pointing information in the first recognition result is used as the target recognition result.

[0120] If the confidence level of the first recognition result is less than the preset confidence threshold, the second pointing information in the second recognition result is taken as the target recognition result.

[0121] For example, the interface element information can include element granularity (i.e. the level of the current UI hierarchy), element confidence, element attribute, etc. Generally, the finer the element granularity, i.e. the lower the level of the current UI hierarchy, the more target elements are in the finer nodes in the tree structure, the higher the confidence level of the first recognition result, the higher the element confidence, the more element attributes in the first recognition result, and the higher the confidence level of the first recognition result. The confidence level of the first recognition result can be determined by comprehensively considering the element granularity (i.e. the level of the current UI hierarchy), the element confidence, and the element attribute. If the confidence level of the first recognition result is greater than or equal to the preset confidence threshold, the first pointing in the first recognition result is taken as the target recognition result. Otherwise, if the confidence level of the first recognition result is less than the preset confidence threshold, the second pointing information in the second recognition result is taken as the target recognition result. It can be understood that the determination of the first recognition result mainly depends on the interface structure information obtained from the mobile terminal. However, the relevant interface structure information cannot be obtained in all scenarios, so the confidence level of the first recognition result is low, and accordingly, the second recognition result is used as the target recognition result. Thus, the recognition and positioning of each target element can be ensured.

[0122] It can be understood that the target element actually needed to be recognized is compared and matched with each historical element in the preset element library in the present application. When the matching is successful, the element recognition mode is automatically recommended, and the traditional element extraction recognition mode or the model recognition mode is used. When the matching is unsuccessful, the preset element extraction recognition mode and the model recognition mode are started simultaneously for recognition, and the better one is selected as the final element recognition result from the two recognition results. On the one hand, the appropriate recognition mode can be quickly found by matching with the preset element library. On the other hand, if no matching is found, the two modes can be used for recognition, so that each target element can be recognized, and the better one is selected as the target result from the two recognition results, so that the accuracy of the target result is ensured.

[0123] Reference Figure 4 Based on the first embodiment of the present application, the second embodiment of the present application is proposed. The same parts as the above embodiments can be referred to the above content, which will not be described here. The step of identifying the target element based on the target historical element recognition mode to obtain the target recognition result includes:

[0124] Step A10, determining the target identification mode of the target historical element from the preset element exploration identification mode and the model identification mode by comparing the first identification parameter of the target historical element and the second identification parameter of the target historical element, wherein the first identification parameter is a parameter for identifying the target historical element by the preset element exploration identification mode, and the second identification parameter is a parameter for identifying the target historical element by the model identification mode;

[0125] Step A20, identifying the target element by the target identification mode to obtain a target identification result.

[0126] For example, the first identification parameter of the target historical element is compared with the second identification parameter of the target historical element. The first identification parameter is a parameter for identifying the historical element by the preset element exploration identification mode, and the second identification parameter is a parameter for identifying the target historical element by the model identification mode. That is, the first identification parameter and the second identification parameter include the number of times of use, the identification time consumption and the selection mode when the corresponding identification mode is used. Taking the first identification parameter as an example, the first identification parameter can include the number of times of identifying the target historical element by the preset element exploration identification mode, the time consumed for completing the identification, and whether the selection mode is a user-initiated selection. These factors are considered to determine an optimal element identification mode. In addition, the user's preference is considered to be higher than the user's preference, so the factor selection mode can be set to have a higher weight. For example, if the user manually selects the model identification mode to identify the target element multiple times, there is a higher probability that the model identification mode will be selected as the target identification mode of the target historical element. In this way, the number of times of use, the identification time consumption and the selection mode of the two modes are scored, and finally the identification mode with the highest score is selected as the target identification mode. Generally, the more the number of times of use, the shorter the identification time consumption, and the higher the score, and the higher the score of the user's manual selection mode. The target element is selected as the target identification mode, so that the identification mode suitable for the target element is quickly determined. Finally, the target element is identified by the target identification mode to obtain a target identification result.

[0127] Reference Figure 5 The third embodiment of the present application is based on the second embodiment of the present application. In this embodiment, the same parts as the above embodiments can refer to the above content, which will not be described here. The identification method of the RPA element includes:

[0128] Step B10, obtaining an actual execution result after executing the target control step based on the second identification result in the RPA control flow, wherein the second identification result is an identification result of the target element identified by the model identification mode;

[0129] Step B20, judging whether the actual execution result is same as the expected execution result after executing the target control step in the RPA control flow;

[0130] Step B30, if not same, optimizing the element recognition model corresponding to the model recognition mode according to the correction information for indicating the error of the second recognition result.

[0131] It should be noted that due to the complexity of the use scene and the uncertainty of the use device, the fixed element recognition model in the model recognition mode cannot meet the recognition requirements of each scene, that is, if the element recognition model is re-optimized in the factory once the model recognition is inaccurate, it will be cumbersome and low in efficiency, so online optimization of the element recognition model is very necessary. In the embodiment, the RPA element recognition method is applied to the execution process of the control flow, that is, the corresponding automation control process is realized based on the control flow. It can be understood that, combined with the characteristics of the RPA flow design, the next step of each step in the RPA flow design and the state (execution result) of each step after execution can realize the expected effect. Based on this feature, the optimization of the element recognition model can be realized.

[0132] For example, when the control flow is executed based on the second recognition result of the model recognition mode, the target step determined based on the second recognition result is determined, and the actual execution result when the second recognition result is executed as the basis is determined. Then, it is judged whether the actual execution result and the expected execution result of the target step recorded in the control flow are same as the actual execution result, if same, it indicates that the result of the element recognition model is accurate, if not same, it indicates that the result of the element recognition model is inaccurate, and the element recognition model needs to be optimized. For example, the correction information of the error of the second recognition result is fed back to the preset element recognition model. Based on the difference between the actual execution result and the expected execution result, the model estimation loss of the preset element recognition model is calculated, and the model parameters in the preset element recognition model are optimized based on the model estimation loss, for example, the model parameters in the preset element recognition model are adjusted based on the model estimation loss by gradient descent method and back propagation to optimize the model. Specifically, the optimization process is not described here, and can also refer to the existing scheme.

[0133] In a possible implementation, after the step of optimizing the element recognition model corresponding to the model recognition mode according to the correction information for indicating the error of the second recognition result, the method further comprises:

[0134] Step B40, generating a new second recognition result based on the optimized preset element recognition model;

[0135] Step B50, based on the new second recognition result, returns to execute the step of obtaining the execution result of the target step in the control flow after execution based on the second recognition result until the execution result is the same as the expected result, or a preset optimization number of times is reached.

[0136] After the model is optimized, a new second recognition result is generated based on the optimized preset element recognition model, and based on the new second recognition result, the step of executing the execution result of the target step in the control flow after execution based on the second recognition result is returned until the execution result is the same as the expected result, or a preset optimization number of times is reached, and the optimization is ended.

[0137] It can be understood that the application optimizes the model based on the difference between the expected result and the actual result by using the characteristics of RPA process design, i.e., the execution result of each step is expected, so as to realize online optimization of the model, so that the element recognition method in the application can adapt to complex and variable recognition scenarios.

[0138] Please refer to Figure 7 In addition, the embodiment of the application further provides an RPA element recognition device 100, which comprises:

[0139] The acquisition module 10 is configured to acquire a page image of the intelligent device.

[0140] The matching module 20 is configured to determine a target element selected in the page image, and match a target historical element identical to the target element from a preset element library.

[0141] The historical recognition module 30 is configured to, if the target historical element is matched, identify the target element based on an identification mode of the target historical element to obtain a target recognition result, wherein the identification mode comprises a preset element exploration identification mode and a model identification mode.

[0142] Optionally, the RPA element recognition device 100 further comprises a comprehensive recognition module 40, and the comprehensive module 40 is configured to:

[0143] If the target historical element is not matched, the target element is identified based on the preset element exploration identification mode to obtain a first recognition result, and the target element is identified based on the model identification mode to obtain a second recognition result.

[0144] The comprehensive recognition module 40 is configured to determine the target recognition result based on the first recognition result and the second recognition result.

[0145] Optionally, the comprehensive recognition module 40 is further configured to:

[0146] obtaining a selected target position in the page image and a current UI hierarchy structure in which the page image is located;

[0147] traversing the current UI hierarchy structure to determine a pre-partition region in which the target position is located in the current UI hierarchy structure, and taking an element in the pre-partition region as the target element;

[0148] recognizing the target element in the pre-partition region to obtain a first recognition result of the target element, wherein the first recognition result includes first pointing information pointing to the target element and interface element information of the target element in the current UI hierarchy structure.

[0149] Optionally, the comprehensive recognition module 40 is further configured to:

[0150] matching, by the pre-set element recognition model, a target local region from the page image, the target local region having a target image similar to the target element and having a similarity greater than a pre-set similarity threshold;

[0151] generating second pointing information pointing to the target element based on a relative position of the target local region in the page image, and determining the second recognition result according to the second pointing information.

[0152] Optionally, the first recognition result includes interface element information and first pointing information, and the second recognition result includes second pointing information, and the comprehensive recognition module 40 is further configured to:

[0153] determining a credibility of the first recognition result based on the interface element information;

[0154] if the credibility of the first recognition result is greater than or equal to a pre-set credibility threshold, taking the first pointing information in the first recognition result as the target recognition result;

[0155] if the credibility of the first recognition result is less than the pre-set credibility threshold, taking the second pointing information in the second recognition result as the target recognition result.

[0156] Optionally, the historical recognition module 30 is further configured to:

[0157] determining a target recognition manner of the target historical element from the pre-set element exploration recognition manner and the model recognition manner by comparing a first recognition parameter of the target historical element and a second recognition parameter of the target historical element, wherein the first recognition parameter is a parameter for recognizing the target historical element by the pre-set element exploration recognition manner, and the second recognition parameter is a parameter for recognizing the target historical element by the model recognition manner;

[0158] The target element is identified by the target identification method to obtain a target identification result.

[0159] Optionally, the RPA element identification device 100 further comprises an optimization module 50, which is configured to:

[0160] obtain an actual execution result after a target control step is executed based on the second identification result in the RPA control flow, wherein the second identification result is an identification result of the target element identified by the model identification method;

[0161] determine whether the actual execution result is the same as an expected execution result after the target control step is executed in the RPA control flow;

[0162] If not, the element identification model corresponding to the model identification method is optimized according to the correction information indicating that the second identification result is incorrect.

[0163] The RPA element identification device provided by the present application adopts the RPA element identification method in the above embodiments, and aims to solve the technical problem that the conventional element positioning method cannot accurately position and identify each element. Compared with the prior art, the RPA element identification device provided by the embodiments of the present application has the same beneficial effects as the RPA element identification method provided by the above embodiments, and other technical features in the RPA element identification device are the same as the features disclosed in the above method embodiments, which will not be repeated here.

[0164] In addition, in order to achieve the above-mentioned purpose, the present application also provides an electronic device, which comprises a memory, a processor and an RPA element identification program stored in the memory and executable on the processor, and the RPA element identification program realizes the steps of the RPA element identification method as described above when executed by the processor.

[0165] The specific implementation of the electronic device of the present application is basically the same as that of the above-mentioned RPA element identification method, which will not be repeated here.

[0166] In addition, in order to achieve the above-mentioned purpose, the present application also provides an electronic device, which comprises a memory, a processor and an RPA element identification program stored in the memory and executable on the processor, and the RPA element identification program realizes the steps of the RPA element identification method as described above when executed by the processor.

[0167] The specific implementation of the electronic device of the present application is basically the same as that of the above-mentioned RPA element identification method, which will not be repeated here.

[0168] It should be noted that, in this document, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0169] The above-mentioned sequence numbers of embodiments of the present application are only for description, and do not represent advantages or disadvantages of the embodiments.

[0170] Those skilled in the art can clearly understand the above-mentioned embodiment methods from the description of the above embodiments, which can be realized by software and necessary general hardware platforms, of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) as described above, and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device) to execute the methods described in the embodiments of the present application.

[0171] The above is only a preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for identifying an RPA element, characterized in that, The method comprises: acquiring a page image of a smart device; determining a target element selected in the page image, and matching a target historical element same as the target element from a preset element library; if the target historical element is matched, identifying the target element based on an identification mode of the target historical element to obtain a target identification result, wherein the identification mode comprises a preset element exploration identification mode and a model identification mode. 2.The method of claim 1, wherein, After the step of matching the target historical element same as the target element from the preset element library, the method further comprises: if the target historical element is not matched, identifying the target element based on the preset element exploration identification mode to obtain a first identification result, and identifying the target element based on the model identification mode to obtain a second identification result; determining the target identification result based on the first identification result and the second identification result. 3.The method of claim 2, wherein, The step of identifying the target element based on the preset element exploration identification mode to obtain a first identification result comprises: acquiring a target position selected in the page image, and a current UI hierarchical structure where the page image is located; traversing the current UI hierarchical structure to determine a pre-partitioned region where the target position is located in the current UI hierarchical structure, and taking elements in the pre-partitioned region as the target element; identifying the pre-partitioned region to obtain a first identification result of the target element, wherein the first identification result comprises first pointing information pointing to the target element, and interface element information of the target element in the current UI hierarchical structure. 4.The method of claim 2, wherein, The step of identifying the target element based on the model identification mode to obtain a second identification result comprises: matching, by a preset element identification model, a target local region having a target image similar to the target element and having a similarity greater than a preset similarity threshold from the page image; generating second pointing information pointing to the target element based on a relative position of the target local region in the page image, and determining the second identification result according to the second pointing information. 5.The method of claim 2, wherein, The first identification result comprises interface element information and first pointing information, the second identification result comprises second pointing information, and the step of determining the target identification result based on the first identification result and the second identification result comprises: determining a credibility of the first identification result based on the interface element information; if the credibility of the first identification result is greater than or equal to a preset credibility threshold, taking the first pointing information in the first identification result as the target identification result; if the credibility of the first identification result is less than the preset credibility threshold, taking the second pointing information in the second identification result as the target identification result. 6.The method of claim 1, wherein, The step of identifying the target element based on the identification mode of the target historical element to obtain a target identification result comprises: determine the target identification manner of the target historical element from the preset element exploration identification manner and the model identification manner by comparing a first identification parameter of the target historical element and a second identification parameter of the target historical element, wherein the first identification parameter is a parameter of identifying the target historical element by the preset element exploration identification manner, and the second identification parameter is a parameter of identifying the target historical element by the model identification manner; identify the target element by the target identification manner to obtain a target identification result.

7. The method of claim 1 to 6, wherein The method further includes: obtain an actual execution result after a target control step is executed based on the second identification result in an RPA control flow, wherein the second identification result is an identification result of the target element identified by the model identification manner; determine whether the actual execution result is same as an expected execution result after the target control step is executed in the RPA control flow; if not, optimize an element identification model corresponding to the model identification manner according to correction information indicating that the second identification result is wrong. 8.An RPA element identification device, applied to element identification in an RPA intelligent device automatic control process of generating a control flow, the RPA element identification device comprising: an acquisition module, configured to acquire a page image of an intelligent device; a matching module, configured to determine a target element selected in the page image, and match a target historical element same as the target element from a preset element library; a historical identification module, configured to, if the target historical element is matched, identify the target element based on an identification manner of the target historical element to obtain a target identification result, wherein the identification manner comprises a preset element exploration identification manner and a model identification manner.

9. An electronic device, comprising: The electronic device comprises a memory, a processor, and an RPA element identification program stored in the memory and executable on the processor, wherein the RPA element identification program, when executed by the processor, implements the steps of the RPA element identification method according to any one of claims 1 to 7.

10. A storage medium, characterized by The storage medium stores an RPA element identification program, and the RPA element identification program, when executed by the processor, implements the steps of the RPA element identification method according to any one of claims 1 to 7.

Images