Interface element pickup method and device, equipment, storage medium and program product

Abstract

The invention discloses an interface element pickup method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, the disclosed interface element pickup method comprises the steps that a to-be-picked interface image is acquired, and the to-be-picked interface image comprises a plurality of interface elements; dividing the plurality of interface elements into at least one interface layout block; for each interface layout block, correcting an element connected domain corresponding to each interface element in the interface layout block according to a consistency check result of the interface layout block to obtain a result element connected domain; and performing semantic recognition on each result element connected domain to obtain an element pickup result of the to-be-picked interface image. According to the method and the device, the connected domain detection results of the plurality of interface elements are subjected to block correction according to the visual features of the element connected domains of the interface elements, so that the condition of element missing detection or element false detection is avoided, and the technical problems of how to extract interface element information and improve the reliability of RPA task decision are solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and particularly relates to a method, apparatus, device, storage medium and program product for picking up interface elements. Background Art

[0002] In related technologies, an RPA (Robotic Process Automation) system can utilize the interface element information extracted from a task interface to assist in making task decisions. However, there may be cases of missed detection or misdetection of elements in the interface element information, resulting in low reliability of RPA task decisions. Summary of the Invention

[0003] The main purpose of the present application is to provide a method, apparatus, device, storage medium and program product for picking up interface elements, aiming to solve the technical problem of how to extract interface element information and improve the reliability of RPA task decisions.

[0004] To achieve the above object, the present application proposes a method for picking up interface elements, and the method for picking up interface elements includes:

[0005] Obtain an interface image to be picked up, where the interface image to be picked up includes a plurality of interface elements;

[0006] Divide the plurality of interface elements into at least one interface layout block, where the interface layout block includes a plurality of element groups, the element group includes a plurality of related interface elements, the visual similarity between the element connectivity domains corresponding to the interface elements in different element groups of the interface layout block satisfies the corresponding preset merging condition, and the visual features of the element connectivity domains corresponding to the interface elements in the element group satisfy the corresponding preset grouping condition;

[0007] For each interface layout block, correct the element connectivity domains corresponding to the interface elements in the interface layout block according to the consistency check result of the interface layout block to obtain a result element connectivity domain, where the consistency check result includes the total number consistency check result and / or the visual feature consistency check result of the plurality of element connectivity domains in different element groups;

[0008] Perform semantic recognition on each result element connectivity domain to obtain an element picking result of the interface image to be picked up.

[0009] In one embodiment, the step of dividing the plurality of interface elements into at least one interface layout block includes:

[0010] Cluster the interface elements whose connectivity domain size and connectivity domain position both satisfy their respective corresponding preset clustering conditions to obtain a plurality of element clustering clusters;

[0011] At least two interface elements whose element alignment and first boundary distance satisfy their respective preset grouping conditions are divided into the same element group, obtaining multiple element groups, where each interface element in the same element group belongs to a different element clustering cluster;

[0012] According to the number of elements and the element layout direction in each element group, the multiple element groups are merged into at least one interface layout block.

[0013] In one embodiment, the step of merging the multiple element groups into at least one interface layout block according to the number of elements and the element layout direction in each element group includes:

[0014] For any element group, if the number of elements and the element layout direction of any element group and its adjacent element group satisfy their respective preset merging conditions, then the any element group is merged with the adjacent element group to obtain at least one interface layout block, where the any element group and the adjacent element group are adjacent on the interface image to be picked up.

[0015] In one embodiment, before the step of dividing the multiple interface elements into at least one interface layout block, the interface element picking method further includes:

[0016] According to the system environment where the interface image to be picked up is located, multiple element picking methods are determined, where the multiple element picking methods include at least two of source code parsing, user interface automation, user interface element detection, and target detection;

[0017] Each element picking method is used to pick up multiple initial element connected regions of the interface image to be picked up;

[0018] The connected region picking results corresponding to the multiple element picking methods are fused to obtain the element connected regions of multiple interface elements in the interface image to be picked up.

[0019] In one embodiment, after the step of using each element picking method to pick up multiple initial element connected regions of the interface image to be picked up, the interface element picking method further includes:

[0020] For each element picking method, multiple initial element connected regions with a second boundary distance less than a preset distance threshold are fused to obtain the corresponding connected region picking result.

[0021] In one embodiment, after the step of performing semantic recognition on each result element connected region to obtain the element picking result of the interface image to be picked up, the interface element picking method further includes:

[0022] According to the element picking result, a task decision for robotic process automation is made.

[0023] In addition, to achieve the above object, the present application further provides an interface element picking device, which includes:

[0024] An acquisition module, configured to acquire an interface image to be picked, where the interface image to be picked includes a plurality of interface elements;

[0025] A grouping module, configured to divide the plurality of interface elements into at least one interface layout block, where the interface layout block includes a plurality of element groups, the element group includes a plurality of related interface elements, and the visual similarity between the element connectivity domains corresponding to the interface elements in different element groups of the interface layout block satisfies the corresponding preset merging condition, and the visual features of the element connectivity domains corresponding to the interface elements in the element group satisfy the corresponding preset grouping condition;

[0026] An inspection module, configured to, for each interface layout block, correct the element connectivity domains corresponding to the interface elements in the interface layout block according to the consistency inspection result of the interface layout block, to obtain a result element connectivity domain, where the consistency inspection result includes the total number consistency inspection result and / or the visual feature consistency inspection result of the plurality of element connectivity domains in different element groups;

[0027] A semantic recognition module, configured to perform semantic recognition on each result element connectivity domain to obtain an element picking result of the interface image to be picked.

[0028] In addition, to achieve the above object, the present application further provides an interface element picking device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the computer program is configured to implement the steps of the interface element picking method as described above.

[0029] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the interface element picking method as described above are implemented.

[0030] In addition, to achieve the above object, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the interface element picking method as described above are implemented.

[0031] One or more technical solutions provided by the present application have at least the following technical effects:

[0032] The present application provides a method, apparatus, device, storage medium, and program product for picking up interface elements. After picking up the element connected regions of multiple interface elements in the interface image to be picked up, according to the visual features of each element connected region and the visual similarity between them, the multiple interface elements are divided into at least one interface layout block. In the same interface layout block, since the visual similarity between the element connected regions corresponding to the interface elements belonging to different element groups satisfies the preset merging condition, therefore, for the same interface layout block, a consistency check is performed on the element connected regions in each element group, and it can be determined whether there is an element missing detection or an element misdetection in each element group. Thus, according to the situation of element missing detection or element misdetection, the element connected regions of each interface element in the interface layout block can be corrected, so that the subsequent semantic recognition can be performed according to the corrected element connected region picking result, and the element picking result of the interface image to be picked up can be obtained. It realizes the block correction of the connected region detection results of multiple interface elements according to the visual features of the element connected regions of the interface elements, avoids the situation of element missing detection or element misdetection, improves the accuracy of element picking, enables the RPA system to make task decisions based on more accurate interface element information, and further improves the reliability of RPA task decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0034] To more clearly illustrate the technical solutions in the embodiments of the present application or in the related art, the following will briefly introduce the drawings required for use in the description of the embodiments or the related art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0035] Figure 1 It is a schematic flowchart of the first embodiment of the interface element picking method of the present application;

[0036] Figure 2 It is a schematic diagram of an exemplary interface element clustering result provided by the interface element picking method of the present application;

[0037] Figure 3 For Figure 2 the schematic diagram of the interface element block division result in

[0038] Figure 4 It is a schematic diagram of an exemplary interface element fusion marking result provided by the interface element picking method of the present application;

[0039] Figure 5 It is a schematic diagram of the modules of the first embodiment of the interface element picking apparatus of the present application;

[0040] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the interface element picking method in the embodiments of the present application.

[0041] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. Specific embodiments

[0042] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0043] In order to better understand the technical solutions of the present application, the following will be described in detail with reference to the accompanying drawings of the specification and specific embodiments.

[0044] The main solution of the embodiments of the present application is as follows: obtaining an interface image to be picked, where the interface image to be picked includes a plurality of related interface elements; dividing the plurality of interface elements into at least one interface layout block, where the interface layout block includes a plurality of element groups, the element group includes a plurality of interface elements, and the visual similarity between the element connectivity domains corresponding to the interface elements in different element groups of the interface layout block satisfies the corresponding preset merging condition, and the visual features of the element connectivity domains corresponding to the interface elements in the element group satisfy the corresponding preset grouping condition; for each interface layout block, correcting the element connectivity domains corresponding to the interface elements in the interface layout block according to the consistency check result of the interface layout block to obtain a result element connectivity domain, where the consistency check result includes the total number consistency check result or the visual feature consistency check result of the plurality of element connectivity domains in different element groups; performing semantic recognition on each result element connectivity domain to obtain an element picking result of the interface image to be picked.

[0045] With the gradual popularization of personal computers, a large number of electronic workflows have emerged. Among them, the logic of some electronic workflows is fixed and the repetition rate is high, and the economic benefits of manual operations are low. Therefore, the RPA (Robotic Process Automation) technology that can realize process automation across systems has emerged as the times require.

[0046] In order to enable the RPA system to find software GUI (Graphical User Interface) elements, Web (World Wide Web) elements, etc. that need to be operated during the execution of the electronic workflow, an element picking technology that can identify and locate GUI elements and Web elements is required.

[0047] In the related technology, element picking includes intrusive element picking methods: for example, using the UIA (User Interface Automation) technology and UIA2 technology of the Windows system to directly obtain the software's GUI layout and element ID (Identity Document), etc.; or, using browser plug-ins to obtain the source code and DOM (Document Object Model) information of the web page, and using Xpath (Extensible Markup Language Path Language) or CSS Selector (Cascading Style Sheets Selector) technology to locate Web elements, etc.

[0048] In addition, with the development of big models and intelligent agent technology, the environment understanding and tool calling capabilities demonstrated by agents based on big models make agents a better RPA implementation path; but in the process of implementing RPA, big models not only need the element category and location information provided by element picking technology, but also the semantic information of each element to assist in task reasoning and execution.

[0049] In summary, the intrusive element picking method relies on the API (Application Programming Interface) provided by the operating system or relies on other software intrusion technologies, such as Java software's JAB (Java Access Bridge), etc. For software that actively blocks intrusion APIs, such as security software, it is impossible to obtain interface element information normally.

[0050] In addition, with the development of the information and communication technology industry, more and more companies are using information and communication technology systems to support daily office or back-end software systems. Most information and communication technology systems do not provide system-wide functions similar to UIA and UIA2 to obtain software GUI element information, and thus do not have the ability to complete interface element picking across systems.

[0051] In addition, the interface element picking method based on the large model may miss or misdetect elements, such as nested pages in iframe components, complex tables, etc.

[0052] Therefore, the problem of how to extract interface element information and improve the reliability of RPA task decisions needs to be solved urgently.

[0053] Therefore, the present application provides a solution. After obtaining the element connected components of multiple interface elements in the interface image to be picked up, according to the visual features of each element connected component and the visual similarity between them, the multiple interface elements are divided into at least one interface layout block. In the same interface layout block, since the visual similarity between the element connected components corresponding to the interface elements belonging to different element groups meets the preset merging condition, for the same interface layout block, a consistency check is performed on the element connected components in each element group, so as to determine whether there is an element missed detection or an element misdetection in each element group. Thus, according to the situation of element missed detection or element misdetection, the element connected components of each interface element in the interface layout block can be corrected, so that the subsequent semantic recognition can be performed according to the corrected element connected component picking result, and the element picking result of the interface image to be picked up can be obtained. It realizes the block correction of the connected component detection results of multiple interface elements according to the visual features of the element connected components of the interface elements, avoids the situation of element missed detection or element misdetection, improves the accuracy of element picking, enables the RPA system to make task decisions based on more accurate interface element information, and further improves the reliability of RPA task decisions.

[0054] It should be noted that the execution subject of this embodiment can be a computing service device with functions of interface element picking, network communication, and program running, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, an interface element picking device, etc. that can implement the above functions. Hereinafter, taking the interface element picking device as an example, this embodiment and the following embodiments will be described.

[0055] Based on this, an embodiment of the present application provides an interface element picking method, referring to Figure 1 , Figure 1 is a schematic flowchart of the first embodiment of the interface element picking method of the present application, Figure 2 is a schematic diagram of an exemplary interface element clustering result provided by the interface element picking method of the present application, Figure 3 is Figure 2 a schematic diagram of the interface element block division result in Figure 4 is a schematic diagram of an exemplary interface element fusion marking result provided by the interface element picking method of the present application.

[0056] In this embodiment, the interface element picking method may include steps S100 to S400:

[0057] Step S100, obtain the interface image to be picked up.

[0058] Among them, the interface image to be picked up includes multiple interface elements.

[0059] It should be noted that the image of the interface to be picked up can be a screenshot of the task interface for element picking. The task interface can be the screen interface after the RPA system executes an instruction. Among them, extracting the interface elements (i.e., GUI elements) of the screen interface after executing the current instruction can be used to assist in making a decision on whether to execute the next instruction. Multiple interface elements can include all icon elements, text elements, button elements, etc. of the task interface.

[0060] Step S200, divide multiple interface elements into at least one interface layout block.

[0061] Among them, the interface layout block includes multiple element groups. An element group includes multiple related interface elements. The visual similarity between the element connectivity domains corresponding to the interface elements in different element groups of the interface layout block satisfies the corresponding preset merging condition, and the visual characteristics of the element connectivity domains corresponding to the interface elements in the element group satisfy the corresponding preset grouping condition.

[0062] It should be noted that an element group can include multiple related interface elements, and an interface layout block can include multiple element groups with similar visual characteristics. Among them, the visual characteristics can include shape characteristics, size characteristics, position characteristics, etc. of the element connectivity domain.

[0063] During the process of picking up interface elements, the element connectivity domains corresponding to each interface element can be marked in the image of the interface to be picked up first, and then semantic recognition is performed to determine the element attributes of the interface elements corresponding to each element connectivity domain, so as to obtain the element picking result of the image of the interface to be picked up.

[0064] In specific implementation, before performing semantic recognition, related interface elements can be marked as an element group according to whether the visual characteristics of each element connectivity domain satisfy the preset grouping condition. Then, according to whether the visual similarity between the element connectivity domains corresponding to different element groups satisfies the preset merging condition, the element groups with similar visual characteristics are marked as an interface layout block. Among them, the preset grouping condition can be that the distance between element connectivity domains satisfies the preset distance condition, and / or the layout direction of the element connectivity domains satisfies the preset layout condition. The distance between element connectivity domains can include the boundary distance or the center distance between element connectivity domains; the preset merging condition can be that the shape characteristics, size characteristics, and position characteristics of the element connectivity domains are similar.

[0065] Step S300, for each interface layout block, correct the element connectivity domains corresponding to the interface elements in the interface layout block according to the consistency check result of the interface layout block to obtain the result element connectivity domain.

[0066] Among them, the consistency check result includes the total number consistency check result and / or the visual characteristic consistency check result of multiple element connectivity domains in different element groups.

[0067] It should be noted that in the same interface layout block, the total number of interface elements in different element groups is usually the same and they have the same visual characteristics. If the total number of element connected regions in any one element group is different from that in other element groups, there is a situation of missed detection of elements in any one element group. At this time, the situation of missed detection of elements in any one element group can be corrected, the element connected regions of the missed detected elements are marked, and the result element connected regions of each interface element are obtained. If the visual characteristics of the element connected regions in any one element group are different from those in other element groups, there is a situation of misdetection of elements in any one element group. At this time, the situation of misdetection of elements in any one element group can be corrected, the element connected regions of the misdetected elements are corrected, and the result element connected regions of each interface element are obtained. If there are both situations of missed detection of elements and misdetection of elements in any one element group, at this time, the element connected regions of the missed detected elements can be marked, and the situation of misdetection of elements in any one element group can be corrected, the element connected regions of the misdetected elements are corrected, and the result element connected regions of each interface element are obtained.

[0068] It can be understood that according to the continuity principle, by checking the compositional consistency of each element group in the interface layout block, a consistency check function Consistency Check can be defined to evaluate the consistency of each layout block B, where the consistency check function can be expressed as:

[0069] Corrected Block=Consistency Check(B)

[0070] Step S400, perform semantic recognition on each result element connected region to obtain the element picking result of the interface image to be picked.

[0071] It should be noted that the element picking result can include the element attributes of multiple interface elements of the interface image to be picked. The element attributes can include at least one of basic attributes, layout attributes, style attributes, and interaction attributes, etc. The basic attributes can include at least one of element type, element ID, and element name, etc. The layout attributes can include at least one of element position, element size, and element alignment method, etc. The style attributes can include at least one of element font, element background, and element border, etc. The interaction attributes can include at least one of clickable, editable, and selectable, etc.

[0072] After obtaining the result element connected regions of each interface element in the interface image to be picked, through the semantic prediction model, perform semantic recognition on each result element connected region, and the element picking result can be obtained. Among them, the input of the semantic prediction model can include the element screenshots of each interface element.

[0073] It can be understood that after obtaining the result element connected regions of each interface element, the interface image to be picked up can be segmented into element screenshots of each interface element according to the result element connected regions and input into the semantic prediction model.

[0074] In the semantic prediction stage, the semantic prediction model first needs to classify each element screenshot to obtain the element type. Among them, the element type can include icon type and text type. Icon type elements can be elements with only icons, no text information, and do not support inputting text information. For example, the video number icon. Text type elements can include: pure text, buttons with text, menus with text, chat windows, search windows, and other elements.

[0075] After obtaining the element type, for text type elements, the semantic prediction model can use the OCR (Optical Character Recognition) function to extract the text information corresponding to the element as an element attribute of the text type element.

[0076] For icon type elements, a lightweight network (for example, mobilenetV3) is used as the backbone network for image feature extraction in the semantic prediction model, and DNN (Dynamic Neural Network) is used as the extractor for other features (for example, ID, OCR_Text, etc.). Finally, it is connected to the FC layer to output the element attributes of the element.

[0077] Thus, this embodiment provides a method for picking up interface elements, which realizes block correction of the connected region detection results of multiple interface elements according to the visual features of the result element connected regions of the interface elements, avoids the situation of element missing detection or element misdetection, improves the accuracy of element picking, enables the RPA system to make task decisions based on more accurate interface element information, and further improves the reliability of RPA task decisions.

[0078] In a feasible implementation manner, step S200 may include steps A210 to A230:

[0079] Step A210, clustering the interface elements whose connected region size and connected region position both meet their respective preset clustering conditions to obtain multiple element clustering clusters.

[0080] It should be noted that, based on the similarity principle of Gestalt theory, in this embodiment, multiple GUI elements of the image of the interface to be picked up are clustered according to spatial similarity and size similarity. GUI elements with similar connected domain sizes and similar element alignment methods are clustered into one element cluster. Among them, the element alignment method can include the center alignment method and the boundary alignment method of the element connected domain, and the element alignment method can be determined according to the position of the element connected domain. The preset clustering conditions can include that the difference in connected domain sizes is less than the preset size difference and the element alignment methods are the same.

[0081] In one example, as Figure 2 shown, multiple interface elements in this interface image can be divided into element cluster A with a purple wireframe and element cluster B with a green wireframe. Among them, the difference in connected domain sizes of multiple element connected domains in element cluster A is less than or equal to the preset size difference, and the element alignment methods between the element connected domains are the same. The difference in connected domain sizes of multiple element connected domains in element cluster B is less than or equal to the preset size difference, and the element alignment methods between the element connected domains are the same.

[0082] In this embodiment, the goal of the clustering process is to minimize the squared error within each element cluster. The clustering objective Clusters is as follows in Formula 1:

[0083]

[0084] where C is the set of element clusters, k is the number of element clusters, μ i is the center (mean value) of element cluster C i ∥e - μ i ∥ is the feature distance between interface element e and the center μ i of its element cluster.

[0085] Step A220: Divide at least two interface elements whose element alignment methods and first boundary distances to each other both meet their respective preset grouping conditions into the same element group to obtain multiple element groups.

[0086] Among them, each interface element in the same element group belongs to a different element cluster.

[0087] It should be noted that the first boundary distance can be the horizontal distance or the vertical distance in the horizontal direction between the boundaries of the element connected domains on the plane where the image of the interface to be picked up is located. The preset grouping conditions can include that the element alignment methods are the same and the first boundary distance is less than or equal to the first preset distance. Among them, both the first preset distance and the preset distance range can be preset according to the usage environment of the actual RPA system.

[0088] In Figure 2In the example shown, the central positions of the interface element A1 in the element cluster A and the interface element B1 in the element cluster B are vertically aligned, and the first boundary distance between the element connectivity domain corresponding to the interface element A1 and the element connectivity domain corresponding to the interface element B1 is less than the first preset distance. Therefore, as Figure 3 shown, the interface element A1 and the interface element B1 are classified into the same element group.

[0089] Step A230: According to the number of elements and the element layout directions in each element group, merge multiple element groups into at least one interface layout block.

[0090] It should be noted that the number of elements is the total number of interface elements in an element group, and the element layout direction can be the layout directions of the interface elements in an element group, which can include the horizontal direction and the vertical direction on the plane where the interface image to be picked up is located. Among them, the difference in the number of elements between element groups in the same interface layout block is less than or equal to the preset difference, and the element layout directions are the same. For example, as Figure 3 shown, in the interface layout block, the interface elements corresponding to the element connectivity domains of the same color block are the same element group. There are multiple element groups in this interface layout block that include the same number of interface elements, and the element layout directions are all vertical.

[0091] Thus, this embodiment provides an interface element picking method. First, cluster the interface elements according to the element size and element position, then divide the relevant interface elements into a group according to the element alignment method and the element boundary distance. Finally, according to the number of elements and the element layout directions, merge multiple element groups with high visual similarity into at least one interface layout block, realizing the association of relevant interface elements and the association of element groups with similar visual features, thereby realizing the correction of missed detection and / or misdetection of interface elements.

[0092] In a feasible implementation manner, step A230 may include: for any element group, if the number of elements and the element layout direction of any element group and its adjacent element group both meet their respective preset merging conditions, then merge any element group with the adjacent element group to obtain at least one interface layout block.

[0093] Among them, any element group and its adjacent element group are adjacent on the interface image to be picked up.

[0094] It should be noted that when merging multiple element groups, divide the element groups that are adjacent and have similar spatial arrangements into a larger element layout block according to the law of proximity in Gestalt principles. Among them, the preset merging conditions can be that the difference in the number of elements is less than or equal to the preset difference, and the element layout directions are the same.

[0095] On the image of the interface to be picked up, the feature distance between any two element groups is as shown in Formula 2 below:

[0096] Distance(C i ,C j )=min)∥e i -e j ∥∣e i ∈C i ,e j ∈C j )

[0097] Distance(C i ,C j ) is the feature distance between the i-th element group C i and the j-th element group C j , e i is an interface element in the element group C i , e j is a terminal interface element in the element group C j , ∥e i -e j ∥ is the feature distance between the interface element e i and the interface element e j .

[0098] In a specific implementation, using Formula 3, two element groups are merged. Formula 3 is:

[0099]

[0100] Threshold block is the feature distance threshold, that is, if the feature distance between two element groups is less than or equal to the feature distance threshold, the two element groups are merged.

[0101] It can be understood that when performing element group merging, for any element group, the feature distance between any element group and its adjacent element group can be determined according to whether the number of elements and the element layout direction meet the preset merging conditions. Then, it is judged whether the feature distance is less than or equal to the feature distance threshold to determine whether to merge any element group with its adjacent element group.

[0102] In Figure 3 the example shown, each interface element in the element clustering cluster A corresponds one-to-one with each interface element in the element clustering cluster B, forming multiple element groups. The number of elements in these multiple element groups is the same, and the element component directions are all vertical directions, so these multiple element groups form an interface layout block.

[0103] Accordingly, this embodiment provides an interface element picking method, which combines elements according to whether the number of elements in the element group and the element layout direction meet the preset merging conditions, and associates element groups with similar visual features, thereby realizing the correction of missed detection and / or false detection of interface elements.

[0104] In a feasible implementation manner, before step S200, the interface element picking method may further include S500 to S700:

[0105] Step S500: Determine multiple element picking methods according to the system environment where the interface image to be picked is located.

[0106] Among them, the multiple element picking methods include at least two of native picking, traditional picking, and intelligent picking.

[0107] Step S600: Use each element picking method to pick multiple initial element connected components of the interface image to be picked.

[0108] Step S700: Fuse the connected component picking results corresponding to the multiple element picking methods to obtain the element connected components of multiple interface elements in the interface image to be picked.

[0109] It should be noted that the system environment is the operating system type. Different operating system types can use different element picking methods to mark the element connected components of the interface image to be picked. Among them, native picking may include obtaining element tags from the operating system HTML or obtaining the application GUI layout and element IDs through UIA or UIA2; traditional picking is a UIED (User Interface and Experience Design) technology developed using traditional CV (Computer Vision) technology to pick GUI elements from a software GUI screenshot; intelligent picking uses object detection technology for element picking. After using the trained Yolo model to perform object detection on the interface image to be picked, the output result is the type of the element object predicted by the model and the confidence score (between 0 and 1) of the classification.

[0110] It can be understood that different element picking methods are applicable to different operating system types. In actual use, according to the operating system type where the interface image to be picked is located, multiple applicable element picking methods can be determined. Then, using different element picking methods to pick elements from the interface image to be picked, after obtaining different connected component picking results, taking the union of the different connected component picking results can obtain the element connected components of multiple interface elements in the interface image to be picked. For example Figure 4As shown, in an example of a connected domain picking result, by combining multiple element picking methods, each interface element in the interface image to be picked can be accurately detected.

[0111] In addition, it is understandable that after obtaining the element connectivity domain of multiple interface elements, efficient prior knowledge processing can be used to quickly screen out unreasonable detection results, such as icons, images, drop-down boxes and other elements with abnormal aspect ratios, and icons that are too small.

[0112] Therefore, this embodiment provides an interface element picking method, which realizes interface element picking by integrating the connected domain picking results of multiple element picking methods. It can be applied to different system environments and complete interface element picking across systems. The cost of cross-system migration is low, and a non-invasive interface element picking method is provided, which will not invade the software execution process, thereby ensuring system security.

[0113] In a feasible implementation, after step S600, the interface element picking method may further include: for each element picking method, merging multiple initial element connected domains whose second boundary distance is less than a preset distance threshold to obtain a corresponding connected domain picking result.

[0114] The second boundary distance is the boundary distance between connected domains of different initial elements.

[0115] It should be noted that since the same interface element may include multiple widgets, that is, the initial element connected domain is the connected domain corresponding to the widget, before fusing the multiple connected domain picking results, the initial element connected domains corresponding to multiple widgets of the same interface element can also be fused according to the second boundary distance between the initial element connected domains to obtain the element connected domain of the interface element.

[0116] In a specific implementation, according to the connectivity rule of the Gestalt principle, multiple initial element connected domains whose second boundary distance is less than a preset distance threshold are merged using Formula 4, where Formula 4 is:

[0117]

[0118] Connected Region (E) is the element connected region, E is the widget set, Dist (e, e') is the second boundary distance between the initial element connected regions corresponding to widget e and widget e', Threshold conn is the connectivity threshold. When the second boundary distance between the connected domains of two initial elements is less than or equal to the connectivity threshold, the corresponding two widgets are considered to be connected and belong to the same interface element.

[0119] exist Figure 2In the example shown, the interface element A2 includes two widgets, namely the icon itself and the icon interaction attributes. After detecting the initial element connected components corresponding to these two widgets, the initial element connected components corresponding to these two widgets can be merged into one element connected component according to the second boundary distance between the initial element connected components corresponding to these two widgets.

[0120] Therefore, this embodiment provides an interface element picking method, which detects the boundary distance between the connected components corresponding to each widget in the interface element based on the connectivity rule, and identifies the widgets that meet the connectivity rule as the same interface element, which can improve the accuracy of interface element picking.

[0121] In a feasible implementation manner, after step S300, the interface element picking method may further include: making a task decision for robotic process automation according to the element picking result.

[0122] It should be noted that the basic process of the dynamic programming part of the RPA system is as follows:

[0123] 1. The RPA agent generates a process that can complete the user's target task according to the requirements input by the user in natural language.

[0124] 2. If there are any inappropriate parts in the generated process, the user can modify it through chatting.

[0125] 3. After confirming that the process is correct, the RPA agent needs to execute the process and first convert the process into a list of subtasks to be completed.

[0126] 4. Select a subtask to be executed from the subtask list.

[0127] 5. Call the large model to generate an operation instruction sequence for the subtask.

[0128] 6. Execute an instruction and send the screenshot after execution to the GSUT (GUI Semantic Understanding Technology) system.

[0129] 7. The GSUT system analyzes and outputs the element list and semantic list of the screenshot.

[0130] 8. The large model decides whether to execute the next instruction according to the outputs of 6 and 7.

[0131] 9. Loop 6 - 8 until the current subtask is completed, return to 4, check whether there are still subtasks to be executed, if so, continue to loop; if not, enter the next step.

[0132] 10. Collect the execution results of all subtasks, call the large model to polish and integrate them, and then return them to the user.

[0133] It can be seen that in this embodiment, the GSUT system provides the necessary environment perception ability for the correct execution of the RPA system subtasks. Through the environment perception ability of the GSUT system, the large model of the RPA system can make correct task decisions.

[0134] It can be understood that the GSUT system can include an element picking model and an element picking model. Before executing the interface element picking method of this embodiment, the element picking model and the element picking model can also be trained first to obtain a GSUT system suitable for the current RPA system. The specific model training process is as follows: According to the actual requirements of the RPA system, training data is prepared. First, multiple software GUI screenshots in the RPA system are obtained, and each software GUI screenshot is subjected to element annotation, semantic annotation, and data augmentation processing. Then, the multiple processed software GUI screenshots are split into a training set and an evaluation set. Among them, the training set includes an element picking training sample set and a semantic prediction training sample set, and the evaluation set includes an element picking evaluation sample set and a semantic prediction evaluation sample set. During the model training process, after importing the sample set into the MaaS (Model as a Service) platform, first, a yolo training task is created, the training parameters of the element picking model are configured, a mobilenetv3 training task is created, and the training parameters of the semantic model are configured. Then, the element picking model is used to predict the element picking training sample set to obtain the sample element picking result, and the semantic prediction model is used to predict the semantic prediction training sample set to obtain the sample semantic prediction result. Finally, an evaluation task is created, the evaluation set is imported, the evaluation parameters are configured, and the evaluation indicators are obtained respectively according to the sample element picking result and the sample semantic prediction result, and the model parameters are updated.

[0135] In addition, it can be understood that after the element picking model and the element picking model are trained, these two models together form a complete GSUT system. When picking interface elements, the GSUT system inputs a GUI screenshot and can output an element list with semantic information, which is used to assist in the task decision-making process of the RPAAgent subtask execution.

[0136] Thus, this embodiment provides an interface element picking method. The corrected element picking result is input into the RPA system, so that the large model of the RPA system can implement task decision-making processing based on the more accurate element picking result, improving the accuracy of the RPA system.

[0137] This application provides an interface element picking device, as Figure 5 shown, Figure 5 is a schematic diagram of the modules of the first embodiment of the interface element picking device of this application. The interface element picking device may include:

[0138] An acquisition module 10 for acquiring an image of an interface to be picked up, where the image of the interface to be picked up includes a plurality of interface elements;

[0139] A grouping module 20 for dividing a plurality of interface elements into at least one interface layout block, where the interface layout block includes a plurality of element groups, the element group includes a plurality of interface elements, and the visual similarity between the element connected regions corresponding to the interface elements in different element groups of the interface layout block satisfies the corresponding preset merging condition, and the visual features of the element connected regions corresponding to the interface elements in the element group satisfy the corresponding preset grouping condition;

[0140] An inspection module 30 for, for each interface layout block, correcting the element connected regions corresponding to the interface elements in the interface layout block according to the consistency inspection result of the interface layout block to obtain a result element connected region, where the consistency inspection result includes the total number consistency inspection result or the visual feature consistency inspection result of a plurality of element connected regions in different element groups;

[0141] A semantic recognition module 40 for performing semantic recognition on each result element connected region to obtain an element picking result of the interface image to be picked up.

[0142] For more implementation details in the specific implementation manner of the above interface element picking device, reference can be made to the description of the specific implementation manner of the interface element picking method in the above embodiments. For the sake of brevity of the specification, it will not be repeated here.

[0143] The present application provides an interface element picking device, and the interface element picking device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the interface element picking method in the first embodiment above.

[0144] Reference is made below to Figure 6 , which shows a schematic structural diagram of an interface element picking device suitable for implementing the embodiments of the present application. The interface element picking device in the embodiments of the present application may include, but is not limited to, mobile terminals such as laptop computers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), etc., and fixed terminals such as desktop computers. Figure 6 The interface element picking device shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0145] As Figure 6As shown in the figure, the interface element picking device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the interface element picking device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the interface element picking device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows an interface element picking device having various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or had alternatively.

[0146] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from the network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are executed.

[0147] The interface element picking device provided by the present application adopts the interface element picking method in the above embodiment, and can solve the technical problem of how to extract interface element information and improve the reliability of RPA task decision-making. Compared with the related art, the beneficial effects of the interface element picking device provided by the present application are the same as those of the interface element picking method provided by the above embodiment, and other technical features in the interface element picking device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0148] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0149] As mentioned above, the above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0150] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the interface element picking method in the above embodiments.

[0151] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or combined with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0152] The above computer-readable storage medium can be included in the interface element picking device; it can also exist separately without being assembled into the interface element picking device.

[0153] The above computer-readable storage medium carries one or more programs, which, when executed by an interface element picking device, cause the interface element picking device to: obtain an interface image to be picked, where the interface image to be picked includes a plurality of relevant interface elements; divide the plurality of interface elements into at least one interface layout block, where the interface layout block includes a plurality of element groups, an element group includes a plurality of interface elements, and the visual similarity between the element connected regions corresponding to the interface elements in different element groups of the interface layout block satisfies the corresponding preset merging condition, and the visual features of the element connected regions corresponding to the interface elements in the element group satisfy the corresponding preset grouping condition; for each interface layout block, according to the consistency check result of the interface layout block, correct the element connected regions corresponding to the interface elements in the interface layout block to obtain a result element connected region, where the consistency check result includes the total number consistency check result or the visual feature consistency check result of the plurality of element connected regions in different element groups; perform semantic recognition on each result element connected region to obtain an element picking result of the interface image to be picked.

[0154] Computer program code for performing the operations of the present application may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in a different order than that marked in the accompanying drawings. For example, two consecutively represented boxes may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as combinations of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0156] The modules involved in the embodiments described in the present application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation to the unit itself in some cases.

[0157] The readable storage medium provided by the present application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above interface element picking method, and can solve the technical problem of how to extract interface element information and improve the reliability of RPA task decision-making. Compared with the related art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as those of the interface element picking method provided by the above embodiments, and will not be elaborated here.

[0158] The present application also provides a computer program product, including a computer program, and the steps of the interface element picking method as described above are implemented when the computer program is executed by a processor.

[0159] The computer program product provided by the present application can solve the technical problem of how to extract interface element information and improve the reliability of RPA task decision-making. Compared with the related art, the beneficial effects of the computer program product provided by the present application are the same as those of the interface element picking method provided by the above embodiments, and will not be elaborated here.

[0160] The above are only some embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structural transformation made by using the content of the specification and drawings of the present application under the technical concept of the present application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A method for picking up interface elements, characterized in that, The method for picking up interface elements includes: Obtaining an interface image to be picked up, where the interface image to be picked up includes multiple interface elements; Dividing the multiple interface elements into at least one interface layout block, where the interface layout block includes multiple element groups, the element group includes multiple relevant interface elements, and the visual similarity between the element connectivity domains corresponding to the interface elements in different element groups of the interface layout block satisfies the corresponding preset merging condition, and the visual features of the element connectivity domains corresponding to the interface elements in the element group satisfy the corresponding preset grouping condition; For each interface layout block, according to the consistency check result of the interface layout block, correcting the element connectivity domains corresponding to the interface elements in the interface layout block to obtain the result element connectivity domains, where the consistency check result includes the total number consistency check result and / or the visual feature consistency check result of the multiple element connectivity domains in different element groups; Performing semantic recognition on each of the result element connectivity domains to obtain the element picking result of the interface image to be picked up.

2. The interface element picking method according to claim 1, wherein The step of dividing the multiple interface elements into at least one interface layout block includes: Clustering the interface elements whose connectivity domain size and connectivity domain position both satisfy their respective corresponding preset clustering conditions to obtain multiple element clustering clusters; Dividing at least two interface elements whose element alignment method and first boundary distance between them both satisfy their respective corresponding preset grouping conditions into the same element group to obtain multiple element groups, where each of the interface elements in the same element group belongs to a different element clustering cluster; Merging the multiple element groups into at least one interface layout block according to the number of elements and the element layout direction in each element group.

3. The interface element picking method according to claim 2, wherein The step of merging the multiple element groups into at least one interface layout block according to the number of elements and the element layout direction in each element group includes: For any element group, if the number of elements and the element layout direction of the adjacent element group of the any element group both satisfy their respective corresponding preset merging conditions, then merge the any element group with the adjacent element group to obtain at least one interface layout block, where the any element group and the adjacent element group are adjacent on the interface image to be picked up.

4. The interface element picking method according to claim 1, wherein Before the step of dividing the multiple interface elements into at least one interface layout block, the method for picking up interface elements further includes: Determining multiple element picking methods according to the system environment where the interface image to be picked up is located, where the multiple element picking methods include at least two of source code parsing, user interface automation, user interface element detection, and target detection; Using each of the element picking methods to pick up multiple initial element connectivity domains of the interface image to be picked up; Fusing the connectivity domain picking results corresponding to the multiple element picking methods to obtain the element connectivity domains of the multiple interface elements in the interface image to be picked up.

5. The interface element picking method according to claim 4, wherein After the step of picking up a plurality of initial element connected components of the to-be-picked interface image by using each of the element picking methods, the interface element picking method further includes: For each of the element picking methods, fusing a plurality of initial element connected components with a second boundary distance less than a preset distance threshold to obtain a corresponding connected component picking result, where the second boundary distance is the boundary distance between different initial element connected components.

6. The interface element picking method according to claim 1, wherein After the step of performing semantic recognition on each of the result element connected components to obtain an element picking result of the to-be-picked interface image, the interface element picking method further includes: Making a task decision for robotic process automation according to the element picking result.

7. An interface element picking device, characterized in that, The interface element picking device includes: An acquisition module, configured to acquire a to-be-picked interface image, where the to-be-picked interface image includes a plurality of interface elements; A grouping module, configured to divide the plurality of interface elements into at least one interface layout block, where the interface layout block includes a plurality of element groups, the element group includes a plurality of related interface elements, and the visual similarity between the element connected components corresponding to the interface elements in different element groups of the interface layout block satisfies a corresponding preset merging condition, and the visual features of the element connected components corresponding to the interface elements in the element group satisfy a corresponding preset grouping condition; An inspection module, configured to, for each of the interface layout blocks, correct the element connected components corresponding to the interface elements in the interface layout block according to the consistency inspection result of the interface layout block to obtain result element connected components, where the consistency inspection result includes a total quantity consistency inspection result and / or a visual feature consistency inspection result of a plurality of element connected components in different element groups; A semantic recognition module, configured to perform semantic recognition on each of the result element connected components to obtain an element picking result of the to-be-picked interface image.

8. An interface element picking device, characterized in that The interface element picking device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and is configured to implement the steps of the interface element picking method according to any one of claims 1 to 6 through the computer program.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the interface element picking method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the interface element picking method according to any one of claims 1 to 6 are implemented.

Images