Control identification method, device, electronic device and storage medium

Abstract

The embodiments of the present application disclose a control identification method, device, electronic device and storage medium. The method includes: obtaining a control to be identified; inputting an image of the control to be identified into a deep learning classification module in a control identification model, and obtaining a feature vector corresponding to the control to be identified output by the deep learning classification module; inputting the feature vector into a machine learning classification module in the control identification model, and obtaining a control identification result of the control to be identified output by the machine learning classification module. Thus, by constructing a control identification model including a deep learning classification module and a machine learning classification module, in the process of identifying the control to be identified through the control identification model, the machine learning classification module can classify the control to be identified based on the feature vector output by the deep learning classification module, thereby improving the accuracy of control identification.

Description

Technical Field

[0001] The present application belongs to the field of machine learning technology, and specifically relates to a control identification method, device, electronic device and storage medium. Background Art

[0002] In the process of interaction between users and terminal devices, interactive operations are often performed through the user interface (UI), where the interactive operations are implemented based on UI controls, such as buttons, text fields, positioning bars, check boxes, zoom buttons, toggle buttons, etc. In the process of interaction, UI controls need to be identified. However, the accuracy of related control identification methods when identifying controls needs to be improved. Summary of the invention

[0003] In view of the above problems, the present application proposes a control identification method, device, electronic device and storage medium to improve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a control identification method, the method comprising: obtaining a control to be identified; inputting an image of the control to be identified into a deep learning classification module in a control identification model, and obtaining a feature vector corresponding to the control to be identified output by the deep learning classification module; inputting the feature vector into a machine learning classification module in the control identification model, and obtaining a control identification result of the control to be identified output by the machine learning classification module.

[0005] In the second aspect, an embodiment of the present application provides a control identification device, which includes: a control acquisition unit, used to acquire a control to be identified; a vector acquisition unit, used to input the image of the control to be identified into a deep learning classification module in a control identification model, and obtain a feature vector corresponding to the control to be identified output by the deep learning classification module; and a result acquisition model, used to input the feature vector into a machine learning classification module in the control identification model, and obtain a control identification result of the control to be identified output by the machine learning classification module.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, wherein the above method is executed when the program code is run.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instructions, which implement the steps of the above method when executed by a processor.

[0009] The embodiments of the present application provide a control identification method, device, electronic device and storage medium. First, the control to be identified is obtained, and then the image of the control to be identified is input into a composite control identification model including a deep learning classification module and a machine learning classification module to obtain a control identification result corresponding to the control to be identified. Thus, by constructing a control identification model including a deep learning classification module and a machine learning classification module, in the process of identifying the control to be identified by the control identification model, the machine learning classification module can classify the control to be identified based on the feature vector output by the deep learning classification module, thereby improving the accuracy of control identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram of an application scenario of a control identification method proposed in an embodiment of the present application is shown;

[0012] Figure 2 A schematic diagram of an application scenario of a control identification method proposed in an embodiment of the present application is shown;

[0013] Figure 3 A flowchart of a control identification method proposed in an embodiment of the present application is shown;

[0014] Figure 4 A schematic diagram of a process of performing control identification on a control to be identified by using a control identification model proposed in an embodiment of the present application is shown;

[0015] Figure 5 A flow chart of a control identification method proposed in another embodiment of the present application is shown;

[0016] Figure 6 A schematic diagram of a user interface control proposed in another embodiment of the present application is shown;

[0017] Figure 7 A schematic diagram of a control tree proposed in another embodiment of the present application is shown;

[0018] Figure 8A flowchart of a control identification method proposed in another embodiment of the present application is shown;

[0019] Fig. 9 A flowchart of step S330 in another embodiment of the present application is shown;

[0020] Fig.10 A flowchart of a control identification method proposed in another embodiment of the present application is shown;

[0021] Fig.11 A flowchart of step S440 in another embodiment of the present application is shown;

[0022] Fig.12 A schematic diagram of determining a control to be identified in another embodiment of the present application is shown;

[0023] Fig.13 A structural block diagram of a control identification device proposed in an embodiment of the present application is shown;

[0024] Fig.14 A structural block diagram of an electronic device or server for executing a control identification method according to an embodiment of the present application is shown in real time of the present application;

[0025] Fig.15 A storage unit in real time of the present application for storing or carrying program codes for implementing a control identification method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] In the process of interaction between the user and the terminal device, the user interface (UI) is often used for interactive operations, wherein the interactive operations can be implemented based on the UI controls in the user interface. The UI controls may include buttons, text fields, positioning bars, check boxes, zoom buttons, toggle buttons, etc. In the interactive process, in order to obtain the user's interactive intention, the electronic device needs to identify the UI controls.

[0028] However, the inventors found in their research on related control identification methods that the accuracy of related control identification methods when identifying controls needs to be improved.

[0029] Therefore, the inventors proposed the control identification method, device, electronic device and storage medium in the present application. First, the control to be identified is obtained, and then the image of the control to be identified is input into a composite control identification model including a deep learning classification module and a machine learning classification module to obtain a control identification result corresponding to the control to be identified. Thus, by constructing a control identification model including a deep learning classification module and a machine learning classification module, in the process of identifying the control to be identified by the control identification model, the machine learning classification module can classify the control to be identified based on the feature vector output by the deep learning classification module, thereby improving the accuracy of control identification.

[0030] In the embodiment of the present application, the control identification method provided can be executed by an electronic device. In this way, all steps in the control identification method provided by the embodiment of the present application can be executed by the electronic device. For example, Figure 1 As shown, the control to be identified can be acquired by the control acquisition device of the electronic device 100, and then the acquired control to be identified is transmitted to the processor, so that the processor can identify the control to be identified in real time to obtain a control identification result of the control to be identified.

[0031] Furthermore, the control identification method provided in the embodiment of the present application can also be executed by a server (cloud). Correspondingly, in this method executed by the server, the electronic device can obtain the control to be identified, and synchronously send the control to be identified to the server, and then the server can identify the control to be identified in real time to obtain the control identification result of the control to be identified.

[0032] In addition, the electronic device and the server may also perform the process in collaboration. In this manner, some steps of the control identification method provided in the embodiment of the present application are performed by the electronic device, while other steps are performed by the server.

[0033] For example, Figure 2 As shown, the electronic device 100 can execute the control identification method including: obtaining the control to be identified, and then the server 200 executes the input of the image of the control to be identified into the deep learning classification module in the control identification model, and obtains the feature vector corresponding to the control to be identified output by the deep learning classification module; inputs the feature vector into the machine learning classification module in the control identification model, and obtains the control identification result of the control to be identified output by the machine learning classification module.

[0034] It should be noted that in this method of collaborative execution by the electronic device and the server, the steps respectively executed by the electronic device and the server are not limited to the methods introduced in the above examples. In actual applications, the steps respectively executed by the electronic device and the server can be dynamically adjusted according to actual conditions.

[0035] It should be noted that the electronic device 100 can be Figure 1 and Figure 2 In addition to the smartphone shown in the figure, it can also be a car device, a wearable device, a tablet computer, a laptop computer, a smart speaker, etc. The server 120 can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers.

[0036] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0037] See also Figure 3 , a control identification method provided in an embodiment of the present application is applied to Figure 1 or Figure 2 The electronic device or server shown, the method comprising:

[0038] Step S110: Obtain the control to be identified.

[0039] As a method, the control to be identified is a control whose corresponding function cannot be determined and needs to be identified. In an embodiment of the present application, the control to be identified is a control whose corresponding control information is missing. Among them, the missing control information refers to the control information being partially missing or completely missing.

[0040] Optionally, the control to be identified may be selected from the controls included in the user interface according to a preset rule.

[0041] Among them, the preset rules may be NAF (Not Accessibility Friendly) determination rules, size ratio determination rules, control type screening rules, etc., which are not specifically limited here.

[0042] Step S120: input the image of the control to be identified into the deep learning classification module in the control recognition model, and obtain the feature vector corresponding to the control to be identified output by the deep learning classification module.

[0043] In an embodiment of the present application, the control recognition model is a composite recognition model that integrates a CNN (Convolutional Neural Network) machine learning algorithm and a deep learning algorithm. Among them, the deep learning algorithm is good at discovering high-level abstract features of images, and the machine learning algorithm is good at exploring the classification boundaries between categories. As a way, the deep learning classification module can be any image classification network, such as YOLO, MobileNet, etc., which is not specifically limited here. However, considering the operating efficiency, a small model suitable for deployment on the electronic device side can be selected.

[0044] The high-dimensional feature vector of the image of the control to be identified is extracted through the deep learning classification module in the control recognition model. The high-dimensional feature vector contains high-dimensional abstract nonlinear classification features.

[0045] Step S130: input the feature vector into the machine learning classification module in the control recognition model, and obtain the control recognition result of the control to be recognized output by the machine learning classification module.

[0046] In the embodiment of the present application, the machine learning classification module can be a machine learning network with strong nonlinear fitting ability. As a method, the machine learning classification module can be an integrated learning algorithm such as LightGBM, XGBoost, etc. based on a gradient boosting tree.

[0047] Optionally, the control recognition result is a semantic recognition result of the control to be recognized. The semantic recognition result can be used to determine the function of the control. Exemplarily, the control recognition result can be that control A is a like control. Among them, "like" can be understood as the semantic recognition result of the control to be recognized.

[0048] The schematic diagram of the process of identifying the control to be identified through the control identification model can be as follows Figure 4 The images of multiple controls to be identified are sequentially input into the deep learning module of the control identification model to obtain the feature vector corresponding to each control to be identified, and then the feature vector corresponding to each control to be identified is input into the machine learning classification module, and the output of the machine learning classification module is used as the control identification result of the control to be identified.

[0049] The present application provides a control recognition method, which first obtains a control to be recognized, and then inputs the image of the control to be recognized into a composite control recognition model including a deep learning classification module and a machine learning classification module to obtain a control recognition result corresponding to the control to be recognized. Thus, by constructing a control recognition model including a deep learning classification module and a machine learning classification module, in the process of recognizing the control to be recognized by the control recognition model, the machine learning classification module can classify the control to be recognized based on the high-dimensional feature vector output by the deep learning classification module, thereby improving the accuracy of control recognition.

[0050] See also Figure 5 , a control identification method provided in an embodiment of the present application is applied to Figure 1 or Figure 2 The electronic device or server shown, the method comprising:

[0051] Step S210: Obtain the control tree corresponding to the current user interface.

[0052] In the VGUI (Voice Graphic User Interface) interaction scenario, the user can be provided with a voice control service to operate the UI (User Interface) in combination with the information on the UI. The information on the UI can be understood as UI controls. The user can send corresponding voice control instructions based on the UI controls displayed on the user interface of the electronic device, and then the electronic device can switch the user interface to the interface corresponding to the voice control instruction for display after receiving the voice control instruction. For example, the current user interface includes a Bluetooth switch. As long as the user says "turn on the Bluetooth switch" or "turn off the Bluetooth switch", the system will click the Bluetooth switch.

[0053] In an embodiment of the present application, the current user interface may be a user interface currently displayed by the electronic device. The current user interface may include controls that need to be identified. The control tree may be constructed based on the parent-child, grandparent-grandchild relationship between controls. Different user interfaces correspond to different control trees.

[0054] As a method, the target interface can be first identified based on code parsing to obtain the ID, type, position, size, inclusion relationship and description information of the controls included in the target interface, and then a control tree can be constructed based on the ID, type, position, size, inclusion relationship and description information of the identified controls. In the control tree, there will be multiple nodes, each of which represents a control. In addition, the control corresponding to the child node is included in the control corresponding to the parent node corresponding to the child node. It should be noted that in the control tree, child nodes and parent nodes exist relatively. If a control corresponding to a node is included in the control corresponding to a node adjacent to the node, then the node is a child node relative to the adjacent node, and correspondingly, the adjacent node is the parent node of the node.

[0055] For example, Figure 6 As shown, Figure 6 A schematic diagram of a user interface control is exemplarily described. The current user interface includes control 1, control 2, and control 3. Figure 6 The interface shown can be analyzed to obtain Figure 7 As shown in the control relationship structure diagram. Figure 7 As shown, node 2 is adjacent to node 5, and from top to bottom, the level of the control corresponding to the node is lower, the level of node 2 is higher than that of node 5, and node 5 is a child node relative to node 2, and node 2 is a parent node relative to node 5. Figure 7 In the control tree shown, the node arranged at the top is the root node, where the root node in the control tree represents the most basic control in the target interface. Among all the controls included in the interface, except for the most basic control, all other controls are included in the most basic control. Moreover, the closer the node is to the top, the closer the level of the control represented by the node is to the most basic control.

[0056] Step S220: traverse the control tree to obtain control information of all controls corresponding to the control tree.

[0057] In the embodiment of the present application, the traversal refers to visiting each node in the control tree once and only once in sequence along a certain search route.

[0058] As a way, the control tree can be traversed according to the preset traversal strategy. As one of the ways, the control tree can be traversed according to the depth-first traversal strategy. Depth-first traversal starts from the initial access node. The initial access node may have multiple adjacent nodes. The strategy of depth-first traversal is to first visit the first adjacent node, and then use this visited adjacent node as the initial node to visit its first adjacent node. That is, each time after visiting the current node, the first adjacent node of the current node is visited first. The access strategy of depth-first traversal is to prioritize vertical digging, rather than horizontally visiting all adjacent nodes of a node.

[0059] As another method, the control tree can be traversed according to the breadth-first traversal strategy. The breadth-first traversal strategy is similar to a hierarchical search process. Breadth-first traversal requires the use of a queue to maintain the order of visited nodes so that the adjacent nodes of these nodes can be visited in this order.

[0060] In the embodiment of the present application, traversing the control tree is mainly to obtain the control information of each control in the control tree. Among them, the control information may include text information, coordinate information, control type, and attribute information, etc. Among them, the attribute information is some system-defined attributes, such as border line size, border line color, etc. In general, the identification of controls is based on the attributes of the controls. For example, the appium test tool uses the uiautomatorviewer.bat tool to view the attributes of the controls. The name positioning corresponds to the text attribute of the control, the id positioning corresponds to the resource-id attribute of the control, the classname positioning corresponds to the class attribute of the control, and the xpath positioning corresponds to the attribute combination of the control. The specific information of the current control is identified by comparing these attributes.

[0061] Step S230: Screening all controls corresponding to the control tree based on the control information to obtain controls to be identified.

[0062] In an embodiment of the present application, after the control information of each control is obtained by the above method, each control information can be compared with the preset control information, and then all controls corresponding to the control tree can be screened to obtain the control to be identified. Among them, the preset control information can be pre-set control information, for example, each attribute or text information can be represented by a different field, and then the format of each field can be defined. When comparing the control information of each control with the preset control information, what is compared is whether each field in the control information is in the preset format.

[0063] Step S240: input the image of the control to be identified into the deep learning classification module in the control recognition model, and obtain the feature vector corresponding to the control to be identified output by the deep learning classification module.

[0064] Step S250: input the feature vector into the machine learning classification module in the control recognition model, and obtain the control recognition result of the control to be recognized output by the machine learning classification module.

[0065] The present application provides a control recognition method, which can reduce the images of controls to be recognized that are input into the control recognition model by traversing the control tree, obtaining the control information of all controls corresponding to the control tree, and then screening the controls based on the control information, so that the control can only recognize a small number of controls to be recognized, thereby speeding up the control recognition. Furthermore, by constructing a control recognition model including a deep learning classification module and a machine learning classification module, in the process of identifying the controls to be recognized by the control recognition model, the machine learning classification module can classify the controls to be recognized based on the feature vectors output by the deep learning classification module, thereby improving the accuracy of control recognition.

[0066] See also Figure 8 , a control identification method provided in an embodiment of the present application is applied to Figure 1 or Figure 2 The electronic device or server shown, the method comprising:

[0067] Step S310: Obtain the control tree corresponding to the current user interface.

[0068] Step S320: traverse the control tree to obtain control information of all controls corresponding to the control tree.

[0069] Step S330: Based on the control information, all controls corresponding to the control tree are screened to obtain reference controls.

[0070] Among them, Fig. 9 As shown, step S330 may include the following steps:

[0071] Step S331: Determine a first control to be referenced from all controls corresponding to the control tree based on the control information, wherein the first control to be referenced is a control among all controls corresponding to the control tree, except that the corresponding control determined based on the control information is a user interactive control, and the text information of the user interactive control and the sub-control of the user interactive control is not empty.

[0072] In an embodiment of the present application, the first control to be referenced can be determined from all controls corresponding to the control tree according to the NAF determination rule based on the control information. Among them, NAF is a determination interface provided by the operating system to determine whether a control is a control friendly to accessibility services. The principle is to determine whether a control is a user-interactive control and whether the text information of the control and its sub-controls is empty.

[0073] Among them, user-interactive controls can be understood as controls that users can click. Text information can include Text information and Content Description information. Text information is visible to users, and Content Description information is invisible to users. All controls corresponding to the control tree include user-interactive controls and user-non-interactive controls.

[0074] In the embodiment of the present application, the user-interactive controls are mainly screened to determine whether the fields corresponding to the Text information and Content Description information of the corresponding user-interactive controls and their sub-controls are empty. If the Text information and Content Description information of the corresponding user-interactive controls and their sub-controls are both empty, then it is determined that the control needs to be identified and does not need to be screened out, thereby obtaining a first reference control that is less in number than all controls corresponding to the control tree; if one of the Text information and Content Description information of the corresponding user-interactive controls and their sub-controls is not empty, then it is determined that the control has rich information and does not need to be identified, i.e., it is screened out.

[0075] Step S332: Determine a second control to be referenced from the first control to be referenced based on the control information of the first control to be referenced, wherein the second control to be referenced is a control in the first control to be referenced except for the parent control of the corresponding control determined based on the control information and the sibling control including the control of text information.

[0076] In the embodiment of the present application, the text information of some controls is stored in their parent controls or sibling controls. Therefore, after filtering the user-interactive controls in all controls corresponding to the control tree in the above manner, the remaining controls can be filtered according to the control information of the remaining controls. Among them, the remaining controls may include all user-non-interactive controls and some user-interactive controls in all controls corresponding to the control tree.

[0077] As a method, check whether the parent control and sibling controls of the remaining controls contain Text information and Content Description information. If so, it means that the control does not need to be identified, that is, it is screened out; if not, it means that the control needs to be identified and does not need to be screened out, thereby obtaining a second set of controls to be referenced that are less in number than the first set of controls to be referenced.

[0078] Step S333: Determine a third control to be referenced from the second control to be referenced based on the control information of the second control to be referenced, wherein the third control to be referenced is a control in the second control to be referenced except for the control whose text information and attribute information of the corresponding control determined based on the control information are complete.

[0079] In the embodiment of the present application, the second controls to be referenced are screened mainly based on the control information of the second controls to be referenced. If it is determined that the text information or attribute information of the corresponding control is missing, it is determined that the control needs to be identified and does not need to be screened out, that is, a third set of controls to be referenced that is less in number than the second set of controls to be referenced is obtained.

[0080] Step S334: Determine a fourth control to be referenced from the third controls to be referenced based on the control information of the third control to be referenced, wherein the fourth control to be referenced is a control in the third controls to be referenced except for controls whose size ratios of corresponding controls determined based on the control information do not meet the preset size ratios.

[0081] In the embodiment of the present application, the preset size ratio is the ratio of width to height of the shape of the preset control. Generally speaking, the preset ratio of the control is usually within a certain range. For example, the shape of common controls that need to be identified, such as play controls or search controls, is usually close to a square, that is, the ratio of width to height is close to 1.

[0082] Here, the main thing is to filter out the controls whose aspect ratios in the third reference controls obviously do not conform to the preset aspect ratios. The aspect ratios of the third reference controls can be calculated based on the coordinate information in the control information.

[0083] Step S335: Determine a fifth control to be referenced from the fourth control to be referenced based on the control information of the fourth control to be referenced, wherein the fifth control to be referenced is a control in the fourth control to be referenced except for a control whose control type determined based on the control information is not a preset control type.

[0084] As a way, due to the image attributes of the control, the control types in the development framework are often limited. In the Android system, the control types can only include Image Button, ImageView, Button, and TextView.

[0085] In the embodiment of the present application, the controls whose control types are not the preset control types in the fourth controls to be referenced are mainly screened out based on the control information of the fourth controls to be referenced.

[0086] Step S336: Determine the reference control from the fifth control to be referenced based on the control information of the fifth control to be referenced, wherein the reference control is a control in the fifth control to be referenced, except for the corresponding control determined based on the control information that has an overlapping control, and the corresponding control and the overlapping control are completely covered.

[0087] Finally, the controls that may overlap on the image in the fifth reference control are processed to screen out the controls that are completely covered in the fifth reference control. Whether the controls overlap can also be calculated based on the coordinate information in the control information.

[0088] Optionally, in the embodiment of the present application, step S333 must be before step S335, and the order of implementing step S331, step S332, step S334 and step S336 is not specifically limited here.

[0089] Step S340: Obtain the similarity between the reference control and the controls included in the control template library.

[0090] In the embodiment of the present application, the similarity between the reference control and the control included in the control template library can be understood as the similarity between the shape and appearance of the reference control and the shape and appearance of the control included in the control template library.

[0091] Step S350: Filter the reference controls based on the similarity to obtain the controls to be identified.

[0092] In an embodiment of the present application, the controls that need to be identified in the reference controls can be determined based on the calculated similarity, that is, the controls to be identified can be determined.

[0093] Step S360: input the image of the control to be identified into the deep learning classification module in the control recognition model, and obtain the feature vector corresponding to the control to be identified output by the deep learning classification module.

[0094] Step S370: input the feature vector into the machine learning classification module in the control recognition model, and obtain the control recognition result of the control to be recognized output by the machine learning classification module.

[0095] The present application provides a control identification method, which can reduce the images of controls to be identified that are input into the control identification model by traversing the control tree, obtaining the control information of all controls corresponding to the control tree, and then screening the controls multiple times based on the control information, so that the control can only identify a small number of controls to be identified, thereby speeding up the control identification. At the same time, due to the reduction in the number of controls to be identified, the demand for memory in the control identification stage can be reduced. Furthermore, by constructing a control identification model including a deep learning classification module and a machine learning classification module, in the process of identifying the controls to be identified by the control identification model, the machine learning classification module can classify the controls to be identified based on the feature vectors output by the deep learning classification module, thereby improving the accuracy of control identification.

[0096] See also Fig.10 , a control identification method provided in an embodiment of the present application is applied to Figure 1 or Figure 2 The electronic device or server shown, the method comprising:

[0097] Step S410: Obtain the control tree corresponding to the current user interface.

[0098] Step S420: traverse the control tree to obtain control information of all controls corresponding to the control tree.

[0099] Step S430: Based on the control information, all controls corresponding to the control tree are screened to obtain reference controls.

[0100] As a method, based on the control information, all controls corresponding to the control tree are screened to obtain a reference control, including at least one of the following methods: based on the control information, it is determined that the corresponding control is a user-interactive control, and the text information of the user-interactive control and the child control of the user-interactive control is empty, and the user-interactive control is used as a reference control; based on the control information, it is determined that the parent control and the brother control of the corresponding control do not include text information, and the control corresponding to the control information is used as a reference control; based on the control information, it is determined that the text information or attribute information of the corresponding control is missing, and the control corresponding to the control information is used as a reference control; based on the control information, it is determined that the size ratio of the corresponding control meets the preset size ratio, and the control corresponding to the control information is used as a reference control; based on the control information, it is determined that the control type of the corresponding control is a preset control type, and the control corresponding to the control information is used as a reference control; and based on the control information, it is determined that the corresponding control corresponds to an overlapping control, and the control that is not covered or not completely covered between the corresponding control and the overlapping control is used as a reference control.

[0101] In an embodiment of the present application, when determining a reference control, the reference control can be determined by combining the above-listed filtering methods. Each of the above-mentioned filtering methods can filter out a portion of all controls corresponding to the control tree that do not require control identification, and then all controls corresponding to the control tree can be filtered by combining the above-mentioned methods. In an embodiment of the present application, how the above-mentioned filtering methods are specifically combined is not specifically limited here, and can be combined according to specific needs, wherein the specific needs can be the demand for the number of controls to be identified, etc.

[0102] Step S440: Obtain the similarity between the reference control and the controls included in the control template library.

[0103] As a way, Fig.11 As shown, step S440 may include the following steps:

[0104] Step S441: input the image of the reference control into the feature extraction model, and obtain the feature vectors corresponding to each of the reference controls output by the feature extraction model.

[0105] The feature vector is obtained by concatenating the texture feature corresponding to the image and the shape feature corresponding to the image.

[0106] In the embodiment of the present application, the number of reference controls may be multiple. The texture feature corresponding to the image may be an LBP (Local Binary Pattern) feature, and the shape feature corresponding to the image may be a HOG (Histogram of Oriented Gradient) feature.

[0107] Optionally, the feature extraction model may include an LBP operator and a HOG operator, which are used to extract LBP features and HOG features of the image of the reference control, respectively. When the image of the reference control is input into the feature extraction model, the LBP operator and the HOG operator in the feature extraction model may be used to extract features of the image of the reference control.

[0108] Among them, the LBP operator is an effective texture feature extraction operator, which measures the local texture information of the image and is invariant to the image brightness. The main idea is to collect multiple pixels within a certain sampling radius of a certain pixel point, obtain the LBP feature values ​​corresponding to each of the multiple pixels, and take the average value of the LBP feature values ​​corresponding to each of the multiple pixels to obtain the LBP feature of the pixel point. A reference control image can be sampled at a certain sampling rate to obtain a specified number of pixels, and the LBP features of the specified number of pixels are extracted by the LBP operator, and the LBP features of the specified number of pixels are used as the LBP features corresponding to the reference control. In an embodiment of the present application, the LBP features of images of other reference controls can also be extracted in the above manner.

[0109] The HOG feature is constructed by calculating and counting the gradient direction histogram of the local area of ​​the image. It measures the local shape information of the image. Its main idea is to divide the image into cell units, count the gradient direction histograms in the divided cell units, and combine the gradient direction histograms of the divided cell units to obtain the HOG feature of the image. For an image of a reference control, the image of the reference control can be divided into multiple cell units, and then the gradient direction histograms in the multiple cell units are counted to obtain the gradient direction histograms corresponding to each of the multiple cell units, and then the gradient direction histograms corresponding to each of the multiple cell units are combined to obtain the HOG feature of the image of the reference control. In an embodiment of the present application, the HOG features of images of other reference controls can also be extracted in the above manner.

[0110] After obtaining the LBP features and HOG features corresponding to the image of the reference control, the LBP features and HOG features corresponding to each reference control image can be spliced ​​to obtain the feature vector corresponding to each reference control. Among them, splicing the LBP features and HOG features can be understood as connecting the HOG features to the LBP features.

[0111] Step S442: Input the images of the controls included in the control template library into the feature extraction model, and obtain the feature vectors corresponding to the controls included in the control template library output by the feature extraction model.

[0112] In the embodiment of the present application, the control template library may include standard templates of various types of controls, wherein the standard templates of various types of controls may be pre-set appearances, shapes, etc. of the controls.

[0113] Similarly, the feature vectors corresponding to the controls included in the control template library can be obtained by the method described above.

[0114] Step S443: Based on the feature vectors corresponding to each of the reference controls and the feature vectors corresponding to each of the controls included in the control template library, determine the similarity between the reference controls and the controls included in the control template library.

[0115] As a method, after obtaining the feature vectors corresponding to each reference control and the feature vectors corresponding to each control included in the control template library, the similarity between the feature vectors corresponding to each reference control and the feature vectors corresponding to each control included in the control template library can be calculated by using methods such as Hamming distance or cosine distance, that is, calculating the similarity between the reference control and the control included in the control template library.

[0116] When calculating the similarity between the feature vectors corresponding to the reference controls and the feature vectors corresponding to the controls included in the control template library, the feature vectors corresponding to each reference control and the feature vectors corresponding to the controls included in the control template library can be calculated to obtain the similarity between each reference control and each control in the control template library. In other words, a reference control can correspond to multiple similarities. Exemplarily, if the number of reference controls is 2, namely control 1 and control 2, the control template library includes 5 controls, namely control A, control B, control C, control D and control E. When calculating the similarity between the feature vectors corresponding to the reference controls and the feature vectors corresponding to the controls included in the control template library, the similarity between control 1 and control A, control B, control C, control D and control E will be calculated respectively; and the similarity between control 2 and control A, control B, control C, control D and control E will be calculated respectively. After the above calculation, control 1 can correspond to 5 similarities, and control 2 can correspond to 5 similarities.

[0117] Step S450: Determine a reference control whose similarity is greater than or equal to a preset similarity as the control to be identified.

[0118] In the embodiment of the present application, the preset similarity is a preset similarity for determining that the reference control needs to be identified. For example, the preset similarity can be set to 70%. The control to be identified is the control that needs to be identified.

[0119] As a method, after calculating the similarity between each reference control and the controls included in the control template library by the above method, multiple similarities corresponding to each reference control can be compared with the preset similarity. If the multiple similarities corresponding to a reference control are all less than the preset similarity, it is determined that the reference control does not need to be identified. That is, the reference control is definitely not the required control type.

[0120] On the contrary, if one of the multiple similarities corresponding to a reference control is greater than or equal to the preset similarity, then the reference control can be determined as the control to be identified. The controls included in the reference control whose corresponding similarity is greater than or equal to the preset similarity are determined as the controls to be identified.

[0121] like Fig.12 As shown, Fig.12 The schematic diagram of the control to be identified is determined based on the similarity between the control included in the control template library and the reference control. First, the image of the control in the control template library and the image of the reference control are input into the feature extraction model, and the LBP feature and HOG feature of the image are extracted by the LBP operator and the HOG operator in the feature extraction model, and then the LBP feature and the HOG feature are spliced ​​to obtain the feature vectors corresponding to the controls included in the control template library and the feature vectors corresponding to the reference control, and then the similarity is calculated based on the feature vectors corresponding to the controls included in the control template library and the feature vectors corresponding to the reference control, and the calculated similarity is compared with the preset similarity to determine the reference control to be retained and the reference control to be screened out. Among them, the retained reference control is determined as the control to be identified, and the screened out reference control is determined as the control that does not need to be identified.

[0122] Step S460: input the image of the control to be identified into the deep learning classification module in the control recognition model, and obtain the feature vector corresponding to the control to be identified output by the deep learning classification module.

[0123] Step S470: input the feature vector into the machine learning classification module in the control recognition model, and obtain the control recognition result of the control to be recognized output by the machine learning classification module.

[0124] The present application provides a control identification method, which first screens all controls corresponding to the control tree through control information to obtain reference controls, and then screens the reference controls through the similarity between the reference controls and the controls in the control template library, which can further reduce the number of controls input into the control identification model, thereby improving the speed of control identification. Furthermore, since the number of controls to be identified input into the control identification model is further reduced, it is conducive to deploying the control identification method on a mobile terminal. Furthermore, by constructing a control identification model including a deep learning classification module and a machine learning classification module, in the process of identifying the controls to be identified through the control identification model, the machine learning classification module can classify the controls to be identified based on the feature vectors output by the deep learning classification module, thereby improving the accuracy of control identification.

[0125] See also Fig.13 , an embodiment of the present application provides a control identification device 500, the device 500 comprising:

[0126] The control acquisition unit 510 is used to acquire the control to be identified.

[0127] As a method, the control acquisition unit 510 is specifically used to obtain a control tree corresponding to the current user interface; traverse the control tree to obtain control information of all controls corresponding to the control tree; and filter all controls corresponding to the control tree based on the control information to obtain controls to be identified.

[0128] Optionally, the control acquisition unit 510 is used to filter all controls corresponding to the control tree based on the control information to obtain a reference control; obtain the similarity between the reference control and the controls included in the control template library; and filter the reference control based on the similarity to obtain the control to be identified.

[0129] Optionally, the control acquisition unit 510 is further used to input the image of the reference control into the feature extraction model to obtain the feature vectors corresponding to each of the reference controls output by the feature extraction model; input the image of the control included in the control template library into the feature extraction model to obtain the feature vectors corresponding to each of the controls included in the control template library output by the feature extraction model; based on the feature vectors corresponding to each of the reference controls and the feature vectors corresponding to each of the controls included in the control template library, determine the similarity between the reference control and the controls included in the control template library. The feature vector is obtained by splicing the texture feature corresponding to the image and the shape feature corresponding to the image.

[0130] Optionally, the control acquisition unit 510 is further used to determine a reference control whose similarity is greater than or equal to a preset similarity as the control to be identified.

[0131] Optionally, the control acquisition unit 510 is also used to determine based on the control information that the parent control and sibling controls of the corresponding control do not include text information, and use the control corresponding to the control information as a reference control; if it is determined based on the control information that the text information or attribute information of the corresponding control is missing, then use the control corresponding to the control information as a reference control; if it is determined based on the control information that the size ratio of the corresponding control meets a preset size ratio, then use the control corresponding to the control information as a reference control; if it is determined based on the control information that the control type of the corresponding control is a preset control type, then use the control corresponding to the control information as a reference control; and if it is determined based on the control information that the corresponding control corresponds to an overlapping control, then use the control that is not covered or not completely covered between the corresponding control and the overlapping control as a reference control.

[0132] Optionally, the control acquisition unit 510 is further used to determine a first control to be referenced from all controls corresponding to the control tree based on the control information, the first control to be referenced being a control among all controls corresponding to the control tree, except that the corresponding control determined based on the control information is a user-interactive control, and the text information of the user-interactive control and the child control of the user-interactive control is not empty; determine a second control to be referenced from the first control to be referenced based on the control information of the first control to be referenced, the second control to be referenced being a control among the first control to be referenced, except that the parent control of the corresponding control determined based on the control information and the sibling control include text information; determine a third control to be referenced from the second control to be referenced based on the control information of the second control to be referenced, the third control to be referenced being a control among the second control to be referenced, except that the text information of the corresponding control determined based on the control information is not empty. and controls other than controls with complete attribute information; determining a fourth control to be referenced from the third control to be referenced based on the control information of the third control to be referenced, the fourth control to be referenced being controls in the third control to be referenced except controls whose size ratio of the corresponding control determined based on the control information does not satisfy a preset size ratio; determining a fifth control to be referenced from the fourth control to be referenced based on the control information of the fourth control to be referenced, the fifth control to be referenced being controls in the fourth control to be referenced except controls whose control type of the corresponding control determined based on the control information is not a preset control type; determining the reference control from the fifth control to be referenced based on the control information of the fifth control to be referenced, the reference control being controls in the fifth control to be referenced except controls that have overlapping controls corresponding to the corresponding control determined based on the control information, and controls that are completely covered by the corresponding control and the overlapping control.

[0133] The vector acquisition unit 520 is used to input the image of the control to be identified into the deep learning classification module in the control recognition model, and obtain the feature vector corresponding to the control to be identified output by the deep learning classification module.

[0134] The result acquisition model 530 is used to input the feature vector into the machine learning classification module in the control recognition model, and obtain the control recognition result of the control to be identified output by the machine learning classification module.

[0135] It should be noted that the device embodiment in the present application corresponds to the aforementioned method embodiment. The specific principles in the device embodiment can be found in the contents of the aforementioned method embodiment and will not be repeated here.

[0136] The following will be combined Fig.14 An electronic device or server provided in the present application is described.

[0137] See also Fig.14 Based on the above control identification method and device, the embodiment of the present application also provides another electronic device or server 800 that can execute the above control identification method. The electronic device or server 800 includes one or more (only one is shown in the figure) processors 802, a memory 804 and a network module 806 that are coupled to each other. Among them, the memory 804 stores a program that can execute the content of the above embodiment, and the processor 802 can execute the program stored in the memory 804.

[0138] Among them, the processor 802 may include one or more processing cores. The processor 802 uses various interfaces and lines to connect various parts within the entire server 800, and executes various functions and processes data of the server 800 by running or executing instructions, programs, code sets or instruction sets stored in the memory 804, and calling data stored in the memory 804. Optionally, the processor 802 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 802 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 802, but may be implemented separately through a communication chip.

[0139] The memory 804 may include a random access memory (RAM) or a read-only memory (ROM). The memory 804 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 804 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device or server 800 during use.

[0140] The network module 806 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices, such as communicating with an audio playback device. The network module 806 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM) cards, memories, etc. The network module 806 may communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices via wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. For example, the network module 806 may exchange information with a base station.

[0141] Please refer to Fig.15 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 900 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0142] The computer-readable storage medium 900 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has storage space for program code 910 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 910 can be compressed, for example, in an appropriate form.

[0143] The present application provides a control recognition method, device, electronic device and storage medium, which first obtains the control to be recognized, and then inputs the image of the control to be recognized into a composite control recognition model including a deep learning classification module and a machine learning classification module to obtain the control recognition result corresponding to the control to be recognized. Thus, by constructing a control recognition model including a deep learning classification module and a machine learning classification module, in the process of identifying the control to be recognized by the control recognition model, the machine learning classification module can classify the control to be recognized based on the feature vector output by the deep learning classification module, thereby improving the accuracy of control recognition.

[0144] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A control identification method, characterized in that: The method comprises: Get the control tree corresponding to the current user interface; Traversing the control tree to obtain control information of all controls corresponding to the control tree; Based on the control information, all controls corresponding to the control tree are screened to obtain reference controls; The filtering of all controls corresponding to the control tree based on the control information to obtain a reference control includes: Determining based on the control information that the corresponding control is a user-interactive control, and that the text information of the user-interactive control and a sub-control of the user-interactive control is empty, and using the user-interactive control as a reference control; Obtaining similarities between the reference control and controls included in a control template library; Screening the reference controls based on the similarity to obtain controls to be identified; Input the image of the control to be identified into a deep learning classification module in a control recognition model, and obtain a feature vector corresponding to the control to be identified output by the deep learning classification module; The feature vector is input into the machine learning classification module in the control recognition model, and the control recognition result of the control to be recognized output by the machine learning classification module is obtained.

2. The method according to claim 1, characterized in that The obtaining of the similarity between the reference control and the control included in the control template library includes: Inputting the image of the reference control into a feature extraction model, and obtaining the feature vectors corresponding to each of the reference controls output by the feature extraction model; Inputting images of controls included in the control template library into the feature extraction model, and obtaining feature vectors corresponding to the controls included in the control template library output by the feature extraction model; Based on the feature vectors corresponding to the reference controls and the feature vectors corresponding to the controls included in the control template library, the similarity between the reference controls and the controls included in the control template library is determined.

3. The method according to claim 2, characterized in that The feature vector is obtained by concatenating the texture feature corresponding to the image and the shape feature corresponding to the image.

4. The method according to claim 1, characterized in that The step of screening the reference controls based on the similarity to obtain the control to be identified includes: A reference control whose similarity is greater than or equal to a preset similarity is determined as the control to be identified.

5. The method according to claim 1, characterized in that The filtering of all controls corresponding to the control tree based on the control information to obtain a reference control includes at least one of the following methods: Determining based on the control information that the corresponding control is a user-interactive control, and that the text information of the user-interactive control and a sub-control of the user-interactive control is empty, and using the user-interactive control as a reference control; Determine based on the control information that the parent control and the sibling controls of the corresponding control do not include text information, and use the control corresponding to the control information as a reference control; Determining based on the control information that text information or attribute information of a corresponding control is missing, then using the control corresponding to the control information as a reference control; Determining based on the control information that the size ratio of the corresponding control meets the preset size ratio, then using the control corresponding to the control information as a reference control; Determining, based on the control information, that the control type of the corresponding control is a preset control type, and taking the control corresponding to the control information as a reference control; as well as If it is determined based on the control information that the corresponding control corresponds to an overlapping control, the control that is not covered or not completely covered between the corresponding control and the overlapping control is used as a reference control.

6. The method according to claim 1, characterized in that The filtering of all controls corresponding to the control tree based on the control information to obtain a reference control includes: Determine a first control to be referenced from all controls corresponding to the control tree based on the control information, wherein the first control to be referenced is a control among all controls corresponding to the control tree, except that the corresponding control determined based on the control information is a user-interactive control, and text information of the user-interactive control and a sub-control of the user-interactive control is not empty; Determine a second control to be referenced from the first control to be referenced based on the control information of the first control to be referenced, the second control to be referenced being a control in the first control to be referenced except a parent control of the corresponding control determined based on the control information and a control whose sibling controls include text information; Determine a third control to be referenced from the second control to be referenced based on the control information of the second control to be referenced, wherein the third control to be referenced is a control in the second control to be referenced except for the control whose text information and attribute information of the corresponding control determined based on the control information are complete; Determine a fourth control to be referenced from the third control to be referenced based on the control information of the third control to be referenced, the fourth control to be referenced being a control in the third control to be referenced except for a control whose size ratio of the corresponding control determined based on the control information does not satisfy a preset size ratio; Determine a fifth control to be referenced from the fourth control to be referenced based on the control information of the fourth control to be referenced, the fifth control to be referenced being a control in the fourth control to be referenced except for a control whose control type of the corresponding control determined based on the control information is not a preset control type; The reference control is determined from the fifth control to be referenced based on the control information of the fifth control to be referenced, and the reference control is a control in the fifth control to be referenced, except for the corresponding control determined based on the control information that has an overlapping control, and the corresponding control and the overlapping control are completely covered.

7. A control identification device, characterized in that: The device comprises: A control acquisition unit, used to acquire a control tree corresponding to a current user interface; traverse the control tree to acquire control information of all controls corresponding to the control tree; based on the control information, screen all controls corresponding to the control tree to obtain a reference control; the screening of all controls corresponding to the control tree based on the control information to obtain a reference control includes: determining that the corresponding control is a user-interactive control based on the control information, and the text information of the user-interactive control and the sub-control of the user-interactive control is empty, and taking the user-interactive control as a reference control; acquiring the similarity between the reference control and the controls included in the control template library; screening the reference control based on the similarity to obtain a control to be identified; A vector acquisition unit, used for inputting the image of the control to be identified into a deep learning classification module in a control recognition model, and acquiring a feature vector corresponding to the control to be identified output by the deep learning classification module; The result acquisition model is used to input the feature vector into the machine learning classification module in the control recognition model, and obtain the control recognition result of the control to be identified output by the machine learning classification module.

8. An electronic device, characterized in that: The method comprises one or more processors; one or more programs are stored in a memory and configured to execute the method according to any one of claims 1 to 6 by the one or more processors.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, wherein when the program code is executed by a processor, the method according to any one of claims 1 to 6 is executed.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images