Control detection model training method, control detection method and device

Abstract

The application provides a control detection model training method, a control detection method and device; the method comprises the following steps: identifying a first control in a first application program to obtain an identification result; generating initial training data based on the identification result; wherein the identification result comprises the position of the first control in the page of the first application program; training a preset initial model based on the initial training data to obtain an intermediate model; detecting a second control in a second application program through the intermediate model to obtain a detection result, updating the initial training data based on the detection result to obtain final training data; wherein the detection result comprises the position of the second control in the page of the second application program; and training the intermediate model based on the final training data to obtain a control detection model. This method can obtain rich training data at a low labor cost, so that the control detection model obtained by training has high universality and detection accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a control detection model training method, a control detection method and device. BACKGROUND

[0002] In order to ensure the stability of the application program, the application program needs to be tested. In the process of automatically testing the application program, the controls in each page of the software need to be identified first, and then the controls are traversed according to a certain traversal strategy. In related technologies, a control identification model can be trained using training data. The control identification model identifies the controls from the page screenshot of the application program through image recognition. The training data needs to be pre-collected and labeled, which requires a lot of human cost, and the generality of the model is low, and the control identification accuracy of the model for some application programs is low. SUMMARY

[0003] Therefore, the present application aims to provide a control detection model training method, a control detection method and device, so as to obtain rich training data at a lower human cost, so that the control detection model trained has high generality and detection accuracy.

[0004] In a first aspect, an embodiment of the present application provides a control detection model training method, which comprises: identifying a first control in a first application program to obtain an identification result; generating initial training data based on the identification result; wherein the identification result comprises the position of the first control in the page of the first application program; training a preset initial model based on the initial training data to obtain an intermediate model; detecting a second control in a second application program through the intermediate model to obtain a detection result, updating the initial training data based on the detection result to obtain final training data; wherein the detection result comprises the position of the second control in the page of the second application program; and training the intermediate model based on the final training data to obtain a control detection model.

[0005] In a second aspect, an embodiment of the present application provides a control detection method, which comprises: determining a target page from a target application program; inputting the target page into a control detection model trained in advance to obtain an output result; wherein the control detection model is trained by the control detection model training method described above; the output result comprises a control indication identifier; the control indication identifier indicates the display area of the detected control in the target page; and determining the control detection result of the target page based on the output result.

[0006] In a third aspect, an embodiment of the present application provides a training device of a control detection model, the device comprising: a first data generation module configured to identify a first control in a first application program to obtain an identification result; generate initial training data based on the identification result; wherein the identification result comprises a position of the first control in a page of the first application program; a first training module configured to train a preset initial model based on the initial training data to obtain an intermediate model; a second data generation module configured to detect a second control in a second application program through the intermediate model to obtain a detection result, update the initial training data based on the detection result to obtain final training data; wherein the detection result comprises a position of the second control in a page of the second application program; and a second training module configured to train the intermediate model based on the final training data to obtain the control detection model.

[0007] In a fourth aspect, an embodiment of the present application provides a control detection device, the device comprising: a page determination module configured to determine a target page from a target application program; a detection module configured to input the target page into a control detection model trained in advance to obtain an output result; wherein the control detection model is trained by the training device of the control detection model; the output result comprises a control indication identifier; the control indication identifier indicates a display area of a detected control in the target page; and a result determination module configured to determine a control detection result of the target page based on the output result.

[0008] In a fifth aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the training method of the control detection model or the control detection method.

[0009] In a sixth aspect, an embodiment of the present application provides a machine readable storage medium, wherein the machine readable storage medium stores machine executable instructions, and the machine executable instructions, when invoked and executed by a processor, cause the processor to implement the training method of the control detection model or the control detection method.

[0010] The embodiments of the present application have the following beneficial effects:

[0011] The control detection model training method, the control detection method, the device and the electronic equipment determine a target page from a target application; input the target page into a control detection model trained in advance to obtain an output result; the control detection page is trained by the control detection model training method; the control detection model training method includes: obtaining initial training data; training an intermediate model based on the initial training data; obtaining a first detection result based on the intermediate model; updating the initial training data based on the first detection result; and training the intermediate model based on the updated initial training data to obtain a final control detection model. The control detection model training method can obtain rich training data at a low cost, so that the control detection model has high universality and detection accuracy, and the control detection accuracy can be improved by using the control detection model.

[0012] Other features and advantages of the present application will be set forth in the descriptions below, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and attained by the structure particularly pointed out in the description, claims and drawings.

[0013] In order to make the above objectives, characteristics and advantages of the present application more apparent, the following preferred embodiments are specifically described below, and the accompanying drawings are referred to for detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0015] Figure 1 A flowchart of a control detection model training method provided by an embodiment of the present application;

[0016] Figure 2 A schematic diagram of a control recognition result in a page provided by an embodiment of the present application;

[0017] Figure 3 Another schematic diagram of a control recognition result in a page provided by an embodiment of the present application;

[0018] Figure 4 A flowchart of a control detection method provided by an embodiment of the present application;

[0019] Figure 5A structural schematic diagram of a control detection model training device provided by an embodiment of the present application is shown in the figure.

[0020] Figure 6 A structural schematic diagram of a control detection device provided by an embodiment of the present application is shown in the figure.

[0021] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions, and advantages of embodiments of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the accompanying drawings. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall within the scope of the present application.

[0023] In the field of software automated testing, obtaining accurate UI (User Interface) controls is a prerequisite for the automated testing task. For example, in an automated stability testing task, the UI controls in the software to be tested need to be identified first, and then these UI controls are traversed according to a certain strategy. Among various UI controls, buttons are the most common control type, and the identification accuracy of buttons will greatly affect the final effect of the automated testing task.

[0024] In related technologies, there are automated frameworks that can implement the identification of buttons. The technical routes of these automated frameworks for identifying buttons can be divided into two categories, namely a control recognition-based scheme and an image-based recognition method.

[0025] Among them, for the control recognition-based scheme, this scheme can obtain the complete UI tree structure of the current page of an App (Application), can obtain the affiliation of each node in the page, and can obtain detailed information of each node, such as node name, size, position, etc. Representative tools of the control recognition-based scheme include Fastbot, moneky tool, and Airtest poco tool, etc. These tools can obtain the UI information of the App from the code level through program instrumentation and other methods.

[0026] However, most of these tools can only identify the UI controls of native Apps under the Android system, and for game Apps and Apps of other platforms, complex access operations are required, and the versatility is poor. In addition, the identification effect of these tools is not accurate enough, and a large number of invalid controls will be identified, especially on non-native Apps.

[0027] The reason for identifying the invalid control is that different controls have different display levels, and the control with low display level is blocked by the control with high display level. The existing tool cannot obtain the display level relationship of the UI, and therefore the blocked control is also misidentified. In addition, it takes a long time to obtain the UI tree of the App using the UI automation tool, and in some Apps, it takes several seconds or even tens of seconds to obtain the UI tree once, which seriously affects the efficiency.

[0028] For the image-based recognition method, such a method recognizes potential UI controls from a screenshot of an App based on computer vision technology, and generally uses a neural network to recognize through target detection. This method can identify which UI controls are in the current page and can estimate the size and position of the controls, but cannot obtain the detailed attributes and hierarchical relationship of the controls. The image-based recognition method has strong universality and low access cost, and as long as the screenshot of the device can be obtained, the controls in the current page of the device can be recognized.

[0029] However, the existing image-based recognition scheme is highly dependent on large-scale high-quality training data. To obtain a general and high-accuracy button detection model, a large amount of labeled data needs to be collected for training. This process generally requires manual annotation of all buttons in various pages of various Apps, which consumes a large amount of human cost. When applied to some Apps with special UI design, the recognition effect of the model may be poor, and corresponding data needs to be supplemented to improve the recognition accuracy.

[0030] Based on the above problems, the embodiment of the present application provides a control detection model training method, device and electronic equipment, which can be applied to control detection of various application programs.

[0031] To facilitate the understanding of the present embodiment, first, a control detection model training method disclosed by the present embodiment is introduced in detail. The method can be executed by a server, a terminal device or other electronic equipment, as shown in Figure 1 The method includes the following steps:

[0032] Step S102, identifying a first control in a first application program to obtain an identification result; generating initial training data based on the identification result; wherein the identification result includes the position of the first control in the page of the first application program;

[0033] The first application program can be a communication program, a game program, etc. The embodiment aims to train a control detection model with high detection accuracy. If the control detection model is used for control detection in a specific type of program, the first application program can be a program of the type. For example, if the control detection model is dedicated to detection of game programs, the first application program can be a game program. If the control detection model needs to be applied to various programs, the first application program can also be a program of any type.

[0034] In actual implementation, a tool in the related art can be used to identify the first control in the first application program. For example, a UI tree in the first application program is obtained, the UI tree including object attributes, types, and dependencies between objects in each program page of the first application program. The first control in each page of the first application program is identified through the UI tree.

[0035] The identification result includes the position of the first control in the page of the first application program. In actual implementation, the position of the control in the program page can be indicated in the form of a rectangular box or a bounding box. The identification result can also include the size of the first control. For example, the size of the rectangular box refers to the size of the first control.

[0036] The first application program usually includes multiple program pages. Therefore, in the identification result, the first program is also distributed in multiple program pages. In actual implementation, each program page in the identification result can be taken as a training sample, and the position indication of the first control in the program page, such as the aforementioned rectangular box, can be taken as the labeling information of the training sample. Part of the program pages in the identification result can also be obtained, and each program page can be taken as a training sample. Based on this, the initial training data can include multiple program pages and the labeling information of the first control in each program page.

[0037] In step S104, the initial model is trained based on the initial training data to obtain an intermediate model.

[0038] The initial model can be a neural network model or other machine learning model. For example, the initial model can be a general object detection model, such as a YOLO model. The initial model is trained through the initial training data. In the training process, a loss value can be calculated through a preset loss function. Based on the loss value, the parameters of the initial model are updated in a gradient descent manner until the initial model converges, and the aforementioned intermediate model is obtained.

[0039] In step S106, the second control in the second application program is detected through the intermediate model to obtain a detection result, the initial training data is updated based on the detection result to obtain final training data; the detection result includes the position of the second control in the page of the second application program.

[0040] In this embodiment, in order to improve the detection accuracy of the model, after obtaining the intermediate model, the training data needs to be updated, and then the model is trained again using the updated training data. In order to improve the detection accuracy of the model while reducing the labor cost of the training data, in this embodiment, the initial training data is updated based on the detection result output by the intermediate model.

[0041] Specifically, first, the second application program is obtained, which can be any type of application program, such as shopping, communication, game, etc.; the program page is obtained from the second application program, and then the screenshot of the program page is input into the intermediate model to obtain a detection result; the detection result includes the position indication of the second control in the program page, such as a rectangular box label, and each rectangular box label indicates a second control. After obtaining the detection result, part of the page can be selected, each page is used as a training sample, the position indication of the second control on the page is used as the label information of the training sample, and the training sample and the label information are saved in the initial training data. The amount of training samples in the initial training data is increased, and the updated initial training data is used as the final training data.

[0042] In actual implementation, in order to improve the generality of the model, the above-mentioned second application program can include multiple second application programs, and the program types of the multiple second application programs can be different, so that the finally trained control detection model can detect the controls in various application programs.

[0043] In step S108, the intermediate model is trained based on the final training data to obtain a control detection model.

[0044] After obtaining the final training data, the intermediate model is trained through the final training data. In the training process, the loss value can be calculated through a preset loss function, and based on the loss value, the parameters of the intermediate model are updated in a gradient descent manner until the intermediate model converges, that is, the above-mentioned intermediate model is obtained.

[0045] The training method of the control detection model, recognizes a first control in a first application program, obtains a recognition result; generates initial training data based on the recognition result; wherein the recognition result includes: the position of the first control in the page of the first application program; train the preset initial model based on the initial training data to obtain an intermediate model; detect a second control in a second application program through the intermediate model to obtain a detection result, update the initial training data based on the detection result to obtain final training data; wherein the detection result includes: the position of the second control in the page of the second application program; train the intermediate model based on the final training data to obtain the control detection model. In this way, after the intermediate model is trained based on the initial training data, the training data is updated based on the detection result of the intermediate model, and then the updated final training data is used to continue training the intermediate model to obtain the final control detection model. This way can obtain rich training data at a lower cost, so that the control detection model has higher universality and detection accuracy.

[0046] The following embodiment describes a specific implementation of recognizing a first control in a first application program.

[0047] Determine a first page from a first application program; recognize the control in the first page based on the page information of the first page; wherein the page information includes: the page nodes in the first page, and the dependency relationship between the page nodes; based on a preset traversal strategy, perform traversal operation on the control in the first page to obtain the first control in the first page that meets the preset control condition; continue to execute the step of determining the first page from the first application program until the first application program is traversed to obtain the recognition result of the first control of the first application program.

[0048] In the initial state, the first page can be the home page of the first application program, and in the process of traversal, the first page can be the page displayed after a certain control is triggered. In this embodiment, the control in the first page is recognized based on the page information of the first page. The page information includes each page node in the first page, which can be a control, text, picture, video, etc. in the page, and the dependency relationship between the page nodes, for example, page node A is a video player, and page node B is a video operation control in the video player; at this time, page node B belongs to page node A; the node attribute information of each page node is also included in the page information, and the page node with the attribute of control can be recognized through the node attribute information. The control in this embodiment can also be called a button.

[0049] The traversal strategy can be a depth-first search-based traversal strategy, a breadth-first search-based traversal strategy, a reinforcement learning-based traversal strategy, etc. Based on the traversal strategy, the traversal order of the controls in the first page can be determined, and then the traversal operation, such as a click operation, a sliding operation, etc., is performed on the controls. After the traversal operation is performed on the controls in the first page, the first control that meets the preset control condition can be selected from the controls. The preset control condition is used to filter the effective controls from the controls in the first page. The effective control can be understood as a control that, when triggered, can trigger a specific event, and can be a page that changes in a certain way. Therefore, the preset control condition can be whether a specified event is triggered after the control is triggered, or whether the page changes, or whether the page text changes, etc.

[0050] After the first control is obtained from the first page, the step of determining the first page from the first application program is executed in a loop until the first application program is traversed. The judgment condition for determining that the first application program is traversed can be that the maximum traversal time is reached, or that all the controls recognized from the first application program are triggered. After the first application program is traversed, the first control in each traversed program page in the first application program can be obtained.

[0051] Further, in order to improve the traversal efficiency, the identified controls need to be filtered during the control identification process. Specifically, based on the page information of the first page, the first page is subjected to control identification to obtain an initial identification result; based on the relative positions of the controls in the initial identification result, the controls in the initial identification result are filtered to obtain a final identification result.

[0052] The relative position here can be understood as the distance between the controls. Since the controls usually occupy a certain display area, the relative position can also be the overlap degree of the display areas between the controls, or the positional relationship between the specified positions in the display areas of the controls and the display areas of other controls, etc. Based on these relative positions, some redundant controls can be filtered out. For example, the same control is identified as multiple controls, and only one of the multiple controls needs to be retained, and the other controls are redundant controls.

[0053] In a specific implementation, if the area of the control display area of the third control is greater than the area of the control display area of the fourth control in the initial identification result, and the center position of the control display area of the third control is located in the control display area of the fourth control, the third control is deleted from the initial identification result to obtain the final identification result.

[0054] For example, the controls can be arranged in the order of the area of the control display region from large to small. If the center position of the display region of the control with a larger area is located in the center position of the display region of the control with a smaller area, the control with a larger area is deleted in this case. The center position of the display region can be understood as the geometric center of the control. The reason for deleting the control with a larger area is that triggering the control with a larger area is equivalent to triggering the control with a smaller area, and therefore, the control with a larger area can be filtered as a redundant control.

[0055] As shown in Figure 2 In the case shown in FIG. 6, for the controls arranged in the left vertical direction, there is one bounding box that encloses all the controls, and there are five bounding boxes that respectively enclose each control. At this time, the bounding box that encloses all the five controls is a redundant control recognition result, and needs to be deleted. For the three controls arranged in the right vertical direction, the bounding box that encloses all the three controls is a redundant control recognition result, and needs to be deleted. Figure 3 In the case shown in FIG. 7, after the bounding box that encloses multiple controls is deleted, each bounding box encloses one control, which avoids redundant controls.

[0056] For example, the redundant controls can be deleted by using the intersection over union method. If the intersection over union of the display regions of two controls in the initial recognition result is greater than a preset intersection over union threshold, one of the two controls can be deleted at this time.

[0057] In the case shown in FIG. 6, for the controls arranged in the left vertical direction, there is one bounding box that encloses all the controls, and there are five bounding boxes that respectively enclose each control. At this time, the bounding box that encloses all the five controls is a redundant control recognition result, and needs to be deleted. For the three controls arranged in the right vertical direction, the bounding box that encloses all the three controls is a redundant control recognition result, and needs to be deleted.

[0058] According to the above traversal order, one trigger control is determined each time, and a trigger operation such as a click operation is performed on the trigger control. For a valid control, the trigger operation usually has a certain impact on the page content, and therefore, in this embodiment, the valid control is obtained from the first control by comparing the first page content before the trigger operation is performed and the second page content after the trigger operation is performed. Specifically, the content change degree of the second page content relative to the first page content of the first page before the trigger operation is performed needs to satisfy a preset change condition. When the content of the second page content is different from the content of the first page content, or the content change reaches a certain degree, the control is determined as the first control.

[0059] Based on this, the preset control condition is used to: filter out the control that can trigger the specified event, or the control that can cause the change of the page content, or the control that can cause the larger change of the page content, from the control of the first page.

[0060] In a specific implementation, the preset change condition at least includes one of the following:

[0061] Condition (1): the overall change degree of the second page content relative to the first page content is higher than a first preset threshold;

[0062] The difference degree between the first page content and the second page content can be calculated by a preset difference degree calculation formula. When the difference degree is higher than the first preset threshold, the condition is met. In actual implementation, the overall change degree of the page can be represented by the similarity between the first page content and the second page content. For example, the similarity between the first page content and the second page content can be calculated by a preset similarity calculation formula. When the similarity is lower than a preset similarity threshold, it can be understood that the overall change degree of the page is higher than the first preset threshold.

[0063] Condition (2): the change degree of the second page content relative to the first page content in the local page region where the first control is located is higher than a second preset threshold;

[0064] For some controls, the overall change degree of the page before and after being triggered is small, but the change degree in the local page region is large. The local page region where the first control is located can be intercepted from the first page, and the local page region where the second control is located can be intercepted from the second page. For example, the region size can be preset, and then the local page region can be intercepted with the first control as the center.

[0065] Then, for the local page region intercepted from the first page and the local page region intercepted from the second page, one way is to calculate the difference degree between the two local page regions by a preset difference degree calculation formula. When the difference degree is higher than the second preset threshold, the condition is met.

[0066] In another way, the change degree of the local page region can be represented by the similarity between the local page region intercepted from the first page and the local page region intercepted from the second page. For example, the similarity between the local page region intercepted from the first page and the local page region intercepted from the second page can be calculated by a preset similarity calculation formula. When the similarity is lower than a preset similarity threshold, it can be understood that the change degree in the local page region is higher than the second preset threshold.

[0067] Condition (3): the second page content and the control text in the first page content change.

[0068] For some controls, the degree of change of the overall page and the local page area before and after triggering is low, but the control text in the page will change. Based on this, the control text in the first page content and the control text in the second page content before and after the control is triggered are counted. If the control texts of the two page contents are different, it can also be explained that the control is an effective control.

[0069] In actual implementation, for a certain control, one of the above three conditions is met, that is, the control is understood as an effective control, which can be used as a first control identified from the first application.

[0070] As can be seen from the foregoing embodiments, after identifying the first control in the first application, the position of the first control in the page of the first application is included in the identification result; since the first application usually includes multiple pages, the first control is distributed in each page. In order to facilitate traversal of the first application and save the identification result, a page library can be established in advance for the first application, each page being an object, saving a screenshot of the page, controls included in the page, for each control, also saving bounding box data of the control, effectiveness verification, and number of times the control is triggered. The bounding box data provides the size and page position of the control, and the effectiveness verification includes the following: the control satisfying the foregoing preset change condition is an effective control, that is, the foregoing first control, and the control not satisfying the foregoing preset change condition is an invalid control. There can also be unverified controls, that is, controls that have not been triggered.

[0071] Before each traversal operation, a screenshot of the current page is obtained, and the current page is matched with the pages in the page library through the screenshot similarity. If the matching fails, the current page is saved to the page library; if the matching with a page in the page library is successful, the information of the page is used to replace the information of the current page. According to a certain traversal strategy, a control to be clicked is selected, a click operation is performed on the control, and the button is labeled according to the subsequent changes. If the image of the entire current page changes significantly after clicking the button, or the image of the local area of the control changes significantly, or the control text in the page changes, the control is marked as an effective control, that is, the foregoing first control; otherwise, it is marked as an invalid control.

[0072] Based on this, the above identification result further includes: multiple first pages in the first application, and the first control in the first page; based on the total number of controls in the first page and the number of controls that are executed traversal operations, the traversal coverage rate of the first page is determined; the target first page with a traversal coverage rate higher than a preset coverage rate threshold, and the first control in the target first page, are determined as initial training data.

[0073] The recognition result includes a plurality of first pages, and a part of target first pages is filtered through traversal coverage, each target first page is taken as a training sample, and the first control indicated by the bounding box in the target first page is taken as the label data of the training sample. In this way, initial training data including a plurality of training samples can be obtained.

[0074] In actual implementation, the total number of controls of each first page and the number of controls on which the traversal operation is performed in the first page can be counted, and the ratio of the number of controls on which the traversal operation is performed to the total number of controls, that is, the traversal coverage of the first page, is obtained. The first page with high traversal coverage is taken as the target first page.

[0075] For example, for a first page with a traversal coverage of 100%, the page screenshot of the first page and the bounding box data of all first controls in the first page are saved. The page screenshot of a first page and the bounding box data of all first controls thereof constitute a training data. If the traversal efficiency is low, resulting in a small number of first pages with a traversal coverage of 100%, a page with a coverage close to 100% can be supplemented as a candidate page. For example, the aforementioned preset coverage threshold is set to 90%, and then manual triggering operation is performed on the unverified controls in the candidate page. If the aforementioned preset change condition is met, the control is taken as a first control, and the bounding box data is set to supplement the label. After the label of the candidate page is completed, the screenshot of the candidate page and the bounding box data of the controls are saved as a training data.

[0076] In order to improve the efficiency of manual supplement labeling, a matching labeling tool can be developed. If the control recognition accuracy is very high, the control can also be manually checked whether it meets the aforementioned preset change condition to improve the quality of the training data. Compared with the traditional target detection training data labeling scheme, this method can greatly save the labor cost.

[0077] After the intermediate model is obtained through the aforementioned initial training data, the intermediate model needs to be retrained. First, a page to be detected is obtained from a second application; the page to be detected is input into the intermediate model to detect the display area of a second control in the page to be detected through the intermediate model, and the confidence of the detected second control is output.

[0078] The second application program here can be the same as the first application program or different. In order to improve the generality of the model, the second application program is usually different from the first application program. The pages can be randomly obtained from the second application program, or the pages can be obtained from the second application program according to a certain traversal strategy, and the obtained pages are input into the intermediate model as the aforementioned to-be-detected pages. It can be understood that the intermediate model is trained by the initial training data and has a certain control detection accuracy. After the to-be-detected page is input into the intermediate model, the to-be-detected page can be recognized to contain a control, i.e., a second control. In actual implementation, the recognized second control can be identified in the form of a bounding box. Meanwhile, the intermediate model also outputs the confidence of each recognized second control.

[0079] In obtaining the to-be-detected page, in one manner, a home page of the second application program is obtained, and the home page is determined as the to-be-detected page. Alternatively, a to-be-traversed control is determined from the current to-be-detected page according to a preset traversal strategy, a triggering operation is performed on the to-be-traversed control, and a page displayed after the triggering operation is performed is determined as an updated to-be-detected page.

[0080] In the traversal process, starting from the home page, after a to-be-traversed control is determined from the home page, a page displayed after the to-be-traversed control is triggered is taken as an updated to-be-detected page, and the process is repeated, thereby obtaining a plurality of to-be-detected pages. The traversal strategy can be a depth-first search traversal strategy, a breadth-first search traversal strategy, a reinforcement learning-based traversal strategy, or the like.

[0081] In the process of traversing the second application program, a to-be-detected page is obtained from the second application program according to the traversal strategy, and the to-be-detected page is input into the intermediate model to output a corresponding detection result. In one specific implementation manner, the detection result includes a display region of the second control in the program page and a confidence of the second control. A target second control with a confidence higher than a preset confidence threshold is obtained from the detection result. A page where the target second control is located and a display region of the target second control in the page are determined. A bounding box data corresponding to the display region is generated, and the bounding box data and the page where the target second control is located are taken as updated training data. The updated training data is saved into the initial training data to obtain final training data.

[0082] For example, the confidence threshold can be set to 80%, 90%, or the like, so as to filter out second controls with high confidence. The page where the second control is located is taken as a training sample, and the bounding box data of all second controls on the page is taken as the labeling data of the training sample and saved into the initial training data. Thus, compared with the initial training data, the number of training samples in the final training data is increased, and the types of the training samples are also more diverse, which is conducive to making the model trained to have high generality and accuracy.

[0083] Further, after the control detection model is trained by the final training data, the detection accuracy of the control detection model is determined; if the detection accuracy is lower than a preset accuracy threshold, the following steps are continuously executed until the detection accuracy reaches the accuracy threshold: the control detection model is taken as an intermediate model, the second control in the second application program is detected by the intermediate model to obtain a detection result, the initial training data is updated based on the detection result to obtain the final training data; and the intermediate model is trained based on the final training data to obtain the control detection model.

[0084] In this way, the control detection model can be trained multiple times, and different second application programs can be detected each time, so that the training data becomes richer, and the generality and accuracy of the control detection model become higher and higher.

[0085] Corresponding to the embodiments of the training method of the control detection model, the following embodiments also provide a control detection method, as shown in the following table: Figure 4 The method comprises the following steps:

[0086] Step S402, a target page is determined from a target application program;

[0087] The target page can be randomly determined from the target application program, or the target page can be determined by traversing the target application program according to a traversal strategy.

[0088] Step S404, the target page is input into a control detection model that is pre-trained to obtain an output result; wherein the control detection model is trained by the training method of the control detection model; and the control indication identifier is included in the output result; and the control indication identifier indicates a display area of a detected control in the target page.

[0089] Step S406, a control detection result of the target page is determined based on the output result.

[0090] The control indication identifier can be bounding box data, each bounding box corresponds to a detected control, and the output result further includes a confidence corresponding to each control indication identifier, thereby indicating the confidence degree of the detected control. In actual implementation, the detected control corresponding to the control indication identifier with a confidence higher than a preset threshold can be taken as the final control detection result.

[0091] The control detection method determines a target page from a target application program, inputs the target page into a control detection model that is pre-trained, and obtains an output result. The control detection page is trained by the control detection model training method. The output result includes a control indication identifier. The control indication identifier indicates a display area of a detected control in the target page. The control detection result of the target page is determined based on the output result. The control detection model in the method is trained by initial training data to obtain an intermediate model. Based on the detection result of the intermediate model, the training data is updated. The updated final training data is used to continue training the intermediate model to obtain a final control detection model. The method can obtain rich training data at a low cost, thereby improving the universality and detection accuracy of the control detection model. Therefore, the control detection model can improve the accuracy of control detection.

[0092] The control detection model training method and the control detection method provided by the embodiment have high universality, speed and accuracy compared with the button recognition scheme based on controls in the related art. The self-training scheme of the control detection model greatly reduces the human cost of collecting training data. Large-scale training data can be obtained at a low cost, which helps to improve the precision of the button detection model and reduce the human cost of adapting to specific Apps.

[0093] As shown in Figure 5 The device includes:

[0094] The first data generation module 50 is configured to identify a first control in a first application program to obtain an identification result, and generate initial training data based on the identification result. The identification result includes the position of the first control in a page of the first application program.

[0095] The first training module 52 is configured to train a preset initial model based on the initial training data to obtain an intermediate model.

[0096] The second data generation module 54 is configured to detect a second control in a second application program by the intermediate model to obtain a detection result, update the initial training data based on the detection result to obtain final training data. The detection result includes the position of the second control in a page of the second application program.

[0097] The second training module 56 is configured to train the intermediate model based on the final training data to obtain a control detection model.

[0098] The training device of the control detection model, identifies a first control in a first application program, obtains an identification result; generates initial training data based on the identification result; wherein the identification result includes: the position of the first control in the page of the first application program; train the preset initial model based on the initial training data, obtain an intermediate model; detect a second control in a second application program through the intermediate model, obtain a detection result, update the initial training data based on the detection result, obtain final training data; wherein the detection result includes: the position of the second control in the page of the second application program; train the intermediate model based on the final training data, obtain the control detection model. In this way, after the intermediate model is obtained by training based on the initial training data, the training data is updated based on the detection result of the intermediate model, and then the updated final training data is used to continue training the intermediate model to obtain the final control detection model. This way can obtain rich training data at a lower human cost, so that the control detection model has high universality and detection accuracy.

[0099] The first data generation module is configured to: determine a first page from a first application program; identify a control in the first page based on page information of the first page; wherein the page information includes: a page node in the first page, and a dependency relationship between the page nodes; perform an iteration operation on the control in the first page based on a preset iteration strategy, to obtain a first control in the first page that meets a preset control condition; continue to perform the step of determining the first page from the first application program until the iteration of the first application program is completed, to obtain an identification result of the first control of the first application program.

[0100] The first data generation module is configured to: perform control identification on the first page based on the page information of the first page, to obtain an initial identification result; perform screening on the control in the initial identification result based on the relative positions of the controls in the initial identification result, to obtain a final identification result.

[0101] The first data generation module is configured to: if the area of the control display area of the third control is greater than the area of the control display area of the fourth control, and the center position of the control display area of the third control is located in the control display area of the fourth control in the initial identification result, delete the third control from the initial identification result, to obtain the final identification result.

[0102] The first data generation module is configured to: determine an iteration order of the control in the first page based on a preset iteration strategy, and perform a trigger operation on the control in the first page according to the iteration order; determine a first control in the first page that meets a preset control condition based on the first page content of the first page before the trigger operation is performed and the second page content of the first page after the trigger operation is performed; wherein the preset control condition includes: the content change degree of the second page content relative to the first page content meets a preset change condition.

[0103] The preset change condition at least includes one of the following: a page overall change degree of the second page content relative to the first page content is higher than a first preset threshold; a change degree of the second page content relative to the first page content in a local page region where the first control is located is higher than a second preset threshold; and the second page content changes with control text in the first page content.

[0104] The identification result further includes a plurality of first pages in the first application program and the first control in the first page; and the first data generation module is configured to: determine a traversal coverage rate of the first page based on a total amount of controls of the first page and a number of controls on which the traversal operation is performed; and determine a target first page with a traversal coverage rate higher than a preset coverage rate threshold and the first control in the target first page as initial training data.

[0105] The second data generation module is further configured to: obtain a to-be-detected page from the second application program; and input the to-be-detected page into the intermediate model to detect a display region of the second control in the to-be-detected page through the intermediate model and output a confidence of the detected second control.

[0106] The second data generation module is further configured to: obtain a home page of the second application program and determine the home page as the to-be-detected page; or determine a to-be-traversed control from the current to-be-detected page according to a preset traversal strategy, perform a trigger operation on the to-be-traversed control, and determine a page displayed after the trigger operation is performed as an updated to-be-detected page.

[0107] The detection result includes a display region of the second control in the program page and the confidence of the second control; and the second data generation module is further configured to: obtain a target second control with a confidence higher than a preset confidence threshold from the detection result; determine a page where the target second control is located and a display region of the target second control in the page; generate bounding box data corresponding to the display region, take the bounding box data and the page where the target second control is located as updated training data, save the updated training data into the initial training data, and obtain final training data.

[0108] The apparatus further includes a continuous training module configured to: determine a detection accuracy of the control detection model; and if the detection accuracy is lower than a preset accuracy threshold, continuously perform the following steps until the detection accuracy reaches the accuracy threshold: take the control detection model as the intermediate model, take the final training data as the initial training data, detect the second control in the second application program through the intermediate model to obtain a detection result, update the initial training data based on the detection result to obtain the final training data, and train the intermediate model based on the final training data to obtain the control detection model.

[0109] Referring to Figure 6 Fig. 1 shows a structural schematic diagram of a control detection device according to an embodiment of the present application. The device comprises:

[0110] a page determination module 60 configured to determine a target page from a target application program;

[0111] a detection module 62 configured to input the target page into a control detection model trained in advance to obtain an output result; wherein the control detection page is obtained through training of the control detection model; and the output result comprises a control indication identifier; and the control indication identifier indicates a display area of a detected control in the target page;

[0112] a result determination module 64 configured to determine a control detection result of the target page based on the output result.

[0113] The control detection device determines a target page from a target application program; inputs the target page into a control detection model trained in advance to obtain an output result; wherein the control detection page is obtained through a training method of the control detection model; the output result comprises a control indication identifier; the control indication identifier indicates a display area of a detected control in the target page; and a control detection result of the target page is determined based on the output result. In the method, an intermediate model is obtained through training of initial training data, the training data is updated based on a detection result of the intermediate model, and the intermediate model is further trained using the updated final training data to obtain a final control detection model. The method can obtain rich training data at a low cost, so that the control detection model has high universality and detection accuracy. Therefore, the control detection model can improve the accuracy of control detection.

[0114] The embodiment also provides an electronic device comprising a processor and a memory. The memory stores machine executable instructions capable of being executed by the processor. The processor executes the machine executable instructions to implement the training method of the control detection model or the control detection method. The electronic device can be a server or a terminal device.

[0115] Referring to Figure 7 The electronic device comprises a processor 100 and a memory 101. The memory 101 stores machine executable instructions capable of being executed by the processor 100. The processor 100 executes the machine executable instructions to implement the training method of the control detection model or the control detection method.

[0116] Further, Figure 7 The electronic device further comprises a bus 102 and a communication interface 103. The processor 100, the communication interface 103 and the memory 101 are connected through the bus 102.

[0117] The memory 101 can include a random access memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used. The bus 102 can be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the figure to represent the system, but it does not mean that there is only one bus or one type of bus.

[0118] The processor 100 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 100 or the instructions in the form of software. The processor 100 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block disclosed in the embodiment of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiment of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101, and combines the hardware to complete the steps of the method of the above embodiment.

[0119] The processor in the above electronic device can realize the following operations in the control detection model training method or control detection method by executing machine executable instructions:

[0120] The first control in the first application program is identified to obtain an identification result; initial training data is generated based on the identification result; wherein the identification result includes: the position of the first control in the page of the first application program; an initial model is trained based on the initial training data to obtain an intermediate model; a second control in a second application program is detected through the intermediate model to obtain a detection result, and the initial training data is updated based on the detection result to obtain final training data; wherein the detection result includes: the position of the second control in the page of the second application program; the intermediate model is trained based on the final training data to obtain a control detection model.

[0121] A first page is determined from a first application program; a control in the first page is identified based on page information of the first page; wherein the page information includes: a page node in the first page, and a dependency relationship between the page nodes; the control in the first page is operated based on a preset traversal strategy to obtain a first control in the first page that meets a preset control condition; the step of determining the first page from the first application program is continuously executed until the first application program is traversed to obtain an identification result of the first control of the first application program.

[0122] The first page is identified based on the page information of the first page to obtain an initial identification result; the control in the initial identification result is screened based on the relative position of the control to obtain a final identification result.

[0123] If the area of the control display area of the third control is greater than the area of the control display area of the fourth control in the initial identification result, and the center position of the control display area of the third control is located in the control display area of the fourth control, the third control is deleted from the initial identification result to obtain the final identification result.

[0124] The traversal order of the control in the first page is determined based on a preset traversal strategy, and the control in the first page is triggered according to the traversal order; the first control in the first page that meets the preset control condition is determined based on the first page content of the first page before the triggering operation is executed and the second page content of the first page after the triggering operation is executed; wherein the preset control condition includes: the content change degree of the second page content relative to the first page content meets a preset change condition.

[0125] The preset change condition at least includes one of the following: the overall page change degree of the second page content relative to the first page content is higher than a first preset threshold; the change degree of the second page content relative to the first page content in the local page area where the first control is located is higher than a second preset threshold; the control text in the second page content changes from the first page content.

[0126] The identification result further includes: a plurality of first pages in the first application program, and a first control in the first page; based on a total amount of controls of the first page and a number of controls on which the traversal operation is performed, a traversal coverage of the first page is determined; a target first page with a traversal coverage higher than a preset coverage threshold and a first control in the target first page are determined as initial training data.

[0127] The second application program obtains a to-be-detected page; the to-be-detected page is input into the intermediate model to detect a display area of the second control in the to-be-detected page through the intermediate model, and output a confidence of the detected second control.

[0128] The first page of the second application program is obtained, and the first page is determined as the to-be-detected page; or, according to a preset traversal strategy, a to-be-traversed control is determined from the current to-be-detected page, a triggering operation is performed on the to-be-traversed control, and a page displayed after the triggering operation is performed is determined as an updated to-be-detected page.

[0129] The detection result includes: a display area of the second control in the program page, and a confidence of the second control; a target second control with a confidence higher than a preset confidence threshold is obtained from the detection result; a page where the target second control is located and a display area of the target second control in the page are determined; a bounding box data corresponding to the display area is generated, the bounding box data and the page where the target second control is located are taken as updated training data, the updated training data is saved into the initial training data, and final training data is obtained.

[0130] The detection accuracy of the control detection model is determined; if the detection accuracy is lower than a preset accuracy threshold, the following steps are continuously performed until the detection accuracy reaches the accuracy threshold: the control detection model is taken as an intermediate model, the final training data is taken as initial training data, the second control in the second application program is detected through the intermediate model to obtain a detection result, the initial training data is updated based on the detection result to obtain final training data; the intermediate model is trained based on the final training data to obtain the control detection model.

[0131] A target page is determined from a target application program; the target page is input into a control detection model that is pre-trained to obtain an output result; the control detection model is trained by the training method of the control detection model; the output result includes a control indication identifier; the control indication identifier indicates a display area of a detected control in the target page; based on the output result, a control detection result of the target page is determined.

[0132] The control detection model in the above manner is obtained by training an intermediate model through initial training data, updating the training data based on a detection result of the intermediate model, and then continuing to train the intermediate model using the updated final training data to obtain a final control detection model. This manner can obtain rich training data at a lower human cost, so that the control detection model has higher universality and detection accuracy, and thus the accuracy of control detection can be improved by detecting the control through the control detection model.

[0133] The embodiment also provides a machine readable storage medium storing machine executable instructions. When the machine executable instructions are invoked and executed by a processor, the machine executable instructions cause the processor to implement the training method of the control detection model or the control detection method.

[0134] The machine executable instructions stored in the machine readable storage medium can implement the following operations in the training method of the control detection model or the control detection method by executing the machine executable instructions:

[0135] recognize a first control in a first application program to obtain a recognition result, generate initial training data based on the recognition result, wherein the recognition result comprises a position of the first control in a page of the first application program, train a preset initial model based on the initial training data to obtain an intermediate model, detect a second control in a second application program through the intermediate model to obtain a detection result, update the initial training data based on the detection result to obtain final training data, wherein the detection result comprises a position of the second control in a page of the second application program, and train the intermediate model based on the final training data to obtain a control detection model.

[0136] determine a first page from a first application program, recognize a control in the first page based on page information of the first page, wherein the page information comprises a page node in the first page and a dependency relationship between the page nodes, perform an iteration operation on the control in the first page based on a preset iteration strategy to obtain a first control in the first page that meets a preset control condition, and continue to perform the step of determining the first page from the first application program until the first application program is iterated completely to obtain a recognition result of the first control of the first application program.

[0137] perform control recognition on the first page based on the page information of the first page to obtain an initial recognition result, and perform screening on the control in the initial recognition result based on a relative position of the control in the initial recognition result to obtain a final recognition result.

[0138] If the area of the control display region of the third control is greater than the area of the control display region of the fourth control in the initial recognition result, and the center position of the control display region of the third control is located in the control display region of the fourth control, the third control is deleted from the initial recognition result to obtain the final recognition result.

[0139] Based on the preset traversal strategy, the traversal order of the controls in the first page is determined, and the controls in the first page are triggered according to the traversal order; based on the first page content before the triggering operation is executed and the second page content after the triggering operation is executed, the first control in the first page that meets the preset control condition is determined; wherein the preset control condition includes that the content change degree of the second page content relative to the first page content meets the preset change condition.

[0140] The preset change condition at least includes one of the following: the overall page change degree of the second page content relative to the first page content is higher than a first preset threshold; the change degree of the second page content relative to the first page content in the local page region where the first control is located is higher than a second preset threshold; the control text in the second page content changes from the first page content.

[0141] The recognition result further includes: a plurality of first pages in the first application program, and the first control in the first page; based on the total amount of controls of the first page and the number of controls subjected to the traversal operation, the traversal coverage rate of the first page is determined; the target first page whose traversal coverage rate is higher than a preset coverage rate threshold, and the first control in the target first page are determined as the initial training data.

[0142] The second application program is obtained from the second application program; the detection page is input into the intermediate model to detect the display region of the second control in the detection page through the intermediate model, and the confidence of the detected second control is output.

[0143] The home page of the second application program is obtained, and the home page is determined as the detection page; or, according to the preset traversal strategy, the control to be traversed is determined from the current detection page, the triggering operation is performed on the control to be traversed, and the page displayed after the triggering operation is performed is determined as the updated detection page.

[0144] The detection result includes: the display region of the second control in the program page, and the confidence of the second control; the target second control whose confidence is higher than a preset confidence threshold is obtained from the detection result; the page where the target second control is located and the display region of the target second control in the page are determined; the bounding box data corresponding to the display region is generated, the bounding box data and the page where the target second control is located are taken as the updated training data, the updated training data is saved into the initial training data, and the final training data is obtained.

[0145] determining the detection accuracy of the control detection model; if the detection accuracy is lower than a preset accuracy threshold, the following steps are continuously executed until the detection accuracy reaches the accuracy threshold: taking the control detection model as an intermediate model, taking the final training data as initial training data, detecting a second control in a second application program through the intermediate model to obtain a detection result, updating the initial training data based on the detection result to obtain the final training data; and training the intermediate model based on the final training data to obtain the control detection model.

[0146] determining a target page from a target application program; inputting the target page into the control detection model trained in advance to obtain an output result; wherein the control detection page is obtained through the training method of the control detection model; the control indication identifier is included in the output result; the control indication identifier indicates the display area of the detected control in the target page; and the control detection result of the target page is determined based on the output result.

[0147] In the above manner, after the intermediate model is trained through the initial training data, the training data is updated based on the detection result of the intermediate model, and then the intermediate model is continuously trained using the updated final training data to obtain the final control detection model. This manner can obtain rich training data at a lower human cost, so that the control detection model has higher universality and detection accuracy, and thus the accuracy of control detection can be improved through the control detection model.

[0148] The training method of the control detection model, the control detection method, the device and the computer program product of the electronic equipment provided in the embodiments of the present application include a machine-readable storage medium storing program codes, the instructions included in the program codes can be used to execute the methods described in the foregoing method embodiments, and the specific implementation can be referred to the method embodiments, which will not be described herein.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and device can refer to the corresponding process in the foregoing method embodiments, which will not be described herein.

[0150] In addition, in the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection" and "connection" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through an intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0151] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a machine readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0152] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0153] Finally, it should be noted that the above embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, and are not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training a control detection model, the method comprising: The method comprises: identifying a first control in a first application program to obtain an identification result; generating initial training data based on the identification result; wherein the identification result comprises a position of the first control in a page of the first application program; training a preset initial model based on the initial training data to obtain an intermediate model; detecting a second control in a second application program through the intermediate model to obtain a detection result, updating the initial training data based on the detection result to obtain final training data; wherein the detection result comprises a position of the second control in a page of the second application program; training the intermediate model based on the final training data to obtain a control detection model; The step of identifying a first control in a first application program to obtain an identification result comprises: determining a first page from the first application program; determining an iteration order of the controls in the first page based on a preset iteration strategy, and performing a trigger operation on the controls in the first page according to the iteration order; determining a first control in the first page that meets a preset control condition based on first page content of the first page before the trigger operation is performed and second page content of the first page after the trigger operation is performed; wherein the preset control condition comprises that a content change degree of the second page content relative to the first page content meets a preset change condition.

2. The method of claim 1, wherein, The step of identifying a first control in a first application program to obtain an identification result comprises: identifying the controls in the first page based on page information of the first page; wherein the page information comprises page nodes in the first page and a dependency relationship between the page nodes; performing an iteration operation on the controls in the first page based on a preset iteration strategy to obtain a first control in the first page that meets a preset control condition; continuing to perform the step of determining a first page from the first application program until the first application program is iterated to completion to obtain an identification result of the first control of the first application program.

3. The method of claim 2, wherein, The step of identifying the controls in the first page based on page information of the first page comprises: performing control identification on the first page based on the page information of the first page to obtain an initial identification result; performing screening on the controls in the initial identification result based on relative positions of the controls in the initial identification result to obtain a final identification result.

4. The method of claim 3, wherein, The step of performing screening on the controls in the initial identification result based on relative positions of the controls in the initial identification result to obtain a final identification result comprises: if the area of the control display area of the third control is greater than the area of the control display area of the fourth control in the initial identification result, and the center position of the control display area of the third control is located in the control display area of the fourth control, the third control is deleted from the initial identification result to obtain a final identification result.

5. The method of claim 1, wherein, The preset change condition at least comprises one of: a page overall change degree of the second page content relative to the first page content is higher than a first preset threshold value; The degree of change of the second page content relative to the first page content in the local page region where the first control is located is higher than a second preset threshold value; The second page content and the control text in the first page content change.

6. The method of claim 1, wherein, The identification result further includes a plurality of first pages in the first application program and first controls in the first pages; the step of generating initial training data based on the identification result includes: determining a traversal coverage rate of the first page based on a total amount of controls of the first page and a number of controls on which the traversal operation is performed; determining a target first page in which the traversal coverage rate is higher than a preset coverage rate threshold value and first controls in the target first page as the initial training data.

7. The method of claim 1, wherein, The step of detecting second controls in a second application program through the intermediate model to obtain a detection result includes: obtaining a page to be detected from the second application program; inputting the page to be detected into the intermediate model to detect a display region of the second controls in the page to be detected through the intermediate model and output a confidence of the detected second controls.

8. The method of claim 7, wherein, The step of obtaining a page to be detected from the second application program includes: obtaining a home page of the second application program and determining the home page as the page to be detected; or, determining a control to be traversed from a current page to be detected according to a preset traversal strategy, performing a trigger operation on the control to be traversed, and determining a page displayed after the trigger operation is performed as an updated page to be detected.

9. The method of claim 1, wherein, The detection result includes a display region of the second controls in a program page and a confidence of the second controls; the step of updating the initial training data based on the detection result to obtain final training data includes: obtaining a target second control with a confidence higher than a preset confidence threshold value from the detection result; determining a page in which the target second control is located and a display region of the target second control in the page; generating bounding box data corresponding to the display region, taking the bounding box data and the page in which the target second control is located as updated training data, saving the updated training data into the initial training data to obtain the final training data.

10. The method of claim 1, wherein, After the step of training the intermediate model based on the final training data to obtain a control detection model, the method further includes: determining a detection accuracy of the control detection model; if the detection accuracy is lower than a preset accuracy threshold value, the following steps are continuously performed until the detection accuracy reaches the accuracy threshold value: taking the control detection model as an intermediate model, taking the final training data as initial training data, detecting second controls in a second application program through the intermediate model to obtain a detection result, updating the initial training data based on the detection result to obtain final training data, and training the intermediate model based on the final training data to obtain a control detection model.

11. A control detection method, characterized by, The method includes: determining a target page from a target application program; input the target page into a control detection model trained in advance to obtain an output result; wherein the control detection page is trained by the training method of the control detection model according to any one of claims 1-10; the output result includes a control indication mark; the control indication mark indicates a display area of a detected control in the target page; based on the output result, determine a control detection result of the target page.

12. A training device for a control detection model, characterized in that, The apparatus comprises: a first data generation module configured to identify a first control in a first application program to obtain an identification result, and generate initial training data based on the identification result; wherein the identification result includes a position of the first control in a page of the first application program; a first training module configured to train a preset initial model based on the initial training data to obtain an intermediate model; a second data generation module configured to detect a second control in a second application program by using the intermediate model to obtain a detection result, and update the initial training data based on the detection result to obtain final training data; wherein the detection result includes a position of the second control in a page of the second application program; a second training module configured to train the intermediate model based on the final training data to obtain a control detection model; The first data generation module is further configured to determine a first page from the first application program, determine an iteration order of controls in the first page based on a preset iteration strategy, perform a trigger operation on the controls in the first page according to the iteration order, determine a first control in the first page that meets a preset control condition based on a first page content of the first page before the trigger operation is performed and a second page content of the first page after the trigger operation is performed; wherein the preset control condition includes that a content change degree of the second page content relative to the first page content meets a preset change condition.

13. A control detection apparatus, characterized by comprising: The apparatus comprises: a page determination module configured to determine a target page from a target application program; a detection module configured to input the target page into a control detection model trained in advance to obtain an output result; wherein the control detection page is trained by the training apparatus of the control detection model according to any one of claims 1-11; the output result includes a control indication mark; the control indication mark indicates a display area of a detected control in the target page; a result determination module configured to determine a control detection result of the target page based on the output result.

14. An electronic device, comprising: An apparatus includes a processor and a memory storing machine executable instructions executable by the processor to implement the training method of the control detection model according to any one of claims 1-10, or implement the control detection method according to claim 11.

15. A machine-readable storage medium, characterized in that, The machine readable storage medium stores machine executable instructions which, when invoked and executed by a processor, cause the processor to implement the control detection model training method of any one of claims 1-10, or implement the control detection method of claim 11.

Images