Unique control area positioning model training and positioning method, device and equipment

Through the reinforcement learning method, data annotation is eliminated and training data is obtained using interactive screenshots, which solves the problem of large workload of data annotation in supervised learning, and realizes efficient control area positioning model training, and improves the writing efficiency of automated test scripts.

CN117454968BActive Publication Date: 2024-11-26BEIJING URBAN CONSTR INTELLIGENT CONTROL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311435873.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-31
Publication Date
2024-11-26
Estimated Expiration
2043-10-31

AI Technical Summary

Technical Problem

In the prior art, model training based on supervised learning requires a large amount of data labeling, resulting in a large amount of data labeling work, which seriously reduces the training efficiency of the model.

Method used

Using reinforcement learning method, the training data is obtained through interactive screenshots, the data annotation process is eliminated, and the status information, positioning information and rewards of the target control area are used as training data, and the initial policy network is iterated until the iteration termination condition is met to form a control area positioning model. .

Benefits of technology

It improves the training efficiency of the model and can automatically intercept the control area that is uniquely recognized and includes the target control area, thereby improving the writing efficiency of automated test scripts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117454968B_ABST
    Figure CN117454968B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of artificial intelligence, and provides a unique control area positioning model training and positioning method, device and equipment, the training method comprising: obtaining an initial policy network, and a first interface image containing a target control area; using the first state information of the target control area, the first positioning information, the reward of the first positioning information and the second state information of the target demarcated area as a set of training data; based on multiple sets of training data, using a reinforcement learning method to iterate the parameters of the initial policy network until the iteration termination condition is met, and using the initial policy network after the parameter iteration as the control area positioning model. The unique control area positioning model training and positioning method, device and equipment provided by the present invention obtains multiple sets of training data by means of interactive screenshots, thereby eliminating the data labeling process, and can automatically capture the control area with unique identifiability and containing the target control area, thereby improving the writing efficiency of automated test scripts.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a unique control area positioning model training and positioning method, device and equipment. Background Art

[0002] In the application process of automated testing tools based on image recognition (such as Airtest), the formation of an automated test script for a test case requires capturing images of control areas involved in the user interface (UI) of the test.

[0003] In related technologies, model training methods based on supervised learning require a large amount of labeled data. However, the workload of data labeling is large, which seriously reduces the training efficiency of the model. Summary of the invention

[0004] The present invention provides a unique control area positioning model training and positioning method, device and equipment to solve the defects in the prior art that the workload of data annotation is large and the training efficiency of the model is seriously reduced.

[0005] The present invention provides a unique control area positioning model training method, comprising:

[0006] Obtaining an initial strategy network and a first interface image including a target control area;

[0007] Determine the first state information of the target control area based on the position information and uniqueness information of the target control area in the first interface image and the image features of the target control area; the uniqueness information indicates whether the target control area is uniquely identifiable in the first interface image;

[0008] Determining first positioning information based on the first state information of the target control area and the initial strategy network;

[0009] Based on the first positioning information, the control area of ​​the first interface image is positioned to obtain a target demarcated area, and based on the position information and uniqueness information of the target demarcated area in the first interface image, a reward for the first positioning information is determined;

[0010] Determine second state information of the target defined area based on position information and uniqueness information of the target defined area in the first interface image and image features of the target defined area;

[0011] The first state information of the target control area, the first positioning information, the reward of the first positioning information, and the second state information of the target demarcated area are used as a set of training data;

[0012] Based on multiple groups of training data, a reinforcement learning method is used to iterate the parameters of the initial strategy network until an iteration termination condition is met, and the initial strategy network after completing the parameter iteration is used as a control area positioning model.

[0013] According to the unique control area positioning model training method provided by the present invention, the first positioning information is determined based on the first state information of the target control area and the initial strategy network, including:

[0014] Inputting the first state information of the target control area into the initial strategy network to obtain initial positioning information output by the initial strategy network;

[0015] The normal distribution is constructed by taking the initial positioning information as the average value of the normal distribution, and the first positioning information is randomly generated from the normal distribution.

[0016] According to the unique control area positioning model training method provided by the present invention, the first state information of the target control area is determined based on the position information and uniqueness information of the target control area in the first interface image and the image features of the target control area, including:

[0017] Determine a target control area, and position information and uniqueness information of the target control area in the first interface image;

[0018] Extracting features from the first interface image to obtain first interface image features;

[0019] Based on the position information of the target control area in the first interface image and the first interface image features, image features of the target control area are extracted to obtain image features of the target control area;

[0020] The position information and uniqueness information of the target control area in the first interface image and the image features of the target control area are spliced ​​to obtain the first state information of the target control area.

[0021] According to the unique control area positioning model training method provided by the present invention, the reward for the first positioning information is determined based on the position information and uniqueness information of the target demarcated area in the first interface image, including:

[0022] Matching the target defined area with other control areas in the first interface image, and determining unique information of the target defined area in the first interface image based on the matching result;

[0023] Based on the position information of the target delimited area in the first interface image, determining the ratio of the length and height of the target delimited area to the length and height of the target control area;

[0024] Based on the uniqueness information of the target demarcated area in the first interface image and the ratio of the length and height of the target demarcated area to the length and height of the target control area, the reward for the first positioning information is determined.

[0025] According to the unique control area positioning model training method provided by the present invention, the reward of the first positioning information is determined based on the unique information of the target demarcated area in the first interface image and the ratio of the length and height of the target demarcated area to the length and height of the target control area, including:

[0026] The reward for the first positioning information is determined based on the following formula:

[0027]

[0028] Wherein, G is a constant, λ1 is the ratio of the length of the target demarcation area to the length of the target control area, λ2 is the ratio of the height of the target demarcation area to the height of the target control area, and both λ1 and λ2 are greater than or equal to 1.

[0029] According to the unique control area positioning model training method provided by the present invention, based on the multiple sets of training data, the initial strategy network is iterated by a reinforcement learning method until the iteration termination condition is met, including:

[0030] For any set of training data among the multiple sets of training data, input the first positioning information and the first state information in the any set of training data into the initial value network to obtain the first positioning value output by the initial value network;

[0031] Inputting the second state information in any one set of training data into the target policy network to obtain the second positioning information output by the target policy network, inputting the second positioning information and the second state information into the target value network to obtain the second positioning value output by the target value network;

[0032] Based on the second positioning value, the reward, and the first positioning value, performing parameter iteration on the initial value network to obtain a value network after parameter iteration;

[0033] Inputting the first positioning information and the first state information in any one set of training data into the value network after parameter iteration to obtain a third positioning value output by the value network after parameter iteration;

[0034] Based on the third positioning value, parameters of the initial strategy network are iterated until an iteration termination condition is met.

[0035] According to the unique control area positioning model training method provided by the present invention, the parameter iteration of the initial value network based on the second positioning value, the reward and the first positioning value includes:

[0036] Add and subtract the second positioning value, the reward and the first positioning value to obtain a value loss;

[0037] Based on the value loss, parameters of the initial value network are iterated.

[0038] The present invention also provides a unique control area positioning method, comprising:

[0039] Acquire a second interface image and a control area to be located in the second interface image;

[0040] Determine third state information of the control area to be located based on the position information and uniqueness information of the control area to be located in the second interface image and the image features of the control area to be located;

[0041] The third state information is input into the control area positioning model to obtain the third positioning information output by the control area positioning model, and the control area of ​​the second interface image is positioned based on the third positioning information. The control area positioning model is trained based on the unique control area positioning model training method.

[0042] The present invention also provides a unique control area positioning model training device, comprising:

[0043] A first acquisition unit, used to acquire an initial strategy network and a first interface image including a target control area;

[0044] A first state determination unit, configured to determine first state information of the target control area based on position information and uniqueness information of the target control area in the first interface image and image features of the target control area; the uniqueness information indicates whether the target control area is uniquely identifiable in the first interface image;

[0045] A positioning information determining unit, which determines first positioning information based on the first state information of the target control area and the initial strategy network;

[0046] a reward determination unit, configured to locate a control area of ​​the first interface image based on the first positioning information to obtain a target demarcated area, and determine a reward for the first positioning information based on position information and uniqueness information of the target demarcated area in the first interface image;

[0047] a second state determining unit, configured to determine second state information of the target defined area based on position information and uniqueness information of the target defined area in the first interface image and image features of the target defined area;

[0048] A training data acquisition unit, which uses the first state information of the target control area, the first positioning information, the reward of the first positioning information, and the second state information of the target demarcated area as a set of training data;

[0049] A parameter iteration unit is used to perform parameter iteration on the initial strategy network based on multiple groups of training data using a reinforcement learning method until an iteration termination condition is met, and use the initial strategy network after parameter iteration as a control area positioning model.

[0050] The present invention also provides a unique control area positioning device, comprising:

[0051] A second acquisition unit, used to acquire a second interface image and a control area to be located in the second interface image;

[0052] A third state determining unit, configured to determine third state information of the control area to be located based on the position information and uniqueness information of the control area to be located in the second interface image and the image features of the control area to be located;

[0053] An area positioning unit is used to input the third state information into a control area positioning model to obtain third positioning information output by the control area positioning model, and to perform control area positioning on the second interface image based on the third positioning information. The control area positioning model is trained based on the unique control area positioning model training method.

[0054] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any one of the above-described unique control area positioning model training methods or unique control area positioning methods.

[0055] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements any of the unique control area positioning model training methods or unique control area positioning methods described above.

[0056] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the unique control area positioning model training methods or unique control area positioning methods described above.

[0057] The unique control area positioning model training and positioning method, device and equipment provided by the present invention locates the control area of ​​the first interface image containing the target control area to obtain the target demarcated area, and uses the first state information, first positioning information, reward of the first positioning information and second state information of the target control area as a set of training data, and obtains multiple sets of training data by interactive screenshots, thereby eliminating the data labeling process. The control area positioning model obtained by the reinforcement learning method can automatically capture the control area that is uniquely identifiable and contains the target control area, thereby improving the efficiency of writing automated test scripts. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 It is a flow chart of the unique control area positioning model training method provided by the present invention;

[0060] Figure 2 is a schematic diagram of a process for determining first state information provided by the present invention;

[0061] Figure 3 is a schematic diagram of a determination process of the second state information provided by the present invention;

[0062] Figure 4 It is a schematic diagram of a reward determination process of the first positioning information provided by the present invention;

[0063] Figure 5 It is a schematic diagram of the initial strategy network training process provided by the present invention;

[0064] Figure 6 It is a flowchart of the unique control area positioning method provided by the present invention;

[0065] Figure 7 It is a structural schematic diagram of a unique control area positioning model training device provided by the present invention;

[0066] Figure 8 It is a structural schematic diagram of a unique control area positioning device provided by the present invention;

[0067] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

[0070] In the application process of automated testing tools based on image recognition (such as Airtest), the formation of an automated test script for a test case requires manual capture of images of the control areas involved in the test UI.

[0071] However, there may be multiple controls with the same content in the software user interface. In order to prevent the test tool from detecting multiple target images at the same time, the tester needs to repeatedly capture images of the area surrounding the control to ensure that the test tool can uniquely identify the operation control and perform the test operation.

[0072] In related technologies, model training methods based on supervised learning require a large amount of labeled data. However, the workload of data labeling is large, which seriously reduces the training efficiency of the model.

[0073] Based on the above considerations, in order to improve the training efficiency of the model, the inventive concept of the present invention is: by adopting the reinforcement learning method, the data labeling process can be omitted, and the training data can be obtained by interactive screenshots, thereby improving the training efficiency of the model.

[0074] Based on the above-mentioned inventive concept, the present invention provides a unique control area positioning model training and positioning method, device and equipment, which are applied to control area positioning scenarios in artificial intelligence technology, such as automatic test case writing scenarios that require control area images, to improve the efficiency of writing automated test scripts.

[0075] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings. Figure 1The flowchart of the unique control area positioning model training method provided by the present invention is shown in FIG. The execution subject of each step in the method can be a unique control area positioning model training device, which can be implemented by software and / or hardware. The device can be integrated in an electronic device, which can be a terminal device (such as a smart phone, a personal computer, etc.), a server (such as a local server or a cloud server, or a server cluster, etc.), a processor, or a chip, etc. Figure 1 As shown, the method may include the following steps:

[0076] Step 110: acquiring an initial policy network and a first interface image including a target control area.

[0077] Specifically, the first interface image may be a test UI that needs to be automated tested, and the first interface image includes multiple control areas. The target control area may be any one or more control areas in the first interface image, such as controls such as "OK" and "Cancel".

[0078] The initial policy network is a neural network that can make correct actions based on feedback from the environment. In the control area positioning scenario, the initial policy network can be used to control positioning, that is, the positioning information determined by the initial policy network can be used to intercept the target area including the target control area.

[0079] Step 120, based on the position information and uniqueness information of the target control area in the first interface image, and the image features of the target control area, determine the first state information of the target control area; the uniqueness information represents whether the target control area is uniquely identifiable in the first interface image.

[0080] Specifically, by performing target control detection on the first interface image, position information of the target control area in the first interface image can be obtained, and the position information here can be represented by coordinates.

[0081] The uniqueness information is used to characterize whether the target control area is uniquely identifiable in the first interface image. The uniqueness information can reflect whether there are distinguishing points and differences between the target control area and other control areas in the first interface image. If there are distinguishing points and differences, and the target control area can be uniquely identified through the distinguishing points and differences, it means that the target control area is uniquely identifiable; otherwise, it means that the target control area is not uniquely identifiable.

[0082] The image features of the target control area represent the image features of each pixel in the target control area, as well as the associated features between the pixels. The image features can be obtained through an image feature extraction network. The first interface image can be first segmented to obtain the target control area, and then the target control area can be input into the image feature extraction network to obtain the image features of the target control area output by the image feature extraction network; or the first interface image can be first input into the image feature extraction network to obtain the image features of the first interface image output by the image feature extraction network, and then the image features of the target control area can be obtained based on the position information of the target control area in the first interface image. The embodiment of the present invention does not specifically limit this.

[0083] After obtaining the position information and uniqueness information of the target control area in the first interface image, as well as the image features of the target control area, the position information, uniqueness information and image features may be fused to obtain the first state information of the target control area.

[0084] Step 130, determining first positioning information based on the first state information of the target control area and the initial strategy network.

[0085] Specifically, the first positioning information reflects the automatic capture position, and the area including the target control area can be automatically captured according to the first positioning information.

[0086] With respect to determining the first positioning information, the first state information of the target control area may be input into the initial strategy network, where the positioning information output by the initial strategy network may be directly used as the first positioning information; or data processing may be performed on the positioning information output by the initial strategy network, such as constructing a normal distribution, and then randomly generating positioning information based on the normal distribution as the first positioning information, which is not specifically limited in the embodiments of the present invention.

[0087] Step 140, locate the control area of ​​the first interface image based on the first positioning information to obtain a target demarcated area, and determine the reward of the first positioning information based on the position information and uniqueness information of the target demarcated area in the first interface image.

[0088] Specifically, the target defined area refers to an area that includes the target control area and is uniquely identifiable in the first interface image. The target defined area can be a rectangular area, and the first positioning information can be four coordinate points in the first interface image, which are the upper left, lower left, upper right and lower right coordinate points of the target defined area.

[0089] After obtaining the target defined area, it can be further determined whether the target defined area is uniquely identifiable in the first interface image, that is, the uniqueness information of the target defined area in the first interface image, where the uniqueness information may include unique or non-unique. The position information of the target defined area in the first interface image can be determined according to the first positioning information.

[0090] In reinforcement learning, the evaluation criteria for the first positioning information are defined by rewards. In the control area positioning scenario, the ultimate goal is to obtain a demarcated area that is uniquely identifiable and contains the target control area. Therefore, the reward for the first positioning information can be determined based on the position information and uniqueness information of the target demarcated area in the first interface image.

[0091] Step 150 : determining second state information of the target defined area based on the position information and uniqueness information of the target defined area in the first interface image and the image features of the target defined area.

[0092] It should be noted that the second state information here is for the target delimited area, and the first state information in step 120 is for the target control area. The second state information includes the position information and uniqueness information of the target delimited area in the first interface image, and the image features of the target delimited area. For example, the position information, uniqueness information, and image features of the target delimited area in the first interface image can be spliced ​​to obtain the second state information.

[0093] Step 160, taking the first state information of the target control area, the first positioning information, the reward of the first positioning information, and the second state information of the target demarcated area as a set of training data;

[0094] Step 170, based on multiple sets of training data, a reinforcement learning method is used to iterate the parameters of the initial policy network until an iteration termination condition is met, and the initial policy network after the parameter iteration is completed is used as a control area positioning model.

[0095] Specifically, the model training method based on supervised learning requires a large amount of labeled data. However, the data labeling workload is large, which seriously reduces the efficiency of model training. The embodiment of the present invention adopts a reinforcement learning method and obtains training data by interactive screenshots, thereby eliminating the data labeling process and improving the training efficiency of the model.

[0096] Each set of training data includes first state information of the target control area, first positioning information, a reward for the first positioning information, and second state information of the target demarcated area.

[0097] It is understandable that for any target control area, one or more target delimited areas can be obtained, and for each target control area in the first interface image, a corresponding target delimited area can be obtained. That is, by repeatedly executing steps 120 to 150, multiple sets of training data can be obtained.

[0098] After obtaining multiple sets of training data, the reinforcement learning method can be used to iterate the parameters of the initial policy network until the iteration termination condition is met. The iteration termination condition here can be a preset number of episodes, such as 250 episodes. The control area positioning model thus obtained can realize the screenshot positioning of the target control area, and automatically capture the control area with unique identifiability and containing the target control area, thereby improving the writing efficiency of the automated test script.

[0099] The method provided by the embodiment of the present invention obtains the target demarcated area by locating the control area of ​​the first interface image including the target control area, and uses the first state information, first positioning information, reward of the first positioning information and second state information of the target control area as a set of training data, and obtains multiple sets of training data by interactive screenshots, thereby eliminating the data labeling process. The control area positioning model obtained by the reinforcement learning method can automatically capture the control area that is uniquely identifiable and includes the target control area, thereby improving the efficiency of writing automated test scripts.

[0100] Based on the above embodiment, based on the first state information of the target control area and the initial strategy network, the first positioning information is determined, that is, step 130 specifically includes:

[0101] Step 131, inputting the first state information of the target control area into the initial strategy network to obtain initial positioning information output by the initial strategy network;

[0102] Step 132: construct a normal distribution by taking the initial positioning information as the average value of the normal distribution, and randomly generate first positioning information from the normal distribution.

[0103] Specifically, with respect to determining the first positioning information, the first state information of the target control area may be firstly input into the initial strategy network to obtain the initial positioning information output by the initial strategy network.

[0104] On this basis, the initial positioning information is taken as the average value of the normal distribution, and the parameter VAR is added to construct a normal distribution. Then a new positioning information is randomly generated from the normal distribution, that is, the first positioning information.

[0105] The method provided by the embodiment of the present invention constructs a normal distribution by taking the initial positioning information as the average value of the normal distribution, and randomly generates the first positioning information from the normal distribution, thereby improving the diversity of training data.

[0106] Based on any of the above embodiments, based on the position information and uniqueness information of the target control area in the first interface image and the image features of the target control area, the first state information of the target control area is determined, that is, step 120 specifically includes:

[0107] Step 121, determining a target control area, and position information and uniqueness information of the target control area in the first interface image;

[0108] Step 122, extracting features from the first interface image to obtain first interface image features;

[0109] Step 123, based on the position information of the target control area in the first interface image and the first interface image features, image features of the target control area are extracted to obtain image features of the target control area;

[0110] Step 124 , the position information and uniqueness information of the target control area in the first interface image and the image features of the target control area are spliced ​​to obtain the first state information of the target control area.

[0111] Specifically, Figure 2 is a schematic diagram of the determination process of the first state information provided by the present invention, such as Figure 2 As shown, the target control area is an area that only includes the target control. The target control may be a user-operated control, and the target control may be determined in response to a click operation by the user in the first interface image.

[0112] After the target control is determined, target control detection and image segmentation are performed on the first interface image to obtain a block containing the target control, namely, the target control area. At the same time, the position information of the target control area in the first interface image can also be obtained.

[0113] The uniqueness information can be obtained by image matching the target control area with other control areas in the first interface image. When there is a control area that matches the target control area, it is considered that the target control area is not uniquely identifiable, that is, the uniqueness information is not unique; conversely, when there is no control area that matches the target control area, it is considered that the target control area is uniquely identifiable, that is, the uniqueness information is unique.

[0114] Then, feature extraction is performed on the first interface image to obtain the image features of the first interface image. The first interface image with information about the control position, height, and whether the control is unique in the user interface can be scaled to a fixed size, for example, to obtain an image of size M×N. Then, the image features of the first interface image are obtained through a pre-trained image feature extraction network, where the standard VGG-16 can be used as an example. The obtained image feature map is then convolved with 4 1×1 convolution kernels to form a feature map containing 4 channels.

[0115] In step 123, ROI Align processing can be performed in combination with the position information of the target control area in the first interface image and the feature extraction network pooling layer to obtain image features of the target control area after bilinear interpolation processing. The embodiment of the present invention can select a 5×5 control area feature map.

[0116] Then, the image features of the target control area are Flattened and then spliced ​​together based on the information of whether the target control area is unique and the position information of the target control area in the first interface image (which may include the position information of the center coordinates of the target control area relative to the length and height of the first interface image, and the length and height information of the target control area relative to the length and height of the first interface image). The first state information of the target control area can be represented by the state feature vector s.

[0117] Figure 3 is a schematic diagram of the determination process of the second state information provided by the present invention, such as Figure 3 As shown, based on the position information and uniqueness information of the target defined area in the first interface image and the image features of the target defined area, the second state information of the target defined area is determined, which can be represented by a state feature vector s_.

[0118] Based on any of the above embodiments, based on the position information and uniqueness information of the target demarcated area in the first interface image, determining the reward of the first positioning information, that is, step 140 specifically includes:

[0119] Step 141, matching the target defined area with other control areas in the first interface image, and determining unique information of the target defined area in the first interface image based on the matching result;

[0120] Step 142, determining a ratio of the length and height of the target defined area to the length and height of the target control area based on the position information of the target defined area in the first interface image;

[0121] Step 143, based on the uniqueness information of the target defined area in the first interface image, and the ratio of the length and height of the target defined area to the length and height of the target control area, determine the reward of the first positioning information.

[0122] Specifically, Figure 4: is a schematic diagram of a reward determination process for the first positioning information provided by the present invention, such as Figure 4 As shown, the uniqueness information of the target delimited area in the first interface image can be realized by template matching. The target delimited area is matched with other control areas in the first interface image to obtain a matching result, which includes a match or a mismatch. If it matches, it means that there are other control areas in the first interface image that match the target delimited area, and the uniqueness information is not unique; if it does not match, it means that there are no other control areas in the first interface image that match the target delimited area, and the uniqueness information is unique.

[0123] In some embodiments, the target defined area and the target control area are both rectangular, and based on the position information of the target defined area in the first interface image, the ratio of the length and height of the target defined area to the length and height of the target control area can be determined.

[0124] It should be noted that the target defined area and the target control area have the same center position. In order to make the target defined area uniquely identifiable, the area of ​​the target defined area is usually larger than the area of ​​the target control area.

[0125] The reward for the first positioning information is determined based on the uniqueness information and the ratio of the length and height of the target demarcation area to the length and height of the target control area. When the target demarcation area is uniquely identifiable in the first interface image, the reward for the first positioning information is greater; conversely, when the target demarcation area is not uniquely identifiable in the first interface image, the reward for the first positioning information is smaller.

[0126] In addition, the larger the ratio of the length and height of the target demarcated area to the length and height of the target control area, the more pixels the target demarcated area contains outside the target control area, and the smaller the reward for the first positioning information; conversely, the smaller the ratio of the length and height of the target demarcated area to the length and height of the target control area, the fewer pixels the target demarcated area contains outside the target control area, and the greater the reward for the first positioning information.

[0127] Optionally, based on the uniqueness information of the target delimited area in the first interface image and the ratio of the length and height of the target delimited area to the length and height of the target control area, the reward of the first positioning information is determined, that is, step 143 specifically includes:

[0128] The reward for the first positioning information is determined based on the following formula:

[0129]

[0130] Wherein, G is a constant, λ1 is the ratio of the length of the target demarcation area to the length of the target control area, λ2 is the ratio of the height of the target demarcation area to the height of the target control area, and both λ1 and λ2 are greater than or equal to 1.

[0131] It is understandable that when the target demarcated area is uniquely identifiable in the first interface image, that is, unique in the first interface image, the reward for the first positioning information is a positive number, and is determined based on the ratio of the length and height of the target demarcated area to the length and height of the target control area.

[0132] When the target demarcated area is not uniquely identifiable in the first interface image, that is, it is not unique in the first interface image, the reward for the first positioning information is zero.

[0133] The method provided by an embodiment of the present invention determines the reward of the first positioning information through the uniqueness information of the target control area and the ratio of the length and height of the target demarcated area to the length and height of the target control area, so that the model can quickly learn to intercept a control area that is uniquely identifiable, contains the target control area and has a smaller area.

[0134] Based on any of the above embodiments, in step 170, based on multiple sets of training data, a reinforcement learning method is used to iterate the parameters of the initial policy network until an iteration termination condition is met, specifically including:

[0135] Step 171, for any set of training data among the multiple sets of training data, input the first positioning information and the first state information in any set of training data into the initial value network to obtain the first positioning value output by the initial value network;

[0136] Step 172, inputting the second state information in any set of training data into the target policy network to obtain the second positioning information output by the target policy network, inputting the second positioning information and the second state information into the target value network to obtain the second positioning value output by the target value network;

[0137] Step 173, based on the second positioning value, the reward and the first positioning value, iterate the parameters of the initial value network to obtain a value network after parameter iteration;

[0138] Step 174, inputting the first positioning information and the first state information in any set of training data into the value network after parameter iteration, and obtaining a third positioning value output by the value network after parameter iteration;

[0139] Step 175: Based on the third positioning value, perform parameter iteration on the initial strategy network until an iteration termination condition is met.

[0140] Specifically, Figure 5 It is a schematic diagram of the initial strategy network training process provided by the present invention, such as Figure 5 As shown in Figure 1, the value network is introduced in the process of iterating the parameters of the initial policy network. The positioning value output by the value network represents the expected total reward of the agent after it performs the action of region division based on the positioning information until the final state.

[0141] First, the first positioning information and the first state information in any set of training data are input into the initial value network to obtain the first positioning value output by the initial value network.

[0142] The second state information in any set of training data is input into the target policy network to obtain the second positioning information output by the target policy network, and the second positioning information and the second state information are input into the target value network to obtain the second positioning value output by the target value network.

[0143] The target strategy network and target value network here can be target neural networks that have undergone multiple rounds of iterations. The target neural network parameters are updated using a sliding average method, for example, the ratio of new and old parameter elements can be 8:2 each time.

[0144] On this basis, the parameters of the initial value network can be iterated based on the second positioning value, reward and the first positioning value to obtain the value network after parameter iteration.

[0145] Optionally, based on the second positioning value, the reward and the first positioning value, performing parameter iteration on the initial value network includes:

[0146] The second positioning value, the reward and the first positioning value are added and subtracted to obtain the value loss; based on the value loss, the parameters of the initial value network are iterated.

[0147] In some embodiments, the loss of value may be expressed as follows:

[0148] TD-error=Q′+Reward-Q

[0149] In the formula, TD-error represents the value loss, Q′ represents the second positioning value, Reward represents the reward, and Q represents the first positioning value.

[0150] Use TD-error as the loss function to update the parameters of the initial value network and obtain the value network after parameter iteration.

[0151] Then, the first positioning information and the first state information in any set of training data are input into the value network after parameter iteration to obtain the third positioning value output by the value network after parameter iteration; the negative value of the third positioning value is used as the loss function to iterate the parameters of the initial strategy network until the iteration termination condition is met.

[0152] Based on any of the above embodiments, a unique control area positioning model training method is provided, including:

[0153] Step 1: The software user interface with the control position, height information and whether the control is unique in the user interface is passed through the CNN network and the control area ROI Align process is performed, and finally a state feature vector s containing information on whether the control area is unique, the position information of the center coordinates of the control area relative to the height of the interface, the height of the control area relative to the height of the interface and the image features of the control area is formed, that is, the first state information s of the target control area;

[0154] Step 2: Input the first state information s in step 1 into the initial policy network (Actor neural network), output the initial positioning information a0, and then take the initial positioning information a0 as the average value of a normal distribution, add the parameter VAR, and construct a normal distribution. Finally, randomly select a new action vector from the normal distribution, that is, the first positioning information a;

[0155] Step 3: Input the first positioning information a in step 2 and the first state information s in step 1 into the initial value network (Critic neural network), and output the corresponding first positioning value Q;

[0156] Step 4: The screenshot generated by the first positioning information a in step 2 is used to determine whether it is unique in the user interface, and the reward for the first positioning information is generated by combining the position information of the center of the control area relative to the height of the interface and the height of the demarcated area relative to the height of the control;

[0157] Step 5: The screenshot generated by the first positioning information a in step 4, the information on whether the target delimited area is unique in the user interface, the position information of the center of the control area relative to the height of the interface, the information of the height of the target delimited area relative to the height of the interface and the characteristics of the target delimited area are combined to generate a state feature vector s_, that is, the second state information s_ of the target delimited area;

[0158] Step 6: Repeat steps 2-5, and organize the first state information s in step 1 and the first positioning information a in step 2, the reward Reward of the first positioning information in step 4, and the second state information s_ of the target demarcated area in step 5 into the form of (s, a, Reward, s_) for data organization and storage as a training data set;

[0159] Step 7: Input the second state information of a set of training data in the training data set in step 6 into the target strategy network (Actor target neural network), and output the action vector a' containing the information of the length and height of the screenshot area in the horizontal and vertical directions relative to the length and height of the demarcated area, that is, the second positioning information a';

[0160] Step 8: Input the second positioning information a' and the second state information s_ in step 7 into the target value network (Critic target neural network), and output the corresponding second positioning value Q' value;

[0161] Step 9: Add the second positioning value Q' in step 8 to the reward Reward in the set of training data, and then subtract the first positioning value Q in the set of training data to obtain TD-error, which is used as the loss function to update the Critic network parameters;

[0162] Step 10: Input the first state information s of the training data in step 7 into the Actor network, input the output first positioning information a and the first state information s into the Critic network after the parameters are updated in step 9, output the third positioning value, use the negative value of the third positioning value as the loss function, and update the parameters of the initial strategy network (Actor neural network);

[0163] Step 11: Repeat steps 2-10 until the training ends. This example uses 250 episodes. The initial policy network after parameter iteration is used as the control area positioning model. The control area positioning model can be used to locate the control area and automatically capture a control area that is uniquely identifiable, contains the target control area, and is small in area.

[0164] Based on any of the above embodiments, Figure 6 is a flow chart of the unique control area positioning method provided by the present invention, such as Figure 6 As shown, a unique control area positioning method is provided, including:

[0165] Step 610, obtaining a second interface image and a control area to be located in the second interface image;

[0166] Step 620, determining third state information of the control area to be located based on the position information and uniqueness information of the control area to be located in the second interface image and the image features of the control area to be located;

[0167] Step 630, input the third state information into the control area positioning model to obtain the third positioning information output by the control area positioning model, and locate the control area of ​​the second interface image based on the third positioning information. The control area positioning model is trained based on the above-mentioned unique control area positioning model training method.

[0168] Specifically, after obtaining the trained control area positioning model, the model can be used to automatically locate the control area.

[0169] Here, the second interface image may be a user interface to be automatically tested. The second interface image includes multiple controls, and the control area to be located is any one or more control areas in the second interface image that need to be located.

[0170] Then, target detection and image segmentation can be performed on the second interface image to obtain the location information of the control area to be located in the second interface image. The method based on image similarity detection obtains the uniqueness information of the control area to be located in the second interface image. The image features of the control area to be located are obtained by image feature extraction.

[0171] On this basis, the position information and uniqueness information of the control area to be located in the second interface image and the image features of the control area to be located are spliced ​​to obtain the third state information of the control area to be located.

[0172] Then, the third state information is input into the trained control area positioning model to obtain the third positioning information output by the control area positioning model, and the control area of ​​the second interface image is positioned based on the third positioning information.

[0173] For example, the third positioning information can be expressed as a=(λ1,λ2), and the control area positioning is performed according to the third positioning information to obtain the position and size information of the designated area that is uniquely identifiable and has a smaller area and includes the control to be positioned.

[0174]

[0175] Wherein, l, h are the length and height of the demarcated area with unique identification and small area including the control to be located, l0, h0 are the length and height of the control area to be located, and the center position of the demarcated area is consistent with the center position of the control area to be located.

[0176] The method provided in the embodiment of the present invention locates the control area through the trained control area positioning model, and can automatically obtain a demarcated area with unique identification and a small area including the control to be located, which can be used to write automated test scripts. Compared with the prior art of manual screenshots multiple times, the efficiency of automated test script writing is greatly improved.

[0177] The unique control area positioning model training device provided by the present invention is described below. The unique control area positioning model training device described below and the unique control area positioning model training method described above can be referenced to each other.

[0178] Based on the above embodiments, Figure 7 : is a structural diagram of a unique control area positioning model training device provided by the present invention, such as Figure 7 As shown, the unique control area positioning model training device includes:

[0179] A first acquisition unit 710 is used to acquire an initial policy network and a first interface image including a target control area;

[0180] The first state determination unit 720 is used to determine the first state information of the target control area based on the position information and uniqueness information of the target control area in the first interface image and the image features of the target control area; the uniqueness information indicates whether the target control area is uniquely identifiable in the first interface image;

[0181] A positioning information determining unit 730 determines first positioning information based on the first state information of the target control area and the initial strategy network;

[0182] A reward determination unit 740 is used to locate the control area of ​​the first interface image based on the first positioning information to obtain a target demarcated area, and determine a reward for the first positioning information based on the position information and uniqueness information of the target demarcated area in the first interface image;

[0183] A second state determining unit 750, configured to determine second state information of the target defined area based on position information and uniqueness information of the target defined area in the first interface image and image features of the target defined area;

[0184] A training data acquisition unit 760 uses the first state information of the target control area, the first positioning information, the reward of the first positioning information, and the second state information of the target demarcated area as a set of training data;

[0185] The parameter iteration unit 770 is used to perform parameter iteration on the initial strategy network based on the multiple sets of training data using a reinforcement learning method until an iteration termination condition is met, and use the initial strategy network after the parameter iteration as a control area positioning model.

[0186] The unique control area positioning model training device provided in the embodiment of the present invention obtains the target demarcated area by locating the control area of ​​the first interface image including the target control area, and uses the first state information, first positioning information, reward of the first positioning information and second state information of the target demarcated area as a set of training data, and obtains multiple sets of training data by interactive screenshots, thereby eliminating the data labeling process. The control area positioning model obtained by the reinforcement learning method can automatically capture the control area that is uniquely identifiable and includes the target control area, thereby improving the efficiency of writing automated test scripts.

[0187] Based on any of the above embodiments, the positioning information determining unit 730 is specifically configured to:

[0188] Inputting the first state information of the target control area into the initial strategy network to obtain initial positioning information output by the initial strategy network;

[0189] The normal distribution is constructed by taking the initial positioning information as the average value of the normal distribution, and the first positioning information is randomly generated from the normal distribution.

[0190] Based on any of the foregoing embodiments, the first state determining unit 720 is specifically configured to:

[0191] Determine a target control area, and position information and uniqueness information of the target control area in the first interface image;

[0192] Extracting features from the first interface image to obtain first interface image features;

[0193] Based on the position information of the target control area in the first interface image and the first interface image features, image features of the target control area are extracted to obtain image features of the target control area;

[0194] The position information and uniqueness information of the target control area in the first interface image and the image features of the target control area are spliced ​​to obtain the first state information of the target control area.

[0195] Based on any of the above embodiments, the reward determination unit 740 is specifically used for:

[0196] Matching the target defined area with other control areas in the first interface image, and determining unique information of the target defined area in the first interface image based on the matching result;

[0197] Based on the position information of the target delimited area in the first interface image, determining the ratio of the length and height of the target delimited area to the length and height of the target control area;

[0198] Based on the uniqueness information of the target demarcated area in the first interface image and the ratio of the length and height of the target demarcated area to the length and height of the target control area, the reward for the first positioning information is determined.

[0199] Based on any of the above embodiments, the reward determination unit 740 is further specifically configured to:

[0200] The reward for the first positioning information is determined based on the following formula:

[0201]

[0202] Wherein, G is a constant, λ1 is the ratio of the length of the target demarcation area to the length of the target control area, λ2 is the ratio of the height of the target demarcation area to the height of the target control area, and both λ1 and λ2 are greater than or equal to 1.

[0203] Based on any of the above embodiments, the parameter iteration unit 770 is specifically used for:

[0204] For any set of training data among the multiple sets of training data, input the first positioning information and the first state information in the any set of training data into the initial value network to obtain the first positioning value output by the initial value network;

[0205] Inputting the second state information in any one set of training data into the target policy network to obtain the second positioning information output by the target policy network, inputting the second positioning information and the second state information into the target value network to obtain the second positioning value output by the target value network;

[0206] Based on the second positioning value, the reward, and the first positioning value, performing parameter iteration on the initial value network to obtain a value network after parameter iteration;

[0207] Inputting the first positioning information and the first state information in any one set of training data into the value network after parameter iteration to obtain a third positioning value output by the value network after parameter iteration;

[0208] Based on the third positioning value, parameters of the initial strategy network are iterated until an iteration termination condition is met.

[0209] Based on any of the above embodiments, the parameter iteration unit 770 is further specifically configured to:

[0210] Add and subtract the second positioning value, the reward and the first positioning value to obtain a value loss;

[0211] Based on the value loss, parameters of the initial value network are iterated.

[0212] Based on any of the above embodiments, Figure 8 is a structural schematic diagram of a unique control area positioning device provided by the present invention, such as Figure 8 As shown, the unique control area positioning device includes:

[0213] A second acquisition unit 810 is used to acquire a second interface image and a control area to be located in the second interface image;

[0214] A third state determining unit 820, configured to determine third state information of the control area to be located based on the position information and uniqueness information of the control area to be located in the second interface image and the image features of the control area to be located;

[0215] The area positioning unit 830 is used to input the third state information into the control area positioning model, obtain the third positioning information output by the control area positioning model, and locate the control area of ​​the second interface image based on the third positioning information. The control area positioning model is trained based on the unique control area positioning model training method.

[0216] The unique control area positioning device provided in the embodiment of the present invention can automatically obtain a demarcated area with unique identification and a small area including the control to be positioned, by using a trained control area positioning model to locate the control area. It can be used to write automated test scripts. Compared with the prior art of manual screenshots multiple times, it greatly improves the efficiency of writing automated test scripts.

[0217] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the unique control area positioning model training method, which includes:

[0218] Obtaining an initial strategy network and a first interface image including a target control area;

[0219] Determine the first state information of the target control area based on the position information and uniqueness information of the target control area in the first interface image and the image features of the target control area; the uniqueness information indicates whether the target control area is uniquely identifiable in the first interface image;

[0220] Determining first positioning information based on the first state information of the target control area and the initial strategy network;

[0221] Based on the first positioning information, the control area of ​​the first interface image is positioned to obtain a target demarcated area, and based on the position information and uniqueness information of the target demarcated area in the first interface image, a reward for the first positioning information is determined;

[0222] Determine second state information of the target defined area based on position information and uniqueness information of the target defined area in the first interface image and image features of the target defined area;

[0223] The first state information of the target control area, the first positioning information, the reward of the first positioning information, and the second state information of the target demarcated area are used as a set of training data;

[0224] Based on multiple groups of training data, a reinforcement learning method is used to iterate the parameters of the initial strategy network until an iteration termination condition is met, and the initial strategy network after completing the parameter iteration is used as a control area positioning model.

[0225] The processor may call logic instructions in the memory to execute a unique control area positioning method, the method comprising:

[0226] Acquire a second interface image and a control area to be located in the second interface image;

[0227] Determine third state information of the control area to be located based on the position information and uniqueness information of the control area to be located in the second interface image and the image features of the control area to be located;

[0228] The third state information is input into the control area positioning model to obtain the third positioning information output by the control area positioning model, and the control area of ​​the second interface image is positioned based on the third positioning information. The control area positioning model is trained based on the unique control area positioning model training method.

[0229] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0230] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the unique control area positioning model training method provided by the above methods, the method includes:

[0231] Obtaining an initial strategy network and a first interface image including a target control area;

[0232] Determine the first state information of the target control area based on the position information and uniqueness information of the target control area in the first interface image and the image features of the target control area; the uniqueness information indicates whether the target control area is uniquely identifiable in the first interface image;

[0233] Determining first positioning information based on the first state information of the target control area and the initial strategy network;

[0234] Based on the first positioning information, the control area of ​​the first interface image is positioned to obtain a target demarcated area, and based on the position information and uniqueness information of the target demarcated area in the first interface image, a reward for the first positioning information is determined;

[0235] Determine second state information of the target defined area based on position information and uniqueness information of the target defined area in the first interface image and image features of the target defined area;

[0236] The first state information of the target control area, the first positioning information, the reward of the first positioning information, and the second state information of the target demarcated area are used as a set of training data;

[0237] Based on multiple groups of training data, a reinforcement learning method is used to iterate the parameters of the initial strategy network until an iteration termination condition is met, and the initial strategy network after completing the parameter iteration is used as a control area positioning model.

[0238] When the computer program is executed by a processor, the computer can execute the unique control area positioning method provided by the above methods, and the method includes:

[0239] Acquire a second interface image and a control area to be located in the second interface image;

[0240] Determine third state information of the control area to be located based on the position information and uniqueness information of the control area to be located in the second interface image and the image features of the control area to be located;

[0241] The third state information is input into the control area positioning model to obtain the third positioning information output by the control area positioning model, and the control area of ​​the second interface image is positioned based on the third positioning information. The control area positioning model is trained based on the unique control area positioning model training method.

[0242] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the unique control area positioning model training method provided by the above methods, the method comprising:

[0243] Obtaining an initial strategy network and a first interface image including a target control area;

[0244] Determine the first state information of the target control area based on the position information and uniqueness information of the target control area in the first interface image and the image features of the target control area; the uniqueness information indicates whether the target control area is uniquely identifiable in the first interface image;

[0245] Determining first positioning information based on the first state information of the target control area and the initial strategy network;

[0246] Based on the first positioning information, the control area of ​​the first interface image is positioned to obtain a target demarcated area, and based on the position information and uniqueness information of the target demarcated area in the first interface image, a reward for the first positioning information is determined;

[0247] Determine second state information of the target defined area based on position information and uniqueness information of the target defined area in the first interface image and image features of the target defined area;

[0248] The first state information of the target control area, the first positioning information, the reward of the first positioning information, and the second state information of the target demarcated area are used as a set of training data;

[0249] Based on multiple groups of training data, a reinforcement learning method is used to iterate the parameters of the initial strategy network until an iteration termination condition is met, and the initial strategy network after completing the parameter iteration is used as a control area positioning model.

[0250] When the computer program is executed by a processor, the unique control area positioning method provided by the above methods is implemented, and the method includes:

[0251] Acquire a second interface image and a control area to be located in the second interface image;

[0252] Determine third state information of the control area to be located based on the position information and uniqueness information of the control area to be located in the second interface image and the image features of the control area to be located;

[0253] The third state information is input into the control area positioning model to obtain the third positioning information output by the control area positioning model, and the control area of ​​the second interface image is positioned based on the third positioning information. The control area positioning model is trained based on the unique control area positioning model training method.

[0254] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0255] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0256] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A unique control area positioning model training method, characterized in that: include: Obtaining an initial strategy network and a first interface image including a target control area; Determine the first state information of the target control area based on the position information and uniqueness information of the target control area in the first interface image and the image features of the target control area; the uniqueness information indicates whether the target control area is uniquely identifiable in the first interface image; Determining first positioning information based on the first state information of the target control area and the initial strategy network; Based on the first positioning information, the control area of ​​the first interface image is positioned to obtain a target demarcated area, and based on the position information and uniqueness information of the target demarcated area in the first interface image, a reward for the first positioning information is determined; Determine second state information of the target defined area based on position information and uniqueness information of the target defined area in the first interface image and image features of the target defined area; The first state information of the target control area, the first positioning information, the reward of the first positioning information, and the second state information of the target demarcated area are used as a set of training data; Based on the multiple sets of training data, a reinforcement learning method is used to iterate the parameters of the initial strategy network until an iteration termination condition is met, and the initial strategy network after the parameter iteration is completed is used as a control area positioning model; The step of using a reinforcement learning method to iterate parameters of the initial strategy network based on the multiple sets of training data until an iteration termination condition is met includes: For any group of training data among the multiple groups of training data, input the first positioning information and the first state information in the any group of training data into the initial value network to obtain the first positioning value output by the initial value network; Inputting the second state information in any one set of training data into the target policy network to obtain the second positioning information output by the target policy network, inputting the second positioning information and the second state information into the target value network to obtain the second positioning value output by the target value network; Based on the second positioning value, the reward, and the first positioning value, performing parameter iteration on the initial value network to obtain a value network after parameter iteration; Inputting the first positioning information and the first state information in any one set of training data into the value network after parameter iteration to obtain a third positioning value output by the value network after parameter iteration; Based on the third positioning value, parameters of the initial strategy network are iterated until an iteration termination condition is met.

2. The unique control area positioning model training method according to claim 1 is characterized in that: The determining the first positioning information based on the first state information of the target control area and the initial strategy network includes: Inputting the first state information of the target control area into the initial strategy network to obtain initial positioning information output by the initial strategy network; The normal distribution is constructed by taking the initial positioning information as the average value of the normal distribution, and the first positioning information is randomly generated from the normal distribution.

3. The unique control area positioning model training method according to claim 1 is characterized in that: The determining the first state information of the target control area based on the position information and uniqueness information of the target control area in the first interface image and the image feature of the target control area includes: Determine a target control area, and position information and uniqueness information of the target control area in the first interface image; Extracting features from the first interface image to obtain first interface image features; Based on the position information of the target control area in the first interface image and the first interface image features, image features of the target control area are extracted to obtain image features of the target control area; The position information and uniqueness information of the target control area in the first interface image and the image features of the target control area are spliced ​​to obtain the first state information of the target control area.

4. The unique control area positioning model training method according to claim 1 is characterized in that: The step of determining a reward for the first positioning information based on the position information and uniqueness information of the target demarcated area in the first interface image includes: Matching the target defined area with other control areas in the first interface image, and determining unique information of the target defined area in the first interface image based on the matching result; Based on the position information of the target delimited area in the first interface image, determining the ratio of the length and height of the target delimited area to the length and height of the target control area; Based on the uniqueness information of the target demarcated area in the first interface image and the ratio of the length and height of the target demarcated area to the length and height of the target control area, the reward for the first positioning information is determined.

5. The unique control area positioning model training method according to claim 4 is characterized in that: The determining of the reward for the first positioning information based on the uniqueness information of the target delimited area in the first interface image and the ratio of the length and height of the target delimited area to the length and height of the target control area includes: The reward for the first positioning information is determined based on the following formula: Wherein, G is a constant, λ1 is the ratio of the length of the target demarcation area to the length of the target control area, λ2 is the ratio of the height of the target demarcation area to the height of the target control area, and both λ1 and λ2 are greater than or equal to 1.

6. The unique control area positioning model training method according to claim 1 is characterized in that: The performing parameter iteration on the initial value network based on the second positioning value, the reward and the first positioning value includes: Add and subtract the second positioning value, the reward and the first positioning value to obtain a value loss; Based on the value loss, parameters of the initial value network are iterated.

7. A unique control area positioning method, characterized in that: include: Acquire a second interface image and a control area to be located in the second interface image; Determine third state information of the control area to be located based on the position information and uniqueness information of the control area to be located in the second interface image and the image features of the control area to be located; The third state information is input into the control area positioning model to obtain the third positioning information output by the control area positioning model, and the control area of ​​the second interface image is positioned based on the third positioning information, and the control area positioning model is trained based on the unique control area positioning model training method according to any one of claims 1 to 6.

8. A unique control area positioning model training device, characterized in that: include: A first acquisition unit, used to acquire an initial strategy network and a first interface image including a target control area; A first state determination unit, configured to determine first state information of the target control area based on position information and uniqueness information of the target control area in the first interface image and image features of the target control area; the uniqueness information indicates whether the target control area is uniquely identifiable in the first interface image; A positioning information determining unit, which determines first positioning information based on the first state information of the target control area and the initial strategy network; a reward determination unit, configured to locate a control area of ​​the first interface image based on the first positioning information to obtain a target demarcated area, and determine a reward for the first positioning information based on position information and uniqueness information of the target demarcated area in the first interface image; a second state determining unit, configured to determine second state information of the target defined area based on position information and uniqueness information of the target defined area in the first interface image and image features of the target defined area; A training data acquisition unit, which uses the first state information of the target control area, the first positioning information, the reward of the first positioning information, and the second state information of the target demarcated area as a set of training data; A parameter iteration unit, configured to perform parameter iteration on the initial strategy network using a reinforcement learning method based on the plurality of sets of training data until an iteration termination condition is satisfied, and use the initial strategy network after parameter iteration as a control area positioning model; The parameter iteration unit is specifically used for: For any group of training data among the multiple groups of training data, input the first positioning information and the first state information in the any group of training data into the initial value network to obtain the first positioning value output by the initial value network; Inputting the second state information in any one set of training data into the target policy network to obtain the second positioning information output by the target policy network, inputting the second positioning information and the second state information into the target value network to obtain the second positioning value output by the target value network; Based on the second positioning value, the reward and the first positioning value, performing parameter iteration on the initial value network to obtain a value network after parameter iteration; Inputting the first positioning information and the first state information in any one set of training data into the value network after parameter iteration to obtain a third positioning value output by the value network after parameter iteration; Based on the third positioning value, parameters of the initial strategy network are iterated until an iteration termination condition is met.

9. A unique control area positioning device, characterized in that: include: A second acquisition unit, used to acquire a second interface image and a control area to be located in the second interface image; A third state determining unit, configured to determine third state information of the control area to be located based on the position information and uniqueness information of the control area to be located in the second interface image and the image features of the control area to be located; An area positioning unit is used to input the third state information into a control area positioning model to obtain third positioning information output by the control area positioning model, and to perform control area positioning on the second interface image based on the third positioning information, wherein the control area positioning model is trained based on the unique control area positioning model training method according to any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the unique control area positioning model training method as described in any one of claims 1 to 6, or the unique control area positioning method as described in claim 7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the unique control area positioning model training method as described in any one of claims 1 to 6, or the unique control area positioning method as described in claim 7.