A data enhancement model training method, data enhancement method, and device
By calculating the geometric feature difference of the detection frame, the automatic parameterization data enhancement model is solved, and the efficient training and generalization ability of the object detection model is achieved.
Patent Information
- Application Number
- CN202411184316.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-27
AI Technical Summary
During the training of the object detection model, improper setting of data enhancement hyperparameters will lead to the image enhancement amplitude being too large or too small, which will not effectively improve the generalization ability and accuracy of the model. It also requires professionals to manually adjust parameters, adding artificial cost.
By generating the variable values of the data augmentation variables, the model loss value is calculated using the geometric feature differences of the detection box, and the parameters of the data augmentation model are automatically adjusted until the preset convergence conditions are met, reducing manual intervention.
The automated data augmentation model training process is realized, which reduces labor costs, ensures that the geometric features of the data augmented detection frame fit with the detection frame in the test image, and improves the generalization ability and accuracy of the model.
Smart Images

Figure CN118710889B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of machine learning technology, and in particular to a data enhancement model training method, a data enhancement method, and a device. Background Art
[0002] When training a target detection model, it is usually necessary to perform data augmentation processing on the training images used to train the target detection model to increase the diversity of the training images, thereby improving the generalization ability and accuracy of the target detection model.
[0003] At present, data enhancement often involves a series of hyperparameters. If the hyperparameters are set too aggressively, the data enhancement will be large, and some enhanced images may not be helpful for real scenes or performance tests. If the hyperparameters are set too conservatively, the existing images may not be fully utilized. Therefore, professionals are often required to manually adjust the hyperparameters involved, which results in high labor costs. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a data enhancement model training method, a data enhancement method, and a device, which can save the process of manually adjusting hyperparameters by professionals and reduce labor costs. The specific technical solution is as follows:
[0005] In a first aspect, an embodiment of the present application provides a method for training a data enhancement model, the method comprising:
[0006] Get training images;
[0007] Generate variable values of data augmentation variables for each training image using the data augmentation model of the initial structure;
[0008] Using the variable value of the data enhancement variable of each training image, transforming the detection frame of the object in the training image to obtain a first detection frame;
[0009] Determining a model loss value based on a difference in geometric features between a detection box representing an object of the same category in a test image and a first detection box representing the object of the same category;
[0010] Based on the model loss value, the model parameters of the data enhancement model of the initial structure are adjusted until the preset convergence conditions are reached to obtain a trained data enhancement model.
[0011] Optionally, the data enhancement model of the initial structure is a neural network model, and includes a first sub-network and a second sub-network;
[0012] The data enhancement model using the initial structure generates the variable value of the data enhancement variable of each training image, including:
[0013] For each training image, the first sub-network is used to extract features from the label information of the detection frame in the training image, and the extracted features are aggregated to obtain aggregated features; wherein the label information of each detection frame includes: information representing the geometric features of the detection frame;
[0014] Generate random information corresponding to the training image; wherein each random information includes multiple random numbers;
[0015] The obtained aggregated features, the size of the training image, and the random information corresponding to the training image are input into the second sub-network to obtain the variable value of the data enhancement variable output by the second sub-network.
[0016] Optionally, after transforming the detection frame of the object in each training image by using the variable value of the data enhancement variable of the training image to obtain the first detection frame, the method further includes:
[0017] Return to the step of generating random information corresponding to the training image until a first specified number of first detection frames corresponding to each detection frame in the training image is obtained.
[0018] Optionally, the model parameters of the data enhancement model of the initial structure include: an upper limit and a lower limit of a variable value of at least one data enhancement variable;
[0019] The data enhancement model using the initial structure generates the variable value of the data enhancement variable of each training image, including:
[0020] For each training image, a random number is assigned to each data augmentation variable;
[0021] Based on the random number assigned to each data augmentation variable, a numerical value is determined from the range represented by the variable value upper limit and the variable value lower limit of the data augmentation variable as the variable value of the training image for the data augmentation variable.
[0022] Optionally, the determining, based on the random number assigned to each data augmentation variable, a value from a range represented by an upper limit and a lower limit of a variable value of the data augmentation variable as a variable value of the training image for the data augmentation variable comprises:
[0023] For each data augmentation variable, using the random number assigned to each data augmentation variable, obtain a weight weighted by the variable value upper limit and the variable value lower limit of the data augmentation variable;
[0024] According to the determined weight, the variable value upper limit and the variable value lower limit of the data enhancement variable are weightedly added to obtain the variable value of the data enhancement variable of the training image.
[0025] Optionally, determining the model loss value based on the difference in geometric features between a detection box representing an object of the same category in the test image and a first detection box representing the object of the same category includes:
[0026] For each first detection frame, determine a detection frame in the test image that represents an object of the same category as the first detection frame as the detection frame to be compared corresponding to the first detection frame; determine the difference in geometric features between the first detection frame and the corresponding detection frames to be compared, and determine the smallest difference as the first difference corresponding to the first detection frame; determine the average level of the first differences corresponding to the first detection frames as the first sub-loss value;
[0027] and / or,
[0028] For each second detection frame in the test image, determine a first detection frame representing an object of the same category as the second detection frame as the detection frame to be compared corresponding to the second detection frame; determine the difference in geometric features between the second detection frame and the corresponding detection frames to be compared, and determine the smallest difference as the second difference corresponding to the second detection frame; determine the average level of the second differences corresponding to the second detection frames as the second sub-loss value;
[0029] Based on the determined sub-loss values, a model loss value is obtained.
[0030] Optionally, the first difference corresponding to each first detection frame is calculated according to the following formula:
[0031] ;
[0032] in, is the first detection frame obtained by the k-th transformation of the i-th detection frame in the acquired training image; The first detection frame The corresponding j-th detection frame to be compared; Indicates the first detection box The detection box to be compared When the center points of the first detection frame coincide with The detection box to be compared The difference between the coordinates of the endpoints of The number of corresponding detection boxes to be compared; The first detection frame The corresponding first difference; Indicates taking the minimum value;
[0033] and / or,
[0034] The second difference corresponding to each second detection frame is calculated according to the following formula:
[0035] ;
[0036] in, is the jth second detection frame; The second detection box in the acquired training image represents the first detection box obtained by the k-th transformation of the i-th detection box of the same category object; S Indicates the first detection box With the second detection box When the center points of the first detection frame coincide with With the second detection box The difference between the coordinates of the endpoints of represents the number of detection boxes of objects of the same category, and P represents the first specified number; Represents the second detection box The corresponding second difference; Indicates taking the minimum value.
[0037] Optionally, before obtaining the model loss value based on the determined sub-loss value, the method further includes:
[0038] For each second detection frame in the test image, determine a detection frame in the acquired training image that represents an object of the same category as the second detection frame as an original detection frame corresponding to the second detection frame;
[0039] Determine a correlation value between the second detection frame and the original detection frame based on a maximum value of geometric feature similarities between each first detection frame obtained after transformation processing of each original detection frame corresponding to the second detection frame and the second detection frame;
[0040] Determine a diversity degree value corresponding to the second detection frame based on a correlation value between the second detection frame and corresponding original detection frames;
[0041] Determine a third sub-loss value based on an average level of diversity values corresponding to each second detection frame; wherein the third sub-loss value is negatively correlated with the average level of diversity values corresponding to each second detection frame;
[0042] The step of obtaining a model loss value based on the determined sub-loss value comprises:
[0043] The sum of the determined sub-loss values is calculated to obtain the model loss value.
[0044] Optionally, a growth rate of the diversity degree value corresponding to each second detection frame is negatively correlated with each correlation value corresponding to the second detection frame.
[0045] Optionally, the similarity of the geometric features of the two detection frames is an intersection-over-union ratio when the center points of the two detection frames coincide.
[0046] The diversity value corresponding to each second detection frame is calculated according to the following formula:
[0047] ;
[0048] in, is the jth second detection frame; The second detection box in the acquired training image Represents the i-th detection box of the same category object The first detection frame obtained after the k-th transformation process; The first detection frame and the second detection box The intersection-and-union ratio when the center points coincide; P represents the first specified number; Indicates taking the maximum value; Represents the second detection box With detection box M is the correlation value between the second detection frame and the acquired training image The number of detection boxes representing objects of the same category; and All are preset constants; The second detection frame The corresponding diversity degree value, D is the first threshold.
[0049] Optionally, the training process of the data enhancement model of the initial structure includes multiple rounds of parameter adjustment;
[0050] The obtaining of training images comprises:
[0051] Selecting a second specified number of images from a training set containing multiple images as training images required for adjusting parameters of the data augmentation model of the initial structure in this round;
[0052] and / or,
[0053] Before determining the model loss value based on the difference in geometric features between the detection box representing the object of the same category in the test image and the first detection box representing the object of the same category, the method further includes:
[0054] From the multiple test images, select a third specified number of images as the test images required for adjusting the parameters of the data augmentation model of the initial structure in this round.
[0055] Optionally, the test images selected in each round include the difficult test images among the test images selected in the previous round;
[0056] Follow these steps to determine the hard test images from the selected test images in each round:
[0057] For each second detection frame in the test image selected in each round, calculate the similarity of geometric features between the second detection frame and the first detection frame representing the object of the same category as the similarity corresponding to the second detection frame;
[0058] If the maximum value of the similarities corresponding to the second detection box is less than a second threshold, the test image to which the second detection box belongs is determined as a difficult test image; wherein the second threshold is determined based on the average level of the similarities corresponding to the second detection boxes in the test image selected in this round.
[0059] In a second aspect, an embodiment of the present application provides a data enhancement method, the method comprising:
[0060] Get training images;
[0061] Generate variable values of data enhancement variables of each training image using the trained data enhancement model; wherein the data enhancement model is trained according to the training method of the data enhancement model described in any one of the above items;
[0062] The data enhancement processing is performed on each training image using the variable value of the data enhancement variable of the training image to obtain a data enhanced training image.
[0063] Optionally, the data enhancement model is a neural network model, and includes: a first subnetwork and a second subnetwork; the step of generating a variable value of a data enhancement variable of each training image using the trained data enhancement model includes:
[0064] For each training image, the first sub-network is used to extract features from label information of each detection frame in the training image, and the extracted features are aggregated to obtain aggregated features; wherein the label information of each detection frame includes: information representing geometric features of the detection frame;
[0065] Generate random information corresponding to the training image; wherein each random information includes multiple random numbers;
[0066] The obtained aggregated features, the size representing the training image, and the random information corresponding to the training image are input into the second sub-network to obtain the variable value of the data enhancement variable output by the second sub-network.
[0067] Optionally, the model parameters of the data augmentation model include: an upper limit and a lower limit of a variable value of at least one data augmentation variable; and the step of generating a variable value of the data augmentation variable of each training image using the trained data augmentation model includes:
[0068] For each data augmentation variable, based on the random number assigned to each training image, a variable value of the data augmentation variable for the training image is determined from a range represented by an upper limit and a lower limit of the variable value of the data augmentation variable.
[0069] In a third aspect, an embodiment of the present application provides a training device for a data enhancement model, the device comprising:
[0070] A training image acquisition module, used for acquiring training images;
[0071] A variable value generation module, used to generate variable values of data enhancement variables of each training image using the data enhancement model of the initial structure;
[0072] A detection frame transformation module, used to transform the detection frame of the object in each training image by using the variable value of the data enhancement variable of the training image to obtain a first detection frame;
[0073] A loss value determination module, configured to determine a model loss value based on a difference in geometric features between a detection box representing an object of the same category in a test image and a first detection box representing the object of the same category;
[0074] The parameter adjustment module is used to adjust the model parameters of the data enhancement model of the initial structure based on the model loss value until the preset convergence conditions are reached to obtain a trained data enhancement model.
[0075] In a fourth aspect, an embodiment of the present application provides a data enhancement device, the device comprising:
[0076] An acquisition module, used for acquiring training images;
[0077] A variable value generation module, used to generate variable values of data enhancement variables of each training image using the trained data enhancement model; wherein the data enhancement model is trained according to the training method of the data enhancement model described in any one of the above items;
[0078] The enhancement module is used to perform data enhancement processing on each training image using the variable value of the data enhancement variable of the training image to obtain a data-enhanced training image.
[0079] In a fifth aspect, an embodiment of the present application provides an electronic device, including:
[0080] Memory, used to store computer programs;
[0081] The processor is used to implement the training method of the data enhancement model or the data enhancement method described in any of the above items when executing the program stored in the memory.
[0082] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the training method of the data enhancement model described in any one of the above items, or the data enhancement method, is implemented.
[0083] In the seventh aspect, an embodiment of the present application provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes any one of the data enhancement model training methods or data enhancement methods described above.
[0084] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described data enhancement model training methods or data enhancement methods.
[0085] Beneficial effects of the embodiments of the present application:
[0086] The training method of the data enhancement model provided in the embodiment of the present application obtains a training image; generates variable values of data enhancement variables of each training image by using the data enhancement model of the initial structure; transforms the detection frame of the object in the training image by using the variable values of the data enhancement variables of each training image to obtain a first detection frame; determines a model loss value based on the difference in geometric features between a detection frame representing an object of the same category in a test image and a first detection frame representing the object of the same category; and adjusts model parameters of the data enhancement model of the initial structure based on the model loss value until a preset convergence condition is reached to obtain a trained data enhancement model.
[0087] In this solution, the model loss value of the data enhancement model is determined based on the difference in geometric features between the detection frame representing the same category of objects in the test image and the first detection frame representing the same category of objects, so that the model loss value can constrain the geometric features of the detection frame after data enhancement to fit the geometric features of the detection frame representing the same category of objects in the test image, that is, the distribution of the detection frames of each category in the training image after data enhancement fits the actual distribution of the test image. In addition, the above process can be achieved by automatically adjusting the model parameters of the data enhancement model through electronic equipment. Compared with the solutions in the prior art, it can save the process of manual adjustment of hyperparameters by professionals, reducing labor costs.
[0088] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.
[0090] Figure 1 A flowchart of a method for training a data enhancement model provided in an embodiment of the present application;
[0091] Figure 2 A schematic diagram of variable values of data enhancement variables output based on a neural network model in an embodiment of the present application;
[0092] Figure 3 Another flowchart of the training method of the data enhancement model provided in the embodiment of the present application;
[0093] Figure 4 A flowchart of a data enhancement method provided in an embodiment of the present application;
[0094] Figure 5 Schematic diagram of a detection frame in a test image in an embodiment of the present application;
[0095] Figure 6 Schematic diagram of a detection frame in a training image in an embodiment of the present application;
[0096] Figure 7 A schematic diagram of a detection frame in a training image after data enhancement in an embodiment of the present application;
[0097] Figure 8 Another schematic diagram of a detection frame in a training image after data enhancement in an embodiment of the present application;
[0098] Fig. 9 A schematic diagram of the structure of a training device for a data enhancement model provided in an embodiment of the present application;
[0099] Fig.10 A schematic diagram of the structure of a data enhancement device provided in an embodiment of the present application;
[0100] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0101] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.
[0102] In order to save the process of manually adjusting hyperparameters by professionals and reduce labor costs, the present application embodiment provides a data enhancement model training method, a data enhancement method, and a device. The data enhancement model training method and the data enhancement method can be applied to various electronic devices, such as personal computers, servers, and other devices with data processing capabilities.
[0103] The following first introduces the training method of the data enhancement model provided in the embodiment of the present application. The method may include the following steps:
[0104] Get training images;
[0105] Generate variable values of data augmentation variables for each training image using the data augmentation model of the initial structure;
[0106] Using the variable value of the data enhancement variable of each training image, transforming the detection frame of the object in the training image to obtain a first detection frame;
[0107] Determining a model loss value based on a difference in geometric features between a detection box representing an object of the same category in a test image and a first detection box representing the object of the same category;
[0108] Based on the model loss value, the model parameters of the data enhancement model of the initial structure are adjusted until the preset convergence conditions are reached to obtain a trained data enhancement model.
[0109] In this embodiment, the model loss value of the data enhancement model is determined based on the difference in geometric features between the detection frame representing the same category of objects in the test image and the first detection frame representing the same category of objects, so that the model loss value can constrain the geometric features of the detection frame after data enhancement to fit the geometric features of the detection frame representing the same category of objects in the test image, that is, the distribution of the detection frames of each category in the training image after data enhancement fits the actual distribution of the test image. In addition, the above process can be achieved by automatically adjusting the model parameters of the data enhancement model through electronic equipment. Compared with the solutions in the prior art, it can save the process of manual adjustment of hyperparameters by professionals, thereby reducing labor costs.
[0110] The following is an introduction to the training method of the data enhancement model provided in the embodiment of the present application in conjunction with the accompanying drawings. Figure 1As shown, the method comprises the following steps:
[0111] S101, obtaining a training image;
[0112] In this embodiment, the acquired training images are images used to train the target detection model. The target detection model is a deep learning model used to implement the target detection task. The target detection task can be a detection task of identifying and locating one or more specific target objects in an image or video frame, for example, in a detection scenario for identifying a vehicle, a detection task for identifying a license plate object. Exemplarily, the target detection model can be an SSD (Single Shot MultiBox Detector) model, a DETR (Detection Transformer) model, and the like.
[0113] In this step, all or part of the training images can be obtained from the training set for training the target detection model as the training images for training the target detection model in this embodiment, and a training set for training the data enhancement model can be constructed based on the obtained training images. Among them, the training set for training the target detection model can be a training set composed of images in an open source image library on a website, or a training set composed of images generated by AI (Artificial Intelligence), which is reasonable.
[0114] It is understandable that the training of the data enhancement model often requires multiple rounds of training iterations. In each round of training, all images in the training set can be used to train the data enhancement model. Alternatively, in order to save computing resources, some images can be selected from the training set for each round of training as training images for the data enhancement model of the initial structure in this round. For example, a random small batch of images can be selected from the training set for the data enhancement model of the initial structure in this round. Compared with using all images in the training set for model training in this round in each round, this can save computing resources and improve training efficiency.
[0115] In the actual implementation process, a pre-configured sampler can be used to select training images from the training set. The sampler can select training images according to a preset selection rule. For example, the training set can be randomly sampled in small batches to obtain training images. The sampler can also be a sampler used in the process of training the target detection model.
[0116] S102, generating variable values of data enhancement variables of each training image using the data enhancement model of the initial structure;
[0117] The data enhancement variables in this embodiment may include: variables for performing affine transformation on an image, and / or variables for performing perspective projection transformation on an image. The variables for performing affine transformation on an image may be a transformation matrix for performing affine transformation on an image, which usually includes 6 variable values, or may be only a scaling ratio for scaling an image, and / or a rotation angle for rotating an image, etc. The variables for performing perspective projection transformation on an image may be a transformation matrix for performing perspective projection transformation on an image, which usually includes 8 variable values.
[0118] During the training process, the data enhancement model of the initial structure is used to generate the variable values of the data enhancement variables, which can be a neural network model or other models. For example, a simple data enhancement model is introduced below:
[0119] The model parameters of the data enhancement model of the initial structure may include: an upper limit and a lower limit of the variable value of at least one data enhancement variable, for example, a lower limit of the variable value of the scaling ratio , the upper limit of the variable value of the scaling ratio ; The lower limit of the variable value of the rotation angle , the upper limit of the variable value of the rotation angle The process of generating variable values by the data augmentation model may include the following steps:
[0120] Step A1, for each training image, assign a random number to each data augmentation variable;
[0121] Step A2, based on the random number assigned to each data augmentation variable, determine a value from the range represented by the upper and lower limits of the variable value of the data augmentation variable as the variable value of the training image for the data augmentation variable.
[0122] In this implementation, a random number can be generated from a preset value range using a random number generation function, and the generated random number can be assigned to each data enhancement variable. For each data enhancement variable, by using the random number assigned to the data enhancement variable, the variable value of the data enhancement variable can be determined from the range represented by the variable value upper limit and variable value lower limit of the data enhancement variable. For example, in one implementation, for the data enhancement variable of the scaling ratio, the variable value lower limit of the scaling ratio is , the upper limit of the variable value of the scaling ratio , if a random number z is generated from the numerical range of (x1, x2), the variable value of the data augmentation variable can be For example, if is 1, is 2, (x1,x2) is (0,100), and z is 10, then the calculated variable value is 1.1.
[0123] In another implementation, step A2 may specifically include:
[0124] Step A21, for each data enhancement variable, using the random number assigned to each data enhancement variable, obtain a weight obtained by weighting the variable value upper limit and the variable value lower limit of the data enhancement variable;
[0125] Step A22: According to the determined weight, weighted addition is performed on the variable value upper limit and the variable value lower limit of the data enhancement variable to obtain the variable value of the data enhancement variable of the training image.
[0126] In this implementation, the random number assigned to each data augmentation variable can be a random number in the interval [0,1], and then the random number assigned to each data augmentation variable and the difference between 1 and the random number can be directly used as the weighted weight. For example, the random number assigned to the scaling ratio is denoted as r1, and the random number assigned to the rotation angle is denoted as r2. Then the variable value of the data augmentation variable under the scaling ratio dimension can be ; The variable value in the rotation angle dimension can be .
[0127] It can be seen that this embodiment obtains the weight of the weighted upper and lower limits of the variable value of the data enhancement variable by utilizing the random number assigned to each data enhancement variable, and then weightedly adds the upper and lower limits of the variable value of the data enhancement variable according to the determined weight to obtain the variable value of the data enhancement variable of the training image, which can provide a basis for generating variable values of data enhancement variables for the data enhancement model of the initial structure.
[0128] S103, using the variable value of the data enhancement variable of each training image, transforming the detection frame of the object in the training image to obtain a first detection frame;
[0129] Since the present embodiment mainly utilizes the detection frame to train the target detection model, in this step, it is only necessary to transform the detection frame of the object in the training image, that is, to transform the rotation angle and size of the detection frame, without transforming the entire training image.
[0130] If the data augmentation variables include variables for performing an affine transformation on an image, after obtaining the variable values of the data augmentation variables for each training image, the detection box of the object in the training image can be subjected to an affine transformation; similarly, if the data augmentation variables include variables for performing a perspective projection transformation on an image, the detection box of the object in the training image can be subjected to a perspective projection transformation.
[0131] For example, if the data enhancement variable is the scaling ratio and the variable value is 0.8, the size of the detection frame can be scaled to 0.8 times the original size; if the data enhancement variable is the rotation angle and the variable value is 10, the detection frame can be rotated 10 degrees clockwise around the center point.
[0132] S104, determining a model loss value based on a difference in geometric features between a detection box representing an object of the same category in the test image and a first detection box representing the object of the same category;
[0133] Exemplarily, in a detection scenario for identifying a vehicle, the detection frame may include a body detection frame and a license plate detection frame. In this case, the body detection frame in each image is a detection frame of an object of the same category, and the license plate detection frame in each image is a detection frame of an object of the same category. Alternatively, if the detection frame only includes a license plate detection frame, the license plate detection frames corresponding to the license plates of the same color in each image are detection frames of an object of the same category. In this case, the yellow license plate is an object of the same category, and the green license plate is an object of the same category. It should be noted that in actual applications, objects of the same category can be set according to actual needs, and the embodiments of the present application do not limit this.
[0134] The geometric features of the detection frame may include the rotation angle and size of the detection frame. In one implementation, each detection frame is a quadrilateral, including four endpoints at the upper left corner, lower left corner, upper right corner and lower right corner, and the difference in geometric features between two detection frames can be calculated by the following steps:
[0135] Align the center points of the two detection frames, and then calculate the distance between the upper left corner endpoints, the lower left corner endpoints, the upper right corner endpoints, and the lower right corner endpoints of the two detection frames respectively, and then calculate the average value of each distance as the difference in geometric features between the two detection frames.
[0136] In this embodiment, the test image is used to provide a reference for the real distribution of the detection frame. Exemplarily, in practical applications, the test image can be an image from a test set of the target detection model, or an image from a training set of the target detection model, or an image from an actual detection scene, which is reasonable.
[0137] In one implementation, the test image is preset according to the detection scene of the target detection model. In this implementation, the test image will be closer to the actual detection scene than the training image. Then, the model loss value can constrain the training image after data enhancement to fit the detection scene of the target detection model. Then, based on the model loss value, the model parameters of the data enhancement model are adjusted, so that the image obtained after data enhancement processing according to the variable values of the data enhancement variables output by the trained data enhancement model can fit the detection scene, so as to improve the quality of the training data after data enhancement, and then improve the performance of the trained target detection model.
[0138] In one implementation, when the target detection model is used to detect targets in images captured by an image acquisition device, the test image can be an image captured by the image acquisition device in an actual detection scenario, and the image can also be manually annotated or other existing models can be used to perform target recognition on the image to generate a detection frame representing the object in the image. In this embodiment, multiple test images can be pre-acquired to form a test set for training the target detection model.
[0139] Correspondingly, detection frames in some images may be selected from multiple test images as detection frames of test images for training the data enhancement model of the initial structure in this round.
[0140] In one implementation, the model loss value can be calculated as follows:
[0141] The first detection frame and the detection frame in the test image (hereinafter referred to as the second detection frame) representing the same category of objects are divided into a group respectively; for each group of detection frames, the difference in geometric features between each first detection frame in the group and the second detection frame in the group is calculated respectively, and the sum of the differences in each group is calculated as the total difference value corresponding to the group; finally, the average value of the total difference values corresponding to each group is taken as the model loss value.
[0142] S105, adjusting the model parameters of the data enhancement model of the initial structure based on the model loss value until the preset convergence conditions are reached to obtain a trained data enhancement model.
[0143] The specific method of adjusting the parameters is not limited in this embodiment. For example, the parameters can be adjusted by a simulated annealing algorithm. If the data enhancement model is a fully gradient-differentiable model, such as a neural network model, the parameters can also be adjusted by a back propagation algorithm. The preset convergence condition can be the convergence of the model loss value.
[0144] After obtaining the trained data enhancement model, the trained data enhancement model can be used to generate the variable value of the data enhancement variable for each training image, and then the variable value of the data enhancement variable of each training image can be used to perform data enhancement processing on the training image to obtain the data enhanced training image. While performing data enhancement processing on the training image, the detection frame of the object in the training image is actually transformed, so that the data enhanced training image can be used to train the target detection model.
[0145] In this embodiment, the model loss value of the data enhancement model is determined based on the difference in geometric features between the detection frame representing the same category of objects in the test image and the first detection frame representing the same category of objects, so that the model loss value can constrain the geometric features of the detection frame after data enhancement to fit the geometric features of the detection frame representing the same category of objects in the test image, that is, the distribution of the detection frames of each category in the training image after data enhancement fits the actual distribution of the test image. In addition, the above process can be achieved by automatically adjusting the model parameters of the data enhancement model through electronic equipment. Compared with the solutions in the prior art, it can save the process of manual adjustment of hyperparameters by professionals, thereby reducing labor costs.
[0146] In one embodiment of the present application, the data enhancement model of the initial structure may be a neural network model, and includes a first sub-network and a second sub-network. In one implementation, the first sub-network and the second sub-network may both be structures of a fully connected layer, such as Figure 2 In this case, the data enhancement model using the initial structure generates the variable value of the data enhancement variable for each training image, which may include:
[0147] Step B1, for each training image, using the first sub-network to extract features from the label information of the detection frame in the training image, and aggregating the extracted features to obtain aggregated features; wherein the label information of each detection frame includes: information representing the geometric features of the detection frame;
[0148] In this step, the label information of each detection frame in a training image can be input into the first sub-network respectively, and the features output by the first sub-network for each detection frame of the training image can be obtained, and then the features output for the training image can be aggregated. The aggregation method can be: adding each feature, or calculating the average value of each feature.
[0149] In one implementation, the label information of each detection box may further include: the category of the object represented by the detection box. Specifically, the category of the object represented by the detection box may be represented by a one-hot vector.
[0150] Step B2, generating random information corresponding to the training image; wherein each random information includes a plurality of random numbers;
[0151] In this step, multiple random numbers may be generated for each training image, and the random numbers of the training image may be combined into a vector for subsequent model input. Each of the generated random numbers may be a number in the interval [0,1].
[0152] Step B3, inputting the obtained aggregated features, the size of the training image, and the random information corresponding to the training image into the second sub-network to obtain the variable value of the data enhancement variable output by the second sub-network.
[0153] The size of the training image may be the width and height of the image. In this step, for each training image, the obtained aggregated features, the size of the training image, and the random information corresponding to the training image may be combined into a vector, and then the vector is input into the second sub-network to obtain the variable value of the data augmentation variable output by the second sub-network for the training image.
[0154] In one implementation, in order to improve the diversity of the first detection frame obtained after the transformation process, after the detection frame of the object in the training image is transformed using the variable value of the data enhancement variable of each training image to obtain the first detection frame, the step of generating the random information corresponding to the training image can be returned to perform the above step, and the obtained aggregated features, the size of the training image, and the random information corresponding to the training image are input into the second sub-network again to obtain the variable value of the data enhancement variable output by the second sub-network; the detection frame of the object in the training image is transformed using the variable value of the data enhancement variable of each training image to obtain the first detection frame. Until the corresponding first specified number of first detection frames are obtained for each detection frame in the training image. That is, for each training image, the random information corresponding to the training image can be generated multiple times to obtain different variable values using multiple different random information. Then, the detection frame in the training image is transformed using the different variable values to obtain multiple first detection frames, thereby achieving the expansion of the training samples.
[0155] The first designated number can be set according to experience and requirements. For example, the first designated number can be set to 8. Then, if there are N training images, and there is a detection frame of an object in each training image, 8N first detection frames can be obtained after processing in the above processing method. It can be understood that by performing multiple transformations on each detection frame using random information, the diversity of the detection frames of the training images can be improved, and the effect of data enhancement model training can be improved.
[0156] In this embodiment, the variable value of the data enhancement variable is obtained by inputting the aggregated features, the size of the training image, and the random information corresponding to the training image into the neural network model. By utilizing the neural network model, more complex data enhancement strategies can be expressed, thereby achieving better data enhancement effects.
[0157] In one embodiment of the present application, determining the model loss value based on the difference in geometric features between a detection box representing an object of the same category in a test image and a first detection box representing the object of the same category may include:
[0158] For each first detection frame, determine a detection frame in the test image that represents an object of the same category as the first detection frame as the detection frame to be compared corresponding to the first detection frame; determine the difference in geometric features between the first detection frame and the corresponding detection frames to be compared, and determine the smallest difference as the first difference corresponding to the first detection frame; determine the average level of the first differences corresponding to the first detection frames as the first sub-loss value;
[0159] And / or, for each second detection frame in the test image, determine a first detection frame representing an object of the same category as the second detection frame as the detection frame to be compared corresponding to the second detection frame; determine the difference in geometric features between the second detection frame and the corresponding detection frames to be compared, and determine the smallest difference as the second difference corresponding to the second detection frame; determine the average level of the second differences corresponding to the second detection frames as the second sub-loss value;
[0160] Finally, based on the determined sub-loss values, the model loss value is obtained. For example, after determining the first sub-loss value and the second sub-loss value between the detection frame representing the same category of objects in the test image and the first detection frame representing the same category of objects, the sum of the first sub-loss value and the second sub-loss value corresponding to each group (i.e., each category of objects) can be used as the model loss value. In one implementation, the difference in geometric features between the two detection frames can be the inverse of the intersection-over-union ratio when the center points of the two detection frames coincide.
[0161] The average level of the first differences corresponding to the above-mentioned first detection frames can be the average value or standard deviation of the first differences corresponding to the first detection frames, and the average level of the second differences corresponding to the second detection frames can be the average value or standard deviation of the second differences corresponding to the second detection frames, and the like.
[0162] In this implementation, the first sub-loss value can constrain the similarity of the geometric features of each detection frame of the training image after data enhancement and the detection frame of the test image, so as to avoid the result of data enhancement falling outside the distribution of the test image, thereby avoiding additional burden on the training of the target detection model. The second loss value can constrain the similarity of the geometric features of each detection frame of the test image and the detection frame of the training image after data enhancement, that is, constrain each detection frame of the test image to find a similar detection frame in the detection frame of the training image after data enhancement, thereby avoiding underfitting of the target detection model.
[0163] Specifically, in one implementation, the first difference corresponding to each first detection frame may be calculated according to the following formula:
[0164] ;
[0165] At this time, the first sub-loss value can be calculated according to the following formula:
[0166] ;
[0167] in, is the first detection frame obtained by the k-th transformation of the i-th detection frame in the acquired training image; The first detection frame The corresponding j-th detection frame to be compared; Indicates the first detection box The detection box to be compared When the center points of the first detection frame coincide with The detection box to be compared The difference between the coordinates of the endpoints of The number of corresponding detection boxes to be compared; The first detection frame The corresponding first difference; Indicates taking the minimum value; represents the first sub-loss value, M' represents the number of detection boxes in the acquired training image, and P represents the first specified number.
[0168] Exemplarily, the first detection frame The detection box to be compared The difference between the coordinates of the endpoints The calculation method can be: determine the first detection frame When the center point of the first detection box is located at the first specified point, the first detection box The coordinates of the four endpoints of the vector 1, and determine the detection box to be compared When the center point of is located at the first specified point, the detection box to be compared Then, the difference between vector 1 and vector 2 is calculated using the preset loss function to obtain the first detection box. The detection box to be compared The difference between the coordinates of the endpoints of . The preset loss function may be an L1 (absolute value error) loss function, an L2 (mean square error) loss function, or the like.
[0169] Similarly, the second difference corresponding to each second detection frame can be calculated according to the following formula:
[0170] ;
[0171] At this time, the second sub-loss value can be calculated according to the following formula:
[0172] ;
[0173] in, is the jth second detection frame; The second detection box in the acquired training image represents the first detection box obtained by the k-th transformation of the i-th detection box of the same category object; S Indicates the first detection box With the second detection box When the center points of the first detection frame coincide with With the second detection box The difference between the coordinates of the endpoints of The number of detection boxes representing objects of the same category; Represents the second detection box The corresponding second difference; Indicates taking the minimum value; represents the second sub-loss value, and N' is the number of second detection boxes.
[0174] Exemplarily, the first detection frame With the second detection box The difference S between the coordinates of the endpoints The calculation method can be: determine the first detection frame When the center point of the first detection box is located at the second specified point, The coordinates of the four endpoints of the vector 3, and determine the second detection frame When the center point of is located at the second specified point, the second detection box Then, the difference between vector 3 and vector 4 is calculated using the preset loss function to obtain the first detection frame. With the second detection box The difference between the coordinates of the endpoints of . The preset loss function may be an L1 loss function, an L2 loss function, or the like.
[0175] The first designated point and the second designated point may be any coordinate points. For the convenience of calculation, the first designated point and the second designated point may be the coordinate origin for mapping the first detection frame and the second detection frame to a preset image coordinate system.
[0176] In one embodiment of the present application, before obtaining the model loss value based on the determined sub-loss value, the training method of the data augmentation model may further include the following steps:
[0177] Step C1, for each second detection frame in the test image, determining a detection frame in the acquired training image that represents an object of the same category as the second detection frame as an original detection frame corresponding to the second detection frame;
[0178] Step C2, determining a correlation value between the second detection frame and the original detection frame based on the maximum value of geometric feature similarities between each first detection frame obtained after transformation processing of each original detection frame corresponding to the second detection frame and the second detection frame;
[0179] Exemplarily, a method for determining the association value between the second detection frame and each original detection frame may be: first calculate the geometric feature similarity between at least one first detection frame obtained after the transformation of the original detection frame and the second detection frame to obtain multiple similarity values, and then compare the maximum value of the multiple similarity values with a first threshold; if the maximum value is greater than the first threshold, then determine that the association value between the second detection frame and the original detection frame is 0; if the maximum value is less than the first threshold, then determine that the association value between the second detection frame and the original detection frame is the maximum value.
[0180] The similarity of the geometric features of two detection frames may be the intersection-and-union ratio when the center points of the two detection frames coincide, or may be the inverse of the difference in geometric features calculated by the above embodiment, and is certainly not limited thereto.
[0181] Step C3, determining a diversity degree value corresponding to the second detection frame based on the correlation value between the second detection frame and the corresponding original detection frames;
[0182] Exemplarily, the sum or average value of the correlation values between the second detection frame and the corresponding original detection frames may be used as the diversity degree value corresponding to the second detection frame, which is reasonable.
[0183] Step C4, determining a third sub-loss value based on the average level of the diversity values corresponding to each second detection frame; wherein the third sub-loss value is negatively correlated with the average level of the diversity values corresponding to each second detection frame.
[0184] For example, the average level of the diversity values corresponding to the second detection frames may be the average value, standard deviation, etc. of the diversity values corresponding to the second detection frames. The third sub-loss value may be the opposite number of the average level of the diversity values corresponding to the second detection frames, or the reciprocal, etc.
[0185] Furthermore, in order to balance the influence of the third sub-loss value and other sub-loss values on the model loss value, that is, to maintain the balance between the influence of each sub-loss value on the model loss value, the growth rate of the diversity degree value corresponding to each second detection frame can be limited to be negatively correlated with each associated value corresponding to the second detection frame. For example, the growth rate of the diversity degree value corresponding to each second detection frame and each associated value corresponding to the second detection frame can be a logarithmic function relationship.
[0186] Specifically, in one implementation, the diversity value corresponding to each second detection frame may be calculated according to the following formula:
[0187] ;
[0188] At this time, the third sub-loss value can be calculated according to the following formula:
[0189] ;
[0190] in, is the jth second detection frame; The second detection box in the acquired training image Represents the i-th detection box of the same category object The first detection frame obtained after the k-th transformation process; The first detection frame and the second detection box The intersection-over-union ratio when the center points coincide is used to indicate the similarity of the geometric features of the two detection frames; P indicates the first specified number; Indicates taking the maximum value; Represents the second detection box With detection box The correlation value between the original detection frame from which it comes; M is the correlation value between the second detection frame in the acquired training image The number of detection boxes representing objects of the same category; and are preset constants. Can be used to adjust the weight of the third sub-loss value, It can be used to avoid the value after taking the logarithm being too small; The second detection frame The corresponding diversity degree value, D is the first threshold; represents the third sub-loss value, and N' is the number of second detection boxes. , The first threshold value can be set according to actual conditions and experience. For example, the first threshold value can be 0.5. is 0.3, is 1.
[0191] In this case, the above-mentioned method of obtaining the model loss value based on the determined sub-loss value may include:
[0192] The sum of the determined sub-loss values is calculated to obtain the model loss value.
[0193] Exemplarily, if the determined sub-loss values include a first sub-loss value, a second sub-loss value, and a third sub-loss value, the sum of the first sub-loss value, the second sub-loss value, and the third sub-loss value is calculated to obtain the model loss value. Calculating the sum of the first sub-loss value, the second sub-loss value, and the third sub-loss value means calculating the sum of the first sub-loss value, the second sub-loss value, and the third sub-loss value corresponding to each group.
[0194] In addition, it is worth mentioning that in one implementation, the maximum number of times the i-th detection box in the above-obtained training image is transformed can be 1 time. At this time, P=1 can be substituted into the above formula to calculate the first difference, the second difference and the diversity value.
[0195] In this embodiment, for each second detection frame in the test image, based on the number of first detection frames matching the second detection frame, the diversity value corresponding to the second detection frame is determined; wherein the diversity value corresponding to the second detection frame is positively correlated with the number of matched first detection frames; the matched first detection frame and the second detection frame represent objects of the same category, and the similarity of geometric features is greater than a first threshold, and then based on the average level of the diversity values corresponding to each second detection frame, a third sub-loss value is determined, and the third sub-loss value is negatively correlated with the average level of the diversity values corresponding to each second detection frame. The third sub-loss value can constrain the second detection frame in the test image to match as many similar detection frames as possible in the training image after data enhancement, thereby generating the variable value of the data enhancement variable using the trained data enhancement model, and after the training image is subjected to data enhancement processing, the detection frame in the enhanced image obtained has a higher diversity, thereby further improving the quality of the training data of the training target detection model.
[0196] In one embodiment of the present application, the training process of the data enhancement model of the initial structure includes multiple rounds of parameter adjustment; accordingly, in the present embodiment, obtaining a training image in the above step S101 includes:
[0197] Selecting a second specified number of images from a training set containing multiple images as training images required for adjusting parameters of the data augmentation model of the initial structure in this round;
[0198] and / or,
[0199] Before determining the model loss value based on the difference in geometric features between the detection box representing the object of the same category in the test image and the first detection box representing the object of the same category in the above step S104, the method further includes:
[0200] From the multiple test images, select a third specified number of images as the test images required for adjusting the parameters of the data augmentation model of the initial structure in this round.
[0201] For example, the second specified number may be 1 / 10, 1 / 100, etc. of the total number of images in the training set. The third specified number may be 1 / 5, 1 / 10, etc. of the total number of images preset according to the detection scenario of the target detection model. The second specified number of images may be selected from the training set, and the third specified number of images may be selected randomly or in the order of the images in the image set, which is reasonable.
[0202] It is understandable that in each round of parameter adjustment training, a random small batch of images is selected from the training set and the test set to be used for training the data augmentation model of the initial structure in this round. This can save computing resources and improve training efficiency compared to using all the images in the training set for model training in this round in each round.
[0203] In one implementation, the test images selected in each round include the hard test images in the test images selected in the previous round; the hard test images can be determined from the test images selected in each round according to the following steps:
[0204] Step D1, for each second detection frame in the test image selected in each round, calculate the similarity of geometric features between the second detection frame and the first detection frame representing the same category of objects as the similarity corresponding to the second detection frame;
[0205] The similarity of the geometric features of the two detection frames in this step may be calculated in the manner described above, such as calculating the intersection-over-union ratio when the center points of the detection frames coincide.
[0206] Step D2, if the maximum value of the similarity corresponding to the second detection frame is less than the second threshold, the test image to which the second detection frame belongs is determined as a difficult test image; wherein the second threshold is determined based on the average level of the similarities corresponding to each second detection frame in the test image selected in this round.
[0207] The average level of similarities corresponding to the second detection frames may be the average value of the similarities corresponding to the second detection frames, and the average value is used as the second threshold. Alternatively, the standard deviation of the similarities corresponding to the second detection frames may be calculated, and the difference between the average value and the standard deviation of the similarities corresponding to the second detection frames may be used as the second threshold. If the maximum value of the similarities corresponding to any second detection frame is less than the second threshold, the test image to which the second detection frame belongs is determined as a difficult test image.
[0208] Furthermore, for each round of difficult test images determined, a specified proportion of difficult test images can be randomly discarded, such as a specified proportion of 10%, to prevent abnormal data from existing in the difficult test images, which would cause the round data augmentation model to be unable to fit the difficult test images.
[0209] In this embodiment, since there may be some difficult test images in the test image that are difficult to be covered by the first detection frame, that is to say, the similarity of the geometric features of the difficult test images and all the first detection frames used in this round of training is relatively low. Therefore, in this embodiment, in each round of data enhancement model training, the difficult test images are determined from the test images by online difficult example mining, and the difficult test images are used for the next round of model training. In this way, the difficult samples will be used for training more frequently, thereby further improving the training effect of the data enhancement model, so that the detection frame in the training image after data enhancement can better cover the detection frame with different geometric features in the test image, thereby improving the training effect of the target detection model trained using the training image after data enhancement.
[0210] For ease of understanding, the following Figure 3 The training method of the data enhancement model provided in this application is exemplarily introduced.
[0211] like Figure 3 As shown, sampler A is used to select training images from the training set, and sampler B is used to select second detection frames from the test set. The data enhancement model is used multiple times to generate variable values of the data enhancement variables for each detection frame in the training image, and the detection frame is transformed using the obtained variable values until a first specified number of first detection frames are obtained for each detection frame.
[0212] Afterwards, a score evaluation is performed based on the difference in geometric features between the detection box representing the same category of objects in the test image and the first detection box representing the same category of objects, that is, the model loss value is determined, and the model parameters of the data enhancement model are optimized based on the model loss value. The whole process can be performed multiple times until the preset convergence condition is reached, and a trained data enhancement model is obtained.
[0213] Based on the training method of the above data enhancement model, the present application embodiment provides a data enhancement method, such as Figure 4 As shown, the data enhancement method includes the following steps:
[0214] S401, obtaining a training image;
[0215] S402, generating a variable value of a data enhancement variable of each training image using the trained data enhancement model;
[0216] The data enhancement model is obtained by training according to the training method of the above data enhancement model;
[0217] S403, performing data enhancement processing on each training image using the variable value of the data enhancement variable of the training image to obtain a data enhanced training image.
[0218] For example, if the data augmentation variable is a variable used to perform an affine transformation on an image, the training image can be affine transformed according to the variable value of the variable; if the data augmentation variable is a variable used to perform a perspective projection transformation on an image, the training image can be perspective projected according to the variable value of the variable.
[0219] In one implementation, the data enhancement model is a neural network model and includes: a first subnetwork and a second subnetwork; then the variable value of the data enhancement variable for each training image generated by the trained data enhancement model may include:
[0220] Step E1, for each training image, using the first sub-network to extract features from the label information of each detection frame in the training image, and aggregating the extracted features to obtain aggregated features; wherein the label information of each detection frame includes: information representing the geometric features of the detection frame;
[0221] Step E2, generating random information corresponding to the training image; wherein each random information includes a plurality of random numbers;
[0222] Step E3, inputting the obtained aggregated features, the size representing the training image, and the random information corresponding to the training image into the second sub-network to obtain the variable value of the data enhancement variable output by the second sub-network.
[0223] It should be noted that step E1 can refer to the relevant description of step B1, step E2 can refer to the relevant description of step B2, and step E3 can refer to the relevant description of step B3, which will not be repeated here.
[0224] In this implementation, steps E2-E3 may be performed multiple times to generate variable values of multiple data enhancement variables for each training image, and then the training image may be subjected to multiple data enhancement processing using each variable value to obtain multiple data enhanced training images, so as to improve the diversity of the data enhanced training images.
[0225] In another implementation, the model parameters of the data augmentation model include: an upper limit and a lower limit of a variable value of at least one data augmentation variable; then the variable value of the data augmentation variable for each training image generated by the trained data augmentation model may include:
[0226] For each data augmentation variable, based on the random number assigned to each training image, a variable value of the data augmentation variable for the training image is determined from a range represented by an upper limit and a lower limit of the variable value of the data augmentation variable.
[0227] In this implementation, multiple random numbers may be assigned to each training image to generate multiple variable values of data enhancement variables for each training image, and then multiple data enhancement processes may be performed on the training image using each variable value to obtain multiple data enhanced training images. This implementation may refer to the relevant description of steps A1-A2 above, which will not be repeated here.
[0228] In an actual application scenario, the target detection model in the embodiment of the present application may be a license plate detection model, which can identify the location of the license plate in the image and generate a detection frame to indicate the location of the license plate. The test image preset according to the detection scenario of the target detection model can be as follows: Figure 5 As shown in the figure, the black dotted frame is the detection frame where the license plate is located. The license plate in the detected image has a larger deflection angle than the image acquisition device that acquired the image, while the training image is as follows Figure 6 For the image with a small deflection angle shown in the figure, the traditional manual adjustment method is used to perform data enhancement on such training images. The obtained training images after data enhancement are often quite different from the test images, that is, the geometric features of the detection frame indicating the location of the license plate are quite different, such as Figure 7 shown.
[0229] The data enhancement method provided in the embodiment of the present application is used to perform data enhancement on such a training image, and the obtained data enhanced training image is as follows: Figure 8 As shown, it can be seen that the training image after data enhancement obtained by this embodiment is more similar to the test image, that is, the geometric features of the detection frame representing the location of the license plate are more similar. Therefore, the training image after data enhancement obtained by this solution can better fit the actual detection scene of the target detection model.
[0230] In this embodiment, the model loss value of the data enhancement model is determined based on the difference in geometric features between the detection box representing the same category of objects in the test image and the first detection box representing the same category of objects, and the test image is preset according to the detection scene of the target detection model, so that the model loss value can constrain the geometric features of the detection box after data enhancement to fit the detection scene of the target detection model. Then, the model parameters of the data enhancement model are adjusted based on the model loss value, which can save the process of manually adjusting the hyperparameters by professionals and reduce labor costs. In addition, the image obtained after data enhancement processing according to the variable values of the data enhancement variables output by the trained data enhancement model can fit the detection scene, so as to improve the quality of the training data after data enhancement, and then improve the performance of the trained target detection model.
[0231] Based on the same inventive concept, the present application embodiment also provides a training device for a data enhancement model, such as Fig. 9 As shown, the device comprises:
[0232] A training image acquisition module 901 is used to acquire a training image;
[0233] A variable value generating module 902, for generating a variable value of a data enhancement variable of each training image using a data enhancement model of an initial structure;
[0234] A detection frame transformation module 903 is used to transform the detection frame of the object in each training image by using the variable value of the data enhancement variable of the training image to obtain a first detection frame;
[0235] A loss value determination module 904, configured to determine a model loss value based on a difference in geometric features between a detection box representing an object of the same category in a test image and a first detection box representing the object of the same category;
[0236] The parameter adjustment module 905 is used to adjust the model parameters of the data enhancement model of the initial structure based on the model loss value until the preset convergence conditions are reached to obtain a trained data enhancement model.
[0237] Optionally, the data enhancement model of the initial structure is a neural network model, and includes a first sub-network and a second sub-network;
[0238] The variable value generating module 902 includes:
[0239] A feature extraction submodule, for extracting features of the label information of the detection frame in each training image using the first subnetwork, and aggregating the extracted features to obtain aggregated features; wherein the label information of each detection frame includes: information representing the geometric features of the detection frame;
[0240] A random information generation submodule is used to generate random information corresponding to the training image; wherein each random information includes a plurality of random numbers;
[0241] The variable value generation submodule is used to input the obtained aggregated features, the size of the training image, and the random information corresponding to the training image into the second subnetwork to obtain the variable value of the data enhancement variable output by the second subnetwork.
[0242] Optionally, the parameter adjustment module 905 is also used to, after transforming the detection frame of the object in the training image using the variable value of the data enhancement variable of each training image to obtain the first detection frame, trigger the random information generation submodule to execute the step of generating random information corresponding to the training image until a corresponding first specified number of first detection frames are obtained for each detection frame in the training image.
[0243] Optionally, the label information of each detection frame also includes: the category of the object represented by the detection frame.
[0244] Optionally, the model parameters of the data enhancement model of the initial structure include: an upper limit and a lower limit of a variable value of at least one data enhancement variable;
[0245] The variable value generating module 902 includes:
[0246] A random number allocation submodule, for assigning a random number to each data augmentation variable for each training image;
[0247] The prediction submodule is used to determine a numerical value from the range represented by the upper limit and the lower limit of the variable value of each data enhancement variable based on the random number assigned to the data enhancement variable, as the variable value of the training image for the data enhancement variable.
[0248] Optionally, the prediction submodule includes:
[0249] A weight determination unit, for each data enhancement variable, using a random number assigned to each data enhancement variable to obtain a weight weighted by an upper limit of a variable value and a lower limit of a variable value of the data enhancement variable;
[0250] According to the determined weight, the variable value upper limit and the variable value lower limit of the data enhancement variable are weightedly added to obtain the variable value of the data enhancement variable of the training image.
[0251] Optionally, the loss value determination module 904 includes:
[0252] A sub-loss value determination module is used to determine, for each first detection frame, a detection frame in the test image that represents an object of the same category as the first detection frame as the detection frame to be compared corresponding to the first detection frame; determine the difference in geometric features between the first detection frame and the corresponding detection frames to be compared, and determine the smallest difference as the first difference corresponding to the first detection frame; determine the average level of the first differences corresponding to the first detection frames as the first sub-loss value; and / or, for each second detection frame in the test image, determine the first detection frame that represents an object of the same category as the second detection frame as the detection frame to be compared corresponding to the second detection frame; determine the difference in geometric features between the second detection frame and the corresponding detection frames to be compared, and determine the smallest difference as the second difference corresponding to the second detection frame; determine the average level of the second differences corresponding to the second detection frames as the second sub-loss value;
[0253] The loss value determination submodule is used to obtain the model loss value based on the determined sub-loss value.
[0254] Optionally, the first difference corresponding to each first detection frame is calculated according to the following formula:
[0255] ;
[0256] in, is the first detection frame obtained by the k-th transformation of the i-th detection frame in the acquired training image; The first detection frame The corresponding j-th detection frame to be compared; Indicates the first detection box The detection box to be compared When the center points of the first detection frame coincide with The detection box to be compared The difference between the coordinates of the endpoints of The number of corresponding detection boxes to be compared; The first detection frame The corresponding first difference; Indicates taking the minimum value;
[0257] And / or, the second difference corresponding to each second detection frame is calculated according to the following formula:
[0258] ;
[0259] in, is the jth second detection frame; The second detection box in the acquired training image represents the first detection box obtained by the k-th transformation of the i-th detection box of the same category object; S Indicates the first detection box With the second detection box When the center points of the first detection frame coincide with With the second detection box The difference between the coordinates of the endpoints of represents the number of detection boxes of objects of the same category, and P represents the first specified number; Represents the second detection box The corresponding second difference; Indicates taking the minimum value.
[0260] Optionally, the device further comprises:
[0261] an original detection frame determination module, configured to determine, before the loss value determination submodule obtains the model loss value based on the determined sub-loss value, a detection frame in the acquired training image that represents an object of the same category as the second detection frame, as an original detection frame corresponding to the second detection frame;
[0262] an association value determination module, configured to determine an association value between the second detection frame and the original detection frame based on a maximum value of geometric feature similarities between each first detection frame obtained after transformation processing of each original detection frame corresponding to the second detection frame and the second detection frame;
[0263] A diversity degree value determination module, used to determine the diversity degree value corresponding to the second detection frame based on the correlation value between the second detection frame and the corresponding original detection frames;
[0264] A third sub-loss value determination module, configured to determine a third sub-loss value based on an average level of diversity values corresponding to each second detection frame; wherein the third sub-loss value is negatively correlated with the average level of diversity values corresponding to each second detection frame;
[0265] The loss value determination submodule is specifically used to calculate the sum of the determined sub-loss values to obtain the model loss value.
[0266] Optionally, a growth rate of the diversity degree value corresponding to each second detection frame is negatively correlated with each correlation value corresponding to the second detection frame.
[0267] Optionally, the similarity of the geometric features of the two detection frames is an intersection-over-union ratio when the center points of the two detection frames coincide.
[0268] The diversity value corresponding to each second detection frame is calculated according to the following formula:
[0269] ;
[0270] in, is the jth second detection frame; The second detection box in the acquired training image Represents the i-th detection box of the same category object The first detection frame obtained after the k-th transformation process; The first detection frame and the second detection box The intersection-and-union ratio when the center points coincide; P represents the first specified number; Indicates taking the maximum value; Represents the second detection box With detection box M is the correlation value between the second detection frame and the acquired training image The number of detection boxes representing objects of the same category; and All are preset constants; The second detection frame The corresponding diversity degree value, D is the first threshold.
[0271] Optionally, the training process of the data enhancement model of the initial structure includes multiple rounds of parameter adjustment; the training image acquisition module 901 is specifically used to: select a second specified number of images from a training set containing multiple images as training images required for adjusting the parameters of the data enhancement model of the initial structure in this round; and / or, the device also includes: a detection frame selection module, which is used to select a third specified number of detection frames from multiple test images as detection frames for test images for training the data enhancement model of the initial structure in this round before the loss value determination module 904 performs the step of determining the model loss value based on the difference in geometric features between the detection frame representing the same category of objects in the test image and the first detection frame representing the same category of objects.
[0272] Optionally, the test images selected in each round include the difficult test images among the test images selected in the previous round;
[0273] The detection box selection module is specifically used to determine the difficult test images from the test images selected in each round according to the following steps:
[0274] For each second detection frame in the test image selected in each round, calculate the similarity of geometric features between the second detection frame and the first detection frame representing the object of the same category as the similarity corresponding to the second detection frame;
[0275] If the maximum value of the similarities corresponding to the second detection box is less than a second threshold, the test image to which the second detection box belongs is determined as a difficult test image; wherein the second threshold is determined based on the average level of the similarities corresponding to the second detection boxes in the test image selected in this round.
[0276] Optionally, the acquired training image is an image used to train a target detection model; and the test image is preset according to a detection scenario of the target detection model.
[0277] Optionally, the data augmentation variables include: variables for performing an affine transformation on an image, and / or variables for performing a perspective projection transformation on an image.
[0278] The present application also provides a data enhancement device, such as Fig.10 As shown, the device comprises:
[0279] An acquisition module 1001 is used to acquire a training image;
[0280] A variable value generating module 1002 is used to generate a variable value of a data enhancement variable of each training image using a trained data enhancement model; wherein the data enhancement model is trained by any of the data enhancement model training methods described above;
[0281] The enhancement module 1003 is used to perform data enhancement processing on each training image using the variable value of the data enhancement variable of the training image to obtain a data enhanced training image.
[0282] Optionally, the data enhancement model is a neural network model, and includes: a first sub-network and a second sub-network; a variable value generation module 1002, including:
[0283] an aggregation submodule, for extracting features from the label information of each detection frame in each training image using the first subnetwork, and aggregating the extracted features to obtain aggregated features; wherein the label information of each detection frame includes: information representing the geometric features of the detection frame;
[0284] A random information submodule generates random information corresponding to the training image; wherein each random information includes a plurality of random numbers;
[0285] The variable value generation submodule is used to input the obtained aggregated features, the size representing the training image, and the random information corresponding to the training image into the second subnetwork to obtain the variable value of the data enhancement variable output by the second subnetwork.
[0286] Optionally, the model parameters of the data enhancement model include: a variable value upper limit and a variable value lower limit of at least one data enhancement variable; a variable value generating module 1002, specifically configured to:
[0287] For each data augmentation variable, based on the random number assigned to each training image, a variable value of the data augmentation variable for the training image is determined from a range represented by an upper limit and a lower limit of the variable value of the data augmentation variable.
[0288] The present application also provides an electronic device, such as Fig.11 As shown, including:
[0289] The memory 1101 is used to store computer programs; the processor 1102 is used to implement any of the above-mentioned data enhancement model training methods or data enhancement methods when executing the programs stored in the memory 1101.
[0290] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1102, the communication interface, and the memory 1101 communicate with each other via the communication bus.
[0291] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0292] The communication interface is used for communication between the above electronic device and other devices.
[0293] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0294] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0295] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements any of the above-mentioned data enhancement model training methods or data enhancement method steps.
[0296] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the training method or data enhancement method of the data enhancement model described in any of the above embodiments.
[0297] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.
[0298] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0299] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and electronic device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0300] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. A training method for a data augmentation model, characterized in that: The method comprises: Get training images; Generate variable values of data augmentation variables for each training image using the data augmentation model of the initial structure; Using the variable value of the data enhancement variable of each training image, transforming the detection frame of the object in the training image to obtain a first detection frame; Determining a model loss value based on a difference in geometric features between a detection box representing an object of the same category in a test image and a first detection box representing the object of the same category; Based on the model loss value, the model parameters of the data enhancement model of the initial structure are adjusted until the preset convergence conditions are reached to obtain a trained data enhancement model.
2. The method according to claim 1, characterized in that The data enhancement model of the initial structure is a neural network model, and includes a first sub-network and a second sub-network; The data enhancement model using the initial structure generates the variable value of the data enhancement variable of each training image, including: For each training image, the first sub-network is used to extract features from the label information of the detection frame in the training image, and the extracted features are aggregated to obtain aggregated features; wherein the label information of each detection frame includes: information representing the geometric features of the detection frame; Generate random information corresponding to the training image; wherein each random information includes multiple random numbers; The obtained aggregated features, the size of the training image, and the random information corresponding to the training image are input into the second sub-network to obtain the variable value of the data enhancement variable output by the second sub-network.
3. The method according to claim 2, characterized in that After transforming the detection frame of the object in the training image using the variable value of the data enhancement variable of each training image to obtain the first detection frame, the method further includes: Return to the step of generating random information corresponding to the training image until a first specified number of first detection frames corresponding to each detection frame in the training image is obtained.
4. The method according to claim 1, characterized in that The model parameters of the data enhancement model of the initial structure include: an upper limit and a lower limit of a variable value of at least one data enhancement variable; The data enhancement model using the initial structure generates the variable value of the data enhancement variable of each training image, including: For each training image, a random number is assigned to each data augmentation variable; Based on the random number assigned to each data augmentation variable, a numerical value is determined from the range represented by the variable value upper limit and the variable value lower limit of the data augmentation variable as the variable value of the training image for the data augmentation variable.
5. The method according to claim 4, characterized in that The step of determining a value from a range represented by an upper limit and a lower limit of a variable value of each data augmentation variable based on a random number assigned to the data augmentation variable as a variable value of the training image for the data augmentation variable comprises: For each data augmentation variable, using the random number assigned to each data augmentation variable, obtain a weight weighted by the variable value upper limit and the variable value lower limit of the data augmentation variable; According to the determined weight, the variable value upper limit and the variable value lower limit of the data enhancement variable are weightedly added to obtain the variable value of the data enhancement variable of the training image.
6. The method according to claim 3, characterized in that The determining of the model loss value based on the difference in geometric features between a detection frame representing an object of the same category in the test image and a first detection frame representing the object of the same category includes: For each first detection frame, determine a detection frame in the test image that represents an object of the same category as the first detection frame as the detection frame to be compared corresponding to the first detection frame; determine the difference in geometric features between the first detection frame and the corresponding detection frames to be compared, and determine the smallest difference as the first difference corresponding to the first detection frame; determine the average level of the first differences corresponding to the first detection frames as the first sub-loss value; and / or, For each second detection frame in the test image, determine a first detection frame representing an object of the same category as the second detection frame as the detection frame to be compared corresponding to the second detection frame; determine the difference in geometric features between the second detection frame and the corresponding detection frames to be compared, and determine the smallest difference as the second difference corresponding to the second detection frame; determine the average level of the second differences corresponding to the second detection frames as the second sub-loss value; Based on the determined sub-loss values, a model loss value is obtained.
7. The method according to claim 6, characterized in that The first difference corresponding to each first detection frame is calculated according to the following formula: ; in, is the first detection frame obtained by the k-th transformation of the i-th detection frame in the acquired training image; The first detection frame The corresponding j-th detection frame to be compared; Indicates the first detection box The detection box to be compared When the center points of the first detection frame coincide with The detection box to be compared The difference between the coordinates of the endpoints of The number of corresponding detection boxes to be compared; The first detection frame The corresponding first difference; Indicates taking the minimum value; and / or, The second difference corresponding to each second detection frame is calculated according to the following formula: ; in, is the jth second detection frame; The second detection box in the acquired training image represents the first detection box obtained by the k-th transformation of the i-th detection box of the same category object; S Indicates the first detection box With the second detection box When the center points of the first detection frame coincide with With the second detection box The difference between the coordinates of the endpoints of represents the number of detection boxes of objects of the same category, and P represents the first specified number; Represents the second detection box The corresponding second difference; Indicates taking the minimum value.
8. The method according to claim 6, characterized in that Before obtaining the model loss value based on the determined sub-loss value, the method further includes: For each second detection frame in the test image, determine a detection frame in the acquired training image that represents an object of the same category as the second detection frame as an original detection frame corresponding to the second detection frame; Determine a correlation value between the second detection frame and the original detection frame based on a maximum value of geometric feature similarities between each first detection frame obtained after transformation processing of each original detection frame corresponding to the second detection frame and the second detection frame; Determine a diversity degree value corresponding to the second detection frame based on a correlation value between the second detection frame and corresponding original detection frames; Determine a third sub-loss value based on an average level of diversity values corresponding to each second detection frame; wherein the third sub-loss value is negatively correlated with the average level of diversity values corresponding to each second detection frame; The step of obtaining a model loss value based on the determined sub-loss value comprises: The sum of the determined sub-loss values is calculated to obtain the model loss value.
9. The method according to claim 8, characterized in that The growth rate of the diversity degree value corresponding to each second detection frame is negatively correlated with each correlation value corresponding to the second detection frame.
10. The method according to claim 9, characterized in that The similarity of the geometric features of two detection frames is the intersection-over-union ratio when the center points of the two detection frames coincide. The diversity value corresponding to each second detection frame is calculated according to the following formula: ; in, is the jth second detection frame; The second detection box in the acquired training image Represents the i-th detection box of the same category object The first detection frame obtained after the k-th transformation process; The first detection frame and the second detection box The intersection-and-union ratio when the center points coincide; P represents the first specified number; Indicates taking the maximum value; Represents the second detection box With detection box M is the correlation value between the second detection frame and the acquired training image The number of detection boxes representing objects of the same category; and All are preset constants; The second detection frame The corresponding diversity degree value, D is the first threshold.
11. The method according to claim 1, characterized in that: The training process of the data enhancement model of the initial structure includes multiple rounds of parameter adjustment; The obtaining of training images comprises: Selecting a second specified number of images from a training set containing multiple images as training images required for adjusting parameters of the data augmentation model of the initial structure in this round; and / or, Before determining the model loss value based on the difference in geometric features between the detection box representing the object of the same category in the test image and the first detection box representing the object of the same category, the method further includes: From the multiple test images, a third specified number of images are selected as the test images required for adjusting the parameters of the data augmentation model of the initial structure in this round.
12. The method according to claim 11, characterized in that The test images selected in each round include the hard-to-solve test images in the test images selected in the previous round; Follow these steps to determine the hard test images from the selected test images in each round: For each second detection frame in the test image selected in each round, calculate the similarity of geometric features between the second detection frame and the first detection frame representing the object of the same category as the similarity corresponding to the second detection frame; If the maximum value of the similarities corresponding to the second detection box is less than a second threshold, the test image to which the second detection box belongs is determined as a difficult test image; wherein the second threshold is determined based on the average level of the similarities corresponding to the second detection boxes in the test image selected in this round.
13. A data enhancement method, characterized in that: The method comprises: Get training images; Generate variable values of data enhancement variables of each training image using the trained data enhancement model; wherein the data enhancement model is trained according to the method according to any one of claims 1 to 12; The data enhancement processing is performed on each training image using the variable value of the data enhancement variable of the training image to obtain a data enhanced training image.
14. The method according to claim 13, characterized in that The data enhancement model is a neural network model, and includes: a first sub-network and a second sub-network; the variable value of the data enhancement variable of each training image generated by the trained data enhancement model includes: For each training image, the first sub-network is used to extract features from label information of each detection frame in the training image, and the extracted features are aggregated to obtain aggregated features; wherein the label information of each detection frame includes: information representing geometric features of the detection frame; Generate random information corresponding to the training image; wherein each random information includes multiple random numbers; Inputting the obtained aggregated features, the size representing the training image, and the random information corresponding to the training image into the second sub-network to obtain the variable value of the data augmentation variable output by the second sub-network; or, The model parameters of the data enhancement model include: an upper limit and a lower limit of a variable value of at least one data enhancement variable; the variable value of the data enhancement variable of each training image generated by using the trained data enhancement model includes: For each data augmentation variable, based on the random number assigned to each training image, a variable value of the data augmentation variable for the training image is determined from a range represented by an upper limit and a lower limit of the variable value of the data augmentation variable.
15. A training device for a data augmentation model, characterized in that: The device comprises: A training image acquisition module, used for acquiring training images; A variable value generation module, used to generate variable values of data enhancement variables of each training image using the data enhancement model of the initial structure; A detection frame transformation module, used to transform the detection frame of the object in each training image by using the variable value of the data enhancement variable of the training image to obtain a first detection frame; A loss value determination module, configured to determine a model loss value based on a difference in geometric features between a detection box representing an object of the same category in a test image and a first detection box representing the object of the same category; The parameter adjustment module is used to adjust the model parameters of the data enhancement model of the initial structure based on the model loss value until the preset convergence conditions are reached to obtain a trained data enhancement model.
