Target detection model training method, target detection method, device and equipment

Abstract

The invention provides a target detection model training method, a target detection method, a target detection device and target detection equipment. Relates to the technical field of computers. The method comprises the steps that a real data set and a simulation data set are acquired, a preset initial model is trained according to the real data set and the simulation data set, and a target detection model is obtained. Wherein the simulation data set is obtained through simulation according to the first probability of each target in the real data set, and the second probability of different types of targets in the simulation data set is larger than a preset value. According to the method provided by the invention, the simulation data set is added, uncommon targets in the real data set are supplemented, and the initial model is trained by combining the simulation data set and the real data set, so that the detection precision of the model when the uncommon targets are detected is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a target detection model training method, a target detection method, a device and equipment. Background Art

[0002] In the process of building a smart city with vehicle-road collaboration, all-round and multi-angle roadside equipment (such as roadside cameras) can be used to capture road traffic conditions, and accurate detection of traffic participants including motor vehicles, non-motor vehicles, pedestrians, traffic obstacles, etc. can be achieved by adopting target detection models.

[0003] In the related art, the road data collected by the roadside equipment is used as training samples to train the model to obtain the detection model. However, the road data collected by the roadside equipment usually presents a long-tail distribution characteristic, that is, the number of samples of common targets (such as cars) is much larger than that of uncommon targets (such as tricycles). This sample imbalance will result in the detection accuracy of the obtained detection model being low when detecting uncommon targets. Summary of the invention

[0004] The present application provides a target detection model training method, a target detection method, a device and equipment to improve the detection accuracy of the detection model when detecting uncommon targets.

[0005] In a first aspect, the present application provides a method for training a target detection model, comprising:

[0006] Acquire a real data set and a simulation data set; the simulation data set is obtained by simulating a first probability of each target in the real data set, and a second probability of different types of targets in the simulation data set is greater than a preset value;

[0007] The preset initial model is trained according to the real data set and the simulation data set to obtain a target detection model.

[0008] In a possible implementation, the initial model includes a backbone network, a detection head, and a domain classifier;

[0009] The step of training a preset initial model according to the real data set and the simulation data set to obtain a target detection model includes:

[0010] According to the real data set and the simulation data set, the initial model is trained, and the parameters of the backbone network, the detection head, and the domain classifier are updated to obtain a model with updated parameters;

[0011] The domain classifier in the model after the parameter update is removed to obtain the object detection model.

[0012] In a possible implementation, the training of the initial model according to the real data set and the simulation data set, updating the parameters of the backbone network, the detection head, and the domain classifier, and obtaining the model after the updated parameters includes:

[0013] According to the real data set and the simulation data set, the initial model is trained for the first round, and the parameters of the backbone network, the detection head, and the domain classifier are updated respectively to obtain an intermediate training model;

[0014] According to the real data set, a second round of training is performed on the intermediate training model, and the parameters of the detection head in the model are updated again to obtain the model with updated parameters.

[0015] In a possible implementation, the obtaining of the real data set and the simulation data set includes:

[0016] Calibrate the type and position of the target in each frame of the real video data collected in the real scene to obtain the real data set;

[0017] Obtaining a first probability of each type of target in the real data set in the total number of targets corresponding to the real data set;

[0018] For each type of target, determine a second probability of the target according to a first probability of the target in the real data set, where the second probability is used to indicate the proportion of the target in the total number of targets in the simulation data;

[0019] According to the second probability of each type of target, a simulation simulator is used to perform data simulation to obtain the simulation data set.

[0020] In a possible implementation manner, determining the second probability of the target according to the first probability of the target in the real data set includes:

[0021] A second probability of the target is calculated based on the first probability of the target, the number of types of targets in the real data set, and a preset hyperparameter.

[0022] In a possible implementation, performing a first round of training on the initial model according to the real data set and the simulated data set, respectively updating the parameters of the backbone network, the detection head, and the domain classifier to obtain an intermediate training model, includes:

[0023] Inputting the real data sets into the initial model for training respectively, and obtaining the real data detection loss and the real data domain classification loss;

[0024] Inputting the simulation data set into the initial model for training, and obtaining the simulation data detection loss and the simulation data domain classification loss;

[0025] According to the real data detection loss, the real data domain classification loss, the simulation data detection loss and the simulation data domain classification loss, a gradient descent algorithm is adopted to update the parameters of the initial model to obtain the intermediate training model.

[0026] In a possible implementation, performing a second round of training on the intermediate training model according to the real data set, and updating the parameters of the detection head in the model again to obtain the model after the updated parameters, includes:

[0027] Inputting the real data sets into the intermediate training models respectively for a second round of training to obtain secondary real data detection losses;

[0028] According to the secondary real data detection loss, a gradient descent algorithm is adopted to update the parameters of the detection head in the intermediate training model again to obtain the model after the updated parameters.

[0029] In a second aspect, the present application provides a target detection method, comprising:

[0030] Acquire the image to be detected;

[0031] The image to be detected is input into a target detection model to perform target detection to obtain a detection result; wherein the target detection model is trained according to the method described in any one of the first aspects.

[0032] In a possible implementation manner, the image to be detected is a road image collected by a roadside device;

[0033] The detection results include: types and positions of different types of traffic participants.

[0034] In a third aspect, the present application provides a training device for a target detection model, comprising:

[0035] A first processing module is used to obtain a real data set and a simulation data set; the simulation data set is obtained by simulating a first probability of each target in the real data set, and a second probability of different types of targets in the simulation data set is greater than a preset value;

[0036] The second processing module is used to train a preset initial model according to the real data set and the simulation data set to obtain a target detection model.

[0037] In a fourth aspect, the present application provides a target detection device, comprising:

[0038] An acquisition module, used for acquiring an image to be detected;

[0039] A processing module is used to input the image to be detected into a target detection model to perform target detection and obtain a detection result; wherein the target detection model is trained according to the method described in any one of the first aspects.

[0040] In a fifth aspect, the present application provides a computer device, including: a processor, a memory, and a communication interface;

[0041] The memory stores computer-executable instructions;

[0042] The processor executes the computer-executable instructions stored in the memory to implement the target detection model training method as described in any one of the first aspects, or the target detection method as described in any one of the second aspects.

[0043] In a sixth aspect, the present application provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the training method of the target detection model as described in any one of the first aspect, or the target detection method as described in any one of the second aspect.

[0044] In the seventh aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the training method of the target detection model as described in any one of the first aspect, or the target detection method as described in any one of the second aspect.

[0045] The present application provides a training method, a target detection method, an apparatus and a device for a target detection model. The computer device can obtain a real data set and a simulation data set, and train a preset initial model according to the real data set and the simulation data set to obtain a target detection model. Among them, the simulation data set is obtained by simulating the first probability of each target in the real data set, and the second probability of different types of targets in the simulation data set is greater than a preset value. The simulation data set obtained by this simulation method can effectively supplement the uncommon targets in the real data set. By combining the simulation data set and the real data set to train the initial model, the detection accuracy of the model for uncommon targets can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application;

[0048] Figure 2 A flowchart of a first embodiment of a method for training a target detection model provided in the present application;

[0049] Figure 3 A flowchart of the second embodiment of the training method of the target detection model provided in this application;

[0050] Figure 4 A schematic diagram of a target location provided in an embodiment of the present application;

[0051] Figure 5 A flowchart of the third embodiment of the training method of the target detection model provided in this application;

[0052] Figure 6 A flowchart of an example of a training method for a target detection model provided in this application;

[0053] Figure 7 A schematic diagram of a target detection method provided in an embodiment of the present application;

[0054] Figure 8 A schematic diagram of the structure of a training device for a target detection model provided in an embodiment of the present application;

[0055] Fig. 9 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application;

[0056] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application.

[0057] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0058] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0059] First, let’s explain the terms involved in this application:

[0060] Simulator: A tool for creating and running a virtual environment that can be placed in a computer device, combining traffic participants with a static background, generating high-quality simulation video data, and simulating real-world traffic scenarios.

[0061] Domain classifier: A neural network component that determines which domain the input data belongs to. In transfer learning and adversarial training, the goal of the domain classifier is to identify whether the input data is from the simulated domain or the real domain.

[0062] Hyperparameters: Parameters that need to be set manually, which are usually selected by users based on experience.

[0063] In the construction of intelligent cities, the application of vehicle-road cooperative technology is becoming more and more extensive. By deploying multi-angle roadside equipment such as roadside cameras, road traffic conditions can be captured comprehensively and in real time. By designing a reasonable detection model, it is possible to accurately detect and classify various traffic participants, thereby improving the monitoring efficiency of traffic flow, optimizing traffic signal control, reducing the probability of traffic accidents, and promoting the development of urban transportation systems in a more intelligent, safe and efficient direction.

[0064] In the related art, the road data collected by the roadside equipment is usually used as training samples to train the detection model. However, the collected data usually presents the characteristics of long-tail distribution, and the number of samples of common targets, i.e., head data (such as cars), is much larger than that of uncommon targets, i.e., tail data (such as tricycles). This problem of unbalanced sample distribution will lead to low accuracy of the detection model when detecting uncommon targets.

[0065] In response to the above problems, the inventors consider that the number of samples can be expanded to construct a new data set, so as to train the detection model and improve the detection accuracy of uncommon targets. Specifically, a real data set and a simulated data set can be obtained, wherein the simulated data set is obtained based on the first probability simulation of each target in the real data set. In addition, the second probability of different types of targets in the simulated data set must be greater than a preset value to ensure that the detection model can obtain more training samples about uncommon targets, and then the preset initial model can be trained according to the real data set and the simulated data set to obtain a target detection model, thereby significantly improving the detection accuracy of the target detection model for uncommon targets. Based on this, the present application provides a target detection model training method and a target detection method, which significantly improves the recognition accuracy of uncommon targets in the image to be detected by utilizing an optimized trained detection model.

[0066] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 , the image to be detected can be input into the target detection model, and the image to be processed can be processed by the target detection model to obtain the detection result.

[0067] For example, a roadside camera may capture an image 1 to be detected, which includes a tricycle. After processing the image 1 to be detected by the target detection model, a detection result may be obtained. The detection result may include the type of the target as a tricycle, and the position of the tricycle, wherein the position of the tricycle may be represented by the bounding box coordinates of the tricycle.

[0068] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0069] The technical solution of this application mainly involves two processes, namely the model training process and the model use process. Figure 2 ,First, the process of model training is explained.

[0070] Figure 2 A flowchart of the first embodiment of the training method of the target detection model provided in this application.

[0071] See also Figure 2 , the method may include:

[0072] S201. Obtain a real data set and a simulation data set.

[0073] The execution subject of the embodiment of the present application can be a computer device, or a training device for a target detection model set in a computer device. The training device for the target detection model can be implemented by software, or by a combination of software and hardware. The training device for the target detection model can be a processor in a computer device. For ease of understanding, the following description is taken as an example of the execution subject being a computer device.

[0074] In this step, a real data set and a simulated data set can be obtained; the simulated data set is obtained by simulating the first probability of each target in the real data set, and the second probability of different types of targets in the simulated data set is greater than a preset value.

[0075] For the real data set, the type and position of the target in each frame of the real video data collected in the real scene can be calibrated to obtain the real data set. Specifically, each frame of the real video data can be manually annotated to obtain an image label, and the image label includes the type and position of the target. The real data set can be obtained through each frame of the image and the image label of each frame of the image.

[0076] The type of the target may be a traffic participant on the road, wherein the type and number of the target may be preset according to actual needs. For example, the type of target 1 (c1) may be a car, the type of target 2 (c2) may be a tricycle, the type of target 3 (c3) may be a pedestrian, and the number of target types K may be 3.

[0077] The position of the target may be represented by the upper left corner coordinates and the lower right corner coordinates in the bounding box coordinates where the target is located, or by the upper left corner coordinates, the width of the bounding box, and the height of the bounding box in the bounding box coordinates where the target is located.

[0078] For example, the real data set may include M training samples, each training sample includes an image and an image label corresponding to the image, and the image label may include the type and position of at least one target.

[0079] In real scenarios, the target probability distribution usually presents a long-tail distribution. The probability of common targets (such as cars) is higher, while the probability of uncommon targets (such as tricycles) is larger. After obtaining the real data set, the second probability of the target can be determined for each type of target based on the first probability of the target in the real data set. The second probability is used to indicate the proportion of the target in the total number of targets in the simulation data.

[0080] Specifically, the real data set can contain c k(k=1, 2, ..., K) types of targets, the target probability distribution in the real data set can be determined according to the number of targets of each type, and the first probability of each type of target in the real data set can be obtained. R The calculation formula is:

[0081]

[0082] In formula (1), k = 1, 2, ..., K, n k Represents c k The number of targets of the class in the real data set.

[0083] After obtaining the first probability of each type of target in the real data set, the second probability of the target can be determined for each type of target according to the first probability of the target in the real data set. The second probability is used to indicate the proportion of the target in the total number of targets in the simulation data. S It can be calculated according to the first probability of the target, the number of target types K in the real data set, and the preset hyperparameter α.

[0084] In an optional implementation, the preset value can be determined by the ratio of the value of the hyperparameter α to the number of target types K in the real data set. For example, if the value of the hyperparameter α is 0.6, the preset value can be calculated to be 0.15 based on the number of target types K in the real data set being 4.

[0085] Optionally, after obtaining the second probability P S After that, a simulation data set can be generated through a simulation simulator.

[0086] For example, the total number of targets in the simulation data set can be set according to the second probability P of each type of target. S (c k ), generate N training samples, the sum of all targets in the N training samples is the total number of targets set in advance. Each training sample includes an image and an image label corresponding to the image, and the image label may include the type and position of at least one target.

[0087] It should be noted that when a simulation data set is generated through a simulation simulator, a static background and traffic participants can be combined to generate simulated video data, and each frame of the image can be extracted from the simulated video data to automatically generate an image label for each image (including the type and location of the target) without manual labeling.

[0088] S202: Train a preset initial model according to the real data set and the simulation data set to obtain a target detection model.

[0089] In this step, the preset initial model can be trained according to the training samples in the real data set and the simulation data set to obtain the target detection model.

[0090] Optionally, the initial model may include a backbone network, a detection head, and a domain classifier. The initial model can be trained based on a real data set and a simulated data set, and the parameters of the backbone network, the detection head, and the domain classifier can be updated to obtain a model with updated parameters. The domain classifier in the model with updated parameters can be removed to obtain a target detection model.

[0091] In the specific implementation, the training samples in the real data set and the simulated data set can be input into the initial model for training. The parameters of the backbone network, detection head, and domain classifier are continuously adjusted through optimization algorithms (such as gradient descent algorithm and Newton's method) to enable it to recognize different types of targets and accurately locate the position of the target in the image.

[0092] In an embodiment of the present application, a computer device can obtain a real data set and a simulated data set, and train a preset initial model according to the real data set and the simulated data set to obtain a target detection model. The simulated data set is obtained by simulating the first probability of each target in the real data set, and the second probability of different types of targets in the simulated data set is greater than a preset value. In the above process, the simulated data set obtained by simulation can effectively make up for the lack of uncommon targets in the real data set, thereby reducing the imbalance problem of the distribution of training samples and improving the detection accuracy of the model for these uncommon targets.

[0093] exist Figure 2 Based on the embodiment shown below, combined with Figure 3 , the training method of the above target detection model is further explained in detail.

[0094] Figure 3 This is a flow chart of the second embodiment of the training method of the target detection model provided in this application. Figure 3 , the initial model includes a backbone network, a detection head, and a domain classifier, and the method may include:

[0095] S301 , calibrating the type and position of the target in each frame of the real video data collected in the real scene to obtain a real data set.

[0096] In this step, the type and position of the target in each frame of the real video data collected by the roadside equipment can be calibrated to obtain the image label of each frame, and a real data set is obtained based on each frame and the image label, where the image label includes the type and position of the target.

[0097] Figure 4A schematic diagram of a target location provided in an embodiment of the present application. Figure 4 After annotating each frame of the real video data, we can get that the target type is a pedestrian, the coordinates of the upper left corner of the bounding box where the pedestrian is located are A(x1, y1), and the coordinates of the lower right corner of the bounding box are B(x2, y2).

[0098] Optionally, after manually annotating an image, the acquired image label may include at least one target type and a target location.

[0099] For example, after annotating image 1, image label 1 can be obtained, which includes the type of an object and the location of the object, wherein the object can be a car, the upper left corner coordinate of the car's bounding box is A1, and the lower right corner coordinate of the bounding box is B1.

[0100] For another example, after annotating image 2, image label 2 can be obtained, which includes the types of two targets and the positions of the two targets, wherein target 1 can be a car, the upper left corner coordinate of the car's bounding box is A2, and the lower right corner coordinate of the bounding box is B2; target 2 can be a tricycle, the upper left corner coordinate of the tricycle's bounding box is A3, and the lower right corner coordinate of the bounding box is B3.

[0101] Exemplarily, after calibrating the type and position of the target in each frame of the real video data collected in the real scene, the obtained real data set may be:

[0102]

[0103] Among them, D R represents a real data set, represents the i-th image after framing, represents the image label of the i-th image; i represents the image sequence number, and M represents the number of training samples.

[0104] S302: Obtain a first probability of each type of target in the total number of targets corresponding to the real data set.

[0105] In this step, after obtaining the real data set, the first probability P R The calculation formula is used to obtain the first probability of each type of target in the total number of targets corresponding to the real data set.

[0106] For example, there are 4 types of targets, namely cars, tricycles, pedestrians, and motorcycles. The number of training samples M is 100, and the total number of targets is 200. Among them, there are 100 cars, 2 tricycles, 80 pedestrians, and 18 motorcycles. Then, according to the first probability PR Calculate the formula to get the first probability P of the four types of targets in the total number of targets corresponding to the real data set R (c k ). The first probability P of the target is calculated R (c k ) can be shown in Table 1:

[0107] Table 1

[0108]

[0109] S303. For each type of target, determine a second probability of the target according to a first probability of the target in the real data set, where the second probability is used to indicate a proportion of the target in the total number of targets in the simulation data.

[0110] In this step, for each type of target, the second probability of the target can be calculated based on the first probability of the target in the real data set, the number of types of targets in the real data set, and preset hyperparameters.

[0111] Optionally, the second probability P S The calculation formula is:

[0112]

[0113] In formula (2), k = 1, 2, ..., K, and α is a hyperparameter.

[0114] In an optional implementation, the value of α can be preset. Specifically, α can be selected from a value range of 0.6 to 0.9.

[0115] For example, if the number of target types K in the real data set is 4 and the preset hyperparameter α is 0.6, then the second probability P of the target is calculated. S (c k ) can be shown in Table 2:

[0116] Table 2

[0117] Target Type Target <![CDATA[First probability P R > <![CDATA[Second probability P S > <![CDATA[c1]]> Sedan <![CDATA[P R (c1)=0.5]]> <![CDATA[P S (c1)=0.35]]> <![CDATA[c2]]> Three-wheeler <![CDATA[P R (c2)=0.01]]> <![CDATA[P S (c2)=0.154]]> <![CDATA[c3]]> pedestrian <![CDATA[P R (c3)=0.4]]> <![CDATA[P s (c3)=0.31]]> <![CDATA[c4]]> motorcycle <![CDATA[P R (c4)=0.09]]> <![CDATA[P s (c4)=0.186]]>

[0118] S304: Perform data simulation using a simulation simulator according to the second probability of each type of target to obtain a simulation data set.

[0119] In this step, according to the second probability of each type of target, as well as the preset number of training samples and the total number of targets in the simulation data set, a simulation simulator can be used to perform data simulation to obtain simulated video data and then obtain a simulation data set.

[0120] Optionally, the simulated video data may include two parts: a static roadside environment background and dynamic traffic participants. The simulation simulator may add dynamic traffic participants to the static background, and generate simulated video data under a specific perspective according to the camera placement angle and internal parameters, thereby obtaining a simulated data set.

[0121] For example, if the number of training samples is N, and the total number of targets in the simulation data set is T, then the second probability P of each type of target can be S (c k ), use a simulation simulator to simulate the data, obtain simulated video data, and then obtain a simulation data set D S .

[0122] Optionally, the simulation data set obtained by simulating video data may be:

[0123]

[0124] Among them, D S represents the simulation data set, represents the i-th image after framing, represents the image label of the i-th image; i represents the image sequence number, and N represents the number of training samples.

[0125] It should be noted that, since the simulation video data is automatically generated by a simulation simulator, the image label of each frame of the image is also automatically generated without manual labeling.

[0126] S305 . Train the initial model according to the real data set and the simulation data set, and update the parameters of the backbone network, the detection head, and the domain classifier to obtain a model with updated parameters.

[0127] In this step, the initial model can be trained based on the real data set and the simulated data set to obtain the loss of each training batch. The parameters of the backbone network, the detection head, and the domain classifier are updated based on the loss of each training batch to obtain the model with updated parameters. The calculation formula of the loss L of each training batch during the training process is:

[0128] L=λL R +ηL S +γL C Formula (3)

[0129] In formula (3), λ, η, and γ are preset weight coefficients, and L R represents the real data detection loss, L S represents the detection loss of simulation data, L C represents the classification loss in the data domain.

[0130] Detection loss L for real data R , the loss is the deviation between the real data detection prediction result and the real data image label, and the calculation formula is:

[0131]

[0132] In formula (4), To test the prediction results for real data, is the image label of the real data, L DET (,) is the detection loss function corresponding to the main target detection network.

[0133] Detection loss L for simulation data S , the loss is the deviation between the simulated data detection prediction result and the image label of the simulated data, and the calculation formula is:

[0134]

[0135] In formula (5), To test the prediction results for the simulation data, is the image label of the simulated data, L DET (,) is the detection loss function corresponding to the main target detection network.

[0136] Specifically, the real data detection prediction results and the simulation data detection prediction results may include the coordinates of the bounding box where the target is located, the type of the target category, and the confidence score for each detection result.

[0137] Optionally, the target detection task mainly includes a positioning task and a classification task. Therefore, the detection loss function L in this application is DET (,) can include two parts: positioning loss and classification loss. For example, the positioning loss can be smooth L1 loss or intersection over union loss (IOU), and the classification loss can be cross-entropy loss.

[0138] For the data domain classification loss L C , the calculation formula is:

[0139]

[0140] In formula (6), is the predicted probability that the image belongs to the real domain, is the predicted probability that the image belongs to the simulation domain.

[0141] In an optional implementation, the network to be trained in the initial model may include two parts, namely, a main target detection network and an adversarial data domain classification structure. The main target detection network is a target detection framework based on a convolutional neural network and can be decomposed into two parts in series: a backbone network f for target feature extraction and a backbone network f for target feature extraction. b (·) and the detection head f for outputting the detection results h (·); the adversarial data domain classification structure includes a domain classifier g(·) composed of convolutional layers. The parameters of the backbone network can be obtained by θ b It means that the parameters of the detection head can be obtained by θ h Represents; the parameters of the domain classifier can be represented by φ.

[0142] For the domain classifier, this structure can determine whether the target belongs to the real data set (real domain) or the simulated data set (simulated domain) based on the input image features.

[0143] Exemplarily, the i-th real image can be selected Input to backbone network f b (·) to obtain the real image features This feature At the same time, input to the detection head f h (·) and domain classifier g(·), respectively, to obtain the real data detection prediction results and the predicted probability that the image belongs to the true domain Test the prediction results based on the real data of M training samples and image tags The real data detection loss L can be obtained R .

[0144] Exemplary: Select the jth simulation image Input to backbone network f b (·) to obtain the simulated image features This feature At the same time, input to the detection head f h (·) and domain classifier g(·), respectively obtain the simulation data detection prediction results and the predicted probability that the image belongs to the simulation domain Test the prediction results based on the simulated data of N training samples and image tags The simulation data detection loss L can be obtained S .

[0145] For example, the predicted probability of each of the M training samples belonging to the real domain can be And the predicted probability that each of the N training samples belongs to the simulation domain Get the data domain classification loss L C .

[0146] Furthermore, the parameters θ of the backbone network, the detection head, and the domain classifier can be adjusted by using the gradient descent algorithm according to the minimization loss L. b ,θ h , and φ are updated, and the updating process is:

[0147]

[0148]

[0149]

[0150] Among them, γ1 is the learning rate and β is the hyperparameter of the balanced loss.

[0151] In an optional implementation, the values ​​of γ1 and β can be preset. Specifically, γ1 can be selected in the value range of 0.0001 to 0.1, and β can be selected in the value range of 0.001 to 0.1. For example, the value of γ1 can be 0.1, and the value of β can be 0.001.

[0152] Optionally, use the data domain classification loss L C Update the parameters θ of the backbone network b When , its gradient can be reversed to reduce the difference between the simulation domain and the real domain. That is, the introduction of gradient reversal makes the domain classifier and the backbone network confront each other.

[0153] S306. Remove the domain classifier in the model after the updated parameters to obtain a target detection model.

[0154] In this step, the domain classifier in the model with updated parameters can be removed to obtain the object detection network.

[0155] It should be noted that the adversarial data domain classification structure, as an independent and flexible structure, can be additionally inserted between the backbone network and the detection head of the main target detection network. This structure does not affect the function of the main detection network and only plays a role in updating the model parameters during the training phase. It can be removed in the actual target detection task.

[0156] In an embodiment of the present application, a computer device can obtain a real data set, and for each type of target, determine a second probability of the target based on the first probability of the target in the real data set, and use a simulation simulator to perform data simulation based on the second probability to obtain a simulated data set. Then, the initial model can be trained based on the real data set and the simulated data set, and the parameters of the backbone network, the detection head, and the domain classifier are updated to obtain a model with updated parameters, and the domain classifier in the model with updated parameters is removed to obtain a target detection model. In the above process, the simulated data set generated by the simulation simulator can effectively expand the real data set, increase sample diversity, avoid sample bias in the real data set, and improve the detection accuracy of the model for uncommon targets.

[0157] Figure 5 This is a flow chart of the third embodiment of the training method of the target detection model provided in this application. Figure 5 Based on the above embodiment, in the specific implementation of the training method of the target detection model, in step S305, the initial model is trained according to the real data set and the simulation data set, and the parameters of the backbone network, the detection head, and the domain classifier are updated. The specific implementation of obtaining the model after the updated parameters may include the following steps:

[0158] S501. Perform the first round of training on the initial model according to the real data set and the simulated data set, and update the parameters of the backbone network, the detection head, and the domain classifier respectively to obtain an intermediate training model.

[0159] In this step, the initial model can be trained according to the real data set and the simulated data set to obtain the real data detection loss, the real data domain classification loss, the simulated data detection loss, and the simulated data domain classification loss. The gradient descent algorithm is used to update the parameters of the backbone network, the detection head, and the domain classifier to obtain the intermediate training model.

[0160] The real data set can be input into the initial model for training to obtain the real data detection loss and the real data domain classification loss.

[0161] For example, the real data set D R The M training samples in are input into the initial model for training respectively to obtain the real data detection loss and the real data domain classification loss.

[0162] Among them, the real data detection loss can be obtained by formula (4), and the real data domain classification loss is

[0163] The simulation data sets can be input into the initial model for training to obtain the simulation data detection loss and the simulation data domain classification loss.

[0164] For example, the simulation data set D S The N training samples in are input into the initial model for training respectively to obtain the simulation data detection loss and the simulation data domain classification loss.

[0165] Among them, the simulation data detection loss can be obtained by formula (5), and the simulation data domain classification loss is

[0166] Then, according to the real data detection loss, the real data domain classification loss, the simulation data detection loss and the simulation data domain classification loss, the gradient descent algorithm can be used to update the parameters of the initial model to obtain the intermediate training model.

[0167] Specifically, the data domain classification loss L can be obtained by formula (6): C , combined with the obtained real data detection loss L R And the simulation data detection loss L S , using the gradient descent algorithm to adjust the parameters θ of the backbone network, detection head, and domain classifier b ,θ h , and φ are updated to obtain an intermediate training model. The parameter updating process can refer to step S305 and will not be described in detail here.

[0168] S502. Perform a second round of training on the intermediate training model based on the real data set, and update the parameters of the detection head in the model again to obtain a model with updated parameters.

[0169] In this step, the intermediate training model obtained in the first round of training can be retrained using the real data set, and the parameters of the detection head can be optimized again to obtain a model with updated parameters. The parameters of the detection head are the parameters of the detection head obtained in the first round of training.

[0170] The real data sets can be input into the intermediate training model for a second round of training to obtain the secondary real data detection loss.

[0171] Detection loss L for secondary real data R2 , the loss is the deviation between the real data re-detection prediction result and the real data image label, and the calculation formula is:

[0172]

[0173] In formula (7), Re-test the prediction results for real data. is the image label of the real data, L DET (,) is the detection loss function corresponding to the main target detection network.

[0174] Exemplarily, the i-th real image can be selected Input to backbone network f b (·) to obtain the real image features This feature Input to the sensor head h (·), get the real data and test the prediction results again Re-test the prediction results based on the real data of M training samples and image tags The secondary real data detection loss L can be obtained R2 .

[0175] Furthermore, according to the secondary real data detection loss, the gradient descent algorithm is used to update the parameters of the detection head in the intermediate training model again to obtain the model with updated parameters.

[0176] Optionally, the loss L can be minimized R2 , the parameters θ of the detection head are adjusted using the gradient descent algorithm h To update, the update process is:

[0177]

[0178] Among them, γ2 is the learning rate.

[0179] Similarly, the value of γ2 can be preset. Specifically, γ2 can be selected from the value range of 0.0001 to 0.1, for example, the value of γ2 can be 0.1.

[0180] Optionally, a second round of training is performed on the intermediate training model, and only the parameters of the detection head obtained in the first round of training can be optimized again, and the parameters of the backbone network and the domain classifier remain fixed, that is, consistent with the parameters of the backbone network and the domain classifier obtained in the first round of training. The detection head parameters obtained in the second round of training, the parameters of the backbone network and the domain classifier obtained in the first round of training are used as the final optimization parameters of the model. During the use of the model, the domain classifier in the model after the updated parameters is removed to obtain the target detection model.

[0181] In an embodiment of the present application, the initial model can be trained for the first round according to the real data set and the simulated data set, and the parameters of the backbone network, the detection head, and the domain classifier can be updated respectively to obtain an intermediate training model; the intermediate training model can be trained for the second round according to the real data set, and the parameters of the detection head in the model can be updated again to obtain a model with updated parameters. In the above process, by first using the real data set and the simulated data set to train the model for the first round, and then using the real data set to train the intermediate training model for the second time, the interference caused by the difference between the simulated data set and the real data set can be reduced, the adaptability of the model to real scenes can be enhanced, and the detection accuracy of uncommon targets can be improved.

[0182] Figure 6 This is a flowchart of an example of a training method for a target detection model provided in this application. Figure 6 , the process may include:

[0183] S601: Acquire real video data.

[0184] For example, road traffic conditions can be captured by roadside cameras to obtain real video data.

[0185] S602: Manually label the target box.

[0186] Optionally, the type and position of the target in each frame of the real video data may be calibrated manually.

[0187] S603: Acquire a real data set.

[0188] For example, the acquired real data set may include M training samples, each training sample includes an image and an image label corresponding to the image.

[0189] S604: Obtain a first probability.

[0190] In this step, the first probability of each type of target in the real data set in the total number of targets corresponding to the real data set can be obtained.

[0191] S605: Obtain a second probability.

[0192] In this step, the second probability of the target can be determined according to the first probability of the target in the real data set, and the second probability is used to indicate the proportion of the target in the total number of targets in the simulation data.

[0193] S606: Construct a simulation data set.

[0194] For example, a simulation data set generated by a simulation simulator may include N training samples, each of which includes an image and an image label corresponding to the image.

[0195] S607: Perform a first round of training on the initial model to obtain an intermediate training model.

[0196] In this step, the initial model can be trained for the first round based on the real data set and the simulated data set, and the parameters of the backbone network, the detection head, and the domain classifier can be updated respectively to obtain an intermediate training model.

[0197] S608: Perform a second round of training on the intermediate training model to obtain a target detection model.

[0198] In this step, the intermediate training model can be trained for the second round based on the real data set, the parameters of the detection head in the model can be updated again to obtain the model with updated parameters, and the domain classifier in the model with updated parameters can be removed to obtain the target detection model.

[0199] The training method example of the target detection model provided in the embodiment of the present application, during the model training process, can refer to the process shown in the above method embodiment, its implementation principle and beneficial effects are similar, and will not be repeated here.

[0200] The contents shown in the above embodiments are all the process of model training. Figure 7 , explaining the process of using the model.

[0201] Figure 7 A flowchart of a target detection method provided in an embodiment of the present application. Figure 7 , the method may include:

[0202] S701: Acquire an image to be detected.

[0203] In this step, the computer device can obtain the image to be detected, wherein the image to be detected is a road image collected by the roadside equipment.

[0204] For example, the computer device may obtain a real-time road image from a roadside camera and use the road image as the image to be detected, wherein the road image may be obtained by dividing the real video data obtained by the roadside camera into frames.

[0205] S702: Input the image to be detected into the target detection model to perform target detection and obtain a detection result.

[0206] In this step, the image to be detected may be input into a target detection model for target detection to obtain a detection result, which may include the type and position of different types of traffic participants.

[0207] The target detection model may be a model obtained by training the domain classifier in the above embodiment and removing the update parameters.

[0208] For example, the image to be detected can be input into the target detection model for target detection, and the obtained detection results include: the target is a tricycle, the upper left corner coordinates of the bounding box where the tricycle is located are A(x1, y1), and the lower right corner coordinates of the bounding box are B(x2, y2).

[0209] In an optional embodiment, the detection results may include, in addition to the type and location of different types of traffic parameters, a confidence score for the detection results, wherein the closer the confidence score is to 1, the stronger the confidence of the model in the detection results; conversely, a score close to 0 indicates that the confidence of the model in the detection results is weaker.

[0210] For example, the image to be detected can be input into the target detection model for target detection, and the detection results obtained include: the target is a tricycle, the coordinates of the upper left corner of the bounding box where the tricycle is located are A(x1, y1), and the coordinates of the lower right corner of the bounding box are B(x2, y2), and the confidence score of the detection result is 0.9. Among them, the confidence score of 0.9 means that the model believes that there is a 90% probability that the target is the identified type (tricycle) within the given bounding box.

[0211] In the embodiment of the present application, the computer device can obtain the image to be detected, and input the image to be detected into the target detection model for target detection to obtain the detection result. Since the target detection model is obtained by optimizing and training the initial model using the real data set and the simulation data set, the detection accuracy of uncommon targets can be improved by optimizing the trained target detection model.

[0212] Figure 8 This is a schematic diagram of the structure of the training device of the target detection model provided in the embodiment of the present application. Figure 8 , the training device 10 of the target detection model may include:

[0213] A first processing module 11 is used to obtain a real data set and a simulation data set; the simulation data set is obtained by simulating a first probability of each target in the real data set, and a second probability of different types of targets in the simulation data set is greater than a preset value;

[0214] The second processing module 12 is used to train a preset initial model according to the real data set and the simulation data set to obtain a target detection model.

[0215] The training device for the target detection model provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar and will not be repeated here.

[0216] In a possible implementation, the initial model includes a backbone network, a detection head, and a domain classifier;

[0217] The second processing module 12 is specifically used for:

[0218] According to the real data set and the simulation data set, the initial model is trained, and the parameters of the backbone network, the detection head, and the domain classifier are updated to obtain a model with updated parameters;

[0219] The domain classifier in the model after the parameter update is removed to obtain the object detection model.

[0220] In a possible implementation manner, the second processing module 12 is specifically configured to:

[0221] According to the real data set and the simulation data set, the initial model is trained for the first round, and the parameters of the backbone network, the detection head, and the domain classifier are updated respectively to obtain an intermediate training model;

[0222] According to the real data set, a second round of training is performed on the intermediate training model, and the parameters of the detection head in the model are updated again to obtain the model with updated parameters.

[0223] In a possible implementation manner, the first processing module 11 is specifically configured to:

[0224] Calibrate the type and position of the target in each frame of the real video data collected in the real scene to obtain the real data set;

[0225] Obtaining a first probability of each type of target in the real data set in the total number of targets corresponding to the real data set;

[0226] For each type of target, determine a second probability of the target according to a first probability of the target in the real data set, where the second probability is used to indicate the proportion of the target in the total number of targets in the simulation data;

[0227] According to the second probability of each type of target, a simulation simulator is used to perform data simulation to obtain the simulation data set.

[0228] In a possible implementation manner, the first processing module 11 is specifically configured to:

[0229] A second probability of the target is calculated based on the first probability of the target, the number of types of targets in the real data set, and a preset hyperparameter.

[0230] In a possible implementation manner, the second processing module 12 is specifically configured to:

[0231] Inputting the real data sets into the initial model for training respectively, and obtaining the real data detection loss and the real data domain classification loss;

[0232] Inputting the simulation data set into the initial model for training, and obtaining the simulation data detection loss and the simulation data domain classification loss;

[0233] According to the real data detection loss, the real data domain classification loss, the simulation data detection loss and the simulation data domain classification loss, a gradient descent algorithm is adopted to update the parameters of the initial model to obtain the intermediate training model.

[0234] In a possible implementation manner, the second processing module 12 is specifically configured to:

[0235] Inputting the real data sets into the intermediate training models respectively for a second round of training to obtain secondary real data detection losses;

[0236] According to the secondary real data detection loss, a gradient descent algorithm is adopted to update the parameters of the detection head in the intermediate training model again to obtain the model after the updated parameters.

[0237] The training device for the target detection model provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar and will not be repeated here.

[0238] Fig. 9 This is a schematic diagram of the structure of the target detection device provided in the embodiment of the present application. Fig. 9 , the target detection device 20 may include:

[0239] An acquisition module 21 is used to acquire an image to be detected;

[0240] The processing module 22 is used to input the image to be detected into the target detection model to perform target detection and obtain a detection result; wherein the target detection model is trained according to the training method of the above-mentioned target detection model.

[0241] In a possible implementation manner, the image to be detected is a road image collected by a roadside device;

[0242] The detection results include: types and positions of different types of traffic participants.

[0243] The target detection device provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0244] Fig.10 This is a schematic diagram of the structure of the computer device provided in the embodiment of the present application. Fig.10 The computer device 30 may include a processor 31, a memory 32 and a communication interface 34. Exemplarily, the processor 31, the memory 32 and the communication interface 34 are interconnected via a bus 33.

[0245] The memory 32 stores computer-executable instructions;

[0246] The processor 31 executes the computer-executable instructions stored in the memory 32, so that the processor 31 executes the target detection model training method or the target detection method as described in the above method embodiment.

[0247] Accordingly, an embodiment of the present application provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the target detection model training method or target detection method described in the above method embodiment.

[0248] Accordingly, an embodiment of the present application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the target detection model training method or target detection method described in the above method embodiment.

[0249] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0250] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for training a target detection model, characterized in that: include: Acquire a real data set and a simulation data set; the simulation data set is obtained by simulating a first probability of each target in the real data set, and a second probability of different types of targets in the simulation data set is greater than a preset value; The preset initial model is trained according to the real data set and the simulation data set to obtain a target detection model.

2. The method according to claim 1, characterized in that: The initial model includes a backbone network, a detection head, and a domain classifier; The step of training a preset initial model according to the real data set and the simulation data set to obtain a target detection model includes: According to the real data set and the simulation data set, the initial model is trained, and the parameters of the backbone network, the detection head, and the domain classifier are updated to obtain a model with updated parameters; The domain classifier in the model after the parameter update is removed to obtain the object detection model.

3. The method according to claim 2, characterized in that The initial model is trained according to the real data set and the simulation data set, and the parameters of the backbone network, the detection head, and the domain classifier are updated to obtain a model with updated parameters, including: According to the real data set and the simulation data set, the initial model is trained for the first round, and the parameters of the backbone network, the detection head, and the domain classifier are updated respectively to obtain an intermediate training model; According to the real data set, a second round of training is performed on the intermediate training model, and the parameters of the detection head in the model are updated again to obtain the model with updated parameters.

4. The method according to any one of claims 1 to 3, characterized in that: The obtaining of the real data set and the simulation data set comprises: Calibrate the type and position of the target in each frame of the real video data collected in the real scene to obtain the real data set; Obtaining a first probability of each type of target in the real data set in the total number of targets corresponding to the real data set; For each type of target, determine a second probability of the target according to a first probability of the target in the real data set, where the second probability is used to indicate the proportion of the target in the total number of targets in the simulation data; According to the second probability of each type of target, a simulation simulator is used to perform data simulation to obtain the simulation data set.

5. The method according to claim 4, characterized in that The determining, according to the first probability of the target in the real data set, a second probability of the target, comprises: A second probability of the target is calculated based on the first probability of the target, the number of types of targets in the real data set, and a preset hyperparameter.

6. The method according to claim 3, characterized in that The first round of training of the initial model is performed according to the real data set and the simulation data set, and the parameters of the backbone network, the detection head, and the domain classifier are updated respectively to obtain an intermediate training model, including: Inputting the real data sets into the initial model for training respectively, and obtaining the real data detection loss and the real data domain classification loss; Inputting the simulation data set into the initial model for training, and obtaining the simulation data detection loss and the simulation data domain classification loss; According to the real data detection loss, the real data domain classification loss, the simulation data detection loss and the simulation data domain classification loss, a gradient descent algorithm is adopted to update the parameters of the initial model to obtain the intermediate training model.

7. The method according to claim 3 or 6, characterized in that: The step of performing a second round of training on the intermediate training model according to the real data set and updating the parameters of the detection head in the model again to obtain the model after the updated parameters includes: Inputting the real data sets into the intermediate training models respectively for a second round of training to obtain secondary real data detection losses; According to the secondary real data detection loss, a gradient descent algorithm is adopted to update the parameters of the detection head in the intermediate training model again to obtain the model after the updated parameters.

8. A target detection method, characterized in that: include: Acquire the image to be detected; The image to be detected is input into a target detection model to perform target detection to obtain a detection result; wherein the target detection model is trained according to the method according to any one of claims 1 to 7.

9. The method according to claim 8, characterized in that The image to be detected is a road image collected by a roadside device; The detection results include: types and positions of different types of traffic participants.

10. A training device for a target detection model, characterized in that: include: A first processing module, used for acquiring a real data set and a simulation data set; The simulation data set is obtained by simulating a first probability of each target in the real data set, and a second probability of different types of targets in the simulation data set is greater than a preset value; The second processing module is used to train a preset initial model according to the real data set and the simulation data set to obtain a target detection model.

11. A target detection device, characterized in that: include: An acquisition module, used for acquiring an image to be detected; A processing module is used to input the image to be detected into a target detection model for target detection to obtain a detection result; wherein the target detection model is trained according to the method according to any one of claims 1 to 7.

12. A computer device, characterized in that: include: processor, memory and communication interface; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the target detection model training method described in any one of claims 1 to 7, or the target detection method described in claim 8 or 9.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the training method for a target detection model as described in any one of claims 1 to 7, or the target detection method as described in claim 8 or 9.

Images