Target detection method, and training method and device of target detection model

Abstract

The present disclosure provides a target detection method and a training method and device of a target detection model, comprising: inputting a to-be-detected image into a backbone network included in the target detection model, and confirming that an output of the backbone network is a feature corresponding to the to-be-detected image; inputting the feature corresponding to the to-be-detected image into a prediction network included in the target detection model, and confirming that an output of the prediction network is target information included in the to-be-detected image; wherein the target information included in the to-be-detected image comprises at least one of position information, size information, orientation information, depth information and occlusion information of each target in the to-be-detected image; the target detection model is trained based on a sample image comprising a target sub-region and a non-target sub-region, so that rich target information can be detected, richer scene information can be provided for subsequent navigation, and more appropriate decisions can be made by assisting navigation.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular to a target detection method and a training method and device of a target detection model. BACKGROUND

[0002] Target detection is a technology for identifying targets (or objects) from image data; according to the spatial dimension of the output result, it is divided into 2D detection and 3D detection. To perform monocular 3D target detection, a large number of 2D images and 3D labels are needed to train the model, but due to the complexity of the actual scene and the high cost, it is difficult to collect a large number of 2D images and 3D labels. The 2D images and 3D labels spliced and synthesized later have irrationality and flaws, which will affect the result of model training. SUMMARY

[0003] The present disclosure provides a target detection method and a training method and device of a target detection model to at least solve the above technical problems in the prior art.

[0004] According to a first aspect of the present disclosure, a target detection method is provided, comprising:

[0005] inputting a to-be-detected image into a backbone network included in a target detection model, and confirming an output of the backbone network as a feature corresponding to the to-be-detected image;

[0006] inputting the feature corresponding to the to-be-detected image into a prediction network included in the target detection model, and confirming an output of the prediction network as target information included in the to-be-detected image;

[0007] wherein the target information included in the to-be-detected image includes at least one of position information, size information, orientation information, depth information and occlusion information of each target in the to-be-detected image; and the target detection model is obtained based on training of a sample image including a target sub-region and a non-target sub-region.

[0008] According to a second aspect of the present disclosure, a training method of a target detection model is provided, comprising:

[0009] pasting a target sub-region including a target and a non-target sub-region not including a target to a first image to generate a first sample image;

[0010] inputting the first sample image into a backbone network included in the target detection model, and confirming an output of the backbone network as a feature corresponding to the first sample image;

[0011] inputting the feature corresponding to the first sample image into a prediction network included in the target detection model, and confirming a first sub-loss;

[0012] update parameters of the backbone network and the prediction network based on the first sub-loss.

[0013] In the above scheme, the target sub-region including the target and the non-target sub-region not including the target are pasted into the first image to generate the first sample image, including:

[0014] Based on the depth information of the target sub-region and the non-target sub-region, the target sub-region and the non-target sub-region are pasted into the region corresponding to the depth information in the first image, and the pasted image is confirmed as the first sample image.

[0015] In the above scheme, the method further includes:

[0016] Obtain at least one sub-region corresponding to each original target in the first image;

[0017] Based on a predetermined probability, the at least one sub-region is retained or deleted;

[0018] The non-target sub-region with consistent depth information is pasted into the deleted sub-region in the first image.

[0019] In the above scheme, the target sub-region including the target and the non-target sub-region not including the target are pasted into the first image to generate the first sample image, including:

[0020] Based on the color distribution of each target in the first image and the color of the target corresponding to the target sub-region, the target sub-region including the target and the non-target sub-region not including the target are pasted into the first image to generate the first sample image.

[0021] In the above scheme, after updating the parameters of the backbone network and the prediction network based on the first sub-loss, the method further includes:

[0022] Input the feature corresponding to the second sample image into the feature identification network with updated parameters to determine a fourth sub-loss;

[0023] Update the parameters of the feature identification network based on the fourth sub-loss;

[0024] Repeat training the backbone network, the prediction network and the feature identification network.

[0025] In the above scheme, the first sub-loss includes a minimized loss function corresponding to the backbone network and the prediction network;

[0026] The second sub-loss is determined based on the output of the feature identification network under the condition of natural pixel input, and the output of the feature identification network when the output of the backbone network after pasting the pixel into the backbone network is input into the feature identification network.

[0027] The third sub-loss is determined based on a minimum loss function corresponding to the backbone network and the prediction network, a minimum loss function of the backbone network when the pixel is pasted into the backbone network, and an output of the feature identification network when the output of the backbone network after the pixel is pasted into the backbone network is taken as an input of the feature identification network.

[0028] In the above solution, the second sub-loss is determined based on an output of the feature identification network when a natural pixel is input, and an output of the feature identification network when the output of the backbone network after the pixel is pasted into the backbone network is taken as an input of the feature identification network, and the second sub-loss is determined by summing an expectation of a logarithmic value of the output of the feature identification network when the natural pixel is input and an expectation of a logarithmic value of the output of the feature identification network.

[0029] In the above solution, the second sub-loss is determined based on an output of the feature identification network when a natural pixel is input, and an output of the feature identification network when the output of the backbone network after the pixel is pasted into the backbone network is taken as an input of the feature identification network, and the second sub-loss is determined by summing an expectation of a logarithmic value of the output of the feature identification network when the natural pixel is input and an expectation of a logarithmic value of the output of the feature identification network.

[0030] In the above solution, the third sub-loss is determined based on a minimum loss function corresponding to the backbone network and the prediction network, a minimum loss function of the backbone network when the pixel is pasted into the backbone network, and an output of the feature identification network when the output of the backbone network after the pixel is pasted into the backbone network is taken as an input of the feature identification network.

[0031] In the above solution, the third sub-loss is determined based on a minimum loss function corresponding to the backbone network and the prediction network, a minimum loss function of the backbone network when the pixel is pasted into the backbone network, and an output of the feature identification network when the output of the backbone network after the pixel is pasted into the backbone network is taken as an input of the feature identification network.

[0032] According to a third aspect of the present disclosure, a target detection device is provided, comprising:

[0033] A first input unit is configured to input a to-be-detected image into a backbone network included in a target detection model, and determine an output of the backbone network as a feature corresponding to the to-be-detected image.

[0034] A detection unit is configured to input the feature corresponding to the to-be-detected image into a prediction network included in the target detection model, and determine an output of the prediction network as target information included in the to-be-detected image.

[0035] The target information included in the to-be-detected image includes at least one of position information, size information, orientation information, depth information, and occlusion information of each target in the to-be-detected image, and the target detection model is trained based on a sample image including a target sub-region and a non-target sub-region.

[0036] According to a fourth aspect of the present disclosure, a training device of a target detection model is provided, comprising:

[0037] a sample generation unit configured to paste a target sub-region including a target and a non-target sub-region not including the target into a first image to generate a first sample image;

[0038] a second input unit configured to input the first sample image into a backbone network included in the target detection model to obtain a feature corresponding to the first sample image as an output of the backbone network;

[0039] a third input unit configured to input the feature corresponding to the first sample image into a prediction network included in the target detection model to obtain a first sub-loss;

[0040] an updating unit configured to update parameters of the backbone network and the prediction network based on the first sub-loss.

[0041] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising:

[0042] at least one processor; and

[0043] a memory connected with the at least one processor; wherein,

[0044] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the present disclosure.

[0045] According to a sixth aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, and the computer instructions are used to make the computer perform the method of the present disclosure.

[0046] The target detection method and the training method of the target detection model provided by the present disclosure generate a first sample image by pasting a target sub-region including a target and a non-target sub-region not including a target into a first image in the model training stage; input the first sample image into a backbone network included in the target detection model, confirm that the output of the backbone network is the feature corresponding to the first sample image; input the feature corresponding to the first sample image into a prediction network included in the target detection model, confirm the first sub-loss; update the parameters of the backbone network and the prediction network based on the first sub-loss; in the model inference stage, input the image to be detected into the backbone network included in the target detection model, confirm that the output of the backbone network is the feature corresponding to the image to be detected; input the feature corresponding to the image to be detected into the prediction network included in the target detection model, and confirm that the output of the prediction network is the target information included in the image to be detected; in this way, the sample image including the target sub-region including the target and the non-target sub-region not including the target is trained, which to some extent offsets the influence of the artificially synthesized image on the training of the target detection model, and in addition, the target information of the image is fully utilized to improve the rationality of the sample image and further improve the accuracy of the target detection model trained based on the sample image.

[0047] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0048] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will be more apparent from the following detailed description read in conjunction with the accompanying drawings, in which:

[0049] In the drawings, identical or corresponding reference numerals indicate identical or corresponding parts.

[0050] Figure 1 An optional flowchart of a target detection method provided by an embodiment of the present disclosure is shown;

[0051] Figure 2 An optional flowchart of a training method of a target detection model provided by an embodiment of the present disclosure is shown;

[0052] Figure 3 Another optional flowchart of a training method of a target detection model provided by an embodiment of the present disclosure is shown;

[0053] Figure 4 A schematic diagram of generating a sample image provided by an embodiment of the present disclosure is shown;

[0054] Figure 5 A structural diagram of a target detection model provided by an embodiment of the present disclosure is shown.

[0055] Figure 6 A result diagram of a target detection method provided by an embodiment of the present disclosure is shown Figure 1 .

[0056] Figure 7 A result diagram of a target detection method provided by an embodiment of the present disclosure is shown Figure 2 .

[0057] Figure 8 A result diagram of a target detection method provided by an embodiment of the present disclosure is shown Figure 3 .

[0058] Figure 9 An optional structural diagram of a target detection device provided by an embodiment of the present disclosure is shown.

[0059] Figure 10 An optional structural diagram of a training device of a target detection model provided by an embodiment of the present disclosure is shown.

[0060] Figure 11 A structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0061] In order to make the objectives, characteristics and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present disclosure.

[0062] Target detection is a technology for recognizing targets (objects) from image data. According to the spatial dimension of the output result, it is divided into 2D detection and 3D detection. In the automatic driving scene, the output result of the 2D target detection technology lacks the 3D information of the target required for vehicle decision-making, which can cause the vehicle to be unable to make effective decisions or make wrong decisions, and has an adverse effect on the safety of the vehicle and the like. Monocular 3D target detection is to take 2D images captured by a single camera as input, detect and recognize the depth (distance), size (length, width and height), orientation, category (person or vehicle, etc.) and other 3D information of the objects or obstacles in the 2D images, so as to provide the vehicle with behavior decision-making. To perform monocular 3D target detection, a large number of 2D images and 3D labels are required to train the model, but due to the complexity of the actual scene and the high cost, it is difficult to collect a large number of 2D images and 3D labels. Moreover, due to safety and ethical issues, it is difficult to obtain scenes such as car accidents, so a large amount of monocular 3D data needs to be artificially produced for use by the model during training.

[0063] In view of the lack of monocular 3D data, the related art center will adopt the following several ways:

[0064] Scheme 1) Cutout, a rectangular area is removed from the image to be enhanced and filled with 0 to generate a new sample image;

[0065] Scheme 2) Box-MixUp, a rectangular box containing a target object is selected from an image and pasted into another image in a semi-transparent manner;

[0066] Scheme 3) Box-Cut-paste, a rectangular box containing a target object is selected from an image and pasted into another image in an opaque manner;

[0067] Scheme 4) Mosaic-Tile, multiple images are spliced into one image;

[0068] Scheme 5) Geometric consistent copy-paste, a target object is taken out from an image using instance segmentation, and a 3D object is projected onto a 2D image by randomly selecting a depth value.

[0069] However, in the above solutions, on the one hand, the semantic information of the background where the target is located is ignored, for example, the solution 5) may paste the vehicle in the air; on the other hand, during the pasting process, the edges of the rectangular frame will produce blurred jagged edges, which will induce the target detection model to learn the blurred jagged edges, so that the trained model cannot be generalized to the verification set including natural images; the cut-out target may not be completely integrated into the new background, and the target pasted into the new background is not a natural picture after all, and the model may learn a feature different from the natural target during the training process, thereby affecting the generalization ability of the model on the verification set; in addition, the cut-out target may be pasted to a background area with a different depth from the target, resulting in pasting a far target to a near background area or pasting a near target to a far background area, causing the finally generated image to be unreasonable.

[0070] Based on the defects in the related art, the present disclosure provides a training method of a target detection model, which performs semantic segmentation on an image to identify roads, vegetation, buildings and the like, to prevent the target from being pasted to the vegetation and the building; copies a background area (non-target sub-area) not including the target to paste into a picture to be enhanced, to suppress the model from learning the pasted edges; and introduces adversarial training to make the model generate the same feature distribution for the pasted pixels and the natural pixels.

[0071] Figure 1 An optional flowchart of a target detection method provided by an embodiment of the present disclosure is shown, which will be described according to each step.

[0072] In step S101, a to-be-detected image is input into a backbone network included in a target detection model, and the output of the backbone network is confirmed as a feature corresponding to the to-be-detected image.

[0073] In some embodiments, the target detection model is trained based on a sample image including a target sub-area and a non-target sub-area, and the specific training method will be described later.

[0074] The embodiment of the present disclosure is applied to a monocular camera scene, that is, after a to-be-detected image is acquired by a monocular camera on a vehicle (or a device with a moving ability, such as a forklift in a logistics sorting scene), target information included in the to-be-detected image is confirmed based on a target detection model, and the target information is used to prompt or instruct a subsequent driving operation. For example, based on the to-be-detected image acquired by the monocular camera, it is confirmed that the travel route of the target in the to-be-detected image partially coincides with or intersects with the current travel route of the vehicle, and then the vehicle can be prompted to change the travel route in advance for avoidance.

[0075] In step S102, the feature corresponding to the image to be detected is input into a prediction network included in the target detection model, and it is determined that the output of the prediction network is target information included in the image to be detected.

[0076] In some embodiments, the target information included in the image to be detected includes at least one of position information, size information, orientation information, depth information, and occlusion information of each target in the image to be detected. This can help the automatic navigation device of the vehicle to make route decisions (such as whether to change the route, etc.).

[0077] In this way, by using the target detection method provided in the embodiments of the present disclosure, the target detection model trained based on the sample image including the target sub-region and the non-target sub-region can accurately obtain the target information of the target in the image to be detected, thereby providing strong support for subsequent navigation.

[0078] Figure 2 An optional flowchart of a training method of a target detection model provided in the embodiments of the present disclosure is shown, which will be described in detail according to each step.

[0079] In step S201, a target sub-region including a target and a non-target sub-region not including a target are pasted into a first image to generate a first sample image.

[0080] In some embodiments, a training device of a target detection model (hereinafter referred to as a device) generates a sample set including a first sample image; the device can paste the target sub-region and the non-target sub-region into a region corresponding to the depth information of the target sub-region and the non-target sub-region in the first image based on the depth information of the target sub-region and the non-target sub-region, and determine that the pasted image is the first sample image; further, the device can also delete the region of the first image to which the target sub-region and the non-target sub-region are pasted.

[0081] In some embodiments, the device can obtain at least one sub-region corresponding to each original target in the first image; based on a predetermined probability, the at least one sub-region is retained or deleted, and a non-target sub-region with consistent depth information is pasted into the deleted sub-region in the first image. Wherein, the original target refers to a target included in the first image without any processing, and the sub-region corresponding to the original target can be a region surrounded by the contour of the original target, a region surrounded by the contour of a regular figure with the smallest area around the original target (such as a region surrounded by the edge of a rectangle with the smallest area around the original target), or a region surrounded by the contour of a figure (which can be a regular figure or an irregular figure) around the original target; the predetermined probability can be determined according to actual needs and experimental results.

[0082] That is, when generating the first sample image, the device can paste the target sub-regions and the non-target sub-regions into the first image according to the depth information of the target sub-regions and the non-target sub-regions, and delete the corresponding regions in the first image. For example, the target sub-regions include A1, A2 and A3, the non-target sub-regions include B1 and B2, and the regions in the first image corresponding to the target sub-regions and the non-target sub-regions in terms of depth information, shape, size and area are A1', A2', A3', B1' and B2' in sequence. Then the device deletes A1', A2', A3', B1' and B2' from the first image, pastes A1, A2, A3, B1 and B2 into the first image, and generates the first sample image.

[0083] In some optional embodiments, the device can also paste the target sub-regions including the target and the non-target sub-regions not including the target into the first image based on the color distribution of each target in the sample set and the color of the target corresponding to the target sub-regions, and generate the first sample image.

[0084] For example, if there are fewer red cars in the sample set, a greater probability is set for changing the color of the car to red (color change according to the data distribution in the target color database) or directly selecting a red car when pasting the target.

[0085] In other optional embodiments, the device can also paste the target sub-regions including the target and the non-target sub-regions not including the target into the first image based on the orientation of each target in the sample set and the orientation of the target corresponding to the target sub-regions, and generate the first sample image.

[0086] For example, if there are fewer targets in a certain orientation in the sample set, a greater probability is set for selecting the target in the orientation when selecting the target.

[0087] Step S202 inputs the first sample image into a backbone network included in the target detection model, and confirms that the output of the backbone network is the feature corresponding to the first sample image.

[0088] In some embodiments, the backbone network can be a backbone network, and the device inputs the first sample image into the backbone network, and confirms that the output of the backbone network is the feature of the first sample image; subsequently, the device can input the feature of the first sample image into a prediction network for sample image prediction, or input the output of the backbone network into a feature discriminator included in the target detection model, and optimize the feature of the sample image output by the backbone network, so that the feature of the sample image output by the backbone network is closer to the feature of a natural pixel.

[0089] Specifically, the output of the backbone network is the feature distribution of the first sample image. Since the first sample image is a synthetic image (a pasted image), the feature distribution thereof is different from the feature distribution of a natural image (or a natural pixel), which further misleads the training result of the target detection model. In the present disclosure, a feature discriminator is introduced, so that the feature distribution of the image output by the backbone network is closer to the feature distribution of a natural image, so that the model inference using a natural image in the later stage can have an accurate output result.

[0090] Step S203: inputting the feature corresponding to the first sample image into a prediction network included in the target detection model, and confirming a first sub-loss.

[0091] In some embodiments, the prediction network can be a prediction head. The first sub-loss includes a minimum loss function corresponding to the backbone network and the prediction network.

[0092] Step S204: updating the parameters of the backbone network and the prediction network based on the first sub-loss.

[0093] In specific implementation, the device fixes the parameters of the feature discriminator, inputs the first sample image into the backbone network and the prediction network, confirms a first sub-loss value, updates the parameters of the backbone network and the prediction network based on the first sub-loss value. Then, the device sets the parameters of the backbone network and the prediction network as the updated parameters, inputs the output of the backbone network into the feature discriminator (i.e., trains the feature discriminator), determines a second sub-loss, and updates the parameters of the feature discriminator based on the second sub-loss.

[0094] Further, the device confirms the second sample image based on step S201, confirms that the parameters of the feature identification network are updated parameters, inputs the second sample image into the backbone network and the prediction network, confirms the third sub-loss value, and updates the parameters of the backbone network and the prediction network based on the third sub-loss value. Then set the parameters of the backbone network and the prediction network to the updated parameters, input the output of the backbone network into the feature identification network (i.e. train the feature identification network), determine the fourth sub-loss, and update the parameters of the feature identification network based on the fourth sub-loss. Wherein, the first sample image and the second sample image are any sample image in the training set.

[0095] In some embodiments, the first sub-loss includes a minimum loss function corresponding to the backbone network and the prediction network; the second sub-loss is determined based on the output of the feature identification network under the condition of natural pixel input, and the output of the feature identification network when the output of the backbone network after pasting pixel input into the backbone network is input into the feature identification network; and the third sub-loss is determined based on the minimum loss function corresponding to the backbone network and the prediction network, the minimum loss function of the backbone network under the condition of pasting pixel input into the backbone network, and the output of the feature identification network when the output of the backbone network after pasting pixel input into the backbone network is input into the feature identification network.

[0096] In specific implementation, the device determines the expectation of the logarithmic value of the output of the feature identification network under the condition of natural pixel input, sums the expectation of the logarithmic value of the output of the feature identification network, and confirms that the result of the summation is the second sub-loss; sums the minimum loss function corresponding to the backbone network and the prediction network, the minimum loss function of the backbone network under the condition of pasting pixel input into the backbone network, and the expectation of the logarithmic value of the output of the feature identification network, and determines the summation result as the third sub-loss.

[0097] Such repeated training of the backbone network, the prediction network and the feature identification network, and then verifying the target detection model based on the verification set.

[0098] The generation method of the sample images included in the verification set is the same as step S201, which is not repeated here. Optionally, after generating multiple sample images according to step S201, the multiple sample images can be divided into a training set and a verification set.

[0099] The verification set verifies the target detection model in the same process as the training process, that is, the sample image is input into the backbone network and the prediction network, the sub-loss value is obtained, then the output of the backbone network is input into the feature identification network, and the loss corresponding to the verification set is confirmed based on the outputs of the prediction network, the backbone network and the feature identification network; if the loss converges to the expected range, it is confirmed that the target detection model training is completed; if the loss does not converge to the expected range, steps S202 to S204 are repeated until the loss corresponding to the verification set converges to the expected range. The expected range can be determined according to experimental results or actual needs.

[0100] In this way, the training method of the target detection model provided by the embodiments of the present disclosure trains based on the sample image including the target sub-region of the target and the non-target sub-region not including the target, which to some extent offsets the influence of artificially synthesized images on the training of the target detection model, and further improves the rationality of the sample image by fully utilizing the target information of the image, and further improves the accuracy of the target detection model trained based on the sample image. In addition, through the adversarial training of the feature distribution of the natural image and the feature distribution of the synthesized image (i.e. the sample image), the model generates the same feature distribution for the pasted pixels and the natural pixels.

[0101] Figure 3 Another optional flowchart of the training method of the target detection model provided by the embodiments of the present disclosure is shown, which will be described according to each step.

[0102] Step S301, generating a sample set.

[0103] In some embodiments, the sample set can be generated based on steps S301a to S301d, and further, the sample set can be divided into a training set and a verification set, the training set is used for training the target detection model, and the verification set is used for verifying the target detection model trained based on the training set.

[0104] Step S301a, constructing a 3D target database.

[0105] In some embodiments, the training device of the target detection model (hereinafter referred to as the device) extracts 3D targets (hereinafter referred to as targets) from the target detection data set obtained based on a monocular camera (hereinafter referred to as a camera), and stores them in a database, records the position coordinates (x, y, z) of the target in the image, the orientation of the target (global azimuth angle and observation angle in the camera coordinate system), the size of the target (length, width, height), the 2D rectangular frame of the target periphery, the occlusion degree of the target (the degree of being occluded by other objects), the truncation degree, the target category, the color, the target original picture, the camera internal parameter and the target segmentation graph (which can be obtained by instance segmentation), point cloud and other data.

[0106] Step S301b, constructing a target color database.

[0107] In some embodiments, the device confirms the HSV color distribution of each target in the sample set or training set, and constructs a target color database based on the HSV color distribution of each target.

[0108] Step S301c, confirming the data distribution of the 3D target database.

[0109] In some embodiments, the device determines the data distribution of the position, orientation, size, depth information and other attributes of each target in the 3D target database.

[0110] Step S301d, generating a first sample image.

[0111] In some embodiments, the device can identify the road and ground in the first image based on the method of semantic segmentation; further, based on the method of instance segmentation, it can identify all the targets included in the first image, and for each target in the first image, it can retain or eliminate it according to a predetermined probability. When eliminating, the target area is covered with the background information around the target, or the target area is covered based on the target in the 3D target database.

[0112] In other embodiments, the device can also select targets that meet the semantic background and depth from the 3D target database, and then according to the projection of the 3D position of the target in the 2D image, paste the target sub-area corresponding to the target to the corresponding position in the first image.

[0113] When selecting targets, the data distribution of the target attributes in the 3D target database should also be considered. For example, if there are fewer red cars in the 3D target database, when pasting the target, a larger probability will be set to change the color of the car to red (color change according to the data distribution in the target color database), or directly select a red car. If there are fewer targets in a certain orientation in the 3D target database, a larger probability will be set to select targets in that orientation when selecting targets.

[0114] According to the data distribution of the target attributes in the 3D target database, the depth of the target can be scaled down or up, according to the scaled depth, the 2D projection of the target is calculated, and then the size of the 2D rectangular frame is calculated. Then, according to the size of the 2D rectangular frame, the target segmentation graph is scaled.

[0115] Figure 4 A schematic diagram of generating a sample image is shown, as shown in Figure 4 The data in the 3D target database is pasted into the first image according to the depth information and the semantics of the first image, and a first sample image is generated.

[0116] Step S302, training the target detection model based on the training set.

[0117] Figure 5 A structural diagram of the target detection model provided by the embodiments of the present disclosure is shown.

[0118] As shown in the figure, the target detection model provided by the embodiments of the present disclosure includes a backbone, a feature discriminator and a prediction head. The output of the backbone is the feature distribution of the image, which is input to the feature discriminator and the prediction head. The backbone generates features in the training process, the prediction head generates prediction results using the features generated by the backbone, and the feature discriminator is used to determine whether each point on the feature map comes from a natural target or a pasted target in the training stage. Figure 5

[0119] The loss function of the model training is determined based on the following formula:

[0120]

[0121] wherein D represents the feature discriminator, G represents the backbone, p data represents the distribution generated by natural pixels (pixels included in natural images), p z represents the distribution generated by pasted pixels (pixels included in synthetic images), H represents the loss function of the conventional 3D target detection model (i.e. the backbone and the prediction head in the present disclosure); further, when x represents a natural pixel, D(x) represents the output of the feature discriminator when the natural pixel is input; when z represents a pasted pixel, D(z) represents the output of the feature discriminator when the pasted pixel is input. Furthermore, G(z) represents the output of the backbone when the pasted pixel is input, and D(G(x)) represents the output of the feature discriminator when the output of the backbone after the pasted pixel is input to the backbone is input to the feature discriminator; minH(G(x)) represents the minimum loss function of the backbone when the natural pixel is input, and minH(G(z)) represents the minimum loss function of the backbone when the pasted pixel is input.

[0122] ​In some embodiments, the device keeps the parameters of the feature identification network from updating, first trains the backbone network and the prediction network for k1 iterations (steps); wherein k1 is a hyperparameter, which can be determined according to actual needs or experimental results, the first sub-loss is minH (that is, the minimum loss function of the backbone network and the prediction network, which can also be understood as the minimum loss function of a conventional 3D detection model (the model includes the backbone network and the prediction network)); update the parameters of the backbone network and the prediction network based on the first sub-loss, and keep the parameters of the backbone network and the prediction network from updating after the update (the process of training the feature identification network does not update), train the feature identification network for k2 steps; wherein k2 is a hyperparameter, which can be determined according to actual needs or experimental results, and the maximum expectation (the second sub-loss) can be expressed as:

[0123]

[0124] wherein D(G(x)) represents the output of the feature identification network when the natural pixels are input into the backbone network and the output of the backbone network is taken as the input of the feature identification network; and D(G(z)) represents the output of the feature identification network when the pasted pixels (that is, the pixels of the sample image) are input into the backbone network and the output of the backbone network is taken as the input of the feature identification network. represents the expectation of the logarithmic value of the output of the feature identification network when the natural pixels are input into the backbone network and the output of the backbone network is taken as the input of the feature identification network. represents the expectation of the result of the logarithmic operation after the output of the feature identification network when the pasted pixels are input into the backbone network and the output of the backbone network is taken as the input of the feature identification network.

[0125] update the parameters of the feature identification network based on the second sub-loss, then keep the parameters of the feature identification network from updating (the process of training the backbone network and the prediction network does not update), train the backbone network and the prediction network for k3 steps; wherein k3 is a hyperparameter, which can be determined according to actual needs or experimental results, and the third sub-loss is:

[0126]

[0127] wherein, represents the expectation of the result of the logarithmic operation after the output of the feature identification network when the pasted pixels are input into the backbone network and the output of the backbone network is taken as the input of the feature identification network.

[0128] The above process is repeated, that is, the target detection model is first trained using the training set, and the parameters of the backbone network, the prediction network and the feature identification network are updated during the training process, until the training based on all sample images in the training set is completed, then the target detection model with updated parameters is verified using the verification set, until the loss on the verification set converges to the desired range (the loss function determined based on the third sub-loss and the fourth sub-loss satisfies the first condition, it is determined that the training of the target detection model is completed), wherein p test represents the data distribution of the verification set.

[0129] Figure 6 The results of the target detection method provided by the embodiments of the present disclosure are shown in the following figure Figure 1 .

[0130] As Figure 6 shown, the higher the score MonoInsight of the target detection model trained by the training method of the target detection model provided by the embodiments of the present disclosure is, the better the model inference is.

[0131] Figure 6 The three columns of scores Easy, Moderate, and Hard in the figure represent three detection scores under different IOU thresholds, and the higher the score is, the better the detection effect is. The size relationship of the IOU threshold is: Easy < Moderate < Hard.

[0132] Figure 7 The results of the target detection method provided by the embodiments of the present disclosure are shown in the following figure Figure 2 .

[0133] Figure 7 There are also three columns of scores in the figure, from left to right, Moderate, Easy, and Hard, which have the same meaning as the above figure. The three columns of scores represent three detection scores under different IOU thresholds, and the higher the score is, the better the detection effect is. The size relationship of the IOU threshold is: Easy < Moderate < Hard.

[0134] The score of the target detection model provided by the present disclosure MonoInsight exceeds that of the model MonoCon and MonoFlex ranking at the top in the related art.

[0135] Figure 8 The results of the target detection method provided by the embodiments of the present disclosure are shown in the following figure Figure 3 .

[0136] As Figure 8The upper part is the result of 2D detection, and the lower part is the detection result provided by the present disclosure. It can be seen that the detection result provided by the present disclosure can detect the information of each target in the image, such as the distance, size, orientation, direction, etc. between the monocular camera, and provide more rich scene information for subsequent navigation, and assist the navigation to make more appropriate decisions.

[0137] Figure 9 An optional structural schematic diagram of the target detection device provided by the embodiments of the present disclosure is shown, which will be described according to each step.

[0138] In some embodiments, the target detection device 400 includes a first input unit 401 and a detection unit 402.

[0139] The first input unit 401 is configured to input a to-be-detected image into a backbone network included in a target detection model, and confirm that an output of the backbone network is a feature corresponding to the to-be-detected image.

[0140] The detection unit 402 is configured to input the feature corresponding to the to-be-detected image into a prediction network included in the target detection model, and confirm that an output of the prediction network is target information included in the to-be-detected image.

[0141] The target information included in the to-be-detected image includes at least one of position information, size information, orientation information, depth information and occlusion information of each target in the to-be-detected image; and the target detection model is obtained based on a sample image including a target sub-region and a non-target sub-region.

[0142] Figure 10 An optional structural schematic diagram of a training device of a target detection model provided by the embodiments of the present disclosure is shown, which will be described according to each step.

[0143] In some embodiments, the training device 500 of the target detection model includes a sample generation unit 501, a second input unit 502, a third input unit 503 and an update unit 504.

[0144] The sample generation unit 501 is configured to paste a target sub-region including a target and a non-target sub-region not including a target to a first image to generate a first sample image.

[0145] The second input unit 502 is configured to input the first sample image into a backbone network included in the target detection model, and confirm that an output of the backbone network is a feature corresponding to the first sample image.

[0146] The third input unit 503 is configured to input the feature corresponding to the first sample image into a prediction network included in the target detection model, and confirm a first sub-loss.

[0147] The updating unit 504 is configured to update parameters of the backbone network and the prediction network based on the first sub-loss.

[0148] The sample generation unit 501 is specifically configured to paste the target sub-region and the non-target sub-region to regions corresponding to the depth information in the first image based on the depth information of the target sub-region and the non-target sub-region, and confirm the pasted image as the first sample image.

[0149] The sample generation unit 501 is further configured to obtain at least one sub-region corresponding to each original target in the first image, retain or delete the at least one sub-region based on a predetermined probability, and paste a non-target sub-region with consistent depth information to a deleted sub-region in the first image.

[0150] The sample generation unit 501 is specifically configured to paste the target sub-region including the target and the non-target sub-region not including the target to the first image based on color distribution of each target in a sample set including the first sample image and color of the target corresponding to the target sub-region, and generate the first sample image.

[0151] The second input unit 502 is further configured to input the feature corresponding to the first sample image to a feature identification network included in the target detection model after updating the parameters of the backbone network and the prediction network based on the first sub-loss, determine a second sub-loss value, and update parameters of the feature identification network based on the second sub-loss value.

[0152] The sample generation unit 501 is further configured to confirm a second sample image after updating the parameters of the feature identification network based on the second sub-loss value.

[0153] The second input unit 502 is further configured to input the second sample image to the backbone network with updated parameters, and confirm that an output of the backbone network is a feature corresponding to the second sample image.

[0154] The third input unit 503 is further configured to input the feature corresponding to the second sample image to the prediction network with updated parameters, and confirm a third sub-loss.

[0155] The updating unit 504 is further configured to update the parameters of the backbone network and the prediction network based on the third sub-loss.

[0156] The updating unit 504 is further configured to input the features corresponding to the second sample image into the feature identification network with updated parameters, determine a fourth sub-loss based on the fourth sub-loss, update parameters of the feature identification network based on the fourth sub-loss, and repeat the training of the backbone network, the prediction network and the feature identification network.

[0157] In the above scheme, the first sub-loss includes a minimized loss function corresponding to the backbone network and the prediction network; the second sub-loss is determined based on an output of the backbone network when natural pixels are input into the backbone network and the output of the backbone network is used as input of the feature identification network, and an output of the feature identification network when pasted pixels are input into the backbone network and the output of the backbone network is used as input of the feature identification network; and the third sub-loss is determined based on a minimized loss function corresponding to the backbone network and the prediction network, a minimized loss function of the backbone network when pasted pixels are input into the backbone network, and an output of the feature identification network when the output of the backbone network after the pasted pixels are input into the backbone network is used as input of the feature identification network.

[0158] The updating unit 504 is specifically configured to determine a sum of an expectation of a logarithmic value of an output of the feature identification network when natural pixels are input into the backbone network and the output of the backbone network is used as input of the feature identification network, and an expectation of a logarithmic value of an output of the feature identification network when pasted pixels are input into the backbone network and the output of the backbone network is used as input of the feature identification network, and confirm that the sum is the second sub-loss.

[0159] The updating unit 504 is specifically configured to determine a sum of a minimized loss function corresponding to the backbone network and the prediction network, a minimized loss function of the backbone network when pasted pixels are input into the backbone network, and an expectation of a logarithmic value of an output of the feature identification network, and determine that the sum is the third sub-loss.

[0160] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device and a readable storage medium.

[0161] Figure 11 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0162] like Figure 11 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0163] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0164] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as object detection methods and / or object detection model training methods. For example, in some embodiments, the object detection methods and / or object detection model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the object detection methods and / or object detection model training methods described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured by any other suitable means (e.g., by means of firmware) to perform object detection methods and / or training methods for object detection models.

[0165] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0166] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0167] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0168] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0169] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0170] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is generally established by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0171] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, in series, or in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0172] In addition, the terms "first", "second", etc., are used herein only to describe different instances, and do not imply or suggest relative importance or an implied number of the indicated technical features. Thus, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0173] The above merely provides the specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present disclosure, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A target detection method characterized by, The method comprises: inputting the to-be-detected image into a backbone network included in the target detection model, and confirming an output of the backbone network as features corresponding to the to-be-detected image; inputting the features corresponding to the to-be-detected image into a prediction network included in the target detection model, and confirming an output of the prediction network as target information included in the to-be-detected image; wherein the target information included in the to-be-detected image comprises at least one of position information, size information, orientation information, depth information and occlusion information of each target in the to-be-detected image; the target detection model is trained based on sample images including target sub-regions and non-target sub-regions, and the sample images are obtained based on the following manner: obtaining at least one sub-region corresponding to each original target in a first image; retaining or deleting the at least one sub-region based on a predetermined probability; pasting non-target sub-regions with consistent depth information to the deleted sub-regions in the first image, and confirming the pasted image as a first sample image. 2.A method for training a target detection model, the method comprising: The method comprises: pasting target sub-regions including targets and non-target sub-regions not including targets into a first image to generate a first sample image; comprising: obtaining at least one sub-region corresponding to each original target in the first image; retaining or deleting the at least one sub-region based on a predetermined probability; pasting non-target sub-regions with consistent depth information to the deleted sub-regions in the first image, and confirming the pasted image as the first sample image; inputting the first sample image into a backbone network included in the target detection model, and confirming an output of the backbone network as features corresponding to the first sample image; inputting the features corresponding to the first sample image into a prediction network included in the target detection model, and confirming a first sub-loss; updating parameters of the backbone network and the prediction network based on the first sub-loss.

3. The method of claim 2, wherein, The method comprises: pasting target sub-regions including targets and non-target sub-regions not including targets into a first image to generate a first sample image; comprising: pasting the target sub-regions and the non-target sub-regions into regions corresponding to depth information of the target sub-regions and the non-target sub-regions in the first image based on the depth information, and confirming the pasted image as the first sample image.

4. The method of claim 2, wherein, The method comprises: pasting target sub-regions including targets and non-target sub-regions not including targets into a first image to generate a first sample image; comprising: pasting the target sub-regions and the non-target sub-regions into regions corresponding to depth information of the target sub-regions and the non-target sub-regions in the first image based on the depth information, and confirming the pasted image as the first sample image.

5. The method of claim 2, wherein, After updating the parameters of the backbone network and the prediction network based on the first sub-loss, the method further comprises: inputting the features corresponding to the first sample image into a feature identification network included in the target detection model, and determining a second sub-loss value; updating parameters of the feature identification network based on the second sub-loss value.

6. The method of claim 5, wherein, After updating the parameters of the feature identification network based on the second sub-loss value, the method further comprises: confirming a second sample image; inputting the second sample image into the backbone network with updated parameters to confirm an output of the backbone network as features corresponding to the second sample image; inputting the features corresponding to the second sample image into the prediction network with updated parameters to confirm a third sub-loss; updating parameters of the backbone network and the prediction network based on the third sub-loss.

7. The method of claim 6, wherein, After the parameters of the backbone network and the prediction network are updated based on the third sub-loss, the method further comprises: inputting the features corresponding to the second sample image into the feature identification network with updated parameters to determine a fourth sub-loss; updating parameters of the feature identification network based on the fourth sub-loss; repeating the training of the backbone network, the prediction network and the feature identification network.

8. The method of claim 7, wherein, the first sub-loss comprises a minimized loss function corresponding to the backbone network and the prediction network; the second sub-loss is determined based on an output of the feature identification network when the output of the backbone network is input into the feature identification network after natural pixels are input into the backbone network, and an output of the feature identification network when the output of the backbone network is input into the feature identification network after the pasted pixels are input into the backbone network; the third sub-loss is determined based on a minimized loss function corresponding to the backbone network and the prediction network, a minimized loss function of the backbone network in the case that the pasted pixels are input into the backbone network, and an output of the feature identification network when the output of the backbone network is input into the feature identification network after the pasted pixels are input into the backbone network.

9. The method of claim 8, wherein, the determination of the second sub-loss based on the output of the feature identification network when the output of the backbone network is input into the feature identification network after natural pixels are input into the backbone network, and the output of the feature identification network when the output of the backbone network is input into the feature identification network after the pasted pixels are input into the backbone network comprises: summing an expectation of a logarithmic value of the output of the feature identification network when the output of the backbone network is input into the feature identification network after natural pixels are input into the backbone network, and an expectation of a logarithmic value of a sum of a reciprocal of the output of the feature identification network and 1 when the output of the backbone network is input into the feature identification network after the pasted pixels are input into the backbone network, to confirm a result of the summation as the second sub-loss.

10. The method of claim 8, wherein, the determination of the third sub-loss based on the minimized loss function corresponding to the backbone network and the prediction network, the minimized loss function of the backbone network in the case that the pasted pixels are input into the backbone network, and the output of the feature identification network when the output of the backbone network is input into the feature identification network after the pasted pixels are input into the backbone network comprises: summing the minimized loss function corresponding to the backbone network and the prediction network, the minimized loss function of the backbone network in the case that the pasted pixels are input into the backbone network, and an expectation of a logarithmic value of the output of the feature identification network, to determine a result of the summation as the third sub-loss.

11. A target detection apparatus characterized by comprising: the apparatus comprises: a first input unit configured to input a to-be-detected image into a backbone network included in a target detection model to confirm an output of the backbone network as features corresponding to the to-be-detected image; The detection unit is configured to input the feature corresponding to the image to be detected into a prediction network included in the target detection model, and confirm that output of the prediction network is target information included in the image to be detected. The target information included in the image to be detected includes at least one of position information, size information, orientation information, depth information and occlusion information of each target in the image to be detected; and the target detection model is trained based on sample images including target sub-regions and non-target sub-regions, and the sample images are obtained based on the following manner: at least one sub-region corresponding to each original target in a first image is acquired; the at least one sub-region is retained or deleted based on a predetermined probability; a non-target sub-region with consistent depth information is pasted to a deleted sub-region in the first image, and the pasted image is confirmed as a first sample image. 12.A device for training a target detection model, characterized in that, The apparatus comprises: The sample generation unit is configured to paste target sub-regions including targets and non-target sub-regions not including targets into a first image to generate a first sample image, and specifically configured to: acquire at least one sub-region corresponding to each original target in the first image; retain or delete the at least one sub-region based on a predetermined probability; paste a non-target sub-region with consistent depth information to a deleted sub-region in the first image, and confirm that the pasted image is the first sample image; The second input unit is configured to input the first sample image into a backbone network included in the target detection model, and confirm that output of the backbone network is a feature corresponding to the first sample image; The third input unit is configured to input the feature corresponding to the first sample image into a prediction network included in the target detection model, and confirm a first sub-loss; The update unit is configured to update parameters of the backbone network and the prediction network based on the first sub-loss.

13. An electronic device, comprising: comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of claim 1; or, the method of any one of claims 2-10 can be executed.

14. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to make the computer execute the method according to claim 1; or, the method according to any one of claims 2-10 is executed.

Images