3D Object Detection Model Training Method, Apparatus, and 3D Object Detection Method

Abstract

The present application provides a 3D object detection model training method, apparatus, and 3D object detection method, which relate to the technical field of image data processing. The 3D object detection model training method includes: obtaining a labeled source domain data set; training a 3D object detection model to be trained through the labeled source domain data set to obtain an initial 3D object detection model; obtaining an unlabeled target domain data set; inputting a target domain image into the initial 3D object detection model to obtain a target domain class probability; calculating a collection score of the target domain image according to the target domain class probability; if the collection score meets a preset target, performing annotation on the target domain image to obtain a labeled target domain data set; and training the initial 3D object detection model through the labeled target domain data set to obtain a 3D object detection model. This method can improve the domain adaptability, generalization ability of the 3D object detection model, and the 3D object detection accuracy for target domain images.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of image data processing, and in particular, to a 3D object detection model training method, apparatus, and 3D object detection method. Background Art

[0002] 3D object detection can detect and locate target objects in a three-dimensional scene. Compared with 2D object detection, 3D object detection also needs to detect depth information to more accurately understand the position and shape of objects in the scene. Therefore, 3D object detection is widely used in fields such as autonomous driving, augmented reality, and robot vision.

[0003] The bird's-eye view (BEV) based on a camera has advantages in terms of perception and understanding, rich semantic information, high computational efficiency, etc., and is applied to 3D object detection. However, due to the cross-domain differences between different data domains, the performance of a BEV model based only on a camera will degrade when applied to other datasets. Therefore, in order to enable a 3D object detection algorithm to be applied to different scenarios, a domain adaptation method needs to be adopted. That is, source domain data and labels as well as target domain data are obtained, a model is trained on the source domain, and it is applied to the target domain to reduce the distribution differences between domains, so as to achieve object detection under the condition that the training and test data distributions are inconsistent.

[0004] However, due to the lack of labeled data in the target domain, when the gap between the source domain and the target domain is large, the adaptability and generalization ability of the model are poor. In addition, there are problems of large intra-class differences and small inter-class differences for different classes, resulting in low detection accuracy of the model. Summary of the Invention

[0005] To solve the above problems, the present application provides a 3D object detection model training method, apparatus, and 3D object detection method, which can solve the technical problems of poor adaptability and generalization ability of the 3D object detection model and low detection accuracy.

[0006] To achieve the above object, in a first aspect, the present application provides a 3D object detection model training method, and the 3D object detection model training method includes:

[0007] Obtain a labeled source domain dataset, where the labeled source domain dataset includes source domain images and label information of the source domain images. The label information includes source domain object detection boxes and source domain object categories. Among them, the source domain images are images collected by multiple cameras, the source domain object detection boxes are coordinate information of object features in the source domain images, and the source domain object categories are category information of object features in the source domain images. Train the 3D object detection model to be trained using the labeled source domain dataset to obtain an initial 3D object detection model. Obtain an unlabeled target domain dataset, where the unlabeled target domain dataset includes target domain images. The target domain images are images collected by multiple cameras, and the types of the target domain images are different from those of the source domain images. Input the target domain images into the initial 3D object detection model to obtain target domain category probabilities, where the target domain category probabilities are the probabilities that the object features in the target domain images belong to each category. Calculate the acquisition score of the target domain images according to the target domain category probabilities. If the acquisition score meets the preset target, perform annotation on the target domain images to obtain a labeled target domain dataset. Train the initial 3D object detection model using the labeled target domain dataset to obtain a 3D object detection model.

[0008] In the above method, first train an initial 3D object detection model according to the labeled source domain dataset, then input the unlabeled target domain dataset into the initial 3D object detection model for prediction to obtain target domain category probabilities; calculate the acquisition score according to the target domain category probabilities, select target domain images for annotation according to the acquisition score, and then obtain a labeled target domain dataset; finally, train the initial 3D object detection model according to the labeled target domain dataset to obtain a 3D object detection model. This method can improve the domain adaptability, generalization ability of the 3D object detection model and the 3D object detection accuracy for target domain images.

[0009] In one implementable manner of the first aspect, training the 3D object detection model to be trained using the labeled source domain dataset to obtain an initial 3D object detection model includes: performing preprocessing on the source domain images to obtain a BEV feature map; inputting the BEV feature map into a detection probe to obtain predicted object detection boxes and predicted object categories; cropping the BEV feature map according to the source domain object detection boxes to obtain object features, and storing the object features in a storage body; performing latent space data augmentation on the object features; calculating the latent space data augmentation loss according to the latent space data augmentation method; calculating the object perception contrastive learning loss according to the object features; calculating the object detection loss according to the source domain object detection boxes, source domain object categories, predicted object detection boxes and predicted object categories; adding the latent space data augmentation loss, the object perception contrastive learning loss and the object detection loss to obtain a model training loss; optimizing the 3D object detection model to be trained according to the model training loss to obtain an initial 3D object detection model.

[0010] In the above method, performing latent space data augmentation on the target features obtained from the BEV feature map can achieve data augmentation and improve detection accuracy; the model training loss obtained by adding the latent space data augmentation loss, object perception contrastive learning loss, and object detection loss can improve the accuracy of model training, thereby improving the model accuracy.

[0011] In one implementable manner of the first aspect, preprocessing the source domain image to obtain a BEV feature map includes: cropping the source domain image to obtain a standard source domain image; performing data augmentation on the standard source domain image to obtain an augmented source domain image; performing feature extraction on the augmented source domain image through a feature extractor to obtain a feature image; inputting the feature image into a depth prediction network to obtain depth distribution data; inputting the feature image into a convolutional network to obtain a content-aware feature map; multiplying the depth distribution data and the content-aware feature map to obtain a BEV feature map.

[0012] In the above method, after cropping, data augmentation, and feature extraction are performed on the source domain image, a feature image is obtained. Then, the feature image is processed by a depth prediction network and a convolutional network respectively, and the processing results are multiplied to obtain a BEV feature map for implementing 3D object detection.

[0013] In one implementable manner of the first aspect, cropping the BEV feature map according to the source domain object detection box to obtain target features includes: cropping the BEV feature map according to the source domain object detection box to obtain target features; wherein, the source domain object detection box is associated with the source domain image from which the BEV feature map is obtained.

[0014] In the above method, the source domain object detection box is the label data in the training dataset. Cropping the BEV feature map through the source domain object detection box can obtain accurate target features and improve the accuracy of model training.

[0015] In one implementable manner of the first aspect, performing latent space data augmentation on the target features includes: obtaining the target features in the storage body; performing latent space data augmentation on the target features through a Gaussian distribution, and the formula is as follows:

[0016] ,

[0017] where represents the Gaussian distribution; represents the th target feature of the th BEV feature map; represents the ratio of the current training epoch to the maximum training epoch; represents the covariance of the source domain target class.

[0018] In the above method, Gaussian distribution can be used to achieve latent space data augmentation based on data sampling, increase the diversity of training data, and improve the accuracy of model training.

[0019] In one implementation of the first aspect, calculating the latent space data augmentation loss according to the latent space data augmentation method includes: determining the calculation upper bound based on the sampled features obtained after performing latent space data augmentation on the target features; calculating the latent space data augmentation loss according to the calculation upper bound, and the formula is as follows:

[0020]

[0021] ,

[0022] Among them, represents the latent space data augmentation loss; represents the number of BEV feature maps; represents the th target number of the th BEV feature map; represents the th target feature of the and respectively represent the weight matrix and bias of the last fully connected layer of the depth prediction network; ; represents the th column of the weight matrix parameter of the last fully connected layer; represents the number of source domain target classes.

[0023] In the above method, by introducing an upper bound that is easy to calculate as the latent space data augmentation loss, the expectation of the target features can be directly calculated, avoiding directly calculating the target features after Gaussian distribution augmentation, reducing the calculation overhead, and improving the speed of model training.

[0024] In one implementation of the first aspect, the preset goals include: the acquisition score is greater than or equal to the preset threshold, or, the number of digits in the descending order of the acquisition score is the preset number of digits.

[0025] In the above method, by setting the preset goals and selecting the target domain images with acquisition scores meeting the expectations, the accuracy of the target domain data can be improved, and thus the model accuracy can be enhanced.

[0026] In one implementation of the first aspect, performing annotation on the target domain images to obtain a labeled target domain data set includes: importing the target domain images into the annotation tool; obtaining the targets to be annotated in the target domain images; drawing the bounding boxes of the targets to be annotated, and saving the data information of the bounding boxes as the target detection boxes of the target features in the target domain images.

[0027] In the above method, the target domain image is labeled by a labeling tool, so that the target detection box of the target feature can be accurately identified, the accuracy of the target domain data is improved, and thus the accuracy of 3D target detection is improved.

[0028] In a second aspect, the present application further provides a 3D target detection model training device. The 3D target detection model training device includes: a labeled source domain data set acquisition module configured to acquire a labeled source domain data set, where the labeled source domain data set includes source domain images and label information of the source domain images, and the label information includes source domain target detection boxes and source domain target categories; wherein, the source domain images are images acquired by multiple cameras, the source domain target detection boxes are coordinate information of target features in the source domain images, and the source domain target categories are category information of target features in the source domain images; a source domain training module configured to train a 3D target detection model to be trained through the labeled source domain data set to obtain an initial 3D target detection model; an unlabeled target domain data set acquisition module configured to acquire an unlabeled target domain data set, where the unlabeled target domain data set includes target domain images, the target domain images are images acquired by multiple cameras, and the target domain images are of a different type from the source domain images; a category probability prediction module configured to input the target domain images into the initial 3D target detection model to obtain target domain category probabilities, where the target domain category probabilities are the probabilities that the target features in the target domain images belong to each category; an acquisition score calculation module configured to calculate an acquisition score of the target domain images according to the target category probabilities; a labeling module configured to perform labeling on the target domain images to obtain a labeled target domain data set if the acquisition score meets a preset target; and a target domain training module configured to train the initial 3D target detection model through the labeled target domain data set to obtain a 3D target detection model.

[0029] In a third aspect, the present application further provides a 3D target detection method. The 3D target detection method includes: acquiring test set data, where the test set data includes test images, and the test images are images acquired by multiple cameras; performing feature extraction on the test images through a feature extractor to obtain feature images; inputting the feature images into a depth prediction network to obtain depth distribution data; inputting the feature images into a convolutional network to obtain a content-aware feature map; multiplying the depth distribution data and the content-aware feature map to obtain a BEV feature map; and inputting the BEV feature map into a detection probe to obtain detection boxes and detection categories.

[0030] It can be understood that the beneficial effects that can be achieved by the technical solutions provided in the second and third aspects above can refer to the beneficial effects in the first aspect and any one of its optional implementation manners, and will not be elaborated here.

[0031] As can be seen from the above technical solutions, the present application provides a 3D object detection model training method, device, and 3D object detection method. The 3D object detection model training method includes: obtaining a labeled source domain dataset, where the labeled source domain dataset includes source domain images and label information of the source domain images, and the label information includes source domain object detection frames and source domain object categories; wherein, the source domain images are images collected by multiple cameras, the source domain object detection frames are coordinate information of object features in the source domain images, and the source domain object categories are category information of object features in the source domain images; training a 3D object detection model to be trained through the labeled source domain dataset to obtain an initial 3D object detection model; obtaining an unlabeled target domain dataset, where the unlabeled target domain dataset includes target domain images, the target domain images are images collected by multiple cameras, and the types of the target domain images are different from those of the source domain images; inputting the target domain images into the initial 3D object detection model to obtain target domain category probabilities, where the target domain category probabilities are the probabilities that the object features in the target domain images belong to each category; calculating the acquisition score of the target domain images according to the target domain category probabilities; if the acquisition score meets a preset target, performing annotation on the target domain images to obtain a labeled target domain dataset; and training the initial 3D object detection model through the labeled target domain dataset to obtain a 3D object detection model.

[0032] In the above method, first, an initial 3D object detection model is trained according to the labeled source domain dataset, then the unlabeled target domain dataset is input into the initial 3D object detection model for prediction to obtain target domain category probabilities; and the acquisition score is calculated according to the target domain category probabilities, and target domain images are selected for annotation according to the acquisition score, so as to obtain a labeled target domain dataset; finally, the initial 3D object detection model is trained according to the labeled target domain dataset to obtain a 3D object detection model. This method can improve the domain adaptability, generalization ability of the 3D object detection model, and the 3D object detection accuracy for target domain images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the present application, the drawings required for the embodiments will be briefly introduced below. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0034] Figure 1 It is a schematic flowchart of a 3D object detection model training method provided by an embodiment of the present application;

[0035] Figure 2 It is a schematic diagram of a method for training a 3D object detection model using a labeled source domain dataset provided by an embodiment of the present application;

[0036] Figure 3Schematic diagram of a method for generating a BEV feature map provided by an embodiment of the present application;

[0037] Figure 4 Schematic diagram of a method for annotating a target domain image provided by an embodiment of the present application;

[0038] Figure 5 Schematic diagram of a 3D object detection model training device provided by an embodiment of the present application;

[0039] Figure 6 Schematic diagram of a 3D object detection method flow provided by an embodiment of the present application. Detailed implementation manners

[0040] The embodiments will be described in detail below, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following embodiments do not represent all implementation manners consistent with the present application.

[0041] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the implementation manners described next, rather than intending to limit the implementation manners of the present application. Unless otherwise specified, these terms should be understood in their ordinary and common meanings.

[0042] The terms "first", "second", "third", etc. in the specification and the above-mentioned drawings of the present application are used to distinguish similar or homogeneous objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that such terms can be interchanged under appropriate circumstances.

[0043] For the convenience of understanding the solution, the following explains relevant terms:

[0044] 3D object detection: 3D object detection refers to detecting and recognizing objects in a three-dimensional space. It is necessary to not only determine the position of the object in the image, but also obtain its precise position, pose, and size in the three-dimensional world, etc., in order to achieve a comprehensive understanding of the objects in the scene.

[0045] BEV feature map: A BEV feature map is an image or matrix that represents the features of a scene from a bird's-eye view. It projects the scene information in a three-dimensional space onto a two-dimensional plane, usually presented from a top-down perspective, and at the same time extracts and encodes the features of various elements in the scene, using different numerical values or vectors to represent the feature information at different positions in the scene. These features can include the category, position, shape, texture, etc. of the object.

[0046] 3D object detection not only identifies the position and category of objects on a two-dimensional plane but also detects the depth information of the objects. The depth information can comprehensively and accurately understand the actual positions, true shapes of the objects in the scene, and the spatial relationships between them. For example, in the context of autonomous driving, accurately obtaining the three-dimensional position information of the vehicles, pedestrians, and obstacles ahead can ensure the safe and efficient driving of the vehicle. Through 3D object detection, the autonomous driving system can accurately judge the distance, orientation, and approximate size of other objects relative to the vehicle, thus making reasonable decisions in advance, such as accelerating, decelerating, avoiding, etc.

[0047] 3D object detection mainly analyzes and processes the BEV feature map to achieve the purpose of object detection and localization. From the perspective of perceptual understanding, the BEV feature map can provide an overhead view, enabling the system to more comprehensively perceive the distribution of objects in the scene. In terms of semantic information, the BEV feature map contains rich semantic content, which can clearly present information such as the categories and position relationships of different objects, providing strong support for subsequent analysis and decision-making.

[0048] However, in the actual application process, the 3D object detection model based on the BEV feature map has certain defects. There are obvious cross-domain differences between different data domains, mainly due to differences in aspects such as the data collection environment, equipment, methods, and the characteristics of the data itself. For example, the data collected in different geographical locations, weather conditions, and lighting intensities have different presented features and distribution laws. When the 3D object detection model based on the BEV feature map from only one data source is applied to other datasets with different training data sources, due to the differences in data distribution, the performance of the model will significantly decline.

[0049] To solve this problem and enable the 3D object detection model to be applied to different scenarios, the domain adaptation method emerged. Its basic principle is to obtain the source domain data and its corresponding labels, as well as the target domain data. Train the model on the source domain data, and by continuously adjusting the model's parameters, make it better fit the distribution characteristics of the source domain data. Then, apply the trained model to the target domain, and through a series of technical means, such as adversarial training, feature alignment, etc., reduce the distribution differences between the source domain and the target domain. In this way, even when the training data and the test data have inconsistent distributions, the model can accurately detect the objects in the target domain to a certain extent.

[0050] However, there are still some problems with current domain adaptation methods in practical applications. One of the main problems is the lack of labeled data in the target domain, which makes it difficult for the knowledge learned by the model in the source domain to be effectively transferred to the target domain. Especially when the gap between the source domain and the target domain is large, the adaptability and generalization ability of the model will deteriorate. In addition, there are also problems of large intra-class differences and small inter-class differences between different categories. For example, when detecting vehicles, vehicles of different brands, models, and colors constitute a large intra-class difference, while some objects of different categories with similar shapes, such as small cars and large toy cars, may result in small inter-class differences. These problems make it easy for the model to be confused when distinguishing objects of different categories, thereby reducing the detection accuracy.

[0051] To solve the above problems, the embodiments of the present application provide a 3D object detection model training method, device, and 3D object detection method. By means of latent space data augmentation, the generalization ability of the model is improved, and by jointly optimizing the model parameters with multiple losses, the detection accuracy of the model is improved. In addition, a method for labeling target domain data is added to expand the target domain data set, and then the 3D object detection model is further trained with the expanded target domain data set to improve the accuracy of the 3D object detection model for detecting target domain data.

[0052] Next, the specific implementation of the 3D object detection model training method will be introduced. The execution subject of this method can be an electronic device.

[0053] Figure 1 It is a schematic flowchart of a 3D object detection model training method provided by the embodiments of the present application. As Figure 1 shown, the 3D object detection model training method includes steps S100 - S700.

[0054] S100: Obtain a labeled source domain data set.

[0055] In some embodiments, to improve the generalization ability of the 3D object detection model, the NuScenes data set can be used as the labeled source domain data set for training. Among them, the labeled source domain data set includes source domain images, source domain object detection frames, and source domain object categories. Each source domain image is associated with at least one source domain object detection frame and one source domain object category, and the source domain object detection frame is associated with the source domain object category.

[0056] It should be noted that each set of data in the NuScenes dataset includes lane information images collected by multiple cameras, 3D detection boxes for determining objects in the multi-camera images, and the categories of each object determined by the 3D detection boxes. Among them, the 3D detection boxes correspond one-to-one with the categories of the objects. In some embodiments, the lane information images collected by multiple cameras in the NuScenes dataset are used as labeled source domain images, the 3D detection boxes are used as source domain object detection boxes, and the categories of the objects are used as source domain object categories.

[0057] Exemplarily, in a set of NuScenes data, it includes 6 lane information images taken by 6 cameras at different angles. There are a total of 10 objects in the 6 lane information images. Each object is framed by a source domain object detection box, and each object framed by the source domain object detection box corresponds to a source domain object category.

[0058] S200: Train the 3D object detection model to be trained using the labeled source domain dataset to obtain an initial 3D object detection model.

[0059] In some embodiments, to improve the accuracy of the 3D object detection model in detecting objects, it is first trained using the labeled source domain dataset. Specifically, using the source domain images as input data and the source domain object detection boxes and source domain object categories as labeled data, train the 3D object detection model to be trained to obtain an initial 3D object detection model.

[0060] Figure 2 The figure is a schematic diagram of a method for training a 3D object detection model using a labeled source domain dataset provided by an embodiment of the present application. As Figure 2 shown, training the 3D object detection model using the labeled source domain dataset includes steps S1 - S9.

[0061] Step S1: Perform preprocessing on the source domain images to obtain a BEV feature map.

[0062] The BEV feature map is to convert the image data obtained by the camera into a feature representation from an aerial view perspective, present the scene information in the form of a two-dimensional plane, and extract and encode the features in the scene, such as information about the position, shape, and category of objects.

[0063] Figure 3 The figure is a schematic diagram of a method for generating a BEV feature map provided by an embodiment of the present application. As Figure 3 shown, generating the BEV feature map includes steps S11 - S16.

[0064] S11: Crop the source domain images to obtain standard source domain images.

[0065] In some embodiments, to improve the adaptability of the model to data and thus improve the training efficiency of the model, it is first necessary to crop the source domain images to obtain standard source domain images with consistent sizes. Exemplarily, the source domain images are cropped into standard source domain images with a size of 256×704 pixels.

[0066] It should be understood that the source domain images can also be cropped into other sizes as long as the training requirements of the model are met, and no specific limitations are imposed in the embodiments of this application.

[0067] S12: Perform data augmentation on the standard source domain images to obtain augmented source domain images.

[0068] In some embodiments, to improve the diversity of the source domain data and thus improve the generalization ability of the model, data augmentation can be performed in the preprocessing stage of the source domain images to obtain augmented source domain images. Exemplarily, data augmentation methods such as random cropping, random scaling, random flipping, and random rotation are performed on the standard source domain images.

[0069] S13: Perform feature extraction on the augmented source domain images through a feature extractor to obtain feature images.

[0070] Exemplarily, the ResNet101 network is used as the feature extractor to perform feature extraction on the augmented source domain images to obtain feature images. The ResNet101 network is constructed by stacking multiple residual blocks, which avoids the problems of gradient disappearance and gradient explosion that occur during model training, enabling the network to train deeper layers. During feature extraction, the augmented source domain images are successively subjected to operations such as multiple convolutional layers, residual blocks, and pooling layers, gradually extracting features at different levels and with different degrees of abstraction.

[0071] S14: Input the feature images into a depth prediction network to obtain depth distribution data.

[0072] Specifically, the depth prediction network first parameterizes the camera internal parameters through a Multilayer Perceptron (MLP), and then processes them through convolutional layers and residual blocks, finally obtaining depth distribution data.

[0073] S15: Input the feature images into a convolutional network to obtain content-aware feature maps.

[0074] S16: Multiply the depth distribution data and the content-aware feature maps to obtain BEV feature maps.

[0075] After obtaining the depth distribution data and the content-aware feature maps, the depth distribution data can provide the depth information of the target object, and the content-aware feature maps can provide the semantic information of the target object. Multiplying the two can obtain the BEV feature maps.

[0076] Step S2: Input the BEV feature map into the detection probe to obtain the predicted target detection box and the predicted target category.

[0077] In some embodiments, after obtaining the BEV feature map, detection is performed on the BEV feature map to obtain the predicted target detection box and the predicted target category.

[0078] Among them, the predicted target detection box is used to frame the target features in the BEV feature map, the predicted target category is used to label the category to which the target features belong, and the predicted target detection box and the predicted target category are associated.

[0079] Specifically, the predicted target detection box is composed of constitute. represents the coordinates of the center position of the predicted target detection box; respectively represent the length, width and height of the predicted target detection box; represents the measured direction angle. The predicted target categories include multiple categories such as "car", "pedestrian", "road sign", "animal", "building", etc.

[0080] Step S3: Crop the BEV feature map according to the source domain target detection box to obtain the target features, and store the target features in the storage body.

[0081] In some embodiments, to improve the generalization ability and accuracy of the 3D target detection model, the target features can be obtained from the BEV feature map through the source domain target detection box associated with the source domain image in the source domain dataset, and the target features are stored in the storage body for subsequent latent space data augmentation.

[0082] It should be noted that the storage body can store the target features of multiple categories in the current training round. Performing latent space data augmentation on all the target features in the storage body can improve the generalization ability of the model.

[0083] Step S4: Perform latent space data augmentation on the target features.

[0084] In some embodiments, after storing the target features in the storage body, the target features in the storage body can be obtained, and latent space data augmentation is performed on the target features through a Gaussian distribution. The formula for constructing the Gaussian distribution for the target features is shown in (1):

[0085] , (1)

[0086] Among them, represents the Gaussian distribution; represents the th target feature of the th BEV feature map; represents the ratio of the current training round to the maximum training round; Represents the covariance of the source domain target class.

[0087] It should be understood that the Gaussian distribution for latent space data augmentation aims to generate a Gaussian distribution based on the within-class covariance for sampling by online computing the within-class feature covariance. Compared with random sampling, this method uses the covariance as prior information to provide a more semantically guiding optimization direction for deep network feature extraction.

[0088] In some embodiments, due to the low accuracy of the initial covariance, is defined as the ratio of the current training epoch to the maximum training epoch to reduce the impact of inaccurate early covariance on the training accuracy of the model.

[0089] Step S5: Calculate the latent space data augmentation loss according to the latent space data augmentation method.

[0090] In some embodiments, to improve the training accuracy of the 3D object detection model, this application calculates three different training losses, and jointly optimizes the model by the three training losses.

[0091] First, calculate the latent space data augmentation loss as the first training loss according to the augmented target features after latent space data augmentation. However, if the Gaussian distribution shown in formula (1) is directly used to explicitly sample each target feature multiple times to obtain multiple augmented target features, the computational cost increases when the number of sampling times is large. This application introduces an easy-to-compute upper bound as the latent space data augmentation loss, which can directly calculate the expectation of the augmented target features, avoiding explicitly generating the augmented target features, thereby improving the computational efficiency.

[0092] Specifically, the formula for calculating the latent space data augmentation loss according to the upper bound calculation is as shown in (2):

[0093]

[0094] , (2)

[0095] where, represents the latent space data augmentation loss; represents the number of BEV feature maps; represents the number of targets in the th BEV feature map; represents the th target feature in the th BEV feature map; and respectively represent the weight matrix and bias of the last fully connected layer of the depth prediction network; ; represents the th column of the weight matrix parameter of the last fully connected layer; represents the number of source domain target classes.

[0096] It should be understood that, as the upper bound of the cross-entropy loss function when the sampling is infinite, the latent space data augmentation loss can impose constraints on the latent space features during minimization, reduce the learning of meaningless semantic features, and implicitly implement data augmentation of the semantic information in the latent space features.

[0097] It should be noted that since the latent space data augmentation loss cannot be calculated through the target features in the first round of iterative training, the latent space data augmentation loss is set to 0 in the first round of iterative training.

[0098] Step S6: Calculate the object-aware contrastive learning loss according to the target features.

[0099] Another training loss is the object-aware contrastive learning loss. In some embodiments, the calculation of the object-aware contrastive learning loss can be implemented according to the target features obtained in step S3.

[0100] It should be understood that since the target features are obtained by cropping the source domain target detection boxes, they can be used as standard data for calculating the training loss.

[0101] Specifically, the formula for calculating the object-aware contrastive learning loss according to the target features is shown in (3):

[0102]

[0103] , (3)

[0104] where, represents the object-aware contrastive learning loss; represents the target features; ; then its positive sample represents the sample with the same label as the sample , and its negative sample represents the sample with a different label from the sample ; represents the set of target positive samples, represents the set of target negative samples; represents the temperature parameter, which is used to control the scaling degree of the similarity in contrastive learning.

[0105] Step S7: Calculate the object detection loss according to the source domain target detection boxes, source domain target classes, predicted target detection boxes, and predicted target classes.

[0106] Another training loss is the object detection loss. In some embodiments, a detection probe is included in the 3D object detection model. The BEV feature map obtained in step S1 is input into the detection probe to obtain predicted object detection boxes and predicted object categories. Using the source domain object detection boxes and source domain object categories corresponding to the source domain images in the source domain dataset as label data, and the predicted object detection boxes and predicted object categories detected by the detection probe as prediction data, the object detection loss can be calculated.

[0107] Specifically, the calculation formula of the object detection loss is shown in (4):

[0108] , (4)

[0109] where, represents the object detection loss; represents the focal loss; represents the smooth L1 loss; represents the source domain object detection box; represents the source domain object category; represents the predicted object detection box; represents the predicted object category.

[0110] Step S8: Add the latent space data augmentation loss, the object-aware contrastive learning loss, and the object detection loss to obtain the model training loss.

[0111] In some embodiments, after calculating the latent space data augmentation loss, the object-aware contrastive learning loss, and the object detection loss in the above steps S5 - S7, add the three training losses to obtain the final loss function.

[0112] Specifically, the calculation formula of the final loss function is shown in (5):

[0113] , (5)

[0114] where, represents the final loss function.

[0115] Step S9: Optimize the 3D object detection model to be trained according to the model training loss to obtain the initial 3D object detection model.

[0116] In some embodiments, optimize the 3D object detection model through the final loss function to improve the generalization ability and accuracy of the 3D object detection model, and obtain the initial 3D object detection model.

[0117] S300: Obtain an unlabeled target domain dataset.

[0118] In some embodiments, due to the obvious cross-domain differences between different data domains, in order to improve the detection ability of the 3D object detection model for the target data domain, it is also necessary to obtain a target domain dataset for training. Since the target domain dataset mostly consists of unlabeled data, the unlabeled target domain dataset can be obtained first, and then the unlabeled target domain dataset can be labeled, and the labeled target domain data can be used for the training of the 3D object detection model.

[0119] Specifically, obtain target domain images as the unlabeled target domain dataset.

[0120] S400: Input the target domain images into the initial 3D object detection model to obtain the target domain class probabilities, where the target domain class probabilities are the probabilities that the target features in the target domain images belong to each category.

[0121] In some embodiments, when labeling the unlabeled target domain data, it is first necessary to obtain the class probabilities of the target features in the target domain images, that is, the target domain class probabilities.

[0122] The method of using the initial 3D object detection model to perform predictions on the target domain images to obtain the target domain class probabilities is the same as the method of the 3D object detection model performing predictions on the source domain images to obtain the predicted target classes in steps S1 - S2, and will not be elaborated here.

[0123] S500: Calculate the acquisition score of the target domain images according to the target domain class probabilities.

[0124] In some embodiments, after obtaining the target domain class probabilities, the acquisition score of the target domain images can be calculated according to the target domain class probabilities, and then the target domain images can be selected for labeling according to the acquisition scores.

[0125] Specifically, the formula for calculating the acquisition score of the target domain images according to the target domain class probabilities is as shown in (6):

[0126] , (6)

[0127] Where, represents the acquisition score; represents the target domain class probabilities; ; represents the number of source domain target classes.

[0128] S600: If the acquisition score meets the preset target, label the target domain images to obtain a labeled target domain dataset.

[0129] In some embodiments, the acquisition scores are judged according to a preset target. If the acquisition score of the target domain image meets the preset target, it indicates that the target domain class probability predicted by the initial 3D target detection model for the target domain image meets the standard, and the target domain image can be labeled to obtain labeled target domain data. Further, multiple groups of target domain images with acquisition scores meeting the preset target are obtained for labeling to generate a labeled target domain data set.

[0130] Specifically, the preset target can be that the acquisition score is greater than or equal to a preset threshold. A preset threshold is set. When the acquisition score calculated from the target domain class probability is greater than or equal to this preset threshold, the target domain image corresponding to the target domain class probability is selected for labeling.

[0131] The preset target can also be that the number of digits of the acquisition scores sorted from high to low is a preset number. According to the acquisition scores calculated from the target domain class probabilities, all target domain images are sorted, and the top images with the highest acquisition scores are selected for labeling.

[0132] Figure 4 This is a schematic diagram of a method for labeling target domain images provided by an embodiment of the present application. As Figure 4 shown, labeling the target domain image includes steps S401 - S404.

[0133] S401: Import the target domain image into a labeling tool.

[0134] Specifically, the target domain image with an acquisition score meeting the preset target is imported into a labeling tool. Tools such as VGG ImageAnnotator (VIA), LabelImg, etc. can be used to label the coordinates of targets in the image.

[0135] S402: Obtain the target to be labeled in the target domain image.

[0136] S403: Draw a bounding box for the target to be labeled.

[0137] Specifically, all the targets to be labeled in the target domain image are selected, and the bounding boxes for the targets to be labeled are drawn respectively. For example, if the target domain image includes targets to be labeled such as "animals" and "road signs", the bounding boxes of "animals" and "road signs" are drawn.

[0138] S404: Save the data information of the bounding box as the target detection box of the target feature in the target domain image.

[0139] Specifically, the data information of each bounding box is saved as the target detection box of the target feature, including and other data.

[0140] In some embodiments, the target categories of each target to be labeled can also be marked, corresponding to the target detection boxes of the target features. Further, the target domain image, the target detection box, and the target category together constitute a labeled target domain data set.

[0141] S700: Train the initial 3D target detection model with the labeled target domain data set to obtain a 3D target detection model.

[0142] In some embodiments, after obtaining the labeled target domain data set, the labeled target domain data set can be used to continue training the initial 3D target detection model that has been trained by the source domain data set, so as to improve the accuracy of the 3D target detection model for target domain detection.

[0143] It should be understood that the method of training the initial 3D target detection model with the labeled target domain data set is the same as the above steps S1 - S9, and will not be elaborated here.

[0144] Some embodiments of the present application also provide a 3D target detection model training device. Figure 5 It is a schematic diagram of a 3D target detection model training device provided by an embodiment of the present application. As Figure 5 shown, the 3D target detection model training device includes:

[0145] A labeled source domain data set acquisition module 501, configured to: acquire a labeled source domain data set, where the labeled source domain data set includes source domain images and label information of the source domain images, and the label information includes source domain target detection boxes and source domain target categories; wherein, the source domain images are images collected by multiple cameras, the source domain target detection boxes are coordinate information of target features in the source domain images, and the source domain target categories are category information of target features in the source domain images; a source domain training module 502, configured to: train the 3D target detection model to be trained with the labeled source domain data set to obtain an initial 3D target detection model; an unlabeled target domain data set acquisition module 503, configured to: acquire an unlabeled target domain data set, where the unlabeled target domain data set includes target domain images, the target domain images are images collected by multiple cameras, and the target domain images are of a different type from the source domain images; a category probability prediction module 504, configured to: input the target domain images into the initial 3D target detection model to obtain target domain category probabilities, where the target domain category probabilities are the probabilities that the target features in the target domain images belong to each category; an acquisition score calculation module 505, configured to: calculate the acquisition score of the target domain images according to the target category probabilities; a labeling module 506, configured to: if the acquisition score meets a preset target, perform labeling on the target domain images to obtain a labeled target domain data set; a target domain training module 507, configured to: train the initial 3D target detection model with the labeled target domain data set to obtain a 3D target detection model.

[0146] Some embodiments of the present application also provide a 3D object detection method, which has high generalization ability and can improve the accuracy of 3D object detection in test images.

[0147] Figure 6 It is a schematic flowchart of a 3D object detection method provided by an embodiment of the present application. As Figure 6 shown, the 3D object detection method includes steps S601 - S606.

[0148] S601: Obtain test set data, where the test set data includes test images, and the test images are images collected by multiple cameras.

[0149] S602: Perform feature extraction on the test images through a feature extractor to obtain feature images.

[0150] S603: Input the feature images into a depth prediction network to obtain depth distribution data.

[0151] S604: Input the feature images into a convolutional network to obtain content - aware feature maps.

[0152] S605: Multiply the depth distribution data and the content - aware feature maps to obtain BEV feature maps.

[0153] S606: Input the BEV feature maps into a detection probe to obtain detection frames and detection categories.

[0154] As can be seen from the above technical solutions, the present application provides a 3D object detection model training method, device, and 3D object detection method. The 3D object detection model training method includes: obtaining a labeled source domain data set, where the labeled source domain data set includes source domain images and label information of the source domain images, and the label information includes source domain object detection frames and source domain object categories; wherein, the source domain images are images collected by multiple cameras, the source domain object detection frames are coordinate information of target features in the source domain images, and the source domain object categories are category information of target features in the source domain images; training a 3D object detection model to be trained through the labeled source domain data set to obtain an initial 3D object detection model; obtaining an unlabeled target domain data set, where the unlabeled target domain data set includes target domain images, the target domain images are images collected by multiple cameras, and the types of the target domain images are different from those of the source domain images; inputting the target domain images into the initial 3D object detection model to obtain target domain category probabilities, where the target domain category probabilities are the probabilities that the target features in the target domain images belong to each category; calculating the acquisition score of the target domain images according to the target domain category probabilities; if the acquisition score meets a preset target, performing annotation on the target domain images to obtain a labeled target domain data set; training the initial 3D object detection model through the labeled target domain data set to obtain a 3D object detection model.

[0155] In the above method, first, an initial 3D object detection model is trained based on the labeled source domain dataset, and then the unlabeled target domain dataset is input into the initial 3D object detection model for prediction to obtain the target domain class probabilities; and the acquisition scores are calculated according to the target domain class probabilities, and the target domain images are selected for annotation according to the acquisition scores, so as to obtain the labeled target domain dataset; finally, the initial 3D object detection model is trained according to the labeled target domain dataset to obtain the 3D object detection model. This method can improve the domain adaptability, generalization ability of the 3D object detection model and the 3D object detection accuracy for target domain images.

[0156] For the similarities between the embodiments provided in this application, reference can be made to each other. The specific embodiments provided above are only several examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation manner extended based on the solution of this application without creative efforts belongs to the protection scope of this application.

Claims

1. A method for training a 3D object detection model, characterized in that, The method includes: Obtain a labeled source domain dataset, where the labeled source domain dataset includes source domain images and label information of the source domain images, and the label information includes source domain object detection frames and source domain object categories; wherein, the source domain images are images collected by multiple cameras, the source domain object detection frames are coordinate information of target features in the source domain images, and the source domain object categories are category information of the target features in the source domain images; Perform preprocessing on the source domain images to obtain a bird's-eye view feature map; Input the bird's-eye view feature map into a detection probe to obtain predicted object detection frames and predicted object categories; Crop the bird's-eye view feature map according to the source domain object detection frames to obtain the target features, and store the target features in a storage body; Perform latent space data augmentation on the target features to obtain sampled features; Determine a computational upper bound according to the sampled features; Calculate the latent space data augmentation loss according to the computational upper bound, and the formula is as follows: ， Among them, represents the latent space data augmentation loss; represents the number of the bird's-eye view feature maps; represents the th target number of the th bird's-eye view feature map; represents the th target feature of the th bird's-eye view feature map; and respectively represent the weight matrix and bias of the last fully-connected layer of the depth prediction network; represents the th column of the weight matrix parameters of the last fully-connected layer; represents the number of source domain target classes; represents the ratio of the current training round to the maximum training round; represents the covariance of the source domain target classes; Calculate the object-aware contrastive learning loss according to the target features; Calculate the object detection loss according to the source domain object detection frames, the source domain object categories, the predicted object detection frames and the predicted object categories; Add the latent space data augmentation loss, the object-aware contrastive learning loss and the object detection loss to obtain the model training loss; Optimize the to-be-trained 3D object detection model according to the model training loss to obtain an initial 3D object detection model; Obtain an unlabeled target domain dataset, where the unlabeled target domain dataset includes target domain images, the target domain images are images collected by the multiple cameras, and the target domain images are of a different type from the source domain images; Input the target domain images into the initial 3D object detection model to obtain target domain category probabilities, where the target domain category probabilities are the probabilities that the target features in the target domain images belong to each category; Calculate the acquisition score of the target domain images according to the target domain category probabilities; If the acquisition score meets a preset target, perform annotation on the target domain images to obtain a labeled target domain dataset; Train the initial 3D object detection model through the labeled target domain dataset to obtain a 3D object detection model.

2. The 3D object detection model training method according to claim 1, wherein, The performing preprocessing on the source domain images to obtain a bird's-eye view feature map includes: Perform cropping on the source domain images to obtain standard source domain images; Perform data augmentation on the standard source domain images to obtain augmented source domain images; Perform feature extraction on the augmented source domain images through a feature extractor to obtain feature images; Input the feature images into a depth prediction network to obtain depth distribution data; Input the feature images into a convolutional network to obtain content-aware feature maps; Multiply the depth distribution data and the content-aware feature maps to obtain the bird's-eye view feature map.

3. The 3D object detection model training method according to claim 1, wherein The cropping the bird's-eye view feature map according to the source domain object detection frames to obtain the target features includes: Crop the bird's-eye view feature map according to the source domain target detection box to obtain the target feature; wherein, the source domain target detection box is associated with the source domain image from which the bird's-eye view feature map is obtained.

4. The 3D object detection model training method according to claim 1, wherein Performing latent space data augmentation on the target feature includes: Obtain the target feature in the storage body; Perform the latent space data augmentation on the target feature through a Gaussian distribution, and the formula is as follows: ， Among them, represents a Gaussian distribution; represents the th target feature of the th bird's-eye view feature map; represents the ratio of the current training round to the maximum training round; represents the covariance of the source domain target class.

5. The 3D object detection model training method according to claim 1, characterized in that The preset target includes: The acquisition score is greater than or equal to a preset threshold, or the number of digits sorted from high to low of the acquisition score is a preset number of digits.

6. The 3D object detection model training method according to claim 1, wherein Performing annotation on the target domain image to obtain a labeled target domain data set includes: Import the target domain image into an annotation tool; Obtain the target to be annotated in the target domain image; Draw a bounding box for the target to be annotated, and save the data information of the bounding box as the target detection box of the target feature in the target domain image.

7. A 3D object detection model training device, characterized in that The device includes: A labeled source domain data set acquisition module, configured to: obtain a labeled source domain data set, the labeled source domain data set includes a source domain image and label information of the source domain image, the label information includes a source domain target detection box and a source domain target category; wherein, the source domain image is an image collected by multiple cameras, the source domain target detection box is the coordinate information of the target feature in the source domain image, and the source domain target category is the category information of the target feature in the source domain image; A source domain training module, configured to: perform preprocessing on the source domain image to obtain a bird's-eye view feature map; Input the bird's-eye view feature map into a detection probe to obtain a predicted target detection box and a predicted target category; Crop the bird's-eye view feature map according to the source domain target detection box to obtain the target feature, and store the target feature in a storage body; Perform latent space data augmentation on the target feature to obtain a sampled feature; Determine a calculation upper bound according to the sampled feature; Calculate the latent space data augmentation loss according to the calculation upper bound, and the formula is as follows: ， Among them, represents the latent space data augmentation loss; represents the number of the bird's-eye view feature maps; represents the th target number of the th bird's-eye view feature map; represents the th target feature of the and respectively represent the weight matrix and bias of the last fully-connected layer of the depth prediction network; ; represents the th column of the weight matrix parameter of the last fully-connected layer; represents the number of source domain target classes; represents the ratio of the current training round to the maximum training round; represents the covariance of the source domain target classes; Calculate the object-aware contrastive learning loss according to the target feature; Calculate the target detection loss according to the source domain target detection box, the source domain target category, the predicted target detection box and the predicted target category; Add the latent space data augmentation loss, the object-aware contrastive learning loss and the target detection loss to obtain a model training loss; Optimize the to-be-trained 3D target detection model according to the model training loss to obtain an initial 3D target detection model; An unlabeled target domain data set acquisition module, configured to: obtain an unlabeled target domain data set, the unlabeled target domain data set includes a target domain image, the target domain image is an image collected by multiple cameras, and the target domain image is of a different type from the source domain image; A class probability prediction module, configured to: input the target domain image into the initial 3D target detection model to obtain a target domain class probability, and the target domain class probability is the probability that the target feature in the target domain image belongs to each class; An acquisition score calculation module, configured to: calculate the acquisition score of the target domain image according to the target class probability; The annotation module is configured to: if the acquisition score meets the preset target, perform annotation on the target domain image to obtain a labeled target domain dataset; The target domain training module is configured to: train the initial 3D object detection model through the labeled target domain dataset to obtain a 3D object detection model.

8. A 3D object detection method, which is executed by a 3D object detection model trained by the 3D object detection model training method according to any one of claims 1-6, characterized in that, The method includes: Obtain test set data, where the test set data includes test images, and the test images are images collected by multiple cameras; Perform feature extraction on the test images through a feature extractor to obtain feature images; Input the feature images into a depth prediction network to obtain depth distribution data; Input the feature images into a convolutional network to obtain content-aware feature maps; Multiply the depth distribution data and the content-aware feature maps to obtain bird's-eye view feature maps; Input the bird's-eye view feature maps into a detection probe to obtain detection boxes and detection categories.

Images