Training method and device of target detection model and monocular detection method and device

Abstract

The present disclosure belongs to the field of computer vision, and particularly relates to a target detection model training method and device and a monocular 3D detection method and device. The target detection model training method comprises the following steps: acquiring time and space synchronous image data and point cloud data in a preset space, constructing a training data set for target detection by taking the image data as a sample, generating auxiliary training data based on the point cloud data, and adding the auxiliary training data to the training data set; constructing a target detection neural network, the target detection neural network at least comprising a 3D branch for performing 3D target detection, and a space embedding branch for mapping image features to an implicit coding space to generate an image feature vector; and performing joint training on the target detection neural network based on the training data set, wherein the training of the space embedding branch comprises adjusting parameters of the target detection neural network to optimize a voxel occupancy prediction loss.

Description

Technical Field

[0001] This disclosure pertains to the field of computer vision, specifically relating to training methods and apparatus for object detection models, and methods and apparatus for monocular 3D detection. Background Technology

[0002] Monocular cameras are widely used in autonomous driving perception systems due to their low cost and flexible deployment. However, monocular cameras lack direct depth measurement, making it difficult to accurately reconstruct the three-dimensional geometry of a scene.

[0003] Existing monocular 3D detection methods are either based on depth estimation, i.e., first predicting pixel-level depth maps and then performing 3D detection, or directly regressing 3D bounding box parameters from 2D images. The former relies on depth estimation algorithms whose accuracy is difficult to guarantee, while the latter requires complex 3D modeling, which is computationally intensive. Summary of the Invention

[0004] This disclosure proposes a monocular 3D detection scheme to address the problem that existing monocular 3D detection methods are either inaccurate or computationally complex due to the lack of direct depth information.

[0005] A first aspect of this disclosure provides a method for training an object detection model, comprising:

[0006] Acquire spatiotemporally synchronized image data and point cloud data within a preset space, construct a training dataset for target detection using the image data as samples, generate auxiliary training data based on the point cloud data, and add it to the training dataset;

[0007] Construct an object detection neural network, the object detection neural network including at least a 3D branch for performing 3D object detection, and a spatial embedding branch for mapping image features to an implicit coding space to generate image feature vectors;

[0008] The object detection neural network is jointly trained based on the training dataset, wherein the training of the spatial embedding branch includes adjusting the parameters of the object detection neural network to optimize the voxel occupancy prediction loss.

[0009] In some embodiments, the backbone network of the target detection neural network is based on a DLA structure.

[0010] In some embodiments, generating auxiliary training data based on the point cloud data includes:

[0011] The first point cloud data is filtered by field of view and light sampling, and outliers are filtered by KNN combined with KD tree to obtain the second point cloud data;

[0012] The auxiliary training data is composed of voxels obtained by voxelizing the second point cloud data as samples and the voxel's occupancy attribute as a label.

[0013] In some embodiments, the field-of-view filtering and ray sampling of the first point cloud data includes:

[0014] In the first point cloud data, the point cloud that falls within the camera's field of view is retained;

[0015] For the point cloud falling within the camera's field of view, points are randomly sampled along the camera's imaging path to generate positive samples, and then points are randomly sampled within a preset range along the line connecting the camera's optical center to the positive samples to generate negative samples. The distance between the positive samples and the negative samples is not less than a first preset threshold. The positive samples refer to voxels containing more than a second preset threshold, and the negative samples refer to voxels containing no more than the second preset threshold.

[0016] In some embodiments, the step of filtering outliers based on KNN combined with KD-tree to obtain the second point cloud data includes:

[0017] Construct a KD tree for the point cloud. For each first point in the point cloud, query a preset number of adjacent second points within a preset distance around the first point and calculate the average distance from the first point to the second point.

[0018] The filtering threshold is determined based on the average distance of all the first points in the point cloud;

[0019] In the point cloud, points whose average distance is less than the filtering threshold are retained to generate the second point cloud data.

[0020] In some embodiments, adjusting the parameters of the target detection neural network to optimize the voxel occupancy prediction loss includes:

[0021] The image feature vector generated by the spatial embedding branch is compared with the voxel position vector generated by Fourier encoding the positions of the samples in the auxiliary training data to generate Focal Loss. Based on the Focal Loss result, the parameters of the object detection neural network are adjusted to optimize the voxel occupancy prediction loss of the spatial embedding branch, wherein:

[0022] FL(p t )=-α t (1-p t ) γ log(p t ),

[0023] Where FL stands for Focal Loss.

[0024] p t It is the probability that the model predicts correctly for each class; for positive samples, p t =p, for negative samples, p t = 1-p, where p is the probability of the model output;

[0025] α t It is a balancing parameter used to mitigate the effects of class imbalance. For the positive samples, α t =α, for the negative sample, α t =1-α.

[0026] A second aspect of this disclosure provides a monocular 3D detection method, including:

[0027] Acquire image data of the target area;

[0028] The image data is input into a monocular 3D detection model, and the 3D detection result of the target region is output. The monocular 3D detection model is composed of a target detection model trained according to the method of the first aspect of this disclosure, with the spatial embedding branch removed.

[0029] A third aspect of this disclosure provides a training apparatus for an object detection model, comprising:

[0030] The data processing module is used to acquire spatiotemporally synchronized image data and point cloud data within a preset space, construct a training dataset for target detection using the image data as samples, generate auxiliary training data based on the point cloud data, and add it to the training dataset.

[0031] A model building module is used to build an object detection neural network, which includes at least a 3D branch for performing 3D object detection and a spatial embedding branch for mapping image features to an implicit coding space.

[0032] The model training module is used to jointly train the object detection neural network based on the training dataset, wherein the training of the spatial embedding branch includes adjusting the parameters of the object detection neural network to optimize the voxel occupancy prediction loss.

[0033] A fourth aspect of this disclosure provides a monocular 3D detection device, comprising:

[0034] The data acquisition module is used to acquire image data of the target area;

[0035] A 3D detection module is used to input the image data into a monocular 3D detection model and output the 3D detection result of the target region, wherein the monocular 3D detection model is composed of a target detection model trained according to any one of the methods of the first aspect of this disclosure, with the spatial embedding branch removed.

[0036] The target detection model training method and apparatus, and monocular 3D detection method and apparatus proposed in this disclosure train the 3D target recognition model using the main task, while simultaneously training the 3D spatial structure of the scene using an auxiliary task. Because the model can utilize more spatial structure information during the learning process by implicitly encoding spatial information, it can infer the depth of instances using more contextual information and improve the accuracy of 3D target recognition based on joint training and depth information. At the same time, when the model performs inference, auxiliary branches can be removed, thus allowing the model to improve detection accuracy without increasing computational complexity. Attached Figure Description

[0037] The features and advantages of this disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the scope of this disclosure in any way.

[0038] Figure 1 This is a schematic diagram of a computer system to which this disclosure applies;

[0039] Figure 2 This is a flowchart illustrating a training method for an object detection model according to some embodiments of the present disclosure;

[0040] Figure 3 yes Figure 2 A schematic diagram of the target detection model shown;

[0041] Figure 4 This is a flowchart illustrating a monocular 3D detection method according to some embodiments of the present disclosure;

[0042] Figure 5 It is based on and Figure 4 The method shown is validated on the KITTI dataset, showing the coordinate distribution of voxels.

[0043] Figure 6 yes Figure 4 A comparison of the detection accuracy and time required with other models using the described method;

[0044] Figure 7 This is a schematic diagram of a training apparatus for an object detection model according to some embodiments of the present disclosure;

[0045] Figure 8 This is a schematic diagram of a monocular 3D detection device according to some embodiments of the present disclosure. Detailed Implementation

[0046] In the following detailed description, numerous specific details of this disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this disclosure are a method of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if they can achieve the same purpose.

[0047] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to, or communicate with, other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used in this disclosure includes any one and all combinations of one or more of the associated listed items.

[0048] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As shown in this specification and claims, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified features, integrals, steps, operations, elements, and / or components, and such expressions do not constitute an exclusive list, in which other features, integrals, steps, operations, elements, and / or components may also be included.

[0049] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this disclosure can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this disclosure. It is understood that the drawings are not drawn to scale.

[0050] Various structural diagrams are used in this disclosure to illustrate various variations of embodiments according to this disclosure. It should be understood that the preceding or following structures are not intended to limit this disclosure. The scope of protection of this disclosure is defined by the claims.

[0051] Figure 1 This is a schematic diagram of a computer system applicable to this disclosure. Figure 1 The system shown includes a monocular image acquisition device for acquiring image data of a target area, and a 3D target detection server for performing 3D target detection based on the image data. Wherein:

[0052] The monocular image acquisition device is used to acquire image data of a target area, which generally lacks depth information. The monocular image acquisition device can be a camera that acquires a single frame of image data, or a camera that continuously acquires multiple frames of image data. The monocular image acquisition device can be a standalone device or embedded in other devices, specifically, an image acquisition module embedded in a 3D target detection server.

[0053] The 3D target detection server is deployed with a monocular 3D detection model. The monocular 3D detection model takes the data to be detected as input and performs 3D target detection on the target area. The 3D target detection server can be any of a single machine, a cluster, or a distributed server.

[0054] Figure 2 This is a flowchart illustrating a training method for an object detection model according to some embodiments of the present disclosure. In some embodiments, the training method for the object detection model is executed by a model training server, and the training method for the object detection model includes the following steps:

[0055] S210: Acquire spatiotemporally synchronized image data and point cloud data within a preset space; construct a training dataset for target detection using the image data as samples; generate auxiliary training data based on the point cloud data; and add the auxiliary training data to the training dataset.

[0056] Specifically, in one embodiment of this disclosure, training data is collected by a vehicle equipped with an onboard camera and an onboard LiDAR, wherein the onboard camera and the onboard LiDAR need to be spatiotemporally synchronized through hard synchronization or soft synchronization. After data collection, the 2D and 3D attribute information of each target (such as pedestrians, vehicles, bicycles, etc.) in the images is manually labeled. Here, it is formally denoted as X. img X LiDAR Y 2D Y 3D X img X represents the image captured by the vehicle's onboard camera. LiDAR This represents the corresponding LiDAR point cloud data. Y 2D It is a collection of 2D attributes of the target object in the image, including the 2D bounding box, pixel coordinates, and corresponding category, Y. 3D It is a collection of 3D attributes of the target object in the image, including the 3D bounding box, the target's 3D position, the target's 3D size, depth, and yaw angle. The X value... img and Y 2D Y 3D The samples and labels are used respectively to form the main training data.

[0057] In addition to basic image feature extraction, the training samples disclosed herein require additional samples and labels for spatial embedding training. The preparation of samples and labels can be formally represented as follows: Let the original point cloud dataset be X. LiDAR The point cloud is filtered using a calibration matrix C and a ray sampling function L to obtain the filtered point cloud P′. The filtered point cloud P′ is then voxened into a voxel mesh V, where the voxenening function is denoted as Vox. The above steps can be expressed as follows:

[0058] X voxel =Vox(L(C(X) LiDAR )))

[0059] When the number of point clouds within a voxel exceeds a preset threshold, the voxel is occupied and marked as a positive sample, denoted as Y. occupied The remaining samples were not occupied and were marked as negative samples, denoted as C. U Y occupied .

[0060] Y occupied ={(x voxel y voxel , z voxe1 )| count(x voxel y voxe1 , z voxel )≥threshold}

[0061] Using voxels obtained by voxelizing the filtered original point cloud as samples and the voxel occupancy attribute as labels, auxiliary training data is formed, which together with the main training data forms the training dataset.

[0062] In some embodiments of this disclosure, the original point cloud dataset X is... LiDAR Filtering methods include:

[0063] LiDAR filtering (implementation in C)

[0064] FOV (Field of View) filtering in camera point cloud data processing refers to the process of retaining only the portion of the point cloud that falls within the camera's field of view. Due to the limited field of view of a camera, not all points acquired from a LiDAR sensor will be captured by the camera. Therefore, FOV filtering is typically performed when integrating LiDAR data into camera data to ensure data consistency and availability. This can be achieved using a camera calibration matrix.

[0065] The camera calibration matrix is ​​typically used to describe the mapping relationship between the three-dimensional world coordinate system and the two-dimensional image coordinate system. It consists of two parts: the intrinsic matrix and the extrinsic matrix.

[0066] The steps to transform the point cloud from the LiDAR coordinate system to the camera coordinate system and then map it onto the image plane are as follows:

[0067] 1. Use an extrinsic parameter matrix to transform point cloud data from the LiDAR coordinate system to the camera coordinate system:

[0068]

[0069] 2. Use an intrinsic parameter matrix to transform points in the camera coordinate system to the image plane:

[0070]

[0071] 3. Convert the obtained homogeneous image coordinates p to non-homogeneous coordinates:

[0072]

[0073] 4. Filter according to image size, keeping only points that fall inside the image: if u≥0, v≥0, u<image width, v<image height, P′z>0, then keep the point.

[0074] After the above processing steps, data filtered by the camera's field of view (FOV) was obtained.

[0075] Ray sampling (implementation of L)

[0076] In global random sampling, samples may be overly sensitive to outliers in the dataset, and this sampling method cannot guarantee truly uniform sampling in space. This is mainly because global random sampling does not fully consider the spatial distribution of the data, resulting in a sample distribution that is inconsistent with the actual positive sample distribution. This inconsistency prevents the model from effectively learning spatially uniformly distributed features, thus affecting the model's understanding and predictive ability of the entire detection space.

[0077] To improve this, this study employs a ray tracing-based sampling method. This method first samples positive samples along the camera's imaging path, then randomly samples negative samples within a certain range along the path connecting the camera's optical center to the positive sample points. This helps simulate realistic spatial distribution and object occlusion relationships. This sequential sampling strategy also helps avoid misclassifying object regions that are actually occluded due to viewpoint issues as negative sample regions. To further improve model training efficiency and prevent overfitting, a minimum interval α between positive and negative samples is set during the sampling process. This parameter controls the learning difficulty of the model and avoids misclassification of occluded regions. In this way, the sampled samples more closely match the actual spatial and object distribution. The sample distribution obtained through ray sampling is more uniform and can better cover the entire detection space, thereby improving the model's generalization ability and detection performance.

[0078] Vox and KNN filtering (implementation of Vox)

[0079] The impact of outliers is a significant issue during voxelization. Outliers, those points that differ significantly from the majority of data points, can negatively affect the sampling process, impacting the overall effectiveness and efficiency of the data processing workflow. Particularly in the case of global average negative sampling, these outliers may be classified as hard negatives due to their high prediction difficulty. This not only weakens the model's learning efficiency for other normal data points but may also negatively affect the model's predictive performance.

[0080] Furthermore, if the sampling strategy is based on methods such as ray tracing, the presence of outliers may lead to uneven sampling distribution. This uneven sampling distribution may cause the point cloud in some voxel regions to become too sparse, while in other regions the point cloud may become too dense, further degrading the quality of the voxelized data and hindering subsequent data processing and model training.

[0081] To address the aforementioned issues, this invention implements an outlier filtering algorithm based on KNN and KD-trees. This method utilizes KD-trees to accelerate KNN search, effectively identifying and removing outliers, thereby improving the overall quality of the data and providing a more reliable data foundation for subsequent data processing and model learning.

[0082] Specifically, the algorithm's steps include: first, constructing a KD tree for the given dataset to improve the efficiency of KNN search; then, for each point in the dataset, querying its k nearest neighbors and calculating the average distance to these neighbors; next, determining a filtering threshold based on the average and standard deviation of the average nearest neighbor distances of all points; and finally, selecting those points whose average nearest neighbor distance is less than the threshold to form the filtered dataset.

[0083] In summary, using global average sampling during training sample construction can lead to a severe imbalance between positive and negative samples. Furthermore, because the samples are constructed from LiDAR point clouds, positive samples obtained through global average sampling always contain 3D entities, while negative samples may not originate from 3D entities; they could be caused by occlusion from nearby objects. Therefore, ray sampling, which samples from both the LiDAR and camera optical centers, alleviates the imbalance and improves sample accuracy, preventing the introduction of more noise into the model. Using KNN to filter outoutpoints allows for flexibility by adjusting the sensitivity of K to select the appropriate level of sensitivity for outoutpoint filtering. KD-trees, on the other hand, help the algorithm find K-neighbors more quickly by establishing a tree structure.

[0084] S220, Construct an object detection neural network, the object detection neural network including at least a 3D branch for performing 3D object detection and a spatial embedding branch for mapping image features to an implicit coding space to generate image feature vectors.

[0085] In some embodiments of this disclosure, Figure 3 yes Figure 2 The diagram shows a target detection model. Figure 3 As shown, the backbone network of the model is based on the DLA structure. DeepLayer Aggregation (DLA) is an advanced architecture for deep learning, specifically designed to improve the information aggregation process in visual recognition tasks. DLA enhances the capabilities of the base network through a more efficient information fusion method, iteratively fusing feature information from different layers to achieve higher accuracy and fewer parameters. With the same number of layers and convolutional blocks, DLA achieves higher accuracy with fewer parameters compared to ResNet and ResNeXt. Both DLA and DenseNet employ feature reuse strategies, but DLA, through its unique aggregation method, has an advantage in memory efficiency while maintaining or exceeding the performance of DenseNet.

[0086] The working principle of the DLA network can be viewed as a series of feature iterations and hierarchical aggregation processes, which helps the network effectively combine deep and shallow information, thereby improving the overall performance of the model.

[0087] The image features extracted by the backbone network are input into multiple branches to perform multiple prediction tasks, with each branch responsible for a specific optimization objective. The tasks include the following:

[0088] 1. Heatmap: A heatmap used for class prediction in object detection, where the number of channels represents the number of classes to be identified.

[0089] 2. 2D offset (offset_2d): Used to precisely locate the position of an object in 2D space. It usually has two channels to represent the offset in the x and y directions respectively.

[0090] 3. Two-dimensional dimension (size_2d): Used to estimate the size of an object in two-dimensional space, including two channels: width and height.

[0091] 4. Depth: Used to estimate the distance from the object to the camera.

[0092] 5. 3D offset (offset_3d): Used to position an object in 3D space.

[0093] 6. 3D size (size_3d): Used to estimate the size of an object in three-dimensional space, typically including length, width, and height.

[0094] 7. Heading: Used to estimate the orientation of an object. Different orientation categories can be represented by a 24-channel vector.

[0095] Building upon this, a Space Embedding task was added to map image features to an implicit coding space to generate image feature vectors and to predict voxel occupancy based on these feature vectors.

[0096] This disclosure proposes an efficient "spatial embedding module" designed to map the features obtained from an image through a feature extractor into a vector space, thereby enhancing the model's ability to represent image content. The design of this module includes the following key steps:

[0097] First, the module performs convolution operations using 1x1 convolution kernels. This convolution operation does not change the spatial size of the feature map, but rather integrates the information from each input channel through linear transformation, achieving feature fusion between channels and thus improving the expressive power of the features. Next, the Flatten operation is used to flatten the convolutional feature map, transforming the multidimensional feature array into a one-dimensional vector. This step is to convert the spatial features into a form that can be processed by fully connected layers, facilitating subsequent dense feature generation.

[0098] The module then performs pooling on the flattened features to reduce feature dimensionality and enhance the model's generalization ability. Pooling helps to abstract and extract effective information while reducing computation and preventing overfitting.

[0099] Finally, the sparse features are further encoded using a neural network (NN) layer, transforming them into a denser feature representation. This step involves learning the complex relationships between different features to obtain an embedding vector that can effectively describe the input image.

[0100] To achieve the objectives of this invention, the network structure of the model includes at least a 3D branch for performing 3D object detection and a spatial embedding branch for mapping image features to an implicit coding space to generate image feature vectors for voxel occupancy prediction.

[0101] The spatial embedding branch also includes a module for encoding voxel position features to generate voxel position vectors and for interacting between image feature vectors and voxel position vectors. In the feature extraction stage, the projection function F is used... proj and F′ proj This allows two types of feature vectors (image feature vectors and voxel position vectors) to interact within the same feature space to obtain a loss function for voxel occupancy prediction. This step lays the foundation for subsequent feature fusion and learning tasks. The projection function used in this disclosure is the identity function.

[0102] In the projection function F proj and F′ proj In its design, this disclosure deliberately maintains a simple structure, allowing the model to focus more on constructing smooth spatial structures during learning. This design philosophy aims to encourage the model to learn and infer spatial structures through simple geometric assumptions, thereby improving the model's performance in utilizing consistency and spatial reasoning capabilities.

[0103] Some embodiments of this disclosure employ Fourier encoding to obtain voxel position vectors, referred to as FFTcast. The purpose is to encode the spatial position information of voxels through Fourier transform and to arrange the complex representation of spatial position information at the end of the model for learning, thereby making the network tend to learn a simple and smooth spatial structure during the learning process. Specifically, the basic form of position encoding is as follows:

[0104] γ(p)=(sin(2 0 πp), cos(2 0 πp), ..., sin(2 L-1 πp), cos(2 L-1 πp)).

[0105] Here, p represents the position vector of the original input, and L is the level of the positional encoding, which determines the length and hierarchical details of the encoding. γ(p) maps p to a high-dimensional space through a series of sine and cosine functions, where each pair of sine and cosine functions corresponds to a different frequency, with frequencies increasing in increments of 2. i The form of π increases.

[0106] For 3D coordinates, this disclosure uses a frequency of L=1024 for encoding. Although the new mapping cannot guarantee that the input space is full-rank or completely distributed, it means that each spatial dimension will be mapped to a combination of a large number of sine and cosine functions. While such a mapping cannot guarantee that the mapped space is completely full-rank or that each dimension is completely independent, this high-dimensional representation does significantly improve the network's information representation ability and discriminative power.

[0107] S230, the object detection neural network is jointly trained based on the training dataset, wherein the training of the spatial embedding branch includes adjusting the parameters of the object detection neural network to optimize the voxel occupancy prediction loss.

[0108] The object detection model is jointly trained using the training data prepared by S210. Specifically, based on X... img X voxel For multi-task joint training, the loss function is:

[0109] L total (x, y) = L seg (x, y) + L offset2d (x, y) + L size2d (x, y)

[0110] +L offset3d (x, y) + L depth (x, y)

[0111] +L size3d (x, y) + L heading (x, y) + L voxel (x, y)

[0112] Where Ltotal is the overall loss, Lseg is the classification loss, Loffset2d is the 2D offset loss, Lsize2d is the 2D size loss, Loffset3d is the 3D offset loss, Ldepth is the depth loss, Lsize3d is the 3D size loss, Lheading is the heading loss, and Lvoxel is the voxel occupancy prediction loss.

[0113] Because positive samples are very sparse in the sample space of spatial embedding tasks, the model faces a severe imbalance problem. This imbalance causes the model to overfit to the majority class (usually negative samples) during training, while neglecting the correct classification of the minority class (i.e., positive samples), thus reducing the model's generalization ability and performance in practical applications. To alleviate the problem of imbalance, this disclosure performs negative sampling, selectively sampling a portion of the negative samples to balance the training data. Furthermore, this disclosure introduces Focal Loss, enabling the model to focus on hard-to-classify samples to further improve the imbalance problem.

[0114] This disclosure obtains a voxel occupancy prediction loss function by comparing the image feature vector generated by the spatial embedding branch with the voxel position vector. In some embodiments of this disclosure, the voxel occupancy prediction loss function is Focal Loss.

[0115] Focal Loss is a modification of the cross-entropy loss function, and its formula is as follows:

[0116] FL(p t )=-α t (1-p t ) γ log(p t )

[0117] in:

[0118] p t It is the probability that the model predicts correctly for each class; for positive samples, p t =p, for negative samples, p t =1-p, where p is the probability of the model output.

[0119] α t It is a balancing parameter used to mitigate the effects of class imbalance. For positive samples, α t =α, for negative samples, α t =1-α.

[0120] γ is an exponent that adjusts the focus parameter, used to reduce the contribution of easily classified samples to the loss, making the model pay more attention to those difficult-to-classify samples.

[0121] This disclosure improves the model's understanding of 3D information in images by training a 3D object recognition model using a spatial embedding task as an auxiliary training method, while simultaneously training the model using the main task. By implicitly encoding spatial information, the model can utilize more spatial structure information during the learning process, allowing it to infer the depth of instances using more contextual information. Based on joint training, it leverages depth information to improve the accuracy of 3D object recognition. The spatial embedding task maintains the implicit encoding of space in the original 3D object recognition task. Directly predicting 3D attributes or depth implicitly encodes the 3D information in the image for 3D object detection. If explicit encoding methods are used, such as having the model predict multiple object planes, it will lead to inconsistencies in the spatial encoding representation between tasks, resulting in negative transfer. During training, neural networks typically first fit the low-frequency components of the data. Since the model needs to learn the consistency of image point clouds, this requires the model not only to predict accurately at a single location but also to ensure consistency across multiple locations.

[0122] Figure 4 This is a flowchart illustrating a monocular 3D detection method according to some embodiments of the present disclosure. In some embodiments, the monocular 3D detection method is executed by a 3D object detection server, and the monocular 3D detection method includes the following steps:

[0123] S410, acquire image data of the target area.

[0124] S420, the image data is input into a monocular 3D detection model, and the 3D detection result of the target region is output, wherein the monocular 3D detection model is based on... Figure 2 The object detection model trained by the method removes the spatial embedding branch.

[0125] When performing 3D object detection on input image data based on an object detection model Figure 2 The target detection model only needs to retain the 3D branch used to perform 3D target detection, while removing the spatial embedding branch and the corresponding voxel position encoding module.

[0126] This disclosure validates the method to be protected on the KITTI dataset. The KITTI dataset is an important evaluation dataset for computer vision algorithms widely used in the field of autonomous driving. For 3D object detection, the KITTI dataset provides 7481 training images and 7518 test images, along with corresponding point cloud data. This part of the dataset contains a total of 80256 labeled objects.

[0127] In the validation, the training data size was set to 48, the test data batch size was set to 16, data augmentation was performed through random pruning and flipping with a probability of 0.5, the maximum training period was set to 140 epochs, the optimizer used was Adam, the initial learning rate was set to 0.00125, the weight decay coefficient was set to {10}^{-5}, warmup was used, and the learning rate was reduced at the 90th and 120th epochs with a learning rate decay ratio of 0.1.

[0128] Figure 5 The distribution of all data along the x, y, and z dimensions is shown. The left plot represents the distribution of the minimum values, and the right plot represents the distribution of the maximum values. By observing these distributions, a suitable origin and reasonable partitioning of the detection space can be determined. Since voxels need to be segmented, the space needs to be pre-divided. The validation process statistically analyzed and visualized the coordinate distribution of the voxels, such as... Figure 5 As shown, x is roughly distributed between [-50, 50], y is roughly distributed between [-4, 10], and z is roughly distributed between [0, 80]. In this verification, the size of the spatial voxelization is defined as (200, 30, 160) and the Voxelsize is set to 0.5.

[0129] The trained model was compared with several other mature models in terms of accuracy and runtime for 3D and BEV (bird's-eye view) object detection within the Car category. The results are as follows: Figure 6 It is evident that, under various modes, the method protected by this disclosure exhibits superior detection accuracy while also demonstrating a significant advantage in runtime. This indicates that the method protected by this disclosure improves detection accuracy without increasing computational complexity.

[0130] Figure 7 This is a schematic diagram of a training apparatus for an object detection model according to some embodiments of the present disclosure. Figure 7 As shown, the training device 700 for the object detection model includes a data processing module 710, a model building module 720, and a model training module 730. In some embodiments of this disclosure, the training function of the object detection model can be executed by a model training server. Wherein:

[0131] The data processing module 710 is used to acquire spatiotemporally synchronized image data and point cloud data within a preset space, construct a training dataset for target detection using the image data as samples, generate auxiliary training data based on the point cloud data, and add it to the training dataset.

[0132] The model building module 720 is used to build an object detection neural network, which includes at least a 3D branch for performing 3D object detection and a spatial embedding branch for mapping image features to an implicit coding space to generate image feature vectors.

[0133] The model training module 730 is used to jointly train the object detection neural network based on the training dataset, wherein the training of the spatial embedding branch includes adjusting the parameters of the object detection neural network to optimize the voxel occupancy prediction loss.

[0134] Figure 8 This is a schematic diagram of a monocular 3D detection device according to some embodiments of the present disclosure. Figure 8 As shown, the monocular 3D detection device 800 includes a data acquisition module 810 and a 3D detection module 820. In some embodiments of this disclosure, the monocular 3D detection function is performed by a 3D target detection server. Wherein:

[0135] Data acquisition module 810 is used to acquire image data of the target area;

[0136] 3D detection module 820 is used to input the image data into a monocular 3D detection model and output the 3D detection result of the target region, wherein the monocular 3D detection model is based on... Figure 2 The object detection model trained by the method removes the spatial embedding branch.

[0137] In summary, the target detection model training method and apparatus, and the monocular 3D detection method and apparatus proposed in this disclosure train the 3D target recognition model using the main task while simultaneously training the 3D spatial structure of the scene using an auxiliary task. Because spatial information is implicitly encoded, the model can utilize more spatial structure information during the learning process, thereby enabling the model to infer the depth of instances using more contextual information and improve the accuracy of 3D target recognition based on joint training using depth information. At the same time, when the model performs inference, auxiliary branches can be removed, thus allowing the model to improve detection accuracy without increasing computational complexity.

[0138] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.

[0139] Although the subject matter described herein is provided in the general context of execution on a computer system in conjunction with an operating system and applications, those skilled in the art will recognize that other implementations can also be executed in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.

[0140] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0141] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this disclosure and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this disclosure should be included within the protection scope of this disclosure. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A training method for an object detection model, characterized in that, include: Acquire spatiotemporally synchronized image data and point cloud data within a preset space, construct a training dataset for target detection using the image data as samples, generate auxiliary training data based on the point cloud data, and add it to the training dataset; A target detection neural network is constructed, which includes at least a 3D branch for performing 3D target detection and a spatial embedding branch for mapping image features to an implicit coding space to generate image feature vectors. The spatial embedding branch includes: a feature processing module for sequentially performing 1×1 convolution, Flatten flattening, pooling, and neural network encoding on the image features extracted by the backbone network; a position encoding module for performing Fourier encoding on voxel positions to generate voxel position vectors; and a loss calculation module for comparing the image feature vectors with the voxel position vectors to generate voxel occupancy prediction loss. The object detection neural network is jointly trained based on the training dataset, wherein the training of the spatial embedding branch includes adjusting the parameters of the object detection neural network to optimize the voxel occupancy prediction loss, and the object detection model for monocular 3D detection is obtained by removing the spatial embedding branch from the object detection neural network obtained by the joint training.

2. The method according to claim 1, characterized in that, The backbone network of the target detection neural network is based on the DLA structure.

3. The method according to claim 1, characterized in that, The generation of auxiliary training data based on the point cloud data includes: The first point cloud data is filtered by field of view and light sampling, and outliers are filtered by KNN combined with KD tree to obtain the second point cloud data; The auxiliary training data is composed of voxels obtained by voxelizing the second point cloud data as samples and the voxel's occupancy attribute as a label.

4. The method according to claim 3, characterized in that, The process of performing field-of-view filtering and light sampling on the first point cloud data includes: In the first point cloud data, the point cloud that falls within the camera's field of view is retained; For the point cloud falling within the camera's field of view, points are randomly sampled along the camera's imaging path to generate positive samples, and then points are randomly sampled within a preset range along the line connecting the camera's optical center to the positive samples to generate negative samples. The distance between the positive samples and the negative samples is not less than a first preset threshold. Here, the positive samples refer to voxels containing more than a second preset threshold, and the negative samples refer to voxels containing no more than the second preset threshold.

5. The method according to claim 3, characterized in that, The second point cloud data obtained by filtering outout points based on KNN combined with KD tree includes: Construct a KD tree for the point cloud. For each first point in the point cloud, query a preset number of adjacent second points within a preset distance around the first point and calculate the average distance from the first point to the second point. The filtering threshold is determined based on the average distance of all the first points in the point cloud; In the point cloud, points whose average distance is less than the filtering threshold are retained to generate the second point cloud data.

6. The method according to claim 1, characterized in that, The adjustment of the parameters of the target detection neural network to optimize the voxel occupancy prediction loss includes: The image feature vector generated by the spatial embedding branch is compared with the voxel position vector generated by Fourier encoding the positions of the samples in the auxiliary training data to generate Focal Loss. Based on the Focal Loss result, the parameters of the object detection neural network are adjusted to optimize the voxel occupancy prediction loss of the spatial embedding branch, wherein: ， Where FL stands for Focal Loss. It is the probability that the model correctly predicts each class; for positive samples, For negative samples, ,in It is the probability output by the model; It is a balancing parameter used to mitigate the effects of class imbalance. For the positive samples, For the negative sample, .

7. The method according to claim 6, characterized in that, The Fourier encoding of the location of the sample includes: ， in, This represents the position vector of the original input. It is the level of the positional encoding, L=1024.

8. A monocular 3D detection method, characterized in that, include: Acquire image data of the target area; The image data is input into a monocular 3D detection model, which outputs a 3D detection result for the target region. The monocular 3D detection model is composed of the target detection neural network trained by the method according to any one of claims 1-7, with the spatial embedding branch removed.

9. A training device for an object detection model, characterized in that, include: The data processing module is used to acquire spatiotemporally synchronized image data and point cloud data within a preset space, construct a training dataset for target detection using the image data as samples, generate auxiliary training data based on the point cloud data, and add it to the training dataset. A model building module is used to build an object detection neural network. The object detection neural network includes at least a 3D branch for performing 3D object detection and a spatial embedding branch for mapping image features to an implicit coding space. The spatial embedding branch includes: a feature processing module for sequentially performing 1×1 convolution, Flatten flattening, pooling, and neural network encoding on the image features extracted by the backbone network; a position encoding module for performing Fourier encoding on voxel positions to generate voxel position vectors; and a loss calculation module for comparing the image feature vector with the voxel position vector to generate voxel occupancy prediction loss. The model training module is used to jointly train the object detection neural network based on the training dataset, wherein the training of the spatial embedding branch includes adjusting the parameters of the object detection neural network to optimize the voxel occupancy prediction loss, and the object detection model for monocular 3D detection is obtained by removing the spatial embedding branch from the object detection neural network obtained by the joint training.

10. A monocular 3D detection device, characterized in that, include: The data acquisition module is used to acquire image data of the target area; A 3D detection module is used to input the image data into a monocular 3D detection model and output the 3D detection result of the target region, wherein the monocular 3D detection model is composed of the target detection neural network jointly trained according to any one of claims 1-7, with the spatial embedding branch removed.

Images