A monocular vision 3D object labeling method based on multimodal data

Abstract

The present invention discloses a monocular vision 3D target labeling method based on multimodal data, which relates to the field of image processing technology. The method comprises: obtaining a 2D data set generated for a training area; determining the camera extrinsic parameters corresponding to the camera generating the 2D data set, and determining the camera matrix corresponding to the camera in combination with the camera extrinsic parameters; determining the projection matrix of the training model in combination with the auxiliary information file corresponding to each RGB image, and training a 3D target detection model based on the projection matrix, in combination with all RGB images, the coordinate position of the 2D target detection frame corresponding to each RGB image, and the camera matrix; obtaining 2D image data collected for the area to be detected, and outputting the 3D detection frame corresponding to each 2D image data in combination with the trained 3D target detection model. The present invention, combined with the projection matrix, can enable the network model to learn the perspective relationship and corner point information of the target in the image, help the parameters of the neural network converge, and achieve ideal 3D detection frame prediction results.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a monocular vision 3D target labeling method based on multimodal data. Background Art

[0002] In traffic scene target detection tasks, 3D target detection algorithms are widely used due to their advantages such as being able to accurately locate target objects and avoid interference from redundant background information. Compared with 2D target detection algorithms, 3D target detection can achieve higher accuracy in downstream tasks such as recognition and tracking. However, although 3D target detection has significant advantages in traffic scenes, its dataset is affected by the limitations of acquisition equipment and the difficulty of annotation, making it difficult to achieve large-scale data collection. In current roadside 3D target detection tasks, 3D annotation boxes are generally obtained through the following two methods:

[0003] 1. Image data is acquired using roadside cameras mounted on light poles or traffic lights. LiDAR sensors are installed on parked or moving vehicles to capture 3D point clouds of the same scene. The data annotation process requires three coordinate systems: the world coordinate system, the camera coordinate system, and the LiDAR coordinate system. To achieve accurate 2D-3D joint annotation results, calibration between the sensors is required. First, the road test camera is calibrated using checkerboard detection to obtain camera intrinsic parameters. Then, the LiDAR coordinate system is calibrated to the world coordinate system using the vehicle positioning module. For world-to-camera calibration, a high-definition map containing lane and crosswalk endpoints is projected onto the 2D image for alignment. After alignment and optimization, a transformation matrix is ​​generated. Finally, the LiDAR-to-World and World-to-Camera transformation matrices are multiplied to obtain the LiDAR-to-Camera transformation matrix. Once the transformation relationships between the three coordinate systems are determined, the ground plane can be fitted using ground points [x, y, z] in the camera coordinate system to calculate the ground equation. During the annotation process, the input of the annotation platform is the image captured by the road test camera and the point cloud scanned by the LiDAR. Through the calibration and calibration between multiple sensors, the transformation between the LiDAR, the world coordinate system and the camera coordinate system, as well as the ground equations and the camera internal parameters are obtained. The point cloud is projected onto the image and the 3D frame position is manually adjusted to adapt the 2D frame for joint annotation. The problem with this method is that it requires multiple sensors such as roadside cameras and on-board LiDAR to collect data. In common urban intersections, highways and other scenes, the above-mentioned multi-sensor data collection conditions are difficult to meet. At the same time, the 3D point cloud data will be lost for targets that are not scanned by the LiDAR. Therefore, only 2D supplementary annotations can be performed on the image for the lost targets. The 3D data set obtained by this method requires multi-sensor support and complex calibration and calibration. Data cleaning is still required during the data processing stage.

[0004] 2. Manually annotate the target with a 3D box on the image. During the annotation process, the annotation platform inputs the image captured by the road test camera. The annotator uses the Cuboid annotation tool to determine the plane position of the target in front and behind, and adjusts the tilt angle of the 3D annotation box to fit the target's outer contour to complete the annotation. This method requires the annotator to have a certain level of spatial imagination and a high level of proficiency in the use of 3D annotation tools. In actual application, the 3D boxes annotated by different annotators cannot be standardized, making it difficult to generate standardized data on a large scale, affecting the training and prediction accuracy of the 3D object detection model. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and specifically provide a monocular vision 3D target labeling method based on multimodal data, which is as follows:

[0006] 1) In the first aspect, the present invention provides a monocular vision 3D target annotation method based on multimodal data, and the specific technical solution is as follows:

[0007] Obtain a 2D dataset generated for the training area, wherein the 2D dataset includes at least one RGB image, an auxiliary information file corresponding to each RGB image, and the coordinate position of a 2D object detection box corresponding to each RGB image;

[0008] Determine the camera extrinsic parameters corresponding to the camera that generates the 2D data set, perform intrinsic calibration on the camera by using a checkerboard detection method, and determine the camera matrix corresponding to the camera in combination with the camera extrinsic parameters;

[0009] Determine the projection matrix of the training model based on the auxiliary information file corresponding to each RGB image, and train the 3D object detection model based on the projection matrix, all RGB images, the coordinate position of the 2D object detection box corresponding to each RGB image, and the camera matrix;

[0010] Obtain the 2D image data collected for the area to be detected, input the 2D image data into the trained 3D object detection model, and output the 3D detection box corresponding to each 2D image data.

[0011] The beneficial effects of the monocular vision 3D target labeling method based on multimodal data provided by the present invention are as follows:

[0012] By performing intrinsic camera calibration and other methods to determine the camera matrix corresponding to the camera, the camera matrix is ​​integrated into the image data as prior information to improve the accuracy of 3D detection box prediction. Combined with the projection matrix, the network model learns the perspective relationship and corner information of the target in the image, helping the neural network parameters converge and achieve ideal 3D detection box prediction results. This solution not only improves prediction accuracy but also saves development resources and funds.

[0013] Based on the above solution, the present invention can also be improved as follows.

[0014] Furthermore, the 2D data set is determined in the following manner:

[0015] Determined by the historical 2D dataset corresponding to the training area;

[0016] Alternatively, based on the image data collected by the roadside monocular camera for the training area, the image data is calibrated, and a 2D target detection frame corresponding to each image data is generated and then stored and determined.

[0017] Furthermore, the process of training the 3D object detection model is as follows:

[0018] Through the convolutional neural network and the feature pyramid network, the input RGB image is feature extracted to obtain four predicted feature maps of different sizes. The projection matrix is ​​used to perform feature sampling in each predicted feature map, and the sampling results are input into the decoder to generate a prediction result. The prediction result includes: the eight corner point coordinates and category index of the 3D detection box corresponding to each 2D target detection box in the RGB image. The loss value is calculated based on the prediction result and the actual label corresponding to the RGB image, and the network parameters of the 3D target detection model are updated by backpropagation until the loss value meets the preset conditions. The 3D target detection model is completed and a trained 3D target detection model is obtained.

[0019] Furthermore, the process of performing feature sampling in each predicted feature map using the projection matrix is ​​specifically as follows:

[0020] The center point of any 2D target detection box is projected onto predicted feature maps of different sizes. The size of the 2D target detection box is used to sample pixel features in the area enclosed by the projection point corresponding to the center point and the preset distance as the radius. The sampled area is subjected to feature fusion through bilinear interpolation and target query column vector to obtain the sampling result.

[0021] Furthermore, the loss value is calculated as follows:

[0022] The loss function is determined by the Hungarian algorithm, and the loss function is specifically:

[0023]

[0024]

[0025] in, is the total loss value, is the minimum matching value between the predicted bounding box and the Ground Truth calculated by the Hungarian algorithm, Represents all permutations and combinations of N prediction results and Ground Truth, i.e., the solution space N represents the number of prediction targets, y i represents the true value of the target, which includes the category to which the target belongs and the coordinates of the corner points of the 3D bounding box, represents the target prediction value, c i represents the target category, Means no goal, represents the prediction category loss, b i represents the true target bounding box coordinates, represents the predicted target bounding box coordinates, represents the bounding box loss, represents the classification loss, Represents the predicted target bounding box corresponding to the true target under the minimum matching arrangement.

[0026] 2) In a second aspect, the present invention further provides a monocular vision 3D target annotation system based on multimodal data, the specific technical solution of which is as follows:

[0027] The acquisition module is used to: acquire a 2D data set generated for the training area, wherein the 2D data set includes at least one RGB image, an auxiliary information file corresponding to each RGB image, and the coordinate position of the 2D target detection box corresponding to each RGB image;

[0028] The determination module is used to: determine the camera extrinsic parameters corresponding to the camera that generates the 2D data set, perform intrinsic parameter calibration on the camera through a checkerboard detection method, and determine the camera matrix corresponding to the camera in combination with the camera extrinsic parameters;

[0029] The training module is used to determine the projection matrix of the training model based on the auxiliary information file corresponding to each RGB image, and train the 3D object detection model based on the projection matrix, all RGB images, the coordinate position of the 2D object detection box corresponding to each RGB image, and the camera matrix;

[0030] The labeling module is used to obtain 2D image data collected for the area to be detected, input the 2D image data into a trained 3D object detection model, and output a 3D detection box corresponding to each 2D image data.

[0031] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0032] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0033] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0035] Figure 1 Schematic diagram of a flow chart of a monocular vision 3D target labeling method based on multimodal data according to an embodiment of the present invention;

[0036] Figure 2 Schematic diagram of a model architecture of a monocular vision 3D object labeling method based on multimodal data according to an embodiment of the present invention;

[0037] Figure 3 This is a structural framework diagram of an electronic device. DETAILED DESCRIPTION

[0038] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0039] like Figure 1 As shown, a monocular vision 3D target labeling method based on multimodal data according to an embodiment of the present invention includes the following steps:

[0040] S1, obtaining a 2D dataset generated for a training area, wherein the 2D dataset includes at least one RGB image, an auxiliary information file corresponding to each RGB image, and the coordinate position of a 2D object detection box corresponding to each RGB image;

[0041] S2, determining camera extrinsic parameters corresponding to the camera that generates the 2D data set, and performing intrinsic parameter calibration on the camera by a checkerboard detection method, and determining a camera matrix corresponding to the camera in combination with the camera extrinsic parameters;

[0042] S3, combining the auxiliary information file corresponding to each RGB image, determining the projection matrix of the training model, and training the 3D object detection model based on the projection matrix, combining all RGB images, the coordinate position of the 2D object detection box corresponding to each RGB image, and the camera matrix;

[0043] S4, obtaining 2D image data collected for the area to be detected, inputting the 2D image data into a trained 3D object detection model, and outputting a 3D detection box corresponding to each 2D image data.

[0044] The beneficial effects of the monocular vision 3D target labeling method based on multimodal data provided by the present invention are as follows:

[0045] By performing intrinsic camera calibration and other methods to determine the camera matrix corresponding to the camera, the camera matrix is ​​integrated into the image data as prior information to improve the accuracy of 3D detection box prediction. Combined with the projection matrix, the network model learns the perspective relationship and corner information of the target in the image, helping the neural network parameters converge and achieve ideal 3D detection box prediction results. This solution not only improves prediction accuracy but also saves development resources and funds.

[0046] In S1, there are two ways to obtain 2D datasets:

[0047] The first method is to obtain an existing 2D dataset, which includes RGB images captured by a camera for the training area, the 2D object detection box corresponding to each RGB image, and auxiliary information files in txt format corresponding to each RGB image.

[0048] The second method uses a roadside monocular camera to capture RGB images of the training area, the 2D target detection box corresponding to each RGB image, the coordinate position corresponding to the 2D target detection box, and the auxiliary information file in txt format corresponding to each RGB image.

[0049] It's important to note that in this solution, to improve the accuracy of the 3D object detection model, we further optimized the selection of the training area, ensuring the richest possible data without wasting excessive resources. Richness refers not only to quantity, but also encompasses diverse scenes, weather conditions, lighting intensities, and camera perspectives.

[0050] In addition, in order to facilitate subsequent data processing, the collected image data is further preprocessed to improve the fusion efficiency. The specific preprocessing process is as follows:

[0051] Enhance the collected image data to make it more visually appealing, i.e. adjust the brightness and contrast to values ​​suitable for feature extraction;

[0052] The collected image data can be subjected to noise reduction and grayscale processing to improve image clarity while reducing data processing volume and improving processing efficiency.

[0053] In this solution, the enhancement and adjustment process of image data is specifically as follows:

[0054] Determine the scene corresponding to the image to be enhanced, which includes but is not limited to: traffic intersections, schools, warehouses, mountainous areas, etc.

[0055] In combination with the scene, the image to be enhanced is analyzed through semantic analysis to obtain the corresponding image description result, and through the above scene and image description result, the historical database is searched for the enhancement adjustment scheme corresponding to the historical image corresponding to the scene and image description result, where the historical database contains the scene corresponding to each historical image, the image description result corresponding to each historical image, and the enhancement adjustment scheme corresponding to each historical image.

[0056] If no enhancement adjustment scheme corresponding to the historical image corresponding to the scene and image description results is found in the historical database, the scene is taken as the main focus to determine whether there is a historical image similar to the image description result in the scene, and the enhancement adjustment scheme is determined based on the search results.

[0057] The process of determining whether there is a historical image similar to the image description result in this scenario can be:

[0058] At least one keyword in the image description corresponding to the image to be enhanced is determined. A search is then performed in a historical database based on all the keywords. The historical images are sorted based on the similarity of the results. The top n historical images in the sort are selected, and the time periods corresponding to the top n historical images are determined. The period corresponding to each historical image is determined based on a preset period. Based on the results of this determination, each of the n historical images is assigned a corresponding weight. Due to the rapid advancement of science and technology, data extraction methods and data enhancement schemes are constantly being optimized. Therefore, the older a historical image matching the image to be enhanced is compared to the current time, the lower its reference value. The preset period is the period during which an algorithm or enhancement scheme is updated. A final enhancement scheme corresponding to the image to be enhanced is determined by combining the weights corresponding to each historical image and the enhancement schemes corresponding to each historical image. Specifically, the weights corresponding to each historical image are multiplied by the corresponding enhancement schemes, and the sum is calculated to obtain the final enhancement scheme.

[0059] It should be noted that the sorting does not sort all the historical images in the historical database. According to the needs, only the first ten images in descending order can be extracted for display. The above number is set according to actual needs and is not limited here.

[0060] If there is no scene identical to the image to be enhanced in the historical database, the image to be enhanced is input into the pre-trained adjustment model, and the required parameters are input to obtain the adjustment plan corresponding to the image to be enhanced.

[0061] S2, determining the camera extrinsic parameters corresponding to the camera that generates the 2D data set, and calibrating the camera intrinsic parameters by a checkerboard detection method, and determining the camera matrix corresponding to the camera in combination with the camera extrinsic parameters.

[0062] The camera external parameters include: the rotation matrix and translation vector that describe the camera coordinates with respect to the world coordinate system.

[0063] The camera is calibrated using checkerboard detection, which is the process of determining the internal parameters of the camera. The intrinsic parameter matrix includes the focal length, optical center, and pixel scale factor, and the distortion coefficient includes radial distortion and tangential distortion. The intrinsic parameter matrix and the extrinsic parameter matrix together constitute the camera matrix. In practical applications, the intrinsic and extrinsic parameter matrices of each camera are different, so the collected image data has a corresponding camera matrix. In order to ensure that the 3D detection frame generated by the annotation has an accurate perspective relationship on the image and fits the target edge, the present invention integrates the camera matrix into the image data as prior information to improve the accuracy of 3D detection frame prediction.

[0064] Distortion occurs when the camera coordinate system is converted to the image physical coordinate system. There are usually radial distortion and tangential distortion. The formula is as follows:

[0065]

[0066] r=x' 2 +y' 2

[0067] x"=x'*(1+k1*r 2 +k2*r 4 +k3*r 6 +2*p1*(x'*y')+p2*(r+2*x'

[0068] *x'))

[0069] y"=y'*(1+k1*r 2 +k2*r 4 +k3*r 6 +2*p1*(x'*y')+p2*(r+2*y'

[0070] *y'))

[0071] x' represents the horizontal coordinate in the normal image pixel coordinates, y' represents the vertical coordinate in the normal image pixel coordinates, and x" and y" represent the distorted image pixel coordinates.

[0072] At this time, the camera coordinate system is [X c , Y c , Z c ]express:

[0073]

[0074] Here, k1, k2, and k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients. Distortion coefficients can be used to correct the distortion of images captured by the camera, improving geometric accuracy and visual quality, making the images more realistic. This is particularly important in monocular 3D annotation tasks.

[0075] S3, combining the auxiliary information file corresponding to each RGB image, determining the projection matrix of the training model, and training the 3D object detection model based on the projection matrix, combining all RGB images, the coordinate position of the 2D object detection box corresponding to each RGB image, and the camera matrix.

[0076] During the training phase, the data is read into memory together with the image data through the data loader to generate the projection matrix required by the model. This matrix contains key elements such as target size, target category, driving posture, camera installation height, and pitch angle. Combined with the image feature vector extracted by the convolutional neural network, the network model can learn the perspective relationship and corner point information of the target in the image, helping the neural network parameters converge and achieve ideal 3D detection box prediction results.

[0077] Combining multimodal data, a neural network model based on the Transformer architecture is used to train a 3D object detection algorithm. During the inference process, the algorithm takes RGB images, 2D object detection box coordinates, object aspect ratio, and camera internal and external parameters as input, extracts target features such as texture and contour, and combines existing prior information to predict the corresponding 3D detection box for each target. The 3D detection box includes the (x, y) coordinates of the eight corner points of the cube and the category index.

[0078] It should be further explained that the 3D object detection algorithm is specifically: a 3D object detection neural network model based on Transformer, the network structure is as follows Figure 2 As shown in the figure, the input of the 3D object detection neural network model consists of two parts: a single RGB image and a projection matrix. Common convolutional neural networks (such as ResNet50) and feature pyramid networks (FPN) are used to extract features from the RGB image to obtain four predicted feature maps of different sizes, the feature map F = {F1, F2, F3, F4}. The four predicted feature maps contain semantic information of objects of different sizes in the image. The projection matrix can sample features in different predicted feature maps according to the size of each 2D object and feed them into the Transformer decoder. After feature embedding, the projection matrix combines the camera intrinsic parameter matrix, the camera extrinsic parameter matrix, and auxiliary information. The projection matrix P can be expressed as:

[0079]

[0080] Among them, f x Indicates the focal length in the x-axis direction described by the pixel, f y Indicates the focal length in the y-axis direction described by the pixel, u0 indicates the abscissa of the actual position of the principal point, v0 indicates the ordinate of the actual position of the principal point, R indicates the rotation matrix, T indicates the translation matrix, A base is the auxiliary information parameter matrix. The auxiliary information includes constant information such as 2D target detection frame coordinates, category, camera height and pitch angle, etc., recorded as X base , these constant information can be mapped into the same dimensional space as the camera matrix through the following linear transformation:

[0081] A base =W static ·X base

[0082] W static It is the transformation matrix of static constant information. The transformation matrix is ​​a learnable parameter matrix, which is randomly initialized at the beginning of model training and the weight parameters are updated after multiple rounds of training and iterative updates.

[0083] Since the projection matrix already contains the 2D coordinate information of the target on the image (that is, each image corresponds to a 2D target label file, which contains the category, x, y, w, h, i.e. the coordinates of the upper left corner, as well as the width and height of all the targets to be detected in the image. The projection matrix is ​​composed of 2D annotation information in the current image, constant information such as camera height and pitch angle), the center point of the 2D annotation box can be projected onto the multi-scale prediction feature map, and the size information of each target is used to sample the pixel features around the projection point. The acquisition of the projection point can be specifically expressed as:

[0084]

[0085] in is the center point of the i-th 2D detection box corresponding to the l-th decoder layer, Indicates matrix splicing, and converts plane coordinates into homogeneous coordinates by adding 1, T c is the camera matrix, Sigmoid is the activation function, c li is the projection point.

[0086] It should be noted that the intrinsic and extrinsic parameter matrices together form the camera matrix, which is used to project the center point of the 2D annotation box onto each predicted feature map. To avoid the impact of different sizes of multiple predicted feature maps and to fuse feature information across layers, the projection points need to be normalized to [-1, 1] using a sigmoid activation function. Each projection matrix contains all 2D object detection boxes in the current image, and multiple objects share the same camera matrix for projection.

[0087] The benefit of this top-down feature sampling approach is that it directly decodes the features of the target region using prior information, removes interference from redundant background information in the image, avoids cross-attention calculations based on global image features, accelerates model training convergence, and achieves ideal prediction results. The sampled feature region is fused with the target query vector (Object Query) through bilinear interpolation, which can be expressed as:

[0088] f kli =f bilinear (Fk ,c li )

[0089] q (l+1)i =f li +q li

[0090] Among them, F k represents the kth prediction feature layer generated by the feature pyramid network, c li represents the center point of the 2D target, i represents the target index in the current image, l represents the number of layers of the Transformer decoder, and the sampling feature calculated by bilinear interpolation is recorded as f kli ; The target query vector q of each layer is obtained by fusing the target query vector of the previous layer and the sampling feature f, f bilinear Indicates bilinear interpolation calculation, f li represents the sampling characteristics, q li represents the target query vector of the lth decoding layer, q (l+1)i represents the (l+1)th layer target query vector.

[0091] In the Transformer decoder part, it is necessary to randomly initialize and generate an N×256-dimensional target query vector (Object Query). In practical applications, N can be greater than the actual target to be detected in the image, and can usually be set to N=300. In the present invention, the Transformer decoder module repeatedly stacks L layers, which can be set to L=6 according to experience. The attention mechanism operation of each layer and the input and output of the module are the same. First, the target query vector obtained by random initialization communicates with each other through a multi-head self-attention mechanism, which facilitates further feature extraction on the 2D feature map. Secondly, the multi-scale feature map generated by the feature pyramid obtains sampling features through the projection matrix, and then interacts with the target query vector in two ways. The first way is to sample the features f li Directly add it to the target query vector to obtain the target query vector after feature fusion. The second way is to use it as the key vector (Key) and value vector (Value) and perform a multi-head cross-attention operation with the target query vector. The calculation of the attention mechanism is as follows:

[0092] [Q; K; V] = [W'1(f l +q l );W'2f l ;W'3f l ]

[0093] MultiHead(Q,K,V)=Concat(head1,...,head n )W o

[0094] head i =Attention(Q,K,V)

[0095] In the above formula, W'1, W'2, W'3 and W o are the target query vector, key vector, value vector and parameter matrix of the multi-head attention mechanism, respectively, l is the sampling feature matrix, Q; K; V represent the target query space, key space and value space respectively, q l represents the target query vector, MultiHead(Q,K,V) represents the multi-head attention mechanism operation, Concat(head1,...,head n ) indicates a cascade operation on n attention heads, head1 indicates the first attention head, head n represents the nth attention head.

[0096] After decoding by the L-layer decoder module, each target query vector is predicted by a feed-forward neural network (FFN) to generate the coordinates of the eight corner points of the corresponding 3D detection box and the category index. The prediction result will be used to calculate the loss with the actual label and backpropagate to update the overall network parameters.

[0097] Since the present invention adopts a deep neural network based on the Transformer structure, the model prediction result is a fixed set of N predicted values, and the model does not introduce the non-maximum suppression algorithm (NMS). It is necessary to use bipartite graph matching to find the unique corresponding ground truth for the predicted 3D bounding box. Therefore, the Hungarian algorithm is used to solve the bipartite graph matching problem and calculate the final loss function. The calculation formula is as follows:

[0098]

[0099] in, It is the minimum match between the predicted bounding box and the Ground Truth calculated by the Hungarian algorithm, and its loss value is determined by the predicted category loss and 3D object box L1 loss Joint decision, find the minimum match and use Calculate the total loss of the model.

[0100] S4, obtaining 2D image data collected for the area to be detected, inputting the 2D image data into a trained 3D object detection model, and outputting a 3D detection box corresponding to each 2D image data.

[0101] Furthermore, the 2D data set is determined in the following manner:

[0102] Determined by the historical 2D dataset corresponding to the training area;

[0103] Alternatively, based on the image data collected by the roadside monocular camera for the training area, the image data is calibrated, and a 2D target detection frame corresponding to each image data is generated and then stored and determined.

[0104] Furthermore, the process of training the 3D object detection model is as follows:

[0105] Through the convolutional neural network and the feature pyramid network, the input RGB image is feature extracted to obtain four predicted feature maps of different sizes. The projection matrix is ​​used to perform feature sampling in each predicted feature map, and the sampling results are input into the decoder to generate a prediction result. The prediction result includes: the eight corner point coordinates and category index of the 3D detection box corresponding to each 2D target detection box in the RGB image. The loss value is calculated based on the prediction result and the actual label corresponding to the RGB image, and the network parameters of the 3D target detection model are updated by backpropagation until the loss value meets the preset conditions. The 3D target detection model is completed and a trained 3D target detection model is obtained.

[0106] Furthermore, the process of performing feature sampling in each predicted feature map using the projection matrix is ​​specifically as follows:

[0107] The center point of any 2D target detection box is projected onto predicted feature maps of different sizes. The size of the 2D target detection box is used to sample pixel features in the area enclosed by the projection point corresponding to the center point and the preset distance as the radius. The sampled area is subjected to feature fusion through bilinear interpolation and target query column vector to obtain the sampling result.

[0108] Furthermore, the loss value is calculated as follows:

[0109] The loss function is determined by the Hungarian algorithm, and the loss function is specifically:

[0110]

[0111] in, is the total loss value, is the minimum matching value between the predicted bounding box and the Ground Truth calculated by the Hungarian algorithm, Represents all permutations and combinations of N prediction results and Ground Truth, i.e., the solution space N represents the number of prediction targets, y i represents the true value of the target, which includes the category to which the target belongs and the coordinates of the corner points of the 3D bounding box, represents the target prediction value, c i represents the target category, Means no goal, represents the prediction category loss, b i represents the true target bounding box coordinates, represents the predicted target bounding box coordinates, represents the bounding box loss, represents the classification loss, Represents the predicted target bounding box corresponding to the true target under the minimum matching arrangement.

[0112] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the scope of protection of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0113] For ease of understanding, this solution is now explained through the following examples:

[0114] 1. Use a roadside monocular camera to collect image data for different scenes. The sensor type is 1 / 1.8" CMOS, the sampling frame rate is 30-60Hz, and the image format is RGB format. The images are compressed and saved as JPEG images with a resolution of 1920×1080. Alternatively, an existing 2D object detection dataset can be used. Obtain the 2D detection box annotation information of the target to be detected in the image, including its category and coordinate representation, denoted as (x, y, w, h), where x and y represent the upper left corner of the 2D object detection box, and w and h represent the width and height of the target, respectively.

[0115] 2. Get the external parameter matrix to realize the conversion from world coordinates to camera coordinate system. The external parameter matrix is ​​described by the orthogonal rotation matrix R and translation vector T, which can be expressed as

[0116]

[0117] 3. Get the intrinsic parameter matrix, prepare a chessboard of known size, take multiple chessboard images, use image processing algorithms (such as the findChessboardCorners() function provided by OpenCV) to automatically detect chessboard corners, and pass the detected corner data to the camera calibration function, which will calculate the camera's internal parameters based on the corner data, including the intrinsic parameter matrix and distortion coefficient. The intrinsic parameter matrix is ​​expressed as Denoted as:

[0118]

[0119] where f x and f y They represent the pixel length of the focal length in the x-axis and y-axis directions respectively, u0 and v0 represent the actual position of the principal point, and the intrinsic parameter matrix can realize the conversion from the camera coordinate system to the image pixel coordinate system.

[0120] Combining steps 2 and 3, the transformation from the world coordinate system to the pixel coordinate system depends on the camera's intrinsic parameter matrix and extrinsic parameter matrix, which can be specifically expressed as:

[0121]

[0122] Create an auxiliary information file corresponding to each image, and generate a projection matrix in the subsequent model training process. The content is: (cls, x, y, w, h, ratio, alpha, rotation_y, camera_h, angle), where cls represents the target category. The dataset in this invention contains three major categories, including motor vehicles, pedestrians, and non-motor vehicles, which are further divided into seven subcategories, namely car, truck, bus, person, cyclist, motorcyclist, and tricyclist. The target category will be encoded in the data fusion stage. The code is a 7×7 one-hot vector; ratio is the aspect ratio of the 2D target detection box; alpha is the observation angle of the object, ranging from -π to +π. In the camera coordinate system, the object is rotated around the y-axis to the camera z-axis with the camera origin as the center and the line connecting the camera origin to the object center as the radius. The angle between the object direction and the camera x-axis at this time is; rotation_y is the spatial direction of the target, which is the angle between the forward direction of the object in the camera coordinate system and the x-axis of the camera coordinate system, ranging from -π to +π; camera_h and angle are the camera installation height and pitch angle, respectively.

[0123] Using the Pytorch deep learning framework, load the RGB image data, 3D label file, and auxiliary information file prepared in the above steps, instantiate a deep neural network based on the Transformer structure, train the network model until the loss function converges, and generate a PTH model file for persistent storage;

[0124] The model is called to perform inference on the dataset to be tested, and the test results are saved as a txt file, which includes (cls_id, x1, y1, x2, y2...... x8, y8), where cls_id is the target category and the last 16 points represent the image coordinates of the 8 corner points (x, y) of the predicted 3D annotation box, thus completing the 3D target annotation. Optionally, if the dataset to be tested cannot obtain auxiliary information such as camera height, pitch angle, and target posture during the inference phase, then only the 2D target detection box coordinate position and category label are provided to still perform 3D annotation of the target, and the unknown information is filled with 0. Since the transformation matrix of the static constant information is a learnable parameter matrix, even if some information is missing, the projection matrix can still extract the target area in the image to achieve relatively ideal 3D target detection, further providing the convenience of data collection and can be applied to actual annotation tasks on a large scale.

[0125] The present invention also provides a monocular vision 3D target labeling system based on multimodal data, and the specific technical solution is as follows:

[0126] The acquisition module is used to: acquire a 2D data set generated for the training area, wherein the 2D data set includes at least one RGB image, an auxiliary information file corresponding to each RGB image, and the coordinate position of the 2D target detection box corresponding to each RGB image;

[0127] The determination module is used to: determine the camera extrinsic parameters corresponding to the camera that generates the 2D data set, perform intrinsic parameter calibration on the camera through a checkerboard detection method, and determine the camera matrix corresponding to the camera in combination with the camera extrinsic parameters;

[0128] The training module is used to determine the projection matrix of the training model based on the auxiliary information file corresponding to each RGB image, and train the 3D object detection model based on the projection matrix, all RGB images, the coordinate position of the 2D object detection box corresponding to each RGB image, and the camera matrix;

[0129] The labeling module is used to obtain 2D image data collected for the area to be detected, input the 2D image data into a trained 3D object detection model, and output a 3D detection box corresponding to each 2D image data.

[0130] It should be noted that the beneficial effects of the monocular vision 3D target labeling system based on multimodal data provided by the above embodiment are the same as the beneficial effects of the above monocular vision 3D target labeling method based on multimodal data, which will not be repeated here. In addition, when the system provided by the above embodiment realizes its functions, it only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0131] like Figure 3 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, which is coupled to a memory 310. The memory 310 stores at least one computer program 330. The at least one computer program 330 is loaded and executed by the processor 320 to enable the electronic device 300 to implement any of the above methods. Specifically:

[0132] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320 to enable the electronic device 300 to implement a monocular vision 3D target labeling method based on multimodal data provided in the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for realizing device functions, which will not be described in detail here.

[0133] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0134] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0135] In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the above methods.

[0136] It should be noted that the terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and to define a specific order or precedence. Where appropriate, the order used for similar objects may be interchanged, such that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0137] Those skilled in the art will appreciate that the present invention may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention may be implemented in the form of a computer program product embodied in one or more computer-readable media containing computer-readable program code.

[0138] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0139] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A monocular vision 3D object labeling method based on multimodal data, characterized in that: include: Obtain a 2D dataset generated for the training area, wherein the 2D dataset includes at least one RGB image, an auxiliary information file corresponding to each RGB image, and the coordinate position of a 2D object detection box corresponding to each RGB image; Determine the camera extrinsic parameters corresponding to the camera that generates the 2D data set, perform intrinsic calibration on the camera by using a checkerboard detection method, and determine the camera matrix corresponding to the camera in combination with the camera extrinsic parameters; Determine the projection matrix of the training model based on the auxiliary information file corresponding to each RGB image, and train the 3D object detection model based on the projection matrix, all RGB images, the coordinate position of the 2D object detection box corresponding to each RGB image, and the camera matrix; Obtain 2D image data collected for the area to be detected, input the 2D image data into a trained 3D object detection model, and output a 3D detection box corresponding to each 2D image data; The process of training a 3D object detection model is as follows: Using a convolutional neural network and a feature pyramid network, feature extraction is performed on the input RGB image to obtain four predicted feature maps of different sizes. Feature sampling is performed in each predicted feature map using the projection matrix, and the sampling results are input into the decoder to generate a prediction result, which includes the eight corner point coordinates and category index of the 3D detection box corresponding to each 2D object detection box in the RGB image. A loss value is calculated based on the prediction result and the actual label corresponding to the RGB image, and the network parameters of the 3D object detection model are updated by backpropagation until the loss value meets the preset conditions. The 3D object detection model is completed to obtain a trained 3D object detection model. The loss value is calculated as follows: The loss function is determined by the Hungarian algorithm, and the loss function is specifically: in, is the total loss value, is the minimum matching value between the predicted bounding box and the Ground Truth calculated by the Hungarian algorithm, Represents all permutations and combinations of N prediction results and Ground Truth, i.e., the solution space N represents the number of prediction targets, y i represents the true value of the target, which includes the category to which the target belongs and the coordinates of the corner points of the 3D bounding box, represents the target prediction value, c i represents the target category, Means no goal, represents the prediction category loss, b i represents the true target bounding box coordinates, represents the predicted target bounding box coordinates, represents the bounding box loss, represents the classification loss, Represents the predicted target bounding box corresponding to the true target under the minimum matching arrangement.

2. The monocular vision 3D target labeling method based on multimodal data according to claim 1, characterized in that: The 2D dataset is determined by: Determined by the historical 2D dataset corresponding to the training area; Alternatively, based on the image data collected by the roadside monocular camera for the training area, the image data is calibrated, and a 2D target detection frame corresponding to each image data is generated and then stored and determined.

3. The monocular vision 3D target labeling method based on multimodal data according to claim 1, characterized in that: Using the projection matrix, the process of performing feature sampling in each predicted feature map is as follows: The center point of any 2D target detection box is projected onto predicted feature maps of different sizes. The size of the 2D target detection box is used to sample pixel features in the area enclosed by the projection point corresponding to the center point and the preset distance as the radius. The sampled area is subjected to feature fusion through bilinear interpolation and target query column vector to obtain the sampling result.

4. A monocular vision 3D target labeling system based on multimodal data, using the monocular vision 3D target labeling method based on multimodal data according to claim 1, characterized in that: The system comprises: The acquisition module is used to: acquire a 2D data set generated for the training area, wherein the 2D data set includes at least one RGB image, an auxiliary information file corresponding to each RGB image, and the coordinate position of the 2D target detection box corresponding to each RGB image; The determination module is used to: determine the camera extrinsic parameters corresponding to the camera that generates the 2D data set, perform intrinsic parameter calibration on the camera through a checkerboard detection method, and determine the camera matrix corresponding to the camera in combination with the camera extrinsic parameters; The training module is used to determine the projection matrix of the training model based on the auxiliary information file corresponding to each RGB image, and train the 3D object detection model based on the projection matrix, all RGB images, the coordinate position of the 2D object detection box corresponding to each RGB image, and the camera matrix; The labeling module is used to obtain 2D image data collected for the area to be detected, input the 2D image data into a trained 3D object detection model, and output a 3D detection box corresponding to each 2D image data.

5. The monocular vision 3D target labeling system based on multimodal data according to claim 4, characterized in that: The 2D dataset is determined by: Determined by the historical 2D dataset corresponding to the training area; Alternatively, based on the image data collected by the roadside monocular camera for the training area, the image data is calibrated, and a 2D target detection frame corresponding to each image data is generated and then stored and determined.

6. An electronic device, characterized in that: The electronic device includes a processor coupled to a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable a computer to implement the method according to any one of claims 1 to 3.

Images