Monocular vision 3D target labeling method based on multi-modal data

Abstract

The invention discloses a monocular vision 3D target labeling method based on multi-modal data, and relates to the technical field of image processing, and the method comprises the steps: obtaining a 2D data set generated for a training region; determining camera external parameters corresponding to the camera generating the 2D data set, and determining a camera matrix corresponding to the camera in combination with the camera external parameters; determining a projection matrix of the training model in combination with the auxiliary information file corresponding to each RGB image, and training a 3D target detection model based on the projection matrix in combination with all the RGB images, the coordinate position of the 2D target detection frame corresponding to each RGB image and the camera matrix; and obtaining 2D image data collected for the to-be-detected area, and outputting a 3D detection frame corresponding to each piece of 2D image data in the trained 3D target detection model. In combination with the projection matrix, the network model can learn the perspective relation and angular point information of the target in the image, parameter convergence of the neural network is helped, and an ideal 3D detection frame prediction result is achieved.

Description

Background Art

[0002] In traffic scene target detection tasks, 3D target detection algorithms are widely used due to their advantages such as being able to accurately locate target objects and avoid interference from background redundant information. Compared with 2D target detection algorithms, 3D target detection can achieve higher accuracy in downstream tasks such as recognition and tracking. However, although 3D target detection has significant advantages in traffic scenes, its data set is limited by the acquisition equipment and the difficulty of annotation, making it difficult to achieve large-scale data acquisition. In the current roadside 3D target detection task, the 3D annotation box is generally obtained in the following two ways:

[0003] 1. Use roadside cameras installed on streetlight poles or traffic light poles to obtain image data, and install LiDAR on parked or moving vehicles to collect 3D point clouds of the same scene. Three coordinate systems are required in the data annotation process, namely the world coordinate system, the camera coordinate system, and the LiDAR coordinate system. In order to obtain accurate 2D-3D joint annotation results, calibration is required between different sensors. First, the road test camera is calibrated through checkerboard detection to obtain the camera internal parameters, and then the LiDAR coordinate system is calibrated to the world coordinate system through the vehicle positioning module. For the calibration of the world coordinate system to the camera coordinate system, the high-definition map containing the lane and the pedestrian crossing endpoint needs to be projected onto the 2D image for matching, and the conversion matrix is ​​obtained after adjustment and optimization. Finally, the LiDAR-to-World and World-to-Camera conversion matrices are multiplied to obtain the LiDAR-to-Camera conversion matrix. After obtaining the conversion relationship between the three coordinate systems, the ground point [x, y, z] in the camera coordinate system can be used to fit the ground plane to calculate the ground equation. During the annotation process, the input of the annotation platform is the image captured by the road test camera and the point cloud scanned by the LiDAR. Through the calibration and calibration between multiple sensors, the transformation between the LiDAR, world coordinate system and camera coordinate system, as well as the ground equation and camera intrinsic parameters are obtained. The point cloud is projected onto the image and the 3D box position is manually adjusted to adapt the 2D box for joint annotation. The problem with this method is that it requires roadside cameras and vehicle-mounted LiDAR multi-sensors to collect data. In common urban intersections, highways and other scenes, the above multi-sensor data collection conditions are difficult to meet. At the same time, for targets that are not scanned by the LiDAR, the 3D point cloud data will be lost. Therefore, only 2D supplementary annotations can be performed on the image for the lost targets. The 3D data set annotated in this way requires multi-sensor support and complex calibration and calibration. Data cleaning is still required in the data processing stage;

[0004] 2. Manually mark the target with a 3D box on the image. During the marking process, the input of the marking platform is the image captured by the road test camera. The annotator needs to use the Cuboid marking tool to determine the plane position of the target in front and behind, and adjust the inclination angle of the 3D marking box so that it fits the outer contour of the target to complete the marking. This method requires the annotator to have a certain spatial imagination ability and a high level of proficiency in the use of 3D marking tools. In actual application, the 3D boxes annotated by different annotators cannot be standardized, and it is difficult to generate standardized data on a large scale, which affects the training and prediction accuracy of the 3D target detection model. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a monocular vision 3D target labeling method based on multimodal data, which is as follows:

[0006] 1) In the first aspect, the present invention provides a monocular vision 3D target labeling method based on multimodal data, and the specific technical solution is as follows:

[0007] Obtain a 2D data set generated for the training area, wherein the 2D data set includes at least one RGB image, an auxiliary information file corresponding to each RGB image, and a coordinate position of a 2D target detection box corresponding to each RGB image;

[0008] Determine the camera extrinsic parameters corresponding to the camera that generates the 2D data set, perform intrinsic parameter calibration on the camera by means of a checkerboard detection method, and determine the camera matrix corresponding to the camera in combination with the camera extrinsic parameters;

[0009] Combined with the auxiliary information file corresponding to each RGB image, determine the projection matrix of the training model, and based on the projection matrix, combine all RGB images, the coordinate position of the 2D target detection box corresponding to each RGB image, and the camera matrix, train the 3D target detection model;

[0010] The 2D image data collected for the area to be detected is obtained, the 2D image data is input into a trained 3D object detection model, and a 3D detection frame corresponding to each 2D image data is output.

[0011] The beneficial effects of the monocular vision 3D target labeling method based on multimodal data provided by the present invention are as follows:

[0012] The camera matrix corresponding to the camera is determined by performing internal calibration on the camera, and the camera matrix is ​​fused into the image data as prior information to improve the accuracy of 3D detection frame prediction. Combined with the projection matrix, the network model can learn the perspective relationship and corner point information of the target in the image, which helps the neural network parameters converge and achieve the ideal 3D detection frame prediction results. This solution can not only improve the prediction accuracy, but also save development resources and money.

[0013] Based on the above solution, the present invention can also be improved as follows.

[0014] Furthermore, the 2D data set is determined in the following manner:

[0015] Determined by the historical 2D dataset corresponding to the training area;

[0016] Alternatively, based on the image data collected by the roadside monocular camera for the training area, the image data is calibrated, and a 2D target detection frame corresponding to each image data is generated and then stored and determined.

[0017] Furthermore, the process of training the 3D object detection model is as follows:

[0018] The input RGB image is feature extracted through a convolutional neural network and a feature pyramid network to obtain four predicted feature maps of different sizes. The projection matrix is ​​used to perform feature sampling in each predicted feature map, and the sampling results are input into a decoder to generate a prediction result, which includes eight corner point coordinates and a category index of a 3D detection box corresponding to each 2D target detection box in the RGB image. The loss value is calculated based on the prediction result and the actual label corresponding to the RGB image, and the network parameters of the 3D target detection model are updated by back propagation until the loss value meets the preset conditions, the 3D target detection model is completed, and a trained 3D target detection model is obtained.

[0019] Furthermore, the process of performing feature sampling in each predicted feature map using the projection matrix is ​​specifically as follows:

[0020] The center point of any 2D target detection box is projected onto predicted feature maps of different sizes. The size of the 2D target detection box is used to perform pixel feature sampling on the area enclosed by the projection point corresponding to the center point and the preset distance as the radius. The sampled area is feature fused through bilinear interpolation and target query column vector to obtain the sampling result.

[0021] Furthermore, the loss value is calculated as follows:

[0022] The loss function is determined by the Hungarian algorithm, and the loss function is specifically:

[0023]

[0024]

[0025] in, is the total loss value, is the minimum match value between the predicted bounding box and the Ground Truth calculated by the Hungarian algorithm, Represents all permutations and combinations of N prediction results and Ground Truth, i.e., the solution space N represents the number of prediction targets, y i represents the true value of the target, which includes the category to which the target belongs and the coordinates of the corner points of the 3D bounding box, represents the target prediction value, c i represents the target category, Indicates no goal, represents the prediction category loss, b i represents the true target bounding box coordinates, represents the predicted target bounding box coordinates, represents the bounding box loss, represents the classification loss, Represents the predicted target bounding box corresponding to the true target under the minimum matching arrangement.

[0026] 2) In the second aspect, the present invention also provides a monocular vision 3D target labeling system based on multimodal data, and the specific technical solution is as follows:

[0027] The acquisition module is used to: acquire a 2D data set generated for a training area, wherein the 2D data set includes at least one RGB image, an auxiliary information file corresponding to each RGB image, and a coordinate position of a 2D target detection frame corresponding to each RGB image;

[0028] The determination module is used to: determine the camera extrinsic parameters corresponding to the camera that generates the 2D data set, calibrate the intrinsic parameters of the camera by a checkerboard detection method, and determine the camera matrix corresponding to the camera in combination with the camera extrinsic parameters;

[0029] The training module is used to: determine the projection matrix of the training model in combination with the auxiliary information file corresponding to each RGB image, and train the 3D object detection model based on the projection matrix, in combination with all RGB images, the coordinate position of the 2D object detection box corresponding to each RGB image, and the camera matrix;

[0030] The labeling module is used to obtain 2D image data collected for the area to be detected, input the 2D image data into a trained 3D object detection model, and output a 3D detection box corresponding to each 2D image data.

[0031] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0032] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above methods.

[0033] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0035] Figure 1 A schematic diagram of a flow chart of a monocular vision 3D target labeling method based on multimodal data according to an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of a model architecture of a monocular vision 3D target labeling method based on multimodal data according to an embodiment of the present invention;

[0037] Figure 3 The figure is a structural framework diagram of an electronic device. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0039] like Figure 1 As shown, a monocular vision 3D target labeling method based on multimodal data in an embodiment of the present invention includes the following steps:

[0040] S1, obtaining a 2D data set generated for a training area, wherein the 2D data set includes at least one RGB image, an auxiliary information file corresponding to each RGB image, and a coordinate position of a 2D target detection box corresponding to each RGB image;

[0041] S2, determining the camera extrinsic parameters corresponding to the camera that generates the 2D data set, and calibrating the camera intrinsic parameters by a checkerboard detection method, and determining the camera matrix corresponding to the camera in combination with the camera extrinsic parameters;

[0042] S3, combining the auxiliary information file corresponding to each RGB image, determining the projection matrix of the training model, and training the 3D object detection model based on the projection matrix, combining all RGB images, the coordinate position of the 2D object detection box corresponding to each RGB image, and the camera matrix;

[0043] S4, obtaining 2D image data collected for the area to be detected, inputting the 2D image data into a trained 3D object detection model, and outputting a 3D detection frame corresponding to each 2D image data.

[0044] The beneficial effects of the monocular vision 3D target labeling method based on multimodal data provided by the present invention are as follows:

[0045] The camera matrix corresponding to the camera is determined by performing internal calibration on the camera, and the camera matrix is ​​fused into the image data as prior information to improve the accuracy of 3D detection frame prediction. Combined with the projection matrix, the network model can learn the perspective relationship and corner point information of the target in the image, which helps the neural network parameters converge and achieve the ideal 3D detection frame prediction results. This solution can not only improve the prediction accuracy, but also save development resources and money.

[0046] In S1, there are two ways to obtain 2D data sets:

[0047] The first method is to obtain an existing 2D dataset, which includes RGB images captured by a camera for the training area, a 2D target detection box corresponding to each RGB image, and an auxiliary information file in txt format corresponding to each RGB image.

[0048] The second method uses a roadside monocular camera to capture RGB images of the training area, the 2D target detection frame corresponding to each RGB image and the coordinate position corresponding to the 2D target detection frame, and the auxiliary information file in txt format corresponding to each RGB image.

[0049] It should be further explained that in this solution, in order to improve the accuracy of the 3D object detection model, further optimization was performed on the selection of the training area, which ensured that the data was as rich as possible without wasting excessive resources. Richness does not only refer to the richness in quantity, but also includes different scenes, different weather, different light intensities, and different camera perspectives.

[0050] In addition, in order to facilitate subsequent data processing, the collected image data is further preprocessed to improve the fusion efficiency. The specific preprocessing process is as follows:

[0051] Enhance the collected image data to make the visual effect of the image data stronger, that is, adjust the brightness and contrast to values ​​suitable for feature extraction;

[0052] The collected image data can be subjected to noise reduction and grayscale processing to improve image clarity while reducing the amount of data processing and improving processing efficiency.

[0053] In this solution, the enhancement and adjustment process of image data is specifically as follows:

[0054] Determine the scene corresponding to the image to be enhanced, which includes but is not limited to: traffic intersections, schools, warehouses, mountainous areas, etc.

[0055] In combination with the scene, the image to be enhanced is analyzed through semantic analysis to obtain the corresponding image description result, and through the above scene and image description result, the historical database is searched for the enhancement adjustment scheme corresponding to the historical image corresponding to the scene and the image description result, wherein the historical database contains the scene corresponding to each historical image, the image description result corresponding to each historical image, and the enhancement adjustment scheme corresponding to each historical image.

[0056] If no enhancement adjustment scheme corresponding to the historical image corresponding to the scene and image description results is found in the historical database, the scene is taken as the main focus to determine whether there is a historical image similar to the image description result in the scene, and the enhancement adjustment scheme is determined based on the search results.

[0057] The process of determining whether there is a historical image similar to the image description result in this scenario can be:

[0058] Determine at least one keyword in the image description result corresponding to the image to be enhanced, search in the historical database according to all the keywords, and sort the historical images based on the similarity of the search, select the first n historical images in the sorting, determine the time corresponding to the first n historical images, and determine the period corresponding to each historical image according to the preset period, and assign corresponding weights to the n historical images based on the determination result. Due to the rapid development of science and technology in the contemporary era, the data extraction method or the data enhancement adjustment scheme is constantly optimized at intervals. Therefore, the longer the historical image matching the image to be enhanced is compared with the current time, the lower the reference value. The preset period is the period of updating an algorithm or an enhancement adjustment scheme. Combined with the weight corresponding to each historical image and the enhancement adjustment scheme corresponding to each historical image, determine the final enhancement adjustment scheme corresponding to the image to be enhanced, that is, according to the weight corresponding to each historical image, multiply the corresponding weight by the corresponding enhancement adjustment scheme and perform summation to obtain the final enhancement adjustment scheme.

[0059] It should be noted that the sorting does not sort all the historical images in the historical database. According to the requirements, only the first ten images in descending order can be extracted for display. The above number is set according to actual requirements and is not limited here.

[0060] If there is no scene identical to the image to be enhanced in the historical database, the image to be enhanced is input through the pre-trained adjustment model, and the required parameters are input to obtain an adjustment plan corresponding to the image to be enhanced.

[0061] S2, determining the camera extrinsic parameters corresponding to the camera that generates the 2D data set, and calibrating the camera intrinsic parameters by means of a checkerboard detection method, and determining the camera matrix corresponding to the camera in combination with the camera extrinsic parameters.

[0062] The camera external parameters include: the rotation matrix and translation vector that describe the camera coordinates with respect to the world coordinate system.

[0063] The camera is calibrated with internal parameters through checkerboard detection, that is, the process of determining the internal parameters of the camera. The internal parameter matrix includes focal length, optical center and pixel scale factor, and the distortion coefficient includes radial distortion and tangential distortion. The internal parameter matrix and the external parameter matrix together constitute the camera matrix. In practical applications, the internal and external parameter matrices of each camera are not the same, so the collected image data has a corresponding camera matrix. In order to make the 3D detection frame generated by the annotation have an accurate perspective relationship on the image and fit the edge of the target, the present invention fuses the camera matrix into the image data as prior information to improve the accuracy of 3D detection frame prediction.

[0064] Among them, the distortion occurs in the process of converting the camera coordinate system into the image physical coordinate system. Usually there is radial distortion and tangential distortion. The formula is as follows:

[0065]

[0066] r=x' 2 +y' 2

[0067] x"=x'*(1+k1*r 2 +k2*r 4 +k3*r 6 +2*p1*(x'*y')+p2*(r+2*x'

[0068] *x'))

[0069] y"=y'*(1+k1*r 2 +k2*r 4 +k3*r 6 +2*p1*(x'*y')+p2*(r+2*y'

[0070] *y'))

[0071] x' represents the horizontal coordinate in the normal image pixel coordinates, y' represents the vertical coordinate in the normal image pixel coordinates, and x" and y" represent the distorted image pixel coordinates.

[0072] At this time, the camera coordinate system is [X c , Y c , Z c ]express:

[0073]

[0074] Among them, k1, k2, k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients. The distortion coefficients can be used to correct the distortion of the image captured by the camera, improve the geometric accuracy and visual quality, and make the image more consistent with the real scene, which is particularly important in the monocular camera 3D annotation task.

[0075] S3, combining the auxiliary information file corresponding to each RGB image, determining the projection matrix of the training model, and training the 3D object detection model based on the projection matrix, combining all RGB images, the coordinate position of the 2D object detection box corresponding to each RGB image, and the camera matrix. Wherein:

[0076] During the training phase, the image data is read into the memory through the data loader to generate the projection matrix required by the model. The matrix contains key elements such as target size, target category, driving posture, camera installation height, and pitch angle. Combined with the image feature vector extracted by the convolutional neural network, the network model can learn the perspective relationship and corner point information of the target in the image, help the neural network parameters converge, and achieve ideal 3D detection frame prediction results.

[0077] Combined with multimodal data, a neural network model based on the Transformer architecture is used to train a 3D object detection algorithm. During the inference process, the algorithm takes RGB images, 2D object detection box coordinates, object aspect ratio, and camera internal and external parameters as inputs, extracts texture, contour and other features of the target, and combines the existing prior information to predict the corresponding 3D detection box for each target. The 3D detection box includes the (x, y) coordinates of the eight corner points of the cube and the category index.

[0078] It should be further explained that the 3D object detection algorithm is specifically: a 3D object detection neural network model based on Transformer, the network structure is as follows Figure 2 As shown in the figure, the input of the 3D object detection neural network model includes two parts, a single RGB image and a projection matrix. The common convolutional neural network (such as ResNet50) and feature pyramid network (FPN) are used to extract features from the RGB image to obtain four predicted feature maps of different sizes, the feature map F = {F1, F2, F3, F4}, The four predicted feature maps contain semantic information of objects of different sizes in the image. The projection matrix can sample features in different predicted feature maps according to the size of each 2D object and send them to the Transformer decoder. After feature embedding, the projection matrix combines the camera intrinsic parameter matrix, camera extrinsic parameter matrix and auxiliary information. The projection matrix P can be expressed as:

[0079]

[0080] Among them, f x Indicates the focal length in the x-axis direction described by the pixel, f y represents the focal length in the y-axis direction described by the pixel, u0 represents the abscissa of the actual position of the principal point, v0 represents the ordinate of the actual position of the principal point, R represents the rotation matrix, T represents the translation matrix, and A base is the auxiliary information parameter matrix. The auxiliary information includes constant information such as 2D target detection frame coordinates, category, camera height and pitch angle, etc., recorded as X base , these constant information can be mapped into the same dimensional space as the camera matrix through the following linear transformation:

[0081] A base =W static ·X base

[0082] W static It is the transformation matrix of static constant information. The transformation matrix is ​​a learnable parameter matrix. It is randomly initialized at the beginning of model training and the weight parameters are updated after multiple rounds of training and iteration.

[0083] Since the projection matrix already contains the 2D coordinate information of the target on the image (that is, each image corresponds to a 2D target label file, which contains the category, x, y, w, h, i.e. the coordinates of the upper left corner, as well as the width and height of all the targets to be detected in the image. The projection matrix is ​​composed of the 2D annotation information in the current image, constant information such as the camera height and pitch angle), the center point of the 2D annotation box can be projected onto the multi-scale prediction feature map, and the size information of each target is used to sample the pixel features around the projection point. The acquisition of the projection point can be specifically expressed as:

[0084]

[0085] in is the center point of the ith 2D detection box corresponding to the lth decoder layer, Indicates matrix concatenation, converting plane coordinates into homogeneous coordinates by adding 1, T c is the camera matrix, Sigmoid is the activation function, c li is the projection point.

[0086] It should be noted that the intrinsic matrix and the extrinsic matrix together form the camera matrix, which is used to project the center point of the 2D annotation box onto each predicted feature map. In order to avoid the impact of different sizes of multiple predicted feature maps and fuse cross-layer feature information, the projection point needs to be normalized to [-1,1] using the sigmoid activation function. Each projection matrix contains all 2D target detection boxes in the current image, and multiple targets share the same camera matrix for projection.

[0087] The benefit of this top-down feature sampling method is that it uses prior information to directly decode the features of the target area, removes the interference of redundant information in the background part of the image, avoids cross-attention calculation based on global image features, speeds up the convergence of model training and achieves ideal prediction results. The sampled feature area is fused through bilinear interpolation and the target query vector (Object Query), which can be specifically expressed as:

[0088] f kli =f bilinear (Fk ,c li )

[0089] q (l+1)i =f li +q li

[0090] Among them, F k represents the kth prediction feature layer generated by the feature pyramid network, c li represents the center point of the 2D target, i represents the target index in the current image, l represents the number of layers of the Transformer decoder, and the sampled features calculated by bilinear interpolation are recorded as f kli ; The target query vector q of each layer is obtained by fusing the target query vector of the previous layer and the sampled feature f, f bilinear Represents bilinear interpolation calculation, f li represents the sampling characteristics, q li represents the target query vector of the lth decoding layer, q (l+1)i represents the (l+1)th layer target query vector.

[0091] In the Transformer decoder part, it is necessary to randomly initialize and generate an N×256-dimensional target query vector (Object Query). In practical applications, N is larger than the actual target to be detected in the image, and can usually be set to N=300. In the present invention, the Transformer decoder module repeatedly stacks L layers, which can be set to L=6 according to experience. The attention mechanism operation of each layer and the input and output of the module are the same. First, the target query vector obtained by random initialization communicates with each other through a multi-head self-attention mechanism, which facilitates further feature extraction on the 2D feature map. Secondly, the multi-scale feature map generated by the feature pyramid obtains the sampling features through the projection matrix, and then interacts with the target query vector in two ways. The first way is to sample the features f li Directly add it to the target query vector to obtain the target query vector after feature fusion. The second way is to use it as the key vector (Key) and the value vector (Value) to perform a multi-head cross-attention operation with the target query vector. The calculation of the attention mechanism is as follows:

[0092] [Q; K; V] = [W'1(f l +q l );W'2f l ;W'3f l ]

[0093] MultiHead(Q,K,V)=Concat(head1,...,head n )W o

[0094] head i =Attention(Q,K,V)

[0095] In the above formula, W'1, W'2, W'3 and W o are the target query vector, key vector, value vector and parameter matrix of the multi-head attention mechanism, respectively, l is the sampling feature matrix, Q, K, and V represent the target query space, key space, and value space respectively. l represents the target query vector, MultiHead(Q,K,V) represents the multi-head attention mechanism operation, Concat(head1,...,head n ) indicates a cascade operation on n attention heads, head1 indicates the first attention head, head n represents the nth attention head.

[0096] After decoding by the L-layer decoder module, each target query vector is predicted by the feed-forward neural network (FFN) to generate the 8 corner point coordinates and category index of the corresponding 3D detection box. The prediction result will calculate the loss with the actual label and back-propagate to update the overall network parameters.

[0097] Since the present invention adopts a deep neural network based on the Transformer structure, the model prediction result is a fixed set of N predicted values, and the model does not introduce the non-maximum suppression algorithm (NMS). It is necessary to use bipartite graph matching to find the only corresponding ground truth for the predicted 3D bounding box. Therefore, the Hungarian algorithm is used to solve the bipartite graph matching problem and calculate the final loss function. The calculation formula is as follows:

[0098]

[0099] in, It is the minimum match between the predicted bounding box and the Ground Truth calculated by the Hungarian algorithm, and its loss value is determined by the predicted category loss and 3D object box L1 loss Decide together and use it after finding the minimum match Compute the total loss of the model.

[0100] S4, obtaining 2D image data collected for the area to be detected, inputting the 2D image data into a trained 3D object detection model, and outputting a 3D detection frame corresponding to each 2D image data.

[0101] Furthermore, the 2D data set is determined in the following manner:

[0102] Determined by the historical 2D dataset corresponding to the training area;

[0103] Alternatively, based on the image data collected by the roadside monocular camera for the training area, the image data is calibrated, and a 2D target detection frame corresponding to each image data is generated and then stored and determined.

[0104] Furthermore, the process of training the 3D object detection model is as follows:

[0105] The input RGB image is feature extracted through a convolutional neural network and a feature pyramid network to obtain four predicted feature maps of different sizes. The projection matrix is ​​used to perform feature sampling in each predicted feature map, and the sampling results are input into a decoder to generate a prediction result, which includes eight corner point coordinates and a category index of a 3D detection box corresponding to each 2D target detection box in the RGB image. The loss value is calculated based on the prediction result and the actual label corresponding to the RGB image, and the network parameters of the 3D target detection model are updated by back propagation until the loss value meets the preset conditions, the 3D target detection model is completed, and a trained 3D target detection model is obtained.

[0106] Furthermore, the process of performing feature sampling in each predicted feature map using the projection matrix is ​​specifically as follows:

[0107] The center point of any 2D target detection box is projected onto predicted feature maps of different sizes. The size of the 2D target detection box is used to perform pixel feature sampling on the area enclosed by the projection point corresponding to the center point and the preset distance as the radius. The sampled area is feature fused through bilinear interpolation and target query column vector to obtain the sampling result.

[0108] Furthermore, the loss value is calculated as follows:

[0109] The loss function is determined by the Hungarian algorithm, and the loss function is specifically:

[0110]

[0111] in, is the total loss value, is the minimum match value between the predicted bounding box and the Ground Truth calculated by the Hungarian algorithm, Represents all permutations and combinations of N prediction results and Ground Truth, i.e., the solution space N represents the number of prediction targets, y i represents the true value of the target, which includes the category to which the target belongs and the coordinates of the corner points of the 3D bounding box, represents the target prediction value, c i represents the target category, Indicates no goal, represents the prediction category loss, b i represents the true target bounding box coordinates, represents the predicted target bounding box coordinates, represents the bounding box loss, represents the classification loss, Represents the predicted target bounding box corresponding to the true target under the minimum matching arrangement.

[0112] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0113] For ease of understanding, this solution is now explained through the following examples:

[0114] 1. Use a roadside monocular camera to collect image data of different scenes. The sensor type is 1 / 1.8” CMOS, the sampling frame rate is 30-60Hz, the image format is RGB format, and it is compressed and saved as a JPEG image at a resolution of 1920×1080. In addition, the existing 2D target detection dataset can also be used; obtain the 2D detection box annotation information of the target to be detected in the image, including the category and coordinate representation, recorded as (x, y, w, h), where x and y represent the upper left corner of the 2D target detection box, and w and h represent the width and height of the target respectively;

[0115] 2. Obtain the external parameter matrix to realize the conversion from world coordinates to camera coordinate system. The external parameter matrix is ​​described by the orthogonal rotation matrix R and the translation vector T, which can be expressed as

[0116]

[0117] 3. Get the intrinsic parameter matrix, prepare a chessboard of known size, take multiple chessboard images, use image processing algorithms (such as the findChessboardCorners() function provided by OpenCV) to automatically detect chessboard corners, and pass the detected corner data to the camera calibration function, which will calculate the camera's internal parameters based on the corner data, including the intrinsic parameter matrix and distortion coefficient. The intrinsic parameter matrix is ​​expressed as Denoted as:

[0118]

[0119] where f x and f y They represent the pixel length of the focal length in the x-axis and y-axis directions respectively, u0 and v0 represent the actual positions of the principal points, and the intrinsic parameter matrix can realize the conversion from the camera coordinate system to the image pixel coordinate system.

[0120] Combining steps 2 and 3, the transformation from the world coordinate system to the pixel coordinate system depends on the camera's intrinsic parameter matrix and extrinsic parameter matrix, which can be specifically expressed as:

[0121]

[0122] Create an auxiliary information file corresponding to each image, and generate a projection matrix in the subsequent model training process. The content is: (cls, x, y, w, h, ratio, alpha, rotation_y, camera_h, angle), where cls represents the target category. The data set in this invention contains three major categories, including motor vehicles, pedestrians and non-motor vehicles, which are subdivided into seven subcategories, namely car, truck, bus, person, cyclist, motorcyclist, tricyclist. The target category will be encoded in the data fusion stage. The code is a one-hot vector of size 7×7; ratio is the aspect ratio of the 2D target detection box; alpha is the observation angle of the object, ranging from -π to +π. In the camera coordinate system, the camera origin is taken as the center, and the line connecting the camera origin to the object center is the radius. The object is rotated around the y-axis to the camera z-axis. At this time, the angle between the object direction and the camera x-axis is; rotation_y is the spatial direction of the target. In the camera coordinate system, the angle between the object's forward direction and the camera coordinate system x-axis is in the range of -π to +π; camera_h and angle represent the camera installation height and pitch angle, respectively.

[0123] Use the Pytorch deep learning framework to load the RGB image data, 3D label file, and auxiliary information file prepared in the above steps, instantiate a deep neural network based on the Transformer structure, train the network model until the loss function converges, and generate a pth model file for persistent storage;

[0124] The model is called to perform reasoning on the dataset to be tested, and the test results are saved as a txt file, which includes (cls_id, x1, y1, x2, y2......x8, y8), cls_id is the target category, and the last 16 points represent the image coordinates of the 8 corner points (x, y) of the predicted 3D annotation box, completing the 3D target annotation. Optionally, if the dataset to be tested cannot obtain auxiliary information such as camera height, pitch angle, and target posture during the reasoning stage, then only providing the 2D target detection box coordinate position and category label can still be used to annotate the target in 3D, and the unknown information is filled with 0. Since the transformation matrix of static constant information is a learnable parameter matrix, even if some information is missing, the projection matrix can still extract the target area in the image to achieve a more ideal 3D target detection, which further provides the convenience of data collection and can be applied to actual annotation tasks on a large scale.

[0125] The present invention also provides a monocular vision 3D target labeling system based on multimodal data, and the specific technical solution is as follows:

[0126] The acquisition module is used to: acquire a 2D data set generated for a training area, wherein the 2D data set includes at least one RGB image, an auxiliary information file corresponding to each RGB image, and a coordinate position of a 2D target detection frame corresponding to each RGB image;

[0127] The determination module is used to: determine the camera extrinsic parameters corresponding to the camera that generates the 2D data set, calibrate the intrinsic parameters of the camera by a checkerboard detection method, and determine the camera matrix corresponding to the camera in combination with the camera extrinsic parameters;

[0128] The training module is used to: determine the projection matrix of the training model in combination with the auxiliary information file corresponding to each RGB image, and train the 3D object detection model based on the projection matrix, in combination with all RGB images, the coordinate position of the 2D object detection box corresponding to each RGB image, and the camera matrix;

[0129] The labeling module is used to obtain 2D image data collected for the area to be detected, input the 2D image data into a trained 3D object detection model, and output a 3D detection box corresponding to each 2D image data.

[0130] It should be noted that the beneficial effects of the monocular vision 3D target labeling system based on multimodal data provided by the above embodiment are the same as the beneficial effects of the above monocular vision 3D target labeling method based on multimodal data, which will not be repeated here. In addition, when the system provided by the above embodiment realizes its functions, it only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0131] like Figure 3 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above methods, specifically:

[0132] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements a monocular vision 3D target labeling method based on multimodal data provided in the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for implementing device functions, which will not be repeated here.

[0133] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any one of the above methods.

[0134] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0135] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above methods.

[0136] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0137] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0138] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0139] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A monocular vision 3D target labeling method based on multimodal data, characterized in that: include: Obtain a 2D data set generated for the training area, wherein the 2D data set includes at least one RGB image, an auxiliary information file corresponding to each RGB image, and a coordinate position of a 2D target detection box corresponding to each RGB image; Determine the camera extrinsic parameters corresponding to the camera that generates the 2D data set, perform intrinsic parameter calibration on the camera by means of a checkerboard detection method, and determine the camera matrix corresponding to the camera in combination with the camera extrinsic parameters; Combined with the auxiliary information file corresponding to each RGB image, determine the projection matrix of the training model, and based on the projection matrix, combine all RGB images, the coordinate position of the 2D target detection box corresponding to each RGB image, and the camera matrix, train the 3D target detection model; The 2D image data collected for the area to be detected is obtained, the 2D image data is input into a trained 3D object detection model, and a 3D detection frame corresponding to each 2D image data is output.

2. The monocular vision 3D target labeling method based on multimodal data according to claim 1, characterized in that: The 2D data set is determined in the following manner: Determined by the historical 2D dataset corresponding to the training area; Alternatively, based on the image data collected by the roadside monocular camera for the training area, the image data is calibrated, and a 2D target detection frame corresponding to each image data is generated and then stored and determined.

3. The monocular vision 3D target labeling method based on multimodal data according to claim 1, characterized in that: The process of training a 3D object detection model is as follows: The input RGB image is feature extracted through a convolutional neural network and a feature pyramid network to obtain four predicted feature maps of different sizes. The projection matrix is ​​used to perform feature sampling in each predicted feature map, and the sampling results are input into a decoder to generate a prediction result, which includes eight corner point coordinates and a category index of a 3D detection box corresponding to each 2D target detection box in the RGB image. The loss value is calculated based on the prediction result and the actual label corresponding to the RGB image, and the network parameters of the 3D target detection model are updated by back propagation until the loss value meets the preset conditions, the 3D target detection model is completed, and a trained 3D target detection model is obtained.

4. The monocular vision 3D target labeling method based on multimodal data according to claim 3, characterized in that: Using the projection matrix, the process of performing feature sampling in each predicted feature map is specifically as follows: The center point of any 2D target detection box is projected onto predicted feature maps of different sizes. The size of the 2D target detection box is used to perform pixel feature sampling on the area enclosed by the projection point corresponding to the center point and the preset distance as the radius. The sampled area is feature fused through bilinear interpolation and target query column vector to obtain the sampling result.

5. The monocular vision 3D target labeling method based on multimodal data according to claim 3, characterized in that: The loss value is calculated as follows: The loss function is determined by the Hungarian algorithm, and the loss function is specifically: in, is the total loss value, is the minimum match value between the predicted bounding box and the Ground Truth calculated by the Hungarian algorithm, Represents all permutations and combinations of N prediction results and Ground Truth, i.e., the solution space N represents the number of prediction targets, y i represents the true value of the target, which includes the category to which the target belongs and the coordinates of the corner points of the 3D bounding box, represents the target prediction value, c i represents the target category, Means no target, represents the prediction category loss, b i represents the true target bounding box coordinates, represents the predicted target bounding box coordinates, represents the bounding box loss, represents the classification loss, Represents the predicted target bounding box corresponding to the true target under the minimum matching arrangement.

6. A monocular vision 3D target labeling system based on multimodal data, characterized in that: include: The acquisition module is used to: acquire a 2D data set generated for a training area, wherein the 2D data set includes at least one RGB image, an auxiliary information file corresponding to each RGB image, and a coordinate position of a 2D target detection frame corresponding to each RGB image; The determination module is used to: determine the camera extrinsic parameters corresponding to the camera that generates the 2D data set, calibrate the intrinsic parameters of the camera by a checkerboard detection method, and determine the camera matrix corresponding to the camera in combination with the camera extrinsic parameters; The training module is used to: determine the projection matrix of the training model in combination with the auxiliary information file corresponding to each RGB image, and train the 3D object detection model based on the projection matrix, in combination with all RGB images, the coordinate position of the 2D object detection box corresponding to each RGB image, and the camera matrix; The labeling module is used to obtain 2D image data collected for the area to be detected, input the 2D image data into a trained 3D object detection model, and output a 3D detection box corresponding to each 2D image data.

7. The monocular vision 3D target labeling system based on multimodal data according to claim 6, characterized in that: The 2D data set is determined in the following manner: Determined by the historical 2D dataset corresponding to the training area; Alternatively, based on the image data collected by the roadside monocular camera for the training area, the image data is calibrated, and a 2D target detection frame corresponding to each image data is generated and then stored and determined.

8. The monocular vision 3D target labeling system based on multimodal data according to claim 6, characterized in that: The process of training a 3D object detection model is as follows: The input RGB image is feature extracted through a convolutional neural network and a feature pyramid network to obtain four predicted feature maps of different sizes. The projection matrix is ​​used to perform feature sampling in each predicted feature map, and the sampling results are input into a decoder to generate a prediction result, which includes eight corner point coordinates and a category index of a 3D detection box corresponding to each 2D target detection box in the RGB image. The loss value is calculated based on the prediction result and the actual label corresponding to the RGB image, and the network parameters of the 3D target detection model are updated by back propagation until the loss value meets the preset conditions, the 3D target detection model is completed, and a trained 3D target detection model is obtained.

9. An electronic device, characterized in that: The electronic device comprises a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements the method according to any one of claims 1 to 5.

Images