A vehicle detection method based on visual information and point cloud information
By combining visual information and point cloud information, a dual backbone object detection model is built, which solves the accuracy and small object detection problems of vehicle detection in tunnel environments, and achieves efficient vehicle identification in complex environments.
Patent Information
- Application Number
- CN202411466751.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2044-10-21
AI Technical Summary
In closed environments such as road tunnels, vehicle inspection is difficult to ensure accuracy due to uneven light and background complexity, and small-scale vehicles are prone to missed inspections.
Using vehicle detection methods based on visual information and point cloud information, by obtaining historical image data and radar point cloud data, converting them into image data, and building a dual backbone target detection model, using this model for training and detection to improve detection accuracy and robustness.
In complex environments, the accuracy and adaptability of vehicle detection are improved, the detection accuracy of small targets is enhanced, and effective detection of long-distance target vehicles in traffic scenarios is achieved.
Smart Images
Figure CN118968473B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of vehicle detection, and in particular to a vehicle detection method based on visual information and point cloud information. Background Art
[0002] With the rapid development of the economy, the number of cars has continued to increase, bringing tremendous pressure to the monitoring and management of the transportation system. The consequences of traffic accidents in closed environments such as highway tunnels are disastrous, so it is necessary to improve the level of traffic safety management.
[0003] In a good road environment, vehicle detection has high accuracy and speed, but the detection effect in the special environment of a tunnel has not been considered. The main problems currently exist are:
[0004] The tunnel has uneven lighting and complex background, making it difficult to ensure the accuracy of vehicle detection;
[0005] Small-scale vehicles have a high rate of missed detection;
[0006] Therefore, to address the above problems, a vehicle detection method based on visual information and point cloud information is proposed. Summary of the invention
[0007] The purpose of the present invention is to provide a vehicle detection method based on visual information and point cloud information to solve the problems raised in the above background technology.
[0008] To achieve the above object, the present invention provides the following technical solutions:
[0009] A vehicle detection method based on visual information and point cloud information comprises the following steps:
[0010] S1. Obtain historical image data I and historical radar point cloud data of vehicles in the target scene, and convert the historical radar point cloud data into image data II;
[0011] S2. construct a dual-backbone target detection model, which includes a backbone network, a neck network and a detection head, wherein the output end of the backbone network is connected to the input end of the neck network, and the output end of the neck network is connected to the input end of the detection head;
[0012] S3. Inputting historical image data I and image data II into the dual-backbone target detection model for training;
[0013] S4. Determine whether the dual-backbone target detection model is trained. If so, proceed to step S5; if not, update the parameters of the dual-backbone target detection model and return to step S3;
[0014] S5. Input the real-time image data I and image data II of the vehicle in the target scene into the trained dual-backbone target detection model to obtain the detection result.
[0015] Preferably, in step S1, converting the radar point cloud data into image data II comprises the following steps:
[0016] S11. Obtain the depth d, radial velocity v and radar cross-sectional area rcs in the radar point cloud data;
[0017] S12. Map the depth d, radial velocity v and radar cross-sectional area rcs to the R channel, the G channel and the B channel respectively to obtain a pixel map;
[0018] S13. Determine the height and width weight of the image according to the depth d and the radar cross section rcs;
[0019] S14. Stretch the pixel image to a height of height and a width of weight to obtain image data II.
[0020] Preferably, the depth d, radial velocity v and radar cross-sectional area rcs are mapped to the R channel, the G channel and the B channel respectively by the following formula:
[0021] ;
[0022] ;
[0023] ;
[0024] Among them, d represents depth, v represents radial velocity, and rcs represents radar cross-section.
[0025] Preferably, the height and width weight are calculated using the following formula:
[0026] ;
[0027] ;
[0028] Where d represents depth and rcs represents radar cross section.
[0029] Preferably, the backbone network includes two parallel backbone networks and four splicing modules, the two backbone networks have the same structure, and one backbone network includes an initialization module, four feature extraction modules and a spatial pyramid pooling module in sequence; image data I is input into the first backbone network, and image data II is input into the second backbone network, and the output ends of the first feature extraction module of the first backbone network and the first feature extraction module of the second backbone network are connected to the input end of the first splicing module; the output ends of the second feature extraction module of the first backbone network and the second feature extraction module of the second backbone network are connected to the input end of the second splicing module; the output ends of the third feature extraction module of the first backbone network and the third feature extraction module of the second backbone network are connected to the input end of the third splicing module; the output ends of the spatial pyramid pooling module of the first backbone network and the spatial pyramid pooling module of the second backbone network are connected to the input end of the fourth splicing module; the feature extraction module includes a Clo module and a convolutional FFN module in sequence.
[0030] Preferably, the neck network includes three upsampling modules, six splicing modules, six C2f modules and three CBS modules; the output end of the fourth splicing module of the backbone network is connected to the input end of the first upsampling module, the output end of the first upsampling module is connected to the input end of the first splicing module, the output end of the third splicing module in the backbone network and the output end of the third feature extraction module of the second backbone network are connected to the input end of the first splicing module, the output end of the first splicing module is connected to the input end of the first C2f module, the output end of the first C2f module is connected to the input end of the second upsampling module, the output end of the second upsampling module is connected to the input end of the second splicing module, the output end of the second splicing module in the backbone network and the output end of the second feature extraction module of the second backbone network are connected to the input end of the second splicing module, the output end of the second splicing module is connected to the input end of the second C2f module, the output end of the second C2f module is connected to the input end of the third upsampling module, and the output end of the third upsampling module is connected to the third splicing module. The input end of the block, the output end of the first splicing module in the backbone network and the output end of the first feature extraction module of the second backbone network are connected to the input end of the third splicing module, the output end of the third splicing module is connected to the input end of the third C2f module, the output end of the third C2f module is connected to the input end of the first CBS module, the output end of the first CBS module is connected to the input end of the fourth splicing module, the output end of the fourth splicing module is connected to the input end of the fourth C2f module, the output end of the fourth C2f module is connected to the input end of the second CBS module, the output end of the second CBS module is connected to the input end of the fifth splicing module, the output end of the fifth splicing module is connected to the input end of the fifth C2f module, the output end of the fifth C2f module is connected to the input end of the third CBS module, the output end of the third CBS module is connected to the input end of the sixth splicing module, the input end of the sixth splicing module is also connected to the output end of the fourth splicing module in the backbone network, and the output end of the sixth splicing module is connected to the input end of the sixth C2f module.
[0031] Preferably, the detection head includes four detection modules, the input end of the first detection module is connected to the output end of the third C2f module; the input end of the second detection module is connected to the output end of the fourth C2f module; the input end of the third detection module is connected to the output end of the fifth C2f module; the input end of the fourth detection module is connected to the output end of the sixth C2f module; one detection module includes a parallel size output module and a category output module; the size output module includes a CBS module and a 1×1 convolution module in sequence, and the size output module is used to control the size of the prediction image output; the category output module includes a CBS module and a 1×1 convolution module in sequence, and the category output module is used to determine the category of the input image.
[0032] Preferably, the dual-backbone target detection model adopts the loss function To train:
[0033] ;
[0034] ;
[0035] ;
[0036] ;
[0037] ;
[0038] ;
[0039] Among them, IoU represents the intersection-over-union ratio between the predicted box and the real box, ( , )and( , ) represent the upper left corner coordinates and lower right corner coordinates of the prediction box respectively, ( , )and( , ) represent the coordinates of the upper left corner and lower right corner of the real box, w and h represent the width and height of the input image, respectively. and Represent the height of the real box and the predicted box respectively, and Represent the width of the real box and the predicted box respectively;
[0040] When the dual-backbone target detection model is trained to a preset fitting degree, the dual-backbone target detection model training is completed.
[0041] Compared with the prior art, the present invention has the following beneficial effects:
[0042] 1. Image information can provide rich visual features and semantic information, which is helpful for the recognition of shape, color, texture, etc.;
[0043] 2. Obtain object information and distance information in complex environments such as high light or low light through the point cloud information provided by the radar, further convert the point cloud information into image information, and input the image information and the image information converted from the point cloud into the dual-backbone target detection network, which can improve the accuracy, robustness and adaptability of target detection;
[0044] 3. The use of the group backbone network in the dual-backbone target detection model can avoid the computational redundancy of complex models for simple tasks and improve the detection speed of the model;
[0045] 4. The neck network in the dual-backbone target detection model can make the extracted feature image have more detailed information of the original image, and can also enhance the feature fusion ability of the model, so as to better utilize the feature information at different scales, improve the detection accuracy of small targets in dense traffic, and achieve good detection of target vehicles far away from the surveillance camera in traffic scenes;
[0046] 5. A specific loss function is used for training to penalize the differences in height and width between the predicted box and the true box. Not only the absolute difference but also the relative proportion is considered, thereby strengthening the constraints on height and width and achieving a more detailed penalty for the difference in shape between the predicted box and the true box. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 This is a flow chart of a vehicle detection method based on visual information and point cloud information of the present invention;
[0048] Figure 2 Images converted from radar point cloud data;
[0049] Figure 3 Schematic diagram of the dual-backbone target detection model. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0051] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0052] Example:
[0053] See also Figure 1-3 , this embodiment provides a technical solution:
[0054] A vehicle detection method based on visual information and point cloud information comprises the following steps:
[0055] S1. Obtain historical image data I and historical radar point cloud data of vehicles in the target scene, and convert the historical radar point cloud data into image data II;
[0056] Among them, converting radar point cloud data into image data II includes the following steps:
[0057] S11. Obtain the depth d, radial velocity v and radar cross-sectional area rcs in the radar point cloud data;
[0058] S12. Map the depth d, radial velocity v, and radar cross-sectional area rcs to the R channel, the G channel, and the B channel, respectively, to obtain a pixel map. The depth d, radial velocity v, and radar cross-sectional area rcs are mapped to the R channel, the G channel, and the B channel, respectively, using the following formula:
[0059] ;
[0060] ;
[0061] ;
[0062] Where, d represents depth, v represents radial velocity, and rcs represents radar cross section;
[0063] S13. Determine the height and width weight of the image according to the depth d and the radar cross-sectional area rcs, and calculate the height and width weight by the following formula:
[0064] ;
[0065] ;
[0066] Where, d represents depth, rcs represents radar cross section;
[0067] S14. Stretch the pixel image to a height and a width, and obtain image data II;
[0068] S2. construct a dual-backbone target detection model, which includes a backbone network, a neck network and a detection head, wherein the output end of the backbone network is connected to the input end of the neck network, and the output end of the neck network is connected to the input end of the detection head;
[0069] Among them, the backbone network includes two parallel backbone networks and four splicing modules, the two backbone networks have the same structure, and one backbone network includes an initialization module, four feature extraction modules and a spatial pyramid pooling module in sequence; image data I is input into the first backbone network, and image data II is input into the second backbone network, and the output ends of the first feature extraction module of the first backbone network and the first feature extraction module of the second backbone network are connected to the input end of the first splicing module; the output ends of the second feature extraction module of the first backbone network and the second feature extraction module of the second backbone network are connected to the input end of the second splicing module; the output ends of the third feature extraction module of the first backbone network and the third feature extraction module of the second backbone network are connected to the input end of the third splicing module; the output ends of the spatial pyramid pooling module of the first backbone network and the spatial pyramid pooling module of the second backbone network are connected to the input end of the fourth splicing module; the feature extraction module includes a Clo module and a convolutional FFN module in sequence;
[0070] Further, the neck network includes three upsampling modules, six splicing modules, six C2f modules and three CBS modules; the output end of the fourth splicing module of the backbone network is connected to the input end of the first upsampling module, the output end of the first upsampling module is connected to the input end of the first splicing module, the output end of the third splicing module in the backbone network and the output end of the third feature extraction module of the second backbone network are connected to the input end of the first splicing module, the output end of the first splicing module is connected to the input end of the first C2f module, the output end of the first C2f module is connected to the input end of the second upsampling module, the output end of the second upsampling module is connected to the input end of the second splicing module, the output end of the second splicing module in the backbone network and the output end of the second feature extraction module of the second backbone network are connected to the input end of the second splicing module, the output end of the second splicing module is connected to the input end of the second C2f module, the output end of the second C2f module is connected to the input end of the third upsampling module, and the output end of the third upsampling module is connected to the third splicing module. The output end of the first splicing module in the backbone network and the output end of the first feature extraction module of the second backbone network are connected to the input end of the third splicing module, the output end of the third splicing module is connected to the input end of the third C2f module, the output end of the third C2f module is connected to the input end of the first CBS module, the output end of the first CBS module is connected to the input end of the fourth splicing module, the output end of the fourth splicing module is connected to the input end of the fourth C2f module, the output end of the fourth C2f module is connected to the input end of the second CBS module, the output end of the second CBS module is connected to the input end of the fifth splicing module, the output end of the fifth splicing module is connected to the input end of the fifth C2f module, the output end of the fifth C2f module is connected to the input end of the third CBS module, the output end of the third CBS module is connected to the input end of the sixth splicing module, the input end of the sixth splicing module is also connected to the output end of the fourth splicing module in the backbone network, and the output end of the sixth splicing module is connected to the input end of the sixth C2f module;
[0071] The detection head includes four detection modules, the input end of the first detection module is connected to the output end of the third C2f module; the input end of the second detection module is connected to the output end of the fourth C2f module; the input end of the third detection module is connected to the output end of the fifth C2f module; the input end of the fourth detection module is connected to the output end of the sixth C2f module; a detection module includes a parallel size output module and a category output module; the size output module includes a CBS module and a 1×1 convolution module in sequence, and the size output module is used to control the size of the prediction image output; the category output module includes a CBS module and a 1×1 convolution module in sequence, and the category output module is used to judge the category of the input image;
[0072] S3. Input historical image data I and image data II into the dual-backbone target detection model for training. Specifically, the dual-backbone target detection model uses a loss function To train:
[0073] ;
[0074] ;
[0075] ;
[0076] ;
[0077] ;
[0078] ;
[0079] Among them, IoU represents the intersection-over-union ratio between the predicted box and the real box, ( , )and( , ) represent the upper left corner coordinates and lower right corner coordinates of the prediction box respectively, ( , )and( , ) represent the coordinates of the upper left corner and lower right corner of the real box, w and h represent the width and height of the input image, respectively. and Represent the height of the real box and the predicted box respectively, and Represent the width of the real box and the predicted box respectively;
[0080] S4. Determine whether the dual-backbone target detection model is trained. If so, proceed to step S5; if not, update the parameters of the dual-backbone target detection model and return to step S3;
[0081] When the dual-backbone target detection model is trained to a preset fitting degree, the dual-backbone target detection model training is completed;
[0082] S5. Input the real-time vehicle image data I and image data II in the target scene into the trained dual-backbone target detection model to obtain the detection results; by inputting the image information and the image information converted from the radar point cloud data into the dual-backbone target detection network, more feature information can be obtained, thereby improving the accuracy, robustness and adaptability of target detection.
[0083] The present invention will be further described below in conjunction with the accompanying drawings:
[0084] See also Figure 1-3 , including the following steps:
[0085] S1. Obtain historical image data I and historical radar point cloud data of vehicles in the target scene, and convert the historical radar point cloud data into image data II;
[0086] S2. Build a dual-backbone target detection model;
[0087] S3. Inputting historical image data I and image data II into the dual-backbone target detection model for training;
[0088] S4. Determine whether the dual-backbone target detection model is trained. If so, proceed to step S5; if not, update the parameters of the dual-backbone target detection model and return to step S3;
[0089] S5. Input the real-time vehicle image data I and image data II in the target scene into the trained dual-backbone target detection model to obtain the detection result. By inputting the image information and the image information converted from the radar point cloud data into the dual-backbone target detection network, more feature information can be obtained, thereby improving the accuracy, robustness and adaptability of target detection.
[0090] In this embodiment, in step S1, historical image data I and historical radar point cloud data of the vehicle in the target scene are obtained, and the historical radar point cloud data are converted into image data II; the image data I is preferably obtained by taking a camera; the radar point cloud data is preferably obtained by using a millimeter wave radar;
[0091] Since radar and camera are different types of sensors, the camera collects RGB format image files, while the millimeter-wave radar collects point cloud files containing information such as target speed, distance, and radar scattering cross-sectional area. In order for the dual-backbone target detection model to extract the feature information of the radar point cloud, the millimeter-wave radar point cloud information needs to be converted into pixel information in the image.
[0092] Convert radar point cloud data into image data by following the steps below Ⅱ:
[0093] S11. Obtain the depth d, radial velocity v and radar cross-sectional area rcs in the radar point cloud data;
[0094] Before projecting the millimeter-wave radar point cloud information to generate a point cloud image, the point cloud needs to be filtered according to the actual conditions and differences in sensor performance. The filtering range is set according to demand; for example, the vehicle speed v in the tunnel is generally not more than 33m / s, and the radar scattering cross-sectional area of the vehicle is generally 3-30dBsm (decibel square meter). Therefore, before projecting the point cloud image, the radar point cloud data exceeding 3-30dBsm is filtered out before point cloud mapping;
[0095] S12. Map the depth d, radial velocity v and radar cross-sectional area rcs to the R channel, G channel and B channel respectively. For unknown areas without radar points, set the pixel values in different channels to 0. Obtain the RGB pixel map by the above method, as shown in Figure 2 As shown in (a);
[0096] The depth d, radial velocity v and radar cross-section rcs are mapped to the R channel, G channel and B channel respectively by the following formula:
[0097] ;
[0098] ;
[0099] ;
[0100] Where, d represents depth, v represents radial velocity, and rcs represents radar cross section;
[0101] S13. Determine the height and width weight of the image according to the depth d and the radar cross section rcs;
[0102] The height and width weight are calculated using the following formula:
[0103] ;
[0104] ;
[0105] Where, d represents depth, rcs represents radar cross section;
[0106] Considering that the point cloud of the millimeter-wave radar does not reflect the size information of the object, it increases the difficulty of fusing the already sparse radar point cloud information. In order to make the radar point cloud reflect the height information of the detected vehicle to a certain extent, so as to improve the matching degree between the camera and the point cloud image; the height and width of the point cloud are expanded according to the distance and radar scattering cross-sectional area information contained in the radar point cloud, so that the point cloud height is small at a long distance and large at a short distance. After processing all radar points within the position range of the front camera image plane, a radar image with the same size and resolution as the front camera can be generated;
[0107] S14. Stretch the pixel map to height and width weight, such as Figure 2 As shown in (b), image data II is obtained.
[0108] Through the above method, the radar point cloud information can reflect the size of the object, and the features in the radar point cloud data can be converted into a form that can be learned by the dual-backbone target detection model.
[0109] In this embodiment, in step S2, a dual-backbone target detection model is constructed;
[0110] The dual-backbone target detection model includes a backbone network (Backbone), a neck network (Neck) and a detection head (Head). The output of the backbone network is connected to the input of the neck network, and the output of the neck network is connected to the input of the detection head; Figure 3 As shown, Figure 3 The MPCM (Mask of point cloud mapping) in the image represents the mask of point cloud mapping, which is the image data II. Figure 3 Image in it represents image, which is image data I;
[0111] The backbone network includes two parallel backbone networks and four concatenation modules (Concat modules). The two backbone networks have the same structure. One backbone network includes an initialization module (Steam module), four feature extraction modules and a spatial pyramid pooling module (Spatial Pyramid Pooling, SPPF module) in sequence.
[0112] Among them, image data I is input into the first backbone network, image data II is input into the second backbone network, the output ends of the first feature extraction module of the first backbone network and the first feature extraction module of the second backbone network are connected to the input end of the first splicing module; the output ends of the second feature extraction module of the first backbone network and the second feature extraction module of the second backbone network are connected to the input end of the second splicing module; the output ends of the third feature extraction module of the first backbone network and the third feature extraction module of the second backbone network are connected to the input end of the third splicing module; the output ends of the spatial pyramid pooling module of the first backbone network and the spatial pyramid pooling module of the second backbone network are connected to the input end of the fourth splicing module; the two backbone networks will receive fusion features of different depths, and then perform all-round feature fusion through the splicing module to generate four sets of features with different resolutions;
[0113] The feature extraction module includes a Clo block and a convolutional FFN module (ConvFFN module) in sequence.
[0114] The neck network includes three upsampling modules (Upsample), six splicing modules, six C2f modules (CrossStage Partial Network Fusion, C2f) and three CBS modules (Convolutional-BatchNormalization-SiLU). The structure of the CBS module and the C2f module, as well as the Bottleneck in the C2f module, is as follows: Figure 3 As shown in the upper right corner;
[0115] Among them, the output end of the fourth splicing module of the backbone network is connected to the input end of the first upsampling module, the output end of the first upsampling module is connected to the input end of the first splicing module, the output end of the third splicing module in the backbone network and the output end of the third feature extraction module of the second backbone network are connected to the input end of the first splicing module, the output end of the first splicing module is connected to the input end of the first C2f module, the output end of the first C2f module is connected to the input end of the second upsampling module, the output end of the second upsampling module is connected to the input end of the second splicing module, the output end of the second splicing module in the backbone network and the output end of the second feature extraction module of the second backbone network are connected to the input end of the second splicing module, the output end of the second splicing module is connected to the input end of the second C2f module, the output end of the second C2f module is connected to the input end of the third upsampling module, the output end of the third upsampling module is connected to the input end of the third splicing module, and the output end of the first splicing module in the backbone network is connected to the input end of the third splicing module. The output end and the output end of the first feature extraction module of the second backbone network are connected to the input end of the third splicing module, the output end of the third splicing module is connected to the input end of the third C2f module, the output end of the third C2f module is connected to the input end of the first CBS module, the output end of the first CBS module is connected to the input end of the fourth splicing module, the output end of the fourth splicing module is connected to the input end of the fourth C2f module, the output end of the fourth C2f module is connected to the input end of the second CBS module, the output end of the second CBS module is connected to the input end of the fifth splicing module, the output end of the fifth splicing module is connected to the input end of the fifth C2f module, the output end of the fifth C2f module is connected to the input end of the third CBS module, the output end of the third CBS module is connected to the input end of the sixth splicing module, the input end of the sixth splicing module is also connected to the output end of the fourth splicing module in the backbone network, and the output end of the sixth splicing module is connected to the input end of the sixth C2f module.
[0116] The detection head includes four detection modules, the input end of the first detection module is connected to the output end of the third C2f module; the input end of the second detection module is connected to the output end of the fourth C2f module; the input end of the third detection module is connected to the output end of the fifth C2f module; the input end of the fourth detection module is connected to the output end of the sixth C2f module;
[0117] Among them, a detection module includes a parallel size output module and a category output module; the size output module includes a CBS module and a 1×1 convolution module in sequence, and the size output module is used to control the size of the prediction image output; the category output module includes a CBS module and a 1×1 convolution module in sequence, and the category output module is used to determine the category of the input image;
[0118] The splicing module, initialization module, Clo module, convolutional FFN module, upsampling module and spatial pyramid pooling module in this application are all existing modules and will not be described in detail here.
[0119] The use of the group backbone network in the dual-backbone target detection model of the present application can avoid the computational redundancy of complex models for simple tasks and improve the detection speed of the model.
[0120] The use of the neck network in the dual-backbone target detection model of the present application can make the extracted feature image have more detailed information of the original image, and can also enhance the feature fusion capability of the model, so as to better utilize feature information at different scales, improve the detection accuracy of small targets in dense traffic flow, and achieve good detection of target vehicles that are far away from the surveillance camera in traffic scenarios.
[0121] In this embodiment, in step S3, the historical image data I and the image data II are input into the dual-backbone target detection model for training;
[0122] The Adam optimizer is used for training, and the initial learning rate is 1.25×10 -4 , the batch size (batch_size) is set to 32, and the loss function is used Train the dual-backbone object detection model;
[0123] ;
[0124] ;
[0125] ;
[0126] ;
[0127] ;
[0128] ;
[0129] Among them, IoU represents the intersection-over-union ratio between the predicted box and the real box, ( , )and( , ) represent the upper left corner coordinates and lower right corner coordinates of the prediction box respectively, ( , )and( , ) represent the coordinates of the upper left corner and lower right corner of the real box, w and h represent the width and height of the input image, respectively. and Represent the height of the real box and the predicted box respectively, and Represent the width of the true box and the predicted box respectively.
[0130] The above loss function is used for training to penalize the differences in height and width between the predicted box and the true box. Not only the absolute difference but also the relative proportion is considered, thereby strengthening the constraints on height and width to more finely penalize the differences in shape between the predicted box and the true box.
[0131] In this embodiment, in step S4, it is determined whether the dual-backbone target detection model is trained. If so, the process proceeds to step S5; if not, the parameters of the dual-backbone target detection model are updated, and the process returns to step S3;
[0132] When the dual-backbone target detection model is trained to a preset fitting degree, the dual-backbone target detection model training is completed, the preset fitting degree is set according to demand, and the dual-backbone target detection model is preferably no longer converged.
[0133] In this embodiment, in step S5, image data I and radar point cloud data of the vehicle in the target scene are acquired in real time, and the radar point cloud data is converted into image data II;
[0134] Image data I and image data II are input into the trained dual-backbone target detection model to obtain the vehicle information in the target scene.
[0135] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A vehicle detection method based on visual information and point cloud information, characterized in that: The following steps are involved: S1. Acquire historical image data I and historical radar point cloud data of the vehicle in the target scene, and convert the historical radar point cloud data into image data II; converting the radar point cloud data into image data II includes the following steps: S11. Obtain the depth d, radial velocity v and radar cross-sectional area rcs in the radar point cloud data; S12. Map the depth d, radial velocity v and radar cross-sectional area rcs to the R channel, the G channel and the B channel respectively to obtain a pixel map; S13. Determine the height and width weight of the image according to the depth d and the radar cross section rcs; S14. Stretch the pixel image to a height and a width, and obtain image data II; S2. construct a dual-backbone target detection model, which includes a backbone network, a neck network and a detection head, wherein the output end of the backbone network is connected to the input end of the neck network, and the output end of the neck network is connected to the input end of the detection head; The backbone network includes two parallel backbone networks and four splicing modules, the two backbone networks have the same structure, and one backbone network includes an initialization module, four feature extraction modules and a spatial pyramid pooling module in sequence; image data I is input into the first backbone network, and image data II is input into the second backbone network, and the output ends of the first feature extraction module of the first backbone network and the first feature extraction module of the second backbone network are connected to the input end of the first splicing module; the output ends of the second feature extraction module of the first backbone network and the second feature extraction module of the second backbone network are connected to the input end of the second splicing module; the output ends of the third feature extraction module of the first backbone network and the third feature extraction module of the second backbone network are connected to the input end of the third splicing module; the output ends of the spatial pyramid pooling module of the first backbone network and the spatial pyramid pooling module of the second backbone network are connected to the input end of the fourth splicing module; the feature extraction module includes a Clo module and a convolutional FFN module in sequence; The neck network includes three upsampling modules, six splicing modules, six C2f modules and three CBS modules; the output end of the fourth splicing module of the backbone network is connected to the input end of the first upsampling module, the output end of the first upsampling module is connected to the input end of the first splicing module in the neck network, the output end of the third splicing module in the backbone network and the output end of the third feature extraction module of the second backbone network are connected to the input end of the first splicing module in the neck network, the output end of the first splicing module in the neck network is connected to the input end of the first C2f module, and the output end of the first C2f module is connected to the input end of the first C2f module. The output end of the second upsampling module is connected to the input end of the second splicing module in the neck network, the output end of the second splicing module in the backbone network and the output end of the second feature extraction module of the second backbone network are connected to the input end of the second splicing module in the neck network, the output end of the second splicing module in the neck network is connected to the input end of the second C2f module, the output end of the second C2f module is connected to the input end of the third upsampling module, the output end of the third upsampling module is connected to the input end of the third ... backbone network and the output end of the second feature extraction module of the second backbone network are connected to the input end of the second splicing module in the neck network, The output end of the first splicing module in the trunk network and the output end of the first feature extraction module of the second backbone network are connected to the input end of the third splicing module in the neck network, the output end of the third splicing module in the neck network is connected to the input end of the third C2f module, the output end of the third C2f module is connected to the input end of the first CBS module, the output end of the first CBS module is connected to the input end of the fourth splicing module in the neck network, the output end of the fourth splicing module in the neck network is connected to the input end of the fourth C2f module, and the output end of the fourth C2f module is connected to the second CB The input end of the S module, the output end of the second CBS module is connected to the input end of the fifth splicing module in the neck network, the output end of the fifth splicing module in the neck network is connected to the input end of the fifth C2f module, the output end of the fifth C2f module is connected to the input end of the third CBS module, the output end of the third CBS module is connected to the input end of the sixth splicing module in the neck network, the input end of the sixth splicing module in the neck network is also connected to the output end of the fourth splicing module in the backbone network, and the output end of the sixth splicing module in the neck network is connected to the input end of the sixth C2f module; S3. Inputting historical image data I and image data II into the dual-backbone target detection model for training; S4. Determine whether the dual-backbone target detection model is trained. If so, proceed to step S5; if not, update the parameters of the dual-backbone target detection model and return to step S3; S5. Input the real-time image data Ⅰ and image data Ⅱ of the vehicle in the target scene into the trained dual-backbone target detection model to obtain the detection result; The detection head includes four detection modules, the input end of the first detection module is connected to the output end of the third C2f module; the input end of the second detection module is connected to the output end of the fourth C2f module; the input end of the third detection module is connected to the output end of the fifth C2f module; the input end of the fourth detection module is connected to the output end of the sixth C2f module.
2. The vehicle detection method based on visual information and point cloud information according to claim 1, characterized in that: The depth d, radial velocity v and radar cross-section rcs are mapped to the R channel, G channel and B channel respectively by the following formula: ； ； ； Among them, d represents depth, v represents radial velocity, and rcs represents radar cross-section.
3. The vehicle detection method based on visual information and point cloud information according to claim 2, characterized in that: The height and width weight are calculated using the following formula: ； ； Where d represents depth and rcs represents radar cross section.
4. The vehicle detection method based on visual information and point cloud information according to claim 3, characterized in that: The first detection module includes a parallel size output module and a category output module; the size output module includes a CBS module and a 1×1 convolution module in sequence, and the size output module is used to control the size of the prediction image output; the category output module includes a CBS module and a 1×1 convolution module in sequence, and the category output module is used to determine the category of the input image.
5. The vehicle detection method based on visual information and point cloud information according to claim 4, characterized in that: The dual-backbone target detection model adopts the loss function To train: ； ； ； ； ； ； Among them, IoU represents the intersection-over-union ratio between the predicted box and the real box, ( , )and( , ) represent the upper left corner coordinates and lower right corner coordinates of the prediction box respectively, ( , )and( , ) represent the coordinates of the upper left corner and lower right corner of the real box, w and h represent the width and height of the input image, respectively. and Represent the height of the real box and the predicted box respectively, and Represent the width of the real box and the predicted box respectively; When the dual-backbone target detection model is trained to a preset fitting degree, the dual-backbone target detection model training is completed.