Target detection method and device, equipment and storage medium

Abstract

The invention discloses a target detection method, device and equipment and a storage medium, and the method comprises the steps: optimizing the depth prediction of an image based on the pose conversion between a current frame and a reference frame of a target image and self-supervised optimization, so as to output the depth estimation results of the current frame and the reference frame of the target image; outputting aerial view image features of the current frame and the reference frame according to the image features of the current frame and the reference frame and the depth estimation results of the current frame and the reference frame, and calculating sampling features of the reference frame in the current frame through mapping; splicing the sampling feature and the aerial view feature of the current frame to obtain a target aerial view feature, reconstructing the loss of the target aerial view feature and the aerial view feature obtained by the laser radar, and optimizing the image features of the current frame and the reference frame, depth estimation and the weight of the aerial view feature in the training process; and performing pre-training visual model enhancement based on the weight, and outputting a target detection result through the enhanced pre-training visual model.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a target detection method, device, equipment and storage medium. Background Art

[0002] The existing 3D target detection methods based on LiDAR sensors have high accuracy, but due to their high cost and limited performance in harsh environments, pure vision-based bird's eye view perception has gradually become a hot topic of research. Existing technologies such as BEVDet and BEVDepth realize the conversion of 2D images to 3D space through depth estimation and feature conversion. However, compared with LiDAR, the existing visual detection models still have insufficient detection accuracy and robustness.

[0003] Therefore, how to improve the accuracy of target detection is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the present invention is to provide a target detection method, device, equipment and storage medium, which can improve the detection performance of the model, improve the detection accuracy of the target, and reduce the cost.

[0005] In a first aspect, the present application provides a target detection method, wherein the method comprises the steps of:

[0006] Based on the pose conversion between the current frame and the reference frame of the target image and the self-supervised optimization, the depth prediction of the image is optimized to output the depth estimation results of the current frame and the reference frame of the target image;

[0007] Outputting bird's-eye view image features of the current frame and the reference frame according to the image features of the current frame and the reference frame and the depth estimation results of the current frame and the reference frame, and calculating the sampling features of the reference frame in the current frame through mapping;

[0008] The sampled features are concatenated with the bird's-eye view features of the current frame to obtain the target bird's-eye view features, and the target bird's-eye view image features are reconstructed with the bird's-eye view image features acquired by the lidar to optimize the weights of the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame during the training process;

[0009] The pre-trained visual model is enhanced based on the weights, and the target detection result is output through the enhanced pre-trained visual model.

[0010] In combination with the first aspect above, as an optional implementation method, based on a defined posture prediction network, a coordinate rotation change and a coordinate translation change between a current frame and a reference frame of a target image are estimated, wherein the reference frame is an adjacent frame image data of the current frame;

[0011] The depth data estimated based on the current frame is spliced ​​with the set range sampling points to form 3D sampling points;

[0012] The coordinates are mapped using the rotation change of the coordinates and the translation change of the coordinates to complete the point mapping of the 3D sampling points from the current frame to the reference frame;

[0013] By mapping the points to the reference frame and using grid sampling, the image sampling data of the current frame in the reference frame is obtained to complete the pose conversion between the current frame and the reference frame;

[0014] The image sampling data and the image data of the reference frame are processed using a similarity structure to perform self-supervised optimization to obtain an optimized pose prediction network;

[0015] Based on the acquired target image, depth feature prediction is performed on the current frame and the reference frame of the target image, and the optimized pose prediction network is used to optimize the depth feature prediction results by reverse gradient to obtain the optimized depth feature estimation results of the current frame and the reference frame of the target image.

[0016] In combination with the first aspect, as an optional implementation, calibration parameters of a current frame and calibration parameters of a reference frame are obtained, wherein the calibration parameters of the current frame include: conversion from an ego-vehicle coordinate system to a global coordinate system, and the calibration parameters of the reference frame include: conversion from a global coordinate system to an ego-vehicle coordinate system;

[0017] A reference point of the same size as the feature size of the bird's-eye view is set, and the coordinates of the reference point are mapped from the current frame to the reference frame using the calibration parameters of the current frame and the reference frame as well as the reference point to obtain the sampling features of the reference frame in the current frame.

[0018] In combination with the first aspect above, as an optional implementation method, according to the formula: Calculate the point mapping from the current frame to the reference frame, where T0 is the current frame and T1 is the reference frame. is the transformation from the global coordinate system to the vehicle coordinate system, For the conversion from the vehicle coordinate system to the global coordinate system, is the sampling point in the bird's-eye view space of the current frame;

[0019] Based on the point mapping from the current frame to the reference frame, the sampling features of the reference frame in the current frame are obtained by grid sampling.

[0020] In combination with the first aspect, as an optional implementation method, the sampling features of the reference frame are spliced ​​with the acquired bird's-eye view image features of the current frame to perform a conversion from the surround view image to the bird's-eye view image features to optimize the perception model;

[0021] The image features and image depth estimation results are used as the input of the optimized perception model to perform image-to-bird's-eye view image feature conversion to obtain the target bird's-eye view image features.

[0022] In combination with the first aspect above, as an optional implementation method, based on the bird's-eye view image features generated by the laser radar network;

[0023] The target bird's-eye view image features and the bird's-eye view image features generated by the lidar network are reconstructed using mean square error loss, so as to optimize the weights of the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame during the training process by reverse gradient propagation.

[0024] In combination with the first aspect above, as an optional implementation method, the obtained training weights corresponding to the image feature extraction module of the current frame and the reference frame, the training weights corresponding to the depth estimation module, and the training weights corresponding to the bird's-eye view image feature learning module are loaded into the corresponding modules in the pre-trained visual model to enhance the pre-trained visual model;

[0025] Use the enhanced pre-trained visual model to detect the target and output the detection results.

[0026] In a second aspect, the present application provides a target detection device, comprising:

[0027] An optimization module, which is used to optimize the depth prediction of the image based on the pose conversion between the current frame and the reference frame of the target image and the self-supervised optimization, so as to output the depth estimation results of the current frame and the reference frame of the target image;

[0028] A calculation module, which is used to output bird's-eye view image features of the current frame and the reference frame according to the image features of the current frame and the reference frame and the depth estimation results of the current frame and the reference frame, and calculate the sampling features of the reference frame in the current frame through mapping;

[0029] A processing module, which is used to splice the sampled features with the bird's-eye view features of the current frame to obtain target bird's-eye view features, and reconstruct the loss of the target bird's-eye view image features and the bird's-eye view image features acquired by the laser radar to optimize the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame in the training process.

[0030] An execution module is used to enhance the pre-trained visual model based on the weights, and output a target detection result through the enhanced pre-trained visual model.

[0031] In a third aspect, the present application further provides an electronic device, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method described in any one of the first aspects is implemented.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium storing computer program instructions, which, when executed by a computer, enables the computer to execute any of the methods described in the first aspect.

[0033] The present application provides a method, device, equipment and storage medium for target detection, wherein the method includes the steps of: optimizing the depth prediction of the image based on the pose conversion between the current frame and the reference frame of the target image and self-supervised optimization to output the depth estimation results of the current frame and the reference frame of the target image; outputting the bird's-eye view image features of the current frame and the reference frame according to the image features of the current frame and the reference frame and the depth estimation results of the current frame and the reference frame, and calculating the sampling features of the reference frame in the current frame by mapping; splicing the sampling features with the bird's-eye view features of the current frame to obtain the target bird's-eye view features, and reconstructing the loss of the target bird's-eye view image features and the bird's-eye view image features acquired by the laser radar to optimize the weights of the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame during the training process; enhancing the pre-trained visual model based on the weights, and outputting the target detection results through the enhanced pre-trained visual model. The present application can improve the detection performance of the model, improve the detection accuracy of the target, and reduce the cost at the same time.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0036] Figure 1 A flow chart of a target detection method provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of a target detection device provided in an embodiment of the present application;

[0038] Figure 3 This is an overall architecture diagram provided in the embodiments of the present application;

[0039] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application;

[0040] Figure 5A schematic diagram of a computer-readable program medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0042] Furthermore, the drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. Some of the blocks shown in the drawings are functional entities and do not necessarily correspond to physically or logically separate entities.

[0043] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0044] Reference Figure 1 , Figure 1 FIG. 1 is a flow chart of a target detection method provided by the present invention. Figure 1 As shown, the method comprises the steps of:

[0045] Step S101: Based on the pose conversion between the current frame and the reference frame of the target image and the self-supervised optimization, the depth prediction of the image is optimized to output the depth estimation results of the current frame and the reference frame of the target image.

[0046] Specifically, based on the defined pose prediction network, the coordinate rotation change and coordinate translation change between the current frame and the reference frame of the target image are estimated, wherein the reference frame is the adjacent frame image data of the current frame;

[0047] The depth data estimated based on the current frame is spliced ​​with the set range sampling points to form 3D sampling points;

[0048] The coordinates are mapped using the rotation change of the coordinates and the translation change of the coordinates to complete the point mapping of the 3D sampling points from the current frame to the reference frame;

[0049] By mapping the points to the reference frame and using grid sampling, the image sampling data of the current frame in the reference frame is obtained to complete the pose conversion between the current frame and the reference frame;

[0050] The image sampling data and the image data of the reference frame are processed using a similarity structure to perform self-supervised optimization to obtain an optimized pose prediction network;

[0051] Based on the acquired target image, depth feature prediction is performed on the current frame and the reference frame of the target image, and the optimized pose prediction network is used to optimize the depth feature prediction results by reverse gradient to obtain the optimized depth feature estimation results of the current frame and the reference frame of the target image.

[0052] For the convenience of understanding the specific description, S1.1 In a multi-camera system, the reference frame-guided image view conversion uses one or more pre-selected reference frames to assist the processing of the current frame. Reference frames are usually selected from multiple perspectives and can provide a comprehensive understanding of the scene to adapt to rapidly changing environments. In the present invention, the reference frame (the previous frame or several adjacent frames) is used to enhance the current key frame features (the current frame, where the current frame is acquired by using the vehicle's 6 surround cameras. The target object is the object that needs to be detected in the driving scene). First, according to the method of multi-view camera image view conversion, it is necessary to predict the depth features of the current image in order to obtain more accurate depth information. The depth prediction task of the image is strengthened using a self-supervised method. This step is completed by predicting the pose conversion between the reference frame and the current frame and self-supervised optimization.

[0053] Inter-frame pose conversion prediction. The reference frame represents the image data of the adjacent frames of the current frame. The current frame data is defined as T0, and the reference frame is defined as T1. In order to learn the pose conversion method between T0 and T1 in a self-supervised manner, the pose learning network is defined as P to estimate the coordinate rotation and translation changes between T0 and T1 frames, where the rotation change of the coordinates is defined as P rot (rot stands for rotation), the translation change of the coordinates is defined as P trans (trans stands for translation). It is understandable that, as the formula Using the estimated P rot and P trans Realize the data sampling between two frames, that is, the sampling between T0 and T1. In fact, the sampled data can be considered the same, so self-supervised training can be performed through SSIM. In the training process, since P rot and P trans It is estimated by the network P, so it can be continuously optimized as the network is trained. Finally, accurate posture conversion between T0 and T1 is achieved.

[0054] In addition, for the depth data D (abbreviation for depth) estimated for the current frame, the size of the depth data D is consistent with the size of the image features, both H×W. In order to realize the pose conversion prediction from the T0 frame to the T1 frame, the predicted depth data D is used in the application and spliced ​​with the set range sampling points (the range sampling points are the 2D_points in the formula) to form 3D sampling points (i.e., 3D_points). This process can be calculated using the following formula, where formula (1) is expressed as:

[0055]

[0056] After obtaining the corresponding 3D sampling points, the user obtains P rot and P trans The coordinates are mapped to complete the mapping of the 3D sampling points from the current frame T0 to the adjacent frame T1. The specific mapping steps are as follows:

[0057]

[0058] Through the mapping formula, the sampling points of the T0 frame can be mapped to the T1 frame with the help of the pose conversion method estimated by the P network. In the following self-supervised optimization, it is introduced how to improve the prediction accuracy of the pose prediction network P with the help of the self-supervised neural network training method. Self-supervised optimization. The self-supervised optimization is completed by continuous coordinate mapping and sampling between two adjacent frames, and the optimized loss function is calculated jointly using SSIM and L1 losses. Since the existing annotated pose conversion method was not considered at the beginning of the network design, the result of the pose prediction network P is briefly optimized by optimizing the mapped points and sampling, so as to finally achieve the optimized pose prediction network in a self-supervised manner. Using the optimized pose prediction network, the depth feature prediction results of the optimized target image current frame and reference frame are obtained by reverse gradient optimization. It can be understood that the pose conversion data is predicted by the pose prediction network P, and the point mapping of the sampling points from T0 to T1 frame is realized. With the key points mapped to the T1 frame, the grid sampling (Grid_Sample, GS) method can be used to obtain the sampling value of each mapping point, so as to calculate the difference between the sampled data and the original T0 frame data. In order to obtain more differences between the T0 frame data and the sampled T1 frame data through the loss function, the SSIM evaluation index can be used to comprehensively evaluate the geometric structure, illumination and pixel distribution between the data, so as to achieve better network training optimization.

[0059] Step S102: outputting bird's-eye view image features of the current frame and the reference frame according to the image features of the current frame and the reference frame and the depth estimation results of the current frame and the reference frame, and calculating the sampling features of the reference frame in the current frame through mapping.

[0060] Specifically, the image features of the current frame and the reference frame are obtained through the image feature extraction network, and the image features of the current frame and the depth estimation result of the current frame as well as the image features of the reference frame and the depth estimation result of the reference frame are input into the perception model LSS, and the bird's-eye view image features of the current frame and the reference frame are output.

[0061] Through mapping, the sampling features of the reference frame in the current frame are calculated, including:

[0062] Acquire calibration parameters of the current frame and calibration parameters of the reference frame, wherein the calibration parameters of the current frame include: conversion from the vehicle coordinate system to the global coordinate system, and the calibration parameters of the reference frame include: conversion from the global coordinate system to the vehicle coordinate system;

[0063] Set a reference point of the same size as the feature size of the bird's-eye view, and use the calibration parameters of the current frame, the calibration parameters of the reference frame and the reference point to map the reference point coordinates from the current frame to the reference frame, and obtain the sampling features of the reference frame in the current frame (because sampling is to be done in the reference frame, the points of the current frame must be mapped to the reference frame. The sampling data is obtained from the reference frame).

[0064] For the convenience of understanding the specific description, in order to further improve the accuracy of the BEV perception model, a reference frame guided image view conversion method is proposed. Specifically, the current method of converting multi-view surround camera images to BEV space and obtaining dense BEV perception is completed by the LSS perception model, and the LSS method requires surround images and corresponding depth features as inputs of the LSS method to complete the conversion of surround images to BEV features. However, a well-known problem is that LSS is strongly dependent on the current transmission perception calibration parameters, which causes the data error of the calibration parameters to cause huge errors to the LSS perception model. Taking this problem into consideration, the reference frame guided image view conversion is introduced in this patent. The core idea of ​​this method is to correct the BEV features of the current frame by sampling the reference frame to further improve the accuracy and richness of the BEV generated by the LSS perception model.

[0065] In order to complete the sampling mapping conversion from the current frame to the reference frame, the calibration parameters to be used are divided into two groups. The first group is the calibration parameters of the current frame T0, including the conversion from the vehicle coordinate system (the vehicle coordinate system can also be considered as the BEV coordinate system) to the global coordinate system (the global coordinate system is the IMU positioning data / GPS positioning data), defined as The second group of calibration parameters for the reference frame T1 includes the transformation from the global coordinate system to the ego-vehicle coordinate system, defined as

[0066] The reference point is set to the same size as the BEV characteristic size, that is, Here and represents the length and width of the BEV feature, and 2 represents the corresponding coordinates. and points BEV , to achieve points BEV The coordinates are mapped from T0 to T1 as follows:

[0067] Where T0 is the current frame and T1 is the reference frame. is the transformation from the global coordinate system to the vehicle coordinate system, For the conversion from the vehicle coordinate system to the global coordinate system, is the sampling point in the bird's-eye view space of the current frame. After mapping by this formula, the key points in T0 can be mapped to the reference points in the T1 coordinate system, and then the sampling features can be obtained using the grid sampling method.

[0068] Step S103: splicing the sampled features with the bird's-eye view features of the current frame to obtain the target bird's-eye view features, and reconstructing the loss of the target bird's-eye view image features and the bird's-eye view image features acquired by the lidar to optimize the weights of the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame during the training process.

[0069] Specifically, the sampling features of the reference frame are spliced ​​with the acquired bird's-eye view image features of the current frame to perform a conversion from the surround image to the bird's-eye view image features to optimize the perception model; the image features and the image depth estimation results are used as inputs of the optimized perception model to perform an image-to-bird's-eye view image feature conversion to obtain the target bird's-eye view image features.

[0070] Bird’s-eye view image features generated based on LiDAR network;

[0071] The target bird's-eye view image features and the bird's-eye view image features generated by the lidar network are reconstructed using mean square error loss, so as to optimize the weights of the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame during the training process by reverse gradient propagation.

[0072] For the convenience of understanding the specific instructions, the sampling features of the reference frame (BEV features of the previous second) are spliced ​​with the BEV features of the current frame to complete the image view conversion guided by the reference frame. It is worth mentioning that although the error of the calibration parameters still exists, this error can be improved through the sampling point mapping, grid sampling, and splicing of the sampling features with the BEV features of the current frame under the optimization of network training, and the performance of the network can be improved.

[0073] The image features and image depth estimation results are used as inputs of the optimized perception model to convert the image to a bird's-eye view image feature to obtain the target bird's-eye view image features (which can be continuously optimized as the network is trained). Bird's-eye view image features generated based on the LiDAR network.

[0074] Since camera sensors and LiDAR sensors have different imaging principles, the accuracy of BEV features generated by surround view images is affected by many factors, such as image resolution, camera calibration, and ambient lighting conditions. In scenes with a lack of texture or large lighting changes, depth estimation may not be accurate enough, and the resulting BEV features will also be greatly affected. The BEV features generated by LiDAR usually have higher accuracy and robustness and are not affected by lighting conditions. They can provide accurate distance measurements and are insensitive to environmental changes. The generated LiDAR BEV features can better reflect the true appearance of the driving environment.

[0075] In order to ensure that the final pre-trained visual model can have the same perception ability as LiDAR, a BEV reconstruction loss is designed in the application. For the BEV feature generated by LiDAR, it is defined as B LiDAR The BEV feature generated by the surround image is defined as B Camera To ensure B Camera The feature can have the perception distribution capability of LiDAR. In this application, BEV reconstruction loss is used for optimization. Specifically, mean square error loss (MSE) is used for optimization. The specific calculation is as follows: L recon =MSE(B LiDAR , B Camera ), L recon is the loss function. It is understandable that the loss function is to continuously optimize the training weights of each module in the pre-trained visual model. It needs to be explained that the network is composed of many weights. This loss is to optimize these weights. The specific optimization method is reverse gradient propagation. The calculation of the module is composed of some convolution, normalization and other components. These components are composed of weights and biases, which constitute the underlying calculation logic. Network training is to continuously optimize these weights and biases.

[0076] "The training weight of each module" During network training, the training weights of the entire model can be saved through pytorch's torch.Save method, so it is called the training weight of each module. However, when running the 3D object detection network below, we do not need to use the weights of each module, nor do we need to load the weights of each module. Specifically, only the weights of some modules need to be loaded, including: 1. The image feature extraction network of the current frame and the reference frame, that is, the weights of this network, need to be loaded into the image feature extraction part. 2. The depth estimation module is the same as the image feature loading. 3. The BEV feature learning module also needs to obtain the corresponding weights from the weights saved above, and then load them into the corresponding module.

[0077] Therefore, it can be understood that the reconstruction loss is to optimize the training weights of each module in the pre-trained visual model, and then take out some of the optimized training weights to enhance the pre-trained model.

[0078] Step S104: enhancing the pre-trained visual model based on the weights, and outputting the target detection result through the enhanced pre-trained visual model.

[0079] Specifically, the obtained training weights corresponding to the image feature extraction module of the current frame and the reference frame, the training weights corresponding to the depth estimation module, and the training weights corresponding to the bird's-eye view image feature learning module are loaded into the corresponding modules in the pre-trained visual model to enhance the pre-trained visual model;

[0080] Use the enhanced pre-trained visual model to detect the target and output the detection results.

[0081] That is, step S103 has reconstructed the features to optimize the training weights of each module in the pre-trained model, and loaded the image feature extraction network of the current frame and the reference frame, that is, the weights of this network, into the image feature extraction part. 2. The depth estimation module and the BEV feature learning module are similar to the image feature loading. After loading, the pre-trained visual model is enhanced, and then the model is used to detect the target.

[0082] It can be understood that after "completing the self-supervised depth estimation training between the current frame and the reference frame, and completing the enhanced BEV feature sampling of the current frame with the help of the reference frame, the final BEV feature is obtained by feature splicing. Then, the final BEV feature is compared with the BEV to be completed loss of LiDAR. Through the above training process, the training weights of each module in the network can be obtained. Then, the network model is kept basically unchanged, especially the image feature extraction part, the depth feature estimation part and the BEV feature learning part. While remaining unchanged, the training weights are loaded to the corresponding modules, and then the pre-trained model enhanced 3D object detection is performed.

[0083] Reference Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a target detection device provided by the present invention, Figure 2 As shown, the device comprises:

[0084] Optimization module 201: It is used to optimize the depth prediction of the image based on the pose conversion between the current frame and the reference frame of the target image and self-supervised optimization to output the depth estimation results of the current frame and the reference frame of the target image.

[0085] Calculation module 202: It is used to output bird's-eye view image features of the current frame and the reference frame according to the image features of the current frame and the reference frame and the depth estimation results of the current frame and the reference frame, and calculate the sampling features of the reference frame in the current frame through mapping.

[0086] Processing module 203: It is used to splice the sampled features with the bird's-eye view features of the current frame to obtain the target bird's-eye view features, and reconstruct the loss of the target bird's-eye view image features and the bird's-eye view image features obtained by the lidar to optimize the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame during the training process.

[0087] Execution module 204: It is used to enhance the pre-trained visual model based on the weights, and output the target detection result through the enhanced pre-trained visual model.

[0088] Further, in a possible implementation, the optimization module is further used to estimate the coordinate rotation change and coordinate translation change between the current frame and the reference frame of the target image based on the defined pose prediction network, wherein the reference frame is the adjacent frame image data of the current frame;

[0089] The depth data estimated based on the current frame is spliced ​​with the set range sampling points to form 3D sampling points;

[0090] The coordinates are mapped using the rotation change of the coordinates and the translation change of the coordinates to complete the point mapping of the 3D sampling points from the current frame to the reference frame;

[0091] By mapping the points to the reference frame and using grid sampling, the image sampling data of the current frame in the reference frame is obtained to complete the pose conversion between the current frame and the reference frame;

[0092] The image sampling data and the image data of the reference frame are processed using a similarity structure to perform self-supervised optimization to obtain an optimized pose prediction network;

[0093] Based on the acquired target image, depth feature prediction is performed on the current frame and the reference frame of the target image, and the optimized pose prediction network is used to optimize the depth feature prediction results by reverse gradient to obtain the optimized depth feature estimation results of the current frame and the reference frame of the target image.

[0094] Further, in a possible implementation manner, the calculation module is further used to obtain calibration parameters of the current frame and calibration parameters of the reference frame, wherein the calibration parameters of the current frame include: conversion from the vehicle coordinate system to the global coordinate system, and the calibration parameters of the reference frame include: conversion from the global coordinate system to the vehicle coordinate system;

[0095] A reference point of the same size as the feature size of the bird's-eye view is set, and the coordinates of the reference point are mapped from the current frame to the reference frame using the calibration parameters of the current frame and the reference frame as well as the reference point to obtain the sampling features of the reference frame in the current frame.

[0096] Furthermore, in a possible implementation manner, the calculation module is further configured to calculate the following formula: Calculate the point mapping from the current frame to the reference frame, where T0 is the current frame and T1 is the reference frame. is the transformation from the global coordinate system to the vehicle coordinate system, For the conversion from the vehicle coordinate system to the global coordinate system, is the sampling point in the bird's-eye view space of the current frame;

[0097] Based on the point mapping from the current frame to the reference frame, the sampling features of the reference frame in the current frame are obtained by grid sampling.

[0098] Furthermore, in a possible implementation manner, the processing module is further used to splice the sampling features of the reference frame with the acquired bird's-eye view image features of the current frame to perform a conversion from the surround view image to the bird's-eye view image features to optimize the perception model;

[0099] The image features and image depth estimation results are used as the input of the optimized perception model to perform image-to-bird's-eye view image feature conversion to obtain the target bird's-eye view image features.

[0100] Further, in a possible implementation, the processing module is also used for generating bird's-eye view image features based on the laser radar network;

[0101] The target bird's-eye view image features and the bird's-eye view image features generated by the lidar network are reconstructed using mean square error loss, so as to optimize the weights of the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame during the training process by reverse gradient propagation.

[0102] Further, in a possible implementation, the execution module is further used to load the obtained training weights corresponding to the image feature extraction module of the current frame and the reference frame, the training weights corresponding to the depth estimation module, and the training weights corresponding to the bird's-eye view image feature learning module into the corresponding modules in the pre-trained visual model to enhance the pre-trained visual model;

[0103] Use the enhanced pre-trained visual model to detect the target and output the detection results.

[0104] Reference Figure 3 , Figure 3 The overall architecture diagram provided by the present invention is shown as follows: Figure 3 As shown:

[0105] Target image data is acquired using multiple surround view cameras, and the historical first frame / reference frame and current frame are acquired through the image feature extraction network. The position transformation prediction network P is optimized using the current frame and reference frame and the self-supervised training method. The specific optimization method is shown in step S101.

[0106] The depth features of the current frame and the reference frame are predicted from the image feature extraction network to obtain the depth feature estimation results of the reference frame and the current frame. The results and the image features are input into the perception model LSS to obtain the bird's-eye view BEV features of the reference frame and the BEV features of the current frame. Based on the BEV features of the reference frame, mapping is performed to obtain the BEV sampling features of the reference frame in the current frame (see step S102 for the mapping details).

[0107] The sampled features and the BEV features of the current frame are concatenated to obtain new BEV features, namely the target BEV features, and the new BEV features are combined with the BEV features obtained by the lidar to construct the MSE loss to optimize the weights of the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame during the training process.

[0108] The weights of the optimized image features, depth estimation and bird's-eye view image features of the current frame and the reference frame during the training process are loaded into the module corresponding to the pre-trained visual model to enhance the pre-trained visual model. The enhanced pre-trained visual model is used to detect the target and output the detection result.

[0109] In the self-supervised depth feature estimation enhancement + visual pre-training part, the self-supervised depth estimation training between the current frame and the reference frame is completed, and the BEV feature sampling of the current frame enhancement is completed with the help of the reference frame, and then the final BEV feature is obtained by feature splicing. Then, the final BEV feature and the BEV feature of LiDAR are combined to complete the reconstruction loss. Through the above training process, the training weights of each module in the network can be obtained. Then, the network model is kept basically unchanged, especially the image feature extraction part, the depth feature estimation part, and the BEV feature learning part. While keeping it unchanged, the training weights are loaded to the corresponding modules, and then the pre-trained model enhanced 3D object detection is performed.

[0110] The beneficial effects of the present application include: 1. Pre-training the multi-view surround camera data by pre-training the visual model, compared with the previous method of directly using the image backbone network (the previous method does not use the pre-trained backbone network model), it can improve the detection performance of the visual network, close to the detection accuracy of LiDAR, and reduce the hardware cost. 2. It can realize the view conversion of 2D surround image data, generate rich BEV features, and enhance the adaptability and robustness of the model to dynamic environments. Compared with the previous method of using 8 frames and 16 frames, the present application can improve the detection performance of the model with less time frame sequence data. 3. The self-supervised optimization is completed by the coordinate mapping and sampling between two consecutive adjacent frames, and the prediction accuracy of the pose prediction network is improved by using SSIM and L1 loss joint calculation. For the existing relatively simple depth estimation method, the present application can realize optimization in a self-supervised manner, so that the network is more sensitive to depth information perception and obtains more accurate depth estimation results. 4. Through pre-trained visual model feature learning and BEV reconstruction loss learning, it can ensure that the network has a stronger perception ability and can generate more accurate BEV features.

[0111] Refer to the following Figure 4 An electronic device 400 according to this embodiment of the present invention will be described. Figure 4 The electronic device 400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0112] like Figure 4 As shown, the electronic device 400 is in the form of a general computing device. The components of the electronic device 400 may include but are not limited to: at least one processing unit 410, at least one storage unit 420, and a bus 430 connecting different system components (including the storage unit 420 and the processing unit 410).

[0113] The storage unit stores program codes, which can be executed by the processing unit 410, so that the processing unit 410 executes the steps according to various exemplary embodiments of the present invention described in the above “Embodiment Method” section of this specification.

[0114] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 421 and / or a cache memory unit 422 , and may further include a read-only memory unit (ROM) 423 .

[0115] The storage unit 420 may also include a program / utility 424 having a set (at least one) of program modules 425, such program modules 425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0116] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0117] The electronic device 400 may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 400, and / or communicate with any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 450. Furthermore, the electronic device 400 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 460. As shown, the network adapter 460 communicates with other modules of the electronic device 400 via a bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0118] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0119] According to the solution of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present invention described in the above "Exemplary Method" section of the present specification.

[0120] refer to Figure 5 As shown, a program product 500 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0121] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0122] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0123] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0124] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0125] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0126] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

[0127] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. A target detection method, characterized in that: include: Based on the pose conversion between the current frame and the reference frame of the target image and the self-supervised optimization, the depth prediction of the image is optimized to output the depth estimation results of the current frame and the reference frame of the target image; Outputting bird's-eye view image features of the current frame and the reference frame according to the image features of the current frame and the reference frame and the depth estimation results of the current frame and the reference frame, and calculating the sampling features of the reference frame in the current frame through mapping; The sampled features are concatenated with the bird's-eye view features of the current frame to obtain the target bird's-eye view features, and the target bird's-eye view image features are reconstructed with the bird's-eye view image features acquired by the lidar to optimize the weights of the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame during the training process; The pre-trained visual model is enhanced based on the weights, and the target detection result is output through the enhanced pre-trained visual model.

2. The method according to claim 1, characterized in that The method optimizes the depth prediction of the image based on the pose conversion between the current frame and the reference frame of the target image and the self-supervised optimization to output the depth estimation results of the current frame and the reference frame of the target image, including: Based on the defined pose prediction network, estimate the coordinate rotation change and coordinate translation change between the current frame and the reference frame of the target image, wherein the reference frame is the adjacent frame image data of the current frame; The depth data estimated based on the current frame is spliced ​​with the set range sampling points to form 3D sampling points; The coordinates are mapped using the rotation change of the coordinates and the translation change of the coordinates to complete the point mapping of the 3D sampling points from the current frame to the reference frame; By mapping the points to the reference frame and using grid sampling, the image sampling data of the current frame in the reference frame is obtained to complete the pose conversion between the current frame and the reference frame. The image sampling data and the image data of the reference frame are processed using a similarity structure to perform self-supervised optimization to obtain an optimized pose prediction network; Based on the acquired target image, depth feature prediction is performed on the current frame and the reference frame of the target image, and the optimized pose prediction network is used to optimize the depth feature prediction results by reverse gradient to obtain the optimized depth feature estimation results of the current frame and the reference frame of the target image.

3. The method according to claim 1, characterized in that The mapping to calculate the sampling features of the reference frame in the current frame includes: Acquire calibration parameters of the current frame and calibration parameters of the reference frame, wherein the calibration parameters of the current frame include: conversion from the vehicle coordinate system to the global coordinate system, and the calibration parameters of the reference frame include: conversion from the global coordinate system to the vehicle coordinate system; A reference point of the same size as the feature size of the bird's-eye view is set, and the coordinates of the reference point are mapped from the current frame to the reference frame using the calibration parameters of the current frame and the reference frame as well as the reference point to obtain the sampling features of the reference frame in the current frame.

4. The method according to claim 3, characterized in that include: According to the formula: Calculate the point mapping from the current frame to the reference frame, where T0 is the current frame and T1 is the reference frame. is the transformation from the global coordinate system to the vehicle coordinate system, For the conversion from the vehicle coordinate system to the global coordinate system, is the sampling point in the bird's-eye view space of the current frame; Based on the point mapping from the current frame to the reference frame, the sampling features of the reference frame in the current frame are obtained by grid sampling.

5. The method according to claim 1, characterized in that The step of combining the sampled features with the bird's-eye view features of the current frame to obtain the target bird's-eye view features includes: splicing the sampling features of the reference frame with the acquired bird's-eye view image features of the current frame to convert the surround image into the bird's-eye view image features to optimize the perception model; The image features and image depth estimation results are used as the input of the optimized perception model to perform image-to-bird's-eye view image feature conversion to obtain the target bird's-eye view image features.

6. The method according to claim 1, characterized in that The reconstructing loss of the target bird's-eye view image features and the bird's-eye view image features acquired by the laser radar to optimize the weights of the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame during the training process includes: Bird’s-eye view image features generated based on LiDAR network; The target bird's-eye view image features and the bird's-eye view image features generated by the lidar network are reconstructed using mean square error loss, so as to optimize the weights of the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame during the training process by reverse gradient propagation.

7. The method according to claim 1, characterized in that The step of enhancing the pre-trained visual model based on the weights and outputting the target detection result through the enhanced pre-trained visual model includes: Loading the obtained training weights corresponding to the image feature extraction module of the current frame and the reference frame, the training weights corresponding to the depth estimation module, and the training weights corresponding to the bird's-eye view image feature learning module into the corresponding modules in the pre-trained visual model to enhance the pre-trained visual model; Use the enhanced pre-trained visual model to detect the target and output the detection results.

8. A target detection device, characterized in that: include: An optimization module, which is used to optimize the depth prediction of the image based on the pose conversion between the current frame and the reference frame of the target image and the self-supervised optimization, so as to output the depth estimation results of the current frame and the reference frame of the target image; A calculation module, which is used to output bird's-eye view image features of the current frame and the reference frame according to the image features of the current frame and the reference frame and the depth estimation results of the current frame and the reference frame, and calculate the sampling features of the reference frame in the current frame through mapping; A processing module, which is used to splice the sampled features with the bird's-eye view features of the current frame to obtain target bird's-eye view features, and reconstruct the loss of the target bird's-eye view image features and the bird's-eye view image features acquired by the laser radar to optimize the image features, depth estimation and bird's-eye view image features of the current frame and the reference frame in the training process. An execution module is used to enhance the pre-trained visual model based on the weights, and output a target detection result through the enhanced pre-trained visual model.

9. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer program instructions are stored therein, and when the computer program instructions are executed by a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images