Around view depth estimation model training method, estimation method, device and equipment

Abstract

The invention relates to the technical field of computer vision, and provides a look-around depth estimation model training method and device, equipment and a storage medium, a look-around depth estimation model obtained through training is adaptive to a fisheye camera, and depth information can be estimated from an image shot by the fisheye camera. According to the method, the high-precision dense depth map is generated by using the sparse point cloud and the image, and due to the characteristics of high precision and cross-scene stability of the sparse point cloud, the model has extremely high robustness during cross-scene use. According to the method, the depth map is obtained by directly supervising and training the point cloud instead of supervising and training depth information to obtain the depth map in the prior art, and a new technical thought is provided. The model not only predicts the multi-view depth map, but also directly predicts the look-around point cloud, so that the precision of the point cloud is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a surround depth estimation model training method, estimation method, device and equipment. Background Art

[0002] Surround view depth estimation is to estimate the distance between the object and the camera by collecting images from a ring camera and using a deep learning model. Compared with the traditional monocular depth estimation method, surround view depth estimation can improve the accuracy and robustness of depth estimation by collecting information from multiple directions of the ring camera, and can be widely used in autonomous driving, 3D modeling, augmented reality, virtual reality and other fields. The implementation of the surround view depth estimation model based on deep learning technology usually uses a convolutional neural network (CNN) to extract image features and uses a regression model to convert image features into depth estimation results. Among them, the design and training process of the CNN network determine the accuracy and speed of this depth estimation. In the design of the CNN network, different numbers and structures of convolutional layers can be used, such as introducing residual connections, batch normalization and other technologies to improve the network effect. In the training stage, it is necessary to select a suitable data set for training and verification, and compare and evaluate different network structures and hyperparameters to obtain better model performance. In general, the surround view depth estimation model technology is a high-precision and efficient visual depth estimation method with broad application prospects.

[0003] The existing technology has the following problems. First, the current technology mainly studies the images collected by pinhole cameras as data sources. However, the field of view of general pinhole cameras is generally small. A large number of cameras are required to form a 360-degree surround view, which greatly increases the cost of the ring camera. At present, fisheye cameras are widely used due to their wide-angle characteristics. However, due to the serious image distortion of fisheye cameras, it is difficult to estimate depth information directly from fisheye cameras. Therefore, there is currently no depth estimation network that can adapt to fisheye images. Second, the model has weak generalization when used across scenes. Since the data sets used in model training often lack scene diversity and rely on the image features of specific scenes to restore depth information, there is bound to be a problem of poor generalization across scenes. Third, in actual use, many usage scenarios often hope to convert the surround depth map into a 360-degree point cloud. Then, due to problems such as internal and external parameter calibration errors in the camera calibration process, the point cloud converted from the depth map deviates from the true point cloud. Summary of the invention

[0004] In response to the above technical problems, the purpose of this application is to provide a surround view depth estimation model training method, estimation method, device and equipment, aiming to propose a depth estimation network adapted to fisheye images, improve the robustness of the surround view depth estimation model and reduce the error of the predicted point cloud.

[0005] In a first aspect, an embodiment of the present invention provides a surround depth estimation model training method, comprising:

[0006] Get the image collected by the ring camera;

[0007] Obtain surround sparse point cloud data collected by the radar;

[0008] Randomly downsampling the surround view sparse point cloud data by automatic masking to evenly block part of the point cloud to obtain second sparse point cloud data;

[0009] Input the image captured by the annular camera and the second sparse point cloud data as training data into a model to be trained; the model to be trained includes a point cloud feature extraction network, an image feature extraction network, a feature fusion module and a geometric network module;

[0010] Extracting features from the image captured by the ring camera through an image feature extraction network to obtain image features of different scales;

[0011] Performing point cloud feature extraction on the second sparse point cloud data through the point cloud feature extraction network to obtain point cloud features of different scales;

[0012] Inputting the image features of different scales and the point cloud features of different scales into the feature fusion module for feature fusion to obtain fused image features and fused point cloud features;

[0013] Inputting the fused image features into the geometric network module, transforming the fused image features into a depth map through the geometric network module, and transforming the depth map into a dense point cloud through reverse projection;

[0014] Constructing a loss function for dense point cloud supervision based on the second sparse point cloud data and the dense point cloud;

[0015] Constructing a total loss function based on the loss function of the dense point cloud supervision;

[0016] The model to be trained is trained with the goal of minimizing the total loss function to obtain the surround depth estimation model.

[0017] Furthermore, after the step of transforming the depth map into a dense point cloud, the method further includes:

[0018] Constructing a loss function for sparse point cloud reconstruction based on the second sparse point cloud data and the fused point cloud features;

[0019] The step of constructing a total loss function based on the loss function supervised by the dense point cloud comprises:

[0020] A total loss function is constructed based on the loss function of the dense point cloud supervision and the loss function of the sparse point cloud reconstruction.

[0021] Furthermore, the ring camera includes a plurality of cameras, each camera shoots in a different direction, and there is an overlapping area in the shooting ranges of adjacent cameras. After the step of transforming the depth map into a dense point cloud, the method further includes:

[0022] Get the mask image of the overlapping area;

[0023] According to the mask image, a point cloud geometric consistency loss function of an adjacent region is constructed according to the geometric consistency loss to constrain point clouds generated from different cameras;

[0024] Constructing an occlusion mask of learnable parameters; the occlusion mask of learnable parameters identifies the occluded area, and the occlusion mask is updated during the model training process;

[0025] Constructing a geometric consistency loss function that considers occlusion relationship according to the occlusion mask and the point cloud geometric consistency loss function of the adjacent area; the geometric consistency loss function that considers occlusion relationship does not perform loss calculation when the position is identified as an occlusion area;

[0026] The step of constructing a total loss function based on the loss function of the dense point cloud supervision and the loss function of the sparse point cloud reconstruction comprises:

[0027] A total loss function is constructed based on the loss function of the dense point cloud supervision, the loss function of the sparse point cloud reconstruction, and the geometric consistency loss function considering the occlusion relationship.

[0028] Furthermore, the image feature extraction network includes a deformable convolutional neural network and a decoder, wherein the deformable convolutional neural network is used to generate a multi-scale feature map according to an image input to the deformable convolutional neural network.

[0029] The decoder is provided with a feature map of the same scale as the multi-scale feature map, and the decoder is used to concatenate the multi-scale feature map with the feature map of the same scale of the decoder at the channel level to obtain image features of different scales.

[0030] Furthermore, the feature fusion module is a multi-scale multi-camera cross attention module, which includes n multimodal attention modules and 2 (n-1) feature upsampling modules, where n is the number of feature scales. The step of inputting the image features of different scales and the point cloud features of different scales into the feature fusion module for feature fusion to obtain the fused image features and the fused point cloud features includes:

[0031] Sort the image features of different scales from small to large to obtain the sorted image features X = {x 1 ,x 2 ,...x n};

[0032] Sort the point cloud features of different scales from small to large, and get the sorted point cloud features Y = {y 1 ,y 2 ,...y n}; wherein the scale of the image feature is the same as the scale of the point cloud feature;

[0033] The image feature x 1 And the point cloud feature y 1 Input the first multimodal attention module to perform feature fusion to obtain a first fused feature;

[0034] Point cloud feature y 2 The scale of is taken as the target, and the first fusion feature is upsampled by a first upsampling module to obtain a second fusion feature;

[0035] The second fusion feature and the point cloud feature y 2 Concatenation is performed at the channel level, first concatenating the results;

[0036] Take image feature x 2 The scale of is taken as the target, and the first fusion feature is upsampled by a second upsampling module to obtain a third fusion feature;

[0037] The third fusion feature and the image feature x 2 Concatenation is performed at the channel level, and the second concatenation results;

[0038] Inputting the first concatenation result and the second concatenation result into a second multimodal attention module for feature fusion to obtain a fourth fusion feature;

[0039] Point cloud feature y 3 The fourth fusion feature is upsampled by a third upsampling module to obtain a fifth fusion feature;

[0040] The fifth fusion feature and the point cloud feature y 3 Perform concatenation at the channel level to obtain a third concatenation result;

[0041] Take image feature x 3 The fourth fusion feature is upsampled by a fourth upsampling module to obtain a sixth fusion feature;

[0042] The sixth fusion feature and the image feature x 3Concatenation is performed at the channel level, and the fourth concatenation result;

[0043] By analogy, the output of the last multimodal attention module is used as the fused image features and the fused point cloud features.

[0044] Furthermore, the multimodal attention module includes a first cross attention module, a second cross attention module, a first self attention module, a second self attention module, a third cross attention module, a fourth cross attention module, a third self attention module, and a fourth self attention module. 1 And the point cloud feature y 1 Inputting the first multimodal attention module for feature fusion, the step of obtaining the first fusion feature includes:

[0045] According to the image feature x 1 And the point cloud feature y 1 Generate corresponding retrieval vectors, key vectors and value vectors respectively;

[0046] Inputting the retrieval vector corresponding to the point cloud feature, the key vector corresponding to the image feature, and the value vector corresponding to the point cloud feature into the first cross attention module to obtain a first result;

[0047] Inputting the retrieval vector corresponding to the image feature, the key vector corresponding to the point cloud feature, and the value vector corresponding to the image feature into the second cross attention module to obtain a second result;

[0048] Inputting the first result into the first self-attention module to obtain a third result;

[0049] Inputting the second result into the second self-attention module to obtain a fourth result;

[0050] Generate a search vector, a key vector, and a value vector corresponding to the third result according to the third result;

[0051] Generate a search vector, a key vector, and a value vector corresponding to the fourth result according to the fourth result;

[0052] Inputting the search vector and the key vector corresponding to the third result and the value vector corresponding to the fourth result into the third cross attention module to obtain a fifth result;

[0053] Inputting the search vector and the key vector corresponding to the fourth result and the value vector corresponding to the third result into the fourth cross-attention module to obtain a sixth result;

[0054] Inputting the fifth result into the third self-attention module to obtain a fusion result of image features;

[0055] The sixth result is input into the fourth self-attention module to obtain a fusion result of the point cloud features.

[0056] Furthermore, the annular cameras are all composed of fisheye cameras.

[0057] In a second aspect, an embodiment of the present application provides a surround depth estimation method, including:

[0058] Get the image collected by the ring camera;

[0059] Obtain surround sparse point cloud data collected by the radar;

[0060] The image collected by the annular camera and the surround view sparse point cloud data collected by the radar are input into a pre-trained surround view depth estimation model to obtain a depth map.

[0061] In a third aspect, an embodiment of the present application provides a point cloud generation method, including:

[0062] Get the image collected by the ring camera;

[0063] Obtain surround sparse point cloud data collected by the radar;

[0064] Get the camera's internal parameters;

[0065] The image collected by the annular camera, the surround view sparse point cloud data collected by the radar, and the internal parameters of the camera are input into a pre-trained surround view depth estimation model to obtain a dense point cloud.

[0066] Furthermore, the surround view depth estimation model is trained using any of the surround view depth model training methods described above.

[0067] In a fourth aspect, an embodiment of the present application provides a surround depth estimation model training device, comprising:

[0068] An image acquisition module is used to acquire images collected by the ring camera;

[0069] Point cloud acquisition module, used to obtain surround sparse point cloud data collected by radar;

[0070] A downsampling module, configured to randomly downsample the surround sparse point cloud data by automatic masking to uniformly block part of the point cloud to obtain second sparse point cloud data;

[0071] A first input module, used to input the image collected by the annular camera and the second sparse point cloud data as training data into a model to be trained; the model to be trained includes a point cloud feature extraction network, an image feature extraction network, a feature fusion module and a geometric network module;

[0072] An image feature extraction module, used to extract features from the image collected by the ring camera through an image feature extraction network to obtain image features of different scales;

[0073] A point cloud feature extraction module, used to extract point cloud features from the second sparse point cloud data through the point cloud feature extraction network to obtain point cloud features of different scales;

[0074] A second input module is used to input the image features of different scales and the point cloud features of different scales into the feature fusion module for feature fusion to obtain fused image features and fused point cloud features;

[0075] A third input module is used to input the fused image features into the geometric network module, transform the fused image features into a depth map through the geometric network module, and transform the depth map into a dense point cloud through reverse projection;

[0076] A dense point cloud supervised loss function construction module, used to construct a dense point cloud supervised loss function based on the second sparse point cloud data and the dense point cloud;

[0077] A total loss function construction module, used to construct a total loss function based on the loss function of the dense point cloud supervision;

[0078] A training module is used to train the model to be trained with the goal of minimizing the total loss function to obtain the surround depth estimation model.

[0079] In a fifth aspect, an embodiment of the present application provides a surround depth estimation device, comprising:

[0080] An image acquisition module is used to acquire images collected by the ring camera;

[0081] Point cloud acquisition module, used to obtain surround sparse point cloud data collected by radar;

[0082] The input module is used to input the image collected by the annular camera and the surround view sparse point cloud data collected by the radar into a pre-trained surround view depth estimation model to obtain a depth map.

[0083] In a sixth aspect, an embodiment of the present application provides a point cloud generation device, characterized in that it includes:

[0084] An image acquisition module is used to acquire images collected by the ring camera;

[0085] The point cloud acquisition module acquires the surround sparse point cloud data collected by the radar;

[0086] The camera internal parameter acquisition module is used to obtain the camera's internal parameters;

[0087] The input module is used to input the image collected by the annular camera, the surround view sparse point cloud data collected by the radar, and the internal parameters of the camera into a pre-trained surround view depth estimation model to obtain a dense point cloud.

[0088] In a seventh aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the methods described above are implemented.

[0089] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the methods described above are implemented.

[0090] The surround depth estimation model trained by the embodiment of the present application is adapted to the fisheye camera and can estimate the depth information from the image taken by the fisheye camera. The embodiment of the present application generates a high-precision dense depth map by using sparse point clouds and images. Thanks to the high precision and cross-scene stability of sparse point clouds, the model has extremely high robustness when used across scenes. The present application obtains a depth map by directly performing supervised training on the point cloud, rather than supervised training on the depth information to obtain the depth map as in the prior art, providing a new technical idea. The model of the present application not only predicts multi-view depth maps, but also directly predicts surround point clouds, greatly improving the accuracy of point clouds. The embodiment of the present application uses two different feature extraction networks in one model, which efficiently extracts features from sparse point clouds and images, while taking into account the characteristics of the two modalities. The embodiment of the present application implements dense point cloud estimation from multiple perspectives in the same model, reduces the resource occupation of deployed equipment, and facilitates maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the implementation manner will be briefly introduced below. Obviously, the drawings described below are only some implementation manners of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0092] Figure 1 It is a flowchart of a surround depth estimation model training method provided in an embodiment of the present application;

[0093] Figure 2 is the image collected by the ring camera;

[0094] Figure 3 It is the sparse point cloud data collected by the radar

[0095] Figure 4 It is the network structure of the surround depth estimation model provided in the embodiment of the present application;

[0096] Figure 5 This is a schematic diagram of the PointNet network structure and its transformation sub-network structure;

[0097] Figure 6 It is a schematic diagram of the structure of the multimodal attention module;

[0098] Figure 7 It is a schematic diagram of the calculation process of the cross-attention and self-attention mechanisms;

[0099] Figure 8 is a schematic diagram of an overlapping area mask provided in an embodiment of the present application;

[0100] Fig. 9 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0101] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0102] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "above", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may also be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any module and all combinations of one or more associated listed items.

[0103] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0104] Embodiment 1:

[0105] like Figure 1 As shown, an embodiment of the present invention provides a surround depth estimation model training method, including steps S1-S11:

[0106] S1. Obtain the image captured by the ring camera.

[0107] In the embodiment of the present application, the ring camera is composed of multiple cameras, each camera shoots in different directions to obtain images with different fields of view. For example, the images collected by the ring camera are as follows: Figure 2 shown.

[0108] S2. Obtain surround sparse point cloud data collected by the radar.

[0109] Exemplarily, the surround sparse point cloud data collected by the radar is as follows: Figure 3 shown.

[0110] S3. Randomly downsample the surround view sparse point cloud data through automatic masking to evenly block part of the point cloud to obtain second sparse point cloud data.

[0111] In the embodiment of the present application, the automatic mask (AutoMask) is mainly used to randomly discard part of the point cloud, for example, if there are 10,000 point clouds, then 1,000 are randomly discarded each time. Therefore, the surround sparse point cloud data can be randomly downsampled by the automatic mask.

[0112] S4, input the image captured by the ring camera and the second sparse point cloud data as training data into the model to be trained; the model to be trained includes a point cloud feature extraction network, an image feature extraction network, a feature fusion module and a geometric network module. In the embodiment of the present application, the network structure of the model to be trained is as follows Figure 4 As shown, Figure 4 PointNet represents the point cloud feature extraction network, Deformable Convolution and Decoder form the image feature extraction network, UM and MMAM form the feature fusion module, namely MSMC Cros-Att in the figure, and GeometryNetwork is the geometry network module. Figure 4 The Sparse GT in the above example represents the sparse point cloud gold standard (groundtruth, also known as the true value) and Figure 4 The Sparse Lidar Input after AutoMask downsampling is the same.

[0113] S5. Perform feature extraction on the image collected by the ring camera through an image feature extraction network to obtain image features of different scales.

[0114] S6. Performing point cloud feature extraction on the second sparse point cloud data through the point cloud feature extraction network to obtain point cloud features of different scales.

[0115] S7. Inputting the image features of different scales and the point cloud features of different scales into the feature fusion module for feature fusion to obtain fused image features and fused point cloud features.

[0116] S8. Input the fused image features into the geometric network module, transform the fused image features into a depth map through the geometric network module, and transform the depth map into a dense point cloud through reverse projection.

[0117] S9. Construct a loss function for dense point cloud supervision based on the second sparse point cloud data and the dense point cloud.

[0118] S10. Constructing a total loss function based on the loss function of the dense point cloud supervision.

[0119] S11. Training the model to be trained with the goal of minimizing the total loss function to obtain the surround depth estimation model.

[0120] The surround depth estimation model trained by the embodiment of the present application is adapted to the fisheye camera and can estimate the depth information from the image taken by the fisheye camera. The embodiment of the present application generates a high-precision dense depth map by using sparse point clouds and images. Thanks to the high precision and cross-scene stability of sparse point clouds, the model has extremely high robustness when used across scenes. The present application obtains a depth map by directly performing supervised training on the point cloud, rather than supervised training on the depth information to obtain the depth map as in the prior art, providing a new technical idea. The model of the present application not only predicts multi-view depth maps, but also directly predicts 360-degree surround point clouds, greatly improving the accuracy of the point clouds. The embodiment of the present application uses two different feature extraction networks in one model, which efficiently extracts features from sparse point clouds and images, while taking into account the characteristics of the two modalities. The embodiment of the present application implements dense point cloud estimation from multiple perspectives in the same model, reduces the resource occupation of deployed equipment, and facilitates maintenance.

[0121] In some embodiments, the surround view sparse point cloud data refers to 360-degree surround view sparse point cloud data.

[0122] In some embodiments, the cameras constituting the ring camera may be pinhole cameras or fisheye cameras.

[0123] In one embodiment, the annular cameras are all composed of fisheye cameras.

[0124] Among them, four fisheye cameras can meet the 360-degree field of view. Compared with the existing method that requires 6 or 8 pinhole cameras to meet the 360-degree field of view, the equipment cost of the ring camera is greatly reduced.

[0125] In one embodiment, the annular cameras are all composed of fisheye cameras.

[0126] It should be noted that fisheye cameras are currently widely used due to their wide-angle characteristics. However, due to the severe image distortion of fisheye cameras, it is difficult to estimate depth information directly from fisheye cameras. Therefore, there is currently almost no solution to estimate depth from fisheye images. The depth estimation network proposed in the embodiment of the present application is adapted to fisheye cameras, which solves the technical problem of difficulty in estimating depth information from fisheye cameras.

[0127] In one embodiment, obtaining image features of different scales means obtaining multi-scale image features FI t s , obtain point cloud features of different scales, that is, obtain multi-scale point cloud features FL s , the above t is the camera (camera) number in each direction, that is, the image number, s is the image or point cloud feature of different scales. In one example, t∈[0,3],s∈[0,2].

[0128] In some embodiments, the point cloud feature extraction network is a PointNet network structure based on a deep learning model, which includes but is not limited to Figure 5 The network structure shown in Figure 1 is as follows. Figure 5 As shown in the figure, the input is n point clouds, each point cloud contains 3 dimensions, which are then passed through the transformation sub-network, and then through the fully connected network to form 64-dimensional features, and finally through the full connection to form 1024-dimensional features.

[0129] In some embodiments, the image feature extraction network includes a deformable convolutional neural network and a decoder, the deformable convolutional neural network is used to generate a multi-scale feature map based on the image input into the deformable convolutional neural network, the decoder is provided with a feature map of the same scale as the multi-scale feature map, and the decoder is used to concatenate the multi-scale feature map with the feature map of the same scale of the decoder at the channel level to obtain image features of different scales.

[0130] The embodiment of the present application simultaneously integrates features of multiple scales in the decoder to perform high-precision dense point cloud estimation for targets of various shapes.

[0131] It should be noted that the deformable convolutional neural network is used for image feature extraction, and therefore, it can be any network structure based on deformable convolution for generating multi-scale features. Figure 4 As shown in the Deformable Convolution (Deformable Convolutional Neural Network), the deformable convolutional neural network contains N blocks, each block contains n layers, and the number of channels between adjacent blocks increases step by step, such as 3, 64, 128, 256, 512. At the same time, the network feature scale of the output of the corresponding block decreases step by step. In one embodiment, the input image size is 384*256, and then the feature scales are 192x128, 96x64, 48x32, 24x16, and 12x8, respectively. The convolution operation involved in each layer is a deformable convolution operation. The decoder is designed to be up-sampled, that is, as the number of layers increases, the scale of the feature map gradually increases.

[0132] In one embodiment, the feature fusion module is a multi-scale multi-camera cross attention module (MSMC Cro-Att), and the structure of the multi-scale multi-camera cross attention module is as follows: Figure 4 As shown in the MSMC Cro-Att in FIG, the multi-scale multi-camera cross attention module includes n multi-modal attention modules (MMAM) and 2 (n-1) feature upsampling modules (UM), n is the number of feature scales, and the steps of inputting the image features of different scales and the point cloud features of different scales into the feature fusion module for feature fusion to obtain the fused image features and the fused point cloud features include:

[0133] S710, sort the image features of different scales from small to large to obtain the sorted image feature X={x 1 ,x 2 ,...x n};

[0134] S711, sort the point cloud features of different scales from small to large, and obtain the sorted point cloud features Y = {y 1 ,y 2 ,...y n}; wherein the scale of the image feature is the same as the scale of the point cloud feature;

[0135] S712: image feature x 1 And the point cloud feature y 1 Input the first multimodal attention module to perform feature fusion to obtain a first fused feature;

[0136] S713, point cloud feature y 2The scale of is taken as the target, and the first fusion feature is upsampled by a first upsampling module to obtain a second fusion feature;

[0137] S714, the second fusion feature and the point cloud feature y 2 Concatenation is performed at the channel level, first concatenating the results;

[0138] S715, using image feature x 2 The scale of is taken as the target, and the first fusion feature is upsampled by a second upsampling module to obtain a third fusion feature;

[0139] S716: Combine the third fusion feature and the image feature x 2 Concatenation is performed at the channel level, and the second concatenation results;

[0140] S717: Input the first concatenation result and the second concatenation result into a second multimodal attention module for feature fusion to obtain a fourth fusion feature;

[0141] S718, point cloud feature y 3 The fourth fusion feature is upsampled by a third upsampling module to obtain a fifth fusion feature;

[0142] S719, the fifth fusion feature and the point cloud feature y 3 Perform concatenation at the channel level to obtain a third concatenation result;

[0143] S720, using image feature x 3 The fourth fusion feature is upsampled by a fourth upsampling module to obtain a sixth fusion feature;

[0144] S721 combines the sixth fusion feature and the image feature x 3 Concatenation is performed at the channel level, and the fourth concatenation result;

[0145] S722, and so on, taking the output of the last multimodal attention module as the fused image features and the fused point cloud features.

[0146] In the embodiment of the present application, taking the scale of a feature a as the target, an upsampling module is used to upsample a feature b, which means that feature b is transformed to a feature of the same scale as feature a after upsampling, that is, the scale of feature a is larger than that of feature b.

[0147] According to the above steps S710-S722, the multi-scale multi-camera cross attention module (MSMC Cro-Att) performs a multi-scale feature FL s (dimensions are: (NL, L)), (The dimensions are (NI, C, H, W). In one embodiment, NI=4, i.e., images in four directions). An MMAM module is independently used to perform multi-module feature fusion at the same scale. The fused features between the two scales are upsampled by UP and then fused through the series connection of the channel levels. Finally, MSMC Cro-Att outputs the fused point cloud features and the fused image features.

[0148] The above steps S710-S722 proposed in the embodiment of the present application realize the multi-scale fusion of image features and point cloud features from multiple cameras, and consider the feature fusion between different scales. The implementation mechanism is excellent and lays the foundation for the subsequent acquisition of high-precision dense depth maps and more accurate reconstructed point clouds.

[0149] It should be understood that the first, second, and nth in the above-mentioned multimodal attention modules only represent different numbers of the multimodal attention modules, and their network structures are the same.

[0150] In one embodiment, for a single multimodal attention module, its structure is as follows Figure 6 As shown, the multimodal attention module includes a first cross attention module, a second cross attention module, a first self-attention module, a second self-attention module, a third cross attention module, a fourth cross attention module, a third self-attention module, and a fourth self-attention module. 1 And the point cloud feature y 1 Inputting the first multimodal attention module to perform feature fusion, the step of obtaining the first fusion feature includes S7121-S712:

[0151] S7121, according to the image feature x 1 And the point cloud feature y 1 Generate corresponding retrieval vector q, key vector k and value vector v respectively;

[0152] S7122, inputting the search vector q (query) corresponding to the point cloud feature, the key vector k (key) corresponding to the image feature, and the value vector v (value) corresponding to the point cloud feature into the first cross-attention module (Cross-Att) to obtain a first result;

[0153] S7123, inputting the retrieval vector q corresponding to the image feature, the key vector k corresponding to the point cloud feature, and the value vector v corresponding to the image feature into the second cross attention module to obtain a second result;

[0154] S7124, inputting the first result into the first self-attention module (self-att) to obtain a third result;

[0155] S7125. Input the second result into the second self-attention module to obtain a fourth result;

[0156] S7126 generates a search vector q, a key vector k, and a value vector v corresponding to the third result according to the third result;

[0157] S7127. Generate a search vector q, a key vector k, and a value vector v corresponding to the fourth result according to the fourth result;

[0158] S7128, inputting the search vector q and the key vector k corresponding to the third result and the value vector v corresponding to the fourth result into the third cross attention module to obtain a fifth result;

[0159] S7129, inputting the search vector q and the key vector k corresponding to the fourth result and the value vector v corresponding to the third result into the fourth cross attention module to obtain a sixth result;

[0160] S71210, inputting the fifth result into the third self-attention module to obtain a fusion result of image features;

[0161] S71211. Input the sixth result into the fourth self-attention module to obtain a fusion result of the point cloud features.

[0162] In an embodiment of the present application, the first cross-attention module to the fourth cross-attention module may be the same or different, and the first self-attention module to the fourth self-attention module may be the same or different.

[0163] The calculation process of the cross attention and self-attention mechanism is as follows Figure 7 As shown, the only difference between the two regions is the source of q, k, v (which can also be expressed in uppercase, i.e., Q, K, V), and the calculation method is the same. Specifically: the cross-attention mechanism refers to the cross-alignment of different modal information in the feature space in the multimodal information scenario to learn the correlation between different modal information to achieve better representation learning effect. In general, the cross-attention mechanism needs to go through the following calculation process:

[0164] 1. Assume that there are two modal input data X and Y, each of which contains n features, that is, X = [x 1 ,x 2 ,...,x n ],Y=[y 1 ,y 2 ,...,y n ];

[0165] 2. Perform linear transformation on X and Y respectively, project them into a common space, and obtain a new feature representation F x and F y , where F x =W x X+b x , F Y =W y Y+b y , W x , W y is the parameter of the linear transformation, b x and b y is the bias term.

[0166] 3. For F x and F Y Calculate the attention weight. There are many ways to calculate the attention weight, such as using the dot product method: A = F x T F Y , and then normalize A (through the softmax function) to obtain the weight distribution a.

[0167] 4. Use the weighted summation of attention weights a to calculate the interaction representation F of input data X and Y in feature space interact :F interact =aF Y T +F X T .

[0168] 5. In F interact Perform a nonlinear transformation of the channel dimension (for example, using a 1×1 convolution) to obtain the final cross-attention feature representation F out .

[0169] 6. Output F out as input to the next layer.

[0170] In one embodiment, after the step of transforming the depth map into a dense point cloud, the method further comprises:

[0171] Constructing a loss function for sparse point cloud reconstruction based on the second sparse point cloud data and the fused point cloud features;

[0172] The step of constructing a total loss function based on the loss function supervised by the dense point cloud comprises:

[0173] A total loss function is constructed based on the loss function of the dense point cloud supervision and the loss function of the sparse point cloud reconstruction.

[0174] like Figure 4As shown, the embodiment of the present application can improve the model's ability to extract point cloud features by also reconstructing sparse point clouds during the training process.

[0175] In one embodiment, the SmoothsL1 loss function is used as the loss function LossRegresionOfPredict for the dense point cloud supervision, and the SmoothsL1 loss function is used as the loss function for the sparse point cloud reconstruction. The SmoothsL1 loss function is shown as follows:

[0176]

[0177] a represents the difference between the true value and the predicted value.

[0178] Then the loss function of dense point cloud supervision or the loss function of sparse point cloud reconstruction in the embodiment of the present invention is:

[0179]

[0180] It should be understood that although the form of the loss function for dense point cloud supervision or sparse point cloud reconstruction is the same, the physical meaning of the formula is different. For the dense point cloud supervision loss function, (x, y, z) represents the coordinates of the second sparse point cloud data, and (x', y', z') represents the coordinates of the dense point cloud. For the loss function for sparse point cloud reconstruction, (x, y, z) represents the coordinates of the second sparse point cloud data, and (x', y', z') represents the coordinates of the fused point cloud features.

[0181] In one embodiment, the loss function of the dense point cloud supervision and the loss function of the sparse point cloud reconstruction may also adopt other differentiable metric functions that can measure the difference between two distributions, such as L2, KL divergence, etc.

[0182] In one embodiment, the ring camera includes a plurality of cameras, each camera shoots in a different direction, and there is an overlapping area in the shooting ranges of adjacent cameras. After the step of transforming the depth map into a dense point cloud, the method further includes:

[0183] Get the mask image of the overlapping area;

[0184] According to the mask image, a point cloud geometric consistency loss function of an adjacent region is constructed according to the geometric consistency loss to constrain point clouds generated from different cameras;

[0185] Constructing an occlusion mask of learnable parameters; the occlusion mask of learnable parameters identifies the occluded area, and the occlusion mask is updated during the model training process;

[0186] Constructing a geometric consistency loss function that considers occlusion relationship according to the occlusion mask and the point cloud geometric consistency loss function of the adjacent area; the geometric consistency loss function that considers occlusion relationship does not perform loss calculation when the position is identified as an occlusion area;

[0187] The step of constructing a total loss function based on the loss function of the dense point cloud supervision and the loss function of the sparse point cloud reconstruction comprises:

[0188] A total loss function is constructed based on the loss function of the dense point cloud supervision, the loss function of the sparse point cloud reconstruction, and the geometric consistency loss function considering the occlusion relationship.

[0189] The embodiment of the present application not only converts the depth estimated by the model into a point cloud, but also introduces geometric consistency constraints on the point clouds formed by adjacent cameras, thereby improving the point cloud accuracy in the area adjacent to the camera.

[0190] In the embodiment of the present application, the steps of obtaining the mask image of the overlapping area and constructing the geometric consistency loss function are performed in the geometric network module. The geometric network module first transforms the fused image features into a depth map through two convolutional layers, and then performs reverse projection based on the internal and external parameters of the camera.

[0191] Reverse projection requires combining the camera's intrinsic parameters and the camera's extrinsic parameters. The reverse projection formula is as follows:

[0192]

[0193]

[0194]

[0195]

[0196] θ * =d -1 (dθ)

[0197] d(θ)=θ+f 1 θ 3 +f 2 θ 5 +f 3 θ 7 +f 4 θ 9

[0198] In the formula, fx, fy, cx, cy represent the intrinsic parameters of the camera, namely horizontal focal length, vertical focal length, horizontal center, vertical center, k1, k2, k3, k4 represent the distortion coefficients of the fisheye lens, Z represents the depth, and x, y represent the coordinate indexes of the image.

[0199] That is, the geometric network module simultaneously predicts the multi-view depth map and the (360-degree) surround dense point cloud. Taking the example of a ring camera composed of four fisheye cameras, four images will be collected. Since there are overlapping areas in the adjacent areas of the four images, the mask images of the four overlapping areas can be obtained in advance. According to the mask images, the position of the overlapping area of ​​the point cloud can be known. According to the geometric relationship, there is a consistency relationship between the point clouds at corresponding positions in the overlapping area of ​​the point cloud. According to the consistency relationship, four sets of constraints between adjacent cameras can be constructed. For each set of constraints, the geometric consistency loss function of the point cloud of the corresponding adjacent area can be constructed, and the geometric consistency loss function of the point cloud of the adjacent area can be minimized. At the same time, according to the principle of geometric vision, due to the change of the camera perspective, the point cloud formed by two cameras in the overlapping area must have inconsistency on the edge of the object because of the occlusion relationship of the object. In order to avoid the occlusion inconsistency affecting the minimization of the geometric consistency loss function of the point cloud of the adjacent area, an occlusion mask with learnable parameters is introduced to identify the occluded area, and the occlusion mask will be updated during the model process. Therefore, the loss function does not perform loss calculation at the location identified as the occluded area, thereby avoiding the impact of occlusion inconsistency on its minimization.

[0200] The following is a detailed description of the point cloud geometric consistency constraints and loss functions of adjacent areas:

[0201] Let P(t, t-1) represent the point cloud of the overlapping area of ​​cameras t and t-1, and P(t-1, t) represent the point cloud of the overlapping area of ​​cameras t-1 and t. According to the geometric relationship of the point clouds, P(t, t-1) = P(t-1, t), that is, LossConsistency(t, t-1) = SmoothL1[P(t, t-1), P(t-1, t)], and the theoretical minimum value is 0, where LossConsistency(t, t-1) is the geometric consistency loss function of the point cloud in the adjacent area. However, due to the perspective occlusion relationship in visual geometry and the precision error of the overlapping area, LossConsistency(t, t-1) will not be perfectly equal to 0.

[0202] The occlusion mask and the geometric consistency loss function considering the occlusion relationship are explained in detail below.

[0203] Occlusion masks can be implemented in two possible ways:

[0204] Method 1: Create an occlusion mask according to the following formula:

[0205]

[0206] Among them, lossConsistency(t,t-1) represents the value of the point cloud geometric consistency loss function of the adjacent area, and θ is the set threshold.

[0207] Method 2: Based on the fused image features, two convolutional layers are used to generate MaskOcc, and the value of MaskOcc is restricted to 0 and 1.

[0208] It should be noted that MaskOcc is a matrix whose size is the same as the image size of a single camera. When it is 0, no constraints are generated when Loss is subsequently multiplied. When it is 1, constraints are generated.

[0209] Geometric consistency loss function considering occlusion relationship:

[0210] The specific calculation formula is:

[0211] LossCstyWithOcc(t,t-1)=MaskOcc*LossConsistency(t,t-1)

[0212] Among them, LossCstyWithOcc(t,t-1) represents the geometric consistency loss function considering the occlusion relationship, and LossConsistency(t,t-1) represents the point cloud geometric consistency loss function of the adjacent area.

[0213] In one embodiment, the overlapping area and the mask map of the overlapping area are calculated as follows:

[0214] Detect key points and use a key point matching algorithm to obtain matching points of adjacent images; wherein the adjacent images are images captured by adjacent cameras;

[0215] Delineating an overlapping area according to matching points of the adjacent images, and creating a mask image of the overlapping area according to the following formula;

[0216]

[0217] In the formula, x and y represent the coordinate index of the image, and KeypointBound is the set of boundary coordinates of the matched key points. Represents the overlap mask of adjacent cameras t and t-1.

[0218] The mask diagram of the overlapping area is as follows: Figure 8 As shown, the non-zero area is the overlapping area of ​​the images of adjacent cameras.

[0219] In one embodiment, the overlapping area can also be calculated in the following way:

[0220] Convert the camera coordinate system to the world coordinate system according to the camera's intrinsic and extrinsic parameters;

[0221] The position and attitude of the cameras in the world coordinate system are calculated based on the external parameters of the two cameras, and the optical centers of the two cameras are adjusted to the same height through the scale factor;

[0222] The position and orientation of the imaging planes of the two cameras in the world coordinate system are calculated using the position and orientation of the cameras in the world coordinate system.

[0223] The positions and directions of the two imaging planes in the world coordinate system are overlapped according to the positions and directions of the imaging planes of the two cameras in the world coordinate system, and the overlapping area is calculated.

[0224] It can be seen from the above that the geometric network structure proposed in the embodiment of the present application not only converts the depth estimated by the model into a point cloud, but also introduces the geometric consistency constraints of the point cloud formed by adjacent cameras, thereby improving the point cloud accuracy of the camera adjacent area.

[0225] In one embodiment, the total loss function for training the model to be trained is, where (t, t-1) is omitted:

[0226] Loss = α*LossCstyWithOcc+β*LossRegresionOfPredict+λ*LossRegresionOfReContruc, where α, β, λ are loss scaling factors, LossCstyWithOcc represents the geometric consistency loss function considering occlusion relationship, LossRegresionOfPredict represents the loss function of dense point cloud supervision, and LossRegresionOfReContruc represents the loss function of sparse point cloud reconstruction.

[0227] Embodiment 2:

[0228] The present application provides a surround depth estimation method, including:

[0229] Get the image collected by the ring camera;

[0230] Obtain surround sparse point cloud data collected by the radar;

[0231] The image collected by the annular camera and the surround view sparse point cloud data collected by the radar are input into a pre-trained surround view depth estimation model to obtain a depth map.

[0232] Embodiment three:

[0233] The present application provides a point cloud generation method, including:

[0234] Get the image collected by the ring camera;

[0235] Obtain surround sparse point cloud data collected by the radar;

[0236] Get the camera's internal parameters;

[0237] The image collected by the annular camera, the surround view sparse point cloud data collected by the radar, and the internal parameters of the camera are input into a pre-trained surround view depth estimation model to obtain a dense point cloud.

[0238] Furthermore, the surround view depth estimation model is trained using the surround view depth model training method described above.

[0239] Embodiment 4:

[0240] The present application provides a surround depth estimation model training device, including:

[0241] An image acquisition module is used to acquire images collected by the ring camera;

[0242] Point cloud acquisition module, used to obtain surround sparse point cloud data collected by radar;

[0243] A downsampling module, configured to randomly downsample the surround sparse point cloud data by automatic masking to uniformly block part of the point cloud to obtain second sparse point cloud data;

[0244] A first input module, used to input the image collected by the annular camera and the second sparse point cloud data as training data into the model to be trained; the model to be trained includes a point cloud feature extraction network, an image feature extraction network, a feature fusion module and a geometric network module;

[0245] An image feature extraction module, used to extract features from the image collected by the ring camera through an image feature extraction network to obtain image features of different scales;

[0246] A point cloud feature extraction module, used to extract point cloud features from the second sparse point cloud data through the point cloud feature extraction network to obtain point cloud features of different scales;

[0247] A second input module is used to input the image features of different scales and the point cloud features of different scales into the feature fusion module for feature fusion to obtain fused image features and fused point cloud features;

[0248] A third input module is used to input the fused image features into the geometric network module, transform the fused image features into a depth map through the geometric network module, and transform the depth map into a dense point cloud through reverse projection;

[0249] A dense point cloud supervised loss function construction module, used to construct a dense point cloud supervised loss function based on the second sparse point cloud data and the dense point cloud;

[0250] A total loss function construction module, used to construct a total loss function based on the loss function of the dense point cloud supervision;

[0251] A training module is used to train the model to be trained with the goal of minimizing the total loss function to obtain the surround depth estimation model.

[0252] Embodiment five:

[0253] The present application provides a surround depth estimation device, including:

[0254] An image acquisition module is used to acquire images collected by the ring camera;

[0255] Point cloud acquisition module, used to obtain surround sparse point cloud data collected by radar;

[0256] The input module is used to input the image collected by the annular camera and the surround view sparse point cloud data collected by the radar into a pre-trained surround view depth estimation model to obtain a depth map.

[0257] Embodiment six:

[0258] The present application provides a point cloud generation device, including:

[0259] An image acquisition module is used to acquire images collected by the ring camera;

[0260] Point cloud acquisition module, used to obtain surround sparse point cloud data collected by radar;

[0261] The camera internal parameter acquisition module is used to obtain the camera's internal parameters;

[0262] The input module is used to input the image collected by the annular camera, the surround view sparse point cloud data collected by the radar, and the internal parameters of the camera into a pre-trained surround view depth estimation model to obtain a dense point cloud.

[0263] Embodiment seven:

[0264] Reference Fig. 9 The present application also provides a computer device, the internal structure of which can be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor designed for the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a surround depth estimation model training method. The network interface of the computer device is used to communicate with an external terminal through a network connection. Furthermore, the above-mentioned computer device can also be provided with an input device and a display screen, etc. When the above-mentioned computer program is executed by a processor, a surround view depth estimation model training method, a surround view depth estimation method and / or a point cloud generation method are implemented. A surround view depth estimation model training method includes the following steps: acquiring an image captured by an annular camera; acquiring surround view sparse point cloud data captured by a radar; randomly downsampling the surround view sparse point cloud data through an automatic mask to uniformly block part of the point cloud to obtain second sparse point cloud data; inputting the image captured by the annular camera and the second sparse point cloud data as training data into a model to be trained; the model to be trained includes a point cloud feature extraction network, an image feature extraction network, a feature fusion module and a geometric network module; extracting features from the image captured by the annular camera through the image feature extraction network to obtain image features of different scales; The second sparse point cloud data is subjected to point cloud feature extraction through the point cloud feature extraction network to obtain point cloud features of different scales; the image features of different scales and the point cloud features of different scales are input into the feature fusion module for feature fusion to obtain fused image features and fused point cloud features; the fused image features are input into the geometric network module, the fused image features are transformed into a depth map through the geometric network module, and the depth map is transformed into a dense point cloud through reverse projection; a loss function of dense point cloud supervision is constructed based on the second sparse point cloud data and the dense point cloud; a total loss function is constructed based on the loss function of dense point cloud supervision; the model to be trained is trained with the goal of minimizing the total loss function to obtain the surround depth estimation model. A surround depth estimation method comprises: obtaining an image collected by a ring camera; obtaining surround sparse point cloud data collected by a radar; inputting the image collected by the ring camera and the surround sparse point cloud data collected by the radar into a pre-trained surround depth estimation model to obtain a depth map. A point cloud generation method comprises: obtaining an image captured by a ring camera; obtaining surround view sparse point cloud data collected by a radar; obtaining an internal parameter of the camera; and inputting the image captured by the ring camera, the surround view sparse point cloud data collected by the radar, and the internal parameter of the camera into a pre-trained surround view depth estimation model to obtain a dense point cloud.Those skilled in the art can understand that. Fig. 9 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0265] Embodiment eight:

[0266] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a surround depth estimation model training method, a surround depth estimation method and / or a point cloud generation method is implemented. A surround depth estimation model training method comprises the following steps: acquiring an image captured by an annular camera; acquiring surround sparse point cloud data captured by a radar; randomly downsampling the surround sparse point cloud data through an automatic mask to uniformly block part of the point cloud to obtain second sparse point cloud data; inputting the image captured by the annular camera and the second sparse point cloud data as training data into a model to be trained; the model to be trained comprises a point cloud feature extraction network, an image feature extraction network, a feature fusion module and a geometric network module; performing feature extraction on the image captured by the annular camera through the image feature extraction network Extract the image features of different scales; extract the point cloud features of the second sparse point cloud data through the point cloud feature extraction network to obtain point cloud features of different scales; input the image features of different scales and the point cloud features of different scales into the feature fusion module for feature fusion to obtain fused image features and fused point cloud features; input the fused image features into the geometric network module, transform the fused image features into a depth map through the geometric network module, and transform the depth map into a dense point cloud through reverse projection; construct a loss function of dense point cloud supervision based on the second sparse point cloud data and the dense point cloud; construct a total loss function based on the loss function of dense point cloud supervision; train the model to be trained with the goal of minimizing the total loss function to obtain the surround depth estimation model. A surround depth estimation method includes: obtaining an image collected by a ring camera; obtaining surround sparse point cloud data collected by a radar; inputting the image collected by the ring camera and the surround sparse point cloud data collected by the radar into a pre-trained surround depth estimation model to obtain a depth map. A point cloud generation method comprises: obtaining an image captured by a ring camera; obtaining surround view sparse point cloud data collected by a radar; obtaining an internal parameter of the camera; and inputting the image captured by the ring camera, the surround view sparse point cloud data collected by the radar, and the internal parameter of the camera into a pre-trained surround view depth estimation model to obtain a dense point cloud.

[0267] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0268] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0269] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A surround depth estimation model training method, characterized in that: include: Get the image collected by the ring camera; Obtain surround sparse point cloud data collected by the radar; Randomly downsampling the surround view sparse point cloud data by automatic masking to evenly block part of the point cloud to obtain second sparse point cloud data; Input the image captured by the annular camera and the second sparse point cloud data as training data into a model to be trained; the model to be trained includes a point cloud feature extraction network, an image feature extraction network, a feature fusion module and a geometric network module; Extracting features from the image captured by the ring camera through an image feature extraction network to obtain image features of different scales; Performing point cloud feature extraction on the second sparse point cloud data through the point cloud feature extraction network to obtain point cloud features of different scales; Inputting the image features of different scales and the point cloud features of different scales into the feature fusion module for feature fusion to obtain fused image features and fused point cloud features; Inputting the fused image features into the geometric network module, transforming the fused image features into a depth map through the geometric network module, and transforming the depth map into a dense point cloud through reverse projection; Constructing a loss function for dense point cloud supervision based on the second sparse point cloud data and the dense point cloud; Constructing a total loss function based on the loss function of the dense point cloud supervision; The model to be trained is trained with the goal of minimizing the total loss function to obtain the surround depth estimation model.

2. The surround depth estimation model training method according to claim 1, characterized in that: After the step of transforming the depth map into a dense point cloud, the method further comprises: Constructing a loss function for sparse point cloud reconstruction based on the second sparse point cloud data and the fused point cloud features; The step of constructing a total loss function based on the loss function supervised by the dense point cloud comprises: A total loss function is constructed based on the loss function of the dense point cloud supervision and the loss function of the sparse point cloud reconstruction.

3. The surround depth estimation model training method according to claim 1 or 2, characterized in that: The ring camera includes a plurality of cameras, each camera shoots in a different direction, and there is an overlapping area in the shooting ranges of adjacent cameras. After the step of transforming the depth map into a dense point cloud, the method further includes: Get the mask image of the overlapping area; According to the mask image, a point cloud geometric consistency loss function of adjacent regions is constructed according to the geometric consistency loss to constrain point clouds generated from different cameras; Constructing an occlusion mask of learnable parameters; the occlusion mask of learnable parameters identifies the occluded area, and the occlusion mask is updated during the model training process; Constructing a geometric consistency loss function that considers occlusion relationship according to the occlusion mask and the point cloud geometric consistency loss function of the adjacent area; the geometric consistency loss function that considers occlusion relationship does not perform loss calculation when the position is identified as an occlusion area; The step of constructing a total loss function based on the loss function of the dense point cloud supervision and the loss function of the sparse point cloud reconstruction comprises: A total loss function is constructed based on the loss function of the dense point cloud supervision, the loss function of the sparse point cloud reconstruction, and the geometric consistency loss function considering the occlusion relationship.

4. The surround depth estimation model training method according to claim 1, characterized in that: The image feature extraction network includes a deformable convolutional neural network and a decoder, wherein the deformable convolutional neural network is used to generate a multi-scale feature map according to an image input to the deformable convolutional neural network. The decoder is provided with a feature map of the same scale as the multi-scale feature map, and the decoder is used to concatenate the multi-scale feature map with the feature map of the same scale of the decoder at the channel level to obtain image features of different scales.

5. The surround depth estimation model training method according to claim 1, characterized in that: The feature fusion module is a multi-scale multi-camera cross attention module, which includes n multimodal attention modules and 2 (n-1) feature upsampling modules, where n is the number of feature scales. The steps of inputting the image features of different scales and the point cloud features of different scales into the feature fusion module for feature fusion to obtain the fused image features and the fused point cloud features include: Sort the image features of different scales from small to large to obtain the sorted image features X = {x1, x2, ...x n }; Sort the point cloud features of different scales from small to large, and get the sorted point cloud features Y = {y1, y2, ...y n }; wherein the scale of the image feature is the same as the scale of the point cloud feature; Input the image feature x1 and the point cloud feature y1 into the first multimodal attention module for feature fusion to obtain the first fused feature; Taking the scale of the point cloud feature y2 as the target, upsampling the first fused feature by using a first upsampling module to obtain a second fused feature; The second fusion feature and the point cloud feature y2 are concatenated at the channel level, and the first concatenation result is obtained; Taking the scale of the image feature x2 as the target, upsampling the first fused feature by using a second upsampling module to obtain a third fused feature; Concatenate the third fusion feature and the image feature x2 at the channel level, and obtain the second concatenation result; Inputting the first concatenation result and the second concatenation result into a second multimodal attention module for feature fusion to obtain a fourth fusion feature; Taking the scale of the point cloud feature y3 as a target, upsampling the fourth fused feature using a third upsampling module to obtain a fifth fused feature; Concatenate the fifth fusion feature and the point cloud feature y3 at the channel level to obtain a third concatenation result; Taking the scale of the image feature x3 as a target, upsampling the fourth fused feature by using a fourth upsampling module to obtain a sixth fused feature; Concatenate the sixth fusion feature and the image feature x3 at the channel level, the fourth concatenation result; By analogy, the output of the last multimodal attention module is used as the fused image features and the fused point cloud features.

6. The surround depth estimation model training method according to claim 5, characterized in that: The multimodal attention module includes a first cross attention module, a second cross attention module, a first self-attention module, a second self-attention module, a third cross attention module, a fourth cross attention module, a third self-attention module, and a fourth self-attention module. The step of inputting the image feature x1 and the point cloud feature y1 into the first multimodal attention module for feature fusion to obtain the first fusion feature includes: Generate corresponding retrieval vector, key vector and value vector respectively according to the image feature x1 and the point cloud feature y1; Inputting the retrieval vector corresponding to the point cloud feature, the key vector corresponding to the image feature, and the value vector corresponding to the point cloud feature into the first cross attention module to obtain a first result; Inputting the retrieval vector corresponding to the image feature, the key vector corresponding to the point cloud feature, and the value vector corresponding to the image feature into the second cross attention module to obtain a second result; Inputting the first result into the first self-attention module to obtain a third result; Inputting the second result into the second self-attention module to obtain a fourth result; Generate a search vector, a key vector, and a value vector corresponding to the third result according to the third result; Generate a search vector, a key vector, and a value vector corresponding to the fourth result according to the fourth result; Inputting the search vector and the key vector corresponding to the third result and the value vector corresponding to the fourth result into the third cross attention module to obtain a fifth result; Inputting the search vector and the key vector corresponding to the fourth result and the value vector corresponding to the third result into the fourth cross-attention module to obtain a sixth result; Inputting the fifth result into the third self-attention module to obtain a fusion result of image features; The sixth result is input into the fourth self-attention module to obtain a fusion result of the point cloud features.

7. The surround depth estimation model training method according to claim 1, characterized in that: The ring cameras are all composed of fisheye cameras.

8. A surround depth estimation method, characterized in that: include: Get the image collected by the ring camera; Obtain surround sparse point cloud data collected by the radar; The image collected by the annular camera and the surround view sparse point cloud data collected by the radar are input into a pre-trained surround view depth estimation model to obtain a depth map.

9. A point cloud generation method, characterized in that: include: Get the image collected by the ring camera; Obtain surround sparse point cloud data collected by the radar; Get the camera's internal parameters; The image collected by the annular camera, the surround view sparse point cloud data collected by the radar, and the internal parameters of the camera are input into a pre-trained surround view depth estimation model to obtain a dense point cloud.

10. The surround depth estimation method according to claim 8 or the point cloud generation method according to claim 9, characterized in that: The surround view depth estimation model is trained using the surround view depth model training method described in any one of claims 1 to 7.

11. A surround depth estimation model training device, characterized in that: include: An image acquisition module is used to acquire images collected by the ring camera; Point cloud acquisition module, used to obtain surround sparse point cloud data collected by radar; A downsampling module, configured to randomly downsample the surround sparse point cloud data by automatic masking to uniformly block part of the point cloud to obtain second sparse point cloud data; A first input module, used to input the image collected by the annular camera and the second sparse point cloud data as training data into a model to be trained; the model to be trained includes a point cloud feature extraction network, an image feature extraction network, a feature fusion module and a geometric network module; An image feature extraction module, used to extract features from the image collected by the ring camera through an image feature extraction network to obtain image features of different scales; A point cloud feature extraction module, used to extract point cloud features from the second sparse point cloud data through the point cloud feature extraction network to obtain point cloud features of different scales; A second input module is used to input the image features of different scales and the point cloud features of different scales into the feature fusion module for feature fusion to obtain fused image features and fused point cloud features; A third input module is used to input the fused image features into the geometric network module, transform the fused image features into a depth map through the geometric network module, and transform the depth map into a dense point cloud through reverse projection; A dense point cloud supervised loss function construction module, used to construct a dense point cloud supervised loss function based on the second sparse point cloud data and the dense point cloud; A total loss function construction module, used to construct a total loss function based on the loss function of the dense point cloud supervision; A training module is used to train the model to be trained with the goal of minimizing the total loss function to obtain the surround depth estimation model.

12. A surround depth estimation device, characterized in that: include: An image acquisition module is used to acquire images collected by the ring camera; Point cloud acquisition module, used to obtain surround sparse point cloud data collected by radar; The input module is used to input the image collected by the annular camera and the surround view sparse point cloud data collected by the radar into a pre-trained surround view depth estimation model to obtain a depth map.

13. A point cloud generating device, characterized in that: include: An image acquisition module is used to acquire images collected by the ring camera; The point cloud acquisition module acquires the surround sparse point cloud data collected by the radar; The camera internal parameter acquisition module is used to obtain the camera's internal parameters; The input module is used to input the image collected by the annular camera, the surround view sparse point cloud data collected by the radar, and the internal parameters of the camera into a pre-trained surround view depth estimation model to obtain a dense point cloud.

14. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

Images