Model training method, model, distance estimation method, device, equipment and medium

Abstract

The invention provides a model training method, a multi-mode look-around depth estimation model, a distance estimation method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: fusing multi-feature-scale image features of scene images of a target scene in different directions of a target object with multi-feature-scale image features of a sparse depth map to obtain target fusion features, decoding the target fusion feature according to the multi-feature-scale image features of the scene image of the target scene in different directions to obtain dense depth maps of the target scene in different directions, and training a multi-modal look-around depth estimation model according to the obtained dense depth maps and sparse depth maps of the target scene in different directions to obtain a multi-modal look-around depth estimation model; therefore, the model has extremely high robustness during cross-scene use, the problem of weak generalization during cross-scene use of the model is solved, and the distance estimation precision can be improved when the distance is estimated through the depth map output by the model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a model training method, a multimodal surround depth estimation model, a distance estimation method, a device, an electronic device, and a computer-readable storage medium in the field of artificial intelligence technology. Background Art

[0002] Surround view depth estimation is to estimate the distance between the object and the camera by collecting images from a ring camera and using a deep learning model. Compared with the traditional monocular depth estimation method, surround view depth estimation can improve the accuracy and robustness of depth estimation by collecting information from multiple directions of the ring camera, and can be widely used in autonomous driving, 3D modeling, augmented reality, virtual reality and other fields.

[0003] The implementation of surround depth estimation model based on deep learning technology usually uses convolutional neural networks (CNN) to extract image features and uses regression models to convert image features into depth estimation results. Among them, the design and training process of the CNN network determines the accuracy and speed of this depth estimation. In the design of the CNN network, different numbers and structures of convolutional layers can be used, such as introducing residual connections, batch normalization and other technologies to improve the network effect. In the training stage, it is necessary to select a suitable data set for training and verification, and compare and evaluate different network structures and hyperparameters to obtain better model performance. However, the CNN network model has weak generalization when used across scenes, that is, the data set used by the CNN network model during training often lacks scene diversity, relies on the image features of specific scenes to restore depth information, and has the problem of poor generalization across scenes. Summary of the invention

[0004] The present application provides a model training method, a multimodal surround depth estimation model, a distance estimation method, an apparatus, an electronic device and a computer-readable storage medium, which can solve the problem of weak generalization of CNN network models when used across scenarios.

[0005] In a first aspect, a model training method is provided, the model training method comprising:

[0006] Acquire a first sample image; wherein the first sample image includes a scene image of a target scene corresponding to each direction of the target object;

[0007] Acquire a second sample image; wherein the second sample image includes a sparse depth map of the target scene corresponding to each direction of the target object;

[0008] Extracting a plurality of first image features of the first sample image; wherein the plurality of first image features have different feature scales;

[0009] Extracting a plurality of second image features of the second sample image; wherein the plurality of second image features have different feature scales;

[0010] According to the feature scale, each of the plurality of first image features is fused with a second image feature corresponding to each of the plurality of first image features to obtain a target fused feature;

[0011] Decoding the target fusion feature according to the multiple first image features to obtain a target sample image; wherein the target sample image includes a dense depth map of the target scene corresponding to each direction of the target object;

[0012] A multimodal surround depth estimation model is trained according to the second sample image and the target sample image.

[0013] In combination with the first aspect, in some possible implementations, the step of acquiring the second sample image includes:

[0014] Acquire point cloud data of the target scene corresponding to each direction of the target object;

[0015] The point cloud data of the target scene corresponding to each of the directions are projected onto an image pixel coordinate system to obtain a sparse depth map of the target scene corresponding to each of the directions of the target object.

[0016] In combination with the first aspect and the above implementation manner, in some possible implementation manners, a radar and a plurality of cameras are provided on the target object, scene images of target scenes corresponding to respective directions of the target object are acquired by cameras at corresponding positions in respective directions of the target object, and point cloud data of target scenes corresponding to respective directions of the target object are acquired by the radar;

[0017] The step of projecting the point cloud data of the target scene corresponding to each direction to the image pixel coordinate system to obtain a sparse depth map of the target scene corresponding to each direction of the target object comprises:

[0018] Acquire a first internal parameter of the camera and a first external calibration parameter of the radar;

[0019] According to the first internal parameter and the first external calibration parameter, the point cloud data of the target scene corresponding to each direction is projected to the image pixel coordinate system to obtain a sparse depth map of the target scene corresponding to each direction of the target object.

[0020] In combination with the first aspect and the above implementation, in some possible implementations, the multimodal surround depth estimation model includes a deformable convolutional neural network and a sparse convolutional neural network;

[0021] The step of extracting a plurality of first image features of the first sample image comprises:

[0022] The deformable convolutional neural network is used to extract image features of the first sample image at each of a plurality of preset feature scales to obtain the plurality of first image features; wherein the plurality of preset feature scales are of different sizes;

[0023] The step of extracting a plurality of second image features of the second sample image comprises:

[0024] The sparse convolutional neural network is used to extract image features of the second sample image at each of the multiple preset feature scales to obtain the multiple second image features.

[0025] In combination with the first aspect and the above implementation manner, in some possible implementation manners, the multimodal surround depth estimation model further includes a plurality of fusion modules, and the plurality of fusion modules correspond to different feature scales;

[0026] The step of fusing each of the plurality of first image features with each of the second image features corresponding to the plurality of first image features according to the feature scale to obtain a target fused feature comprises:

[0027] Determining, from the plurality of fusion modules, target fusion modules corresponding to the plurality of first image features respectively according to the feature scale;

[0028] Using target fusion modules corresponding to the plurality of first image features, each of the plurality of first image features is fused with a second image feature corresponding to the plurality of first image features to obtain a fusion result; wherein the fusion result includes fused image features corresponding to the plurality of first image features and fused depth features corresponding to the plurality of first image features;

[0029] The target fusion feature is generated according to the fusion result.

[0030] In combination with the first aspect and the above implementation manner, in some possible implementation manners, the target fusion module includes a first module group and a second module group, the first module group includes a plurality of first processing modules connected in series, each first processing module includes a first submodule and a second submodule connected to an output end of the first submodule, the second module group includes a plurality of second processing modules connected in series, each second processing module includes a third submodule and a fourth submodule connected to an output end of the third submodule, the first submodule and the third submodule are both cross-attention modules, and the second submodule and the fourth submodule are both self-attention modules;

[0031] The step of using the target fusion modules corresponding to the plurality of first image features to fuse the plurality of first image features with the second image features corresponding to the plurality of first image features to obtain the fusion result comprises:

[0032] Acquire a first linear feature corresponding to each of the plurality of first image features, and acquire a second linear feature corresponding to each of the plurality of second image features; wherein the first linear feature and the second linear feature both include a Q feature, a K feature, and a V feature;

[0033] The first linear feature and the second linear feature are cross-input into the first module group and the first module group to obtain fused image features corresponding to each of the plurality of first image features and fused depth features corresponding to each of the plurality of second image features.

[0034] In combination with the first aspect and the above implementation manner, in some possible implementation manners, the multimodal surround depth estimation model further includes a decoder;

[0035] The step of decoding the target fusion feature according to the multiple first image features to obtain a target sample image comprises:

[0036] The multiple first image features and the target fusion features are input into the decoder, so that the target fusion features are decoded by the decoder to obtain the target sample image.

[0037] In combination with the first aspect and the foregoing implementation manner, in some possible implementation manners, the step of training a multimodal surround depth estimation model according to the second sample image and the target sample image includes:

[0038] Determine the difference between the dense depth map and the sparse depth map of the target scene corresponding to each direction of the target object, and obtain multiple differences;

[0039] Determine a loss function based on an average difference of the plurality of differences;

[0040] The multimodal surround view depth estimation model is trained according to the loss function.

[0041] In a second aspect, a multimodal surround depth estimation model is provided, and the multimodal surround depth estimation model includes:

[0042] A feature processing module is used to extract multiple image features of each of the first image and the second image to obtain a feature set; wherein the first image includes: a scene image of a target scene corresponding to at least one direction of the target object, and the second image includes: a sparse depth map of the target scene corresponding to at least one direction of the target object; the feature set includes: a plurality of first image features corresponding to the first image and a plurality of second image features corresponding to the second image, wherein the plurality of first image features have different feature scales, and the plurality of first image features have different feature scales;

[0043] The feature processing module is further used to fuse each of the plurality of first image features with a second image feature corresponding to each of the plurality of first image features according to a feature scale to obtain a target fused feature;

[0044] A decoder is used to decode the target fusion feature according to the multiple first image features to obtain a target image; wherein the target image includes: a dense depth map of the target scene corresponding to at least one direction of the target object.

[0045] In conjunction with the second aspect, in some possible implementations, the feature processing module includes:

[0046] A deformable convolutional neural network, wherein the deformable convolutional neural network includes M first feature extraction layers, the convolution operation adopted by the M first feature extraction layers is a deformable convolution operation, the preset feature scales corresponding to the first first feature extraction layer to the Mth first feature extraction layer are gradually reduced, and M is an odd number greater than or equal to 5;

[0047] A sparse convolutional neural network, wherein the sparse convolutional neural network comprises M second feature extraction layers, the convolution operation adopted by the M second feature extraction layers is a sparse convolution operation, and the preset feature scales corresponding to the first second feature extraction layer to the Mth second feature extraction layer are gradually reduced;

[0048] M - 1 fusion module;

[0049] A feature series connection layer, wherein the feature series connection layer is connected to the decoder;

[0050] in:

[0051] The output ends of the i-th first feature extraction layer and the i-th second feature extraction layer are connected to the input end of the i-th fusion module, and the output end of the i-th fusion module is connected to the input ends of the i+1-th first feature extraction layer and the i+1-th second feature extraction layer;

[0052] The output ends of the Mth first feature extraction layer and the Mth second feature extraction layer are connected to the feature series connection layer;

[0053] The output ends of the first feature extraction layer from the first feature extraction layer 1 to the first feature extraction layer M-2 are connected to the decoder, i=1, 2, 3, ..., M-1;

[0054] The i-th first feature extraction layer is used to extract the first image feature of the first image according to a preset feature scale corresponding to the i-th first feature extraction layer;

[0055] The i-th second feature extraction layer is used to extract the second image features of the second image according to the preset feature scale corresponding to the i-th second feature extraction layer, and the preset feature scale corresponding to the i-th first feature extraction layer is the same as the preset feature scale corresponding to the i-th second feature extraction layer;

[0056] The i-th fusion module is used to fuse the first image features extracted by the i-th first feature extraction layer and the second image features extracted by the i-th second feature extraction layer.

[0057] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, each fusion module includes a first module group and a second module group, the first module group includes a plurality of first processing modules connected in series, each first processing module includes a first submodule and a second submodule connected to the output end of the first submodule, the second module group includes a plurality of second processing modules connected in series, each second processing module includes a third submodule and a fourth submodule connected to the output end of the third submodule, the first submodule and the third submodule are both cross-attention modules, and the second submodule and the fourth submodule are both self-attention modules.

[0058] In combination with the second aspect and the above implementation, in some possible implementations, the decoder includes M-1 sequentially connected feature decoding layers, and the feature scales corresponding to the first feature decoding layer to the M-1th feature decoding layer increase step by step;

[0059] Among them, the first feature decoding layer is connected to the output end of the feature series layer, the input of the jth feature decoding layer is connected to the output end of the kth first feature extraction layer, and the output end of the M-1th feature decoding layer is the output end of the multimodal surround depth estimation model, j=M-1,M-2,...,2, k=1,2,...,M-2.

[0060] In a third aspect, a distance estimation method is provided, which is applied to a controller of a vehicle, wherein a plurality of ultra-wide-angle cameras are disposed on the vehicle, and the plurality of ultra-wide-angle cameras are dispersedly disposed around a body of the vehicle, and the field of view angles of two adjacent ultra-wide-angle cameras intersect;

[0061] The distance estimation method comprises:

[0062] Acquire a first environment image of the target area captured by each of the plurality of ultra-wide-angle cameras;

[0063] Acquire a second environment image of a target area corresponding to each of the first environment images; wherein the second environment image is a sparse depth map;

[0064] For each of the first environment images, the first environment image and the second environment image corresponding to the first environment image are input into a depth map generation model to obtain a dense depth map of the target area corresponding to each of the first environment images; wherein the depth map generation model is a multimodal surround depth estimation model obtained by the above-mentioned model training method;

[0065] The distance between the vehicle and obstacles in the surrounding environment is determined according to the dense depth maps of the target areas corresponding to each of the plurality of first environment images.

[0066] In conjunction with the third aspect, in some possible implementations, the vehicle is further provided with a detection radar, and the detection radar is used to obtain scene point cloud data around the vehicle;

[0067] The step of acquiring a second environment image of a target area corresponding to each of the plurality of first environment images comprises:

[0068] Acquire second internal parameters of the multiple ultra-wide-angle cameras and second external calibration parameters of the detection radar;

[0069] Acquire regional point cloud data of target areas corresponding to each of the plurality of first environmental images; wherein the regional point cloud data of target areas corresponding to each of the plurality of first environmental images are collected by the detection radar;

[0070] According to the second internal parameter and the second external calibration parameter, the regional point cloud data of the target area corresponding to each of the plurality of first environmental images are projected to an image pixel coordinate system to obtain a second environmental image of the target area corresponding to each of the plurality of first environmental images.

[0071] In a third aspect, a model training device is provided, the model training device comprising:

[0072] A first image acquisition module, used to acquire a first sample image; wherein the first sample image includes a scene image of a target scene corresponding to each direction of the target object;

[0073] A second image acquisition module, used to acquire a second sample image; wherein the second sample image includes a sparse depth map of the target scene corresponding to each direction of the target object;

[0074] A first feature extraction module, used to extract a plurality of first image features of the first sample image; wherein the plurality of first image features have different feature scales;

[0075] A second feature extraction module, used to extract a plurality of second image features of the second sample image; wherein the plurality of second image features have different feature scales;

[0076] A feature fusion module, used for fusing each of the plurality of first image features with a second image feature corresponding to each of the plurality of first image features according to a feature scale, to obtain a target fusion feature;

[0077] A feature decoding module, configured to decode the target fusion feature according to the plurality of first image features to obtain a target sample image; wherein the target sample image includes a dense depth map of the target scene corresponding to each direction of the target object;

[0078] A model training module is used to train a multimodal surround depth estimation model based on the second sample image and the target sample image.

[0079] In a fifth aspect, a distance estimation device is provided, which is configured in a controller of a vehicle, wherein the vehicle is provided with a plurality of ultra-wide-angle cameras, wherein the plurality of ultra-wide-angle cameras are dispersedly arranged around a body of the vehicle, and the field of view angles of two adjacent ultra-wide-angle cameras intersect;

[0080] The distance estimation device comprises:

[0081] A third image acquisition module, used to acquire a first environment image of the target area captured by each of the multiple ultra-wide-angle cameras;

[0082] A fourth image acquisition module, used to acquire a second environment image of a target area corresponding to each of the plurality of first environment images; wherein the second environment image is a sparse depth map;

[0083] A depth map generation module, configured to input the first environment image and a second environment image corresponding to the first environment image into a depth map generation model for each of the first environment images, so as to obtain a dense depth map of the target area corresponding to each of the first environment images; wherein the depth map generation model is a multimodal surround depth estimation model obtained by the above-mentioned model training method;

[0084] The distance determination module is used to determine the distance between the vehicle and obstacles in the surrounding environment according to the dense depth map of the target area corresponding to each of the plurality of first environment images.

[0085] In a sixth aspect, an electronic device is provided, comprising a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the electronic device executes the above-mentioned model training method or distance estimation method.

[0086] In a seventh aspect, a computer program product is provided, which includes: a computer program code, which, when executed on a computer, enables the computer to execute the above-mentioned model training method or distance estimation method.

[0087] In an eighth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the above-mentioned model training method or distance estimation method.

[0088] The model training method, multimodal surround depth estimation model, distance estimation method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application have the following technical effects:

[0089] In the embodiment of the present application, the image features of multiple feature scales of the scene image of the target scene in different directions of the target object are fused with the image features of multiple feature scales of the sparse depth map to obtain the target fusion features, and then the target fusion features are decoded according to the image features of the multiple feature scales of the scene image of the target scene in different directions to obtain dense depth maps of the target scenes in different directions. Then, a multimodal surround view depth estimation model is trained according to the obtained dense depth map and the sparse depth maps of the target scenes in different directions, so that the model has extremely high robustness when used across scenes, solves the problem of weak generalization of the model when used across scenes, and when estimating the distance through the dense depth map output by the multimodal surround view depth estimation model, the estimation accuracy of the distance can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 A schematic flow chart of a model training method provided in an embodiment of the present application is shown;

[0091] Figure 2 A schematic diagram showing the arrangement of cameras and radars on a target object is shown;

[0092] Figure 3 A workflow diagram of a multimodal surround depth estimation model is shown;

[0093] Figure 4 shows a schematic structural diagram of the fusion module;

[0094] Figure 5 The calculation diagram of the cross attention mechanism and the self-attention mechanism is shown;

[0095] Figure 6 An exemplary structural diagram of a multi-modal surround depth estimation model provided in an embodiment of the present application is shown;

[0096] Figure 7 Another exemplary structural diagram of a multimodal surround depth estimation model provided by an embodiment of the present application is shown;

[0097] Figure 8 A schematic flow chart of a distance estimation method provided in an embodiment of the present application is shown;

[0098] Fig. 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown;

[0099] Fig.10 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0100] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0101] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0102] Surround view depth estimation is to estimate the distance between the object and the camera by collecting images from a ring camera and using a deep learning model. Compared with the traditional monocular depth estimation method, surround view depth estimation can improve the accuracy and robustness of depth estimation by collecting information from multiple directions of the ring camera, and can be widely used in autonomous driving, 3D modeling, augmented reality, virtual reality and other fields.

[0103] The implementation of surround depth estimation model based on deep learning technology usually uses convolutional neural networks (CNN) to extract image features and uses regression models to convert image features into depth estimation results. Among them, the design and training process of the CNN network determines the accuracy and speed of this depth estimation. In the design of the CNN network, different numbers and structures of convolutional layers can be used, such as introducing residual connections, batch normalization and other technologies to improve the network effect. In the training stage, it is necessary to select a suitable data set for training and verification, and compare and evaluate different network structures and hyperparameters to obtain better model performance.

[0104] The existing related technologies have the following problems in realizing surround depth estimation:

[0105] 1. The CNN network model has poor generalization when used across scenarios: The data sets used in training the CNN network model often lack scene diversity and rely on the image features of specific scenes to recover depth information, resulting in poor generalization across scenarios.

[0106] 2. A large number of cameras are required: To obtain the depth of a 360-degree field of view, you first need to use multiple cameras to form a ring camera to collect information in multiple directions. Generally, at least 6 cameras are used to form a ring camera. Since the field of view of each camera is relatively small, the total number of cameras is large, the cost of the information collection equipment is high, and it is not easy to maintain.

[0107] Based on the problems existing in the existing related technologies, the example of this application proposes a model training method, a multimodal surround view depth estimation model, a distance estimation method, an apparatus, an electronic device and a computer-readable storage medium. Specifically, a multimodal surround view depth estimation model adapted to the camera image and sparse depth map of an ultra-wide-angle camera is proposed. The ultra-wide-angle camera cooperates with the radar, the camera image is acquired by the ultra-wide-angle camera, and the sparse depth map is generated by the point cloud acquired by the radar. The camera image and the sparse depth map are used as the multimodal surround view depth estimation model, thereby outputting a high-precision dense depth map. Thanks to the assistance of the radar, the model has extremely high robustness when used across scenarios.

[0108] The following is an embodiment of a model training method provided in an embodiment of the present application.

[0109] Figure 1 A schematic flow chart of a model training method provided in an embodiment of the present application is shown, such as Figure 1 As shown, the model training method provided in the embodiment of the present application is applied to electronic devices with computing capabilities, such as computers, servers, etc. The model training method includes the following schemes:

[0110] S110: Acquire a first sample image; wherein the first sample image includes a scene image of a target scene corresponding to each direction of the target object.

[0111] In an exemplary embodiment, the target object may be a stationary object or a moving object, such as a moving car, a stationary car, a target tool, etc. A radar and multiple cameras are arranged on the target object, the radar includes at least one, the number of the multiple cameras is at least 4, the multiple cameras are dispersedly arranged circumferentially on the target object, the field of view angles of two adjacent cameras intersect, each camera corresponds to a direction, and the multiple cameras can obtain images of the target scenes corresponding to each direction of the target object, thereby realizing image acquisition of the environmental scene in the 360-degree direction around the target object, and the radar can rotate 360 ​​degrees to be responsible for the point cloud data acquisition of the environmental scene in the 360-degree direction around the target object.

[0112] The embodiment of the present application is described by taking the number of the multiple cameras as 4, the number of the radar as 1, and the camera as an ultra-wide-angle camera (such as a fisheye camera) as an example. Figure 2 As shown, Figure 2 A schematic diagram showing the arrangement of cameras and radars on a target object is shown. Figure 2In the figure, W represents the target object, R represents the radar, and the radar is set on the top of the target object. It can rotate 360 ​​degrees to collect point cloud data of the environment scene in 360 degrees around the target object. C1-C4 represent cameras. C1-C4 each corresponds to a direction. For example, C1-C4 correspond to the front, back, left and right sides of the target object respectively. FOV1 represents the field of view of C1, FOV2 represents the field of view of C2, FOV3 represents the field of view of C3, and FOV4 represents the field of view of C4. FOV1 intersects with FOV2, FOV2 intersects with FOV3, FOV3 intersects with FOV4, and FOV4 intersects with FOV1. C1-C4 cooperate with each other, and C1-C4 respectively obtain images in one direction, so that images in 360 degrees around the target object can be obtained.

[0113] Acquire a first sample image, the first sample image includes scene images of the target scene corresponding to each direction of the target object, the scene image is also an RGB image, which can also be understood as a visible light image. The scene images of the target scene corresponding to each direction of the target object are acquired by cameras at corresponding positions in each direction of the target object. For example, the first sample image includes: the scene image of the target scene corresponding to the front of the target object, represented as I1, I1 is acquired by C1; the scene image of the target scene corresponding to the back of the target object, represented as I2, I2 is acquired by C2: the scene image of the target scene corresponding to the left side of the target object, represented as I3, I3 is acquired by C3; the scene image of the target scene corresponding to the right side of the target object, represented as I4, I4 is acquired by C4. Among them, I1-I4 can splice out an image of the environmental scene in the 360-degree direction around the target object.

[0114] S120: Acquire a second sample image; wherein the second sample image includes a sparse depth map of the target scene corresponding to each direction of the target object.

[0115] A second sample image is obtained, and the second sample image includes a sparse depth map of the target scene corresponding to each direction of the target object. For example, the second sample image includes: a sparse depth map of the target scene corresponding to the front of the target object, represented by D1, and D1 is generated by the point cloud data of the target scene corresponding to the front of the target object obtained by R; a sparse depth map of the target scene corresponding to the back of the target object, represented by D2, and D2 is generated by the point cloud data of the target scene corresponding to the back of the target object obtained by R: a sparse depth map of the target scene corresponding to the left side of the target object, represented by D3, and D3 is generated by the point cloud data of the target scene corresponding to the left side of the target object obtained by R; a sparse depth map of the target scene corresponding to the right side of the target object, represented by D4, and D4 is generated by the point cloud data of the target scene corresponding to the right side of the target object obtained by R. Among them, D1-D4 can be spliced ​​to form a sparse depth map of the environmental scene in the 360-degree direction around the target object.

[0116] In a possible implementation, the above S120 includes the following solutions:

[0117] Acquire point cloud data of the target scene corresponding to each direction of the target object;

[0118] The point cloud data of the target scene corresponding to each of the directions are projected onto an image pixel coordinate system to obtain a sparse depth map of the target scene corresponding to each of the directions of the target object.

[0119] The process of obtaining the second sample image includes: obtaining point cloud data of the target scene corresponding to each direction of the target object, projecting the point cloud data of the target scene corresponding to each direction to the image pixel coordinate system, obtaining a sparse depth map of the target scene corresponding to each direction of the target object, and obtaining the second sample image. Figure 2 , obtain the point cloud data of the target scenes corresponding to the front, back, left and right sides of the target object through the radar, project the point cloud data of the target scene corresponding to the front of the target object to the image pixel coordinate system, and obtain D1; project the point cloud data of the target scene corresponding to the back of the target object to the image pixel coordinate system, and obtain D2; project the point cloud data of the target scene corresponding to the left side of the target object to the image pixel coordinate system, and obtain D3; project the point cloud data of the target scene corresponding to the right side of the target object to the image pixel coordinate system, and obtain D4.

[0120] In a possible implementation, projecting the point cloud data of the target scene corresponding to each direction to the image pixel coordinate system to obtain the sparse depth map of the target scene corresponding to each direction of the target object includes the following scheme:

[0121] Acquire a first internal parameter of the camera and a first external calibration parameter of the radar;

[0122] According to the first internal parameter and the first external calibration parameter, the point cloud data of the target scene corresponding to each direction is projected to the image pixel coordinate system to obtain a sparse depth map of the target scene corresponding to each direction of the target object.

[0123] The first internal parameters of the camera are also called the internal parameters of the camera. The first external calibration parameters of the radar are pre-calibrated, specifically referring to the external calibration parameters of the image and the radar, which are called the external parameters of the radar.

[0124] After obtaining the point cloud data of the target scene corresponding to each direction of the target object, the point cloud data of the target scene corresponding to each direction is projected to the image pixel coordinate system according to the first internal parameter and the first external calibration parameter to obtain a sparse depth map of the target scene corresponding to each direction of the target object.

[0125] refer to Figure 2 According to the external parameters of the intrinsic participation radar of camera C1, the point cloud data of the target scene corresponding to the front of the target object is projected to the image pixel coordinate system to obtain D1; according to the external parameters of the intrinsic participation radar of camera C2, the point cloud data of the target scene corresponding to the rear of the target object is projected to the image pixel coordinate system to obtain D2; according to the external parameters of the intrinsic participation radar of camera C3, the point cloud data of the target scene corresponding to the left side of the target object is projected to the image pixel coordinate system to obtain D3; according to the external parameters of the intrinsic participation radar of camera C4, the point cloud data of the target scene corresponding to the right side of the target object is projected to the image pixel coordinate system to obtain D4.

[0126] Among them, the sparse depth map is obtained by the following method:

[0127] Since the coordinates of each point in the point cloud data are the coordinates in the world coordinate system, that is, firstly, according to the external parameters of the radar, the coordinates of each point in the point cloud data in the world coordinate system are converted to the camera coordinate system. The coordinates of the points in the point cloud data in the camera coordinate system are expressed as (Xc, Yc, Zc), and then the sparse depth map is obtained by the following formula group, which is as follows:

[0128]

[0129] Where π(x,i) represents the coordinates of a point in the sparse depth map, the intrinsic parameters of the camera are the horizontal focal length fx, the vertical focal length fy, the horizontal center cx, the vertical center cy, k1, k2, ..., kn represent the lens distortion coefficient of the first camera, the lens distortion coefficient of the second camera, ..., the lens distortion coefficient of the nth camera, m is an odd number, mn = n + 1, for example, when n = 1, m = 3, when n = 2, m = 5. Reference Figure 2 , there are 4 cameras, namely C1-C4, then the formula of d(θ) is as follows:

[0130] d(θ)=θ+k1θ 3 +k2θ 5 +k3θ 7 +k4θ 9 ;

[0131] Among them, k1 is the lens distortion coefficient of C1, k2 is the lens distortion coefficient of C2, k3 is the lens distortion coefficient of C3, and k4 is the lens distortion coefficient of C4.

[0132] Since the first sample image includes scene images of the target scene corresponding to each direction of the target object, and the second sample image includes a sparse depth map of the target scene corresponding to each direction of the target object, it is beneficial to achieve depth estimation from multiple perspectives through the same model.

[0133] S130: Extracting a plurality of first image features of the first sample image; wherein the plurality of first image features have different feature scales.

[0134] S140: Extracting a plurality of second image features of the second sample image; wherein the plurality of second image features have different feature scales.

[0135] After obtaining the first sample image, multiple first image features of the first sample image are extracted, that is, the image features of the first sample image are extracted according to preset feature scales of different sizes, and multiple first image features of the first sample image are obtained, and the feature scales of the multiple first image features are different. It can also be understood that: for the scene image of the target scene corresponding to each direction of the target object, multiple first image features of the scene image are extracted, and the feature scales of the multiple first image features are different.

[0136] The first image feature is a feature of the image captured by the camera. Indicates that t represents the serial number of the camera in each direction, s represents the different feature scales of the same image feature, t∈{1,2,3,...,a}, s∈{0,1,2,3,...,b}, for example, when t=4, t∈[1,4], indicating that there are 4 cameras, and the 4 cameras are numbered from 1 to 4, when s=4, s∈[0,4], then the multiple first image features of the scene image corresponding to each direction of the target object are: refer to Figure 2 , the multiple first image features of the scene image corresponding to C1 are: The multiple first image features of the scene image corresponding to C2 are: The multiple first image features of the scene image corresponding to C3 are: The multiple first image features of the scene image corresponding to C4 are:

[0137]

[0138] After obtaining the second sample image, multiple second image features of the second sample image are extracted, that is, the image features of the second sample image are extracted according to preset feature scales of different sizes, and multiple second image features of the second sample image are obtained, and the feature scales of the multiple second image features are different. It can also be understood that: for the sparse depth map of the target scene corresponding to each direction of the target object, multiple second image features of the sparse depth map are extracted, and the feature scales of the multiple second image features are different.

[0139] The second image feature refers to the depth feature. For example, when t=4, t∈[1,4] means there are 4 cameras, which are numbered from 1 to 4. When s=4, s∈[0,4], then the multiple second image features of the sparse depth map corresponding to each direction of the target object are: refer to Figure 2 , the multiple second image features of the sparse depth map corresponding to C1 are: The multiple first image features of the sparse depth map corresponding to C2 are: The multiple first image features of the sparse depth map corresponding to C3 are: The multiple first image features of the sparse depth map corresponding to C4 are:

[0140] In one possible implementation, Figure 3 As shown, Figure 3 The work flow chart of the multimodal surround depth estimation model is shown, and the multimodal surround depth estimation model includes a deformable convolutional neural network (Deforable Convolutional Network), a sparse convolutional neural network (Sparse Convolutional Network), multiple fusion modules FM and a decoder (Decoder). Among them, the feature scales corresponding to the multiple fusion modules are different, the deformable convolutional neural network is used to extract image features of multiple feature scales from the image captured by the camera, and the convolution operation involved in the deformable convolutional neural network is a variable convolution operation. The sparse convolutional neural network is used to extract image features of multiple feature scales from the sparse depth map, and the convolution operation involved in the sparse convolutional neural network is a sparse convolution operation.

[0141] The step of extracting a plurality of first image features of the first sample image comprises:

[0142] The deformable convolutional neural network is used to extract image features of the first sample image at each of a plurality of preset feature scales to obtain the plurality of first image features; wherein the plurality of preset feature scales are of different sizes.

[0143] The first sample image is input into a deformable convolutional neural network, and the image features of the first sample image at each of a plurality of preset feature scales are extracted by the deformable convolutional neural network to obtain a plurality of first image features, wherein the feature scales of the plurality of first image features are different.

[0144] It can be understood that for the scene image of the target scene corresponding to each direction of the target object, the scene image is input into the deformable convolutional neural network, and the image features of the scene image at multiple preset feature scales are extracted by the deformable convolutional neural network to obtain multiple first image features of the scene image. The image feature extraction order of the scene image is to first extract the image features of the largest preset feature scale, then extract the image features of the second largest preset feature scale, and so on, and finally extract the image features of the smallest preset feature scale. The size sorting of the multiple preset feature scales is from large to small. For example, the size of the input scene image is 384×256, and the size sorting of the multiple preset feature scales is: 192×128, 96×64, 48×32, 24×16, 12×8, which also represents the extraction order of the multiple first image features.

[0145] refer to Figure 2 For example, the first image features of the scene image I1 corresponding to C1 are: The feature scale is 192×128, The feature scale is 96×64, The feature scale is 48×32, The characteristic scale is 24×16, IF1 4 The characteristic scale of is 12×8. Similarly, the characteristic scales of the multiple first image features of the scene images corresponding to C2-C4 are also the same as the characteristic scales of the multiple first image features of the scene images corresponding to C1. For example, and The feature scale of each image is 192×128.

[0146] The step of extracting a plurality of second image features of the second sample image comprises:

[0147] The sparse convolutional neural network is used to extract image features of the second sample image at each of the multiple preset feature scales to obtain the multiple second image features.

[0148] The second sample image is input into a sparse convolutional neural network, and the image features of the second sample image at each of a plurality of preset feature scales are extracted through the sparse convolutional neural network to obtain a plurality of second image features, wherein the plurality of second image features have different feature scales.

[0149] It can be understood that for the sparse depth map of the target scene corresponding to each direction of the target object, the sparse depth map is input into the sparse convolutional neural network, and the image features of the sparse depth map at multiple preset feature scales are extracted by the sparse convolutional neural network to obtain multiple second image features of the sparse depth map. The image feature extraction order of the sparse depth map is to first extract the image features of the largest preset feature scale, then extract the image features of the second largest preset feature scale, and so on, and finally extract the image features of the smallest preset feature scale. The size sorting of the multiple preset feature scales is from large to small. For example, the size sorting of the multiple preset feature scales is: 192×128, 96×64, 48×32, 24×16, 12×8, which also represents the extraction order of the multiple second image features.

[0150] refer to Figure 2 , for example, the multiple second image features of the sparse depth map D1 corresponding to C1 are: PF1 0 The feature scale is 192×128, The feature scale is 96×64, PF1 2 The feature scale is 48×32, PF1 3 The feature scale is 24×16, PF1 4 The feature scale of PF1 is 12×8. Similarly, the feature scales of the multiple second image features of the sparse depth maps corresponding to C2-C4 are the same as the feature scales of the multiple second image features of the sparse depth map corresponding to C1. For example, PF1 0 and The feature scale of each image is 192×128.

[0151] Deformable convolutional neural network refers to a form of feature extraction network, which is used to extract image features. Sparse convolutional neural network is another form of feature extraction network, which is used to extract depth features of depth maps. The two different forms of feature extraction networks are used respectively, which can efficiently extract features from sparse depth maps and images captured by cameras, and have the characteristics of taking into account both modalities at the same time.

[0152] S150: According to the feature scale, fuse each of the plurality of first image features with a second image feature corresponding to each of the plurality of first image features to obtain a target fused feature.

[0153] After extracting multiple first image features of the first sample image and multiple second image features of the second sample image, the two image features are fused, that is, according to the feature scale, each of the multiple first image features is fused with the second image features corresponding to each of the multiple first image features to obtain a target fused feature. For each first image feature, according to the feature scale, the first image feature is fused with the second image feature corresponding to the first image feature. After the multiple first image features are fused with the respective corresponding second image features according to the size of the feature scale, the target fused feature of the image features of the first sample image and the image features of the second sample image is obtained, that is, the fusion of the image features of the first sample image and the image features of the second sample image is achieved.

[0154] For example, the first image features of I1 are: The second image features of D1 are: The fusion process is: IF1 0 With PF1 0 Fusion, and Fusion, IF1 2 and Fusion, IF1 3 and Fusion, IF1 4 and After the fusion of the multiple first image features of I1 and the corresponding multiple second image features is completed, the target fusion features of I1 and D1 are obtained. By performing feature fusion through multiple fusion modules, data enhancement can be achieved.

[0155] In a possible implementation, the above S150 includes the following solutions:

[0156] Determining, from the plurality of fusion modules, target fusion modules corresponding to the plurality of first image features respectively according to the feature scale;

[0157] Using target fusion modules corresponding to the plurality of first image features, each of the plurality of first image features is fused with a second image feature corresponding to the plurality of first image features to obtain a fusion result; wherein the fusion result includes fused image features corresponding to the plurality of first image features and fused depth features corresponding to the plurality of first image features;

[0158] The target fusion feature is generated according to the fusion result.

[0159] Since the feature scales corresponding to the multiple fusion modules are different, each fusion module fuses the first image features and the second image features with the corresponding feature scale according to the corresponding feature scale. For example, fusion module 1 is used to fuse the first image features and the second image features with a feature scale of 192×128, and fusion module 2 is used to fuse the first image features and the second image features with a feature scale of 96×64, and so on.

[0160] When fusing multiple first image features of a first sample image with their respective corresponding second image features, it is necessary to select a corresponding fusion module, that is, determine the target fusion modules corresponding to each of the multiple first image features from the multiple fusion modules according to the feature scale, and then, use the target fusion modules corresponding to each of the multiple first image features to fuse each of the multiple first image features with the second image features corresponding to each of the multiple first image features to obtain a fusion result. The fusion result includes fused image features corresponding to each of the multiple first image features and fused depth features corresponding to each of the multiple first image features, and then, the target fusion features of the image features of the first sample image and the image features of the second sample image are generated according to the fusion result.

[0161] For example, the multiple fusion modules are fusion module 1, fusion module 2, ..., fusion module n, where the feature scale corresponding to fusion module 1 is 192×128, and IF1 of I1 0 PF1 with D1 0 The feature scale of is 192×128, so the fusion module 1 is used to IF1 of I1 0 PF1 with D1 0 The remaining first image features of I1 are fused with their respective corresponding second image features in the same way to finally obtain a fusion result, that is, the fusion result includes the fused image features corresponding to I1 and the fused depth features corresponding to D1. The fused image features corresponding to I1 and the fused depth features corresponding to D1 are connected in series to obtain the target fusion features of the first image features of I1 and the second image features of D1.

[0162] In one possible implementation, the target fusion module includes a first module group and a second module group, the first module group includes a plurality of first processing modules connected in series, each first processing module includes a first submodule and a second submodule connected to an output end of the first submodule, the second module group includes a plurality of second processing modules connected in series, each second processing module includes a third submodule and a fourth submodule connected to an output end of the third submodule, the first submodule and the third submodule are both cross-attention modules (Cross-Attention), and the second submodule and the fourth submodule are both self-attention modules (Self-Attention).

[0163] like Figure 4 As shown, Figure 4 The schematic diagram of the structure of the fusion module is shown. For example, the first module group in the target fusion module includes two first processing modules connected in series, and the second module group includes two second processing modules connected in series. The output end of the first submodule H11 in the first first processing module H1 is connected to the input end of the second submodule H12, the input end of the first submodule H21 in the second first processing module H2 is connected to the output end of the second submodule H12, and the output end of the first submodule H21 is connected to the input end of the second submodule H22; the output end of the third submodule G11 in the first second processing module G1 is connected to the input end of the fourth submodule G12, the input end of the third submodule G21 in the second second processing module G2 is connected to the output end of the fourth submodule G12, and the output end of the third submodule G21 is connected to the input end of the fourth submodule G22. Among them, the first submodule H11, the first submodule H21, the third submodule G11 and the third submodule G21 are all cross-attention modules, and the second submodule H12, the second submodule H22, the fourth submodule G12 and the fourth submodule G22 are all self-attention modules.

[0164] The target fusion modules corresponding to the plurality of first image features are used to fuse the plurality of first image features with the second image features corresponding to the plurality of first image features to obtain the fusion results, including the following schemes:

[0165] Acquire a first linear feature corresponding to each of the plurality of first image features, and acquire a second linear feature corresponding to each of the plurality of second image features; wherein the first linear feature and the second linear feature both include a Q feature, a K feature, and a V feature;

[0166] The first linear feature and the second linear feature are cross-input into the first module group and the first module group to obtain fused image features corresponding to each of the plurality of first image features and fused depth features corresponding to each of the plurality of second image features.

[0167] like Figure 4 As shown, IFn represents any first image feature, DFn represents the second image feature corresponding to any first image feature, IFR represents the fused image feature, DFR represents the fused depth feature, the Q feature, K feature and V feature of the first image feature are represented as IQ, IK and IV respectively, and the Q feature, K feature and V feature of the second image feature are represented as DQ, DK and DV respectively.

[0168] The feature scales of IFn and DFn are the same, that is, the feature scale corresponding to the target fusion module is the same as the feature scales of IFn and DFn. Multiple fusion modules adopt a fusion structure of multimodal cross fusion. When performing feature fusion, the target fusion module obtains the first linear features of IFn, namely IQ, IK and IV, and obtains the second linear features of DFn, namely DQ, DK and DV. Then, DQ, DV and IK are input to H11 to obtain the first intermediate quantity C1, and DK, IV and IV are input to G11 to obtain the second intermediate quantity C2, and then C1 is input to H12 to obtain the third intermediate quantity S11, and C2 is input to G12 to obtain the fourth intermediate quantity S12; repeat the above operation, obtain new IQ, IK and IV from S11, obtain new DQ, DK and DV from S12, and then, input the new DK, DQ and IV to H21 to obtain the fifth intermediate quantity C3, and input the new DV, IQ and IK to G21 to obtain the sixth intermediate quantity C4, and then input C3 to H22 to obtain the seventh intermediate quantity S21, and input C4 to G22 to obtain the eighth intermediate quantity S22, that is, S21 is the fused depth feature, and S22 is the fused image feature. In the above manner, the fused image features corresponding to each of the multiple first image features and the fused depth features corresponding to each of the multiple second image features can be obtained.

[0169] Since the fusion module adopts a multimodal cross-fusion structure, which takes image features and depth map features as input and finally outputs the fused two, its implementation mechanism has excellent effect.

[0170] Among them, the cross attention module uses the cross attention mechanism, and the self-attention module uses the self-attention mechanism. The calculation process of the cross attention mechanism and the self-attention mechanism is as follows Figure 5 As shown, Figure 5 The figure shows the calculation diagram of the cross-attention mechanism and the self-attention mechanism.

[0171] The difference between the cross-attention mechanism and the self-attention mechanism is only the different sources of Q, K, and V, and the calculation method is the same. Specifically: the cross-attention mechanism refers to the cross-alignment of different modal information in the feature space in the multimodal information scenario to learn the correlation between different modal information to achieve better representation learning effect. In general, the cross-attention mechanism needs to go through the following calculation process:

[0172] 1. Assume that there are two modal input data X and Y, each of which contains n features, that is: X = [x1, x2, ..., x n ],Y=[y1,y2,...,y n ].

[0173] 2. Perform linear transformation on X and Y respectively, project them into a common space, and obtain new features, denoted as F X and F Y , where: F X =W x X+b x , F X =W y Y+b y , W x , W y is the parameter of the linear transformation, b x , b y is the bias term.

[0174] 3. For F X and F Y Calculate the attention weight. There are many ways to calculate the attention weight, such as using the dot product method: A is then normalized to obtain the weight distribution a. For example, the normalization is achieved through a softmax function.

[0175] 4. Use the weighted summation of attention weight a to calculate the interactive representation of input data X and Y in the feature space:

[0176] 5. In F interact Perform a nonlinear transformation of the channel dimension (for example, using a 1×1 convolution) to obtain the final cross-attention feature representation F out .

[0177] 6. Output F out as input to the next layer.

[0178] S160: Decode the target fusion feature according to the multiple first image features to obtain a target sample image; wherein the target sample image includes a dense depth map of the target scene corresponding to each direction of the target object.

[0179] After obtaining the target fusion feature of the image feature of the first sample image and the image feature of the second sample image, the target fusion feature is decoded according to the multiple first image features of the first sample image to obtain a target sample image, denoted as DP, and the target sample image includes dense depth maps of the target scene corresponding to each direction of the target object, namely, the target sample image includes dense depth maps DP1, DP2, ..., DPn of the target scene corresponding to each direction of the target object. Figure 2 and Figure 3, the target sample images include DP1, DP2, DP3 and DP4, that is, DP1 is obtained by fusing and decoding the image features of I1 and D1, DP2 is obtained by fusing and decoding the image features of I2 and D2, DP3 is obtained by fusing and decoding the image features of I3 and D3, and DP4 is obtained by fusing and decoding the image features of I4 and D4.

[0180] In a possible implementation, S160 includes the following solutions:

[0181] The multiple first image features and the target fusion features are input into the decoder, so that the target fusion features are decoded by the decoder to obtain the target sample image.

[0182] In the decoding stage of image features, multiple first image features of the first sample image and the target fusion features are taken as input of the decoder and then input into the decoder. The decoder decodes the target fusion features by upsampling, that is, upsampling the target fusion features according to the multiple first image features to obtain the target sample image.

[0183] For example, multiple first image features of I1 and target fusion features of I1 and D1 are input into the decoder, and the decoder upsamples the target fusion features according to the multiple first image features of I1 to obtain a dense depth map DP1 of the target scene corresponding to the front of the target object.

[0184] The multiple first image features of I2 and the target fusion features of I2 and D2 are input into the decoder. The decoder upsamples the target fusion features according to the multiple first image features of I3 to obtain a dense depth map DP2 of the target scene corresponding to the back of the target object.

[0185] The multiple first image features of I3 and the target fusion features of I3 and D3 are input into the decoder. The decoder upsamples the target fusion features according to the multiple first image features of I3 to obtain a dense depth map DP3 of the target scene corresponding to the left side of the target object.

[0186] The multiple first image features of I4 and the target fusion features of I4 and D4 are input into the decoder. The decoder upsamples the target fusion features according to the multiple first image features of I4 to obtain a dense depth map DP4 of the target scene corresponding to the right side of the target object.

[0187] After multiple first image features and target fusion features are input into the decoder, the decoder simultaneously aggregates features of multiple feature scales, thereby enabling high-precision depth estimation for targets of various shapes.

[0188] S170: Training a multimodal surround depth estimation model according to the second sample image and the target sample image.

[0189] In the model training phase, the multimodal surround depth estimation model is iteratively trained, and the difference between the depth value of the target sample image output by the model and the depth value of the second sample image after each round of training is calculated. If the difference meets the training expectations, the model training is stopped, and the multimodal surround depth estimation model is completed. The trained multimodal surround depth estimation model can be applied to autonomous driving, 3D modeling, augmented reality, virtual reality and other fields for distance estimation.

[0190] In a possible implementation, S170 includes the following solutions:

[0191] Determine the difference between the dense depth map and the sparse depth map of the target scene corresponding to each direction of the target object, and obtain multiple differences;

[0192] Determine a loss function based on an average difference of the plurality of differences;

[0193] The multimodal surround view depth estimation model is trained according to the loss function.

[0194] Since the input of the multimodal surround depth estimation model includes the scene image and sparse depth map of the target scene corresponding to the target object in different directions, that is, the output is the dense depth map of the target scene corresponding to the target object in different directions, the damage function of the model is set according to the average value. Specifically, during model training, the SmoothsL1 loss function is used for supervised training of the model.

[0195] Calculate the difference between the dense depth map and the sparse depth map of the target scene corresponding to each direction of the target object, obtain multiple differences, calculate the average difference of the multiple differences, and set the SmoothsL1 loss function according to the average difference. The loss function is as follows:

[0196]

[0197] Here, x represents the mean difference.

[0198] After iteratively training the multimodal surround depth estimation model, if the function curve area of ​​the SmoothsL1 loss function is stable, it means that the model has converged, and the training of the multimodal surround depth estimation model is stopped, and the model training is completed. In addition, in addition to SmoothsL1, the loss function of the multimodal surround depth estimation model can also be set to L1 loss function or KL divergence (Kullback-Leibler Divergence).

[0199] In the embodiment of the present application, the image features of multiple feature scales of the scene image of the target scene in different directions of the target object are fused with the image features of multiple feature scales of the sparse depth map to obtain the target fusion features, and then the target fusion features are decoded according to the image features of the multiple feature scales of the scene image of the target scene in different directions to obtain dense depth maps of the target scenes in different directions. Then, a multimodal surround view depth estimation model is trained according to the obtained dense depth map and the sparse depth maps of the target scenes in different directions, so that the model has extremely high robustness when used across scenes, solves the problem of weak generalization of the model when used across scenes, and when estimating the distance through the dense depth map output by the multimodal surround view depth estimation model, the estimation accuracy of the distance can be improved.

[0200] The following is an embodiment of a multi-modal surround depth estimation model provided in an embodiment of the present application.

[0201] Figure 6 An exemplary structural diagram of a multi-modal surround depth estimation model provided in an embodiment of the present application is shown. Figure 6 As shown, the multimodal surround depth estimation model provided by the embodiment of the present application includes a feature processing module and a decoder. The output of the feature processing module is the input of the decoder. The input of the feature processing module includes at least one, each input corresponds to a direction, and all inputs correspond to different directions. Each input of the feature processing module includes a sparse depth map and an RGB image. The output of the decoder includes at least one, each output corresponds to a direction, and all outputs correspond to different directions. Each input of the feature processing module corresponds to an output of the decoder, and each output of the decoder is a dense depth map. The feature processing module can realize image feature extraction of RGB images and depth feature extraction of sparse depth maps, as well as realize the fusion of the two features. The decoder is used to decode the fused features according to the image features of the current RGB image and output a dense depth map.

[0202] A feature processing module is used to extract multiple image features of each of the first image and the second image to obtain a feature set; wherein the first image includes: a scene image of a target scene corresponding to at least one direction of the target object, and the second image includes: a sparse depth map of the target scene corresponding to at least one direction of the target object; the feature set includes: a plurality of first image features corresponding to the first image and a plurality of second image features corresponding to the second image, wherein the plurality of first image features have different feature scales, and the plurality of first image features have different feature scales;

[0203] The feature processing module is further used to fuse each of the plurality of first image features with a second image feature corresponding to each of the plurality of first image features according to a feature scale to obtain a target fused feature;

[0204] A decoder is used to decode the target fusion feature according to the multiple first image features to obtain a target image; wherein the target image includes: a dense depth map of the target scene corresponding to at least one direction of the target object.

[0205] like Figure 2 As shown, it is assumed that the first image includes a scene image of the target scene corresponding to the front direction of the target object, represented as I1, and the second image includes a sparse depth map of the target scene corresponding to the front direction of the target object, represented as D1. I1 and D1 are used as inputs of the feature processing module. The feature processing module extracts multiple image features of I1 and extracts multiple image features of D1 to obtain a feature set. The feature set includes multiple first image features of I1, that is, The feature scale is 192×128, The feature scale is 96×64, IF1 2 The feature scale is 48×32, The characteristic scale is 24×16, IF1 4 The feature scale of is 12×8; multiple second image features of D1, namely PF1 0 The feature scale is 192×128, The feature scale is 96×64, PF1 2 The feature scale is 48×32, PF1 3 The feature scale is 24×16, PF1 4 The characteristic scale is 12×8.

[0206] Then, the feature processing module will With PF1 0 Fusion, IF1 1 and Fusion, IF1 2 With PF1 2 Fusion, IF1 3 and Fusion, and After the fusion of the multiple first image features of I1 and the corresponding multiple second image features is completed, the target fusion features of I1 and D1 are obtained.

[0207] The feature processing module inputs the target fusion feature and multiple first image features of I1 to the decoder, and the decoder decodes the target fusion feature. The decoding adopts an upsampling method to output a dense depth map of the target scene corresponding to the front direction of the target object. The distance between the target object and the obstacles in the target scene corresponding to its front direction can be estimated through the dense depth map.

[0208] In the case where the first image includes scene images of the target scene corresponding to each direction of the target object, and the second image includes sparse depth maps of the target scene corresponding to each direction of the target object, the feature processing module will have multiple inputs, each input is a scene image and a sparse depth map of the target scene corresponding to different directions of the target object, and the decoder has multiple outputs, thereby outputting dense depth maps of the target scene corresponding to different directions of the target object. Among them, the feature fusion process and decoding process of the scene image and the sparse depth map of the target scene corresponding to each direction are the same, and can be referred to the feature fusion process and decoding process of the scene image and the sparse depth map of the target scene corresponding to the front direction of the target object mentioned above, which will not be repeated here.

[0209] In one possible implementation, Figure 7 Another exemplary structural diagram of a multi-modal surround depth estimation model provided in an embodiment of the present application is shown. Figure 7 As shown, the feature processing module includes: a sparse convolutional neural network, a sparse convolutional neural network, M-1 fusion modules FM and a feature series layer, the feature series layer is connected to the decoder, and M is an odd number greater than or equal to 5.

[0210] Deformable convolutional neural network package: M first feature extraction layers, the convolution operation used in the M first feature extraction layers is a deformable convolution operation, and the preset feature scales corresponding to the 1st first feature extraction layer to the Mth first feature extraction layer decrease step by step. For example, the preset feature scale corresponding to the 1st first feature extraction layer is greater than the preset feature scale corresponding to the 2nd first feature extraction layer, and the preset feature scale corresponding to the 2nd first feature extraction layer is greater than the preset feature scale corresponding to the 3rd first feature extraction layer. The number of channels between adjacent first feature extraction layers increases step by step, for example, the number of channels of the 1st first feature extraction layer is 3, the number of channels of the 2nd first feature extraction layer is 64, the number of channels of the 3rd first feature extraction layer is 125, the number of channels of the 4th first feature extraction layer is 256, the number of channels of the 5th first feature extraction layer is 512, and so on.

[0211] The sparse convolutional neural network includes M second feature extraction layers, and the convolution operation adopted by the M second feature extraction layers is a sparse convolution operation. The preset feature scales corresponding to the first second feature extraction layer to the Mth second feature extraction layer decrease step by step. For example, the preset feature scale corresponding to the first second feature extraction layer is greater than the preset feature scale corresponding to the second second feature extraction layer, and the preset feature scale corresponding to the second second feature extraction layer is greater than the preset feature scale corresponding to the third second feature extraction layer. The number of channels between adjacent second feature extraction layers increases step by step. For example, the number of channels of the first second feature extraction layer is 3, the number of channels of the second second feature extraction layer is 64, the number of channels of the third second feature extraction layer is 125, the number of channels of the fourth second feature extraction layer is 256, the number of channels of the fifth second feature extraction layer is 512, and so on.

[0212] The output ends of the i-th first feature extraction layer and the i-th second feature extraction layer are respectively connected to the input end of the i-th fusion module, and the output end of the i-th fusion module is respectively connected to the input ends of the i+1-th first feature extraction layer and the i+1-th second feature extraction layer; the output ends of the M-th first feature extraction layer and the M-th second feature extraction layer are connected to the feature series connection layer; the output ends of the 1st first feature extraction layer to the M-2th first feature extraction layer are respectively connected to the decoder, i=1,2,3,...,M-1.

[0213] For example, when M = 5, see Figure 3 , the output ends of the first first feature extraction layer and the first second feature extraction layer are respectively connected to the input end of the first fusion module, and the output end of the first fusion module is respectively connected to the input end of the second first feature extraction layer and the second second feature extraction layer; the output ends of the second first feature extraction layer and the second second feature extraction layer are respectively connected to the input end of the second fusion module, and the output end of the second fusion module is respectively connected to the input end of the third first feature extraction layer and the third second feature extraction layer; the output ends of the third first feature extraction layer and the third second feature extraction layer are respectively connected to the input end of the third fusion module. The output ends of the third fusion module are connected to the input ends of the fourth first feature extraction layer and the fourth second feature extraction layer respectively; the output ends of the fourth first feature extraction layer and the fourth second feature extraction layer are each connected to the input end of the fourth fusion module, the output end of the fourth fusion module is respectively connected to the input ends of the fifth first feature extraction layer and the fifth second feature extraction layer respectively, the output ends of the fifth first feature extraction layer and the fifth second feature extraction layer are connected to the feature series connection layer, and the output ends of the first first feature extraction layer, the second first feature extraction layer and the third first feature extraction layer are each connected to the decoder.

[0214] The first first feature extraction layer can be understood as the input layer of the deformable convolutional neural network, which has at least one input, each of which is an RGB image. I1, I2, ..., In are each an input, and each RGB image of the input corresponds to a different direction; the second first feature extraction layer can be understood as the input layer of the sparse convolutional neural network, which has at least one input, each of which is a sparse depth map. D1, D2, ..., Dn are each an input, and each sparse depth map of the input corresponds to a different direction. I1 and D1 correspond to the same direction, I2 and D2 correspond to the same direction, and so on.

[0215] The i-th first feature extraction layer is used to extract first image features of the first image according to a preset feature scale corresponding to the i-th first feature extraction layer; the i-th second feature extraction layer is used to extract second image features of the second image according to the preset feature scale corresponding to the i-th second feature extraction layer, and the preset feature scale corresponding to the i-th first feature extraction layer is the same as the preset feature scale corresponding to the i-th second feature extraction layer; the i-th fusion module is used to fuse the first image features extracted by the i-th first feature extraction layer and the second image features extracted by the i-th second feature extraction layer.

[0216] For example, if an input of the multimodal surround depth estimation model is I1 and D1, I1 is an input of the deformable convolutional neural network, D1 is an input of the sparse convolutional neural network, I1 and D1 correspond to the same direction, assuming M = 5, the preset feature scales corresponding to the first feature extraction layer to the fifth first feature extraction layer are: 192×128, 96×64, 48×32, 24×16, 12×; the preset feature scales corresponding to the first second feature extraction layer to the fifth second feature extraction layer are: 192×128, 96×64, 48×32, 24×16, 12×8, then:

[0217] The first first image feature of I1 extracted by the first first feature extraction layer is IF1 0 , IF1 0 The feature scale is 192×128, and the first second image feature of D1 extracted by the first second feature extraction layer is PF1 0 , PF1 0 The feature scale is 192×128, and the first fusion module FM is used for IF1 0 and PF1 0 , and fuse to obtain a first fusion result, the first fusion result includes a first fusion image feature and a first fusion depth feature.

[0218] The first fused image feature is input into the second first feature extraction layer, the first fused depth feature is input into the second second feature extraction layer, and the second first image feature of I1 extracted by the second first feature extraction layer is The feature scale of is 96×64, and the second second image feature of D1 extracted by the first second feature extraction layer is The feature scale of is 96×64, and the second fusion module FM is and The fusion is performed to obtain a second fusion result, where the second fusion result includes the first fusion image feature and the second fusion depth feature.

[0219] And so on;

[0220] The fourth fused image feature is input into the fifth first feature extraction layer, the fourth fused depth feature is input into the fifth second feature extraction layer, and the fifth first image feature of I1 extracted by the fifth first feature extraction layer is The feature scale of is 12×8, and the 5th second image feature of D1 extracted by the 5th second feature extraction layer is The characteristic scale is 12×8, and Input feature concatenation layer, feature concatenation layer pair and The target fusion features of I1 and D1 are obtained by fusion.

[0221] The generation process of the target fusion features of I2 and D2, the target fusion features of I3 and D3, ..., the target fusion features of In and Dn is the same as the generation process of the target fusion features of I1 and D1, and will not be repeated here.

[0222] In one possible implementation, each fusion module includes a first module group and a second module group, the first module group includes a plurality of first processing modules connected in series, each first processing module includes a first submodule and a second submodule connected to an output end of the first submodule, the second module group includes a plurality of second processing modules connected in series, each second processing module includes a third submodule and a fourth submodule connected to an output end of the third submodule, the first submodule and the third submodule are both cross-attention modules, and the second submodule and the fourth submodule are both self-attention modules.

[0223] like Figure 4 As shown, Figure 4The schematic diagram of the structure of the fusion module is shown. For example, the first module group in each fusion module includes two first processing modules connected in series, and the second module group includes two second processing modules connected in series. The output end of the first submodule H11 in the first first processing module H1 is connected to the input end of the second submodule H12, the input end of the first submodule H21 in the second first processing module H2 is connected to the output end of the second submodule H12, and the output end of the first submodule H21 is connected to the input end of the second submodule H22; the output end of the third submodule G11 in the first second processing module G1 is connected to the input end of the fourth submodule G12, the input end of the third submodule G21 in the second second processing module G2 is connected to the output end of the fourth submodule G12, and the output end of the third submodule G21 is connected to the input end of the fourth submodule G22. Among them, the first submodule H11, the first submodule H21, the third submodule G11 and the third submodule G21 are all cross-attention modules, and the second submodule H12, the second submodule H22, the fourth submodule G12 and the fourth submodule G22 are all self-attention modules.

[0224] Each fusion module performs feature fusion based on IF1 0 and PF1 0 Taking fusion as an example, the feature fusion process is as follows:

[0225] IFn is IF1 0 , DFn is PF1 0 , IF1 0 The Q feature, K feature and V feature are represented as IQ, IK and IV respectively, PF1 0 The Q feature, K feature and V feature are represented as DQ, DK and DV respectively.

[0226] Get IF1 0 The first linear features of , namely IQ, IK and IV, are used to obtain PF1 0The second linear feature, namely DQ, DK and DV. Then, DQ, DV and IK are input to H11 to obtain the first intermediate quantity C1, and DK, IV and IV are input to G11 to obtain the second intermediate quantity C2, and then C1 is input to H12 to obtain the third intermediate quantity S11, and C2 is input to G12 to obtain the fourth intermediate quantity S12; repeat the above operation, obtain new IQ, IK and IV from S11, obtain new DQ, DK and DV from S12, and then, input the new DK, DQ and IV to H21 to obtain the fifth intermediate quantity C3, and input the new DV, IQ and IK to G21 to obtain the sixth intermediate quantity C4, and then input C3 to H22 to obtain the seventh intermediate quantity S21, and input C4 to G22 to obtain the eighth intermediate quantity S22, that is, S21 is the fused depth feature, and S22 is the fused image feature.

[0227] The fusion process of the first image features and the second image features at other feature scales is similar to IF1 0 and PF1 0 The fusion process is the same and will not be elaborated here.

[0228] In one possible implementation, Figure 7 As shown, the decoder includes M-1 sequentially connected feature decoding layers, the feature scales corresponding to the 1st feature decoding layer to the M-1th feature decoding layer increase step by step, and the number of channels from the 1st feature decoding layer to the M-1th feature decoding layer decreases step by step. Wherein, the 1st feature decoding layer is connected to the output end of the feature series connection layer, the input of the jth feature decoding layer is connected to the output end of the kth first feature extraction layer, and the output end of the M-1th feature decoding layer is the output end of the multimodal surround depth estimation model, j = M-1, M-2, ..., 2, k = 1, 2, ..., M-2.

[0229] refer to Figure 3 and Figure 7 , when M=5, the decoder includes 4 feature decoding layers connected in sequence, the first feature decoding layer is connected to the output end of the feature series connection layer, the input of the fourth feature decoding layer is connected to the output end of the first first feature extraction layer, the input of the third feature decoding layer is connected to the output end of the second first feature extraction layer, and the input of the second feature decoding layer is connected to the output end of the third first feature extraction layer.

[0230] Taking I1 and D1 as an example, when decoding, The target fusion features of I1 and D1 are input into the decoder, and the decoder is based on The target fusion features are upsampled to obtain a dense depth map DP1 of the target scene in the directions corresponding to I1 and D1.

[0231] The generation process of the dense depth map DP2 of the target scene in the direction corresponding to I2 and D2, the dense depth map DP3 of the target scene in the direction corresponding to I3 and D3, ..., the dense depth map DPn of the target scene in the direction corresponding to In and Dn is the same as the generation process of the dense depth map DP1, and will not be followed here.

[0232] The following is an embodiment of a distance estimation method provided in an embodiment of the present application.

[0233] Figure 8 A schematic flow chart of a distance estimation method provided in an embodiment of the present application is shown. Figure 8 As shown, the distance estimation method provided in the embodiment of the present application is applied to the controller of a vehicle, and a plurality of ultra-wide-angle cameras are arranged on the vehicle, and the plurality of ultra-wide-angle cameras are dispersedly arranged around the body of the vehicle, and the field of view angles of two adjacent ultra-wide-angle cameras intersect, and the number of the plurality of ultra-wide-angle cameras is at least 4; a detection radar is also arranged on the vehicle, and the detection radar is used to obtain scene point cloud data around the vehicle, and the detection radar includes at least one. The ultra-wide-angle camera is, for example, a fisheye camera, and the vehicle can be a vehicle, a ship, an airplane, etc. This embodiment is described by taking the vehicle as an example. Figure 2 As shown, W represents the vehicle, R represents the detection radar, which is set on the roof. The 360-degree rotation can collect scene point cloud data of the environment around the vehicle in 360 degrees. C1-C4 represent ultra-wide-angle cameras. C1-C4 each corresponds to a direction, such as C1-C4 respectively correspond to the front, rear, left and right sides of the vehicle, FOV1 represents the field of view of C1, FOV2 represents the field of view of C2, FOV3 represents the field of view of C3, FOV4 represents the field of view of C4, FOV1 intersects with FOV2, FOV2 intersects with FOV3, FOV3 intersects with FOV4, FOV4 intersects with FOV1, C1-C4 cooperate, C1-C4 respectively obtain images in one direction, so as to obtain images in 360 degrees around the vehicle. Among them, setting up 4 ultra-wide-angle cameras and a detection radar can reduce the cost of information collection equipment while meeting the needs, and also facilitate the maintenance of the equipment.

[0234] The distance estimation method includes the following schemes:

[0235] S210: Acquire a first environment image of the target area captured by each of the plurality of ultra-wide-angle cameras;

[0236] S220: Acquire a second environment image of a target area corresponding to each of the first environment images; wherein the second environment image is a sparse depth map;

[0237] S230: For each of the first environment images, input the first environment image and the second environment image corresponding to the first environment image into a depth map generation model to obtain a dense depth map of the target area corresponding to each of the first environment images; wherein the depth map generation model is a multimodal surround depth estimation model obtained by the above-mentioned model training method;

[0238] S240: Determine the distance between the vehicle and obstacles in the surrounding environment according to the dense depth maps of the target areas corresponding to each of the plurality of first environment images.

[0239] In an exemplary embodiment, the target area can be understood as the range area covered by the field of view of the ultra-wide-angle camera. A first environment image of the target area captured by each of the multiple ultra-wide-angle cameras is obtained, such as Figure 2 As shown, the first environment images captured by C1-C4 are I1, I2, I3 and I4 respectively.

[0240] Obtain second environment images of the target area corresponding to each of the multiple first environment images. For example, the multiple second environment images obtained are the second environment image D1 of the target area corresponding to I1, the second environment image D2 of the target area corresponding to I2, the second environment image D3 of the target area corresponding to I3, and the second environment image D4 of the target area corresponding to I4.

[0241] After obtaining a plurality of first environment images and a plurality of second environment images, the images are grouped according to the corresponding directions, and the first environment images and the second environment images corresponding to the same direction are divided into one image group, and the image groups corresponding to each direction are obtained. For example, I1 and D1 are divided into one image group, I2 and D2 are divided into one image group, I3 and D3 are divided into one image group, and I4 and D4 are divided into one image group.

[0242] Each image group is then used as the input of the depth map generation model, so that all image groups are input into the depth map generation model. The depth map generation model outputs dense depth maps of the target areas corresponding to each direction, and then calculates the distance between the vehicle and obstacles in the surrounding environment based on the dense depth maps of the target areas corresponding to each direction. For example, in the field of intelligent driving, after the depth map generation model outputs dense depth maps of the target areas corresponding to each direction of the vehicle, the distance between the vehicle and obstacles in the surrounding environment is calculated, and then obstacle avoidance operations can be performed based on the calculated distance. Since the depth map generation model gathers multiple scale features during the generation of the dense depth map, it can perform high-precision depth estimation of targets of various shapes in the environment, thereby obtaining an accurate dense depth map, and the dense depth map can be used to accurately estimate the distance between the vehicle and obstacles in the surrounding environment.

[0243] In a possible implementation, the above S220 includes the following solutions:

[0244] Acquire second internal parameters of the multiple ultra-wide-angle cameras and second external calibration parameters of the detection radar;

[0245] Acquire regional point cloud data of target areas corresponding to each of the plurality of first environmental images; wherein the regional point cloud data of target areas corresponding to each of the plurality of first environmental images are collected by the detection radar;

[0246] According to the second internal parameter and the second external calibration parameter, the regional point cloud data of the target area corresponding to each of the plurality of first environmental images are projected to an image pixel coordinate system to obtain a second environmental image of the target area corresponding to each of the plurality of first environmental images.

[0247] The second internal parameters of the ultra-wide-angle camera are also called the internal parameters of the camera. The second external calibration parameters of the detection radar are pre-calibrated, specifically referring to the external calibration parameters of the image and radar, which are called the external parameters of the radar.

[0248] Acquire regional point cloud data of the target area corresponding to each of the multiple first environmental images, convert the regional point cloud data of the target area corresponding to each of the multiple first environmental images to the camera coordinate system according to the second external calibration parameter, obtain converted point cloud data of the regional point cloud data of the target area corresponding to each of the multiple first environmental images in the camera coordinate system, and then project the converted point cloud data corresponding to each of the multiple first environmental images to the image pixel coordinate system according to the second internal parameter, to obtain second environmental images of the target area corresponding to each of the multiple first environmental images.

[0249] refer to Figure 2 , according to the external parameters of the radar, the regional point cloud data of the target area corresponding to I1 is converted to the camera coordinate system to obtain the converted point cloud data corresponding to I1, and then the converted point cloud data corresponding to I1 is projected to the image pixel coordinate system according to the internal parameters of C1-C4 to obtain D1. According to the external parameters of the radar, the regional point cloud data of the target area corresponding to I2 is converted to the camera coordinate system to obtain the converted point cloud data corresponding to I2, and then the converted point cloud data corresponding to I2 is projected to the image pixel coordinate system according to the internal parameters of C1-C4 to obtain D2. According to the external parameters of the radar, the regional point cloud data of the target area corresponding to I3 is converted to the camera coordinate system to obtain the converted point cloud data corresponding to I3, and then the converted point cloud data corresponding to I3 is projected to the image pixel coordinate system according to the internal parameters of C1-C4 to obtain D1. According to the external parameters of the radar, the regional point cloud data of the target area corresponding to I4 is converted to the camera coordinate system to obtain the converted point cloud data corresponding to I4, and then the converted point cloud data corresponding to I4 is projected to the image pixel coordinate system according to the internal parameters of C1-C4 to obtain D4.

[0250] The specific generation process of the sparse depth map is the same as the specific generation process of the sparse depth map in the above-mentioned model training method embodiment, which will not be repeated here.

[0251] The following is an embodiment of the model training device of the present application, which can be used to execute the embodiment of the model training method of the present application. For details not disclosed in the embodiment of the model training device of the present application, please refer to the embodiment of the model training method of the present application.

[0252] Fig. 9 FIG. 1 shows a schematic diagram of the structure of a model training device provided in an embodiment of the present application. Fig. 9 As shown, the model training device 900 includes:

[0253] A first image acquisition module 910 is used to acquire a first sample image; wherein the first sample image includes a scene image of a target scene corresponding to each direction of the target object;

[0254] A second image acquisition module 920 is used to acquire a second sample image; wherein the second sample image includes a sparse depth map of the target scene corresponding to each direction of the target object;

[0255] A first feature extraction module 930 is used to extract a plurality of first image features of the first sample image; wherein the plurality of first image features have different feature scales;

[0256] A second feature extraction module 940 is used to extract a plurality of second image features of the second sample image; wherein the plurality of second image features have different feature scales;

[0257] A feature fusion module 950 is used to fuse each of the plurality of first image features with a second image feature corresponding to each of the plurality of first image features according to a feature scale to obtain a target fused feature;

[0258] A feature decoding module 960 is used to decode the target fusion feature according to the multiple first image features to obtain a target sample image; wherein the target sample image includes a dense depth map of the target scene corresponding to each direction of the target object;

[0259] The model training module 970 is used to train a multimodal surround depth estimation model according to the second sample image and the target sample image.

[0260] In a possible implementation, the second image acquisition module 920 includes:

[0261] A point cloud acquisition unit, used to acquire point cloud data of a target scene corresponding to each direction of the target object;

[0262] The point cloud projection unit is used to project the point cloud data of the target scene corresponding to each direction to the image pixel coordinate system to obtain a sparse depth map of the target scene corresponding to each direction of the target object.

[0263] In a possible implementation, a radar and a plurality of cameras are provided on the target object, scene images of target scenes corresponding to respective directions of the target object are acquired by cameras at corresponding positions of respective directions of the target object, point cloud data of target scenes corresponding to respective directions of the target object are acquired by the radar, and the point cloud projection unit includes:

[0264] a parameter acquisition subunit, configured to acquire a first internal parameter of the camera and a first external calibration parameter of the radar;

[0265] The point cloud projection subunit is used to project the point cloud data of the target scene corresponding to each direction to the image pixel coordinate system according to the first internal parameter and the first external calibration parameter, so as to obtain a sparse depth map of the target scene corresponding to each direction of the target object.

[0266] In a possible implementation, the multimodal surround depth estimation model includes a deformable convolutional neural network and a sparse convolutional neural network;

[0267] The first feature extraction module 930 is specifically configured to use the deformable convolutional neural network to extract image features of the first sample image at each of a plurality of preset feature scales to obtain the plurality of first image features; wherein the plurality of preset feature scales are of different sizes;

[0268] The second feature extraction module 940 is specifically configured to use the sparse convolutional neural network to extract the image features of the second sample image at each of the plurality of preset feature scales to obtain the plurality of second image features.

[0269] In a possible implementation, the multimodal surround depth estimation model further includes a plurality of fusion modules, the plurality of fusion modules correspond to different feature scales, and the feature fusion module 950 includes:

[0270] A module selection unit, configured to determine, from the plurality of fusion modules, target fusion modules corresponding to the plurality of first image features respectively according to feature scales;

[0271] a feature fusion unit, configured to fuse each of the plurality of first image features with a second image feature corresponding to each of the plurality of first image features using a target fusion module corresponding to each of the plurality of first image features, to obtain a fusion result; wherein the fusion result includes a fused image feature corresponding to each of the plurality of first image features and a fused depth feature corresponding to each of the plurality of first image features;

[0272] A feature generating unit is used to generate the target fusion feature according to the fusion result.

[0273] In a possible implementation, the target fusion module includes a first module group and a second module group, the first module group includes a plurality of first processing modules connected in series, each of the first processing modules includes a first submodule and a second submodule connected to an output end of the first submodule, the second module group includes a plurality of second processing modules connected in series, each of the second processing modules includes a third submodule and a fourth submodule connected to an output end of the third submodule, the first submodule and the third submodule are both cross attention modules, the second submodule and the fourth submodule are both self attention modules, and the feature fusion unit includes:

[0274] A feature extraction subunit, configured to obtain first linear features corresponding to each of the plurality of first image features, and to obtain second linear features corresponding to each of the plurality of second image features; wherein the first linear features and the second linear features both include Q features, K features, and V features;

[0275] The feature fusion subunit is used to cross-input the first linear feature and the second linear feature into the first module group and the first module group to obtain the fused image features corresponding to each of the multiple first image features and the fused depth features corresponding to each of the multiple second image features.

[0276] In a possible implementation, the multimodal surround depth estimation model further includes a decoder;

[0277] The feature decoding module 960 is specifically configured to input the plurality of first image features and the target fusion feature into the decoder, so as to decode the target fusion feature through the decoder to obtain the target sample image.

[0278] In a possible implementation, the model training module 970 includes:

[0279] A difference calculation unit, used to determine the difference between the dense depth map and the sparse depth map of the target scene corresponding to each direction of the target object, to obtain a plurality of differences;

[0280] A function determination unit, configured to determine a loss function according to an average difference value of the plurality of differences;

[0281] A type training unit is used to train the multimodal surround depth estimation model according to the loss function.

[0282] It should be noted that the model training device provided in the above embodiment only uses the division of the above functional modules as an example when executing the model training method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the model training device provided in the above embodiment belongs to the same concept as the model training method embodiment. Therefore, for details not disclosed in the device embodiment of the present application, please refer to the above-mentioned model training method embodiment of the present application, which will not be repeated here.

[0283] The following is an embodiment of the distance estimation device of the present application, which can be used to execute the distance estimation method embodiment of the present application. For details not disclosed in the distance estimation device embodiment of the present application, please refer to the distance estimation method embodiment of the present application.

[0284] In a fifth aspect, a distance estimation device is provided, which is configured in a controller of a vehicle, wherein the vehicle is provided with a plurality of ultra-wide-angle cameras, wherein the plurality of ultra-wide-angle cameras are dispersedly arranged around a body of the vehicle, and the field of view angles of two adjacent ultra-wide-angle cameras intersect;

[0285] The distance estimation device comprises:

[0286] A third image acquisition module, used to acquire a first environment image of the target area captured by each of the multiple ultra-wide-angle cameras;

[0287] A fourth image acquisition module, used to acquire a second environment image of a target area corresponding to each of the plurality of first environment images; wherein the second environment image is a sparse depth map;

[0288] A depth map generation module, configured to input the first environment image and a second environment image corresponding to the first environment image into a depth map generation model for each of the first environment images, so as to obtain a dense depth map of the target area corresponding to each of the first environment images; wherein the depth map generation model is a multimodal surround depth estimation model obtained by the above-mentioned distance estimation method;

[0289] The distance determination module is used to determine the distance between the vehicle and obstacles in the surrounding environment according to the dense depth map of the target area corresponding to each of the plurality of first environment images.

[0290] In a possible implementation, the vehicle is further provided with a detection radar, and the detection radar is used to obtain scene point cloud data around the vehicle, and the fourth image acquisition module includes:

[0291] A device parameter acquisition unit, used to acquire second internal parameters of the plurality of ultra-wide-angle cameras and second external calibration parameters of the detection radar;

[0292] A regional data acquisition unit, used to acquire regional point cloud data of target areas corresponding to each of the plurality of first environmental images; wherein the regional point cloud data of target areas corresponding to each of the plurality of first environmental images are collected by the detection radar;

[0293] The regional data conversion unit is used to project the regional point cloud data of the target area corresponding to each of the multiple first environmental images to the image pixel coordinate system according to the second internal parameter and the second external calibration parameter, so as to obtain the second environmental image of the target area corresponding to each of the multiple first environmental images.

[0294] It should be noted that when the distance estimation device provided in the above embodiment executes the distance estimation method, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the distance estimation device provided in the above embodiment belongs to the same concept as the distance estimation method embodiment. Therefore, for details not disclosed in the device embodiment of the present application, please refer to the above-mentioned distance estimation method embodiment of the present application, which will not be repeated here.

[0295] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0296] Fig.10 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown.

[0297] For example, Fig.10 As shown, the electronic device 1000 includes: a memory 1001 and a processor 1002, wherein the memory 1001 stores an executable program code 10011, and the processor 1002 is used to call and execute the executable program code 10011 to perform a model training method or a distance estimation method.

[0298] In this embodiment, the electronic device can be divided into functional modules according to the above method example. For example, each functional module can be corresponded to, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0299] In the case of dividing each functional module according to each function, the electronic device may include: a first image acquisition module, a second image acquisition module, a first feature extraction module, a second feature extraction module, a feature fusion module, a feature decoding module, a model training module, etc., and may also include a third image acquisition module, a fourth image acquisition module, a depth map generation module, a distance determination module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0300] The electronic device provided in this embodiment is used to execute one of the above-mentioned model training methods or distance estimation methods, and thus can achieve the same effect as the above-mentioned implementation method.

[0301] In the case of an integrated unit, the electronic device may include a processing module and a storage module. The processing module may be used to control and manage the actions of the electronic device. The storage module may be used to support the electronic device in executing mutual program codes and data.

[0302] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits disclosed in the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0303] This embodiment also provides a computer-readable storage medium, which stores computer program code. When the computer program code runs on a computer, the computer executes the above-mentioned related method steps to implement a model training method or distance estimation method in the above-mentioned embodiment.

[0304] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement a model training method or a distance estimation method in the above-mentioned embodiment.

[0305] In addition, the electronic device provided in the embodiments of the present application may specifically be a chip, a component or a module, and the electronic device may include a connected processor and a memory; wherein the memory is used to store instructions, and when the electronic device is running, the processor may call and execute instructions so that the chip executes a model training method or a distance estimation method in the above-mentioned embodiments.

[0306] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0307] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0308] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0309] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method, It is characterized in that The model training method comprises: Acquire a first sample image; wherein the first sample image includes a scene image of a target scene corresponding to each direction of the target object; Acquire a second sample image; wherein the second sample image includes a sparse depth map of the target scene corresponding to each direction of the target object; Extracting a plurality of first image features of the first sample image; wherein the plurality of first image features have different feature scales; Extracting a plurality of second image features of the second sample image; wherein the plurality of second image features have different feature scales; According to the feature scale, each of the plurality of first image features is fused with a second image feature corresponding to each of the plurality of first image features to obtain a target fused feature; Decoding the target fusion feature according to the multiple first image features to obtain a target sample image; wherein the target sample image includes a dense depth map of the target scene corresponding to each direction of the target object; A multimodal surround depth estimation model is trained according to the second sample image and the target sample image.

2. The model training method according to claim 1, It is characterized in that The step of acquiring the second sample image comprises: Acquire point cloud data of the target scene corresponding to each direction of the target object; The point cloud data of the target scene corresponding to each of the directions are projected onto an image pixel coordinate system to obtain a sparse depth map of the target scene corresponding to each of the directions of the target object.

3. The model training method according to claim 2, It is characterized in that The target object is provided with a radar and a plurality of cameras, the scene images of the target scenes corresponding to each direction of the target object are acquired by the cameras at the corresponding positions of each direction of the target object, and the point cloud data of the target scenes corresponding to each direction of the target object are acquired by the radar; The step of projecting the point cloud data of the target scene corresponding to each direction to the image pixel coordinate system to obtain a sparse depth map of the target scene corresponding to each direction of the target object comprises: Acquire a first internal parameter of the camera and a first external calibration parameter of the radar; According to the first internal parameter and the first external calibration parameter, the point cloud data of the target scene corresponding to each direction is projected to the image pixel coordinate system to obtain a sparse depth map of the target scene corresponding to each direction of the target object.

4. The model training method according to claim 1, It is characterized in that The multimodal surround depth estimation model includes a deformable convolutional neural network and a sparse convolutional neural network; The step of extracting a plurality of first image features of the first sample image comprises: The deformable convolutional neural network is used to extract image features of the first sample image at each of a plurality of preset feature scales to obtain the plurality of first image features; wherein the plurality of preset feature scales are of different sizes; The step of extracting a plurality of second image features of the second sample image comprises: The sparse convolutional neural network is used to extract image features of the second sample image at each of the multiple preset feature scales to obtain the multiple second image features.

5. The model training method according to claim 1, It is characterized in that The multimodal surround depth estimation model further includes a plurality of fusion modules, and the plurality of fusion modules correspond to different feature scales; The step of fusing each of the plurality of first image features with each of the second image features corresponding to the plurality of first image features according to the feature scale to obtain a target fused feature comprises: Determining, from the plurality of fusion modules, target fusion modules corresponding to the plurality of first image features respectively according to the feature scale; Using target fusion modules corresponding to the plurality of first image features, each of the plurality of first image features is fused with a second image feature corresponding to the plurality of first image features to obtain a fusion result; wherein the fusion result includes fused image features corresponding to the plurality of first image features and fused depth features corresponding to the plurality of first image features; The target fusion feature is generated according to the fusion result.

6. The model training method according to claim 5, It is characterized in that The target fusion module includes a first module group and a second module group, the first module group includes a plurality of first processing modules connected in series, each of the first processing modules includes a first submodule and a second submodule connected to the output end of the first submodule, the second module group includes a plurality of second processing modules connected in series, each of the second processing modules includes a third submodule and a fourth submodule connected to the output end of the third submodule, the first submodule and the third submodule are both cross-attention modules, and the second submodule and the fourth submodule are both self-attention modules; The step of using the target fusion modules corresponding to the plurality of first image features to fuse the plurality of first image features with the second image features corresponding to the plurality of first image features to obtain the fusion result comprises: Acquire a first linear feature corresponding to each of the plurality of first image features, and acquire a second linear feature corresponding to each of the plurality of second image features; wherein the first linear feature and the second linear feature both include a Q feature, a K feature, and a V feature; The first linear feature and the second linear feature are cross-input into the first module group and the first module group to obtain fused image features corresponding to each of the plurality of first image features and fused depth features corresponding to each of the plurality of second image features.

7. The model training method according to claim 1, It is characterized in that The multimodal surround depth estimation model also includes a decoder; The step of decoding the target fusion feature according to the multiple first image features to obtain a target sample image comprises: The multiple first image features and the target fusion features are input into the decoder, so that the target fusion features are decoded by the decoder to obtain the target sample image.

8. The model training method according to claim 1, It is characterized in that The step of training a multimodal surround depth estimation model according to the second sample image and the target sample image comprises: Determine the difference between the dense depth map and the sparse depth map of the target scene corresponding to each direction of the target object, and obtain multiple differences; Determine a loss function based on an average difference of the plurality of differences; The multimodal surround depth estimation model is trained according to the loss function.

9. A multimodal surround depth estimation model, It is characterized in that The multimodal surround depth estimation model includes: A feature processing module is used to extract multiple image features of each of the first image and the second image to obtain a feature set; wherein the first image includes: a scene image of a target scene corresponding to at least one direction of the target object, and the second image includes: a sparse depth map of the target scene corresponding to at least one direction of the target object; the feature set includes: a plurality of first image features corresponding to the first image and a plurality of second image features corresponding to the second image, wherein the plurality of first image features have different feature scales, and the plurality of first image features have different feature scales; The feature processing module is further used to fuse each of the plurality of first image features with a second image feature corresponding to each of the plurality of first image features according to a feature scale to obtain a target fused feature; A decoder is used to decode the target fusion feature according to the multiple first image features to obtain a target image; wherein the target image includes: a dense depth map of the target scene corresponding to at least one direction of the target object.

10. The multimodal surround depth estimation model according to claim 9, It is characterized in that The feature processing module comprises: A deformable convolutional neural network, wherein the deformable convolutional neural network includes M first feature extraction layers, the convolution operation adopted by the M first feature extraction layers is a deformable convolution operation, the preset feature scales corresponding to the first first feature extraction layer to the Mth first feature extraction layer are gradually reduced, and M is an odd number greater than or equal to 5; A sparse convolutional neural network, wherein the sparse convolutional neural network comprises M second feature extraction layers, the convolution operation adopted by the M second feature extraction layers is a sparse convolution operation, and the preset feature scales corresponding to the first second feature extraction layer to the Mth second feature extraction layer are gradually reduced; M - 1 fusion module; A feature series connection layer, wherein the feature series connection layer is connected to the decoder; in: The output ends of the i-th first feature extraction layer and the i-th second feature extraction layer are connected to the input end of the i-th fusion module, and the output end of the i-th fusion module is connected to the input ends of the i+1-th first feature extraction layer and the i+1-th second feature extraction layer; The output ends of the Mth first feature extraction layer and the Mth second feature extraction layer are connected to the feature series connection layer; The output ends of the first feature extraction layer from the first feature extraction layer 1 to the first feature extraction layer M-2 are connected to the decoder, i=1, 2, 3, ..., M-1; The i-th first feature extraction layer is used to extract the first image feature of the first image according to a preset feature scale corresponding to the i-th first feature extraction layer; The i-th second feature extraction layer is used to extract the second image features of the second image according to the preset feature scale corresponding to the i-th second feature extraction layer, and the preset feature scale corresponding to the i-th first feature extraction layer is the same as the preset feature scale corresponding to the i-th second feature extraction layer; The i-th fusion module is used to fuse the first image features extracted by the i-th first feature extraction layer and the second image features extracted by the i-th second feature extraction layer.

11. The multimodal surround depth estimation model according to claim 10, It is characterized in that Each fusion module includes a first module group and a second module group, the first module group includes a plurality of first processing modules connected in series, each first processing module includes a first submodule and a second submodule connected to the output end of the first submodule, the second module group includes a plurality of second processing modules connected in series, each second processing module includes a third submodule and a fourth submodule connected to the output end of the third submodule, the first submodule and the third submodule are both cross-attention modules, and the second submodule and the fourth submodule are both self-attention modules.

12. The multimodal surround depth estimation model according to claim 10, It is characterized in that The decoder comprises M-1 sequentially connected feature decoding layers, and the feature scales corresponding to the first feature decoding layer to the M-1th feature decoding layer increase step by step; Among them, the first feature decoding layer is connected to the output end of the feature series layer, the input of the jth feature decoding layer is connected to the output end of the kth first feature extraction layer, and the output end of the M-1th feature decoding layer is the output end of the multimodal surround depth estimation model, j=M-1,M-2,...,2, k=1,2,...,M-2.

13. A distance estimation method, It is characterized in that A controller applied to a vehicle, wherein the vehicle is provided with a plurality of ultra-wide-angle cameras, wherein the plurality of ultra-wide-angle cameras are dispersedly arranged around a body of the vehicle, and the field of view angles of two adjacent ultra-wide-angle cameras intersect; The distance estimation method comprises: Acquire a first environment image of the target area captured by each of the plurality of ultra-wide-angle cameras; Acquire a second environment image of a target area corresponding to each of the first environment images; wherein the second environment image is a sparse depth map; For each of the first environment images, the first environment image and the second environment image corresponding to the first environment image are input into a depth map generation model to obtain a dense depth map of the target area corresponding to each of the first environment images; wherein the depth map generation model is a multimodal surround depth estimation model obtained by the model training method according to any one of claims 1 to 8; The distance between the vehicle and obstacles in the surrounding environment is determined according to the dense depth maps of the target areas corresponding to each of the plurality of first environment images.

14. The distance estimation method according to claim 13, It is characterized in that The vehicle is also provided with a detection radar, which is used to obtain scene point cloud data around the vehicle; The step of acquiring a second environment image of a target area corresponding to each of the first environment images comprises: Acquire second internal parameters of the multiple ultra-wide-angle cameras and second external calibration parameters of the detection radar; Acquire regional point cloud data of target areas corresponding to each of the plurality of first environmental images; wherein the regional point cloud data of target areas corresponding to each of the plurality of first environmental images are collected by the detection radar; According to the second internal parameter and the second external calibration parameter, the regional point cloud data of the target area corresponding to each of the plurality of first environmental images are projected to an image pixel coordinate system to obtain a second environmental image of the target area corresponding to each of the plurality of first environmental images.

15. A model training device, It is characterized in that The model training device comprises: A first image acquisition module, used to acquire a first sample image; wherein the first sample image includes a scene image of a target scene corresponding to each direction of the target object; A second image acquisition module, used to acquire a second sample image; wherein the second sample image includes a sparse depth map of the target scene corresponding to each direction of the target object; A first feature extraction module, used to extract a plurality of first image features of the first sample image; wherein the plurality of first image features have different feature scales; A second feature extraction module, used to extract a plurality of second image features of the second sample image; wherein the plurality of second image features have different feature scales; A feature fusion module, used for fusing each of the plurality of first image features with a second image feature corresponding to each of the plurality of first image features according to a feature scale, to obtain a target fusion feature; A feature decoding module, configured to decode the target fusion feature according to the plurality of first image features to obtain a target sample image; wherein the target sample image includes a dense depth map of the target scene corresponding to each direction of the target object; A model training module is used to train a multimodal surround depth estimation model based on the second sample image and the target sample image.

16. An electronic device, It is characterized in that The electronic device comprises: A memory for storing executable program codes; A processor, used to call and run the executable program code from the memory, so that the electronic device executes the model training method as described in any one of claims 1 to 8 or executes the distance estimation method as described in claim 13 or 14.

17. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed, it implements the model training method as described in any one of claims 1 to 8 or executes the distance estimation method as described in claim 13 or 14.

Images