Method for acquiring depth information based on surround view image, electronic device and storage medium

Abstract

The disclosure provides a depth information acquisition method based on a surround view image, an electronic device and a storage medium. The depth information acquisition method based on the surround view image comprises: acquiring a surround view image; performing monocular depth estimation on the surround view image based on a monocular depth estimation model to obtain a first depth loss; performing surround depth estimation on the surround view image based on a surround depth estimation model to obtain a depth consistency loss; acquiring a total loss according to the first depth loss and the depth consistency loss; adjusting model parameters of the surround depth estimation model based on the total loss to obtain an adjusted surround depth estimation model; and acquiring depth information of the surround view image based on the adjusted surround depth estimation model.

Description

Technical Field

[0001] This disclosure relates to the fields of autonomous driving and computer vision, and in particular to a method, apparatus, electronic device, storage medium, and computer program product for acquiring depth information based on surround view images. Background Technology

[0002] Depth estimation is a fundamental and challenging task in perception for autonomous driving and robotic systems. Due to the high cost of depth sensors, research has emerged using convolutional neural networks (CNNs) to infer depth maps from images. Utilizing annotated depth data, we can train a regression CNN to predict the depth value for each pixel in a single image.

[0003] However, due to the lack of large-scale, accurate, and dense ground truth depth data for supervised learning, self-supervised methods seeking supervision signals from stereo frame pairs or monocular videos have emerged. These methods treat depth estimation as a novel view synthesis problem and minimize image reconstruction loss. Because collecting stereo data requires complex configuration and data processing, most methods focus on self-supervised approaches that only require monocular video data.

[0004] The relevant technical solutions will be introduced as follows:

[0005] Technical Solution 1: The paper "Digging into self-supervised monocular depthestimation" made the following improvements: 1) It proposed a minimum reprojection loss, changing the averaging of reprojection errors from multiple source images to minimizing them. This effectively solves the occlusion problem of occluded pixels that only appear in some source images; 2) It proposed an automatic masking fixed pixel method, masking pixels that have not changed between consecutive frames to avoid them from participating in the loss calculation. This mask allows the network to ignore objects in the scene moving at the same speed as the vehicle, and to ignore the entire frame when the vehicle stops moving, avoiding these scenes from polluting the photometric reprojection loss. The minimum reprojection loss proposed by this method simply and effectively solves the object occlusion problem and has been widely cited in various subsequent methods. However, it lacks processing for dynamic objects in the scene, only ignoring objects that are relatively stationary compared to the vehicle's movement through masking.

[0006] Technical Solution 2: The paper "Self-supervised Object Motion and Depth Estimation from Video" mainly addresses the processing of dynamic objects. The method involves the following steps: 1) Introducing an instance segmentation network to divide the pixels of a 2D image into static background pixels and dynamic object pixels; 2) Introducing a new object motion estimation network to predict the motion of each dynamic pixel instance; 3) When synthesizing the target image, background pixels are treated as static regions, and object pixels as dynamic regions, and synthesis is performed using camera pose and predicted pose respectively. This method effectively models the motion of rigid objects in a scene through an object pose estimation network, significantly improving the depth prediction of dynamic regions in self-supervised methods. However, this method is based on monocular video sequences and struggles to handle objects moving between multiple cameras on a vehicle. Furthermore, the original paper does not address the occlusion problem caused by moving objects.

[0007] Technical Solution 3: The paper "BEVScope: Enhancing Self-Supervised Depth Estimation Leveraging Bird's-Eye-View in Dynamic Scenarios" extends depth estimation from a monocular to a surround-view setting and advocates utilizing geometric cues inherent in BEV features to enhance image depth estimation. The core idea is to fuse surround-view image features with BEV features, then use self-attention to allow image features to learn scene geometric cues from the BEV features, and finally decode the predicted depth map from the image features. This paper is the first to introduce BEV features into depth estimation; however, the experimental results show limited improvement compared to previous methods, and the proposed fusion method does not effectively enable image features to learn geometric cues in BEV scenes.

[0008] Technical Solution 4: The paper "Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints" primarily contributes by incorporating 3D point cloud information into depth estimation, explicitly considering the constraints of the 3D geometry of the entire scene on depth estimation. After obtaining 3D point clouds from the estimated depth, the ICP algorithm is used to perform optimal matching between two 3D point clouds, and then the 3D point cloud consistency loss is calculated. The introduction of 3D geometric structure information effectively improves the accuracy of depth estimation; however, the ICP algorithm used in point cloud matching is computationally intensive and difficult to extend from monocular to surround-view depth estimation.

[0009] In light of relevant technologies, current self-supervised depth estimation methods based on photometric reprojection loss still have the following shortcomings:

[0010] (1) Cross-utilization of information from multiple cameras: Most methods (technical solutions 1, 2, and 4) still rely on monocular depth estimation, while most recent surround view methods obtain information between multiple viewpoints through attention interaction between surround view feature maps. There are also methods that interact with 2D image features and BEV features through attention (technical solution 3). These methods all utilize attention mechanisms and have a large computational load.

[0011] (2) Utilization of geometric structure information in the scene: Technical solution 4 is the first to introduce the consistency of 3D structure prediction into depth estimation and has achieved good results. However, the method used is computationally intensive and it is difficult to extend from monocular to surround view. Technical solution 3 hopes to learn scene geometric information from BEV features through a self-attention mechanism, but the experimental results are not good.

[0012] (3) Handling of dynamic objects: Many methods for handling dynamic objects have emerged in the monocular setting, such as technical solution 2 which models the motion of dynamic instances through a pose network. However, in the surround view setting, the handling of dynamic objects is still a blank. If the monocular method is directly extended to surround view multi-view, it will result in high computational cost and difficulty in handling objects moving between multiple camera views. Summary of the Invention

[0013] This disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for acquiring depth information based on a panoramic image.

[0014] According to one aspect of this disclosure, a method for obtaining depth information based on a surround view image is provided, comprising:

[0015] Obtain the panoramic image;

[0016] Based on the monocular depth estimation model, monocular depth estimation is performed on the panoramic image to obtain the first depth loss;

[0017] The surround view depth is estimated based on the surround view depth estimation model to obtain the depth consistency loss.

[0018] The total loss is obtained based on the first depth loss and the depth consistency loss;

[0019] Based on the total loss, the model parameters of the surround view depth estimation model are adjusted to obtain the adjusted surround view depth estimation model.

[0020] The depth information of the surround view image is obtained based on the adjusted surround view depth estimation model.

[0021] According to at least one embodiment of the depth information acquisition method based on a surround view image of the present disclosure, acquiring a surround view image includes:

[0022] Obtain monocular image sequences;

[0023] The set of monocular images at the same frame time in each monocular image sequence is used as a circumferential image to obtain a circumferential image sequence.

[0024] According to at least one embodiment of the method for acquiring depth information based on a surround-view image, a monocular depth estimation is performed on the surround-view image based on a monocular depth estimation model to obtain a first depth loss, including:

[0025] Based on monocular vision, the depth map of the monocular target image in the panoramic target image is obtained, and the relative pose between the monocular target image and the monocular source image is obtained.

[0026] Based on the depth map and relative pose corresponding to the monocular target image, the monocular target image is reconstructed from pixels sampled from the monocular source image to obtain a monocular synthetic image;

[0027] The first depth loss is obtained based on the error between the monocular target image and the corresponding monocular synthesized image.

[0028] According to at least one embodiment of the depth information acquisition method based on a panoramic image of the present disclosure, the monocular target image is reconstructed from sampled pixels of the monocular source image based on the depth map corresponding to the monocular target image and the relative pose, to obtain a monocular synthesized image, including:

[0029] Obtain the camera intrinsic parameter matrix and the homogeneous pixel coordinates of the monocular target image;

[0030] Pixel projection coordinates are obtained based on the depth map, the relative pose, the camera intrinsic parameter matrix, and the pixel homogeneous coordinates. The pixel projection coordinates represent the projection of the pixel homogeneous coordinates onto the monocular source image.

[0031] Bilinear interpolation is performed based on the pixel homogeneous coordinates and the pixel projected coordinates to obtain the sampled pixel values;

[0032] The monocular target image is reconstructed based on the sampled pixel values ​​to obtain the monocular synthesized image.

[0033] According to at least one embodiment of the depth information acquisition method based on a panoramic image of the present disclosure, the error between the monocular target image and the monocular synthetic image corresponding to the monocular target image is measured by photometric reprojection loss.

[0034] According to at least one embodiment of the depth information acquisition method based on a panoramic image of the present disclosure, the first depth loss is obtained based on the error between a monocular target image and a corresponding monocular synthesized image, including:

[0035] The monocular target image is reconstructed based on different monocular source images to obtain different monocular synthetic images;

[0036] Based on the error between the monocular target image and different monocular synthesized images, the photometric reprojection loss is calculated to obtain a set of photometric reprojection losses.

[0037] The minimum photometric reprojection loss in the set of photometric reprojection losses is obtained as the first depth loss.

[0038] According to at least one embodiment of the depth information acquisition method based on a surround view image of this disclosure, the surround view depth of the surround view image is estimated based on a surround view depth estimation model to obtain a depth consistency loss, including:

[0039] Bird's-eye view features are obtained based on the panoramic image;

[0040] The bird's-eye view features are used to identify moving objects, and motion estimation is performed on the moving objects to obtain their motion information.

[0041] Based on the object motion information, the vehicle motion information, and the bird's-eye view perspective features, feature reconstruction is performed to obtain synthetic bird's-eye view features;

[0042] The depth consistency loss is obtained based on the similarity between the bird's-eye view features and the synthetic bird's-eye view features.

[0043] According to at least one embodiment of the depth information acquisition method based on a panoramic view image of this disclosure, feature recognition is performed on the perspective features of the bird's-eye view to obtain moving objects, motion estimation is performed on the moving objects, and object motion information is obtained, including:

[0044] The target moving object is obtained based on the target bird's-eye view perspective features of the panoramic target image, and the source moving object is obtained based on the source bird's-eye view perspective features of the panoramic source image.

[0045] Match the target moving object acquired from the surround view target image with the source moving object acquired from the surround view source image to obtain the same moving object;

[0046] The motion information of the object is obtained based on the same moving object.

[0047] According to at least one embodiment of the depth information acquisition method based on a surround view image of this disclosure, feature reconstruction is performed based on the object motion information, the vehicle motion information, and the bird's-eye view perspective features to obtain synthetic bird's-eye view features, including:

[0048] The source BEV features and the target BEV features are aligned based on object motion information and vehicle operation information.

[0049] Pixel sampling is performed on the source BEV features after coordinate alignment, and feature reconstruction is performed on the target BEV features after coordinate alignment to obtain the synthetic bird's-eye view features.

[0050] According to at least one embodiment of the depth information acquisition method based on panoramic images of this disclosure, the depth consistency loss is obtained based on the similarity between the bird's-eye view features and the synthetic bird's-eye view features, including:

[0051] Reconstruct the surrounding target image based on different surrounding source images to obtain different synthetic bird's-eye view features;

[0052] Based on the errors between the bird's-eye view features of the panoramic target image and different synthetic bird's-eye view features, calculate the BEV feature consistency loss respectively to obtain the BEV feature consistency loss set.

[0053] The minimum BEV feature consistency loss in the BEV feature consistency loss set is taken as the deep consistency loss.

[0054] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform a depth information acquisition method based on a surround view image according to any embodiment of this disclosure.

[0055] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the depth information acquisition method based on a surround view image according to any embodiment of this disclosure.

[0056] According to another aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the depth information acquisition method based on a surround view image according to any embodiment of this disclosure. Attached Figure Description

[0057] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0058] Figure 1 This is a flowchart illustrating a method for acquiring depth information based on a panoramic image, according to one embodiment of this disclosure.

[0059] Figure 2 This is a schematic flowchart illustrating the process of obtaining a panoramic image according to one embodiment of this disclosure.

[0060] Figure 3 This is a schematic diagram of a process for obtaining a first depth loss by performing monocular depth estimation on a panoramic image based on a monocular depth estimation model, according to one embodiment of this disclosure.

[0061] Figure 4 This is a schematic diagram of the process for acquiring a monocular synthesized image according to one embodiment of the present disclosure.

[0062] Figure 5 This is a schematic diagram of a process for obtaining a first depth loss based on the error between a monocular target image and a corresponding monocular synthetic image, according to one embodiment of this disclosure.

[0063] Figure 6 This is a schematic diagram of a process for obtaining depth consistency loss by performing surround view depth estimation on a surround view image based on a surround view depth estimation model according to one embodiment of the present disclosure.

[0064] Figure 7 This is a schematic diagram of a process for acquiring a moving object and acquiring object motion information according to one embodiment of this disclosure.

[0065] Figure 8 This is a flowchart of one embodiment of the present disclosure for obtaining features of a synthetic bird's-eye view.

[0066] Figure 9 This is a flowchart of obtaining a first depth loss according to one embodiment of this disclosure.

[0067] Figure 10 This is a schematic block diagram of a depth information acquisition device based on a surround view image, which employs a hardware implementation of a processing system according to one embodiment of the present disclosure. Detailed Implementation

[0068] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0069] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0070] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0071] The use of crosshairs and / or shading in the accompanying drawings is generally used to clarify the boundaries between adjacent components. Thus, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for the specific material, material properties, dimensions, proportions, commonalities between the illustrated components, or any other characteristics, properties, etc., of the components. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of components may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of their description. Furthermore, the same reference numerals denote the same components.

[0072] When a component is referred to as being "on" or "above" another component, "connected to" or "joined to" another component, the component may be directly on, directly connected to, or directly joined to the other component, or there may be intermediate components. However, when a component is referred to as being "directly on" another component, "directly connected to," or "directly joined to" another component, there are no intermediate components. Therefore, the term "connection" can refer to physical connection, electrical connection, etc., and may or may not have intermediate components.

[0073] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “(the)” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0074] The following text combines Figures 1 to 10 The present disclosure provides a detailed description of the method and apparatus for acquiring depth information based on panoramic images.

[0075] Figure 1 This is a flowchart illustrating a method for acquiring depth information based on a panoramic image, according to one embodiment of this disclosure.

[0076] refer to Figure 1 In some embodiments of this disclosure, the depth information acquisition method M100 based on surround view images includes:

[0077] S102. Obtain the panoramic image.

[0078] S104. Perform monocular depth estimation on the panoramic image based on the monocular depth estimation model to obtain the first depth loss.

[0079] S106. Based on the surround view depth estimation model, perform surround view depth estimation on the surround view image to obtain the depth consistency loss.

[0080] S108. Obtain the total loss based on the first depth loss and the depth consistency loss.

[0081] S110. Adjust the model parameters of the surround view depth estimation model based on the total loss to obtain the adjusted surround view depth estimation model.

[0082] S112. Obtain the depth information of the surround view image based on the adjusted surround view depth estimation model.

[0083] The surround view image (i.e. surround view image data) described in this disclosure can be composed of multiple bird's-eye-view (BEV) images taken by multiple vehicle-mounted cameras (monocular cameras). The bird's-eye-view images taken by the vehicle-mounted cameras can be frame images in video image sequence data. The field of view of these vehicle-mounted cameras can cover a 360° annular area around the vehicle.

[0084] Step S104: Perform monocular depth estimation on the panoramic image based on the monocular depth estimation model. The monocular depth estimation model of this disclosure can use existing monocular self-supervised depth estimation models, such as Monodepth2 model, FeatDepth model, LiteMono model, etc. Any selection or adjustment of the monocular depth estimation model by those skilled in the art under the guidance of the technical solution of this disclosure shall fall within the protection scope of this disclosure.

[0085] In this disclosure, the bird's-eye view images taken by different vehicle-mounted cameras in the surround view image can be regarded as independent monocular vision images. The monocular vision images are depth estimated and pose estimated based on a monocular depth estimation model (e.g., Monodepth2 model). The depth estimation and pose estimation can be used as intermediate tasks for view synthesis to obtain a monocular synthesized image.

[0086] Then, the minimum photometric reprojection loss (i.e., the first depth loss described above) is obtained by minimizing the error between the monocular synthesized image and the monocular visual image, and is used as supervision for the intermediate task mentioned above.

[0087] Step S106: The surround view image (i.e., surround view image data) is estimated based on a surround view depth estimation model. The surround view depth estimation model can use existing depth estimation-based BEV models, which can generate corresponding BEV features, such as the LSS model, BEVDet model, etc. Any selection or adjustment of the surround view depth estimation model by those skilled in the art based on the technical solutions of this disclosure falls within the protection scope of this disclosure. This disclosure can supervise the above depth estimation by constraining the dynamically aligned BEV features to obtain the depth consistency loss described above.

[0088] Step S108: Obtain the total loss based on the first depth loss and the depth consistency loss. This can extend monocular depth estimation to surround view depth estimation. It combines the first depth loss supervised by the view synthesis perspective and the depth consistency loss supervised by the BEV synthesis perspective to obtain the total loss.

[0089] Step S110: Adjust the model parameters of the surround view depth estimation model based on the total loss to obtain the adjusted surround view depth estimation model.

[0090] The total loss obtained from the above steps can be used to adjust and optimize the model parameters of the surround view depth estimation model, and can effectively use surround view images and geometric information to constrain the depth estimation performed by the surround view depth estimation model.

[0091] Furthermore, in step S112, the depth information of the surround view image can be obtained based on the parameter-adjusted surround view depth estimation model.

[0092] The depth information acquisition method based on surround view images disclosed herein performs self-supervision of depth estimation (i.e., depth information acquisition) performed by the surround view depth estimation model based on the total loss. It realizes supervision of the depth estimation task of the surround view depth estimation model by using the first depth loss (minimum photometric reprojection loss) and the depth consistency loss (BEV feature consistency loss), which can improve the accuracy of the surround view depth estimation model in acquiring depth information of surround view images.

[0093] This self-supervised method for depth estimation based on total loss can update and optimize the model parameters of the look-around depth estimation model to obtain optimized parameters. The optimized parameters correspond to minimizing the total loss, or to the total loss at a preset number of iterations.

[0094] The adjusted surround view depth estimation model disclosed herein, based on optimized parameters, estimates the depth of surround view images, thereby obtaining more accurate depth information.

[0095] Figure 2 This is a schematic flowchart illustrating the process of obtaining a panoramic image according to one embodiment of this disclosure.

[0096] refer to Figure 2 In some embodiments of this disclosure, step S102, obtaining the panoramic image, includes:

[0097] S1021. Obtain the monocular image sequence (i.e., the monocular visual image sequence). The monocular image sequence is a collection of images taken by monocular cameras in a surround-view camera array, ordered by time. The surround-view camera array includes multiple monocular cameras taking pictures from a bird's-eye view perspective.

[0098] S1022. Take the set of monocular images at the same frame time of each monocular image sequence as a panoramic image to obtain a panoramic image sequence.

[0099] In the process of acquiring the monocular image sequence, the monocular image at the current frame time is recorded as the monocular target image, and the monocular images at other times are recorded as the monocular source images.

[0100] Accordingly, the panoramic image at the current frame time is recorded as the panoramic target image, and the panoramic images at other times are recorded as the panoramic source images.

[0101] For example, suppose the monocular camera is C i , i∈(1,2,…,6), representing the number of 6 monocular cameras in the panoramic camera array. In monocular camera C... i In a sequence of monocular images, the image captured at the current time t (the current frame time) is denoted as the monocular target image. Images at other times are recorded as monocular source images.

[0102] In some embodiments of this disclosure, adjacent frame images of the monocular target image are retained as monocular source images. For example, the images acquired at time t-1 and time t+1 are used as monocular source images, which are then used in subsequent steps to focus on motion changes in adjacent frames.

[0103] Figure 3 This is a schematic diagram of a process for obtaining a first depth loss by performing monocular depth estimation on a panoramic image based on a monocular depth estimation model, according to one embodiment of this disclosure.

[0104] refer to Figure 3 In some embodiments of this disclosure, step S104, performing monocular depth estimation on the panoramic image based on a monocular depth estimation model to obtain a first depth loss, includes:

[0105] S1041. Obtain the depth map of the monocular target image in the surround view target image based on monocular vision, and obtain the relative pose between the monocular target image and the monocular source image.

[0106] S1042. Based on the depth map and relative pose corresponding to the monocular target image, reconstruct the monocular target image from the pixels sampled from the monocular source image to obtain a monocular synthetic image.

[0107] S1043. Based on the error between the monocular target image and the monocular synthesized image corresponding to the monocular target image, obtain the first depth loss.

[0108] In step S1041, acquiring the depth map of the monocular target image and the relative pose between the monocular target image and the corresponding monocular source image based on monocular vision means performing depth estimation and pose estimation based on the monocular viewpoint of each monocular camera in the surround-view camera array. For example, in the case of a surround-view camera array including 6 monocular cameras, the above depth estimation and pose estimation are performed based on the monocular viewpoints of the 6 different monocular cameras. The identifier i (i.e., camera identifier) ​​of different monocular image sequences is omitted in the formulas and terminology expressions below.

[0109] In step S1042, a monocular target image is reconstructed from pixels sampled from the monocular source image based on the depth map and relative pose, thus obtaining a monocular synthesized image. This involves using depth estimation and pose estimation from monocular vision as intermediate tasks for view synthesis, resulting in the monocular synthesized image described above. For example, assume the monocular target image is I. t The corresponding monocular source image is I. s Existing depth estimation networks can be used to predict monocular target images I. t Depth map and using existing pose estimation networks φ E Predicting monocular target image I t With monocular source image I s Relative pose between

[0110] Figure 4 This is a schematic diagram of the process for acquiring a monocular synthesized image according to one embodiment of the present disclosure.

[0111] refer to Figure 4 In some embodiments of this disclosure, step S1042, reconstructing the monocular target image from pixels sampled from the monocular source image based on the depth map corresponding to the monocular target image and the relative pose, to obtain a monocular synthesized image, includes:

[0112] S421. Obtain the camera intrinsic parameter matrix and the pixel homogeneous coordinates of the monocular target image.

[0113] S422. Obtain pixel projection coordinates based on the depth map, the relative pose, the camera intrinsic parameter matrix, and the pixel homogeneous coordinates. The pixel projection coordinates represent the projection of the pixel homogeneous coordinates onto the monocular source image.

[0114] S423. Perform bilinear interpolation calculation based on the pixel homogeneous coordinates and the pixel projection coordinates to obtain the sampled pixel value.

[0115] S424. Reconstruct the monocular target image based on the sampled pixel values ​​to obtain the monocular synthesized image.

[0116] For example, suppose p t p represents the homogeneous coordinates of pixels in a monocular target image, K represents the camera intrinsic parameter matrix, and p represents the homogeneous coordinates of pixels in a monocular target image. s p represents the pixel projection coordinates. s Characterizing p t Projection onto a monocular source image. This can be determined based on the depth map. Relative pose Camera intrinsic parameter matrix K and pixel homogeneous coordinates p t The pixel projection coordinates can be obtained using the following formula.

[0117] Formula 1:

[0118] The pixel projection coordinates p obtained from Formula 1 s It is a continuous value. Assume the monocular synthesized image is... Based on pixel homogeneous coordinates p t and pixel projection coordinates p s The method of bilinear interpolation is used to extract data from a monocular source image I. s Monocular synthetic image is obtained by reconstructing a monocular target image from medium-sampled pixels.

[0119] In some embodiments of this disclosure, for each homogeneous pixel in a monocular target image, its four nearest pixels are found in the monocular source image, and the pixel values ​​of these four pixels are obtained. Based on the weights of these four pixels and their distances to the homogeneous pixels, interpolation is performed to obtain sampled pixel values, which are then used as the pixel values ​​for reconstructing the homogeneous pixels. Similarly, for each homogeneous pixel in the monocular target image, sampled pixel values ​​are obtained from the monocular source image, and the homogeneous pixels of the monocular target image are reconstructed based on these sampled pixel values ​​to obtain the monocular synthetic image described above.

[0120] Preferably, in step S1043, the error between the monocular target image and the corresponding monocular synthetic image is measured by photometric reprojection loss.

[0121] Figure 5 This is a schematic diagram of a process for obtaining a first depth loss based on the error between a monocular target image and a corresponding monocular synthetic image, according to one embodiment of this disclosure.

[0122] refer to Figure 5 In some embodiments of this disclosure, step S1043, obtaining the first depth loss based on the error between the monocular target image and the corresponding monocular synthesized image, includes:

[0123] S431. Reconstruct the monocular target image based on different monocular source images to obtain different monocular synthetic images.

[0124] S432. Calculate the photometric reprojection loss based on the error between the monocular target image and different monocular synthesized images to obtain a set of photometric reprojection losses.

[0125] Different monocular source images can be combined with a monocular target image to form a series of multiple frames. For example, different monocular source images may include the image from the previous frame and the image from the next frame of the monocular target image. Alternatively, different monocular source images may include the image from the previous frame and the image from the two frames preceding the previous frame of the monocular target image. Any selection or adjustment of different monocular source images by those skilled in the art based on the teachings of this disclosure falls within the protection scope of this disclosure.

[0126] S433. Obtain the minimum photometric reprojection loss in the set of photometric reprojection losses as the first depth loss.

[0127] The error between the monocular target image and the corresponding monocular synthesized image is characterized by the photometric reprojection loss. In a preferred embodiment of this disclosure, the photometric reprojection loss is obtained based on the structural similarity and absolute value loss term between the monocular target image and the corresponding monocular synthesized image.

[0128] In some embodiments of this disclosure, the absolute value loss term uses the L1 loss term, assuming the photometric reprojection loss is... Used to measure monocular target image I t Combined with monocular synthetic images Error between them, photometric reprojection loss It can be obtained through the following formula two.

[0129] Formula 2:

[0130] in, Characterizing monocular target image I t Combined with monocular synthetic images Structural similarity between them Let α be the L1 loss term, and let α be a constant. Based on empirical values, α can be taken as 0.85.

[0131] For example, for input of multi-frame image data (monocular image sequence), such as when the monocular source images include images at time t-1 and time t+1, different monocular synthetic images are obtained based on different monocular source images. Photometric reprojection loss is obtained based on the error between the monocular target image and the different monocular synthetic images. The minimum photometric reprojection loss is taken as the first depth loss of the monocular target image. Based on this first depth loss, the pixel occlusion problem that only appears in some monocular source images of consecutive multi-frame images can be solved.

[0132] Assume the first depth loss is L p , Where s∈{t-1,t+1}. t→sI represents the reconstruction of a monocular target image by sampling pixels from either the monocular source image at time t-1 or the monocular source image at time t+1. t The obtained monocular synthetic image.

[0133] Figure 6 This is a schematic diagram of a process for obtaining depth consistency loss by performing surround view depth estimation on a surround view image based on a surround view depth estimation model according to one embodiment of the present disclosure.

[0134] refer to Figure 6 In some embodiments of this disclosure, step S106, estimating the surround view depth of the surround view image based on the surround view depth estimation model and obtaining the depth consistency loss, includes:

[0135] S1061. Obtain the bird's eye view (BEV) features based on the panoramic image.

[0136] S1062. Perform feature recognition on the bird's-eye view features to obtain moving objects, perform motion estimation on the moving objects, and obtain object motion information.

[0137] S1063. Based on the object motion information, the vehicle motion information, and the bird's-eye view features (BEV features), feature reconstruction is performed to obtain synthetic bird's-eye view features.

[0138] S1064. Based on the similarity between the bird's-eye view features and the synthetic bird's-eye view features, obtain the depth consistency loss.

[0139] As described above, in this disclosure, the panoramic image at the current frame time (time t) is denoted as the panoramic target image, and the panoramic images outside the current frame time (time t-1, time t+1, etc.) are denoted as the panoramic source image.

[0140] In some embodiments of this disclosure, S1061, bird's eye view (BEV) features are obtained based on the panoramic image, wherein BEV features of the panoramic image can be obtained using a depth estimation-based BEV model (e.g., LSS model, BEVDet model, etc.).

[0141] In some embodiments of this disclosure, target bird's-eye view features (i.e., target BEV features) are obtained from the surround-view target image, and source bird's-eye view features (i.e., source BEV features) are obtained from the surround-view source image. In this disclosure, the target BEV features characterize the BEV features of the surround-view image at the current time t, and the target BEV features are denoted as feature B. tThe source BEV feature represents the BEV feature of the look-through source image at other times besides the current time t. These other times can be time t-1, time t+1, etc. In this disclosure, the source BEV feature is denoted as feature B. s .

[0142] In some embodiments of this disclosure, S1062, feature recognition is performed on the bird's-eye view features to obtain moving objects, motion estimation is performed on the moving objects, and object motion information is obtained. A pre-trained image segmentation model (e.g., Mask-RCNN model) can be used to distinguish moving objects from the background in the panoramic image (i.e., the bird's-eye view) to identify the moving objects. Furthermore, motion estimation of the moving objects to obtain object motion information can be achieved by modeling the motion of the moving objects.

[0143] Figure 7 This is a schematic diagram of a process for acquiring a moving object and acquiring object motion information according to one embodiment of this disclosure.

[0144] refer to Figure 7 In some embodiments of this disclosure, step S1062, which involves performing feature recognition on the bird's-eye view features to obtain moving objects, estimating the motion of the moving objects, and obtaining object motion information, includes:

[0145] S621. Obtain the target moving object based on the target bird's-eye view perspective features of the panoramic target image, and obtain the source moving object based on the source bird's-eye view perspective features of the panoramic source image.

[0146] S622. Match the target moving object obtained from the surround view target image with the source moving object obtained from the surround view source image to obtain the same moving object.

[0147] S623. Obtain the motion information of the object based on the same moving object.

[0148] In some embodiments of this disclosure, the segmentation mask in the BEV features of two surround view images (surround view target image and surround view source image) must first be matched.

[0149] In step S621, the Mask-RCNN model can be used to obtain the recognition results S0, S1, ..., S based on the input BEV features. n Where S0 represents the background mask, S1, S2, ..., S n Represents a mask for moving objects.

[0150] The recognition results for the surround-view target image include a target moving object mask, and the recognition results for the surround-view source image include a source moving object mask. The target moving object mask represents the moving object within the target's BEV feature B.t The mask in the middle is denoted as The source moving object mask represents the moving object in the source BEV feature B. s The mask in the middle is denoted as The subscript i is used to identify the moving object.

[0151] The mask of the same moving object in the surround view target image matches the mask in the surround view source image; therefore, the target moving object mask of the same moving object... Source moving object mask Form a mask pair.

[0152] The matching strategy described in step S622 preferably employs the maximum intersection-union ratio (MUR) strategy. That is, for the target moving object mask... In the source BEV feature B s Find the source moving object mask with the largest intersection-union ratio. As a matching mask, a mask pair of the same moving object described above is obtained.

[0153] For mask pairs of the same moving object, the background and moving object in the panoramic image can be distinguished based on the binary mask, thus obtaining the moving object.

[0154] For example, a first binary mask O0(S) is defined for a static background, O0(S) = 1 - ∪ i S i , among which, ∪ i S i Represents the sum of sets S i Take the union of each element in 1-∪. i S i , indicating that the set and S i After taking the union of each element in S, the corresponding set is S. i Each binary bit in the array will be cleared (set to 0), while the other bits will remain unchanged (remain at 1), in order to block out all moving objects.

[0155] For each moving object, define a second binary mask O. i (S), O i (S)=S i , (i>0), is used to return the binary mask of the moving object i, thus obtaining the moving object.

[0156] In some embodiments of this disclosure, S623, the motion information of the object is obtained based on the same moving object. This motion information can be obtained by modeling the motion of the moving object, and a pose transformation model can be used as the object motion model to characterize the target BEV feature B of the same moving object.t Heyuan BEV characteristics B s Motion changes between the surrounding target image (i.e., the current frame) and the surrounding source image (e.g., adjacent frames).

[0157] For example, the motion information of moving object i It can be obtained through the following formula three.

[0158] Formula 3:

[0159] In Formula 3, ⊙ represents element-wise multiplication.

[0160] Any adjustments or selections made by those skilled in the art to the method of obtaining motion information of a moving object based on the technical solutions disclosed herein fall within the protection scope of this disclosure.

[0161] In some embodiments of this disclosure, S1063, feature reconstruction is performed based on the object motion information, the vehicle motion information, and the bird's-eye view features (BEV features) to obtain synthetic bird's-eye view features. Bilinear interpolation can be used for feature reconstruction: from the source BEV features B... s Pixel sampling is performed on the target BEV feature B t Feature reconstruction yields the synthetic bird's-eye view features described above.

[0162] In some embodiments of this disclosure, from the source BEV feature B s Before performing pixel sampling, it is necessary to process the source BEV features B. s and target BEV features B t Perform coordinate alignment.

[0163] The coordinates of the region containing a moving object in a panoramic image can be determined based on the object's movement information. Coordinate alignment is performed. For the coordinates of the area containing the stationary background in the surround view image, coordinate alignment can be performed based on the vehicle's motion information. The vehicle's motion information can be obtained based on existing algorithms (e.g., using the pose change parameters of the onboard camera obtained in real time by the onboard attitude sensor as the vehicle's motion information), and this disclosure does not impose any particular limitations on it.

[0164] Figure 8 This is a flowchart of one embodiment of the present disclosure for obtaining features of a synthetic bird's-eye view.

[0165] refer to Figure 8 S1063. Based on the object motion information, the vehicle motion information, and the bird's-eye view features (BEV features), feature reconstruction is performed to obtain synthetic bird's-eye view features, including:

[0166] S631, Based on object motion information and vehicle operation information, the source BEV features B s and target BEV features B t Perform coordinate alignment.

[0167] S632, From the coordinate-aligned source BEV feature B s Pixel sampling is performed to obtain the target BEV feature B after coordinate alignment. t Perform feature reconstruction to obtain the features of the synthesized bird's-eye view.

[0168] In some embodiments of this disclosure, the pixel sampling and feature reconstruction described above can be performed based on the bilinear interpolation method described above.

[0169] For example, in some embodiments of this disclosure, for static areas (background areas) in a surround view image, the vehicle motion information is used. To perform the coordinate alignment described above, refer to Formulas 4 and 5:

[0170] Formula 4: V = O0(S t )⊙O0(S s );

[0171] Formula 5: Where, p t ∈B t ⊙V, B t ⊙V represents the static area where moving objects are ignored in the bird's-eye view feature (BEV feature).

[0172] Where, p t p represents the homogeneous pixel coordinates of the pan-view target image. s p represents the pixel projection coordinates. s Characterizing p t Projection onto the panoramic source image.

[0173] For dynamic regions (moving object regions) in a panoramic image, use motion information. To perform the coordinate alignment described above, refer to Formula Six:

[0174] Formula Six:

[0175] Where i is the identifier for the moving object, and B t ⊙O i (S t ) refers to the moving object i (i = 1, 2, ..., n) in the bird's-eye view feature (BEV feature), where p t p represents the homogeneous pixel coordinates of the pan-view target image. s p represents the pixel projection coordinates. sCharacterizing p t Projection onto the panoramic source image.

[0176] In some embodiments of this disclosure, in step S632, the bilinear interpolation described above can be sampled from the source BEV feature B. s Pixel sampling is performed to obtain the target BEV feature B after coordinate alignment. t Feature reconstruction was performed to obtain the features of the synthesized bird's-eye view.

[0177] In some embodiments of this disclosure, the L1 loss term is used as the BEV feature consistency loss to measure the similarity between the bird's-eye view features and the synthetic bird's-eye view features, as shown in Formula 7:

[0178] Formula 7:

[0179] Preferably, the sampling method is the same as in step S1043 described above, and the loss function (L1 loss term) is extended to multiple frames to obtain the minimum BEV feature consistency loss.

[0180] Figure 9 This is a flowchart of obtaining a first depth loss according to one embodiment of this disclosure.

[0181] refer to Figure 9 In some embodiments of this disclosure, S1064, obtaining the depth consistency loss based on the similarity between the bird's-eye view features and the synthetic bird's-eye view features, includes:

[0182] S641. Reconstruct the surrounding target image based on different surrounding source images to obtain different synthetic bird's-eye view features.

[0183] S642. Calculate the BEV feature consistency loss based on the error between the bird's-eye view features of the surround view target image and different synthetic bird's-eye view features, and obtain the BEV feature consistency loss set.

[0184] In this design, different surround-view source images can form a series of multiple frames with the surround-view target image. For example, different surround-view source images may include the image from the previous frame and the image from the next frame of the surround-view target image. Alternatively, different surround-view source images may include the image from the previous frame and the image from the two frames preceding the surround-view target image. Any selection or adjustment of different surround-view source images by those skilled in the art based on the teachings of this disclosure falls within the protection scope of this disclosure.

[0185] S643. Obtain the minimum BEV feature consistency loss in the BEV feature consistency loss set as the deep consistency loss.

[0186] This disclosure preferably uses the aforementioned minimum BEV feature consistency loss to implicitly address the problem of unfilled areas caused by moving object masks.

[0187] In this disclosure, L is used. b To represent the depth consistency loss, refer to Formula 8:

[0188] Formula 8: L b =min s∈{t-1,t+1} L1(B t B t→s ).

[0189] Where s∈{t-1,t+1}, represents that the surround view source image includes the image at time t-1 and the image at time t+1, B t→s Indicates the source BEV feature B s Features of a synthetic bird's-eye view obtained by pixel sampling.

[0190] In some embodiments of this disclosure, in step S108, L final The total loss can be represented by the first depth loss L obtained from this disclosure. p and depth consistency loss L b Obtain the total loss L final See Formula Nine for reference:

[0191] Formula 9: L final =L p +L b .

[0192] In step S110, the model parameters of the surround-view depth estimation model described above in this disclosure are adjusted based on the total loss mentioned above, to obtain the adjusted surround-view depth estimation model, thus realizing the estimation based on the total loss L. final The self-supervised development of the surround depth estimation model (surround depth estimation network) can effectively improve the accuracy of depth estimation.

[0193] The depth information acquisition method based on panoramic images disclosed herein is a panoramic self-supervised depth estimation method that utilizes the features of bird's-eye view (BEV).

[0194] This disclosed method for acquiring depth information based on surround-view images generates BEV features using a BEV model for depth estimation. Therefore, depth estimation can be constrained by directly comparing BEV features between adjacent frames. The aligned BEV features of consecutive frames obtained through vehicle motion information and object motion information in the scene are consistent. Furthermore, the BEV features themselves represent the scene features surrounding the vehicle, allowing for simple and direct utilization of the intersection information of the surround-view camera array and scene geometry, avoiding the high computational cost and insufficient utilization of BEV features inherent in traditional attention mechanisms.

[0195] This disclosure aligns the dynamic BEV features of consecutive frames and performs moving object segmentation (i.e., instance segmentation) on the BEV features to obtain moving objects and background. For moving objects, this disclosure uses a pose estimation network to predict the pose transformation matrix of each object one by one to align the dynamic features; for the background, it directly uses the vehicle motion information for alignment, thereby aligning the BEV features of the entire scene.

[0196] In some embodiments of this disclosure, the present disclosure proposes to supervise the depth estimation task using the minimum BEV feature consistency loss. For example, in BEV feature generation methods based on the LSS paradigm, the intermediate task of depth estimation is utilized. By imposing consistency constraints on temporal BEV features, the depth estimation can be supervised simply and effectively using look-around information and scene geometric information.

[0197] In some embodiments of this disclosure, alignment of dynamic BEV features is proposed to handle dynamic objects (i.e., moving objects) in a scene. By modeling the motion of objects in the scene, the problem of inaccurate depth estimation caused by object motion can be effectively solved.

[0198] Based on the depth information acquisition method based on panoramic images described above, this disclosure also provides a depth information acquisition device 1000 based on panoramic images, comprising:

[0199] Image acquisition module 1002 acquires a panoramic image.

[0200] The first depth loss acquisition module 1004 performs monocular depth estimation on the panoramic image based on a monocular depth estimation model to acquire the first depth loss.

[0201] The depth consistency loss acquisition module 1006 estimates the depth of the surround view image based on the surround view depth estimation model and acquires the depth consistency loss.

[0202] The total loss acquisition module 1008 acquires the total loss based on the first depth loss and the depth consistency loss.

[0203] The model parameter adjustment module 1010 adjusts the model parameters of the surround view depth estimation model based on the total loss to obtain the adjusted surround view depth estimation model.

[0204] The depth information acquisition module 1012 acquires the depth information of the surround view image based on the adjusted surround view depth estimation model.

[0205] Figure 10 This is a schematic block diagram of a depth information acquisition device 1000 based on a surround view image, which employs a hardware implementation of a processing system according to one embodiment of this disclosure.

[0206] The depth information acquisition device 1000 based on panoramic images disclosed herein may include corresponding modules that perform one or more steps in the flowchart described above. Therefore, each or more steps in the flowchart may be performed by a corresponding module, and the device may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0207] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0208] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0209] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0210] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0211] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0212] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0213] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware. The program can be stored in a readable storage medium, and when executed, the program includes one or a combination of the steps of the method implementation.

[0214] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0215] This disclosure also provides an electronic device, including: a memory storing execution instructions; and a processor or other hardware module executing the execution instructions stored in the memory, causing the processor or other hardware module to execute the above-described method for acquiring depth information based on a surround view image.

[0216] This disclosure also provides a readable storage medium storing executable instructions, which, when executed by a processor, are used to implement the above-described method for acquiring depth information based on a surround view image.

[0217] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0218] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0219] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for acquiring depth information based on a panoramic image, characterized in that, include: Acquiring panoramic images includes: acquiring a monocular image sequence; taking the set of monocular images at the same frame time of each monocular image sequence as a panoramic image to obtain a panoramic image sequence, wherein the monocular image sequence is a set of images taken by monocular cameras in a panoramic camera array ordered by time, and the panoramic camera array includes multiple monocular cameras taking pictures from a bird's-eye view perspective. Based on the monocular depth estimation model, monocular depth estimation is performed on the panoramic image to obtain the first depth loss; The surround-view depth estimation model is used to estimate the surround-view depth of the surround-view images and obtain depth consistency loss. This includes: recording the surround-view image at the current frame as the surround-view target image and recording surround-view images outside the current frame as surround-view source images; obtaining target BEV features from the surround-view target images and source BEV features from the surround-view source images; obtaining target moving objects based on the target BEV features of the surround-view target images and source moving objects based on the source BEV features of the surround-view source images; matching the target moving objects obtained from the surround-view target images with the source moving objects obtained from the surround-view source images according to the moving object masks to obtain identical moving objects; obtaining object motion information based on the identical moving objects; and classifying the object motion information, vehicle motion information, and BEV features. Feature reconstruction is performed on different surround-view source images to obtain multiple different synthetic BEV features; based on the similarity between the target BEV features of the surround-view target image and the multiple different synthetic BEV features, a BEV feature consistency loss set is obtained; the minimum BEV feature consistency loss in the BEV feature consistency loss set is obtained as the depth consistency loss, wherein feature reconstruction to obtain synthetic BEV features includes: aligning the coordinates of the source BEV features and the target BEV features of the moving object region and the background region of the surround-view image based on object motion information and vehicle motion information, respectively; sampling pixels from the coordinate-aligned source BEV features; and reconstructing features from the coordinate-aligned target BEV features to obtain the synthetic BEV features. The total loss is obtained based on the first depth loss and the depth consistency loss; The surround view depth estimation model is adjusted based on the total loss to obtain the adjusted surround view depth estimation model; and The depth information of the surround view image is obtained based on the adjusted surround view depth estimation model.

2. The method for obtaining depth information based on a surround view image according to claim 1, characterized in that, Monocular depth estimation is performed on the panoramic image based on the monocular depth estimation model to obtain the first depth loss, including: The monocular image at the current frame time is recorded as the monocular target image, and the monocular images at other times are recorded as the monocular source images. Based on monocular vision, the depth map of the monocular target image in the panoramic target image is obtained, and the relative pose between the monocular target image and the monocular source image is obtained. Based on the depth map and relative pose corresponding to the monocular target image, the monocular target image is reconstructed from pixels sampled from the monocular source image to obtain a monocular synthesized image; and The first depth loss is obtained based on the error between the monocular target image and the corresponding monocular synthesized image.

3. The method for obtaining depth information based on a surround view image according to claim 2, characterized in that, Based on the depth map and relative pose corresponding to the monocular target image, the monocular target image is reconstructed from pixels sampled from the monocular source image to obtain a monocular synthesized image, including: Obtain the camera intrinsic parameter matrix and the homogeneous pixel coordinates of the monocular target image; Pixel projection coordinates are obtained based on the depth map, the relative pose, the camera intrinsic parameter matrix, and the pixel homogeneous coordinates. The pixel projection coordinates represent the projection of the pixel homogeneous coordinates onto the monocular source image. Bilinear interpolation is performed based on the pixel homogeneous coordinates and the pixel projected coordinates to obtain the sampled pixel value; and The monocular target image is reconstructed based on the sampled pixel values ​​to obtain the monocular synthesized image.

4. The method for obtaining depth information based on a surround view image according to claim 2, characterized in that, The error between the monocular target image and the corresponding monocular synthetic image is measured by photometric reprojection loss.

5. The method for obtaining depth information based on a surround view image according to claim 2, characterized in that, The first depth loss is obtained based on the error between the monocular target image and the corresponding monocular synthesized image, including: The monocular target image is reconstructed based on different monocular source images to obtain different monocular synthetic images; Based on the errors between the monocular target image and different monocular synthesized images, the photometric reprojection loss is calculated to obtain a set of photometric reprojection losses; and The minimum photometric reprojection loss in the set of photometric reprojection losses is obtained as the first depth loss.

6. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes the execution instructions stored in the memory, causing the processor to perform the depth information acquisition method based on a surround view image as described in any one of claims 1 to 5.

7. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the depth information acquisition method based on a surround view image as described in any one of claims 1 to 5.

8. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the depth information acquisition method based on a surround view image as described in any one of claims 1 to 5.

Images