Depth estimation method, device and system

Abstract

The invention provides a depth estimation method, device and system, and relates to the technical field of image processing. The method comprises the following steps: acquiring a two-dimensional feature group (including a fisheye image and a pinhole image) of an image group; projecting the three-dimensional voxel space to the image group based on a forward projection model to obtain a two-dimensional projection coordinate set; performing interpolation sampling on two-dimensional projection coordinates in the two-dimensional projection coordinate set to obtain a three-dimensional voxel feature set; performing feature fusion on the three-dimensional voxel feature set to obtain fused voxel features; re-projecting the fused voxel features to the image group based on a back projection model to obtain a re-projected two-dimensional feature group; and determining a depth estimation result according to the re-projection two-dimensional feature group. Thus, cameras with any pose and different internal and external parameters can be supported to participate in feature fusion and depth estimation, no specific requirements for camera types, number and poses exist, and the applicability of depth estimation is improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a depth estimation method, apparatus and system. Background Technology

[0002] In fields such as low-speed parking, robot obstacle avoidance, and video surveillance, vehicles and robots can perform real-time obstacle detection in the environment based on image vision to identify obstacles that cannot be detected due to blind spots in sensors such as LiDAR. However, the aforementioned image vision-based obstacle detection has high requirements for camera type, number, and pose when performing obstacle depth estimation, resulting in poor applicability. Summary of the Invention

[0003] This application provides a depth estimation method, apparatus, and system that can solve the problem that depth estimation in the above-mentioned image vision-based obstacle detection has high requirements for camera type, number, and pose, and has poor applicability.

[0004] Firstly, a depth estimation method is provided. This method includes: acquiring a two-dimensional feature set of an image group, the image group including at least one fisheye image captured by at least one fisheye camera and at least one pinhole image captured by at least one pinhole camera; projecting a three-dimensional voxel space of a preset size onto the image group based on a forward projection model to obtain a two-dimensional projection coordinate set; interpolating and sampling the two-dimensional projection coordinates in the two-dimensional projection coordinate set to obtain a three-dimensional voxel feature set corresponding to the two-dimensional projection coordinate set; performing voxel feature stitching and feature fusion on the three-dimensional voxel feature set to obtain fused voxel features; reprojecting the fused voxel features onto the image group based on a back projection model to obtain a reprojected two-dimensional feature set; and determining the depth estimation result based on the reprojected two-dimensional feature set.

[0005] The forward projection models include fisheye forward projection and pinhole forward projection. The fisheye forward projection model converts 3D voxel space coordinates into 2D feature space coordinates corresponding to the fisheye camera, while the pinhole forward projection model converts 3D voxel space coordinates into 2D feature space coordinates corresponding to the pinhole camera. The backward projection models include fisheye backward projection and pinhole backward projection. The fisheye backward projection model converts 2D feature space coordinates corresponding to the fisheye camera into 3D voxel space coordinates, while the pinhole backward projection model converts 2D feature space coordinates corresponding to the pinhole camera into 3D voxel space coordinates.

[0006] Based on the aforementioned depth estimation method, a forward projection model projects the 3D voxel space onto the image set for 3D voxel feature sampling, achieving backprojection of distorted and reconstructed image features from different viewpoints. Conversely, a backward projection model reprojects the fused voxel features onto the image set for 2D feature sampling, achieving forward projection of 3D voxel features into the 2D feature space from different viewpoints. Thus, through forward and backward projection between the 3D voxel space and the 2D feature space, depth elements are introduced into the 2D feature space of fisheye or pinhole images from different viewpoints within the image set. This allows for simultaneous support of several heterogeneous camera inputs (fisheye (distorted) and several pinhole (undistorted) cameras, and supports cameras with arbitrary poses and different intrinsic and extrinsic parameters participating in feature fusion and depth estimation. It places no specific requirements on camera type, number, or pose, thus improving the applicability of depth estimation.

[0007] This application does not limit the execution subject of the depth estimation method; the depth estimation method can be executed by any computing device capable of image processing functions such as image recognition. For example, the execution subject of the depth estimation method can be a processor (central processing unit, CPU) or a computing device such as an onboard computer, computer, or robot that contains a processor.

[0008] Combining the depth estimation method provided in the first aspect, as a possible implementation, a three-dimensional voxel space of a preset size is projected onto an image group based on a forward projection model to obtain a set of two-dimensional projection coordinates. This includes: projecting the three-dimensional voxel space onto a two-dimensional plane corresponding to at least one fisheye camera based on a fisheye forward projection model to obtain fisheye two-dimensional feature space coordinates; projecting the three-dimensional voxel space onto a two-dimensional plane corresponding to at least one pinhole camera based on a pinhole forward projection model to obtain pinhole two-dimensional feature space coordinates; and selecting two-dimensional feature space coordinates within the image size range of the two-dimensional feature group from the fisheye two-dimensional feature space coordinates and the pinhole two-dimensional feature space coordinates to obtain a set of two-dimensional projection coordinates.

[0009] Based on the above implementation method, the projection of the three-dimensional voxel space onto the two-dimensional plane of the fisheye camera and the two-dimensional plane of the pinhole camera is performed using the fisheye forward projection model and the pinhole forward projection model, respectively. This determines the mapping relationship between the coordinates of the three-dimensional voxel space and the coordinates of the two-dimensional feature space of the fisheye image and the two-dimensional feature space of the pinhole image, thus enabling the acquisition of the two-dimensional features corresponding to the three-dimensional voxels in the two-dimensional feature space. In this way, the image processing of heterogeneous multi-view images is transformed into image processing in the same three-dimensional space, avoiding inaccurate feature acquisition caused by feature differences between fisheye and pinhole images due to camera heterogeneity, and ensuring the accuracy of image feature acquisition from heterogeneous cameras.

[0010] Combining the depth estimation method provided in the first aspect, as a possible implementation, interpolation sampling is performed on the two-dimensional projected coordinates in the two-dimensional projected coordinate set to obtain the three-dimensional voxel feature set corresponding to the two-dimensional projected coordinate set. This includes: for each two-dimensional projected coordinate in the two-dimensional projected coordinate set, interpolation sampling is performed on the features of the pixels in the neighborhood range of each two-dimensional projected coordinate to obtain the three-dimensional voxel features corresponding to each two-dimensional projected coordinate.

[0011] Based on the above implementation, the features of pixels in the neighborhood of each two-dimensional projection coordinate are interpolated and sampled, which reduces the coordinate error of the three-dimensional voxel space coordinates projected to the two-dimensional feature space and improves the accuracy of interpolation sampling.

[0012] Combining the depth estimation method provided in the first aspect, as a possible implementation, voxel feature concatenation and feature fusion are performed on the three-dimensional voxel feature set to obtain fused voxel features. This includes: determining whether each three-dimensional voxel feature in the three-dimensional voxel feature set belongs to a common-view region or a non-common-view region; for the three-dimensional voxel features of the common-view region, performing concatenation and three-dimensional convolution operations on the three-dimensional voxel features of different images to obtain fused voxel features of the common-view region; for the three-dimensional voxel features of the non-common-view region, adding the three-dimensional voxel features of different images and performing three-dimensional convolution operations to obtain fused voxel features of the non-common-view region; and adding the fused voxel features of the common-view region and the fused voxel features of the non-common-view region to obtain fused voxel features.

[0013] Based on the above implementation, the shared and non-shared viewing regions from each camera are modeled and processed separately in the 3D voxel space. This avoids the limitations of camera heterogeneity and instead leverages the complementary enhancement of the asymmetric viewing ranges of different cameras. The shared viewing regions more effectively help correct fisheye distortion areas, effectively recovering depth information of obstacles in distant regions and improving the accuracy of near-range depth estimation based on pinhole images. This improves the accuracy of depth prediction for objects in 3D space and reduces the dependence on fitting massive amounts of data. Furthermore, this fusion scheme can perform mid-level fusion between the neural network encoder and decoder, or post-fusion in the post-processing module after the encoder, offering flexible placement and enabling multi-level heterogeneous shared-view fusion effects.

[0014] In combination with the above implementation, optionally, when the two-dimensional projection coordinates corresponding to the three-dimensional voxel features in the three-dimensional voxel feature set are located within the image size range of the two-dimensional feature sets corresponding to at least two images in the image set, the three-dimensional voxel features belong to the common viewing region.

[0015] In combination with the above implementation, optionally, when the two-dimensional projection coordinates corresponding to the three-dimensional voxel features in the three-dimensional voxel feature set are located within the image size range of the two-dimensional feature group corresponding to an image in the image group, the three-dimensional voxel features belong to the non-common viewing region.

[0016] Combining the depth estimation method provided in the first aspect, as a possible implementation, the reprojected two-dimensional feature set includes at least one fisheye back projection tensor and at least one pinhole back projection tensor. In the depth estimation method's process, the fused voxel features are reprojected onto the image set based on the back projection model to obtain the reprojected two-dimensional feature set, including: converting at least one fisheye image into at least one fisheye back projection tensor based on the fisheye back projection model; converting at least one pinhole image into at least one pinhole back projection tensor based on the pinhole back projection model; the fisheye back projection tensor and the pinhole back projection tensor are used to indicate the three-dimensional voxel space coordinates of each pixel in the preset tensor when back-projected to the three-dimensional voxel space at different depth distances, and the value of each element of the preset tensor is a preset equally spaced depth value; sampling is performed based on the three-dimensional voxel space coordinates of each pixel at each depth distance in the at least one fisheye back projection tensor and the at least one pinhole back projection tensor to obtain the reprojected two-dimensional feature set.

[0017] Based on the above implementation method, by setting a preset tensor in the three-dimensional voxel space and introducing a depth dimension into the projection from the three-dimensional voxel space to the two-dimensional plane, the correspondence between depth values ​​and coordinates in the two-dimensional feature space is realized.

[0018] In conjunction with the above implementation, optionally, a reprojected two-dimensional feature set is obtained by sampling the three-dimensional voxel space coordinates of each pixel at various depth distances in at least one fisheye inverse tensor and at least one pinhole inverse tensor. This includes: interpolating and sampling the fused voxel features based on the three-dimensional voxel space coordinates of each pixel at various depth distances in at least one fisheye inverse tensor and at least one pinhole inverse tensor to obtain a sampled two-dimensional feature set; and performing a pooling operation on the sampled two-dimensional feature set to obtain the reprojected two-dimensional feature set.

[0019] In conjunction with the above implementation method, optionally, the depth estimation result is determined based on the reprojection two-dimensional feature set, including: determining the disparity prediction value based on the reprojection two-dimensional feature set; and normalizing the disparity prediction value to obtain the depth estimation result.

[0020] In conjunction with the depth estimation method provided in the first aspect, as a possible implementation, the depth estimation method also includes the following steps: extracting bird's-eye view features from the fused voxel features; and determining the pose matrix prediction value corresponding to the bird's-eye view features.

[0021] Based on the above implementation method, the pose matrix prediction value is determined by the bird's-eye view. The image can be reconstructed based on the depth estimation result and the pose matrix prediction value, and then depth estimation is performed. This supports depth estimation in visual sensor scenes under different poses and ensures the flexibility of depth estimation.

[0022] In conjunction with the depth estimation method provided in the first aspect, as a possible implementation, a two-dimensional feature set of an image group is obtained, including: obtaining a two-dimensional feature set of an image group through a feature encoder.

[0023] In conjunction with the above implementation, optionally, voxel feature splicing and feature fusion are performed on the three-dimensional voxel feature set to obtain fused voxel features, including: performing voxel feature splicing and feature fusion on the three-dimensional voxel feature set through a fusion module to obtain fused voxel features; the fusion module includes a splicing layer, an addition layer, a convolutional layer, and a fusion layer, wherein the splicing layer is used to perform voxel feature splicing operations, the addition layer is used to perform voxel feature addition operations, the convolutional layer is used to perform three-dimensional convolution operations on the three-dimensional voxel features on the result of the splicing or addition operations, and the fusion layer is used to perform addition operations on the result output by the convolutional layer to obtain fused voxel features.

[0024] In conjunction with the above implementation, optionally, the depth estimation result is determined based on the reprojected two-dimensional feature set, including: inputting the reprojected two-dimensional feature set into the decoder to obtain the depth estimation result.

[0025] Optionally, in combination with the above implementation methods, the feature encoder, fusion module, and decoder are obtained by training the neural network using a combination of supervised training and self-supervised training methods.

[0026] In conjunction with the above implementation, optionally, the supervised loss function of the supervised training method is used to indicate the loss of the depth estimation result output by the decoder compared to the ground truth depth values ​​of pixels in the fisheye image or pinhole image. The loss function of the self-supervised training method is used to indicate the loss of pixels in the fisheye image at the first time step after being projected onto the pinhole image compared to the ground truth pixel values ​​of the pinhole image at the second time step, the loss of pixels in the pinhole image at the first time step after being projected onto the fisheye image compared to the ground truth pixel values ​​of the fisheye image at the second time step, and the image reconstruction loss after mutual reprojection of pinhole images and fisheye images from adjacent pinhole cameras and fisheye cameras; wherein, the first time step precedes the second time step.

[0027] Based on the above implementation method, a deep learning approach is adopted, which utilizes heterogeneous camera images for mutual supervision. The heterogeneous self-supervised training includes cameras with shared viewing areas, including fisheye cameras and pinhole cameras. Through this learning paradigm, it is possible to use pinhole cameras to assist fisheye cameras in depth estimation, or vice versa, thereby improving the accuracy of depth estimation.

[0028] In a second aspect, a depth estimation apparatus is provided, comprising modules for performing the depth estimation method in the first aspect or any possible implementation thereof.

[0029] As one possible implementation, the depth estimation device may include an acquisition module, a projection module, a sampling module, a fusion module, and a depth estimation module. The acquisition module acquires a two-dimensional feature set of an image group; the image group includes at least one fisheye image captured by at least one fisheye camera and at least one pinhole image captured by at least one pinhole camera. The projection module projects a three-dimensional voxel space of a preset size onto the image group based on a forward projection model, obtaining a two-dimensional projection coordinate set; the forward projection model includes a fisheye forward projection model and a pinhole forward projection model. The fisheye forward projection model converts the three-dimensional voxel space coordinates into two-dimensional feature space coordinates corresponding to the fisheye camera, and the pinhole forward projection model converts the three-dimensional voxel space coordinates into two-dimensional feature space coordinates corresponding to the pinhole camera. The sampling module interpolates and samples the two-dimensional projection coordinates in the two-dimensional projection coordinate set to obtain a three-dimensional voxel feature set corresponding to the two-dimensional projection coordinate set. The fusion module performs voxel feature stitching and feature fusion on the three-dimensional voxel feature set to obtain fused voxel features. The projection module is also used to reproject the fused voxel features onto the image group based on the back projection model, resulting in a reprojected two-dimensional feature group. The back projection model includes a fisheye back projection model and a pinhole back projection model. The fisheye back projection model converts the two-dimensional feature space coordinates corresponding to the fisheye camera into three-dimensional voxel space coordinates, while the pinhole back projection model converts the two-dimensional feature space coordinates corresponding to the pinhole camera into three-dimensional voxel space coordinates. The depth estimation module is used to determine the depth estimation result based on the reprojected two-dimensional feature group.

[0030] Thirdly, a computing device is provided, including a memory and a processor, wherein the memory is used to store a set of computer instructions, and when the processor executes the set of computer instructions, it is used to perform the operational steps of the depth estimation method in the first aspect or any possible implementation of the first aspect.

[0031] Fourthly, a depth estimation system is provided. The depth estimation system includes at least one fisheye camera, at least one pinhole camera, and the aforementioned computing device. The at least one fisheye camera is used to acquire at least one fisheye image, the at least one pinhole camera is used to acquire at least one pinhole image, and the computing device is used to perform the operation steps of the depth estimation method in the first aspect or any possible implementation of the first aspect based on the at least one fisheye image and the at least one pinhole image.

[0032] Furthermore, the technical effects of the depth estimation device described in the second aspect, the computing device described in the third aspect, and the depth estimation system described in the fourth aspect can be referred to with reference to the technical effects of the depth estimation method described in the first aspect, and will not be repeated here.

[0033] Fifthly, a chip is provided, including a processor for calling and executing instructions stored in a memory, such that a computing device on which the chip is mounted performs the depth estimation method described in any possible implementation of the first aspect above.

[0034] In a sixth aspect, another chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, the output interface, the processor, and the memory are connected via an internal connection path, and the processor is used to execute code in the memory, wherein when the code is executed, the processor is used to execute the depth estimation method described in any possible implementation of the first aspect above.

[0035] A seventh aspect provides a readable storage medium. The readable storage medium includes: a computer program or instructions; when the computer program or instructions are executed on a computer, causing the computer to perform the depth estimation method described in any possible implementation of the first aspect.

[0036] Eighthly, a vehicle is provided. The vehicle includes a computing device as described in the third aspect above or a depth estimation system as described in the fourth aspect above.

[0037] A ninth aspect provides a computer program product. The computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the depth estimation method described in any possible implementation of the first aspect. Attached Figure Description

[0038] Figure 1 This application provides a schematic diagram of the architecture of an in-vehicle system;

[0039] Figure 2 A schematic diagram of a camera provided for this application;

[0040] Figure 3 A flowchart illustrating a depth estimation method provided in this application. Figure 1 ;

[0041] Figure 4 A schematic diagram illustrating the relationship between a forward projection model and a backward projection model provided in this application;

[0042] Figure 5 A flowchart illustrating a voxel fusion step provided in this application;

[0043] Figure 6 A schematic diagram of a fisheye-pinhole cross-camera heterogeneous training process provided in this application;

[0044] Figure 7 A flowchart illustrating a depth estimation method provided in this application. Figure 2 ;

[0045] Figure 8 A schematic diagram of a depth estimation device provided in this application;

[0046] Figure 9 A schematic diagram of the structure of a computing device provided in this application. Detailed Implementation

[0047] The technical solutions of this application will now be described with reference to the accompanying drawings. In this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0048] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0049] The depth estimation method provided in this application can be applied to scenarios of autonomous driving (or assisted driving) and automatic parking (or assisted parking) for vehicles, robots, etc. The following is a brief introduction to the technologies that may be involved in this application.

[0050] Depth estimation refers to estimating the depth of a scene in a camera-imaged image using one or more 2D images taken from multiple viewpoints. Depth estimation is the vertical distance from each pixel in the scene to the camera's imaging plane. It is a crucial step in scene reconstruction, such as 3D reconstruction, within the field of computer vision.

[0051] Deep learning is a machine learning technique based on deep neural network models and methods. It has evolved from statistical machine learning and artificial neural network algorithms, combined with the advancements in big data and computing power. The most important technical characteristic of deep learning is its ability to automatically extract features, enabling it to automatically learn and identify complex patterns within large datasets.

[0052] General obstacle detection (GOD) aims to enable vehicles or robots to identify and respond to various non-standard or unusual obstacles through multi-sensor fusion and self-supervised learning. This technology is trained on massive amounts of data, giving vehicles environmental perception capabilities similar to human drivers, enabling them to identify and handle irregular obstacles on the road, such as piles of dirt, overturned vehicles, and fallen cardboard boxes.

[0053] Self-supervised learning is an unsupervised learning method that utilizes the inherent characteristics of data for model training. It involves pre-training using unlabeled data to generate inputs and labels, followed by fine-tuning on a small amount of labeled data. Compared to traditional supervised learning methods, self-supervised learning eliminates the need for manually labeled large amounts of data, saving time and costs, and has achieved excellent results on several tasks.

[0054] Depth estimation in obstacle detection, such as general obstacle detection, can be achieved based on images captured by a single camera or multiple cameras.

[0055] For example, unsupervised learning-based fisheye camera scene depth estimation has achieved the ability to estimate scene depth of monocular fisheye images. However, the method of depth estimation based on monocular images has the problem of scale ambiguity. The generalization of ranging scale is limited in different scenarios. Due to the ambiguity effect, the generalization ability of the depth estimation scenarios is limited, resulting in poor applicability.

[0056] For example, depth estimation based on binocular fisheye cameras involves processing two 2D images of the same scene captured simultaneously by the left and right cameras, including correction, feature extraction, image segmentation, and stereo matching, to reconstruct the scene's 3D information. This method has high hardware requirements, necessitating symmetrical cameras, parallel optical axes, and coplanar imaging planes, making it unsuitable for vehicle-mounted camera hardware configurations. Furthermore, it suffers from low hardware deployment flexibility, struggling to support depth estimation with different poses or numbers of visual sensors, resulting in poor applicability.

[0057] For example, using multiple fisheye cameras to achieve surround-view depth estimation essentially still treats the distorted images as if they were from a binocular pinhole camera for depth estimation. This method only supports depth estimation based on images captured by fisheye cameras in a fixed pose and number. Therefore, it requires symmetrical camera hardware, parallel optical axes, and coplanar imaging. It cannot support arbitrary poses, is unsuitable for camera hardware configurations on vehicles or robots, and suffers from poor applicability.

[0058] It is evident that depth estimation in obstacle detection still places high demands on camera type, number, and pose, and has poor applicability to platforms such as vehicles and robots equipped with different types and numbers of cameras.

[0059] This application provides a depth estimation method, specifically a "depth estimation method supporting depth estimation using heterogeneous cameras of arbitrary number and pose". The method includes: acquiring a two-dimensional feature set of an image group, the image group including at least one fisheye image captured by at least one fisheye camera and at least one pinhole image captured by at least one pinhole camera; projecting a three-dimensional voxel space of a preset size onto the image group based on a forward projection model to obtain a two-dimensional projection coordinate set; interpolating and sampling the two-dimensional projection coordinates in the two-dimensional projection coordinate set to obtain a three-dimensional voxel feature set corresponding to the two-dimensional projection coordinate set; performing voxel feature stitching and feature fusion on the three-dimensional voxel feature set to obtain fused voxel features; reprojecting the fused voxel features onto the image group based on a back projection model to obtain a reprojected two-dimensional feature set; and determining the depth estimation result based on the reprojected two-dimensional feature set. The forward projection model includes a fisheye forward projection model and a pinhole forward projection model. The fisheye forward projection model is used to convert the three-dimensional voxel space coordinates into two-dimensional feature space coordinates corresponding to the fisheye camera, and the pinhole forward projection model is used to convert the three-dimensional voxel space coordinates into two-dimensional feature space coordinates corresponding to the pinhole camera. The back projection model includes the fisheye back projection model and the pinhole back projection model. The fisheye back projection model is used to convert the two-dimensional feature space coordinates corresponding to the fisheye camera into three-dimensional voxel space coordinates, while the pinhole back projection model is used to convert the two-dimensional feature space coordinates corresponding to the pinhole camera into three-dimensional voxel space coordinates.

[0060] Based on the aforementioned depth estimation method, a forward projection model projects the 3D voxel space onto the image set for 3D voxel feature sampling, achieving backprojection of distorted and reconstructed image features from different viewpoints. Conversely, a backward projection model reprojects the fused voxel features onto the image set for 2D feature sampling, achieving forward projection of 3D voxel features into the 2D feature space from different viewpoints. Thus, through forward and backward projection between the 3D voxel space and the 2D feature space, depth elements are introduced into the 2D feature space of fisheye or pinhole images from different viewpoints within the image set. This allows for simultaneous support of several heterogeneous camera inputs (fisheye (distorted) and several pinhole (undistorted) cameras, and supports cameras with arbitrary poses and different intrinsic and extrinsic parameters participating in feature fusion and depth estimation. It places no specific requirements on camera type, number, or pose, thus improving the applicability of depth estimation.

[0061] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0062] Figure 1 This is a schematic diagram of the architecture of an in-vehicle system 100 provided in this application. Figure 1 As shown, the vehicle-mounted system 100 (also known as a depth estimation system) is installed in a vehicle. The vehicle-mounted system 100 may consist of one or more subsystems, each performing a different function. Examples include a sensing system 110, a computing system 120, and a display system 130. The vehicle-mounted system 100 described above is merely an example provided in this application and does not limit the possible structure of the vehicle-mounted system 100. The vehicle-mounted system 100 may include more or fewer subsystems, and each subsystem may include multiple devices. Each subsystem and device of the vehicle-mounted system 100 can be connected via wired or wireless means.

[0063] The sensing system 110 may include one or more sensors for sensing information about the environment near the vehicle system 100. For example, the sensing system 110 may include one or more cameras. Taking a vehicle equipped with four cameras as an example... Figure 2 As shown, cameras 111-114 include camera 111 mounted at the front of the vehicle, camera 112 mounted on the left side of the vehicle, camera 113 mounted on the right side of the vehicle, and camera 114 mounted at the rear of the vehicle. The fields of view of any two adjacent cameras may or may not overlap. Cameras 111-114 may employ standard lenses, wide-angle lenses, or ultra-wide-angle lenses. For example, in this embodiment, all cameras 111-114 are fisheye cameras, or all cameras 111-114 are pinhole cameras, or some cameras 111-114 are fisheye cameras and some are pinhole cameras.

[0064] The computing system 120 may include components that provide data storage, computation, and output control commands for the vehicle. Some or all functions of the in-vehicle system 100 are controlled by the computing system 120. The computing system 120 may include one or more processors 121 for executing instructions stored, for example, in memory 122. The computing system 120 can process and analyze images captured by cameras 111-114 to identify objects and / or features in the vehicle's surrounding environment.

[0065] Processor 121 can be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, processor 121 can be a dedicated device such as a application-specific integrated circuit (ASIC), a graphics processing unit (GPU), an embedded neural network processing unit (NPU), or other hardware-based processors. Although Figure 1 The processor, memory, and other components are functionally illustrated; however, those skilled in the art will understand that the processor, computer, or memory may actually include or may not be stored in the same physical housing. For example, memory may be a hard disk drive or other storage medium located in a housing different from that of computing system 120. Therefore, references to processors or computers will be understood to include a collection of processors or computers or memories that may operate in parallel or not. Unlike using a single processor to perform the steps described herein, some components, such as the steering assembly and deceleration assembly, may each have their own processor that performs calculations only related to the component's specific function.

[0066] In one possible embodiment, the processor may be located remotely from the vehicle system 100 and communicate with the vehicle system 100. In other possible embodiments, the above functions may be partially executed on a processor within the vehicle system 100 and partially executed by a remote processor.

[0067] In some embodiments, memory 122 may contain instructions (e.g., program logic) that can be executed by processor 121 to perform various functions of the vehicle system 100. Memory 122 may also contain additional instructions, including instructions for transmitting data, receiving data from, interacting with, and / or controlling one or more of the sensing system 110, display system 130, etc.

[0068] In addition to instructions, memory 122 can also store data. For example, in this embodiment, memory 122 can store images acquired by cameras 111-114.

[0069] Display system 130 is used to support interaction between the vehicle and the user. For example, display system 130 can receive user input through user operation of the touchscreen and can provide visual information to the user.

[0070] In some embodiments, an artificial intelligence (AI) chip is disposed within the computing system 120. The AI ​​chip periodically trains the received sample data to generate a depth estimation model. The AI ​​chip may also be referred to as a neural network processor (NPU), for example, it may be processor 121. The depth estimation model is stored by processor 121; alternatively, it can be stored in memory via a system bus or other transmission method. The AI ​​chip can be a single chip or other physical component, such as a training chip for building a neural network model or an inference chip for performing inference using the neural network model.

[0071] In some embodiments, the computing system 120 can also transmit data with other computing systems, and data from one computing system can be transferred to another computing system for processing.

[0072] The above Figure 1 The vehicle system 100 shown is merely an illustrative example of the architecture to which the depth estimation method provided in this application can be applied, and does not limit the entity that performs the depth estimation method. In possible embodiments, the vehicle system 100 may also include more or fewer subsystems or devices, and may be applied to devices other than vehicles, such as robots.

[0073] Next, the steps of the vehicle system 100 performing the depth estimation method provided in this application will be described in detail with reference to the accompanying drawings.

[0074] Figure 3 A flowchart illustrating a depth estimation method provided in this application. Figure 1 Here, we will take the example of the processor 121 in the vehicle system 100 performing depth estimation based on the image acquired by the sensor system 110.

[0075] S301, Processor 121 acquires the two-dimensional feature set of the image group.

[0076] The processor 121 extracts features from the image group to obtain a two-dimensional feature group of the image group.

[0077] The image set includes heterogeneous views (or heterogeneous images) captured by heterogeneous cameras. For example, the heterogeneous cameras may include at least one fisheye camera and at least one pinhole camera. The heterogeneous views include at least one fisheye image captured by at least one fisheye camera and at least one pinhole image captured by at least one pinhole camera. The at least one fisheye camera and the at least one pinhole camera can be cameras 111-114, and there is a shared viewing area between each pair of adjacent cameras. The two-dimensional feature set includes the two-dimensional features of each image in the image set. The image set includes one or more sets of images. One set of images consists of at least one fisheye image captured synchronously at a single moment, while multiple sets of images consist of at least one fisheye image captured at consecutive moments.

[0078] As one possible implementation, processor 121 extracts two-dimensional features of the image using a feature encoder based on camera intrinsic parameters and distortion parameters. For example, processor 121 extracts two-dimensional features of an image captured by camera 111 using a feature encoder based on camera intrinsic and extrinsic parameters of camera 111.

[0079] Optionally, camera intrinsic parameters refer to the camera's intrinsic parameter matrix and distortion correction parameters, where the intrinsic parameter matrix is: Where fx represents the pixel length of the focal length along the y-axis, fy represents the pixel length of the focal length along the y-axis, and cx and cy represent the positions of the principal points. The intrinsic parameter matrix is ​​fixed at the time of camera manufacture. Distortion correction parameters mainly refer to the distortion correction parameters of the fisheye camera, which are determined by the fisheye camera's model. The camera extrinsic parameter E refers to the extrinsic parameter matrix of each camera transformed to the vehicle coordinate system, which is composed of the rotation matrix R and the translation vector T.

[0080] Optionally, images taken from different perspectives by the same type of camera (such as a fisheye camera or a pinhole camera) can be used to extract features using the same feature encoder to obtain two-dimensional features for different images. The two-dimensional features of all images in the image group constitute a two-dimensional feature group. The form of the two-dimensional feature can be: H×W×C, where H represents the height of the two-dimensional feature, W represents the width of the two-dimensional feature, and C represents the dimension of the two-dimensional feature.

[0081] The aforementioned feature encoder can be an encoder implemented based on Convolutional Neural Networks (CNNs), such as residual networks (ResNet) or transformer architectures.

[0082] S302 and processor 121 project a three-dimensional voxel space of a preset size onto an image group based on a forward projection model to obtain a two-dimensional projection coordinate set.

[0083] The orthographic projection models include the fisheye orthographic projection model and the pinhole orthographic projection model. The fisheye orthographic projection model is used to convert 3D voxel space coordinates into 2D feature space coordinates corresponding to the fisheye camera. The pinhole orthographic projection model is used to convert 3D voxel space coordinates into 2D feature space coordinates corresponding to the pinhole camera.

[0084] The processor 121 projects the feature map of each image in the image group, i.e. each two-dimensional feature in the two-dimensional feature group, onto a set of three-dimensional voxel spaces with a preset size of X×Y×Z×C, to obtain the image region corresponding to the projection position of each voxel.

[0085] As one possible implementation, processor 121 projects the three-dimensional voxel space onto a two-dimensional plane corresponding to at least one fisheye camera based on a fisheye forward projection model to obtain fisheye two-dimensional feature space coordinates, and projects the three-dimensional voxel space onto a two-dimensional plane corresponding to at least one pinhole camera based on a pinhole forward projection model to obtain pinhole two-dimensional feature space coordinates. Then, processor 121 filters two-dimensional feature space coordinates within the image size range of the two-dimensional feature group from the fisheye and pinhole two-dimensional feature space coordinates to obtain a set of two-dimensional projection coordinates. This set of two-dimensional projection coordinates is used to indicate the image region corresponding to the projection position of the voxels in the aforementioned three-dimensional voxel space.

[0086] Next, combine Figure 4 This paper provides a detailed explanation of how to extract two-dimensional feature space coordinates using fisheye orthogonal projection and pinhole orthogonal projection models. Figure 4 This is a schematic diagram illustrating the relationship between a forward projection model and a backward projection model provided in this application. The fisheye projection model is... Fisheye back projection model is The pinhole projection model is The pinhole back projection model is in, I is the coordinate of the 2D image. p is the coordinate in 3D space. θ is the angle between the incident ray and the optical axis corresponding to this pixel, and f(θ) is the Kannala-Brandt fisheye distortion model f(θ)=θ+k1θ 3 +k2θ 5 +k3θ 7 +k4θ 9 Also known as distortion correction parameters, K is the intrinsic parameter matrix and E is the camera extrinsic parameter.

[0087] For example, processor 121 is based on a fisheye forward projection model. Projecting the coordinates p = (x, y, z) of each voxel in the three-dimensional voxel space onto the two-dimensional plane corresponding to at least one fisheye camera yields the two-dimensional feature space coordinates of the fisheye, based on the pinhole forward projection model. Each voxel coordinate p = (x, y, z) in the three-dimensional voxel space is projected onto a two-dimensional plane corresponding to at least one pinhole camera to obtain the pinhole two-dimensional feature space coordinates. The fisheye two-dimensional feature space coordinates and the pinhole two-dimensional feature space coordinates can be represented as (u, v). Then, the processor 121 filters out the set of two-dimensional projected coordinates that fall within the H×W image range.

[0088] The two-dimensional projection coordinate set can include multiple two-dimensional projection coordinate subsets, each of which corresponds to a fisheye image or a pinhole image.

[0089] S303, processor 121 performs interpolation sampling on the two-dimensional projection coordinates in the two-dimensional projection coordinate set to obtain the three-dimensional voxel feature set corresponding to the two-dimensional projection coordinate set.

[0090] Processor 121 uses bilinear interpolation sampling to interpolate the image features of pixels within the neighborhood of all two-dimensional feature space coordinates within the two-dimensional projected coordinate set. The resulting features are... That is, it serves as the corresponding three-dimensional voxel feature set.

[0091] S304 and processor 121 perform voxel feature splicing and feature fusion on the three-dimensional voxel feature set to obtain fused voxel features.

[0092] Based on the shared viewing area information between heterogeneous views, the processor 121 interactively fuses the shared viewing voxel features corresponding to each view in the three-dimensional voxel features, and then sequentially completes and fuses them with the voxel features of the non-shared viewing areas of each view in the three-dimensional voxel features, finally obtaining a unique three-dimensional voxel space containing the fused voxel features after aggregation.

[0093] Next, combine Figure 5 The document provides a detailed explanation of how to obtain the characteristics of fusion voxels. Figure 5 This is a flowchart illustrating a voxel fusion step provided in this application. Figure 5 It includes multiple combined cubes, each representing multiple 3D voxels corresponding to a fisheye image or a pinhole image. A small cube in the combined cube represents a 3D voxel. Among them, the small cubes filled with horizontal lines represent the 3D voxels corresponding to the fisheye image, the small cubes filled with vertical lines represent the 3D voxels corresponding to the pinhole image, and the small cubes filled with diagonal lines represent the common viewing area.

[0094] Step a: Processor 121 determines whether each three-dimensional voxel feature contained in the three-dimensional voxel feature set belongs to a common viewing region or a non-common viewing region.

[0095] For each 3D voxel in the 3D voxel feature set, if the 2D projected coordinates of a 3D voxel projected onto the 2D feature space simultaneously fall into two or more subsets of 2D projected coordinates, the processor 121 determines that the 3D voxel and its corresponding 3D voxel feature belong to a common viewing region. If the 2D projected coordinates of a 3D voxel projected onto the 2D feature space fall into only one subset of 2D projected coordinates, the processor 121 determines that the 3D voxel and its corresponding 3D voxel feature belong to a non-common viewing region.

[0096] Step b: The processor 121 performs a stitching operation and a three-dimensional convolution operation on the three-dimensional voxel features of different images for the shared viewing region to obtain the fused voxel features of the shared viewing region.

[0097] As one possible implementation, processor 121 can use the concat function to perform the above-mentioned concatenation operation. The concat function is used to concatenate two or more feature images along the channel dimension.

[0098] As one possible implementation, processor 121 can use the conv (convolution) function to perform the above-mentioned three-dimensional convolution operation to obtain fused voxel features of the common viewing region with dimension C′. The conv function is used to extract image features through the convolution kernel. By sliding the convolution kernel on the image, weighted summation is performed on each local region to obtain a new feature map.

[0099] Step c: For the three-dimensional voxel features of non-co-view regions, processor 121 adds the three-dimensional voxel features of different images and performs a three-dimensional convolution operation to obtain the fused voxel features of non-co-view regions.

[0100] As one possible implementation, processor 121 can use the add function to perform the above addition operation. The add function is used to overlay pixels of an image without changing the image's dimensions.

[0101] As one possible implementation, the processor 121 can use the conv function to implement the above three-dimensional convolution operation to obtain the fused voxel features of the non-co-view region with dimension C′.

[0102] Step d: Processor 121 adds the fused voxel features of the co-viewing region and the fused voxel features of the non-co-viewing region to obtain the fused voxel features.

[0103] As one possible implementation, the processor 121 can use the add function to perform the addition operation of the fused voxel features of the non-common viewing regions, where the feature dimensions of the fused voxel features are X×Y×Z×C'.

[0104] In this way, the common and non-common viewing regions from each camera are modeled and processed separately in the three-dimensional voxel space, thus avoiding the limitations of the binocular stereo vision method. Instead, the complementary enhancement of the asymmetric viewing range of different cameras can be utilized to improve the overall accuracy of depth estimation.

[0105] S305 and processor 121 reproject the fused voxel features onto the image group based on the back projection model to obtain the reprojected two-dimensional feature group.

[0106] Processor 121 converts at least one fisheye image into at least one fisheye back projection tensor based on a fisheye back projection model, and at least one pinhole image into at least one pinhole back projection tensor based on a pinhole back projection model. Then, processor 121 samples the three-dimensional voxel space coordinates of each pixel at various depth distances in the at least one fisheye back projection tensor and the at least one pinhole back projection tensor to obtain a reprojected two-dimensional feature set. The fisheye back projection tensor and the pinhole back projection tensor are used to indicate the three-dimensional voxel space coordinates of each pixel in the preset tensor when back-projected to the three-dimensional voxel space at different depth distances. The value of each element of the preset tensor is a preset, equally spaced depth value.

[0107] The aforementioned preset tensor is a set of equally spaced depth bins, such as 0.5m-60.5m, 1m-50.5m, or 0m-70m. An equally spaced depth distance is set at each unit distance (e.g., 0.5m, 1m, etc.), resulting in a tensor D with dimensions H×W×N×1, where N represents the number of depth bins. The value of each element of this tensor D is an equally spaced depth value, such as 0.5m, 1.5m, 2.5m, ..., 60.5m.

[0108] As one possible implementation, processor 121 is based on a fisheye back projection model. Convert at least one fisheye image into at least one fisheye inverse tensor based on the pinhole back projection model. At least one pinhole image is converted into at least one pinhole inverse tensor. Each inverse tensor can be represented as a tensor V of dimension H×W×N×3. This tensor V indicates the 3D voxel space coordinates of each pixel in the preset tensor when it is inversely projected into 3D voxel space at different depth distances. Here, I is a tensor of dimension H×W×3, and the value at each position in this tensor... The two-dimensional feature space coordinates representing the element at that location.

[0109] As one possible implementation, processor 121, based on the three-dimensional voxel space coordinates corresponding to the tensor V at various distance positions of each pixel in at least one fisheye reverse tensor and at least one pinhole reverse tensor, samples in the voxel space with fused voxel features using a trilinear interpolation method. The features obtained after sampling are: The number of features is H×W×N. Then, the processor 121 performs a pooling operation to average the N feature values, resulting in a reprojected two-dimensional feature set with dimension H×W. This feature set is the feature obtained from the re-rendering of the three-dimensional voxel space.

[0110] S306, Processor 121 determines the depth estimation result based on the reprojected two-dimensional feature set.

[0111] Processor 121 determines the disparity prediction value based on the reprojected two-dimensional feature set, and performs normalization on the disparity prediction value to obtain the depth prediction result. Processor 121 extracts bird's-eye view features with dimensions of Y×Z×1 from the fused voxel features, and determines the pose matrix prediction value corresponding to the bird's-eye view features. Then, processor 121 determines the depth estimation result based on the depth prediction result and the pose matrix prediction value.

[0112] As one possible implementation, the processor 121 inputs the reprojected two-dimensional feature set into the disparity decoder to obtain the disparity prediction value output by the disparity decoder, which has a size of H×W×1.

[0113] As one possible implementation, the processor 121 inputs the bird's-eye view features into the pose decoder to obtain the pose matrix prediction value output by the pose decoder.

[0114] Based on the aforementioned depth estimation method, a forward projection model projects the 3D voxel space onto the image set for 3D voxel feature sampling, achieving backprojection of distorted and reconstructed image features from different viewpoints. Conversely, a backward projection model reprojects the fused voxel features onto the image set for 2D feature sampling, achieving forward projection of 3D voxel features into the 2D feature space from different viewpoints. Thus, through forward and backward projection between the 3D voxel space and the 2D feature space, depth elements are introduced into the 2D feature space of fisheye or pinhole images from different viewpoints within the image set. This allows for simultaneous support of several heterogeneous camera inputs (fisheye (distorted) and several pinhole (undistorted) cameras, and supports cameras with arbitrary poses and different intrinsic and extrinsic parameters participating in feature fusion and depth estimation. It places no specific requirements on camera type, number, or pose, thus improving the applicability of depth estimation.

[0115] The above text combined Figures 3-5The depth estimation method provided in this application is described in general. Embodiments of the aforementioned depth estimation method can also be combined with deep learning, introducing depth-guided fisheye-pinhole cross-camera heterogeneous self-supervised learning into the depth estimation process. For example, the feature encoder in S301, the voxel feature stitching and feature fusion in S304, and the disparity decoder and pose decoder in S306 are all obtained by training neural networks using deep learning.

[0116] The following is combined Figure 6 The process of heterogeneous training across fisheye and pinhole cameras is described below. Steps a-f can be executed by the processor 121, or by other devices or apparatuses with neural network training capabilities other than processor 121, and the trained depth estimation model is then run by processor 121. The following description uses processor 121 as the execution entity.

[0117] Step a: Extract the monitoring signal.

[0118] Processor 121 acquires frame data synchronized with the laser point cloud and the image, performs depth ground truth projection from the point cloud to the image, and determines the depth ground truth d of some pixels at the corresponding image resolution based on the depth ground truth projection. The laser point cloud can be obtained by accumulating from a single frame or multiple frames, and the corresponding intrinsic parameter coefficients need to be adjusted according to the image resolution during application.

[0119] The supervision signal is the absolute depth supervision signal used to calculate the supervision loss. The images mentioned above can be pinhole images or fisheye images from the image group, or pinhole images or fisheye images used as training data.

[0120] Step b: Obtain depth estimation results.

[0121] Processor 121 inputs the two-dimensional features obtained from reprojection into the disparity decoder to obtain H×W×1 disparity prediction values. The depth prediction result is obtained by taking the reciprocal and denormalizing the values ​​through the post-processing module.

[0122] As one possible implementation, the two-dimensional features can be features from the reprojected two-dimensional feature set; please refer to [reference needed] for their extraction methods. Figure 3 The relevant content in S305 shown will not be repeated here.

[0123] Step c: Calculate the monitoring loss.

[0124] Processor 121 predicts the depth based on the true depth value d and the depth value. Calculate the value of the supervised loss function.

[0125] As one possible implementation, the loss function can include the L1 loss function (or absolute error loss function), silog, or other two-dimensional supervised loss functions, which will not be elaborated here.

[0126] Step d: Calculate the self-supervised loss.

[0127] Calculate the time-series "fisheye-pinhole" self-supervised loss function: Processor 121 predicts the depth of the fisheye image at second t. The fisheye pixels at second t Pinhole artifact reprojected to t+1 second And the actual image of the pinhole at time t+1. Construct an image reconstruction loss. Similarly, processor 121 can also process the pinhole image at second t. Reprojected to the fisheye pseudo-image at t+1 second The actual fisheye image at second t+1. Calculate the image reconstruction loss.

[0128] The specific calculation is as follows: the temporal self-supervised loss from fisheye to pinhole is... The time-series self-supervised loss from pinhole to fisheye is SSIM is the image structure similarity function.

[0129] Computational spatial "fisheye-pinhole" self-supervised loss function: Similar to the temporal "fisheye-pinhole" self-supervised loss function, the difference is that it only calculates the image reconstruction loss after the left and right adjacent fisheye-pinhole cameras reproject each other at the same time.

[0130] The specific calculation is as follows: the self-supervised loss in the space from the fisheye to the pinhole is... The self-supervised loss from pinhole to fisheye is

[0131] Step e: Calculate the total loss.

[0132] The processor 121 sums the above losses to obtain the final total loss, which is used for backpropagation of the gradient of the neural network to update the model parameters. The neural network can be a depth estimation model, which includes a feature encoder, a layer of voxel feature concatenation and feature fusion, a disparity decoder and a pose decoder, etc., all of which update their parameters based on the gradient backpropagation of the total loss.

[0133] Step f: Iterative training.

[0134] Processor 121 repeats steps a-e above based on different training data.

[0135] Thus, by incorporating deep learning into the depth estimation method provided in this application, depth estimation can be achieved by integrating depth maps from multiple perspectives based on deep learning. The common viewing area integrates the spatial features of adjacent views, making depth estimation more accurate for small obstacles at the edges.

[0136] Steps a-f are combined with Figure 3 In the illustrated embodiment, S304 is executed before S306, but this application does not limit the execution order. For example, in other embodiments, S304 is executed after S306. Through common-view region fusion modeling after depth estimation, and guided by supervised loss and self-supervised loss, the depth map is further optimized, fully utilizing image information to make the post-processed depth map more refined and the prediction of small obstacles more accurate. Figure 7 As shown, depth estimation methods may include:

[0137] Step 1: Preparation of fisheye and pinhole image data.

[0138] Step one includes obtaining heterogeneous views using at least one fisheye camera and at least one pinhole camera as described in S301 above, which will not be repeated here.

[0139] Step 2: Heterogeneous image encoding.

[0140] Step two includes feature extraction of the image group in S301 above, to obtain a two-dimensional feature group of the image group, which will not be described again here.

[0141] Step 3: Heterogeneous image decoding and depth estimation.

[0142] Step three includes S306 mentioned above, and will not be repeated here.

[0143] Step 4: Fusion of multi-level heterogeneous co-view regions based on three-dimensional voxel space aggregation.

[0144] Step four performs operations such as S302-S305 on the depth estimation results output from step three, thereby fusing the depth estimation results in three-dimensional voxel space, performing a weighted average on the non-common viewing regions of the heterogeneous images that are not fused, and finally backpropagating the gradient to the pre-decoded depth estimation results based on supervised loss and depth-guided fisheye-pinhole cross-camera heterogeneous self-supervised loss to further optimize the output of the depth estimation results.

[0145] Thus, by fusing and modeling the common viewing regions after depth estimation, and by using supervised loss and self-supervised loss to guide further optimization of the depth estimation results, the image information is fully utilized to make the post-processed depth estimation results more refined and the prediction of small obstacles more accurate.

[0146] The depth estimation method of this application has been described in detail above. In conjunction with the above-described depth estimation method, this application also provides a depth estimation device. Figure 8 This is a schematic diagram of a possible depth estimation device provided in this embodiment. This depth estimation device can be used to implement the function of the processor 121 in the above method embodiment, and therefore also has the beneficial effects of the above method embodiment. In this embodiment, the depth estimation device can be as follows: Figure 3 The processor 121 shown may also be a module (such as a chip) used in a server.

[0147] like Figure 8 As shown, the depth estimation device 800 includes an acquisition module 810, a projection module 820, a sampling module 830, a fusion module 840, and a depth estimation module 850.

[0148] The depth estimation device 800 can be used to achieve the above. Figure 3 The processor 121 in the method embodiment shown has the function of the depth estimation device 800, and the various modules included in the depth estimation device 800 are specifically used to implement the following functions.

[0149] The acquisition module 810 is used to acquire a two-dimensional feature set of an image set; the image set includes at least one fisheye image taken by at least one fisheye camera and at least one pinhole image taken by at least one pinhole camera.

[0150] The projection module 820 is used to project a three-dimensional voxel space of a preset size onto an image group based on a forward projection model to obtain a set of two-dimensional projection coordinates. The forward projection model includes a fisheye forward projection model and a pinhole forward projection model. The fisheye forward projection model is used to convert the three-dimensional voxel space coordinates into two-dimensional feature space coordinates corresponding to the fisheye camera, and the pinhole forward projection model is used to convert the three-dimensional voxel space coordinates into two-dimensional feature space coordinates corresponding to the pinhole camera.

[0151] The sampling module 830 is used to interpolate and sample the two-dimensional projected coordinates in the two-dimensional projected coordinate set to obtain the three-dimensional voxel feature set corresponding to the two-dimensional projected coordinate set.

[0152] The fusion module 840 is used to perform voxel feature splicing and feature fusion on the three-dimensional voxel feature set to obtain fused voxel features.

[0153] The projection module 820 is also used to reproject the fused voxel features onto the image group based on the back projection model to obtain the reprojected two-dimensional feature group. The back projection model includes the fisheye back projection model and the pinhole back projection model. The fisheye back projection model is used to convert the two-dimensional feature space coordinates corresponding to the fisheye camera into three-dimensional voxel space coordinates, and the pinhole back projection model is used to convert the two-dimensional feature space coordinates corresponding to the pinhole camera into three-dimensional voxel space coordinates.

[0154] The depth estimation module 850 is used to determine the depth estimation result based on the reprojected two-dimensional feature set.

[0155] As one possible implementation, the projection module 820 is specifically used to: project the three-dimensional voxel space onto the two-dimensional plane corresponding to at least one fisheye camera based on the fisheye forward projection model to obtain the fisheye two-dimensional feature space coordinates; project the three-dimensional voxel space onto the two-dimensional plane corresponding to at least one pinhole camera based on the pinhole forward projection model to obtain the pinhole two-dimensional feature space coordinates; and select the two-dimensional feature space coordinates within the image size range of the two-dimensional feature group from the fisheye two-dimensional feature space coordinates and the pinhole two-dimensional feature space coordinates to obtain the two-dimensional projection coordinate set.

[0156] As one possible implementation, the sampling module 830 is specifically used to: for each two-dimensional projection coordinate in the set of two-dimensional projection coordinates, interpolate and sample the features of the pixels in the neighborhood of each two-dimensional projection coordinate to obtain the three-dimensional voxel features corresponding to each two-dimensional projection coordinate.

[0157] As one possible implementation, the fusion module 840 is specifically used to: determine whether each three-dimensional voxel feature contained in the three-dimensional voxel feature set belongs to a common-view region or a non-common-view region; for the three-dimensional voxel features of the common-view region, perform a stitching operation and a three-dimensional convolution operation on the three-dimensional voxel features of different images to obtain the fused voxel features of the common-view region; for the three-dimensional voxel features of the non-common-view region, add the three-dimensional voxel features of different images and perform a three-dimensional convolution operation to obtain the fused voxel features of the non-common-view region; and add the fused voxel features of the common-view region and the fused voxel features of the non-common-view region to obtain the fused voxel features.

[0158] As one possible implementation, when the two-dimensional projection coordinates of the three-dimensional voxel features in the three-dimensional voxel feature set are located within the image size range of the two-dimensional feature sets corresponding to at least two images in the image group, the three-dimensional voxel features belong to the common viewing region; when the two-dimensional projection coordinates of the three-dimensional voxel features in the three-dimensional voxel feature set are located within the image size range of the two-dimensional feature sets corresponding to one image in the image group, the three-dimensional voxel features belong to the non-common viewing region.

[0159] As one possible implementation, the reprojected two-dimensional feature set includes at least one fisheye back projection tensor and at least one pinhole back projection tensor. The projection module 820 is specifically used to: convert at least one fisheye image into at least one fisheye back projection tensor based on a fisheye back projection model; convert at least one pinhole image into at least one pinhole back projection tensor based on a pinhole back projection model; the fisheye back projection tensor and the pinhole back projection tensor are used to indicate the three-dimensional voxel space coordinates of each pixel in the preset tensor when back-projected to the three-dimensional voxel space at different depth distances, where the value of each element of the preset tensor is a preset equally spaced depth value; and sample the three-dimensional voxel space coordinates of each pixel at various depth distances in the at least one fisheye back projection tensor and the at least one pinhole back projection tensor to obtain the reprojected two-dimensional feature set.

[0160] As one possible implementation, the projection module 820 is specifically used to: perform interpolation sampling on the fused voxel features based on the three-dimensional voxel space coordinates of each pixel at various depth distances in at least one fisheye inverse tensor and at least one pinhole inverse tensor to obtain a sampled two-dimensional feature group; and perform pooling operation on the sampled two-dimensional feature group to obtain a reprojected two-dimensional feature group.

[0161] As one possible implementation, the depth estimation module 850 is specifically used to: determine the disparity prediction value based on the reprojected two-dimensional feature set; normalize the disparity prediction value to obtain the depth prediction result; extract the bird's-eye view features from the fused voxel features; determine the pose matrix prediction value corresponding to the bird's-eye view features; and determine the depth estimation result based on the depth prediction result and the pose matrix prediction value.

[0162] As one possible implementation, the acquisition module 810 is specifically used to: acquire a two-dimensional feature set of an image group through a feature encoder.

[0163] As one possible implementation, the fusion module 840 is specifically used to: perform voxel feature stitching and feature fusion on the three-dimensional voxel feature set through the fusion module to obtain fused voxel features; the fusion module includes a stitching layer, an addition layer, a convolutional layer and a fusion layer, the stitching layer is used to perform voxel feature stitching operation, the addition layer is used to perform voxel feature addition operation, the convolutional layer is used to perform three-dimensional convolution operation on the result of the stitching operation or addition operation on the three-dimensional voxel features, and the fusion layer is used to perform addition operation on the result output by the convolutional layer to obtain fused voxel features.

[0164] As one possible implementation, the depth estimation module 850 is specifically used to input the reprojected two-dimensional feature set into the decoder to obtain the depth estimation result.

[0165] It should be understood that the depth estimation device 800 of this embodiment of the present invention can be implemented by a CPU, an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The depth estimation device 800 is implemented in software. Figure 3 The depth estimation method shown can be implemented using a depth estimation device 800 and its various modules, which can also be software modules.

[0166] It should be understood that the processor 121 in the embodiments of this application is related to the depth estimation device 800 in the embodiments of this application, and may correspond to the corresponding subject executing the method according to the embodiments of this application, and the above and other operations and / or functions of each module in the depth estimation device 800 are respectively for implementing Figure 3 The corresponding process of the method in the document will not be elaborated here for the sake of brevity.

[0167] For example, Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. The computing device can be a terminal device, network device, robot, in-vehicle computer, etc., or it can be a chip (system) or other component or assembly that can be disposed in a terminal device, network device, robot, or in-vehicle computer. Figure 9 As shown, the computing device 900 may include a processor 901. Optionally, the computing device 900 may also include a memory 902 and / or a transceiver 903. The processor 901 is coupled to the memory 902 and the transceiver 903, for example, via a communication bus.

[0168] The following is combined Figure 9 A detailed description of each component of the computing device 900 is provided below:

[0169] The processor 901 is the control center of the computing device 900. It can be a single processor or a collective term for multiple processing elements. For example, the processor 901 can be one or more central processing units, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application, such as one or more microprocessors (digital signal processors, DSPs), or one or more field-programmable gate arrays.

[0170] Optionally, the processor 901 can perform various functions of the computing device 900 by running or executing software programs stored in the memory 902 and by calling data stored in the memory 902, such as... Figure 3 The functions implemented by the modules S301-S306 or depth estimation device 800, such as acquisition module 810, projection module 820, sampling module 830, fusion module 840, and depth estimation module 850.

[0171] In a specific implementation, as one example, the processor 901 may include one or more CPUs, for example... Figure 9 CPU0 and CPU1 are shown in the diagram.

[0172] In a specific implementation, as one example, the computing device 900 may also include multiple processors, for example... Figure 9 The processors 901 and 904 are shown. Each of these processors can be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0173] The memory 902 is used to store software programs that execute the solution of this application, such as causing the processor 901 to execute... Figure 3 The software programs for the functions implemented by the modules 810, 820, 830, 840, and 850 in the depth estimation device 800 (S301-S306) shown are controlled and executed by the processor 901. Specific implementation methods can be found in the above method embodiments and will not be repeated here. Furthermore, the memory 902 can also be used to store images acquired by the camera and corresponding point cloud data.

[0174] Optionally, the memory 902 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 902 may be integrated with the processor 901 or may exist independently and be accessible through the interface circuitry of the computing device 900. Figure 9 (Not shown in the image) is coupled to the processor 901, but this embodiment does not specifically limit this.

[0175] Transceiver 903 is used for communication with other devices. For example, if computing device 900 is an in-vehicle computer, transceiver 903 can be used to communicate with a robot or with another in-vehicle computer.

[0176] Alternatively, transceiver 903 may include a receiver and a transmitter. Figure 9 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0177] Optionally, the transceiver 903 can be integrated with the processor 901, or it can exist independently and be connected to the interface circuit of the computing device 900. Figure 9 (Not shown in the image) is coupled to the processor 901, but this embodiment does not specifically limit this.

[0178] It should be noted that, Figure 9 The structure of the computing device 900 shown is not intended to limit the device. Actual devices may include more or fewer components than shown, or may combine certain components, or have different component arrangements.

[0179] Furthermore, the technical effects of the computing device 900 can be referenced from the technical effects of the depth estimation method described in the above method embodiments, and will not be repeated here.

[0180] It should be understood that the processor in the embodiments of this application can be a central processing unit, or it can be other general-purpose processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0181] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be ROM, programmable read-only memory (PROM), EPROM, EEPROM, or flash memory. Volatile memory can be RAM, which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0182] This application also provides a chip, including a processor. The processor is used to retrieve and execute instructions stored in a memory, causing a computing device equipped with the chip to perform the aforementioned depth estimation method.

[0183] As one possible implementation, the chip also includes an input interface, an output interface, and a memory. The input interface, the output interface, the processor, and the memory are connected via internal interconnection paths. The processor is used to execute code in the memory, and when the code is executed, the processor is used to perform the depth estimation method described above.

[0184] This application embodiment also provides a vehicle, the vehicle including... Figure 9 The computing device shown or Figure 1 The vehicle-mounted system or depth estimation system shown is used to perform the depth estimation method described above.

[0185] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0186] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0187] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0189] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0191] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0192] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A depth estimation method, characterized in that, include: Obtain a two-dimensional feature set of an image group; the image group includes at least one fisheye image taken by at least one fisheye camera and at least one pinhole image taken by at least one pinhole camera; Based on the forward projection model, a three-dimensional voxel space of a preset size is projected onto the image group to obtain a two-dimensional projection coordinate set; the forward projection model includes a fisheye forward projection model and a pinhole forward projection model. The fisheye forward projection model is used to convert the three-dimensional voxel space coordinates into two-dimensional feature space coordinates corresponding to the fisheye camera, and the pinhole forward projection model is used to convert the three-dimensional voxel space coordinates into two-dimensional feature space coordinates corresponding to the pinhole camera. Interpolate and sample the two-dimensional projected coordinates in the set of two-dimensional projected coordinates to obtain the three-dimensional voxel feature set corresponding to the set of two-dimensional projected coordinates. The three-dimensional voxel feature set is spliced ​​and fused to obtain fused voxel features; The fused voxel features are reprojected onto the image group based on the back projection model to obtain the reprojected two-dimensional feature group; the back projection model includes a fisheye back projection model and a pinhole back projection model. The fisheye back projection model is used to convert the two-dimensional feature space coordinates corresponding to the fisheye camera into three-dimensional voxel space coordinates, and the pinhole back projection model is used to convert the two-dimensional feature space coordinates corresponding to the pinhole camera into three-dimensional voxel space coordinates. The depth estimation result is determined based on the reprojected two-dimensional feature set.

2. The method according to claim 1, characterized in that, The method of projecting a three-dimensional voxel space of a preset size onto the image group based on a forward projection model to obtain a two-dimensional projection coordinate set includes: Based on the fisheye forward projection model, the three-dimensional voxel space is projected onto the two-dimensional plane corresponding to the at least one fisheye camera to obtain the two-dimensional feature space coordinates of the fisheye. Based on the pinhole forward projection model, the three-dimensional voxel space is projected onto the two-dimensional plane corresponding to the at least one pinhole camera to obtain the pinhole two-dimensional feature space coordinates. The two-dimensional feature space coordinates within the image size range of the two-dimensional feature group are selected from the fisheye two-dimensional feature space coordinates and the pinhole two-dimensional feature space coordinates to obtain the two-dimensional projection coordinate set.

3. The method according to claim 1 or 2, characterized in that, The step of interpolating and sampling the two-dimensional projected coordinates in the two-dimensional projected coordinate set to obtain the three-dimensional voxel feature set corresponding to the two-dimensional projected coordinate set includes: For each two-dimensional projection coordinate in the set of two-dimensional projection coordinates, the features of the pixels in the neighborhood of each two-dimensional projection coordinate are interpolated and sampled to obtain the three-dimensional voxel features corresponding to each two-dimensional projection coordinate.

4. The method according to any one of claims 1-3, characterized in that, The process of concatenating and fusing voxel features in the three-dimensional voxel feature set to obtain fused voxel features includes: Determine whether each three-dimensional voxel feature contained in the three-dimensional voxel feature set belongs to a common-view region or a non-common-view region; For the three-dimensional voxel features of the shared viewing region, the three-dimensional voxel features of different images are stitched together and three-dimensional convolutional to obtain the fused voxel features of the shared viewing region. For the three-dimensional voxel features of the non-co-view region, the three-dimensional voxel features of different images are added together and a three-dimensional convolution operation is performed to obtain the fused voxel features of the non-co-view region. The fusion voxel feature is obtained by adding the fusion voxel feature of the co-view region and the fusion voxel feature of the non-co-view region.

5. The method according to claim 4, characterized in that, When the two-dimensional projection coordinates corresponding to the three-dimensional voxel features in the three-dimensional voxel feature set are located within the image size range of the two-dimensional feature sets corresponding to at least two images in the image group, the three-dimensional voxel features belong to the common viewing region. When the two-dimensional projection coordinates corresponding to the three-dimensional voxel features in the three-dimensional voxel feature set are located within the image size range of the two-dimensional feature set corresponding to an image in the image group, the three-dimensional voxel features belong to the non-common viewing region.

6. The method according to any one of claims 1-5, characterized in that, The reprojected two-dimensional feature set includes at least one fisheye inverse tensor and at least one pinhole inverse tensor. The reprojection of the fused voxel features onto the image set based on the inverse projection model to obtain the reprojected two-dimensional feature set includes: Based on the fisheye back projection model, the at least one fisheye image is converted into the at least one fisheye back tensor; Based on the pinhole back projection model, the at least one pinhole image is converted into the at least one pinhole back tensor; the fisheye back tensor and the pinhole back tensor are used to indicate the three-dimensional voxel space coordinates of each pixel in the preset tensor when it is back projected to the three-dimensional voxel space at different depth distances, and the value of each element of the preset tensor is a preset equally spaced depth value. The reprojected two-dimensional feature set is obtained by sampling the three-dimensional voxel space coordinates of each pixel at various depth distances in the at least one fisheye inverse tensor and the at least one pinhole inverse tensor.

7. The method according to claim 6, characterized in that, The step of sampling the three-dimensional voxel space coordinates of each pixel at various depth distances in the at least one fisheye inverse tensor and the at least one pinhole inverse tensor to obtain the reprojected two-dimensional feature set includes: Based on the three-dimensional voxel space coordinates of each pixel at various depth distances in the at least one fisheye inverse tensor and the at least one pinhole inverse tensor, interpolation sampling is performed on the fused voxel features to obtain a sampled two-dimensional feature group. A pooling operation is performed on the sampled two-dimensional feature set to obtain the reprojected two-dimensional feature set.

8. The method according to any one of claims 1-7, characterized in that, The step of determining the depth estimation result based on the reprojected two-dimensional feature set includes: The disparity prediction value is determined based on the reprojection two-dimensional feature set; Normalize the disparity prediction values ​​to obtain the depth prediction results; Extract bird's-eye view features from the fused voxel features; Determine the pose matrix prediction value corresponding to the bird's-eye view features; The depth estimation result is determined based on the depth value prediction result and the pose matrix prediction result.

9. The method according to any one of claims 1-8, characterized in that, The acquisition of the two-dimensional feature set of the image group includes: The two-dimensional feature set of the image group is obtained by a feature encoder; The process of concatenating and fusing voxel features in the three-dimensional voxel feature set to obtain fused voxel features includes: The fusion module performs voxel feature stitching and feature fusion on the three-dimensional voxel feature set to obtain fused voxel features. The fusion module includes a stitching layer, an addition layer, a convolutional layer, and a fusion layer. The stitching layer is used to perform voxel feature stitching operations, the addition layer is used to perform voxel feature addition operations, the convolutional layer is used to perform three-dimensional convolution operations on the results of the stitching or addition operations, and the fusion layer is used to perform addition operations on the output of the convolutional layer to obtain the fused voxel features. The step of determining the depth estimation result based on the reprojected two-dimensional feature set includes: The reprojected two-dimensional feature set is input into the decoder to obtain the depth estimation result.

10. The method according to claim 9, characterized in that, The feature encoder, the fusion module, and the decoder are obtained by training the neural network using a combination of supervised training and self-supervised training methods.

11. The method according to claim 10, characterized in that, The supervised loss function of the supervised training method is used to indicate the loss of the depth estimation result output by the decoder compared to the ground truth depth value of the pixels in the fisheye image or pinhole image. The loss function of the self-supervised training method is used to indicate the loss of the pixel ground truth of the pinhole image after the pixel of the fisheye image at the first time point is projected onto the pinhole image, compared to the pixel ground truth of the pinhole image at the second time point; the loss of the pixel ground truth of the pinhole image at the first time point is projected onto the fisheye image, compared to the pixel ground truth of the fisheye image at the second time point; and the image reconstruction loss after the pinhole images and fisheye images of adjacent pinhole cameras and fisheye cameras are reprojected onto each other; wherein, the first time point precedes the second time point.

12. A depth estimation device, characterized in that, The apparatus includes modules that perform the operational steps of the method according to any one of claims 1-11.

13. A chip, characterized in that, The chip includes a processor for retrieving and executing instructions stored in a memory, causing the chip to perform the method as described in any one of claims 1-11.

14. A computing device, characterized in that, The computing device includes a memory and a processor, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions, it performs the operation steps of the method according to any one of claims 1-11.

15. A depth estimation system, characterized in that, The device includes at least one fisheye camera, at least one pinhole camera, and a computing device as described in claim 14 above, wherein the at least one fisheye camera is used to acquire at least one fisheye image, the at least one pinhole camera is used to acquire at least one pinhole image, and the computing device is used to perform the operation steps of the method according to any one of claims 1-11 above based on the at least one fisheye image and the at least one pinhole image.

16. A vehicle, characterized in that, The vehicle includes the computing device as described in claim 14 or the depth estimation system as described in claim 15.

17. A readable storage medium, characterized in that, The readable storage medium includes a computer program or instructions that, when executed on a computer, cause the computer to perform the operational steps of the method according to any one of claims 1-11.

Images