Multi-view fisheye camera-based depth estimation method, device, equipment and medium

Abstract

The application discloses a multi-view fisheye camera-based depth estimation method and device, equipment and medium, relates to the technical field of computer vision, and the method comprises the following steps: acquiring fisheye images covering 360-degree panorama through multiple fisheye cameras; constructing a stereo pair of Cassini projection in the visual field overlapping area of fisheye images shot by adjacent two fisheye cameras; performing stereo matching on each stereo pair to obtain a disparity map and a confidence map of each stereo pair; converting each disparity map into a depth map; converting each confidence map and each depth map from Cassini projection to equirectangular projection, and then fusing each depth map according to each confidence map to obtain a fused omnidirectional depth map; and performing refinement processing on the fused omnidirectional depth map to eliminate the difference error generated when converting from Cassini projection to equirectangular projection and the detail error generated during fusion, so as to obtain a target omnidirectional depth map. The application improves the accuracy and calculation efficiency by constructing a stereo pair of Cassini projection.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to depth estimation methods, apparatus, devices and media based on multi-view fisheye cameras. Background Technology

[0002] In numerous navigation applications, fast and reliable omnidirectional 3D perception is crucial. Currently, the main methods for acquiring omnidirectional 3D information are LiDAR and stereo camera solutions. LiDAR solutions suffer from high cost and large size, and the generated depth maps are sparse and lack detailed information. Stereo camera solutions face challenges such as increased system weight, size, and cost due to the increased number of cameras, as well as blind spots between lenses. In contrast, using a relatively small number of fisheye cameras for omnidirectional depth estimation significantly reduces cost and weight, making it a better solution.

[0003] In the field of omnidirectional depth estimation based on multi-view fisheye cameras, various methods have been proposed. Existing convolutional neural network-based methods have made some progress in improving accuracy and robustness, but still have many shortcomings. For example, some methods have poor accuracy in calculating omnidirectional depth, suffer from uneven layering, and have poor generalization ability in unknown scenes, relying heavily on minimum depth prior information. In terms of computational efficiency, most existing methods typically require constructing a 4D cost volume containing a large amount of redundant information or learning complex epipolar geometry, resulting in low efficiency. Summary of the Invention

[0004] The main objective of this application is to propose a depth estimation method, apparatus, device, and medium based on a multi-view fisheye camera, so as to improve the accuracy and efficiency of depth estimation.

[0005] To achieve the above objectives, one aspect of this application proposes a depth estimation method based on a multi-view fisheye camera, the method comprising the following steps:

[0006] Fisheye images of corresponding perspectives are acquired using multiple fisheye cameras; wherein the field of view of the multiple fisheye cameras covers a 360-degree panorama.

[0007] Construct a stereo pair of Cassini projections in the overlapping areas of the fields of view in the fisheye images captured by two adjacent fisheye cameras.

[0008] Stereo matching is performed on each stereo pair to obtain a disparity map and a first confidence map for each stereo pair;

[0009] Each of the disparity maps is converted into a depth map as the first depth map;

[0010] Each of the first confidence maps and each of the first depth maps are converted from the Cassini projection to the equidistant cylindrical projection, and then the first depth maps are fused according to each of the first confidence maps to obtain a fused omnidirectional depth map;

[0011] The fused omnidirectional depth map is refined to eliminate the difference error caused by the conversion from the Cassini projection to the equidistant cylindrical projection and the detail error generated during fusion, thereby obtaining the target omnidirectional depth map.

[0012] In some embodiments, constructing a Cassini projection stereo pair in the overlapping regions of the fields of view in the fisheye images captured by two adjacent fisheye cameras includes the following steps:

[0013] The fisheye images are projected onto a unit sphere in a Cartesian coordinate system using a fisheye camera model. The projections of the fisheye images are then converted from the Cartesian coordinate system to the Cassini spherical coordinate system, and finally mapped onto the Cassini projection image.

[0014] Construct the Cassini projection stereo pair on the Cassini projection image corresponding to the overlapping field of view;

[0015] The Cassini spherical coordinate system is defined as (ρ, φ, θ), where ρ represents the distance between the origin O and point P. Let θ be the angle between line OP and plane yOz, and let θ be the angle between line OP′ and the positive Z-axis, where point P′ is the projection of point P onto plane yOz; the coordinates (x, y, z) of the Cartesian coordinate system and the coordinates of the Cassini spherical coordinate system are... The conversion relationships are as follows:

[0016]

[0017] In the Cassini projection, point P is defined. l and point P r Angular parallax d = φ l -φ r The baseline length is B, and point P is... l Depth ρ to the origin O l Calculated by the following formula:

[0018]

[0019] In some embodiments, performing stereo matching on each stereo pair to obtain a disparity map and a first confidence map for each stereo pair includes the following steps:

[0020] The cost volume is constructed for each of the stereo pairs according to the ACVNet method;

[0021] The relevant stereo matching volume and the cost volume are regularized using an hourglass network;

[0022] The cost volume after regularization is upsampled to obtain the disparity map and the first confidence map for each stereo pair.

[0023] In some embodiments, fusing the first depth maps according to the first confidence maps to obtain a fused omnidirectional depth map includes the following steps:

[0024] Align the first depth maps and the first confidence maps from different perspectives to the same direction and position by rotation and translation to obtain the aligned second confidence map and second depth map;

[0025] The individual second depth maps are fused into an initial omnidirectional depth map based on the confidence information of the second confidence map;

[0026] The fused omnidirectional depth map is obtained by adjusting the depth values ​​of the initial omnidirectional depth map based on the adjacent depth information of each location using a lightweight fusion network with an encoder-decoder architecture and hop connections.

[0027] In some embodiments, the refinement process of the fused omnidirectional depth map to eliminate the difference error caused by the conversion from the Cassini projection to the equidistant cylindrical projection and the detail error generated during fusion, to obtain the target omnidirectional depth map, includes the following steps:

[0028] For each location in the fused omnidirectional depth map, samples are taken within the depth neighborhood to generate a depth hypothesis;

[0029] The feature maps of each fisheye image are mapped to an equidistant cylindrical projection using a fisheye camera imaging model and the aforementioned depth assumption, thereby constructing a 4D cost volume.

[0030] The 4D cost volume is regularized using an hourglass network, and then the target omnidirectional depth map is obtained using the softargmin function based on the regularized 4D cost volume.

[0031] In some embodiments, the method further includes the following steps:

[0032] Navigation information is generated based on the target omnidirectional depth map.

[0033] In some embodiments, acquiring fisheye images from corresponding viewpoints using multiple fisheye cameras includes the following steps:

[0034] The fisheye images are acquired from corresponding perspectives by multiple fisheye cameras installed on vehicles, robots, or drones.

[0035] To achieve the above objectives, another aspect of this application proposes a depth estimation device based on a multi-view fisheye camera, the device comprising:

[0036] The image acquisition unit is used to acquire fisheye images from corresponding perspectives using multiple fisheye cameras; wherein the field of view of the multiple fisheye cameras covers a 360-degree panorama.

[0037] A stereo pair construction unit is used to construct a Cassini projection stereo pair in the overlapping areas of the field of view in the fisheye images captured by two adjacent fisheye cameras.

[0038] A stereo matching unit is used to perform stereo matching on each stereo pair to obtain a disparity map and a first confidence map for each stereo pair.

[0039] A parallax conversion unit is used to convert each of the parallax maps into a depth map as a first depth map;

[0040] The deep fusion unit is used to convert each of the first confidence maps and each of the first depth maps from the Cassini projection to the equidistant cylindrical projection, and then fuse each of the first depth maps according to each of the first confidence maps to obtain a fused omnidirectional depth map.

[0041] An error processing unit is used to refine the fused omnidirectional depth map to eliminate the difference error caused by the conversion from the Cassini projection to the equidistant cylindrical projection and the detail error generated during fusion, so as to obtain the target omnidirectional depth map.

[0042] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described depth estimation method based on a multi-view fisheye camera.

[0043] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned depth estimation method based on a multi-view fisheye camera.

[0044] The embodiments of this application include at least the following beneficial effects:

[0045] This application can acquire fisheye images from corresponding viewpoints using multiple fisheye cameras; wherein the field of view of the multiple fisheye cameras covers a 360-degree panorama; constructing Cassini projection stereo pairs in the overlapping areas of the fields of view in fisheye images captured by two adjacent fisheye cameras; performing stereo matching on each stereo pair to obtain a disparity map and a first confidence map for each stereo pair; converting each disparity map into a depth map as a first depth map; converting each first confidence map and each first depth map from Cassini projection to equidistant cylindrical projection, and then fusing each first depth map according to each first confidence map to obtain a fused omnidirectional depth map; refining the fused omnidirectional depth map to eliminate the difference error caused by the conversion from Cassini projection to equidistant cylindrical projection and the detail error generated during fusion, to obtain a target omnidirectional depth map. This application constructs a stereo pair of Cassini projections to project each fisheye image onto a specific plane, so that the epipolar lines of adjacent views are horizontally aligned after projection. This simplifies the photometric matching process, reduces the computational difficulty of depth estimation, and enables subsequent stereo matching to quickly and accurately obtain the depth map from the stereo pair of Cassini projections, thereby improving accuracy and computational efficiency. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A schematic flowchart illustrating the depth estimation method based on a multi-view fisheye camera provided in this application embodiment;

[0048] Figure 2 A schematic diagram illustrating the arrangement of four fisheye cameras on the roof of a vehicle, as provided in an embodiment of this application;

[0049] Figure 3 An example flowchart of a depth estimation method based on a multi-view fisheye camera provided in an embodiment of this application;

[0050] Figure 4 Example diagrams of constructing Cassini projection stereo pairs from fisheye images provided in this application embodiment;

[0051] Figure 5 An exemplary schematic diagram illustrating the result of depth estimation based on a fisheye image, provided for an embodiment of this application;

[0052] Figure 6 A schematic diagram of the structure of a depth estimation device based on a multi-view fisheye camera provided in an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0055] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0056] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0058] Before providing a detailed description of the embodiments of this application, some terms and related technologies involved in the embodiments of this application will be explained as follows:

[0059] 1. Omnidirectional depth estimation:

[0060] This refers to using multiple cameras to acquire 360° depth information of the surroundings, enabling three-dimensional geometric perception of the entire scene, and has important applications in fields such as autonomous driving, robot navigation, and drone navigation.

[0061] 2. Field of View (FOV):

[0062] 3. An angular measure of the range of the scene that the camera can capture. In the multi-view fisheye camera system of this application, the field of view determines the size of the area that each camera can cover. Fisheye cameras typically have a large field of view, enabling them to capture a wider range of scene information.

[0063] 4. Cassini projection:

[0064] A spherical projection method is used in this application to project fisheye images onto a specific plane, so that the epipolar lines of adjacent views are arranged horizontally after projection, thereby simplifying the photometric matching process and reducing the computational difficulty of depth estimation. This is one of the key solutions for achieving efficient depth estimation in this application.

[0065] 5. Equirectangular Projection (ERP):

[0066] A projection method for mapping the surface of a sphere onto a plane. In omnidirectional depth estimation, fisheye images or other panoramic images are often converted to equidistant cylindrical projections for processing and analysis. In this application, the depth map needs to be converted to equidistant cylindrical projections at a certain stage for operations such as multi-view fusion.

[0067] 6. Stereo matching:

[0068] In computer vision, depth is obtained by calculating disparity information by analyzing the relationship between corresponding pixels in images from different viewpoints. In this application, stereo matching is one of the core components of depth estimation. By constructing an efficient cost volume and combining it with a specific network structure and algorithm, depth information is estimated quickly and accurately from Cassini projection stereo pairs.

[0069] Existing technologies suffer from the following main drawbacks: First, insufficient depth estimation accuracy. Whether based on spherical scanning or epipolar reconstruction, these methods struggle to accurately capture environmental depth details in complex and unfamiliar scenes. Spherical layered scanning methods are limited by the number and uniformity of depth layers and are sensitive to prior information such as minimum depth. Epipolar reconstruction methods suffer from mapping errors and depth discontinuities. Second, low computational efficiency. Most methods exhibit high computational complexity when processing multi-view fisheye images, failing to meet real-time requirements and severely limiting their feasibility in practical applications. Third, weak model generalization ability. Performance drops significantly when facing unknown scenes, making reliable application in various environments difficult. This application aims to address these issues by achieving high-precision, high-efficiency, and highly generalizable full-strength depth estimation through innovative technical solutions.

[0070] This application provides a depth estimation method, apparatus, device, and medium based on a multi-view fisheye camera, relating to the field of computer vision technology. The depth estimation method, apparatus, device, and medium based on a multi-view fisheye camera provided in this application can be applied to terminals, servers, or software running on terminals or servers. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the server can also be a node server in a blockchain network; the software can be an application implementing the depth estimation method based on a multi-view fisheye camera, but is not limited to the above forms.

[0071] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0072] Reference Figure 1 This application provides a depth estimation method based on a multi-view fisheye camera. This method may include, but is not limited to, steps S100 to S150, as follows:

[0073] S100: Acquire fisheye images from corresponding perspectives using multiple fisheye cameras; wherein the field of view of the multiple fisheye cameras covers a 360-degree panorama.

[0074] For example, Figure 2 This is a schematic diagram illustrating the arrangement of four fisheye cameras on the roof of a vehicle, as provided in an embodiment of this application.

[0075] Furthermore, S100 may include the following steps:

[0076] The fisheye images are acquired from corresponding perspectives by multiple fisheye cameras installed on vehicles, robots, or drones.

[0077] S110: Construct a stereo pair of Cassini projections in the overlapping areas of the fields of view in the fisheye images captured by two adjacent fisheye cameras.

[0078] Furthermore, S110 may include the following steps S111 to S112:

[0079] S111: Using a fisheye camera model, each fisheye image is projected onto a unit sphere in a Cartesian coordinate system, and then the projection of each fisheye image is converted from the Cartesian coordinate system to the Cassini spherical coordinate system, thereby mapping each fisheye image onto a Cassini projection image.

[0080] S112: Construct the Cassini projection stereo pair on the Cassini projection image corresponding to the overlapping field of view;

[0081] The Cassini spherical coordinate system is defined as (ρ, φ, θ), where ρ represents the distance between the origin O and point P. Let θ be the angle between line OP and plane yOz, and let θ be the angle between line OP′ and the positive Z-axis, where point P′ is the projection of point P onto plane yOz; the coordinates (x, y, z) of the Cartesian coordinate system and the coordinates of the Cassini spherical coordinate system are... The conversion relationships are as follows:

[0082]

[0083] In the Cassini projection, point P is defined. l and point P r Angular parallax d = φ l -φ r The baseline length is B, and point P is... l Depth ρ to the origin O l Calculated by the following formula:

[0084]

[0085] S120: Perform stereo matching on each stereo pair to obtain a disparity map and a first confidence map for each stereo pair.

[0086] Furthermore, S120 may include the following steps S121 to S123:

[0087] S121: Construct cost volumes for each of the stereo pairs according to the ACVNet method;

[0088] S122: Regularize the relevant stereo matching volume and the cost volume using an hourglass network;

[0089] S123: Upsample the regularized cost volume to obtain the disparity map and the first confidence map for each stereo pair.

[0090] S130: Convert each of the disparity maps into a depth map as the first depth map.

[0091] S140: Convert each of the first confidence maps and each of the first depth maps from the Cassini projection to the equidistant cylindrical projection, and then fuse each of the first depth maps according to each of the first confidence maps to obtain a fused omnidirectional depth map.

[0092] Further, in S140, fusing the first depth maps according to the first confidence maps to obtain the fused omnidirectional depth map may include the following steps S141 to S143:

[0093] S141: Align each of the first depth maps and each of the first confidence maps from different perspectives to the same direction and position by rotation and translation to obtain an aligned second confidence map and second depth map;

[0094] S142: Based on the confidence information of the second confidence map, fuse the individual second depth maps into an initial omnidirectional depth map;

[0095] S143: Using a lightweight fusion network with an encoder-decoder architecture and hop connections, adjust the depth value of the initial omnidirectional depth map according to the adjacent depth information of each location to obtain the fused omnidirectional depth map.

[0096] S150: The fused omnidirectional depth map is refined to eliminate the difference error caused by the conversion from the Cassini projection to the equidistant cylindrical projection and the detail error generated during fusion, so as to obtain the target omnidirectional depth map.

[0097] Furthermore, S150 may include the following steps S151 to S153:

[0098] S151: Sample each location in the depth neighborhood of the fused omnidirectional depth map to generate a depth hypothesis;

[0099] S152: Using the fisheye camera imaging model and the depth assumption, the feature maps of each fisheye image are mapped to an equidistant cylindrical projection, thereby constructing a 4D cost volume.

[0100] S153: The 4D cost volume is regularized using an hourglass network, and then the target omnidirectional depth map is obtained using the softargmin function based on the regularized 4D cost volume.

[0101] As an optional implementation, after S150, the embodiments of this application may further include the following steps:

[0102] S160: Generate navigation information based on the target omnidirectional depth map.

[0103] It is understood that this embodiment can quickly generate accurate navigation information based on the target omnidirectional depth map, thereby providing accurate navigation information for scenarios such as autonomous driving, robot navigation, or drone navigation.

[0104] The following section will provide a detailed introduction and explanation of the solutions in the embodiments of this application, using specific application examples.

[0105] Specifically, refer to Figure 3 This embodiment may include the following solutions:

[0106] 1. Construct the Cassini projection stereo pair.

[0107] Reference Figure 4 This embodiment uses Cassini spherical projection to construct a stereo pair in the overlapping field of view of two adjacent fisheye cameras. The epipolar lines of the Cassini stereo pair are horizontally aligned after projection, which simplifies the stereo matching calculation process and improves the efficiency and accuracy of depth estimation. It should be noted that the steps for constructing the Cassini-projected stereo pair in this embodiment can be replaced by constructing a pinhole-projected stereo pair.

[0108] In the specific implementation, the fisheye image is first projected onto a unit sphere in a Cartesian coordinate system using a fisheye camera model. Then, these Cartesian coordinates are converted to Cassini spherical coordinates, and finally mapped onto the Cassini projected image. The Cassini spherical coordinate system (ρ, φ, θ) is defined, where ρ represents the distance between the origin o and point P. Let θ be the angle between line OP and plane yOz, and let θ be the angle between line OP′ and the positive Z-axis, where point P′ is the projection of point P onto plane yOz. Cartesian coordinates (x, y, z) and Cassini spherical coordinates. The conversion relationships are as follows:

[0109]

[0110] In the Cassini projection, point P is defined. l and point P r Angular parallax d = φ l -φ r The baseline length is B, and point P is...l Depth ρ to the origin O l It can be calculated using the following formula:

[0111]

[0112] This embodiment constructs Cassini stereo pairs through precise coordinate transformation and projection operations, laying the foundation for subsequent stereo matching and depth estimation.

[0113] 2. Lightweight 3D matching.

[0114] This embodiment designs a lightweight stereo matching network containing only 1.8M parameters. The network employs a compact U-Net structure, enabling rapid image feature extraction. Unlike stereo matching methods that directly construct cost volumes with high computational costs and a large amount of redundant information, this embodiment constructs a compact and efficient cost volume based on the ACVNet method, effectively reducing computational overhead. Specifically, a small hourglass network is used to regularize the correlation volume and cost volume. Then, CUBG is applied to upsample the cost volume, obtaining the disparity map and confidence map C for each stereo pair. c Finally, the disparity map is converted into a depth map D. c This serves as a preliminary result for depth estimation.

[0115] 3. Deep integration of multiple perspectives.

[0116] This embodiment proposes a confidence-information-guided multi-view depth fusion module to address depth discontinuities in omnidirectional depth maps. First, depth maps D from different directions are fused... c Confidence graph C c The Cassini projection is converted to an isometric cylindrical projection, and then the depth maps from multiple perspectives are transformed. c Confidence graph C c By aligning them to the same direction and position through rotation and translation, we obtain the alignment confidence map C. u and depth map D u Guided by confidence information, depth maps from multiple perspectives are fused into an initial omnidirectional depth map D. I .

[0117] Initial omnidirectional depth map D I There may be invalid pixels and noise due to alignment issues. In the second step of this module, a lightweight fusion network with an encoder-decoder architecture and skip connections is used to adjust the depth values ​​based on the neighboring depth information at each location, resulting in a fused omnidirectional depth map D. F This will further improve the accuracy of omnidirectional depth maps.

[0118] 4. Refine the omnidirectional depth map.

[0119] This embodiment proposes an omnidirectional depth refinement module to solve the interpolation error generated when correcting fisheye images to Cassini projection and then to equidistant cylindrical projection, as well as the detail blurring caused by the smoothing process of the fusion module.

[0120] This module fuses the omnidirectional depth map D. F Each location is sampled within its depth neighborhood to generate a depth hypothesis d. h Using a fisheye camera imaging model and the depth assumption d h The feature map of the fisheye image is mapped to an equidistant cylindrical projection to construct a small 4D cost volume. Cost regularization is performed through a lightweight hourglass network, and the refined omnidirectional depth map D of the target is obtained using the softargmin function. R This module effectively recovers the detailed information of the omnidirectional depth map, improving the accuracy of the overall depth estimation. For example, Figure 5 This is an example illustration of the result of depth estimation based on a fisheye image.

[0121] In summary, this embodiment includes the following technical solutions:

[0122] 1. Lightweight stereo matching network significantly reduces the number of parameters and computational overhead while maintaining a certain level of accuracy, enabling rapid generation of depth maps and confidence maps.

[0123] 2. Multi-view depth fusion: It uses confidence information to fuse depth maps from multiple perspectives, effectively solving the problem of depth discontinuity caused by directly stitching depth maps from different perspectives, and handling invalid points and noise in omnidirectional depth maps.

[0124] 3. Omnidirectional depth refinement: By sampling from the depth neighborhood and reprojecting, the detail information in the omnidirectional depth map is enhanced, thereby improving the accuracy of the final omnidirectional depth estimation result.

[0125] Beneficial effects:

[0126] High depth estimation accuracy: Experimental results on multiple simulation datasets (such as Urban, OmniHouse, and OmniThings) demonstrate that this embodiment outperforms other existing open-source methods in all evaluation metrics (including MAE, RMSE, SILog, δ1, δ2, and δ3). This embodiment can capture environmental depth details more accurately, providing more reliable depth information for practical applications.

[0127] Fast Depth Estimation: This embodiment achieves real-time omnidirectional depth estimation. When outputting an omnidirectional depth map at a resolution of 640*320, it can achieve a frame rate of 35 FPS on a high-performance device like the RTX TITAN GPU; and on a performance-constrained embedded device like the Jetson AGX Orin, it can achieve a frame rate of 12.3 FPS, demonstrating a significant speed advantage compared to other methods. This embodiment shows significant speed improvements on different datasets while maintaining higher accuracy, effectively solving the problem of low computational efficiency in existing methods, and is more suitable for real-time application scenarios.

[0128] It has excellent generalization ability: When this embodiment is tested on the untrained OmniTown dataset, it can still maintain high accuracy. Compared with other methods, the performance of the latter will drop significantly in new scenarios. This embodiment can better adapt to different environments, reduce the dependence on specific scenarios, broaden its application scope, and has stronger practicality.

[0129] The following will describe a more specific implementation method.

[0130] Specifically, in practical applications, such as the environmental perception system of autonomous vehicles, the method of this embodiment can be deployed on an onboard computing platform. Four fisheye cameras are mounted on the roof of the vehicle; the camera configuration can still be referenced. Figure 2 The network is arranged in a specific layout to ensure comprehensive coverage of the surrounding environment. In terms of hardware, this embodiment can use embedded computing devices such as Jetson AGX Orin, configured according to different performance requirements and cost budgets. For software implementation, this embodiment can build its network model based on the PyTorch framework. First, it is initially trained for 40 epochs on the OmniThing dataset, and then fine-tuned for 30 epochs on a mixed Urban and OmniHouse dataset.

[0131] In practical applications, fisheye cameras in four directions acquire images of the surrounding environment in real time and input them into the method model trained in this embodiment. This method model can quickly output a high-precision all-around depth map and provide accurate environmental perception information.

[0132] Reference Figure 6 This application also provides a depth estimation device based on a multi-view fisheye camera, which can implement the above-mentioned depth estimation method based on a multi-view fisheye camera. The device includes:

[0133] The image acquisition unit is used to acquire fisheye images from corresponding perspectives using multiple fisheye cameras; wherein the field of view of the multiple fisheye cameras covers a 360-degree panorama.

[0134] A stereo pair construction unit is used to construct a Cassini projection stereo pair in the overlapping areas of the field of view in the fisheye images captured by two adjacent fisheye cameras.

[0135] A stereo matching unit is used to perform stereo matching on each stereo pair to obtain a disparity map and a first confidence map for each stereo pair.

[0136] A parallax conversion unit is used to convert each of the parallax maps into a depth map as a first depth map;

[0137] The deep fusion unit is used to convert each of the first confidence maps and each of the first depth maps from the Cassini projection to the equidistant cylindrical projection, and then fuse each of the first depth maps according to each of the first confidence maps to obtain a fused omnidirectional depth map.

[0138] An error processing unit is used to refine the fused omnidirectional depth map to eliminate the difference error caused by the conversion from the Cassini projection to the equidistant cylindrical projection and the detail error generated during fusion, so as to obtain the target omnidirectional depth map.

[0139] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned depth estimation method based on a multi-view fisheye camera. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0141] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0142] Please see Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0143] The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0144] The memory 702 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 702 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701 to execute the depth estimation method based on a multi-view fisheye camera according to the embodiments of this application.

[0145] The input / output interface 703 is used to implement information input and output;

[0146] The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0147] Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704);

[0148] The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0149] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned depth estimation method based on a multi-view fisheye camera.

[0150] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0151] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0152] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0153] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0154] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0156] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0157] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0159] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0161] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for depth estimation based on multi-view fisheye camera, characterized in that, The method comprises the following steps: Obtaining fisheye images of corresponding visual angles through multiple fisheye cameras; wherein the field angles of the multiple fisheye cameras cover 360-degree panorama; Constructing a stereo pair of Cassini projection in the visual field overlap area of the fisheye images photographed by two adjacent fisheye cameras; Stereo matching each stereo pair to obtain a disparity map and a first confidence map of each stereo pair; Converting each disparity map into a depth map as a first depth map; Converting each first confidence map and each first depth map from the Cassini projection to the equirectangular projection, and then fusing each first depth map according to each first confidence map to obtain a fused omnidirectional depth map; Refining the fused omnidirectional depth map to eliminate the difference error generated from the conversion of the Cassini projection to the equirectangular projection and the detail error generated during the fusion, and obtaining a target omnidirectional depth map. 2.The multi-view fisheye camera based depth estimation method of claim 1, wherein, The step of constructing a stereo pair of Cassini projection in the visual field overlap area of the fisheye images photographed by two adjacent fisheye cameras comprises the following steps: Projecting each fisheye image onto a unit sphere in a Cartesian coordinate system using a fisheye camera model, and then converting the projection of each fisheye image from the Cartesian coordinate system to a Cassini spherical coordinate system, and then mapping each fisheye image to a Cassini projection image; Constructing the stereo pair of Cassini projection on the Cassini projection image corresponding to the visual field overlap area; Wherein, the Cassini coordinate system is defined as (p, f, q), wherein p represents the distance between the coordinate origin o and point P, is the angle between straight line OP and plane yOz, q is the angle between straight line OP' and positive Z axis, wherein P' is the projection of point P on plane yOz; the conversion relationship between the coordinates (x, y, z) of the Cartesian coordinate system and the coordinates (p, f, q) of the Cassini coordinate system is as follows: is the angle between straight line OP and plane yOz, q is the angle between straight line OP' and positive Z axis, wherein P' is the projection of point P on plane yOz; the conversion relationship between the coordinates (x, y, z) of the Cartesian coordinate system and the coordinates (p, f, q) of the Cassini coordinate system is as follows: In the Cassini projection, point P is defined. l and point P r Angular parallax d = φ l -φ r The baseline length is B, and point P... l Depth ρ to the origin O l Calculated by the following formula: 3.The multi-view fisheye camera based depth estimation method of claim 1, wherein, The step of stereo matching each stereo pair to obtain a disparity map and a first confidence map of each stereo pair comprises the following steps: Constructing a cost volume according to the ACVNet method for each stereo pair; Regularizing the relevant stereo matching volume and the cost volume respectively using an hourglass network; Up-sampling the regularized cost volume to obtain the disparity map and the first confidence map of each stereo pair. 4.The multi-view fisheye camera based depth estimation method of claim 1, wherein, The step of fusing each first depth map according to each first confidence map to obtain a fused omnidirectional depth map comprises the following steps: Aligning each first depth map and each first confidence map of different visual angles to the same direction and position through rotation and translation to obtain an aligned second confidence map and a second depth map; Fusing each second depth map into an initial omnidirectional depth map according to the confidence information of the second confidence map; Adjusting the depth value of the initial omnidirectional depth map according to the adjacent depth information of each position using a lightweight fusion network with an encoder-decoder architecture and skip connection to obtain the fused omnidirectional depth map. 5.The multi-view fisheye camera based depth estimation method of claim 1, wherein, The step of refining the fused omnidirectional depth map to eliminate the difference error generated from the conversion of the Cassini projection to the equirectangular projection and the detail error generated during the fusion to obtain a target omnidirectional depth map comprises the following steps: Sampling each position of the fused omnidirectional depth map within a depth neighborhood to generate a depth hypothesis; Mapping the feature map of each fisheye image to the equirectangular projection using a fisheye camera imaging model and the depth hypothesis, and then constructing a 4D cost volume; The 4D cost volume is cost-regularized by a sandglass network, and then the target omnidirectional depth map is obtained from the regularized 4D cost volume using a soft argmin function. 6.The multi-view fisheye camera based depth estimation method of claim 1, wherein, The method further comprises the following steps: Generating navigation information according to the target omnidirectional depth map.

7. The multi-view fisheye camera based depth estimation method of any one of claims 1 to 6, characterized in that, The fish-eye images of corresponding visual angles are obtained by a plurality of fish-eye cameras, comprising the following steps: The fish-eye images of corresponding visual angles are obtained by a plurality of fish-eye cameras arranged on a vehicle, a robot or a drone.

8. A depth estimation apparatus based on multi-view fisheye cameras, characterized in that, The device comprises: An image acquisition unit is configured to obtain fish-eye images of corresponding visual angles by a plurality of fish-eye cameras, wherein the field of view angles of the plurality of fish-eye cameras cover 360-degree panorama; A stereo pair construction unit is configured to construct a Cassini projection stereo pair in the visual field overlap area of the fish-eye images captured by adjacent two fish-eye cameras; A stereo matching unit is configured to perform stereo matching on each stereo pair to obtain a disparity map and a first confidence map of each stereo pair; A disparity conversion unit is configured to convert each disparity map into a depth map as a first depth map; A depth fusion unit is configured to convert each first confidence map and each first depth map from the Cassini projection to an equirectangular projection, and then fuse each first depth map according to each first confidence map to obtain a fused omnidirectional depth map; An error processing unit is configured to perform refinement processing on the fused omnidirectional depth map to eliminate the difference error generated by converting from the Cassini projection to the equirectangular projection and the detail error generated during fusion, and obtain a target omnidirectional depth map.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the multi-view fish-eye camera based depth estimation method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the multi-view fish-eye camera based depth estimation method of any one of claims 1 to 7.

Images