Binocular fisheye stereo matching method constructed by fusing monocular depth prior and multi-scale spherical projection cost

Abstract

The invention discloses a binocular fisheye stereo matching method constructed by fusing monocular depth prior and multi-scale spherical projection cost. The binocular fisheye stereo matching method comprises the following specific steps: S1, image input and feature extraction; s2, monocular depth priori estimation; s3, constructing a cost matrix of multi-scale spherical projection scanning; s4, the deep attention network is matched; s5, performing initial parallax estimation; s6, obtaining a final fine disparity map based on a ConvGRU disparity optimization network, and the method can accurately and efficiently carry out stereo matching without image correction, and improves the matching precision and enhances the robustness of an algorithm to a distorted image by introducing monocular depth estimation prior and a multi-scale spherical projection scanning strategy; the method also avoids the quality loss caused by image correction, retains the original view field information of the fisheye image, gives consideration to the precision and efficiency through a multi-scale spherical scanning strategy, adapts to fisheye distortion characteristics, introduces an attention mechanism and a GRU structure, and improves the expression capability of a parallax estimation network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, and particularly relates to a binocular fisheye stereo matching method fusing monocular depth prior and multi-scale spherical projection cost construction. BACKGROUND

[0002] Fisheye cameras have a wide range of applications in automatic driving, robots, security monitoring and other fields due to their ultra-wide field of view (FoV) characteristics. However, the large field of view brought by fisheye lenses also brings serious image distortion. Especially, the image edge part is highly stretched and distorted, and the center area is relatively compressed. This strong distortion will cause the traditional stereo matching algorithm based on pinhole model or epipolar geometry to fail.

[0003] Existing methods usually rectify fisheye images before stereo matching, but the rectification process will significantly reduce the image quality, especially in the edge area, serious spatial sampling degradation occurs, the advantage of the large field of view of the fisheye lens itself is lost, and the computational complexity is increased.

[0004] Therefore, there is an urgent need for a stereo matching algorithm that adapts to the distortion characteristics of fisheye images and fully utilizes the original field of view information to improve matching accuracy and efficiency. SUMMARY

[0005] In view of the above technical problems, the present application provides a binocular fisheye stereo matching method fusing monocular depth prior and multi-scale spherical projection cost construction, which can accurately and efficiently perform stereo matching without image rectification. By introducing monocular depth estimation prior and multi-scale spherical projection scanning strategy, the matching accuracy is improved and the robustness of the algorithm to distorted images is enhanced.

[0006] To achieve the above purpose, the technical scheme of the present application is as follows:

[0007] The binocular fisheye stereo matching method fusing monocular depth prior and multi-scale spherical projection cost construction has the following specific steps:

[0008] S1. Image input and feature extraction: input the left and right fisheye images into the pyramid feature extraction backbone network to extract multi-scale semantic features;

[0009] S2. Monocular depth prior estimation: input the left fisheye image into the monocular depth estimation network to predict its depth prior, providing geometric constraints for subsequent matching;

[0010] S3. Multi-scale spherical projection scanning cost matrix construction: based on the principle of spherical projection, a multi-scale angular interval method is used to construct the disparity candidate set, and the cost volume is constructed on the spherical surface to avoid the failure of traditional scanning lines in fisheye images;

[0011] S4. Deep attention network matching: the attention mechanism is introduced to weight the cost matrix, which improves the feature fusion and matching accuracy;

[0012] S5. Initial disparity estimation: output the preliminary disparity map;

[0013] S6. ConvGRU-based disparity optimization network: using the gated recurrent structure ConvGRU to combine monocular depth prior and initial disparity map, the temporal optimization iteration of disparity is carried out to obtain the final fine disparity map.

[0014] Further, the specific content of step S1 is that the left and right fisheye images I L and I R are input into the pyramid feature extraction network module F(·) composed of multiple scales to obtain multi-scale semantic feature representation:

[0015]

[0016] Wherein, s∈1, 2,..., S represents different scale levels of the pyramid.

[0017] Further, the specific content of step S2 is that the left fisheye image I L is input into the monocular depth estimation network D(·) to obtain the depth prior Z L :

[0018] Z L =D(I L ),

[0019] And the depth map is only used to provide guidance information for the subsequent disparity optimization network, and is not used for initial disparity calculation.

[0020] Further, the specific content of step S3 is as follows:

[0021] 3.1, projection mapping of fisheye image space to spherical direction

[0022] Given any image pixel (u, v), let its relative image center coordinates be:

[0023] Δx = u-u0, Δy = v-v0,

[0024] Let the focal length f, and use the equidistant fisheye model to represent the projection angle φ:

[0025]

[0026] The unit spherical direction vector d(u, v) is calculated as:

[0027]

[0028] 3.2, constructing spherical scanlines and candidate depth set

[0029] To adapt to different depth ranges in multi-scale scenes, a set of uniformly distributed depth candidates is constructed at each scale s:

[0030]

[0031] where α s , β s are the minimum and maximum depth ranges at scale s, respectively, and N s is the number of depth levels;

[0032] 3.3 Cost volume generation based on spherical back-projection

[0033] For each image pixel (u, v) and each depth hypothesis The spatial coordinates of the point in the left camera coordinate system are calculated:

[0034]

[0035] Then, the corresponding right eye coordinates are obtained through the transformation T LR = [R | t] between the left and right cameras:

[0036]

[0037] PR (i) is back-projected to the right fisheye image coordinate system and mapped to the pixel coordinates through the fisheye model function Πfisheye(·):

[0038]

[0039] The cost volume C s (u, v, i) is constructed at each scale s by calculating the feature similarity of the left and right images:

[0040]

[0041] The cost volumes C s at multiple scales will be fused in subsequent steps.

[0042] Further, the specific content of step S4 is that the constructed cost volume is input into the fusion network A(·), and the attention mechanism is used to enhance the response value of the disparity hypothesis, obtaining the matching enhanced cost volume:

[0043]

[0044] Further, the specific content of step S5 is that the Soft-Argmin operation is used to obtain the cost vector The weighted aggregation is performed to obtain an initial disparity map:

[0045]

[0046] wherein, is the disparity value corresponding to the current depth hypothesis, and the mapping between disparity and depth depends on the camera baseline and the spherical geometry model.

[0047] Further, the specific content of the step S6 is to input the initial disparity map left feature map and the depth prior map into the refining network module GRURefine(·):

[0048]

[0049] The module is based on the ConvGRU structure, realizes dynamic guided update of the spatial disparity field, improves the estimation accuracy in the weak texture and serious occlusion area, and obtains the final disparity.

[0050] The beneficial effects of the present application are:

[0051] The present application can accurately and efficiently perform stereo matching without image rectification, improves the matching accuracy and enhances the robustness of the algorithm to distorted images by introducing monocular depth estimation prior and multi-scale spherical projection scanning strategy;

[0052] The present application avoids the quality loss caused by image rectification, retains the original field of view information of the fisheye image, improves the accuracy of the initial disparity by combining the monocular depth estimation prior, considers the accuracy and efficiency by the multi-scale spherical scanning strategy, adapts to the fisheye distortion characteristics, and introduces the attention mechanism and the GRU structure to enhance the expression ability of the disparity estimation network. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a processing flow schematic diagram of the present application. DETAILED DESCRIPTION

[0054] The technical solutions of the present application will be described clearly and completely in combination with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0055] Embodiment 1

[0056] In combination with the drawings, Figure 1 the implementation steps of the method of the present application are as follows:

[0057] Step 1, fisheye image preprocessing and feature extraction

[0058] input left and right fisheye images I L with I R , respectively, into a pyramid feature extraction network module F(·) composed of multiple scales to obtain multi-scale semantic feature representations:

[0059]

[0060] where s∈1, 2,..., S represents different scale levels of the pyramid.

[0061] Step 2, monocular depth estimation module (for prior guidance)

[0062] left fisheye image I L is input into the monocular depth estimation network D(·) to obtain depth prior Z L :

[0063] Z L = D(I L ),

[0064] This depth map is only used to provide guidance information for the subsequent disparity optimization network and is not used for initial disparity calculation.

[0065] Step 3, multi-scale spherical cost volume construction

[0066] 3.1 Projection mapping of fisheye image space to spherical direction

[0067] Given any image pixel (u, v), let its relative image center coordinates be:

[0068] Δx = u - u0, Δy = v - v0,

[0069] Let the focal length f, and use the equidistant fisheye model to represent the projection angle φ:

[0070]

[0071] The unit spherical direction vector d(u, v) is calculated as:

[0072]

[0073] 3.2 Construction of spherical scan lines and candidate depth set

[0074] To adapt to different depth ranges in multi-scale scenes, the present application constructs a set of uniformly distributed depth candidates at each scale s:

[0075]

[0076] where α s ,β smin and max depth range under scale s, respectively, N s is the number of depth layers.

[0077] 3.3 Cost volume generation based on spherical de-projection

[0078] For each image pixel (u, v) and each depth hypothesis Compute the spatial coordinates of this point in the left camera coordinate system:

[0079]

[0080] Then get the corresponding right eye coordinate through the transformation T LR = [R | t] between left and right cameras:

[0081]

[0082] Map the PR (i) to the right fisheye image coordinate system through the fisheye model function Πfisheye(·) to get the pixel coordinate:

[0083]

[0084] The cost volume C s (u, v, i) is constructed through the feature similarity of left and right images at each scale s:

[0085]

[0086] The cost volumes C s at multiple scales will be fused in the following steps.

[0087] Step 4, attention-guided matching enhancement network

[0088] The constructed cost volume is input into the fusion network A(·), and the attention mechanism is used to enhance the response value of the disparity hypothesis, to get the matching enhanced cost volume:

[0089]

[0090] Step 5. Initial disparity estimation (not dependent on depth map)

[0091] The cost vector of each pixel position (u, v) is aggregated by using the Soft-Argmin operation: to get the initial disparity map:

[0092]

[0093] where The mapping between disparity and depth depends on the camera baseline and the spherical geometry model for the current depth hypothesis corresponding disparity value.

[0094] Step 6. ConvGRU disparity optimization network

[0095] The initial disparity map is input into the refinement network module GRURefine(·): The left feature map and the depth prior map are input into the refinement network module GRURefine(·):

[0096]

[0097] The module is based on the ConvGRU structure, realizes dynamic guided update of the spatial disparity field, and improves the estimation accuracy in the weak texture and serious occlusion area. Finally, the final disparity is obtained.

[0098] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A binocular fisheye stereo matching method that integrates monocular depth prior and multi-scale spherical projection cost is constructed. The specific steps are as follows: S1. Image Input and Feature Extraction: The left and right fisheye images are fed into the pyramid feature extraction backbone network to extract multi-scale semantic features. S2. Monocular depth prior estimation: The left fisheye image is fed into the monocular depth estimation network to predict its depth prior, providing geometric constraints for subsequent matching. S3. Cost Matrix Construction for Multi-Scale Spherical Projection Scanning: Based on the principle of spherical projection, a multi-scale angular interval approach is used to construct a disparity candidate set. The cost volume is constructed on the sphere, avoiding the failure of traditional scan lines in fisheye images. S4. Deep Attention Network Matching: Introducing an attention mechanism to weight the cost matrix to improve feature fusion and matching accuracy; S5. Initial disparity estimation: output preliminary disparity map; S6. ConvGRU-based disparity optimization network: Using the gated recurrent structure ConvGRU combined with the monocular depth prior and the initial disparity map, the disparity is optimized iteratively in time sequence to obtain the final refined disparity map.

2. The binocular fisheye stereo matching method based on the fusion of monocular depth prior and multi-scale spherical projection cost construction according to claim 1 is characterized in that The specific content of step S1 is to input the left and right fisheye images I L with I R , are fed into the pyramid feature extraction network module F(·) composed of multiple scales to obtain multi-scale semantic feature representation: Among them, s∈1,2,...,S represents different scale levels of the pyramid.

3. The binocular fisheye stereo matching method based on the fusion of monocular depth prior and multi-scale spherical projection cost construction according to claim 2 is characterized in that The specific content of step S2 is: the left fisheye image I L At the same time, input the monocular depth estimation network D(·) to obtain the depth prior Z L :Z L =D(I L ).

4. The binocular fisheye stereo matching method based on the fusion of monocular depth prior and multi-scale spherical projection cost construction according to claim 3 is characterized in that The specific content of step S3 is as follows: 3.1, Projection mapping of fisheye image space to spherical direction Given any image pixel (u, v), let its coordinates relative to the image center be: Δx=u-u0, Δy=v-v0, Assume the focal length f and use the equidistant fisheye model to represent the projection angle φ: The direction vector d(u,v) on the unit sphere is calculated as: 3.2, Constructing spherical scan lines and candidate depth sets To adapt to different depth ranges in multi-scale scenes, a set of evenly distributed depth candidates is constructed at each scale s: Among them, α s , β s are the minimum and maximum depth ranges under scale s, N s is the depth layer number; 3.3 Cost Volume Generation Based on Spherical Backprojection For each image pixel (u, v) and each depth hypothesis Calculate the spatial coordinates of the point in the left camera coordinate system: Then, through the transformation T between the left and right cameras LR =[R|t] to get the corresponding right eye coordinates: PR (i) Back-project to the right fisheye image coordinate system and map the pixel coordinates using the fisheye model function Πfisheye(·): Cost Body C s (u,v,i) is constructed by calculating the feature similarity of the left and right images at each scale s: Cost volume C at multiple scales s Will be integrated in subsequent steps.

5. The binocular fisheye stereo matching method based on the fusion of monocular depth prior and multi-scale spherical projection cost construction according to claim 4 is characterized in that The specific content of step S4 is to input the constructed cost volume into the fusion network A(·), use the attention mechanism to weight the response value of the disparity hypothesis, and obtain the matched and enhanced cost volume:

6. The binocular fisheye stereo matching method according to claim 5, characterized in that: The specific content of step S5 is to use the Soft-Argmin operation to calculate the cost vector of each pixel position $(u,v)$ Perform weighted aggregation to obtain the initial disparity map: in, is the disparity value corresponding to the current depth hypothesis. The mapping between disparity and depth depends on the camera baseline and spherical geometry model.

7. The binocular fisheye stereo matching method according to claim 6, characterized in that The initial disparity map Left feature map And the depth prior graph is sent to the refinement network module GRURefine(·): This module is based on the ConvGRU structure and implements dynamic guided updates of the spatial disparity field, improving the estimation accuracy in areas with weak texture and severe occlusion to obtain the final disparity.

Images