A depth prediction method based on infrared and visible light cameras

Abstract

The application discloses a kind of depth prediction methods based on infrared camera and visible light camera.The method uses infrared image and visible light image as input, utilizes backbone network to extract different granularity infrared features and visible light features, then according to depth seed sampling mechanism and multi-view camera model, features are aggregated in 3D space using back projection method, finally, the depth prediction result is output through prediction module, in addition, in order to alleviate the contradiction between prediction accuracy and computing resources, the idea of cascaded optimization is used, and the depth is optimized from coarse granularity to fine granularity step by step.The application improves the accuracy and robustness of depth prediction by improving the traditional binocular depth estimation method according to the characteristics of infrared camera and visible light camera, and realizes real-time depth prediction in vehicle-mounted road scene.

Description

Technical Field

[0001] This invention belongs to the field of multi-sensor fusion sensing, specifically a depth prediction method based on infrared and visible light cameras. Background Technology

[0002] With the continuous development of computer vision and deep learning technologies, the demand for scene depth prediction is increasing. Depth prediction is an important task in the field of computer vision, with applications covering multiple areas such as autonomous driving, intelligent monitoring, and augmented reality.

[0003] In existing depth prediction technologies, commonly used sensors include visible light cameras and lidar. LiDAR is limited by its high cost or size and weight, hindering its widespread adoption. Conversely, visible light cameras are favored for their low cost and portability, making them the basis for most methods. However, visible light camera-based methods have limitations in handling complex and varied scenes, particularly in low-light or complex environments, due to significant performance fluctuations in strong or weak light conditions. Infrared cameras perform well in low-light environments, but their depth prediction for general scenes still faces some challenges. Therefore, combining visible light and infrared cameras to improve the accuracy and robustness of depth prediction has become a current research hotspot.

[0004] Current depth prediction methods have certain limitations when processing multimodal information, especially in integrating visible light and infrared information. Firstly, the fusion of visible light and infrared image information has not been fully studied. Secondly, traditional binocular systems impose strong constraints on the two cameras, such as identical camera parameters, parallel optical axes, and epipolar alignment. However, in practical applications using a combination of visible light and infrared cameras, these constraints are difficult to meet. Therefore, it is necessary to extend the binocular camera model to a multi-view camera model, which presents new challenges for information aggregation. Summary of the Invention

[0005] To address the limitations of existing depth prediction methods in terms of light sensitivity and adaptability to specific scenarios, this invention proposes a depth prediction method based on infrared and visible light cameras. This method aims to combine the advantages of both sensors to improve the performance and adaptability of depth prediction.

[0006] This invention includes the following steps:

[0007] S1: It is known that after time synchronization and data preprocessing, the infrared image X is obtained. infraed and visible light image X visible .

[0008] S2: Transfer the infrared image X infraed and visible light image X viosible Input into the backbone network and extract infrared features F at different granularities. infared，i and visible light characteristics F visible，i , where i represents the granularity symbol, i = 0, 1, ..., N-1.

[0009] S3: Sample to generate depth seed S i .

[0010] S4: Based on the multi-view camera model, the infrared feature F is obtained using the back projection method. infared，i and visible light characteristics F visible，i Aggregate in 3D space to obtain 3D features V i .

[0011] S5: Output the current stage depth prediction result via the prediction module. i Depth N-1 This is the final depth prediction result.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0013] This invention uses infrared and visible light images as input, extracts infrared and visible light features of different granularities using a backbone network, and then aggregates these features in 3D space using a back-projection method based on a depth seed sampling mechanism and a multi-view camera model. Finally, the prediction module outputs the depth prediction result, improving the accuracy and robustness of depth prediction. Furthermore, to alleviate the conflict between prediction accuracy and computational resources, a cascaded optimization approach is adopted, progressively optimizing from coarse-grained depth to fine-grained depth, thus improving the real-time performance of depth prediction. Attached Figure Description

[0014] Figure 1 This is a network framework diagram for depth prediction based on infrared and visible light cameras;

[0015] Figure 2 This is a flowchart for depth prediction based on infrared and visible light cameras;

[0016] Figure 3 These are schematic diagrams of a binocular camera model and a multi-view camera model;

[0017] Figure 4 This is a schematic diagram illustrating the depth prediction effect of the present invention. Detailed Implementation

[0018] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0019] like Figure 1 and Figure 2 As shown in the embodiments of this application, a depth prediction method based on an infrared camera and a visible light camera is disclosed, which includes the following steps:

[0020] S1: Given that after time synchronization and data preprocessing, the infrared image X... infraed Visible light image X visible ;

[0021] S2: Transfer the infrared image X infraed and visible light image X visible Input the designed backbone network, such as Figure 1 As shown in (a), infrared features F at different granularities are extracted. infared，i and visible light characteristics F visible，i Where i represents the granularity symbol, i = 0, 1, ..., N-1, as follows:

[0022] S2.1: Set the downsampling rate downsample_rate i Referring to the feature pyramid structure, feature maps of different granularities for each single mode are extracted.

[0023] S2.2: Set the fuse_rate i Common feature extraction is achieved by using a method that shares some convolutional kernel channels;

[0024] S3.2: Using the two strategies in S2.1 and S2.2, extract infrared features F at different granularities. infared，i and visible light characteristics F visible，i ;

[0025] S3: Sample to generate depth seed S i ,like Figure 1 As shown in (b), the details are as follows:

[0026] S3.1: If this granularity is the initial granularity, i.e., i = 0, then set the sampling interval of the depth seed as follows: [Range] min Range max ]:

[0027] Range min (p, q) = D min

[0028] Range max (p, q) = Dmax

[0029] Where p and q represent indices, and D... min D max These represent the minimum and maximum predicted depths, respectively.

[0030] S3.2: If the granularity is not the initial granularity, i.e., i > 0, then set the sampling interval of the depth seed as follows: [Range] min Range max ]:

[0031]

[0032] Center(p, q) = S i-1 (index center (p, q), p, q)

[0033]

[0034]

[0035] Where k, p, and q represent indices, Index center Depth′ represents the center index of the interval. i-1 This represents the depth result of the previous level, which is consistent with the current granularity. The depth prediction result is derived from the previous level's granularity. i-1 S is generated by interpolation. i-1 This represents the depth seed of the previous granularity level, where Center represents the center of the interval, and Interval... i-1 Indicates the sampling interval of the previous granularity;

[0036] S3.3: Using uniform sampling, generate the depth seed S as follows: i :

[0037]

[0038]

[0039] Where k, p, and q represent indices, and Range min Range max K represents the lower and upper limits of the sampling interval, respectively. i Interval represents the number of depth seed samples. i Indicates the sampling interval;

[0040] S4: Based on the multi-view camera model, such as Figure 3 As shown in (b), the infrared feature F is projected using the back projection method. infared，i and visible light characteristics F visible，iAggregate in 3D space to obtain 3D features V i ,like Figure 1 As shown in (c), the details are as follows:

[0041] S4.1: Calculate the infrared 3D feature V as shown below. infrared，i :

[0042] V infrared，i (k, p, q) = F infared，i (p, q)

[0043] Where k, p, and q represent indices, and F... infared，i Infrared features at the current granularity extracted from the backbone network;

[0044] S4.2: Calculate the feature P of the projection matrix as shown below. i :

[0045]

[0046]

[0047]

[0048]

[0049] P i =(Intrinsic infrared，i ·E infared Intrinsic visible，i ·E visible ) -1

[0050] Among them W i H i These represent the width and height of the depth map at the current granularity, respectively; W and H represent the width and height of the depth map at the initial granularity, respectively; and C... infared C visible E represents the intrinsic parameter matrix of the infrared camera and the intrinsic parameter matrix of the visible light camera, respectively. infared E visible These represent the extrinsic parameter matrices of the infrared camera and the visible light camera, respectively.

[0051] S4.3: Combining the multi-view camera model and the back projection method, calculate the projection points as shown below:

[0052] U(:,:,q)=q

[0053] V(:,p,:)=p

[0054]

[0055] Where k, p, and q represent indices, U and V represent the original pixel coordinates, and S... i Represents the original depth seed, U′ and V′ represent the projected pixel coordinates, and S i ′ represents the depth seed after conversion;

[0056] S4.4: Visible light features F extracted from the backbone network at the current granularity visible，i Given the projected pixel coordinates U′ and V′, the visible light 3D feature V is calculated using interpolation methods. visible，i .

[0057] S4.5: For V infrared，i With V visible，i The variance is calculated point by point to obtain the 3D feature V. i ;

[0058] S5: Output the current stage depth prediction result via the prediction module. i Depth N-1 For the final depth prediction result, such as Figure 1 As shown in (d), the details are as follows:

[0059] S5.1: Utilizing the feature pyramid structure formed by 3D convolutional modules, the 3D feature V... i Further feature extraction and channel compression are performed to form a new 2D feature V. i ′;

[0060] S5.2: Calculate the depth prediction value for the current grain size as shown below:

[0061]

[0062]

[0063] Where k, p, q represent indices, and W i S represents the weight. i This represents a depth seed.

[0064] like Figure 3 As shown, this illustrates the differences between a stereo camera model and a multi-view camera model. Figure 3 The binocular camera model shown in (a) needs to meet the following conditions: consistent camera parameters, parallel optical axes, epipolar alignment, etc., which are quite restrictive. Figure 3 The multi-view camera model shown in (b) can be established even under conditions of inconsistent camera parameters, non-parallel optical axes, and misaligned epipolar lines, and has higher universality for practical applications.

[0065] like Figure 4As shown, in order to verify the effectiveness of the invention, ordinary in-vehicle scenario data was used for verification. Figure 4 (a) and Figure 4 (b) shows the infrared and visible light images of the same scene, respectively. Figure 4 Image (c) shows a sparse depth map generated using lidar point cloud projection, used for comparative evaluation of prediction results. Figure 4 (d) shows the depth prediction performance of this invention. It can be observed that the depth prediction results for targets such as vehicles, utility poles, and traffic lights are relatively good. However, the depth prediction results for extremely distant points such as the sky are poor. This is because it is difficult to obtain the true depth value for such points, making it impossible to effectively supervise the model during the training phase. However, for most downstream tasks, the depth value of such points is not important, and such points are often filtered out by post-processing algorithms.

[0066] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A depth prediction method based on infrared and visible light cameras, characterized in that, Includes the following steps: S1: It is known that after time synchronization and data preprocessing, an infrared image is obtained. and visible light images ; S2: Transfer infrared image and visible light images Input the data into the backbone network and extract infrared features at different granularities. and visible light characteristics ,in Indicates the granularity identifier symbol. ; S3: Sample to generate depth seed ; S4: Based on the multi-view camera model, use the back projection method to extract infrared features. and visible light characteristics Aggregate in 3D space to obtain 3D features. ; S5: Output the current stage depth prediction result through the prediction module. , This is the final depth prediction result; Specifically, S3 is: Different sampling intervals are set according to the granularity; A depth seed is generated using a uniform sampling method; The S4 is implemented through the following sub-steps: S4.1: Calculate infrared 3D features : in Indicates an index. Infrared features at the current granularity extracted from the backbone network; S4.2: Calculate the characteristics of the projection matrix : in , These represent the width and height of the depth map at the current granularity, respectively. , These represent the width and height of the depth map at the initial granularity, respectively. , These represent the intrinsic parameter matrices of the infrared camera and the visible light camera, respectively. , These represent the extrinsic parameter matrices of the infrared camera and the visible light camera, respectively. S4.3: Calculate the projection points by combining the multi-view camera model and the back-projection method: in Indicates an index. , Represents the original pixel coordinates. Represents the original depth seed. , Indicates the pixel coordinates after projection. Indicates the depth seed after conversion; S4.4: Visible light features at the current granularity extracted from the backbone network. and projected pixel coordinates , Using interpolation methods, 3D features of visible light are calculated. ; S4.5: To and Calculate the variance point by point to obtain 3D features ; S5 is implemented through the following sub-steps: S5.1: Utilizing the feature pyramid structure formed by 3D convolutional modules to process 3D features Further feature extraction and channel compression are performed to form new 2D features. ; S5.2: Calculate the depth prediction value for the current grain size: in Indicates an index. Indicates weight, This represents a depth seed.

2. The depth prediction method based on an infrared camera and a visible light camera according to claim 1, characterized in that, The S2 is achieved through the following sub-steps: S2.1: Set the downsampling rate, refer to the feature pyramid structure, and extract feature maps of different granularities for each single mode; S2.2: Set the fusion rate and use shared partial convolutional kernel channels to achieve common feature extraction; S2.3: Extract infrared features of different granularities using S2.1 and S2.2 respectively. and visible light characteristics .

3. The depth prediction method based on an infrared camera and a visible light camera according to claim 1, characterized in that, If this particle size is the initial particle size, that is Then set the sampling interval of the depth seed as follows: : in Indicates an index. , These represent the minimum and maximum predicted depths, respectively. If this particle size is not the initial particle size, i.e. Then set the sampling interval of the depth seed as follows: : in Indicates an index. Indicates the center index of the interval. This indicates the depth result of the previous level that is consistent with the current granularity, based on the depth prediction result of the previous level of granularity. Generated by interpolation method This indicates the depth seed of the next higher particle size. Indicates the center of the interval. This indicates the sampling interval for the previous granularity.

4. The depth prediction method based on an infrared camera and a visible light camera according to claim 3, characterized in that, Depth seeds are generated using a uniform sampling method as follows: : in Indicates an index. , These represent the lower and upper limits of the sampling interval, respectively. Indicates the number of depth seed samples. Indicates the sampling interval.

Images