A depth estimation method based on multi-domain feature fusion

Abstract

This invention belongs to the field of depth estimation technology, specifically relating to a depth estimation method based on multi-domain feature fusion. It includes the following steps: First, by adjusting the focal length of the acquisition device to change the focal plane, a multi-focus image sequence covering the entire depth of field is obtained; second, the input multi-focus image sequence undergoes preliminary focus feature extraction using a focus extraction module to capture the spatial correlation between image sequences; based on this, a focus recognition module performs multi-scale enhancement and depth cue extraction on the features; then, in the intermediate layer, a multi-scale processing module, a frequency domain processing module, and a fractional domain processing module are simultaneously introduced to perform more accurate focus feature extraction, and the fractional domain features are used to adjust the multi-domain feature fusion of time-domain and frequency-domain features; finally, a depth estimation map is obtained after depth regression. This invention, by extracting and fusing features from multiple domains, introduces richer time-frequency domain modeling capabilities, which can significantly improve the accuracy of depth estimation for components in fields such as integrated circuits and precision manufacturing.

Description

Technical Field

[0001] This invention belongs to the field of depth estimation in computer vision, and specifically relates to a depth estimation method based on multi-domain feature fusion. Background Technology

[0002] Depth estimation, a core component of 3D reconstruction, aims to perceive the distance from each pixel in an image to the camera and infer the geometric structure of the scene. This task has significant applications in fields such as healthcare, industrial manufacturing, and augmented reality.

[0003] Existing depth estimation methods can be broadly categorized into monocular depth estimation and multicular depth estimation based on the number of visual sensors. Monocular depth estimation infers depth from a single image, enabling a mapping from 2D images to depth information. While these methods have low hardware dependency, they can only infer relative depth. Existing methods primarily rely on learning prior information about the scene from large-scale data, exhibiting inherent limitations such as scale ambiguity and geometric prior dependence, which restricts the model's generalization ability in cross-domain environments and high-precision applications. To overcome this bottleneck, multicular depth estimation introduces stereo geometric constraints and then accurately calculates camera calibration parameters, transforming the problem of solving depth information into a disparity calculation problem. This resolves scale uncertainty and enables the acquisition of more accurate depth. However, in weakly textured regions lacking sufficient feature points, matching accuracy significantly decreases.

[0004] Based on the above analysis, depth estimation of multi-focus image sequences can effectively overcome the lack of prior information in monocular depth estimation and the matching problem in multi-view depth estimation. The implementation paths of depth estimation of multi-focus image sequences can be divided into two categories: model-driven methods based on mathematical operators and data-driven methods based on deep learning. These two approaches differ significantly in their implementation paths and key features. Model-driven methods estimate 3D structures by analyzing feature correspondences and geometric constraints in a series of 2D image sequences, focusing primarily on two core steps: focus measurement models and depth map restoration. They aim to calculate depth information through pixel processing. However, lacking supplementary prior information, their reconstruction performance improvement is limited. Furthermore, due to the varying imaging principles and noise interference in different scenarios, they require design tailored to specific application domains, lacking scenario adaptability. Data-driven methods, based on deep learning technology, utilize the feature learning capabilities of neural networks to establish a mapping relationship between focus information and depth information. They possess a certain degree of anti-interference capability and robustness, and are computationally accurate and efficient. Currently, most depth estimation methods based on multi-focus image sequences employ purely temporal or purely frequency domain feature extraction methods. While temporal methods can capture local features of multi-focus images at multiple scales, they neglect global structural information. On the other hand, frequency-domain methods, although capable of capturing global structural information, have shortcomings in processing non-stationary signals caused by interactions such as object motion and complex occlusion in real-world scenes, i.e., defocus blur caused by non-ideal frequency-domain low-pass filtering.

[0005] In summary, the main challenges of existing data-driven multi-focus image sequence depth estimation techniques are that temporal domain models lack the ability to capture global structure, while frequency domain methods cannot effectively extract scene edge details. How to effectively integrate the advantages of these two types of methods, and significantly improve the accuracy and robustness of depth estimation in complex scenes by introducing richer time-frequency domain modeling capabilities, thereby giving the network better anti-interference capabilities, remains a technical problem to be solved. Summary of the Invention

[0006] To overcome the shortcomings of existing solutions, the purpose of this invention is to provide a depth estimation method based on multi-domain feature fusion.

[0007] The technical solution adopted in this invention is: a depth estimation method based on multi-domain feature fusion, comprising the following steps:

[0008] Step 1: By adjusting the focal length of the acquisition device to change the focal plane, images captured at different focal lengths are arranged in the order of capture to obtain a multi-focus image sequence covering the entire depth of field. , This represents the sequence number, and the size of a single frame image is... , Indicates the height and width of a single image. Indicates the number of channels in an image;

[0009] Step 2: The multifocus image sequence obtained in Step 1... The input is fed into the focus extraction module, where the spatial correlation between image sequences is captured using equation (1) to obtain the focus features. ,

[0010] (1)

[0011] in For the focused extraction module, its calculation method is shown in equation (2).

[0012] (2)

[0013] in For the input tensor, This is a 3D convolution operation. For batch normalization operations, It is a non-linear activation function;

[0014] Step 3: Apply the focusing features obtained in Step 2 Multi-scale enhancement and depth cue extraction are performed using equation (3) to obtain the feature volume. ,

[0015] (3)

[0016] in For the focused area identification module, its calculation method is shown in equation (4).

[0017] (4)

[0018] in For the input tensor, This is a 3D convolution operation. It is a non-linear activation function;

[0019] Step 4: Calculate the feature volume obtained in Step 3. Multi-scale temporal feature extraction is performed using equation (5) to obtain feature representations at different scales. ,

[0020] (5)

[0021] in As a multi-scale processing module, its calculation method is shown in equation (6).

[0022] (6)

[0023] in For the input tensor, This is a 3D convolution operation, with kernel sizes of [sizes to be filled in]. , For pooling operations, the pooling scales are respectively Then, the features at different scales are represented. The temporal feature representation is obtained through equation (7). ,

[0024] (7)

[0025] in The multi-scale feature fusion module is calculated as shown in equation (8).

[0026] (8)

[0027] in, For the input tensor, This is a 3D convolution operation. For splicing operations;

[0028] Step 5: Convert the feature volume obtained in Step 3 into a single volume. By converting to the frequency domain using equation (9), the frequency domain characteristics are obtained. ,

[0029] (9)

[0030] in It is a learnable adaptive weight matrix. This represents element-wise matrix multiplication. The result is a Fourier transform, and then the soft thresholding result is multiplied by the Fourier transform output using equation (10) to obtain the data representation. ,

[0031] (10)

[0032] in This is a soft threshold operation. This represents element-wise matrix multiplication. Then, the processed features are transformed from the frequency domain to the spatial domain by performing an inverse Fourier transform using equation (11), resulting in the spatial representation. ,

[0033] (11)

[0034] in For the inverse Fourier transform, the feature volume obtained in step 3 is finally transformed by equation (12). Spatial representation By performing residual connections on the features, a frequency domain feature representation is obtained. ,

[0035] (12)

[0036] Where + indicates feature residual connection;

[0037] Step 6: Convert the feature volume obtained in Step 3 into a single volume. By transforming to the fractional domain using equation (13), we obtain its fractional domain characteristics. ,

[0038] (13)

[0039] in It is a learnable adaptive weight matrix. This represents element-wise matrix multiplication. For order Fractional Fourier transform, order The data representation is obtained through adaptive learning of the model, and then the soft threshold result is multiplied by the output of the fractional Fourier transform using equation (14). ,

[0040] (14)

[0041] in This is a soft threshold operation. This represents element-wise matrix multiplication, followed by an inverse fractional Fourier transform using equation (15). The processed features are transformed from the frequency domain to the spatial domain to obtain the spatial representation. ,

[0042] (15)

[0043] in For order The inverse fast fractional Fourier transform is then performed, and finally the feature volume obtained in step 3 is expressed by equation (16). Spatial representation Residual connections are performed on the features to obtain fractional domain feature representations. ,

[0044] (16)

[0045] Where + indicates feature residual connection;

[0046] Step 7: Represent the time-domain features obtained in steps 4, 5, and 6. Frequency domain feature representation and fractional domain feature representation By inputting Equation (17) into the multi-domain feature fusion module, a multi-domain feature representation is obtained. ,

[0047] (17)

[0048] in The multi-domain feature fusion module is calculated as shown in equation (18).

[0049] (18)

[0050] in This is a 3D convolution operation. For splicing operations;

[0051] Step 8: Represent the multi-domain features obtained in Step 7 After performing nonlinear normalization, the result is weighted and summed with the preset focal length parameter, and the final depth estimation result is output through equation (19). ,

[0052] (19)

[0053] in For activation function, This represents element-wise matrix multiplication. This is the preset focal length parameter.

[0054] Compared with the prior art, the present invention has the following advantages:

[0055] (1) The depth estimation method based on multi-domain feature fusion proposed in this invention combines time domain, frequency domain and fractional domain information, introduces richer time and frequency domain modeling capabilities, is more sensitive to low texture or uniform regions, and can effectively capture global features and local edge details of the image, thereby reducing the boundary blurring phenomenon in the depth estimation results.

[0056] (2) The depth estimation method based on multi-domain feature fusion proposed in this invention utilizes the characteristic that the fractional domain lies between the time and frequency domains to construct a novel multi-domain feature fusion module. While enhancing important features, it also achieves semantic alignment of cross-domain features, which helps to maintain the clarity of depth boundaries and the consistency of the overall shape in complex scenes. Especially in the fields of integrated circuits and precision manufacturing, it can effectively achieve noise suppression and improve the accuracy of depth estimation in weak texture regions. Attached Figure Description

[0057] Figure 1 This is a flowchart of a depth estimation method based on multi-domain feature fusion according to the present invention;

[0058] Figure 2This is a network diagram illustrating a depth estimation method based on multi-domain feature fusion according to the present invention.

[0059] Figure 3 The first step in Embodiment 1 of this invention involves 15 samples from a multi-focus image sequence on the FlyingThings3D dataset.

[0060] Figure 4 This is a depth map of the sample obtained in step 8 of Embodiment 1 of the present invention;

[0061] Figure 5 This is the depth map of the sample obtained using the AiF-net method.

[0062] Figure 6 This paper compares the performance metrics of the AiF-net method and the method of this invention on the FlyingThings3D dataset. Detailed Implementation

[0063] Example 1

[0064] like Figure 1 , Figure 2 As shown in this embodiment, a depth estimation method based on multi-domain feature fusion includes the following steps:

[0065] Step 1: By adjusting the focal length of the acquisition device to change the focal plane, images captured at different focal lengths are arranged in the order of capture to obtain a multi-focus image sequence covering the entire depth of field. , This represents the sequence number, and the size of a single frame image is... , Indicates the height and width of a single image. This represents the number of channels in an image, as shown in the implementation example. Figure 3 As shown, the sequence number Image high Image width and image channels ;

[0066] Step 2: The multifocus image sequence obtained in Step 1... The input is fed into the focus extraction module, where the spatial correlation between image sequences is captured using equation (1) to obtain the focus features. ,

[0067] (1)

[0068] in For the focused extraction module, its calculation method is shown in equation (2).

[0069] (2)

[0070] in For the input tensor, This is a 3D convolution operation. For batch normalization operations, It is a non-linear activation function;

[0071] Step 3: Apply the focusing features obtained in Step 2 Multi-scale enhancement and depth cue extraction are performed using equation (3) to obtain the feature volume. ,

[0072] (3)

[0073] in For the focused area identification module, its calculation method is shown in equation (4).

[0074] (4)

[0075] in For the input tensor, This is a 3D convolution operation. It is a non-linear activation function;

[0076] Step 4: Calculate the feature volume obtained in Step 3. Multi-scale temporal feature extraction is performed using equation (5) to obtain feature representations at different scales. ,

[0077] (5)

[0078] in As a multi-scale processing module, its calculation method is shown in equation (6).

[0079] (6)

[0080] in For the input tensor, This is a 3D convolution operation, with kernel sizes of [sizes to be filled in]. , For pooling operations, the pooling scales are respectively Then, the features at different scales are represented. The temporal feature representation is obtained through equation (7). ,

[0081] (7)

[0082] in The multi-scale feature fusion module is calculated as shown in equation (8).

[0083] (8)

[0084] in, For the input tensor, This is a 3D convolution operation. For splicing operations;

[0085] Step 5: Convert the feature volume obtained in Step 3 into a single volume. By converting to the frequency domain using equation (9), the frequency domain characteristics are obtained. ,

[0086] (9)

[0087] in It is a learnable adaptive weight matrix. This represents element-wise matrix multiplication. The result is a Fourier transform, and then the soft thresholding result is multiplied by the Fourier transform output using equation (10) to obtain the data representation. ,

[0088] (10)

[0089] in This is a soft threshold operation. This represents element-wise matrix multiplication. Then, the processed features are transformed from the frequency domain to the spatial domain by performing an inverse Fourier transform using equation (11), resulting in the spatial representation. ,

[0090] (11)

[0091] in For the inverse Fourier transform, the feature volume obtained in step 3 is finally transformed by equation (12). Spatial representation By performing residual connections on the features, a frequency domain feature representation is obtained. ,

[0092] (12)

[0093] Where + indicates feature residual connection;

[0094] Step 6: Convert the feature volume obtained in Step 3 into a single volume. By transforming to the fractional domain using equation (13), we obtain its fractional domain characteristics. ,

[0095] (13)

[0096] in It is a learnable adaptive weight matrix. This represents element-wise matrix multiplication. For order Fractional Fourier transform, order The data representation is obtained through adaptive learning of the model, and then the soft threshold result is multiplied by the output of the fractional Fourier transform using equation (14). ,

[0097] (14)

[0098] in This is a soft threshold operation. This represents element-wise matrix multiplication, followed by an inverse fractional Fourier transform using equation (15). The processed features are transformed from the frequency domain to the spatial domain to obtain the spatial representation. ,

[0099] (15)

[0100] in For order The inverse fast fractional Fourier transform is then performed, and finally the feature volume obtained in step 3 is expressed by equation (16). Spatial representation Residual connections are performed on the features to obtain fractional domain feature representations. ,

[0101] (16)

[0102] Where + indicates feature residual connection;

[0103] Step 7: Represent the time-domain features obtained in steps 4, 5, and 6. Frequency domain feature representation and fractional domain feature representation By inputting Equation (17) into the multi-domain feature fusion module, a multi-domain feature representation is obtained. ,

[0104] (17)

[0105] in The multi-domain feature fusion module is calculated as shown in equation (18).

[0106] (18)

[0107] in This is a 3D convolution operation. For splicing operations;

[0108] Step 8: Represent the multi-domain features obtained in Step 7 After performing nonlinear normalization, the result is weighted and summed with the preset focal length parameter, and the final depth estimation result is output through equation (19). ,like Figure 4 As shown,

[0109] (19)

[0110] in For activation function, This represents element-wise matrix multiplication. This is the preset focal length parameter.

[0111] This patented method is compared to Figure 5 , Figure 6 The depth estimation results of the AiF-net method have significant advantages in terms of the integrity of edge details and the consistency of depth homogeneous regions.

[0112] In the FlyingThings3D dataset, compared with the AiF-net method, the proposed method reduced the Abs Rel and LogRMAE metrics by 41.55% and 23.29% respectively, and improved the Acc1, Acc2 and Acc3 metrics by 14.33%, 8.26% and 6.61% respectively.

Claims

1. A depth estimation method based on multi-domain feature fusion, characterized in that: Includes the following steps: Step 1: By adjusting the focal length of the acquisition device to change the focal plane, images captured at different focal lengths are arranged in the order of capture to obtain a multi-focus image sequence covering the entire depth of field. , This represents the sequence number, and the size of a single frame image is... , Indicates the height and width of a single image. Indicates the number of channels in an image; Step 2: The multifocus image sequence obtained in Step 1... The input is fed into the focus extraction module, where the spatial correlation between image sequences is captured using equation (1) to obtain the focus features. , （1） in For the focused extraction module, its calculation method is shown in equation (2). （2） in For the input tensor, This is a 3D convolution operation. For batch normalization operations, It is a non-linear activation function; Step 3: Apply the focusing features obtained in Step 2 Multi-scale enhancement and depth cue extraction are performed using equation (3) to obtain the feature volume. , （3） in For the focused area identification module, its calculation method is shown in equation (4). （4） in For the input tensor, This is a 3D convolution operation. It is a non-linear activation function; Step 4: Calculate the feature volume obtained in Step 3. Multi-scale temporal feature extraction is performed using equation (5) to obtain feature representations at different scales. , （5） in As a multi-scale processing module, its calculation method is shown in equation (6). （6） in For the input tensor, This is a 3D convolution operation, with kernel sizes of [sizes to be filled in]. , For pooling operations, the pooling scales are respectively Then, the features at different scales are represented. The temporal feature representation is obtained through equation (7). , （7） in The multi-scale feature fusion module is calculated as shown in equation (8). （8） in, For the input tensor, This is a 3D convolution operation. For splicing operations; Step 5: Convert the feature volume obtained in Step 3 into a single volume. By converting to the frequency domain using equation (9), the frequency domain characteristics are obtained. , （9） in It is a learnable adaptive weight matrix. This represents element-wise matrix multiplication. The result is a Fourier transform, and then the soft thresholding result is multiplied by the Fourier transform output using equation (10) to obtain the data representation. , （10） in This is a soft threshold operation. This represents element-wise matrix multiplication. Then, the processed features are transformed from the frequency domain to the spatial domain by performing an inverse Fourier transform using equation (11), resulting in the spatial representation. , （11） in For the inverse Fourier transform, the feature volume obtained in step 3 is finally transformed by equation (12). Spatial representation By performing residual connections on the features, a frequency domain feature representation is obtained. , （12） Where + indicates feature residual connection; Step 6: Convert the feature volume obtained in Step 3 into a single volume. By transforming to the fractional domain using equation (13), we obtain its fractional domain characteristics. , （13） in It is a learnable adaptive weight matrix. This represents element-wise matrix multiplication. For order Fractional Fourier transform, order The data representation is obtained through adaptive learning of the model, and then the soft threshold result is multiplied by the output of the fractional Fourier transform using equation (14). , （14） in This is a soft threshold operation. This represents element-wise matrix multiplication, followed by an inverse fractional Fourier transform using equation (15). The processed features are transformed from the frequency domain to the spatial domain to obtain the spatial representation. , （15） in For order The inverse fast fractional Fourier transform is then performed, and finally the feature volume obtained in step 3 is expressed by equation (16). Spatial representation Residual connections are performed on the features to obtain fractional domain feature representations. , （16） Where + indicates feature residual connection; Step 7: Represent the time-domain features obtained in steps 4, 5, and 6. Frequency domain feature representation and fractional domain feature representation By inputting Equation (17) into the multi-domain feature fusion module, a multi-domain feature representation is obtained. , （17） in The multi-domain feature fusion module is calculated as shown in equation (18). （18） in This is a 3D convolution operation. For splicing operations; Step 8: Represent the multi-domain features obtained in Step 7 After performing nonlinear normalization, the result is weighted and summed with the preset focal length parameter, and the final depth estimation result is output through equation (19). , （19） in For activation function, This represents element-wise matrix multiplication. This is the preset focal length parameter.

Images