Depth map prediction method and device based on phase-guided Transform-CNN (Convolutional Neural Network) dual-path fusion

Abstract

The invention discloses a depth map prediction method and device based on phase-guided Transform-CNN (Convolutional Neural Network) dual-path fusion. The method comprises the following steps: acquiring a to-be-processed stripe image; frequency characteristics related to phase information are extracted from the fringe image through a Fourier model, and long-distance and global frequency characteristics are acquired by using a Transform model; extracting detail features from the stripe image by using a CNN model; and carrying out cross fusion on the long-distance and global frequency features and the detail features, and carrying out multi-scale feature enhancement to obtain a depth map of the stripe image. The PG-FTCNet can effectively predict phase information on a complex surface, and meanwhile, the prediction precision of the depth map is remarkably improved. According to the method, a single fringe image in fringe projection profilometry serves as input, phase unwrapping and depth map prediction are achieved at the same time, an efficient end-to-end calculation mode is reserved while the result interpretability is improved, and a reliable and practical solution is provided for a three-dimensional reconstruction task of FPP.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular to a depth map prediction method and device based on phase-guided Transformer-CNN dual-path fusion. Background Art

[0002] Fringe Projection Profilometry (FPP) is a non-contact 3D reconstruction technology widely used in the field of computer vision. Its basic principle is to project a fringe pattern onto the surface of the object to be measured, calculate the wrapped phase through the deformation of the fringe, and then perform phase unwrapping. Finally, the unwrapped phase is mapped to the depth value to generate the depth map required for 3D reconstruction. In this process, the accuracy of phase unwrapping plays a vital role in the quality of the final depth map. Therefore, phase unwrapping is one of the key steps of FPP.

[0003] In recent years, deep learning has been widely used in many fields and has made remarkable progress due to its outstanding performance in automatically learning complex information. In FPP-related research, many scholars have begun to explore how to introduce deep learning technology into phase unwrapping and depth map prediction tasks. For example, convolutional neural networks (CNNs) have been shown to perform well in phase unwrapping, effectively extracting local phase features and handling moderate noise. However, the receptive field of CNNs is limited, and there are deficiencies in capturing global features. It may incorrectly assume the continuity of local phases and have difficulty dealing with discontinuities or large phase jumps on complex surfaces, thus causing phase unwrapping errors.

[0004] In addition, generative adversarial networks (GANs) have also been applied to phase unwrapping tasks. Through adversarial learning between the generator and the discriminator, GANs have performed outstandingly in synthesizing data and suppressing high-frequency noise. However, the results often tend to smooth out details, especially when the generator is overfitted, artifacts may appear, limiting the credibility of the results. On the other hand, Transformer-based models have recently become a hot topic in phase unwrapping research because they can capture long-distance dependencies and rich contextual information. However, Transformer still faces the challenge of balancing when integrating global and local features. It may ignore local details due to excessive focus on the global background, thus limiting its performance in solving phase ambiguity and noise interference problems.

[0005] Although deep learning technology has shown potential in FPP-related research, most of the current mainstream depth prediction neural networks use an end-to-end approach from fringe images directly to depth maps, completely skipping the traditional phase unwrapping step. Although this method has improved computational efficiency, due to the lack of explicit modeling of the phase unwrapping process, the results are not interpretable enough and the phase information cannot be fully utilized to optimize the depth map prediction. This makes it difficult for existing methods to deal with problems such as phase jumps and error propagation when dealing with complex surfaces, resulting in limited quality of the final depth map. Summary of the invention

[0006] In order to solve the above technical problems, the embodiment created by the present invention provides a depth map prediction method based on phase-guided Transformer-CNN dual-path fusion, comprising: Acquire a fringe image to be processed; Extracting frequency features of phase information from the fringe image using a Fourier model, and acquiring long-range and global frequency features using a Transformer model; Extracting detail features from the stripe image using a CNN model; The long-distance and global frequency features and the detail features are cross-fused, and multi-scale feature enhancement is performed to obtain a depth map of the stripe image.

[0007] Furthermore, extracting frequency characteristics of phase information from the fringe image through a Fourier model includes: Converting the fringe image from a spatial domain image to a frequency domain image by Fourier transform; Extracting phase information encoded in the ±1-level spectrum from the frequency domain image and removing zero-frequency features; The image after removing the zero-frequency feature is processed through a feedforward neural network to obtain the frequency feature containing phase information.

[0008] Furthermore, the Transformer model includes: an MHSA model and an FFN model; and obtaining long-distance frequency features using the Transformer model includes: The frequency features are sequentially input into the MHSA model and the FFN model to obtain features with long-distance dependency and global frequency.

[0009] Furthermore, the extracting detail features from the fringe image using a CNN model includes: The stripe image is input into a plurality of convolution blocks, and the detail features of the stripe image are extracted by using a convolution layer with an activation function of LeakyReLU and batch normalization.

[0010] Furthermore, the cross-fusion of the long-distance and global frequency features and the detail features includes: In the encoder, the Transformer branch and the CNN branch perform feature cross-fusion at each layer; In the decoder, the feature image after feature cross-fusion is restored to the resolution of the original stripe image through upsampling, and the feature image after feature cross-fusion in the encoder is merged with the upsampled features of the decoder using jump connections to generate the unfolded phase.

[0011] Furthermore, in the encoder, the Transformer branch and the CNN branch perform feature cross-fusion at each layer, including: Inputting the stripe image containing the detail features into the convolution embedding model to convert it into the data shape features of the Transformer branch; The data shape features converted into the Transformer branch are fused with the long-distance and global frequency features, restored to a standard feature map by an image reconstruction model, and then optimized by a CNN model.

[0012] Furthermore, the performing multi-scale feature enhancement to obtain the depth map of the stripe image includes: The unwrapped phase is used as a physical information reference and fused with the upsampled features and the input image in a multi-scale global-local fusion model MSEF to obtain high-resolution deep features.

[0013] The present invention discloses a depth map prediction device based on phase-guided Transformer-CNN dual-path fusion, comprising: An acquisition model is used to acquire a fringe image to be processed; A processing model for extracting frequency features of phase information from the fringe image using a Fourier model, and obtaining long-range and global frequency features using a Transformer model; The processing model is also used to extract detail features from the stripe image using a CNN model; An execution model is used to cross-fuse the long-range and global frequency features and the detail features, and perform multi-scale feature enhancement to obtain a depth map of the stripe image.

[0014] Furthermore, the acquisition model includes: A first acquisition sub-model is used to convert the fringe image from a spatial domain image to a frequency domain image by Fourier transform; A first processing sub-model is used to extract the phase information encoded in the ±1 level spectrum from the frequency domain image and remove the zero frequency feature; The first execution sub-model is used to process the image after removing the zero-frequency feature through a feedforward neural network to obtain a frequency feature containing phase information.

[0015] Furthermore, the Transformer model includes: an MHSA model and an FFN model; the processing model includes: The second processing sub-model is used to input the frequency features into the MHSA model and the FFN model in sequence to obtain features with long-distance dependency and global frequency.

[0016] Furthermore, the processing sub-model includes: The third processing sub-model is used to input the stripe image into multiple convolution blocks, and use a convolution layer with an activation function of LeakyReLU and batch normalization to extract detail features of the stripe image.

[0017] Furthermore, the execution model includes: The second execution sub-model is used in the encoder to perform feature cross-fusion between the Transformer branch and the CNN branch at each layer through the multi-scale global-local fusion model MGLF; The third execution sub-model is used to restore the feature image after feature cross-fusion to the resolution of the original stripe image through upsampling in the decoder, and to merge the feature image after feature cross-fusion in the encoder with the upsampled features of the decoder using jump connections to generate an unfolded phase.

[0018] Furthermore, the second execution sub-model includes: A fourth execution sub-model, used for inputting the stripe image containing the detail features into the convolution embedding model to convert it into a data shape feature of the Transformer branch; The fifth execution sub-model is used to fuse the data shape features converted into the Transformer branch with the long-distance and global frequency features, and optimize them by the CNN model after restoring them to the standard feature map through the image reconstruction model.

[0019] Furthermore, the execution model includes: The sixth execution sub-model is used to use the unfolded phase as a physical information reference, and fuse it with the upsampled features and the input image in the multi-scale global-local fusion model MSEF to obtain high-resolution depth features.

[0020] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor performs the steps of the depth map prediction method based on phase-guided Transformer-CNN dual-path fusion as described above.

[0021] A storage medium storing computer-readable instructions, when the computer-readable instructions are executed by one or more processors, causes the one or more processors to perform the steps of the depth map prediction method based on phase-guided Transformer-CNN dual-path fusion as described above.

[0022] The present invention is based on a deep learning network architecture - Phase-Guided Fourier-Transformer-CNN Network (PG-FTCNet). Different from the existing end-to-end depth prediction scheme, PG-FTCNet explicitly introduces the phase unwrapping process, and integrates phase unwrapping and depth prediction into a unified network architecture by phase guidance. The network adopts a dual-branch architecture of Transformer and CNN to extract global and local features respectively, and introduces a Fourier transform model in the Transformer branch to extract frequency features. The introduction of Fourier transform enhances the network's ability to capture phase jumps and long-distance dependent features on complex surfaces, while the dual-branch architecture performs well in the balanced integration of global features and local details. PG-FTCNet can effectively predict phase information on complex surfaces and significantly improve the accuracy of depth map prediction. This method uses a single fringe image in fringe projection profilometry as input, and simultaneously realizes phase unwrapping and depth map prediction. While improving the interpretability of the results, it retains an efficient end-to-end computing mode, providing a reliable and practical solution for the three-dimensional reconstruction task of FPP. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A schematic diagram of a flow chart of a depth map prediction method based on phase-guided Transformer-CNN dual-path fusion provided by an embodiment of the present invention; Figure 2A schematic diagram of model processing of another depth map prediction method based on phase-guided Transformer-CNN dual-path fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0027] like Figure 1 As shown, the present invention provides a depth map prediction method based on phase-guided Transformer-CNN dual-path fusion, comprising: S1, obtaining a fringe image to be processed; S2. Extracting frequency features of phase information from the fringe image using a Fourier model, and obtaining long-distance and global frequency features using a Transformer model; S3, extracting detail features from the stripe image using a CNN model; S4, cross-fusing the long-distance and global frequency features and the detail features, and performing multi-scale feature enhancement to obtain a depth map of the stripe image.

[0028] In one embodiment of the present invention, in step S2, extracting frequency characteristics of phase information from the fringe image using a Fourier model includes: Step 1: Convert the fringe image from the spatial domain image to the frequency domain image through Fourier transform; Step 2: extract the phase information encoded in the ±1-level spectrum from the frequency domain image and remove the zero-frequency feature; Step 3: Process the image after removing the zero-frequency feature through a feedforward neural network to obtain the frequency feature containing phase information.

[0029] It should be noted that the Transformer model includes: an MHSA model and an FFN model; using the Transformer model to obtain long-distance frequency features includes: inputting the frequency features into the MHSA model and the FFN model in sequence to obtain features with long-distance dependencies and global frequencies.

[0030] like Figure 2As shown in (a), in the Fourier model, the stripe image is processed by fast Fourier transform (FFT), learnable filter, inverse fast Fourier transform (IFFT), layer normalization (Layer Norm), feedforward neural network (FFN) and regularization (Dropout) in sequence. After FFT, the stripe image is converted from the spatial domain to the frequency domain to obtain the frequency feature, extract the phase information encoded in the ±1 level spectrum, remove the DC component and other irrelevant frequencies through the learnable filter and filter frequency to suppress low-frequency noise, and obtain the frequency feature through IFFT.

[0031] It should be noted that the frequency features after screening and optimization, that is, FFN, provide support for the Transformer model to capture global frequency-related features. It should be noted that after Fourier transform, there are zero-order and ±1-order spectra, and the phase information is contained in the ±1-order spectrum.

[0032] like Figure 2 As shown in (b), (b-1) is a schematic diagram of the Transformer model. The Transformer model consists of a multi-head self-attention mechanism (MHSA) model and a feedforward neural network (FFN) model, which captures long-distance dependencies and global frequency-related features, and provides support for the encoder's global feature extraction.

[0033] Specifically, the frequency features are processed by layer normalization (Layer Norm), self-attention mechanism (MHSA) and regularization (Dropout) in sequence, and the frequency image is processed by layer normalization (Layer Norm), feedforward neural network (FFN) and regularization (Dropout) in sequence.

[0034] The present invention utilizes the phase-frequency feature extraction method in FTP and combines it with the feature learning capability of deep learning to enhance the adaptability and robustness of frequency information extraction. At the same time, the frequency components are screened by learnable parameters, taking into account the retention of high-frequency details and the suppression of low-frequency noise, so as to achieve more accurate frequency domain modeling. And the extracted frequency features are input into the Transformer model to further capture the long-distance frequency-related features, so that the phase information can be globally modeled in deep prediction.

[0035] The Fourier model retains the physical interpretability of phase information by optimizing the extraction of phase features in the frequency domain, making up for the shortcomings of traditional deep learning methods in end-to-end prediction from fringe images to depth images. Inputting frequency information into the network as an explicit feature reduces noise and background interference, and improves the accuracy and robustness of depth prediction. Experiments show that the model can effectively alleviate the problems of phase jumps and error propagation in complex surfaces (such as severe undulations or surface discontinuities).

[0036] In one embodiment of the present invention, in step S3, detail features are extracted from a stripe image using a CNN model, including: inputting the stripe image into multiple convolution blocks, and extracting detail features of the stripe image using a convolution layer carrying an activation function of LeakyReLU and batch normalization.

[0037] like Figure 2 As shown in (b), (b-2) is a schematic diagram of the convolution (CNN) model. In the CNN model, the stripe image is processed by two-dimensional convolution Conv2d, convolution kernel k=3, step size s=1, and padding value p=1, and then batch normalization (Btach Norm) and local features are extracted through the activation function with LeakyReLU. After multiple extractions, the detailed information of the object surface is effectively captured.

[0038] In one embodiment of the present invention, in step S4, cross-fusion of long-range and global frequency features and detail features is performed, including: Step 1: In the encoder, the Transformer branch and the CNN branch perform feature cross-fusion at each layer through the multi-scale global-local fusion model MGLF; It should be noted that the Multi-scale Global-Local Fusion (MGLF) model is used to fuse local and global features at multiple scales. This module converts local features into a data shape consistent with global features through the Convolutional Patch Embedding (CPE) model, and then performs global-local feature fusion. The fused features are input into the Transformer model for further calculation. The obtained global features are transmitted forward through the global branch, and at the same time, the global features are re-expanded into the data shape of local features through the image patch reconstruction model (Patch Reconstruction, PR), and then fused with the initial local features and input into the CNN model for further calculation. The obtained local features are transmitted forward through the local branch. This fusion method can effectively integrate local and global information and improve the richness of feature representation.

[0039] The specific process is as follows: Inputting the stripe image containing the detail features into the convolution embedding model to convert it into the data shape features of the Transformer branch; The data shape features converted into the Transformer branch are fused with the long-distance and global frequency features, restored to a standard feature map by an image reconstruction model, and then optimized by a CNN model.

[0040] Step 2: In the decoder, the feature image after feature cross-fusion is restored to the resolution of the original stripe image by upsampling, and the feature image after feature cross-fusion in the encoder is merged with the upsampled features of the decoder using jump connections to generate the unfolded phase.

[0041] like Figure 2 As shown in (c) in the figure, the input stripe image passes through the Fourier model and Transformer model in the Transformer branch, and the CNN model in the CNN branch, and feature fusion is performed in the MGLF model. After fusion, it is sampled through the Multi-Scale Feature Enhancement Fusion (MSEF) model, and the expanded phase is extracted through 1X1 convolution to obtain the depth map.

[0042] It should be noted that in the MGLF model, in the encoder, the Transformer branch and the CNN branch perform feature cross-fusion at each layer through the MGLF model. Figure 2 As shown in (b-3) in the figure, the local features extracted by CNN (W is width, H is height, and C is the number of channels) are processed by the CPE model through two-dimensional convolution Conv2d, convolution kernel k=4, step size s=2, and padding value p=1, and then converted to the data shape of the Transformer branch through reshaping and permutation (reshape and pemute), that is, the width becomes W / 2, the height is H / 2, and the number of channels is C / 2. After being fused with the global features processed by the Transformer model, as shown in Figure 2 As shown in (b-4) in the figure, the PR model restores the standard feature map, that is, after reshaping and permutation, interpolation and further optimization by two-dimensional convolution Conv2d, convolution kernel k=1. Through this process, the global frequency features and local spatial features are effectively integrated, thereby enhancing the overall feature extraction capability of the network. It should be noted that, Figure 2 As shown in (c), in each layer of the MGLF model, the image processed by the Transformer model is processed by the PR model and then feature fused by the CNN model. At the same time, the image processed by the CNN model is processed by the CPE model and then feature fused by the Transformer model.

[0043] In addition, Figure 2In (c), in the MSEF model, at the decoder stage, the feature map is upsampled layer by layer to restore the resolution of the original image, and the multi-scale fusion features of the encoder are merged layer by layer with the upsampled features of the decoder using skip connections. Finally, the unfolded phase generates a depth map through a 1×1 convolution, which is used as a guidance prior to participate in the final depth prediction.

[0044] In one embodiment of the present invention, in step S4, multi-scale feature enhancement is performed to obtain a depth map of the fringe image, including: using the unfolded phase as a physical information reference, fusing it with the upsampled features and the input image in the MSEF model, and obtaining a high-resolution depth feature.

[0045] It should be noted that the MSEF model is used to fuse features of different scales and generate the final depth map. The module first adjusts the features of different scales to the same number of channels, and then performs upsampling and addition operations layer by layer so that the feature maps of all scales are fused at the highest resolution. Finally, the fused feature map is converted into a depth map through convolution. The MSEF model can effectively integrate features of different scales and improve the accuracy of depth prediction.

[0046] In an embodiment of the present invention, the unfolded phase is used as a physical information reference (physical guidance) and is fused with the upsampled features and input images in the MSEF model to integrate high-resolution and depth features layer by layer. In the last step, the upsampled feature map is restored to the input resolution, and after a 1×1 convolution operation, a high-precision depth map is finally generated. This network architecture effectively captures spatial and frequency-related information, and combines the unfolded phase as physical prior knowledge to guide depth prediction, ensuring the accuracy and robustness of the depth map, especially in complex scenes, and can provide reliable 3D reconstruction results.

[0047] In the decoder, the present invention uses a CNN model to generate an unwrapped phase. The unwrapped phase is combined with the fringe image and the feature map extracted by the network, and participates in the multi-scale feature fusion process of generating the final depth map. The multi-scale feature enhancement fusion MSEF model uses the unwrapped phase as a guiding prior, and fuses it with the feature map and high-resolution feature map sampled layer by layer in the decoder to improve the accuracy and interpretability of depth prediction. This method retains the physical modeling characteristics of fringe projection profilometry (FPP), making the depth prediction process more in line with physical laws. The training process of the deep learning model is optimized through phase guidance, which effectively alleviates the problems of phase jumps and error propagation, and improves the robustness in complex surface scenes. The physical prior of phase unwrapping is introduced into the deep learning model, and the unwrapped phase is explicitly modeled and generated, which is used as important guiding information in the depth prediction process. Performance comparison as shown in Table 1 below: Table 1. PG-FTCNet model performance comparison

[0048] In this invention, PG-FTCNet is used to combine frequency domain processing with Fourier model, global frequency-related feature extraction with Transformer model, and local spatial detail capture with CNN model. These components are unified by MGLF model and MSEF model, which jointly enhance the ability of the model to accurately reconstruct 3D surfaces. At the same time, the root mean square error RMSE of the unfolded phase prediction is 0.0212 mm, the structural similarity SSIM is 0.9924, and the RMSE of the depth map prediction is 0.0220 mm and SSIM is 0.9925. Experiments show that the lack of unfolded phase as a physical prior guide (no unfolded phase in Table 1) leads to a significant decrease in accuracy (RMSE 0.0398 mm, SSIM 0.9728), highlighting its key role in depth prediction. In addition, experiments show that excluding key models such as Fourier model or MGLF model leads to a significant decrease in performance, emphasizing the importance of these models in capturing meaningful frequency-related features and achieving cross-fusion of global and local information. This cross-fusion is crucial for accurately integrating various features, thereby maximizing the prediction accuracy of the model.

[0049] These results show that PG-FTCNet not only outperforms traditional methods, but also becomes a robust and efficient solution for high-precision 3D reconstruction in practical applications by leveraging the synergistic effects of its specialized components. In the field of architecture, it can be used to accurately model buildings to support the design, planning, and monitoring of engineering projects. In the field of medicine, this technology can obtain the 3D structure of human organs and is widely used in surgical planning, orthodontics, facial reconstruction, and dental treatment. In the field of cultural heritage protection and digitization, it can be used for the digitization and 3D reconstruction of cultural relics, ancient buildings, and artworks for more effective protection, restoration, and research. In industrial manufacturing, this technology is used in product quality control, morphological detection, and component inspection to help accurately measure part size, surface defects, and assembly quality. In game development and virtual reality technology, it is used to build realistic virtual scenes and characters, significantly improving the interactive experience of games and the visual effects of virtual reality applications. With the advancement of technology, some smartphones and cameras have integrated 3D reconstruction functions to support consumer-level applications such as augmented reality, face unlocking, and gesture recognition, further promoting the popularization and application of this technology in daily life.

[0050] An embodiment of the present invention also provides a depth map prediction device based on phase-guided Transformer-CNN dual-path fusion, including: The present invention discloses a depth map prediction device based on phase-guided Transformer-CNN dual-path fusion, including: an acquisition model, used to acquire a stripe image to be processed; a processing model, used to extract frequency features about phase information from the stripe image through a Fourier model, and use a Transformer model to obtain long-distance and global frequency features; the processing model is also used to extract detail features from the stripe image using a CNN model; an execution model, used to cross-fuse the long-distance and global frequency features and the detail features, and perform multi-scale feature enhancement to obtain a depth map of the stripe image.

[0051] The present invention is based on a deep learning network architecture - Phase-Guided Fourier-Transformer-CNN Network (PG-FTCNet). Different from the existing end-to-end depth prediction scheme, PG-FTCNet explicitly introduces the phase unwrapping process, and integrates phase unwrapping and depth prediction into a unified network architecture by phase guidance. The network adopts a dual-branch architecture of Transformer and CNN to extract global and local features respectively, and introduces a Fourier transform model in the Transformer branch to extract frequency features. The introduction of Fourier transform enhances the network's ability to capture phase jumps and long-distance dependent features on complex surfaces, while the dual-branch architecture performs well in the balanced integration of global features and local details. PG-FTCNet can effectively predict phase information on complex surfaces and significantly improve the accuracy of depth map prediction. This method uses a single fringe image in fringe projection profilometry as input, and simultaneously realizes phase unwrapping and depth map prediction. While improving the interpretability of the results, it retains an efficient end-to-end computing mode, providing a reliable and practical solution for the three-dimensional reconstruction task of FPP.

[0052] In some embodiments, the acquisition model includes: a first acquisition sub-model, used to convert the fringe image from a spatial domain image to a frequency domain image through Fourier transform; a first processing sub-model, used to extract the phase information encoded in the ±1 level spectrum from the frequency domain image and remove the zero-frequency feature; a first execution sub-model, used to process the image after removing the zero-frequency feature through a feedforward neural network to obtain a frequency feature containing phase information.

[0053] In some embodiments, the Transformer model includes: an MHSA model and an FFN model; the processing model includes: a second processing sub-model, used to input the frequency features into the MHSA model and the FFN model in sequence to obtain features with long-distance dependencies and global frequencies.

[0054] In some embodiments, the processing sub-model includes: a third processing sub-model, used to input the stripe image into multiple convolution blocks, and use a convolution layer carrying an activation function of LeakyReLU and batch normalization to extract detail features of the stripe image.

[0055] In some embodiments, the execution model includes: a second execution sub-model, which is used in the encoder to perform feature cross-fusion at each layer of the Transformer branch and the CNN branch through a multi-scale global-local fusion model MGLF; a third execution sub-model, which is used in the decoder to restore the feature image after feature cross-fusion to the resolution of the original stripe image by upsampling, and use jump connections to merge the feature image after feature cross-fusion in the encoder with the upsampled features of the decoder to generate an unfolded phase.

[0056] In some embodiments, the second execution sub-model includes: a fourth execution sub-model, used to input the stripe image containing the detail features into the convolution embedding model to convert it into a data shape feature of the Transformer branch; a fifth execution sub-model, used to fuse the data shape features converted into the Transformer branch with the long-distance and global frequency features, and after being restored to a standard feature map by an image reconstruction model, it is optimized by a CNN model.

[0057] In some embodiments, the execution model includes: a sixth execution sub-model, which is used to use the unfolded phase as a physical information reference, and fuse it with the upsampled features and the input image in a multi-scale global-local fusion model MSEF to obtain high-resolution depth features.

[0058] To solve the above technical problems, an embodiment of the present invention also provides a computer device. The computer device includes a processor, a non-volatile storage medium, a memory and a network interface connected by a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database and a computer-readable instruction, and the database may store a control information sequence. When the computer-readable instruction is executed by the processor, the processor can implement a depth map prediction method based on phase-guided Transformer-CNN dual-path fusion. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor may execute a method for predicting protein-ligand binding affinity based on distance and atomic contact number. The network interface of the computer device is used to connect and communicate with the terminal. In this embodiment, the processor is used to execute the specific content of the acquisition model and the processing model, and the memory stores the program code and various types of data required to execute the above model. The network interface is used to transmit data between the user terminal or the server. The memory in this embodiment stores the program code and data required to execute all sub-models in the image processing method, and the server can call the program code and data of the server to execute the functions of all sub-models.

[0059] The present invention also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, enables one or more processors to perform the steps of a depth map prediction method based on phase-guided Transformer-CNN dual-path fusion described in any of the above embodiments.

[0060] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0061] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0062] The above descriptions are only some embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0063] The above content is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A depth map prediction method based on phase-guided Transformer-CNN dual-path fusion, characterized in that: include: Acquire a fringe image to be processed; Extracting frequency features of phase information from the fringe image using a Fourier model, and acquiring long-range and global frequency features using a Transformer model; Extracting detail features from the stripe image using a CNN model; The long-distance and global frequency features and the detail features are cross-fused, and multi-scale feature enhancement is performed to obtain a depth map of the stripe image.

2. The method according to claim 1, characterized in that The extracting frequency characteristics of phase information from the fringe image by using a Fourier model comprises: Converting the fringe image from a spatial domain image to a frequency domain image by Fourier transform; Extracting phase information encoded in the ±1-level spectrum from the frequency domain image and removing zero-frequency features; The image after removing the zero-frequency feature is processed through a feedforward neural network to obtain the frequency feature containing phase information.

3. The method according to claim 1, characterized in that The Transformer model includes: an MHSA model and an FFN model; and the method of obtaining long-distance frequency features using the Transformer model includes: The frequency features are sequentially input into the MHSA model and the FFN model to obtain features with long-distance dependency and global frequency.

4. The method according to claim 1, characterized in that: The extracting detail features from the stripe image using a CNN model includes: The stripe image is input into a plurality of convolution blocks, and the detail features of the stripe image are extracted by using a convolution layer with an activation function of LeakyReLU and batch normalization.

5. The method according to claim 1, characterized in that The cross-fusion of the long-distance and global frequency features and the detail features comprises: In the encoder, the Transformer branch and the CNN branch perform feature cross-fusion at each layer; In the decoder, the feature image after feature cross-fusion is restored to the resolution of the original stripe image through upsampling, and the feature image after feature cross-fusion in the encoder is merged with the upsampled features of the decoder using jump connections to generate the unfolded phase.

6. The method according to claim 1, characterized in that In the encoder, the Transformer branch and the CNN branch perform feature cross-fusion at each layer through the multi-scale global-local fusion model MGLF, including: Inputting the stripe image containing the detail features into the convolution embedding model to convert it into the data shape features of the Transformer branch; The data shape features converted into the Transformer branch are fused with the long-distance and global frequency features, restored to a standard feature map by an image reconstruction model, and then optimized by a CNN model.

7. The method according to claim 6, characterized in that The performing multi-scale feature enhancement to obtain the depth map of the stripe image includes: The unwrapped phase is used as a physical information reference and fused with the upsampled features and the input image to obtain high-resolution deep features.

8. A depth map prediction device based on phase-guided Transformer-CNN dual-path fusion, characterized in that: include: An acquisition model is used to acquire a fringe image to be processed; A processing model for extracting frequency features of phase information from the fringe image using a Fourier model, and obtaining long-range and global frequency features using a Transformer model; The processing model is also used to extract detail features from the stripe image using a CNN model; An execution model is used to cross-fuse the long-range and global frequency features and the detail features, and perform multi-scale feature enhancement to obtain a depth map of the stripe image.

9. The depth map prediction device according to claim 8, characterized in that The acquisition model includes: A first acquisition sub-model is used to convert the fringe image from a spatial domain image to a frequency domain image by Fourier transform; A first processing sub-model is used to extract the phase information encoded in the ±1 level spectrum from the frequency domain image and remove the zero frequency feature; The first execution sub-model is used to process the image after removing the zero-frequency feature through a feedforward neural network to obtain a frequency feature containing phase information.

10. The depth map prediction device according to claim 8, characterized in that: The Transformer model includes: an MHSA model and an FFN model; the processing model includes: The second processing sub-model is used to input the frequency features into the MHSA model and the FFN model in sequence to obtain features with long-distance dependency and global frequency.

Images