Single-frame stripe image prediction method, device, computer equipment and storage medium

Abstract

The embodiment of the present invention discloses a prediction method for a single-frame stripe image, including: using a Transformer‑based Channel Fusion (TCF) module to automatically identify objects in a single-frame stripe image, and fusing the overall shape and contour information of the object with the original single-channel image, wherein the TCF module is fine-tuned based on a pre-trained model Dense Prediction Transformer (DPT); extracting features from the fused image through a Res‑Unet module to obtain the depth information of the single-frame stripe image. The proposed TCF‑ResNet model realizes end-to-end prediction that directly maps the stripe image to a depth map. This method significantly simplifies the entire process and improves efficiency, making it more suitable for fast depth estimation and real-time three-dimensional reconstruction. In addition, with the application of neural networks, the anti-noise performance is also enhanced, making the prediction results under different conditions more reliable.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a prediction method, device, computer equipment and storage medium for a single-frame stripe image. Background Art

[0002] Fringe projection profilometry (FPP) is a 3D reconstruction method in computer vision, which is used to obtain the depth information of the scene. Due to its non-contact and simple equipment, it has been applied in many fields. For example, in the field of robotics, fringe projection 3D reconstruction can help robots perceive and understand the environment to perform tasks such as navigation, obstacle avoidance, and object grasping. It is of great significance for applications such as industrial automation, logistics, and warehousing. In the field of architecture, fringe projection 3D reconstruction can be used to build accurate models of buildings to help design, plan, and monitor engineering projects. In the field of medicine, fringe projection can obtain the 3D shape of human organs for surgical planning, orthodontics, facial reconstruction, and dental applications. In the field of cultural heritage protection and digital culture, fringe projection is used to digitize and reconstruct 3D models of cultural relics, ancient buildings, and artworks for protection, restoration, and research. In the industrial field, fringe projection 3D reconstruction is used for product quality control, shape detection, and part detection to help detect the size, surface defects, and assembly problems of parts. In the field of game development and virtual reality, 3D reconstruction technology is used to create realistic virtual environments and characters, which improves the quality of gaming experience and virtual reality applications. At present, some smartphones and cameras have integrated 3D reconstruction technology to support consumer applications such as augmented reality, facial unlocking, gesture recognition, etc., further promoting the popularization of this technology in daily life.

[0003] FPP is a method of projecting a pattern of known frequency and phase onto the surface of a target object through a projector. Due to the shape difference of the object surface, the projected stripes will be deformed. Subsequently, a camera is used to record the deformed stripe images on the irradiated object. These images contain information about the deformation of the object surface. By performing phase calculation, phase unwrapping and unwrapping phase-depth mapping on these deformed stripe images, the depth information of the object under test can be obtained, thereby realizing the reconstruction of the three-dimensional shape. It can be seen that accurate phase calculation has become a key element of this technology. Currently, the commonly used methods for solving the phase are Fourier transform profilometry (FTP) and phase shift profilometry (PSP). Among them, the former has also developed variants such as windowed Fourier transform profilometry (WFT) and wavelet transform (WT). However, the common feature of these methods is that the high-frequency stripe information modulated on the surface of the object needs to be well separated from the background intensity in the frequency domain. If the depth variation of the object surface is too complex, the reconstruction will fail due to spectrum aliasing, especially when processing sharp edges. There are great limitations. The latter can be used to reconstruct objects with complex surfaces, but it requires three or more phase-shifted sinusoidal stripe patterns. This method of capturing multiple deformed fringe patterns is easily disturbed by the environment, which limits it to static scenes and has certain limitations in dynamic or high-speed scenes. In order to reduce the number of fringe patterns, a phase retrieval method based on a single fringe pattern is proposed.

[0004] In the prior art, a neural network based on U-Net is used to predict the order of stripes at each pixel based on the wrapped phase map. An improved version (PhaseNet 2.0) is proposed based on DenseNet to improve the accuracy of phase unfolding. The unfolded phase is obtained by predicting the numerator and denominator of the wrapped phase. Based on this method, the color stripe map acquired in a single shot is also split into three single-channel grayscale images as input, but the depth map predicted by this method lacks information, and the prediction performance still has room for improvement. It can be seen that relying only on a single-frame single-frequency stripe map for high-precision three-dimensional reconstruction is a challenging task. The main problem of this method is that the amount of information and constraints are limited. Compared with the mutli-shot method, using only a single image for three-dimensional reconstruction requires higher algorithm accuracy and data processing skills to solve the problem of insufficient information.

[0005] The above traditional methods, such as those based on multi-step phase shifting and temporal phase unwrapping, usually require the acquisition of multiple fringe images to obtain high-precision depth information. However, this also requires the system to maintain high stability when acquiring multiple images to avoid introducing errors or even mistakes. Summary of the invention

[0006] An embodiment of the present invention provides a single-frame stripe image prediction method, comprising:

[0007] Automatically identify objects in a single-frame stripe image using a TCF module, and fuse the overall shape and contour information of the object with the original single-channel image, wherein the TCF module is obtained based on DPT pre-training;

[0008] The fused image is subjected to feature extraction by the Res-Unet model to obtain a predicted graph of the single-frame stripe image.

[0009] Furthermore, the method of automatically identifying an object in a single-frame fringe image by using the TCF module and fusing the overall shape and contour information of the object with the original single-channel image includes:

[0010] Performing fine-tuned DPT processing on the single-frame fringe image to obtain the overall shape and contour information of the single-frame fringe image;

[0011] The overall shape and contour information of the single-frame fringe image are fused with the original single-channel image.

[0012] Furthermore, the Res-Unet model includes: a first residual block, a second residual block and a conv-relu block.

[0013] Furthermore, extracting features from the fused image through a Res-Unet model to obtain a predicted graph of the single-frame stripe image includes:

[0014] The fused image is encoded and decoded using the first residual block, the +second residual block and the conv-relu block to obtain the predicted depth of the single-frame stripe image.

[0015] Furthermore, it also includes:

[0016] The fused image is passed through the first residual block to perform a convolution operation with a kernel size of 3×3, a step size of 1, and a padding of 1 to keep the size and number of channels of the feature map unchanged;

[0017] The fused image is processed by the second residual block through two paths so that the size of the feature map becomes half of the original one and the number of channels becomes twice the original one. One path undergoes a convolution operation with a convolution kernel of 3×3, a step size of 2, and a padding of 1, and the other path undergoes a convolution operation with a convolution kernel of 1×1, a step size of 2, and a padding of 0.

[0018] The fused image is subjected to a 1×1 convolution operation using the conv-Relu block and then subjected to Relu processing for nonlinear transformation without changing the size of the feature map and the number of channels.

[0019] A single-frame stripe image prediction device, comprising:

[0020] An acquisition module, used for automatically identifying an object in a single-frame stripe image using a TCF module, and fusing the overall shape and contour information of the object with the original single-channel image, wherein the TCF module is obtained based on DPT pre-training;

[0021] The execution module is used to extract features of the fused image through the Res-Unet model to obtain a predicted graph of the single-frame stripe image.

[0022] Furthermore, the acquisition module includes:

[0023] The first processing submodule is used to perform fine-tuned DPT processing on the single-frame fringe image to obtain the overall shape and contour information of the single-frame fringe image;

[0024] The second processing submodule is used to fuse the overall shape and contour information of the single-frame fringe image with the original single-channel image.

[0025] Furthermore, the Res-Unet model includes: a first residual block, a second residual block and a conv-relu block.

[0026] Furthermore, the processing module includes:

[0027] The third processing submodule is used to encode and decode the fused image using the first residual block, the +second residual block and the conv-relu block to obtain the predicted depth of the single-frame stripe image.

[0028] Furthermore, it also includes:

[0029] A first execution submodule is used to perform a convolution operation with a kernel size of 3×3, a step size of 1, and a padding of 1 on the fused image through the first residual block so that the size and the number of channels of the feature map remain unchanged;

[0030] The second execution submodule is used to process the fused image through the second residual block through a dual path so that the size of the feature map becomes half of the original one and the number of channels becomes twice the original number of channels, wherein one path undergoes a convolution operation with a convolution kernel of 3×3, a step size of 2, and a padding of 1, and the other path undergoes a convolution operation with a convolution kernel of 1×1, a step size of 2, and a padding of 0;

[0031] The third execution submodule is used to perform a 1×1 convolution operation on the fused image using a conv-Relu block and then perform Relu processing to perform a nonlinear transformation without changing the size of the feature map and the number of channels.

[0032] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the single-frame stripe image prediction method as described above.

[0033] A storage medium storing computer-readable instructions, when the computer-readable instructions are executed by one or more processors, enables the one or more processors to execute the steps of the single-frame stripe image prediction method as described above.

[0034] The beneficial effects of the embodiments of the present invention are as follows: the proposed TCF-ResNet model realizes end-to-end prediction of directly mapping the fringe map to the depth map, which significantly simplifies the entire process and improves efficiency, making it more suitable for fast depth estimation and real-time 3D reconstruction. In addition, with the application of neural networks, the anti-noise performance is also enhanced, making the prediction results under different conditions more reliable.

[0035] Global information such as the overall shape and contour of the object to be tested is introduced through Transformer, and the image channel is designed to fuse the global information with the original single-channel image to obtain a three-channel image. This design enables the model to understand the input image more comprehensively, thereby more accurately predicting the shape of the object to be tested, significantly enhancing the model's perception ability in different scenes and different objects, thereby improving the accuracy and robustness of single-frame fringe image depth prediction.

[0036] By introducing the image channel design with global information, and then using ResNet-18 as the backbone, and replacing the encoder of Unet with ResNet-18, the potential of the pre-trained model is fully utilized. This improvement not only accelerates the training convergence speed of the model, but also significantly improves the depth prediction performance, with the plane accuracy prediction error of about 6μm and the sphere radius prediction error of about 90μm. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A single-frame stripe image prediction method flow provided by an embodiment of the present invention;

[0039] Figure 2 It is a phase-depth calibration diagram in an embodiment of the present invention;

[0040] Figure 3 A scene data diagram in an embodiment of the present invention;

[0041] Figure 4 4 is a diagram of prediction results in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0043] In the single-frame stripe structured light depth prediction task, the input deformed stripe image can be expressed by the formula: ,in Represents the gray value of a point in the deformed fringe image captured by the camera. Represents the gray value of the deformed fringe pattern background, represents the modulation intensity of the fringe. Represents the amplitude of the sinusoidal stripes reflected at a certain point on the target surface. When the sinusoidal stripes are less modulated by the object, the deformation of the stripes is less significant. will be close to zero, and is most likely the background area; The larger it is, the more likely it is an object area. Distinguish objects from background. Usually, the modulated deformed fringe image With different values, if the object area and the background area are distinguished by setting a threshold, the stability and generalization of the prediction results will be affected due to the inconsistency of the scene. In order to solve this problem, the present application scheme can enable the neural network to automatically identify stripes with small or no modulation, thereby automatically generating a mask of the object, which helps to improve the stability of the model and its ability to adapt to different situations. The following is a prediction method for a single-frame stripe image including a TCF module and a Res-Unet module.

[0044] like Figure 1 A prediction method for a single-frame stripe image is shown, comprising:

[0045] S1. Automatically identify an object in a single-frame stripe image using a TCF module, and fuse the overall shape and contour information of the object with the original single-channel image, wherein the TCF module is obtained based on DPT pre-training;

[0046] Specifically, a single-frame fringe image is processed by fine-tuned DPT to obtain the overall shape and contour information of the single-frame fringe image; and the overall shape and contour information of the single-frame fringe image is fused with the original single-channel image. In one embodiment of the present invention, an input single-frame H×W×1 fringe image is processed by Fine-Tuned DPT to obtain global information of the object, and a H×W×3 image obtained by image channel fusion contains the original information and the overall shape and contour information of the object.

[0047] The DPT pre-trained model has general knowledge of predicting depth information for a single-frame image, but the model does not perform well when processing a single-frame stripe structured light. Therefore, this solution utilizes the general knowledge of the DPT model and fine-tunes it to adapt to the single-frame stripe structured light depth prediction task. In one embodiment of the present invention, a deformed stripe pattern of different contrast and frequency is input into the vanilla DPT to obtain a prediction result. The vanilla DPT has the potential to identify the edge contour of an object, but the predicted contour will appear to diverge from the object. Fine-tuning can significantly improve the prediction quality and obtain more accurate object edge information. The vanilla DPT also has errors in the prediction of the background area, and information that should not appear appears in the non-object area. After fine-tuning, the prediction result is also significantly improved, making the prediction of the background area more accurate.

[0048] S2. Extract features of the fused image through a Res-Unet model to obtain a predicted graph of the single-frame stripe image.

[0049] In one embodiment of the present invention, after a single-frame stripe image passes through the TCF model, 384×384×3 data is obtained. The original Resnet18 usually performs average pooling and full connection in the output layer to achieve multi-classification output. In order to adapt to the depth prediction of a single-frame stripe image, this application constructs a U-shaped network structure based on Resnet18. Resnet18 is used as the encoder and its output layer is transformed. The decoder is composed of convolutional layers and upsampling. First, the three common blocks in the network are introduced.

[0050] Specifically, the Res-Unet model includes: a first residual block, a second residual block and a conv-relu block.

[0051] The present application utilizes the first residual block, the second residual block and the conv-relu block to encode and decode the fused image to obtain a predicted graphic of the single-frame stripe image.

[0052] Specifically, the fused image is passed through the first residual block to perform a convolution operation with a convolution kernel size of 3×3, a step size of 1, and a padding of 1 so that the size and number of channels of the feature map remain unchanged;

[0053] The fused image is processed by the second residual block through two paths so that the size of the feature map becomes half of the original one and the number of channels becomes twice the original one. One path undergoes a convolution operation with a convolution kernel of 3×3, a step size of 2, and a padding of 1, and the other path undergoes a convolution operation with a convolution kernel of 1×1, a step size of 2, and a padding of 0.

[0054] The fused image is subjected to a 1×1 convolution operation using the conv-Relu block and then subjected to Relu processing for nonlinear transformation without changing the size of the feature map and the number of channels.

[0055] In one embodiment of the present invention, in the first residual block, 3×3 conv indicates a convolution operation with a convolution kernel size of 3×3, s=1, i.e., a step size of 1, and p=1, i.e., a padding of 1. After the H×W×N data passes through the first residual block, the size and number of channels of the feature map remain unchanged, and it is still a tensor of H×W×N. In the second residual block, the number of channels becomes twice the original. It is worth noting that the H×W×N data entering the second residual block will be processed by two paths together. One path passes through a convolution operation with a convolution kernel of 3×3, a step size of 2, and a padding of 1. After the convolution operation, the size of the feature map becomes half of the original, generating H / 2×W / 2×2N data; the other path passes through a convolution operation with a convolution kernel of 1×1, a step size of 2, and a padding of 0. After the convolution operation, the size of the feature map becomes half of the original. The remaining convolution operations with convolution kernels of 3×3, a step size of 1, and a padding of 1 will not change the size and number of channels of the feature map. When the data enters the second residual block, the size of the feature map is reduced to half of the original size, and the number of channels is increased to twice the original size. That is, the H×W×N data becomes the output of H / 2×W / 2×2N after passing through the second residual block.

[0056] In addition, the conv-Relu block in this application uses a 1×1 convolution operation followed by a Relu process. This operation does not change the size of the feature map and the number of channels. The main purpose is to introduce nonlinear transformations to improve the network's ability to express complex features. Thus, in the encoding stage of this application, layer0 accepts a three-channel image of size 384×384×3 as input.

[0057] In one embodiment of the present invention, the above three-channel image first undergoes a convolution operation with a 7×7 convolution kernel, a step size of 2, and a padding of 3 to generate a 192×192×64 tensor0. Tensor0 is downsampled by maximum pooling to obtain a 96×96×64 tensor, which enters the first residual block in layer1 to generate a 96×96×64 tensor. The tensor enters the conv-relu block to obtain a 96×96×64 tensor1, and enters the second residual block to reach layer2. For layer2, the 96×96×64 data first enters the second residual block to generate a 48×48×128 tmp_tensor, and the tmp_tensor enters the conv-relu block to obtain a 48×48×128 tensor2, and enters the second residual block to reach layer3. For layer3, the 48×48×128 data first enters the second residual block to generate a 24×24×256 tmp_tensor, and the tmp_tensor enters the conv-relu block to obtain a 24×24×256 tensor2, and then enters the second residual block to reach layer4. For layer4, the 24×24×256 data first enters the second residual block to generate a 12×12×512 tmp_tensor, and the tmp_tensor enters the conv-relu block to obtain a 12×12×512 tensor4. In the decoding stage, the operation of each layer is similar, that is, starting from tensor4_up, upsampling is performed and concatenated with tensor3 to generate tmp_cat_tensor, and then enters the conv-relu block to generate tensor3_up. Repeat the operation for tensor3_up, and so on, gradually restoring the spatial resolution of the feature map and completing the fusion of information. Until reaching layer_initial, through 3×3 convolution, i.e. Relu activation, 384×384×64 data is generated, and finally a single-channel image is obtained through convolution. This application inputs the three-channel image into a U-shaped network with ResNet-18 as the encoder for feature extraction, and obtains a depth map as output. Since ResNet-18 has been pre-trained on a large-scale dataset, this method based on knowledge transfer and network migration makes full use of the common features learned in image recognition tasks, which not only helps the model converge faster, reduces training time and computational costs, but also improves the generalization ability of the model.

[0058] In order to verify the effectiveness of the solution of this application, a projection-camera system was established. A projector and a camera were placed on an optical platform. The optical centers of the two were in the same plane, and the optical axes of the two formed a certain angle so that the field of view contained the complete object. The optical axis of the camera was perpendicular to the object plane, and the distance from the camera to the object plane was about 0.5m.

[0059] The camera uses BASLER acA2440-20gm, whose CMOS sensor size is 8.4×7.1 (mm), CMOS resolution is 2448×2048, equipped with a 12mm lens, and controlled by Pylon Viewer software; the projector uses M11B (LC)-DEMO, whose image DMD sensor size is 10.368×5.832 (mm), DMD resolution is 1920×1080, equipped with LED light source, signal output interface is HDMI, and controlled by SLMOutputApp software.

[0060] In this study, Matlab was used to generate a set of four-step phase-shift fringe patterns containing four frequencies, in which the stripes of different frequencies have different widths. The size of each fringe pattern is 1080×1920 pixels, and a total of 16 images were generated. These fringe patterns were projected onto the object to be tested in turn to collect the deformed fringe patterns. The different columns abcd are fringe patterns collected by projecting stripes with stripe widths of 1920, 200, 40, and 8 (unit: pixel) onto the object to be tested in turn. The smaller the stripe width, the denser the deformed stripes and the greater the number. In the same column, the same frequency is projected from top to bottom and generated in turn. Phase-shifted sinusoidal fringes, the folded phase is calculated by the four-step phase shift method . For folded phase The unwrapped phase is calculated by the time unwrapping algorithm Place planes every 25 mm on the optical platform, calculate the unfolded phases of the seven plane positions in turn, and then perform a second-order fitting on the unfolded phase-depth to obtain the mapping relationship between the two, as shown in Figure 2 shown.

[0061] This application obtains a total of 1062 sets of scene data, the data set includes 5 different objects, these objects will appear separately, such as Figure 3 As shown by a and b in ; or in combination, such as Figure 3As shown in c to i in the figure, the object was rotated 360° during the data collection process to collect object data from different angles. Each set of data includes a single-channel fringe image in PNG format (384×384 resolution) and a depth matrix in MAT format (384×384), where the fringe image is a deformed fringe image collected by projecting a fringe with a fringe width of 8 onto the object and modulating it. These data are randomly divided into three parts, including 646 training sets, 208 validation sets, and 208 test sets.

[0062] In addition, this scheme conducts experiments to compare the depth prediction results of three methods, which are: ① TCF-ResNet of this scheme, ② Res-Unet with the same ResNet-18 encoder as TCF-ResNet, and ③ Unet with the same number of layers as Res-Unet.

[0063] A variety of test objects were used in the experiment, including a David plaster statue, a cat-shaped ceramic, a spherical standard part, a metal convex standard part, etc.

[0064] like Figure 4 As shown in the figure, the columns from left to right are Ground Truth, Unet prediction results, Res-Unet prediction results, and TCF-ResNet prediction results. In the Unet prediction results, there are still cases where deformation stripes are retained, as shown in box 1 in b1, b2, and b3. These residual deformation stripes often appear more on arc lines. The residual stripes indicate that the Unet model did not correctly identify and predict all deformation stripes. However, the recognition performance of Res-Unet and TCF-ResNet for deformation stripes is significantly better than that of Unet, as shown in boxes c1, c2, d1, and d2. Deformed stripes at the same position can be recognized and depth predicted. In addition, Unet's prediction results also have uneven patches, as shown in boxes b2 and b3. These patches indicate that the depth value of the area has undergone a mutation compared with the surrounding area, resulting in discontinuous depth results, which will seriously affect the details of 3D reconstruction. Res-Unet and TCF-ResNet have obvious improvements, as shown in boxes c2, c3, d2, and d3. The depth change is obviously more reasonable than the Unet model. Through the above observations and comparisons, it is found that compared with Unet, Res-Unet and TCF-ResNet have stronger performance and potential in the task of stripe structured light prediction.

[0065] Although both Res-Unet and TCF-ResNet have overcome the Unet's inability to fully recognize and predict deformed stripes and the error of obvious depth jumps, the prediction results of Res-Unet still have deformation defects, as shown in the c3 and c4 boxes, where c4 is an enlarged view of the c3 box. The prediction results of TCF-ResNet have been significantly improved, as shown in d3 and d4, where d4 is an enlarged view of the d3 box. Compared with the arrows d3 and d4, the sphere predicted by TCF-ResNet is more regular and has no obvious deformation defects. By observing the prediction results between the sphere standards, it can be seen that compared with Unet, Res-Unet and TCF-ResNet have stronger performance and potential in the task of stripe structured light prediction.

[0066] By comparing the depth prediction results of the three methods for a single-frame stripe image, we can find that the prediction results of the TCF-ResNet model are significantly better than those of Unet and Res-Unet. The TCF-ResNet model is able to identify all deformed stripes for prediction, and the prediction results do not show uneven patches representing depth mutations, and there is no obvious shape distortion for special shapes such as spheres. These observations show that the TCF-ResNet model has a clear advantage in edge prediction when processing single-frame stripe structured light depth prediction tasks, providing more reliable results for 3D reconstruction.

[0067] The TCF-ResNet model proposed in the prediction method of a single-frame stripe image in this application realizes end-to-end prediction of directly mapping the stripe image to a depth map. This method significantly simplifies the entire process, improves efficiency, and makes it more suitable for fast depth estimation and real-time three-dimensional reconstruction. In addition, with the application of neural networks, the anti-noise performance is also enhanced, making the prediction results under different conditions more reliable. Global information such as the overall shape and contour of the object to be measured is introduced through Transformer, and the image channel is designed to fuse the global information with the original single-channel image to obtain a three-channel image. This design enables the model to understand the input image more comprehensively, thereby more accurately predicting the shape of the object to be measured, significantly enhancing the model's perception ability in different scenes and different objects, thereby improving the accuracy and robustness of single-frame stripe image depth prediction. By introducing the image channel design of global information, and then using ResNet-18 as the backbone, ResNet-18 replaces the encoder of Unet, and gives full play to the potential of the pre-trained model. This improvement not only accelerates the training convergence speed of the model, but also significantly improves the depth prediction performance, with the plane accuracy prediction error being about 6μm and the sphere radius prediction error being about 90μm.

[0068] The present application provides a prediction device for a single-frame stripe image, comprising: an acquisition module, used to automatically identify an object in a single-frame stripe image using a TCF module, and fuse the overall shape and contour information of the object with the original single-channel image, wherein the TCF module is obtained based on a DPT pre-training model; an execution module, used to extract features from the fused image through a Res-Unet model to obtain a predicted graph of the single-frame stripe image.

[0069] In some embodiments, the acquisition module includes: a first processing submodule, used to perform fine-tunedDPT processing on a single-frame stripe image to obtain the overall shape and contour information of the single-frame stripe image; and a second processing submodule, used to fuse the overall shape and contour information of the single-frame stripe image with the original single-channel image.

[0070] In some implementations, the Res-Unet model includes: a first residual block, a second residual block, and a conv-relu block.

[0071] In some embodiments, the processing module includes: a third processing submodule, configured to encode and decode the fused image using the first residual block, the second residual block and the conv-relu block to obtain a predicted image of the single-frame stripe image.

[0072] In some embodiments, it also includes: passing the fused image through the first residual block to perform a convolution operation with a convolution kernel size of 3×3, a step size of 1, and a padding of 1 to keep the size and number of channels of the feature map unchanged; a second execution submodule, used to pass the fused image through the second residual block through dual-path joint processing to make the size of the feature map half of the original, and the number of channels twice the original number of channels, one path passing through a convolution operation with a convolution kernel of 3×3, a step size of 2, and a padding of 1, and the other path passing through a convolution operation with a convolution kernel of 1×1, a step size of 2, and a padding of 0; a third execution submodule, used to use a conv-Relu block to perform a 1×1 convolution operation on the fused image and then perform Relu processing to perform a nonlinear transformation, without changing the size and number of channels of the feature map.

[0073] The TCF-ResNet model proposed in the prediction device of the single-frame stripe image in this application realizes end-to-end prediction of directly mapping the stripe image to the depth map. This method significantly simplifies the entire process and improves efficiency, making it more suitable for fast depth estimation and real-time three-dimensional reconstruction. In addition, with the application of neural networks, the anti-noise performance is also enhanced, making the prediction results under different conditions more reliable. Global information such as the overall shape and contour of the object to be measured is introduced through Transformer, and the image channel is designed to fuse the global information with the original single-channel image to obtain a three-channel image. This design enables the model to understand the input image more comprehensively, thereby more accurately predicting the shape of the object to be measured, significantly enhancing the model's perception ability in different scenes and different objects, thereby improving the accuracy and robustness of the single-frame stripe image depth prediction. By introducing the image channel design of global information, and then using ResNet-18 as the backbone, ResNet-18 replaces the encoder of Unet, and gives full play to the potential of the pre-trained model. This improvement not only accelerates the training convergence speed of the model, but also significantly improves the depth prediction performance, with the plane accuracy prediction error being about 6μm and the sphere radius prediction error being about 90μm.

[0074] To solve the above technical problems, an embodiment of the present invention also provides a computer device. The computer device includes a processor, a non-volatile storage medium, a memory and a network interface connected through a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database and a computer-readable instruction, and the database may store a control information sequence. When the computer-readable instruction is executed by the processor, the processor can implement an image processing method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor can execute an image processing method. The network interface of the computer device is used to connect and communicate with the terminal. In this embodiment, the processor is used to execute the specific contents of the acquisition module and the processing module, and the memory stores the program code and various types of data required to execute the above modules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all sub-modules in the image processing method, and the server can call the program code and data of the server to execute the functions of all sub-modules.

[0075] The present invention also provides a storage medium storing computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the image processing method described in any of the above embodiments.

[0076] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0077] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0078] The above descriptions are only some embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A single-frame stripe image prediction method, characterized in that: include: Automatically identify objects in a single-frame stripe image using a TCF module, and fuse the overall shape and contour information of the object with the original single-channel image, wherein the TCF module is obtained based on a DPT pre-training model; The fused image is subjected to feature extraction by a Res-Unet model to obtain a predicted graph of the single-frame stripe image; The Res-Unet model includes: a first residual block, a second residual block and a conv-relu block, and the fused image is subjected to feature extraction through the Res-Unet model to obtain a predicted graph of the single-frame stripe image, including: Using the first residual block, the second residual block and the conv-relu block, the fused image is encoded and decoded to obtain the predicted depth of the single-frame stripe image; The method further includes: performing a convolution operation on the fused image through the first residual block with a convolution kernel size of 3×3, a step size of 1, and a padding of 1 so that the size and the number of channels of the feature map remain unchanged; The fused image is processed by the second residual block through two paths so that the size of the feature map becomes half of the original one and the number of channels becomes twice the original one. One path undergoes a convolution operation with a convolution kernel of 3×3, a step size of 2, and a padding of 1, and the other path undergoes a convolution operation with a convolution kernel of 1×1, a step size of 2, and a padding of 0. The fused image is subjected to a 1×1 convolution operation using the conv-Relu block and then subjected to Relu processing for nonlinear transformation without changing the size and number of channels of the feature map.

2. The prediction method according to claim 1, characterized in that: The method of automatically identifying an object in a single-frame fringe image by using the TCF module and fusing the overall shape and contour information of the object with the original single-channel image includes: Preprocessing the single-frame fringe image using fine-tuned DPT to obtain the overall shape and contour information of the single-frame fringe image; The overall shape and contour information of the single-frame fringe image are fused with the original single-channel image.

3. A prediction device for a single-frame stripe image, characterized in that: include: An acquisition module, used for automatically identifying an object in a single-frame stripe image using a TCF module, and fusing the overall shape and contour information of the object with the original single-channel image, wherein the TCF module is obtained based on a DPT pre-training model; An execution module, used for extracting features of the fused image through a Res-Unet model to obtain a predicted graph of the single-frame stripe image; The Res-Unet model includes: a first residual block, a second residual block and a conv-relu block; the execution module includes: a third processing submodule, which is used to encode and decode the fused image using the first residual block, the second residual block and the conv-relu block to obtain the predicted depth of the single-frame stripe image; and also includes: A first execution submodule is used to perform a convolution operation with a kernel size of 3×3, a step size of 1, and a padding of 1 on the fused image through the first residual block so that the size and the number of channels of the feature map remain unchanged; The second execution submodule is used to process the fused image through the second residual block through a dual path so that the size of the feature map becomes half of the original one and the number of channels becomes twice the original number of channels, wherein one path undergoes a convolution operation with a convolution kernel of 3×3, a step size of 2, and a padding of 1, and the other path undergoes a convolution operation with a convolution kernel of 1×1, a step size of 2, and a padding of 0; The third execution submodule is used to perform a 1×1 convolution operation on the fused image using a conv-Relu block and then perform Relu processing to perform a nonlinear transformation without changing the size of the feature map and the number of channels.

4. The prediction device according to claim 3, characterized in that The acquisition module comprises: A first processing submodule, used for preprocessing a single-frame fringe image using a fine-tuned DPT to obtain the overall shape and contour information of the single-frame fringe image; The second processing submodule is used to fuse the overall shape and contour information of the single-frame fringe image with the original single-channel image.

5. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the single-frame stripe image prediction method as described in any one of claims 1 to 2.

6. A storage medium storing computer-readable instructions, which, when executed by one or more processors, enables the one or more processors to perform the steps of the single-frame stripe image prediction method as described in any one of claims 1 to 2.

Images