Active sub-pixel displacement fringe structured light super-resolution three-dimensional reconstruction method
By employing an active subpixel displacement fringe structured light super-resolution 3D reconstruction method, combined with an iterative back projection algorithm and a deep learning network, the bottleneck problem of high-frequency detail capture in traditional 3D reconstruction systems is solved, achieving efficient and low-cost high-resolution 3D reconstruction suitable for industrial measurement and other scenarios.
Patent Information
- Application Number
- CN202611173780.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-04
- Publication Date
- 2026-11-06
AI Technical Summary
Traditional 3D reconstruction systems are limited by the resolution of cameras and projectors and the performance of optical systems, making it difficult to improve the ability to capture high-frequency details. Furthermore, high-resolution cameras bring high costs, and existing super-resolution methods suffer from image registration errors and system complexity issues.
An active subpixel displacement fringe structured light super-resolution 3D reconstruction method is adopted. By acquiring a fringe image sequence containing subpixel displacement, high-resolution image reconstruction is achieved by using an iterative back projection algorithm and a deep learning-driven super-resolution network. Combined with data consistency constraints and pixel value range constraints, error propagation and computation are reduced.
Without increasing hardware costs, it achieves 3D reconstruction of high-frequency details, improves reconstruction accuracy and stability, reduces computing platform performance requirements, and is suitable for visual inspection and precision measurement scenarios.
Smart Images
Figure CN123052178A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of fringe projection 3D reconstruction and digital image processing technology. Specifically, it is a fringe projection super-resolution 3D reconstruction method based on controllable subpixel displacement driving, which can be directly applied to high-precision 3D reconstruction of object surface morphology, industrial measurement and other scenarios. Background Technology
[0002] Fringe projection profilometry, due to its high precision, efficiency, and strong environmental adaptability, has been widely applied in fields such as artificial intelligence, industrial defect detection, and medical imaging. However, limited by hardware conditions such as camera and projector resolution, pixel size, and optical system performance, traditional 3D reconstruction systems have inherent bottlenecks in high-frequency detail capture capabilities, making it difficult to further improve reconstruction accuracy and detail fidelity. However, using high-resolution cameras incurs significant product costs. To overcome hardware limitations and improve reconstruction accuracy, existing technologies often employ super-resolution methods to enhance image detail quality.
[0003] Classic image super-resolution techniques include interpolation algorithms, with bicubic interpolation being the most widely used. This algorithm utilizes cubic polynomial weighted interpolation of the 16 surrounding pixels to obtain a relatively smooth magnified image. However, because it only mines local pixel information and lacks the ability to restore high-frequency details, it is prone to edge blurring and phase shift when processing periodic striped images, affecting the accuracy of subsequent 3D reconstruction. Another super-resolution reconstruction method is to perform super-resolution processing on striped coded images captured by a camera. This typically employs a strategy of using high- and low-resolution camera pairs to collect the training data needed for deep learning models, but it introduces additional image registration errors into the system and increases the system's complexity. Summary of the Invention
[0004] The purpose of this invention is to address the shortcomings of the existing technology by proposing an active subpixel displacement fringe structured light super-resolution 3D reconstruction method. This method aims to acquire a dataset of high and low resolution fringe images that more closely approximate the scene, thereby achieving super-resolution 3D reconstruction with a small number of fringe images. This solves the problem of image pairing error in dual-camera acquisition strategies and is applicable to scenarios such as visual inspection and precision measurement.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: The present invention provides a super-resolution 3D reconstruction method using active sub-pixel displacement fringe structured light, comprising the following steps: Step 1: Obtain the stripe image sequence containing subpixel displacement priors { | },in, Indicates the first Frame low-resolution striped image; This represents the total number of striped images, and ,in, This is the super-resolution factor; Indicates the length of a low-resolution striped image. Indicates the width of a low-resolution striped image. Indicates the number of channels; Step 2: Based on data consistency constraints and pixel value range constraints, an iterative back-projection algorithm guided by sub-pixel displacement is used to process the low-resolution striped image { | High-resolution image estimation is performed to obtain a high-resolution stripe reconstruction image of the target. ; Step 3: Obtain the results following the process from Step 1 to Step 2. Zhang target high-resolution stripe reconstruction image { | }, and adopt N Step phase shift method for the first Zhang's high-resolution stripe reconstruction image Perform stripe phase analysis to output high-resolution molecular weight. and high-resolution denominator And further calculation and Mask threshold map and its mask diagram ; Step 4: Construct a stripe analysis super-resolution reconstruction network, including: shallow feature extraction, deep feature extraction modules, and a reconstruction module, and process the first frame of the low-resolution stripe image. The process was performed to obtain the predicted high-resolution molecular weight. and the predicted high-resolution denominator ; Step 5, based on and And predicting high-resolution molecular weight and high-resolution denominator The loss function of the fringe analysis super-resolution reconstruction network is constructed using equation (1). ; (1) Step 6: Iteratively train the fringe analysis super-resolution reconstruction network using the backpropagation algorithm and calculate the training loss. Adjust network parameters until... Once convergence is achieved, the trained fringe analysis super-resolution reconstruction model is used to perform high-resolution phase reconstruction on the input low-resolution fringe image, resulting in a high-resolution 3D reconstruction.
[0006] The super-resolution 3D reconstruction method of active sub-pixel displacement fringe structured light described in this invention is also characterized in that step 2 includes: Step 2.1: Define the current iteration number as... and initialize The total number of iterations is defined as ,by Using the reference frame as a reference, bicubic interpolation is used to perform super-resolution processing on the reference frame to obtain an initial high-resolution stripe image estimate. and as the first High-resolution stripe reconstruction image estimation under the next iteration ; Step 2.2, according to the first Frame subpixel displacement Using equation (2) Degradation is performed to generate the first The simulation of the nth iteration Frame low-resolution striped image : (2) In equation (2), Represents the displacement operator. Represents a fuzzy operator. Indicates the downsampling operator; Step 2.3: Calculate the first step using equation (3). The th iteration Frame residual : (3) Step 2.4: Use equation (4) to obtain the first... In the nth iteration Frame error compensation term : (4) In equation (4), Indicates the upsampling operator; The transpose operator of the fuzzy operator. Indicates transpose; Step 2.5: Use equation (5) to obtain the first... High-resolution stripe reconstruction image pre-estimation in the next iteration : (5) In equation (5), Indicates the relaxation factor; Step 2.6, using equation (6) to... After normalization, we obtain the first... High-resolution stripe reconstruction image estimation under the next iteration : (6) In equation (6), The projection operator for the pixel interval [0,1]; Step 2.7: Determine whether equation (7) is true. If it is true, then it means... Satisfy data consistency constraints, and As Otherwise, Assign to Then, return to step 2.2 and execute sequentially until... achieve until; (7) In equation (7), This is the preset tolerance threshold.
[0007] Furthermore, step 3 includes: Step 3.1, using equation (8) to... To represent; (8) In equation (8), To reconstruct the background light intensity of the high-resolution striped image of the target. The modulation of the target high-resolution stripe reconstructed image. for Phase shift amount, Given the initial phase, we have: (9) (10) (11) Step 3.2: Calculate using equations (12) and (13) respectively. corresponding molecular weight and denominator ; (12) (13) Step 3.3: Calculate the modulation index using formula (14) Thus, the mask threshold map can be obtained using equation (15). ; (14) (15) In equation (15), This indicates the mask factor that is set; Step 3.4, using equation (16) to... Pixel-by-pixel judgment is performed, and pixels that satisfy equation (16) are retained to obtain the mask image. ; (16) In equation (16), express The Middle Line number The mask value of the column, express The Middle Line number The pixel values of the column.
[0008] Furthermore, step 4 includes: Step 4.1, the shallow feature extraction module performs... Processing is performed to obtain initial eigenvalues. ,in, For the expanded number of channels, and 1; Step 4.2, the deep feature extraction module consists of... Each downsampling processing module, bottleneck layer and It consists of several upsampling processing modules, and processes the initial feature values. The process is performed to obtain the depth feature sequence of the downsampled stripe image. } and upsampled stripe image depth feature sequence },in, Indicates the first The depth features of the downsampled stripe image output by each downsampling module. Indicates the first The upsampling stripe image depth features output by each upsampling module; wherein, each downsampling processing module sequentially includes: one The residual block and a downsampling unit; each upsampling processing module includes, in sequence: an upsampling unit, a skip connection layer and a... Residual block; the bottleneck layer includes: A series Residual block; Step 4.3: The reconstruction module consists of a channel-expanded convolutional layer and a sub-pixel layer. The convolutional layer is composed of channels that extend the depth features of the S-th upsampled fringe. The process is performed to obtain the extended feature map. Then through the subpixel layer Will After pixel rearrangement, the output is the predicted high-resolution molecular weight. and the predicted high-resolution denominator ,in, This represents the number of channels in the expanded feature map, and , This is the upsampling factor.
[0009] Furthermore, step 4.2 includes: Step 4.2.1, when At that time, Input to the s-th downsampling processing module The residual block is processed to obtain the corresponding number. Each downsampled residual feature map ;Will Enter the first The downsampling unit in the downsampling processing module processes the data to obtain the first downsampling result. Depth feature map of each downsampled stripe image ; when At that time, the first Depth feature map of each downsampled stripe image Enter the first In each downsampling processing module, and sequentially through After processing the residual block and the downsampling unit, the following is obtained: Thus, by the first The output of the first downsampling processing module is the first... Depth feature map of each downsampled stripe image ; Step 4.2.2, the first Depth features of each downsampled stripe image The input is entered into the bottleneck layer and passes through sequentially. A series Deep feature extraction of residual blocks yields bottleneck residual feature maps. ; Step 4.2.3, when At that time, Enter the first The upsampling unit in the upsampling processing module processes the data to obtain the corresponding result. Upsampled feature maps ;Will Enter the first In the first skip connection layer, and with the first... Each downsampled residual feature map After adding pixel by pixel, input to the first pixel. In each upsampling processing module The residual block is processed to obtain the first... Each upsampled stripe depth feature map ; when At that time, the first Depth feature map of an upsampled stripe image Enter the first In each upsampling processing module, the data sequentially passes through an upsampling unit, a skip connection layer, and... After processing the residual blocks, we obtain Thus, by the first The output of the first upsampling processing module is the first Each upsampled stripe depth feature map .
[0010] Furthermore, in step 4.2.1... In each downsampling processing module The residual block is processed according to the following steps. Processing is performed to obtain the first... Each downsampled residual feature map : Step 4.2.1.1: Calculate the first step using equation (17). Preliminary convolutional feature maps : (17) In equation (17), Indicates a convolutional layer. This indicates a modified linear activation function. Indicates the first Subsampling Residual feature map, when season ; Step 4.2.1.2, using equation (18) to... Perform global average pooling to obtain the first... Pooling feature map : (18) In the formula, This is a global average pooling operation; These are the height and width of the feature map, respectively. for The Middle Line number The characteristic values of the column.
[0011] Step 4.2.1.3: Calculate the first step using equation (19). Each convolutional feature map : (19) Step 4.2.1.4: Use equation (20) to output the final result. Each downsampled residual feature map : (20) In equation (20), This indicates an element-wise multiplication operation along the channel dimension. This indicates an element-wise addition operation along the channel dimension. This represents the sigmoid activation function.
[0012] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program supporting the processor in performing the method described therein, and the processor is configured to execute the program stored in the memory.
[0013] The present invention discloses a computer-readable storage medium storing a computer program, characterized in that the computer program is executed by a processor to perform the steps of the method described thereon.
[0014] Compared with the prior art, the present invention has at least the following beneficial effects: 1. Controllable subpixel displacement at the acquisition end suppresses error: This invention replaces traditional high- and low-resolution camera pairs with a super-resolution industrial camera using an integrated piezoelectric ceramic displacement stage. This eliminates the extrinsic error, baseline error, and synchronization error introduced by the binocular imaging system at the hardware architecture level, suppressing the adverse effects of error propagation on subsequent 3D reconstruction from the source, and improving the stability and reliability of the reconstruction results.
[0015] 2. Iterative back projection lightweight adaptation significantly reduces the dependence on computing platform performance: The proposed subpixel displacement-guided iterative backprojection super-resolution reconstruction method (SDIB) directly embeds a sequence of multiple low-resolution striped images acquired through controllable subpixel displacement into an iterative backprojection framework. This method eliminates the need for complex deep learning model inference, achieving high-resolution image reconstruction solely through iterative optimization. The algorithm is simple in logic, computationally manageable, and does not rely on high-performance GPUs or server platforms. It can be directly deployed in embedded devices or camera hardware in industrial settings, significantly reducing the performance requirements of 3D reconstruction systems on computing platforms.
[0016] 3. Achieving higher resolution 3D reconstruction with a low-cost hardware architecture: This invention establishes a structured light super-resolution 3D reconstruction system based on a super-resolution industrial camera and projector. The proposed deep learning-driven super-resolution network for stripe-coded images (FASRNet) can perform super-resolution processing on low-resolution stripe images, restoring high-frequency detail information. Without increasing hardware costs, it achieves a 3D reconstruction effect with higher detail resolution, breaking through the bottleneck of traditional stripe projection systems limited by the resolution of imaging devices.
[0017] 4. Driven by a small number of stripe images, achieving high-efficiency and high-precision 3D reconstruction: The deep learning-driven super-resolution network for stripe-coded images (FASRNet) proposed in this invention, after being trained on a large-scale, high-quality dataset, only requires three low-resolution stripe-coded images as input to complete high-resolution reconstruction. It directly outputs the corresponding high-resolution numerator and denominator for use in subsequent phase calculation processes, which greatly reduces the number of stripe images required in the 3D reconstruction process, significantly reduces the time cost of data acquisition and subsequent processing, and effectively improves the overall reconstruction efficiency.
[0018] 5. Software and hardware co-optimization to improve adaptability for engineering applications in multiple scenarios: The Subpixel Displacement Guided Iterative Backprojection (SDIB) algorithm proposed in this invention, together with the Deep Learning-Driven Super-Resolution Network for Striped Co-coded Images (FASRNet), forms a complete hardware and software co-optimized solution: SDIB can be deployed in industrial camera systems with integrated controllable subpixel displacement stages, achieving high-precision multi-frame super-resolution reconstruction; while the Deep Learning-Driven Super-Resolution Network for Striped Co-coded Images can be directly applied to traditional FPP 3D imaging systems limited by low-resolution cameras, achieving high-resolution enhancement of striped images without additional hardware modifications. The two solutions complement each other, balancing the needs of static high-precision measurement and low-cost rapid reconstruction, significantly improving the versatility and engineering application adaptability of this invention in different industrial settings. Attached Figure Description
[0019] Figure 1 A schematic diagram of a structured light super-resolution 3D reconstruction method driven by controllable displacement. Figure 2 This is a schematic diagram of an iterative back-projection image super-resolution reconstruction method guided by subpixel displacement. Figure 3 A schematic diagram of a deep learning-driven super-resolution network structure for stripe-coded images; Figure 4 In this invention Schematic diagram of the residual module; Figure 5This is a comparison of the super-resolution reconstruction results of the USAF 1951 resolution board by the iterative back projection algorithm based on subpixel displacement guidance and the traditional Bicubic method in this embodiment. Figure 6 This is a comparison diagram of the modulation schemes obtained by reconstructing the network and other reconstruction methods in this embodiment; Figure 7 This is a comparison diagram of the 3D reconstructions obtained by the reconstruction network and other reconstruction methods in this embodiment; Figure 8 This is a diagram verifying the reconstruction accuracy of the standard step and the standard sphere in this embodiment. Detailed Implementation
[0020] In this embodiment, addressing the dependence of existing fringe projection contour reconstruction accuracy on high-resolution hardware and the cost issues arising from the computational demands of subsequent image processing on high-performance GPUs and service platforms, a subpixel displacement-guided iterative back-projection (SDIB) image super-resolution reconstruction method is proposed. This method utilizes prior information provided by controllable subpixel displacement at the acquisition end to synthesize a super-resolution image from a sequence of subpixel-displaced images. Furthermore, a fringe analysis super-resolution reconstruction network is used to significantly improve the effect of cross-frame image fusion, achieving high-resolution reconstruction of the original low-resolution fringe image sequence. This further enhances the fidelity of high-frequency fringe edges and detail regions, meeting the requirements of relative stability, repeatability, and traceability in industrial measurements, as well as the high-speed, high-detail resolution requirements of 3D reconstruction applications. This overcomes the limitations of existing hardware and software strategies in resolution improvement, acquiring high-quality, high-resolution fringe images in fringe projection 3D reconstruction scenarios, and achieving higher detail resolution in 3D topography reconstruction. Specific implementation details are as follows... Figure 1 As shown, the specific steps of this method are as follows: Step 1: Construct a fringe image acquisition system consisting of a super-resolution camera and a projector integrating a precision piezoelectric ceramic displacement platform. This system utilizes the piezoelectric ceramic displacement stage to drive a CMOS chip, achieving sub-pixel-level high-precision displacement to acquire a fringe image sequence containing sub-pixel displacement priors. | This provides a data foundation for subsequent super-resolution reconstruction; among which, Indicates the first Frame low-resolution striped image; This represents the total number of striped images, and ,in, This is the super-resolution factor; Indicates the length of a low-resolution striped image. Indicates the width of a low-resolution striped image. Indicates the number of channels.
[0021] Since the displacement trajectory is actively applied by the controller according to preset instructions, the displacement relationship between adjacent frames is deterministic and repeatable, providing a reliable explicit displacement prior for subsequent multi-frame fringe image fusion. During operation, the two-dimensional piezoelectric ceramic displacement platform drives the image sensor to make minute displacements along two mutually perpendicular planes, enabling the sensor to sequentially complete fringe image exposure and acquisition at multiple sub-pixel displacements. In this embodiment, a two-fold super-resolution magnification is achieved, and a single high-resolution fringe reconstruction acquires four low-resolution fringe images.
[0022] Step 2: Based on data consistency constraints and pixel value range constraints, an iterative back-projection algorithm guided by sub-pixel displacement is used to process the low-resolution striped image { Perform high-resolution image estimation, such as Figure 2 As shown, based on the prior knowledge of controllable subpixel displacement at the acquisition end, the fringe imaging process is modeled as a multi-frame degradation observation model. Under a unified iterative framework, the high-resolution fringe image estimate is gradually corrected, thereby obtaining the target high-resolution fringe reconstructed image. .
[0023] This method avoids the alignment errors caused by post-motion estimation in traditional multi-frame super-resolution and makes fuller use of complementary sampling information between multi-frame fringe images. Specifically, the reconstruction method works according to the process of "simulated degradation - residual calculation - backprojection compensation - iterative update": the current high-resolution estimated image is projected back into the low-resolution observation space to simulate degradation; the degradation result is compared with the original low-resolution fringe image through residual calculation to quantify the reconstruction error; then, the error is back-mapped to the high-resolution space and the estimated value is updated through the backprojection compensation step; further iterative updates introduce data consistency constraints and pixel value range constraints to ensure the stability and effectiveness of the iterative process, and finally achieve accurate restoration of the target high-resolution fringe image, as detailed below: Step 2.1: Define the current iteration number as... and initialize 0, defining the total number of iterations as ,by Using the reference frame as a reference, bicubic interpolation is used to perform super-resolution processing on the reference frame to obtain an initial high-resolution stripe image estimate. In the In the nth iteration, the nth generation is generated. The high-resolution fringe reconstruction image estimate under the next iteration is denoted as... .
[0024] Step 2.2, according to the first Frame subpixel displacement Using equation (1) based on the known displacement right Degradation processing is performed by projecting the current high-resolution estimated image onto the low-resolution observation space, thereby generating the second... The simulation of the nth iteration Frame low-resolution striped image : (1) In equation (1), Represents the displacement operator. Represents a fuzzy operator. This represents the downsampling operator.
[0025] Step 2.3: Calculate the first step using equation (2). The th iteration Frame residual Calculate the residual between the current high-resolution estimated image projected onto the low-resolution observation space and the original fringe image: (2) Step 2.4: Apply the residuals to the upsampling operator. Upgraded to the HR scale and implemented using the inverse shift operator Mapping it back to the reference HR coordinate system, and using equation (3) to obtain the first... In the nth iteration Frame error compensation term : (3) In equation (3), Indicates the upsampling operator; The transpose operator of the fuzzy operator. This indicates transpose.
[0026] Step 2.5: Introduce an error compensation term and update the HR image with a relaxation factor to replace the original high-resolution estimate. Use equation (4) to obtain the result. High-resolution stripe reconstruction image pre-estimation in the next iteration : (4) In equation (4), This represents the relaxation factor.
[0027] Step 2.6: Due to physical constraints in the actual imaging process, the updated HR image is projected into the legal pixel range, and equation (5) is used to... After normalization, we obtain the first... High-resolution stripe image estimation under the next iteration : (5) In equation (5), This represents the projection operator for the pixel interval [0,1].
[0028] Step 2.7: In each iteration, calculate the residual norm corresponding to all low-resolution stripe images to determine whether the data consistency constraint is satisfied, that is, whether equation (6) holds. If it holds, then it means that... Satisfy data consistency constraints, and As Otherwise, Assign to Then, return to step 2.2 and execute sequentially until... Reaching the maximum number of iterations until; (6) In equation (6), This is the preset tolerance threshold.
[0029] Step 3: Obtain the results following the process from Step 1 to Step 2. Zhang target high-resolution stripe reconstruction image { }, and adopt Step phase shift method for the first Zhang's high-resolution stripe reconstruction image Perform stripe phase analysis to output high-resolution molecular weight. and high-resolution denominator The overall modulation structure of the stripes is separated from local detail information to suppress the influence of noise and invalid regions, and further calculations are performed. and Mask threshold map and its mask diagram The details are as follows: Step 3.1, using equation (7) to... To represent; (7) In equation (7), To reconstruct the background light intensity of the high-resolution striped image of the target. The modulation of the target high-resolution stripe reconstructed image. for Phase shift amount, Given the initial phase, we have: (8) (9) (10) Step 3.2: Calculate using equations (11) and (12) respectively. corresponding molecular weight and denominator ; (12) (13) Step 3.3: Calculate the modulation index using formula (13) Thus, the mask can be obtained using equation (14). ; (13) (14) In equation (14), This indicates the mask factor that is set. The value can be adjusted according to the actual shooting scene.
[0030] Step 3.4: Use equation (15) to judge pixel by pixel. If the condition is met, keep the pixel; otherwise, ignore it to obtain the mask image. In equation (15), express The Middle Line number The mask value of the column, express The Middle Line number The pixel values of the column.
[0031] The absolute phase can be further calculated using the above formula. Through the pre-calibrated phase-space coordinate mapping relationship, the three-dimensional spatial coordinate information of the measured object can be calculated. To facilitate subsequent data processing and phase calculation, this invention pre-calculates the molecular weight and denominator required for absolute phase calculation, providing direct input data for solving the arctangent function.
[0032] Step 4: Construct a fringe analysis super-resolution reconstruction network, such as... Figure 3 As shown, it includes: shallow feature extraction, deep feature extraction modules, and a reconstruction module, and processes the first frame of low-resolution striped image. The process was performed to obtain the predicted high-resolution molecular weight. and the predicted high-resolution denominator This deep learning-driven super-resolution network for stripe-coded images (FASRNet), after being trained on a large-scale, high-quality dataset, can complete high-resolution reconstruction with only a small number of low-resolution stripe images, significantly reducing the number of images acquired and processing time. Details are as follows: Step 4.1, the shallow feature extraction module... Processing is performed to obtain initial eigenvalues. ,in, For the expanded number of channels, and 1.
[0033] Step 4.2: The inference time of deep learning networks increases with the number of network layers, while the feature extraction performance also improves with increasing network depth. Therefore, selecting a reasonable number of downsampling operations is crucial for balancing algorithm execution time and feature extraction accuracy. In this implementation case, the deep feature extraction module consists of three downsampling processing modules, a bottleneck layer, and three upsampling processing modules, as shown in the structure diagram below. Figure 3 As shown, and for the initial eigenvalues The process is performed to obtain the depth feature sequence of the downsampled stripe image. } and upsampled stripe image depth feature sequence },in, Indicates the first The depth features of the downsampled stripe image output by each downsampling module. Indicates the first The upsampling stripe image depth features output by each upsampling module; wherein, each downsampling processing module sequentially includes: one The residual block and a downsampling unit; each upsampling processing module includes, in sequence: an upsampling unit, a skip connection layer and a... Residual blocks; bottleneck layer includes: A series Residual block.
[0034] Step 4.2.1, when At that time, Enter the first In each downsampling processing module The residual block is processed to obtain the corresponding number. Each downsampled residual feature map ;Will Enter the first The downsampling unit in the downsampling processing module processes the data to obtain the first downsampling result. Depth feature map of each downsampled stripe image .
[0035] when At that time, the first Depth feature map of each downsampled stripe image Enter the first In each downsampling processing module, and sequentially through After processing the residual block and the downsampling unit, the following is obtained: Thus, by the first The output of the first downsampling processing module is the first... Depth feature map of each downsampled stripe image .
[0036] like Figure 4 As shown, where, The specific processing flow for residual blocks is as follows: Step 4.2.1.1: Calculate the first step using equation (16). Preliminary convolutional feature maps : (16) In equation (16), Indicates a convolutional layer. This indicates a modified linear activation function. Indicates the first Subsampling Residual feature map, when season .
[0037] Step 4.2.1.2, using equation (17) to... Perform global average pooling to obtain the first... Pooling feature map : (17) In the formula, This is a global average pooling operation; These are the height and width of the feature map, respectively. for The Middle Line number The characteristic values of the column.
[0038] Step 4.2.1.3: Calculate the first step using equation (18). Each convolutional feature map : (18) Step 4.2.1.4: Use equation (19) to output the final result. Each downsampled residual feature map : (19) In equation (19), This indicates an element-wise multiplication operation along the channel dimension. This indicates an element-wise addition operation along the channel dimension. This represents the sigmoid activation function.
[0039] Step 4.2.2, the first Depth feature map of stripe image The input is passed into the bottleneck layer and then sequentially through... A series Deep feature extraction of residual blocks yields bottleneck residual feature maps. .
[0040] Step 4.2.3, when At that time, Enter the first The upsampling unit in the upsampling processing module processes the data to obtain the corresponding result. Upsampled feature maps ;Will With input number In the first skip connection layer, and with the first... Each downsampled residual feature map After adding pixel by pixel, input to the first pixel. In each upsampling processing module The residual block is processed to obtain the first... Each upsampled stripe depth feature map .
[0041] when At that time, the first Depth feature map of an upsampled stripe image Enter the first In each upsampling processing module, and accordingly through the upsampling unit, skip connection layer and S After processing the residual blocks, we obtain Thus, by the first The output of the first upsampling processing module is the first Each upsampled stripe depth feature map .
[0042] Step 4.3: The reconstruction module consists of a channel-expanded convolutional layer and a subpixel layer. Composition, and through channel expansion of the convolutional layer to the first Each upsampled stripe depth feature The process is performed to obtain the extended feature map. Then through the subpixel layer Will After pixel rearrangement, the output is the predicted high-resolution molecular weight. and the predicted high-resolution denominator ,in, This represents the number of channels in the expanded feature map, and , The upsampling factor is 2 in this implementation example.
[0043] Step 5, based on and And predicting high-resolution molecular weight and high-resolution denominator The loss function of the fringe analysis super-resolution reconstruction network is constructed using equation (1). ; (20) Step 6: Iteratively train the fringe analysis super-resolution reconstruction network using the backpropagation algorithm and calculate the training loss. Adjust network parameters until... The trained fringe analysis super-resolution reconstruction model, after convergence, is used to perform high-resolution phase reconstruction on the input low-resolution fringe image, resulting in a high-resolution 3D reconstruction. This model achieves high-detail super-resolution reconstruction while reducing the number of acquired images, providing reliable, high-quality input for phase calculation and 3D reconstruction.
[0044] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.
[0045] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.
[0046] The effects of the present invention will be further illustrated below with experimental results: The proposed method for image super-resolution reconstruction based on subpixel displacement-guided IBP (Integrated Back Projection) in this invention acquires sequential images using an industrial camera with an integrated subpixel displacement platform and combines it with an iterative back projection (IBP) algorithm to achieve high-precision image reconstruction. Its reconstruction results are significantly superior to traditional Bicubic interpolation methods. To verify the performance advantages of this invention, Figure 5 The proposed method was compared with the original low-resolution image (LR) and the traditional Bicubic interpolation method. Through multi-dimensional results such as the overall reconstruction results, magnified local detail images, and grayscale profile curves of typical line bundle positions, the superiority of the present invention in resolution improvement and stripe detail reconstruction was fully demonstrated.
[0047] To further optimize super-resolution image details, this invention proposes a deep learning-driven super-resolution network for stripe-coded images (FASRNet). The designed network model is run on a host machine with an Intel(R) Xeon(R) Gold 6148 CPU (2.40 GHz) based on Python 3.12.7 and PyTorch 2.7.1, and training is accelerated using an NVIDIA GeForce RTX 4090 D graphics card. 900 sub-pixel displacement stripe-coded images of different scenes are acquired using a super-resolution industrial camera. These images are obtained through an iterative back projection (IBP) algorithm and used as training samples. Each training batch contains three images: the original low-resolution stripe image as input and the high-resolution molecular weight and high-resolution denominator as output. During network training, the Adam optimizer is used for optimization and gradient calculation and propagation. The initial learning rate is set to 1×10⁻⁶. -4 The learning rate is halved every 500 epochs.
[0048] To evaluate the performance of the proposed network in super-resolution, it was compared with several advanced methods. The proposed method was tested and compared with other fringe image super-resolution methods in terms of modulation error, absolute phase error, and 3D reconstruction results. Through comparison with various advanced algorithms, the proposed method can enhance the clarity of local structures while better preserving the smoothness of object surfaces, thus obtaining more realistic and detailed 3D reconstruction results. The comparison results of modulation reconstruction effects are shown below. Figure 6 The comparison results of the 3D reconstruction effects are shown in the figure. Figure 7 The reconstruction accuracy error was controlled within 70 μm. The reconstruction results for the standard sphere and the standard step are shown in the figure. Figure 8 ( Figure 8 In the table, a and b represent the metrological standard values of the standard sphere and standard step, respectively, and c and d represent the measured values of the standard sphere and standard step after processing by the experimental system and the super-resolution method, respectively. The comparison results of modulation error and phase error are shown in Table 1.
[0049] In Table 1, DdBSRNet is the super-resolution reconstruction method disclosed in the non-patent document "Super-resolution technique for dense 3D reconstruction in fringe projection profilometry"; SRPRNet is the super-resolution reconstruction method disclosed in the non-patent document "Super-Resolution Phase Retrieval Network for Single-Pattern Structured Light 3D Imaging"; SSSR-FPP (CNN1) is the super-resolution reconstruction method disclosed in the non-patent document "Single-shot super-resolved fringe projection profilometry (SSSR-FPP): 100,000 frames-per second 3D imaging with deep learning"; the method of this invention is a structured light super-resolution 3D reconstruction method driven by controllable subpixel displacement.
[0050] Table 1 Figure 5 In the scenario shown, the modulation scheme PSNR, SSIM, and absolute phase RMSE and MAE are... Figure 7 This paper presents a visual comparison between the proposed Deep Learning-Driven Super-Resolution Network for Striped Coded Images (FASRNet) and other methods. The reconstruction results show that, compared to the original low-resolution input, each super-resolution method improves the ability to represent the details of the 3D model to a certain extent, making the local structure clearer and the surface gradient changes more obvious. However, significant differences still exist between the different methods, with some exhibiting reduced smoothness, disordered reconstructed textures, and insufficient depiction of object detail areas. In contrast, the proposed FASRNet effectively suppresses the generation of pseudo-textures while improving the resolution of object details, and maintains surface smoothness better while enhancing the clarity of local structures, thus achieving a more realistic and delicate 3D reconstruction effect.
Claims
1. A super-resolution 3D reconstruction method using active subpixel displacement fringe structured light, characterized in that, Includes the following steps: Step 1: Obtain the stripe image sequence containing subpixel displacement priors { | },in, Indicates the first Frame low-resolution striped image; This represents the total number of striped images, and ,in, This is the super-resolution factor; Indicates the length of a low-resolution striped image. Indicates the width of a low-resolution striped image. Indicates the number of channels; Step 2: Based on data consistency constraints and pixel value range constraints, an iterative back-projection algorithm guided by sub-pixel displacement is used to process the low-resolution striped image { | High-resolution image estimation is performed to obtain a high-resolution stripe reconstruction image of the target. ; Step 3: Obtain the results following the process from Step 1 to Step 2. Zhang target high-resolution stripe reconstruction image { | }, and adopt N Step phase shift method for the first Zhang's high-resolution stripe reconstruction image Perform stripe phase analysis to output high-resolution molecular weight. and high-resolution denominator And further calculation and Mask threshold map and its mask diagram ; Step 4: Construct a stripe analysis super-resolution reconstruction network, including: shallow feature extraction, deep feature extraction modules, and a reconstruction module, and process the first frame of low-resolution stripe image. The process was performed to obtain the predicted high-resolution molecular weight. and the predicted high-resolution denominator ; Step 5, based on and And predicting high-resolution molecular weight and high-resolution denominator The loss function of the fringe analysis super-resolution reconstruction network is constructed using equation (1). ; (1) Step 6: Iteratively train the fringe analysis super-resolution reconstruction network using the backpropagation algorithm and calculate the training loss. Adjust network parameters until... Once convergence is achieved, the trained fringe analysis super-resolution reconstruction model is used to perform high-resolution phase reconstruction on the input low-resolution fringe image, resulting in a high-resolution 3D reconstruction.
2. The super-resolution 3D reconstruction method of active sub-pixel displacement fringe structured light according to claim 1, characterized in that, Step 2 includes: Step 2.1: Define the current iteration number as... and initialize The total number of iterations is defined as ,by Using the reference frame as a reference, bicubic interpolation is used to perform super-resolution processing on the reference frame to obtain an initial high-resolution stripe image estimate. and as the first High-resolution stripe reconstruction image estimation under the next iteration ; Step 2.2, according to the first Frame subpixel displacement Using equation (2) Degradation is performed to generate the first The simulation of the nth iteration Frame low-resolution striped image : (2) In equation (2), Represents the displacement operator. Represents a fuzzy operator. Indicates the downsampling operator; Step 2.3: Calculate the first step using equation (3). The th iteration Frame residual : (3) Step 2.4: Use equation (4) to obtain the first... In the nth iteration Frame error compensation term : (4) In equation (4), Indicates the upsampling operator; The transpose operator of the fuzzy operator. Indicates transpose; Step 2.5: Use equation (5) to obtain the first... High-resolution stripe reconstruction image pre-estimation in the next iteration : (5) In equation (5), Indicates the relaxation factor; Step 2.6, using equation (6) to... After normalization, we obtain the first... High-resolution stripe reconstruction image estimation under the next iteration : (6) In equation (6), The projection operator for the pixel interval [0,1] is used. Step 2.7: Determine whether equation (7) is true. If it is true, then it means... Satisfy data consistency constraints, and As Otherwise, Assign to Then, return to step 2.2 and execute sequentially until... achieve until; (7) In equation (7), This is the preset tolerance threshold.
3. The super-resolution 3D reconstruction method of active sub-pixel displacement fringe structured light according to claim 1, characterized in that, Step 3 includes: Step 3.1, using equation (8) to... To represent; (8) In equation (8), To reconstruct the background light intensity of the high-resolution striped image of the target. The modulation of the target high-resolution stripe reconstructed image. for Phase shift amount, Given the initial phase, we have: (9) (10) (11) Step 3.2: Calculate using equations (12) and (13) respectively. corresponding molecular weight and denominator ; (12) (13) Step 3.3: Calculate the modulation index using formula (14) Thus, the mask threshold map can be obtained using equation (15). ; (14) (15) In equation (15), This indicates the mask factor that is set; Step 3.4, using equation (16) to... Pixel-by-pixel judgment is performed, and pixels that satisfy equation (16) are retained to obtain the mask image. ; (16) In equation (16), express The Middle Line number The mask value of the column, express The Middle Line number The pixel values of the column.
4. The super-resolution 3D reconstruction method of active sub-pixel displacement fringe structured light according to claim 1, characterized in that, Step 4 includes: Step 4.1, the shallow feature extraction module performs... Processing is performed to obtain initial eigenvalues. ,in, For the expanded number of channels, and 1; Step 4.2, the deep feature extraction module consists of... Each downsampling processing module, bottleneck layer and It consists of several upsampling processing modules, and processes the initial feature values. The process is performed to obtain the depth feature sequence of the downsampled stripe image. } and upsampled stripe image depth feature sequence },in, Indicates the first The depth features of the downsampled stripe image output by each downsampling module. Indicates the first The upsampling stripe image depth features output by each upsampling module; wherein, each downsampling processing module sequentially includes: one The residual block and a downsampling unit; each upsampling processing module includes, in sequence: an upsampling unit, a skip connection layer and a... Residual block; the bottleneck layer includes: A series Residual block; Step 4.3: The reconstruction module consists of a channel-expanded convolutional layer and a sub-pixel layer. The convolutional layer is composed of channels that extend the depth features of the S-th upsampled fringe. The process is performed to obtain the extended feature map. Then through the subpixel layer Will After pixel rearrangement, the output predicts a high-resolution molecular weight. and the predicted high-resolution denominator ,in, This represents the number of channels in the expanded feature map, and , This is the upsampling factor.
5. The super-resolution 3D reconstruction method of active sub-pixel displacement fringe structured light according to claim 4, characterized in that, Step 4.2 includes: Step 4.2.1, when At that time, Input to the s-th downsampling processing module The residual block is processed to obtain the corresponding number. Each downsampled residual feature map ;Will Enter the first The downsampling unit in the downsampling processing module processes the data to obtain the first downsampling result. Depth feature map of each downsampled stripe image ; when At that time, the first Depth feature map of each downsampled stripe image Enter the first In each downsampling processing module, and sequentially through After processing the residual block and the downsampling unit, the following is obtained: Thus, by the first The output of the first downsampling processing module is the first... Depth feature map of each downsampled stripe image ; Step 4.2.2, the first Depth features of each downsampled stripe image The input is entered into the bottleneck layer and passes through sequentially. A series Deep feature extraction of residual blocks yields bottleneck residual feature maps. ; Step 4.2.3, when At that time, Enter the first The upsampling unit in the upsampling processing module processes the data to obtain the corresponding result. Upsampled feature maps ;Will Enter the first In the first skip connection layer, and with the first... Each downsampled residual feature map After adding pixel by pixel, input to the first pixel. In each upsampling processing module The residual block is processed to obtain the first... Each upsampled stripe depth feature map ; when At that time, the first Depth feature map of an upsampled stripe image Enter the first In each upsampling processing module, the data sequentially passes through an upsampling unit, a skip connection layer, and... After processing the residual blocks, we obtain Thus, by the first The output of the first upsampling processing module is the first Each upsampled stripe depth feature map .
6. The super-resolution 3D reconstruction method of active sub-pixel displacement fringe structured light according to claim 5, characterized in that, Step 4.2.1 In each downsampling processing module The residual block is processed according to the following steps. Processing yields the first... Each downsampled residual feature map : Step 4.2.1.1: Calculate the first step using equation (17). Preliminary convolutional feature maps : (17) In equation (17), Indicates a convolutional layer. This indicates a modified linear activation function. Indicates the first Subsampling Residual feature map, when season ; Step 4.2.1.2, using equation (18) to... Perform global average pooling to obtain the first... Pooling feature map : (18) In the formula, This is a global average pooling operation; These are the height and width of the feature map, respectively. for The Middle Line number The characteristic values of the column. Step 4.2.1.3: Calculate the first step using equation (19). Each convolutional feature map : (19) Step 4.2.1.4: Use equation (20) to output the final result. Each downsampled residual feature map : (20) In equation (20), This indicates an element-wise multiplication operation along the channel dimension. This indicates an element-wise addition operation along the channel dimension. This represents the sigmoid activation function.
7. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports a processor in executing the method of any one of claims 1-6, the processor being configured to execute the program stored in the memory.
8. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by a processor to perform the steps of the method according to any one of claims 1-6.