Single-frame three-dimensional imaging method based on multi-frequency RGB stripe compression

Abstract

The invention discloses a single-frame three-dimensional imaging method based on multi-frequency RGB (red, green and blue) stripe compression. According to the method, three groups of structured light stripe patterns with relatively prime frequencies and different directions are respectively embedded into RGB channels by utilizing channel characteristics of a color camera, the patterns are switched at a high speed by a projector, dynamic superposition is completed within single-frame exposure time of the camera, space compression acquisition of time domain information is realized, and a color superposition image fused with multiple pieces of time slice information is obtained. According to the invention, a multi-stage deep neural network architecture is designed to realize separation of fringe images of a color superposition image at different moments, phase information is accurately extracted from the decomposed fringe images, and finally unambiguous high-speed three-dimensional imaging in dynamic scenes at different moments is realized. According to the method, the time resolution of three-dimensional imaging can be remarkably improved without improving the frame rate of the camera, meanwhile, the space precision and the system stability are considered, and good universality and expandability are achieved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computational optical imaging and three-dimensional reconstruction, and particularly relates to a single-frame three-dimensional imaging method based on multi-frequency RGB fringe compression. BACKGROUND

[0002] Three-dimensional imaging technology has wide application prospects in industrial measurement, biomechanical analysis, motion capture, intelligent manufacturing and other fields. Among them, structured light three-dimensional imaging has become one of the mainstream methods due to its non-contact, high precision and high density advantages. Fringe projection profilometry (FPP) as an important implementation form of structured light technology can realize high-precision reconstruction of object surface topography by projecting coded fringe patterns and phase decoding of reflected images.

[0003] However, the traditional FPP method generally relies on a multi-frame image acquisition process, such as multi-step phase shift encoding or spatial projection encoding, which requires the image sensor to continuously capture multiple time fringe images to accurately complete phase calculation and three-dimensional reconstruction. This multi-frame dependent mechanism is prone to image mismatch, fringe misplacement, motion blur and other problems when facing high-speed moving objects or dynamic scenes, seriously affecting the stability and accuracy of the reconstruction results.

[0004] In order to improve the measurement speed, in recent years, scholars have proposed a compression encoding method based on frequency domain multiplexing, which realizes single-frame three-dimensional imaging by superimposing multiple frequency fringe patterns in one frame of image. However, single-channel frequency domain multiplexing is limited by frequency interval and phase decoding stability, and the problem of spectral aliasing is difficult to avoid, resulting in a clear contradiction between compression ratio and decoding accuracy, making it difficult to meet the demand of high spatiotemporal resolution measurement.

[0005] At the same time, the ability of deep learning technology in image decomposition and phase decoding gradually emerges, providing a new way for three-dimensional reconstruction algorithm. Some researches try to build a neural network model to directly regress phase information from composite fringe images, and have achieved preliminary results, but there are still the following main problems: (1) the encoding structure is single, which cannot effectively utilize color channel resources; (2) there is a lack of background and crosstalk suppression mechanism, and the image decoding quality is unstable; (3) the model structure has high coupling degree, and the generalization ability is weak, which is difficult to extend to multi-direction and multi-frequency mixed encoding scene.

[0006] Therefore, there is an urgent need for a three-dimensional imaging method that can efficiently encode multi-frequency and multi-direction information in a single frame of image and realize high-precision decoding through deep neural network, to realize time super-resolution three-dimensional reconstruction under ordinary hardware conditions, break through the speed bottleneck and decoding limit of traditional FPP method, and have stronger robustness and adaptability. SUMMARY

[0007] The application aims to provide a single-frame three-dimensional imaging method based on multi-frequency RGB fringe compression.

[0008] Step 1: Constructing a plurality of fringe pattern sequences, the fringe pattern sequence including a fringe pattern sequence A composed of RGB oblique fringes with three frequencies and different inclination angles; inserting a full white image between adjacent fringe images in the fringe pattern sequence A to form a fringe pattern sequence B; and a fringe pattern sequence C composed of multi-frequency multi-step phase shift fringes corresponding to all fringe patterns of the fringe pattern sequence A;

[0009] Step 2: Constructing a plurality of training data sets, the specific method being:

[0010] A projector projects the fringe pattern sequence A to a scene to be measured, and a color superimposed image is obtained by synchronously shooting under long exposure state through an RGB camera; and the fringe pattern sequence B is projected to the same scene to be measured, and a purified superimposed image is obtained by synchronously shooting under short exposure state through the RGB camera and by calculation and processing; the color superimposed image and the purified superimposed image constitute a data set I;

[0011] The projector projects the fringe pattern sequence B to the scene to be measured, and a purified superimposed image and a purified fringe image are obtained by synchronously shooting under short exposure state through the RGB camera and by calculation and processing, respectively; the purified superimposed image and the corresponding purified fringe image constitute a data set II;

[0012] The projector projects the fringe pattern sequence B to the scene to be measured, and a purified fringe image is obtained by synchronously shooting under short exposure state through the RGB camera and by calculation and processing; and the fringe pattern sequence C is projected to the same scene to be measured, and a wrapped phase image and an absolute phase image are obtained by synchronously shooting under short exposure state through the RGB camera and by calculation and processing; the purified fringe image and the corresponding wrapped phase image and absolute phase image constitute a data set III;

[0013] Step 3: constructing and training the multi-stage neural network; the multi-stage deep neural network architecture comprises a background crosstalk removal network Network1, a superposition image decomposition network Network2 and a direction discrimination multi-expert network Network3; the background crosstalk removal network Network1 is trained using color superposition images in a data set I as input and corresponding purified superposition images as labels; the superposition image decomposition network Network2 is trained using purified superposition images and their spectrum images in a data set II as input and purified fringe images as labels; and the direction discrimination multi-expert network Network3 is trained using purified fringe images in a data set III as input and corresponding wrapped phase images and absolute phase images as labels;

[0014] Step 4: a new color superposition image modulated by an object surface is photographed, and the trained background crosstalk removal network Network1, superposition image decomposition network Network2 and direction discrimination multi-expert network Network3 are sequentially input, to obtain wrapped phase images and rough absolute phase images at different time points, jointly estimate phase orders and calculate a series of high-precision absolute phase images; the high-precision absolute phase images are converted into depth images at multiple time points through a stereo phase unwrapping algorithm combined with system calibration information, and are converted into three-dimensional topography using a three-dimensional reconstruction algorithm, so that three-dimensional information decoding at multiple time points can be realized for a single color superposition image.

[0015] Compared with the prior art, the present application has the following advantages: the present application innovatively proposes a frequency-direction joint fringe encoding method based on RGB three channels, different frequency and direction fringe patterns are embedded into red, green and blue channels respectively, the physical isolation characteristics between color channels are used, and the overlapping problem of multi-frequency fringes in the single channel spectrum in the traditional frequency domain multiplexing is effectively solved. The encoding strategy can significantly improve the time compression ratio and three-dimensional information carrying capacity of a single image under the premise of maintaining the fringe solvability, and realizes high-density embedding of fringe patterns in multiple directions and multiple frequencies. Compared with the traditional method which needs to collect multiple images to complete high-precision reconstruction, the present application can complete phase extraction and three-dimensional reconstruction in multiple time segments only by relying on a single color superposition image, which provides a new solution for high-speed measurement of dynamic processes and breaks through the physical bottleneck caused by the frame rate of the image sensor.

[0016] The decoding network proposed in this invention comprises three functionally distinct and synergistic sub-networks: a background and crosstalk removal network, an image decomposition network, and a direction discrimination expert network. Each module performs tasks such as background crosstalk removal, superimposed image decoding, and direction-aware phase decoding in color striped images. By introducing a background crosstalk suppression mechanism, the influence of background interference and channel crosstalk in long-exposure acquisition is effectively resolved, improving image quality and stripe signal contrast. Simultaneously, the direction discrimination expert network can automatically identify the dominant stripe direction in the image, guiding the phase decoding process to the corresponding sub-network according to the direction, avoiding phase jumps and decoding errors caused by direction aliasing, and enhancing the system's adaptability to real-world scenarios with complex textures and poor structural continuity.

[0017] The method proposed in this invention can be directly deployed in existing structured light 3D imaging systems without replacing existing imaging hardware. The entire system only requires a common color camera and a commercial DLP projector to acquire a single frame of composite color image in a single exposure. It then uses a neural network to complete phase retrieval and 3D shape reconstruction at multiple frequencies and directions, significantly simplifying system design and synchronization control processes. This solution eliminates the need for high-speed image sensors or dedicated optics, avoiding high cost, high power consumption, and complex hardware integration issues. It boasts excellent platform compatibility and integration flexibility, making it suitable for various scenarios requiring high spatiotemporal resolution, such as industrial measurement, structural monitoring, and motion analysis. It possesses broad engineering application potential and industrialization prospects. Attached Figure Description

[0018] Figure 1 This is a flowchart of a single-frame 3D imaging method based on multi-frequency RGB stripe compression.

[0019] Figure 2 A schematic diagram of the Network1 structure for background crosstalk removal.

[0020] Figure 3 A schematic diagram of the direction determination gating module in the Network3 multi-expert network for direction determination.

[0021] Figure 4 The reconstruction process and results of a single-frame 3D imaging method based on multi-frequency RGB stripe compression are shown in the figure. Detailed Implementation

[0022] This invention proposes a single-frame 3D imaging method based on multi-frequency RGB stripe compression, the process of which is as follows: Figure 1 As shown. The overall process includes the following four steps:

[0023] Step 1: Construct a stripe pattern sequence;

[0024] The system consists of one color camera and one DLP projector. In the stage of constructing the sequence of fringe patterns and training the data set, the projector projects the sequence of RGB fringe patterns onto the surface of the object to be measured. Each group of patterns encodes sinusoidal fringe signals with different frequencies and different directions by using red, green and blue color channels respectively.

[0025] The specific fringe images to be projected in the sequence of fringe patterns I are as follows:

[0026] The sequence of fringe patterns A composed of RGB slanted fringes with different inclination angles at three frequencies is designed as follows:

[0027]

[0028] where a = b = 0.5 are the average value and modulation, (x p ,y p ) are the pixel coordinates of the projected image, and the subscript m represents the index of the projected pattern, m = 1, 2, …, 3M. The phase is defined as:

[0029]

[0030] The angle direction θ n is defined as:

[0031]

[0032] where n represents the sequence of the direction of the fringe, n = 1, 2, …, M, represents the period of the red low-frequency fringe in the x and y directions, λ R represents the period of the red low-frequency fringe in its angle direction, represents the period of the blue medium-frequency fringe, λ B represents the period of the blue medium-frequency fringe in its angle direction represents the period of the green high-frequency fringe, λ G represents the period of the green high-frequency fringe in its angle direction. To ensure that the pattern does not become blurred in the entire projection area, the period of the fringe needs to satisfy the principle of minimizing the projection distance, i.e.:

[0033]

[0034] where W x H is the size of the projected pattern, and LCM(·) represents the least common multiple.

[0035] The specific fringe images to be projected in the sequence of fringe patterns B are as follows:

[0036] ​The stripe pattern sequence B is composed of stripes of RGB slanted stripes with different tilt angles at three frequencies, and a full white image is inserted between adjacent stripes to form the stripe pattern sequence B The stripe pattern sequence B can be expressed as

[0037]

[0038] where a = b = 0.5 is the average value and the modulation degree, is defined as formula (2), (x p ,y p ) is the pixel coordinate of the projection image, and subscript m represents the projection pattern index, m = 1, 2, …, 3M.

[0039] The to-be-projected stripe image in the stripe pattern sequence C is specifically:

[0040] The stripe pattern sequence C is composed of multi-frequency and multi-step phase shift stripes corresponding to RGB slanted stripes with different tilt angles at three frequencies, and the designed stripe pattern The stripe pattern sequence C can be expressed as:

[0041] The definition of the nth step phase shift stripe image in the cth channel c ∈ {R, G, B} of the gth frequency group g ∈ {L, M, H} (low, medium, high) and the Mth direction is as follows:

[0042]

[0043] where the parameter meanings are: c ∈ {R, G, B}: color channel index; g ∈ {L, M, H}: frequency group index sequence number, respectively representing three different frequencies.

[0044] N L = 3, N M = 3, N H = 12: phase shift step. θ m : the Mth stripe direction. a (c) , b (c) : background light intensity and modulation degree under channel c.

[0045] Step 2: Construct a data set for network training;

[0046] A sequence of fringe patterns A, B, C obtained in step 1 is used to construct a plurality of training data sets. The training data set includes a color superimposed image obtained by synchronously capturing a color camera in a long exposure state when a projector projects a sequence of fringe patterns A. The projector projects a sequence of fringe patterns B and synchronously captures a color camera in a short exposure state, and calculates and processes to obtain a purified superimposed image and a purified fringe image. The projector projects a sequence of fringe patterns B and synchronously captures a color camera in a short exposure state, and calculates and processes to obtain a wrapped phase image and an absolute phase image, specifically:

[0047] The specific method for constructing data set I is:

[0048] A sequence of fringe patterns A composed of RGB slanted fringes with different inclination angles at three frequencies is projected at high speed by a projector to the surface of the object to be measured, and a color camera is used to capture a color superimposed image I fused with multiple time segments in a long exposure mode sum ; Under the same object and scene to be measured, a sequence of fringe patterns B composed of RGB slanted fringes with different inclination angles at three frequencies and inserted with a full white image is projected at high speed by a projector to the surface of the object to be measured, and a color camera is used to capture a series of color single fringe images and corresponding background images of the color single fringe images in a short exposure mode; a series of purified fringe images are obtained by subtracting the corresponding background images of the color single fringe images from the series of color single fringe images, and a purified superimposed image is obtained by superimposing the corresponding purified fringe images. Finally, a series of color superimposed images I sum are used as input, and a series of purified superimposed images are used as labels to construct data set I.

[0049] The specific method for constructing data set II is:

[0050] A sequence of fringe patterns composed of RGB slanted fringes with different inclination angles at three frequencies and inserted with a full white image is projected at high speed by a high-speed projector to the surface of the object to be measured, and a color camera is used to capture a series of color single fringe images and corresponding background images of the color single fringe images in a short exposure mode; a series of purified fringe images are obtained by subtracting the corresponding background images of the color single fringe images from the series of color single fringe images, and a purified superimposed image is obtained by superimposing the corresponding purified fringe images Finally, a series of purified superimposed images are used as input, and the purified fringe images are used as labels to construct data set II.

[0051] The specific method for constructing data set III is:

[0052] The high-speed projector is used to project the stripe pattern sequence B on the surface of the object to be measured. The same method as that for obtaining the data set II is used to obtain the purified stripe image And in the same scene, the stripe pattern sequence C composed of the multi-frequency multi-step phase-shift stripes corresponding to all the stripes of the stripe pattern sequence A. During the projection, the RGB color camera synchronously collects the corresponding color image sequence, and records the modulated stripe image of the object surface at each time.

[0053] For the same channel c, the same frequency group g, and the same direction θ m Under the N g Step phase shift image, the multi-step phase shift algorithm can be used to calculate the wrapped phase The formula is:

[0054]

[0055] Wherein the parameter meanings are: c∈{R,G,B}: color channel index; g∈{L,M,H}. In the same direction and the same channel, the high-frequency absolute phase can be recovered by the multi-frequency phase unwrapping formula The formula is:

[0056]

[0057] Wherein is the phase order, and the calculation method is as follows:

[0058]

[0059] All Images are arranged in a series of wrapped phase images φ m All Images are arranged in a series of absolute phase images Φ m .

[0060] Finally, the purified stripe image is taken as the input, the wrapped phase image φ m and the absolute phase image Φ m corresponding to the purified stripe image are taken as the labels, and a data set III is constructed;

[0061] Step 3: Construct and train the basic network model;

[0062] A multi-stage neural network is constructed and the network is trained. The multi-stage deep neural network structure includes a background crosstalk removal network Network1, a superimposed image decomposition network Network2, and a direction discrimination multi-specialist network Network3.

[0063] The background crosstalk removal network Network1 takes a series of color superimposed images I sum As input, the input channel is 1, and the output is a purified superimposed image, which is used for subsequent decoding of the purified fringe image. The background crosstalk removal network Network1 is as shown in Figure 2 As shown in the figure, a multi-path convolutional network structure is adopted, including four parallel convolutional processing paths, which respectively extract image features at different scale resolutions. The first path is a shallow detail extraction path, which directly receives the input color composite fringe image without downsampling operation, and extracts spatial detail features at the original resolution of the image through two consecutive 3x3 convolutional layers and a residual module. The second path is a shallow-middle layer feature extraction path. First, the input image is subjected to a maximum pooling operation to realize 2 times downsampling, compressing the image size while expanding the receptive field. Then, two residual convolution modules are sequentially passed through to extract the medium-scale structure and color-related features of the image. The third path is a middle-deep layer semantic extraction path, which performs two maximum pooling operations on the input image to realize 4 times downsampling, extracting higher semantic level background interference patterns and cross-region low-frequency interference distribution information. The fourth path is a deep abstract semantic path, which realizes 8 times downsampling through three maximum pooling operations, significantly compressing the spatial size while expanding the receptive field range, and is used to model the global light distribution, background intensity offset and large-scale crosstalk pattern in the image.

[0064] After the above four paths complete feature extraction at their respective scales, they are all restored to the original image resolution through nearest neighbor interpolation, and then channel dimension splicing is performed in the feature fusion module to form a unified high-dimensional fused feature map, which is used by the subsequent convolutional compression and decoder reconstruction module. This network structure has good multi-scale expression ability and cross-scale alignment characteristics, and is used to extract common features of cross-channel frequency interference to achieve the removal of zero-frequency background and RGB crosstalk signals. The superimposed image decomposition network Network2 takes the purified superimposed image and its frequency spectrum as joint input, uses the purified fringe image as label, and trains Network2 through supervised learning. The superimposed image decomposition network Network2 includes two parallel sub-paths, which are spatial domain encoding path and frequency domain encoding path, respectively, for extracting structural features and frequency features of the image.

[0065] The spatial domain path directly takes the purified superposition image as input and sequentially passes through multiple convolution units for feature extraction. Each convolution unit is composed of a 3x3 convolution layer, a Batch Normalization layer, and a ReLU activation function. A max-pooling operation is inserted between the convolution units, which is used to compress the spatial dimension layer by layer and extract spatial features such as local structure, texture, and edge direction. The frequency domain path takes the frequency spectrum of the superposition image as input. This path is symmetrical in structure to the spatial domain path and is also composed of multiple convolution units and down-sampling modules. Its role is to extract the main frequency components, directional distribution, and energy concentration areas in the frequency spectrum to identify the direction and amplitude features corresponding to different frequency fringes. After the two paths output feature maps, they are spliced in the channel dimension and sent to the fusion module to obtain fusion features.

[0066] The fusion module is designed with two levels of convolution compression units: the first level uses 1x1 convolution for fast compression of the number of channels and preservation of the interaction between channels, and the second level uses 3x3 convolution to further fuse the neighborhood context information and multi-scale feature response. Between the two convolution layers, there are Batch Normalization layers and ReLU activation functions to improve the non-linear expression ability and training stability of the network. The fusion features are finally sent to the decoding module. The decoding module adopts a symmetrical structure layer by layer and is composed of multiple decoding layers. Each decoding layer includes an up-sampling operation, a cross-layer feature connection operation, and two consecutive convolution residual blocks. Each convolution residual block is composed of two serial 3x3 convolution layers, equipped with Batch Normalization and ReLU activation functions, and introduces residual connection to avoid gradient disappearance and improve detail recovery ability. The decoding module gradually restores the spatial resolution of the feature map, preserves the directional fringe edge structure and coherent features, and outputs the purified fringe image.

[0067] The direction discrimination multi-expert network Network3 takes the purified fringe image as input and outputs the purified fringe image corresponding wrapped phase and absolute phase. The direction discrimination multi-expert network Network3 is composed of a direction discrimination gating module and M phase decoding sub-networks.

[0068] The direction discrimination gating module takes the purified fringe image as input and outputs a weight vector for fringe direction perception The direction discrimination gate module is based on a lightweight neural network architecture design, adopts a left-to-right feature extraction and mapping structure, mainly includes multiple convolution layers, pooling layers, flattening layers and fully connected layers, and layer by layer extracts the directional response information in the input image. The gate module first extracts and dimensionally compresses the original fringe image through the encoding path containing multiple 3x3 convolution layers and maximum pooling layers. The convolution layer adopts the ReLU activation function, and is normalized by the Batch Normalization layer to enhance the feature expression ability and network stability. The feature extraction path performs a 2x spatial downsampling operation layer by layer, and finally obtains a deep directional feature map with a size of W / 32xH / 32x32C at the fifth layer. Then the spatial features are mapped into a one-dimensional vector through the flattening layer, and input into two fully connected layers to realize the regression prediction of the direction weight. The output Mx1 weight vector represents the response strength of the current input fringe image in M candidate directions, which can reflect the angle of the input fringe to match the appropriate phase decoding subnetwork for decoding.

[0069] The M phase decoding subnetworks are responsible for the phase decoding of the M kinds of angle purification fringe images. Each phase decoding subnetwork takes an angle purification fringe image as input, and takes the wrapped phase and absolute phase corresponding to the input as label for training. Each phase decoding subnetwork includes a wrapped phase prediction branch and a rough absolute phase prediction branch; adopts a parallel structure of shared encoder+branch decoder. First, the features are extracted through the shared multi-scale encoder, which is composed of multiple encoding modules containing 3x3 convolution, Batch Normalization and ReLU activation function, combined with downsampling operation to extract local texture and directional structure information layer by layer. The wrapped phase prediction branch inputs the decoupled single-frequency fringe image, and the label is the wrapped phase image φ m for training the wrapped phase prediction branch; the rough absolute phase prediction branch takes the same input image I m as the basis, and the label is the absolute phase image Φ m for training the rough absolute phase prediction branch. It is used to estimate the phase order. This subnetwork structure effectively improves the phase prediction accuracy under frequency multiplexing conditions through module collaborative modeling.

[0070] After the Network3 is trained, the following functions will be realized: the purification fringe image I is input into the direction discrimination gate module for direction judgment, and a weight vector w with a length of M is output, and the position sequence number a of the maximum value in the vector is identified, so that the purification fringe image I is input into the a-th phase decoding subnetwork, and the network can predict the wrapped phase image φ m and the rough absolute phase image Φc m

[0071] Step 4: Color superposition image is taken and decoding is pushed;

[0072] The fringe pattern sequence I is projected onto the surface of the object to be measured, and a color superposition image that fuses multi-frequency time domain information is obtained by using a color camera for long exposure shooting in a fringe modulation state The color superposition image is obtained The cleaned superposition image is obtained by inputting the trained background crosstalk removal network Network1 The cleaned superposition image is obtained And The frequency spectrum obtained after Fourier transform is input into the superposition image decomposition network Network2 to obtain a series of cleaned fringe images corresponding to the cleaned superposition image The cleaned fringe images corresponding to the cleaned superposition image The result reflects the spatial structure information of multiple modulation fringe components hidden in the composite image. The series of cleaned fringe images are input into the direction discrimination multi-expert network Network3, and the direction discrimination gate module outputs a direction perception weight vector of each cleaned fringe image, each weight being used for vector weighted activation of M phase decoding sub-networks to predict the wrapped phase image φ corresponding to each cleaned fringe image The wrapped phase image φ 1 m , and the coarse absolute phase image Φ c1 m . According to the following formula, the final high-quality absolute phase image is calculated by phase order restoration

[0073]

[0074] By sequentially executing Network1→Network2→Network3, multiple cleaned fringe images and their corresponding high-quality absolute phase images can be inferred from one color superposition image

[0075] Subsequently, a planar calibration plate is used to jointly calibrate the projector and the camera to obtain their respective intrinsic parameters and extrinsic parameters between them. By projecting a fringe image of a specific frequency on a plane of different known heights, a mapping relationship between the two-dimensional phase image and the known depth value is constructed. For the mth high-quality absolute phase image By fitting the absolute phase values corresponding to the multiple known depth calibration planes with the actual heights, a mapping relationship from the high-quality absolute phase image to the depth value Z mMapping function of (x, y):

[0076]

[0077] wherein is a single variable function constructed based on least square method, polynomial fitting or lookup table interpolation, suitable for phase encoding rules under different fringe directions and frequencies.

[0078] According to the intrinsic matrix of the camera and the imaging model, the two-dimensional pixel point (x, y) and its corresponding depth value Z m (x, y) is back-projected into three-dimensional space coordinates (X, Y, Z).

[0079] To further verify the universality and robustness of the method of the application on multiple types of targets, a data set covering a variety of typical static and dynamic structures is constructed for network training and inference testing. The selected targets include portrait half-length portraits, antique column head models, spherical bodies, metal workpieces, stepped surfaces and other objects, covering from regular geometry to free-form surfaces with complex textures, ensuring good diversity in geometric shape, texture distribution and light response. The experiment uses a digital light processing (DLP) system to project a total of 24 frames of RGB fringe patterns, including 8 directions and 3 frequencies of combinations, respectively encoded in the red, green and blue color channels. The corresponding fringe periods are 1 / 72, 1 / 78 and 1 / 84, respectively, and the frequency spectra are isolated from each other to avoid interference and aliasing.

[0080] The image acquisition process is completed by single exposure, and the camera is set to an exposure time of 16.7 ms, during which the 24 projected structured light patterns are recorded completely and fused into a single color superimposed image. We show four representative sequences as shown in Figure 4 (a) by decoupling the cleaned fringe images obtained by the Network2 network. Further extraction of the 24 cleaned fringe images corresponding to high-quality absolute phase images for three-dimensional reconstruction, the three-dimensional reconstruction results of the four representative sequences are shown in Figure 4 (b), clearly reflecting the dynamic topography evolution process of the target in the 16.7 ms window.

[0081] The experimental results show that: under the condition of only single exposure, the application can recover 24 time segments of high-precision three-dimensional structure from a single color superimposed image, significantly improving the temporal resolution of three-dimensional imaging without sacrificing accuracy. The reconstruction results are structurally complete and continuous, suitable for time super-resolution three-dimensional imaging tasks in dynamic scenes.

Claims

1. A single-frame three-dimensional imaging method based on multi-frequency RGB stripe compression, characterized in that, include: Step 1: Construct multiple stripe pattern sequences, including stripe pattern sequence A composed of RGB diagonal stripes of three frequencies and different tilt angles; inserting a pure white image between adjacent stripe images in stripe pattern sequence A to form stripe pattern sequence B; and stripe pattern sequence C composed of multi-frequency multi-step phase-shifted stripes corresponding to all stripe patterns in stripe pattern sequence A. Step 2: Construct multiple training datasets, specifically as follows: A stripe pattern sequence A is projected onto the scene under test by a projector, and a color superimposed image is obtained by synchronously capturing the image with an RGB camera under long exposure. A stripe pattern sequence B is projected onto the same scene under test, and a cleaned superimposed image is obtained by synchronously capturing the image with an RGB camera under short exposure and processing the image. Data set I consists of the color superimposed image and the cleaned superimposed image. The projector projects a stripe pattern sequence B onto the scene under test, and simultaneously captures images using an RGB camera under short exposure conditions. The cleaned overlay image and the cleaned stripe image are calculated and processed separately to obtain the cleaned overlay image and the corresponding cleaned stripe image. The cleaned overlay image and the corresponding cleaned stripe image constitute dataset II. The projector projects a stripe pattern sequence B onto the scene under test, and simultaneously captures the image using an RGB camera under short exposure conditions, and calculates and processes the image to obtain a cleaned stripe image; the projector also projects a stripe pattern sequence C onto the same scene under test, and simultaneously captures the image using an RGB camera under short exposure conditions, and calculates and processes the image to obtain a wrapped phase image and an absolute phase image; the cleaned stripe image and the corresponding wrapped phase image and absolute phase image constitute dataset III. Step 3: Construct a multi-stage neural network and train it; the multi-stage deep neural network architecture includes a background crosstalk removal network (Network 1), an overlay image decomposition network (Network 2), and a direction discrimination multi-expert network (Network 3). The background crosstalk removal network Network1 is trained using the color overlay images in dataset I as input and the corresponding cleaned overlay images as labels. The Network2 superimposed image decomposition network is trained using the cleaned overlay images and their spectrograms from Dataset II as input and the cleaned stripe images as labels. The orientation discrimination multi-expert network Network3 was trained using the clean stripe images from Dataset III as input, and the corresponding wrap phase images and absolute phase images as labels. Step 4: Capture a new color overlay image modulated by the object surface, and sequentially input it into the trained background crosstalk removal network (Network1), overlay image decomposition network (Network2), and orientation discrimination multi-expert network (Network3) to obtain wrapping phase images and coarse absolute phase images at different times. Jointly estimate the phase order and calculate a series of high-precision absolute phase images. Convert the high-precision absolute phase images into depth images at multiple times using a stereo phase unfolding algorithm combined with system calibration information, and convert them into three-dimensional topography using a three-dimensional reconstruction algorithm. This allows for the decoding of three-dimensional information at multiple times from a single frame of color overlay image.

2. The single-frame three-dimensional imaging method based on multi-frequency RGB stripe compression according to claim 1, characterized in that, Striped image in striped pattern sequence A Specifically: Where a = b = 0.5 are the average value and the modulation index, respectively, (x p ,y p ) represents the pixel coordinates of the projected image, and the subscript m represents the index of the projected pattern, m = 1, 2, ..., 3M, where M is the number of diagonal stripe directions; For phase, specifically: θn is the tilt angle of the diagonal stripes, specifically: Where n represents the direction sequence of the stripes, n = 1, 2, ..., M. These represent the periods of the red low-frequency stripes in the x and y directions, respectively. The period of the blue mid-frequency stripes in the x and y directions is denoted as . The period of the green high-frequency stripe in the x and y directions; The fringe period setting satisfies the principle of minimizing the projected distance, that is: Where W×H is the size of the projected pattern, and LCM(·) represents the least common multiple.

3. The single-frame three-dimensional imaging method based on multi-frequency RGB stripe compression according to claim 1, characterized in that, Striped image in striped pattern sequence B Specifically: Where a = b = 0.5 are the average value and the modulation index, respectively, (x p ,y p ) represents the pixel coordinates of the projected image, N is a non-zero natural number, and the subscript m represents the index of the projected pattern, m = 1, 2, ..., 3M.

4. The single-frame three-dimensional imaging method based on multi-frequency RGB stripe compression according to claim 1, characterized in that, Striped image in striped pattern sequence C Specifically: Where c∈{R,G,B}, R,G,B represent the red, green and blue channels respectively; Indicates low frequency. Indicates intermediate frequency. Indicates high frequency; N L N M N H These represent the phase shift steps corresponding to three different frequencies, θ m Let a be the angle of the m-th fringe. (c) b (c) These represent the background light intensity and modulation index under channel c, respectively.

5. The single-frame three-dimensional imaging method based on multi-frequency RGB stripe compression according to claim 1, characterized in that, The specific method for constructing dataset I is as follows: A stripe pattern sequence A is projected at high speed onto the surface of the object under test using a projector, and a color superimposed image I is directly acquired using a color camera in long exposure mode. sum ; Under the same test object and test scene, a projector is used to project a stripe pattern sequence B onto the surface of the test object at high speed. A color camera is used to continuously acquire color single stripe images and the background image corresponding to the color single stripe images in short exposure mode. Subtracting the background image corresponding to the color single-stripe image from the color single-stripe image yields the purified stripe image. The purified stripe images obtained from each projected stripe pattern sequence B are then superimposed to obtain the purified superimposed image. The superposition formula is: Where m = 1, 2, ..., 3M; Finally, a series of color overlay images I sum As input, a series of cleaned overlay images As labels, construct dataset I.

6. The single-frame three-dimensional imaging method based on multi-frequency RGB stripe compression according to claim 1, characterized in that, The specific method for constructing dataset II is as follows: A high-speed projector projects a stripe pattern sequence B onto the scene under test. A series of color single-stripe images and their corresponding background images are acquired using a color camera in short-exposure mode. After subtracting the background images from the series of color single-stripe images, a series of clean stripe images are obtained. The clean stripe images are then superimposed to obtain a clean superimposed image. Finally, a series of purified overlay images As input, and the clean striped image As labels, construct dataset II.

7. The single-frame three-dimensional imaging method based on multi-frequency RGB stripe compression according to claim 1, characterized in that, The specific method for constructing dataset III is as follows: The stripe pattern sequence B is projected onto the surface of the object under test using a high-speed projector, and the purified stripe image is obtained using the same method as acquiring dataset II. And project a stripe pattern sequence C under the same test scene. During the projection, the RGB color camera synchronously acquires the corresponding color image sequence and records the modulated stripe image of the object surface at each moment. For the same channel c, the same frequency group g, and the same direction θ m N below g Multi-step phase-shifting image, using a multi-step phase-shifting algorithm to calculate the wrap-around phase. The specific formula is as follows: The parameters are defined as follows: c∈{R,G,B} represents the color channel index, where R, G, and B correspond to the red, blue, and green indices, respectively; g∈{L,M,H}, where L, M, and H correspond to the low, medium, and high frequency indices, respectively; the absolute phase of the high frequency is recovered using a multi-frequency phase expansion formula under the same direction and channel. The formula is: in The phase series is calculated as follows: By all The images are arranged in the order of projection to form a series of wrapping phase images φ m , by all A series of absolute phase images Φ are formed according to the order of projection. m . Ultimately, the image was purified with stripes. As input, the wrapped phase image φ corresponding to the purified stripe image. m and absolute phase image Φ m As labels, construct dataset III.

8. The single-frame three-dimensional imaging method based on multi-frequency RGB stripe compression according to claim 1, characterized in that, The background crosstalk removal network Network1 adopts a multi-path convolutional network structure, including four parallel convolutional processing paths, which extract image features at different scale resolutions. The first path is a shallow detail extraction path, which directly receives the input color composite stripe image and extracts spatial detail features at the original resolution of the image through two consecutive 3×3 convolutional layers and a residual module. The second path is a mid-shallow feature extraction path, which first performs a max pooling operation on the input image to achieve a 2x downsampling, compressing the image size while expanding the receptive field. Then, it sequentially extracts the mid-scale structure and color-related features of the image through two residual convolutional modules. The third path is a mid-deep semantic extraction path, which performs two max pooling operations on the input image to achieve a 4x downsampling, extracting background interference patterns and cross-regional low-frequency interference distribution information at a higher semantic level. The fourth path is a deep abstract semantic path, which achieves an 8x downsampling through three max pooling operations. After feature extraction at their respective scales, the four paths described above restore the original image resolution through nearest neighbor interpolation. Then, in the feature fusion module, channel dimensions are stitched together to form a unified high-dimensional fusion feature map.

9. The single-frame three-dimensional imaging method based on multi-frequency RGB stripe compression according to claim 1, characterized in that, The overlay image decomposition network Network2 includes two parallel sub-paths, namely a spatial domain coding path and a frequency domain coding path, which are used to extract the structural features and frequency features of the image. The spatial domain path takes the cleaned and overlaid image as input and extracts features sequentially through multiple convolutional units. Each convolutional unit consists of a 3×3 convolutional layer, a BatchNormalization layer, and a ReLU activation function. Max pooling operations are inserted between the convolutional units to compress the spatial dimension layer by layer and extract spatial features such as local structure, texture, and edge orientation. The frequency domain path takes the spectrogram of the overlaid image as input. The frequency domain path is symmetrical to the spatial domain path and also consists of multiple convolutional units and a downsampling module. After the two paths output feature maps, they are concatenated along the channel dimension and fed into the fusion module to obtain fused features. The fusion module is designed with two levels of convolutional compression units: the first level uses 1×1 convolution to quickly compress the number of channels and retain the interaction between channels, and the second level uses 3×3 convolution to further fuse neighborhood context information and multi-scale feature responses; a Batch Normalization layer and a ReLU activation function are provided between the two convolutional layers. The fused features are finally fed into the decoding module, which adopts a layer-by-layer symmetrical structure, consisting of multiple stacked decoding layers. Each decoding layer includes an upsampling operation, a cross-layer feature connection operation, and two consecutive convolutional residual blocks. Each convolutional residual block consists of two cascaded 3×3 convolutional layers equipped with Batch Normalization and ReLU activation functions, and residual connections are introduced to avoid gradient vanishing and improve detail recovery. The decoding module gradually restores the spatial resolution of the feature map, preserves the directional stripe edge structure and coherent features, and outputs a cleaned stripe image.

10. The single-frame three-dimensional imaging method based on multi-frequency RGB stripe compression according to claim 1, characterized in that, The direction discrimination multi-expert network Network3 consists of a direction discrimination gating module and M phase decoding sub-networks. The direction discrimination gating module purifies the stripe image. As input, the output is a weight vector for stripe direction sensing. The orientation discrimination gating module is based on a lightweight neural network architecture and adopts a left-to-right feature extraction and mapping structure, including multiple convolutional layers, pooling layers, flattening layers, and fully connected layers, to extract directional response information from the input image layer by layer. The gating module first performs feature extraction and dimensionality reduction compression on the original stripe image through an encoding path containing multiple 3×3 convolutional layers and max pooling layers. The convolutional layers use the ReLU activation function and are normalized by a Batch Normalization layer. The feature extraction path performs a 2x spatial downsampling operation layer by layer, finally obtaining a deep orientation feature map with a size of W / 32×H / 32×32C in the fifth layer. Then, the spatial features are mapped into a one-dimensional vector through a flattening layer and fed into two fully connected layers to achieve regression prediction of orientation weights. The output M×1 weight vector represents the response intensity of the current input stripe image in M ​​candidate directions, reflecting the angle of the input stripes, and is used to match the phase decoding subnetwork for decoding; The M phase decoding sub-networks are responsible for phase decoding of cleaned stripe images at M different angles. Each phase decoding sub-network takes a cleaned stripe image at one angle as input and is trained using the corresponding wrapping phase and absolute phase as labels. Each phase decoding sub-network includes a wrapping phase prediction branch and a coarse absolute phase prediction branch. A parallel structure of shared encoder + branch decoder is adopted. First, features are extracted by a shared multi-scale encoder, which consists of multiple encoding modules containing 3×3 convolution, Batch Normalization, and ReLU activation functions. Combined with downsampling operations, local texture and orientation structure information are extracted layer by layer. The wrapping phase prediction branch is input with the decoupled single-frequency stripe image, labeled as the wrapping phase map φ of the corresponding orientation. m , used to train the wrap-around phase prediction branch; coarse absolute phase prediction branch, with the same input image I m Based on this, the absolute phase map Φ corresponding to the tag bit m Train a coarse absolute phase prediction branch to estimate the phase order.

Images