Computer system of composite stripe multi-mode imaging model based on parallel single pixel

Abstract

The invention discloses a computer system of a composite stripe multi-mode imaging model based on parallel single pixels. The computer system comprises a control unit, a projector and a color industrial camera, the control unit performs multi-modal imaging through the following method: constructing at least one stripe pattern with a specified frequency, and synthesizing the stripe patterns into an RGB composite stripe pattern; collecting a color composite image modulated by the surface of the target object; acquiring three-dimensional depth information of the target object; reconstructing a direct illumination texture map of the scene; the scattered field distribution of the scene is modeled and reconstructed; and based on the three-dimensional depth information, the direct illumination texture map information and the scattered field distribution information, a final multi-modal imaging result is obtained. The sampling efficiency is remarkably improved through spectrum channel multiplexing, the imaging interference problem of a complex optical surface is effectively solved, and the method is suitable for synchronous high-precision measurement of geometric and physical attributes in a dynamic scene.

Description

Technical Field

[0001] This disclosure relates to the field of optical computational imaging model construction technology, and in particular to a computer system based on a parallel single-pixel RGB composite stripe multimodal imaging model. Background Technology

[0002] Single-pixel imaging technology utilizes a single-point detector without spatial resolution to acquire spatial information of objects through intensity correlation calculations, exhibiting high sensitivity, wide spectral adaptability, and strong anti-interference capabilities. However, traditional single-pixel imaging relies on modulating the scene with numerous temporal projection patterns, resulting in long imaging times and difficulty in adapting to dynamic scene monitoring. Existing technologies, when processing such scenes, often treat scattering components carrying material properties as noise and simply filter them out, leading to non-single-valued errors in phase calculations or decreased accuracy in topography reconstruction. Furthermore, it is difficult to simultaneously acquire the object's three-dimensional geometry and internal physical properties within the same system.

[0003] Currently, there is a lack of technical means to achieve full-field parallel detection through spectral channel multiplexing and to effectively decouple direct illumination and scattering components to fully utilize light field information. Summary of the Invention

[0004] The purpose of this disclosure is to provide a computer system based on a parallel single-pixel RGB composite stripe multimodal imaging model. The aim is to achieve parallel acquisition of multidimensional information under a single exposure by reusing the RGB channel frequency, thereby realizing high-precision anti-interference multimodal imaging in complex dynamic scenes. It is especially suitable for complex scenes that require simultaneous acquisition of high-precision three-dimensional shape of objects, direct illumination texture and subsurface scattering field distribution, such as high reflectivity industrial workpiece inspection and biomedical tissue imaging.

[0005] The specific steps of this disclosure are as follows:

[0006] A computer system based on a parallel single-pixel composite stripe multimodal imaging model includes:

[0007] Control unit;

[0008] The projector is connected to the control unit;

[0009] Color industrial cameras are used to acquire modulated color composite images of the surface of target objects for multimodal imaging.

[0010] The control unit performs multimodal imaging using the following method:

[0011] Construct at least one stripe pattern with a specified frequency, and combine the stripe patterns into an RGB composite stripe image;

[0012] Acquire a color composite image modulated by the surface of the target object, and separate and extract stripe data of any independent frequency component based on the color composite image;

[0013] The three-dimensional depth information of the target object is obtained by using the uniform sampling frequency data carried by the G channel and the non-uniform sampling frequency data carried by the R channel.

[0014] By jointly utilizing the fringe frequency data carried by the G and R channels, the direct illumination components are filtered to reconstruct the direct illumination texture map of the scene;

[0015] Based on the full-frequency fringe information contained in the RGB full channels, the scattering field distribution of the scene is inverted and reconstructed; and based on the three-dimensional depth information, direct illumination texture map information and scattering field distribution information, the final multimodal imaging result is obtained.

[0016] The process of constructing at least one stripe pattern of a specified frequency and combining the stripe patterns into an RGB composite stripe image includes:

[0017] Construct the first set, selecting continuous components closely arranged along the horizontal and vertical axes starting from the center of the frequency domain, to obtain the basic contour structure of the object and provide an initial phase reference in orthogonal directions;

[0018] A second set is constructed, selecting jump points located along the axis and satisfying a fixed grid interval frequency. Utilizing their sparse distribution characteristics and high signal-to-noise ratio, a refined phase constraint and positioning skeleton are provided.

[0019] A third set is constructed, selecting continuous frequency points distributed along the diagonal direction in the spectrum, and introducing the coupling characteristics of the horizontal and vertical coordinates to eliminate the periodic ambiguity of a single axis.

[0020] Axially continuous stripes are generated for the first set, uniform sparse stripes are generated for the second set, and diagonally coupled stripes are generated for the third set to obtain a stripe pattern.

[0021] The stripe pattern is embedded into each channel of the RGB image according to a preset color encoding rule, and an RGB composite stripe image is constructed by linear superposition. :

[0022] ;

[0023] Wherein, R, G, and B are the R channel, G channel, and B channel, respectively; This is the first set constructed by humans. A second set constructed by humans; A third set constructed by humans; Axial continuous stripes generated for the first set; Uniform sparse stripes generated for the second set; Diagonal coupled stripes generated for the third set.

[0024] The method of obtaining the three-dimensional depth information of the target object using the uniform sampling frequency data carried by the G channel and the non-uniform sampling frequency data carried by the R channel includes:

[0025] For the R channel image, the frequency components of the axial continuous stripes are extracted, and one-dimensional discrete Fourier inverse transforms are performed along the horizontal and vertical directions respectively to generate axial light intensity distribution curves containing the basic contours of the object.

[0026] For the G channel image, the corresponding uniform sparse stripe frequency components are extracted and inverse transformation is performed to generate a uniform sparse light intensity distribution curve with high frequency periodic characteristics.

[0027] Perform a Cartesian product operation on the peak sets in the axial light intensity distribution curve and the uniform sparse light intensity distribution curve, and pair all horizontal peaks with vertical peaks within the interval to construct a fine candidate point grid. ;

[0028] Using epipolar geometry constraints to select from fine candidate point meshes Remove spurious combinations and traverse a fine-grained grid of candidate points. any two-dimensional candidate point Obtain the two-dimensional candidate points The geometric distance to the epipolar line is calculated, and the point with the smallest distance is selected as the best matching point for integer pixels. The coordinates are... ;

[0029] For the optimal matching point of the integer pixel The one-dimensional light intensity distribution curves in the G channel respectively and The response values ​​of three adjacent points are extracted, a local second-order parabolic model is constructed, the peak offset is obtained, and the sub-pixel projection coordinates are obtained.

[0030] Based on the sub-pixel projection coordinates Solve for the three-dimensional spatial coordinates of the target point This enables 3D reconstruction and yields 3D depth information.

[0031] For the R channel image, the axial continuous fringe frequency components are extracted, and a one-dimensional discrete Fourier inverse transform is performed along the horizontal and vertical directions respectively to generate an axial light intensity distribution curve containing the basic contour of the object, including:

[0032] Extract the first set constructed by the corresponding person. The frequency components are subjected to one-dimensional discrete Fourier inverse transforms along the horizontal and vertical directions, respectively, to generate axial light intensity distribution curves that include the basic contours of the object. and ;

[0033] Search for the axial light intensity distribution curve containing the basic outline of the object. and Global maximum response value and And set the radius as the center. Coarse position information interval of 20 pixels:

[0034] ;

[0035] in, .

[0036] The step of extracting the frequency components corresponding to the uniform sparse stripes and performing an inverse transform to generate a uniform sparse light intensity distribution curve with high-frequency periodic characteristics includes:

[0037] Extract the corresponding artificially constructed second set The frequency components are analyzed and an inverse transform is performed to generate a uniform sparse intensity distribution curve with high-frequency periodic characteristics. and ;

[0038] Within the preset coarse positioning range and Within the specified range, a minimum height threshold for peak detection is set to 0.5 of the maximum value. All fine peak points meeting the criteria within this region are extracted to form a horizontal fine peak set. With vertical fine peak set :

[0039] ;

[0040] .

[0041] The method utilizes epipolar geometry constraints to select from a fine candidate point mesh Remove spurious combinations and traverse a fine-grained grid of candidate points. any two-dimensional candidate point Obtain the two-dimensional candidate points The geometric distance to the epipolar line is calculated, and the point with the smallest distance is selected as the best matching point for the integer pixel, including:

[0042] Set an epipolar distance threshold. If the minimum distance is greater than this threshold, the point is considered invalid, and the point with the smallest distance is selected as the best matching point for the integer pixel. :

[0043] ;

[0044] in, The L2 norm of the coefficient vector of the polar equation is denoted as . for The transpose of .

[0045] One-dimensional light intensity distribution curves in channel G and The response values ​​of three adjacent points are extracted, and it is assumed that... , Subpixel offset The calculation formula is:

[0046] ;in, The one-dimensional light intensity distribution curves are respectively Above, using integer pixel peak coordinates The light intensity response values ​​are centered on the light source and corresponding to its left and right adjacent positions.

[0047] The horizontal offsets were obtained respectively. offset in the vertical direction This allows us to obtain the final high-precision sub-pixel projection coordinates. :

[0048] .

[0049] The method of jointly utilizing the fringe frequency data carried by the G and R channels to filter the direct illumination components and reconstruct the direct illumination texture map of the scene includes:

[0050] set up For the camera in pixels The corresponding frequency collected at the location The nth phase-shifted image, where ; Calculate the Fourier coefficient spectrum through complex linear combination ;

[0051] Based on subpixel projection coordinates Addressing is performed in the two-dimensional spectrum H, and the spectral response value is... , which serves as a coefficient characterizing the energy directly reflected.

[0052] Using the spectral response value The corresponding fringe inverse multiplication is performed, and the spectral response value is adjusted. Multiplying the fringe pattern projected onto the G channel at the corresponding frequency, an inverse discrete Fourier transform is performed to demodulate the texture information, yielding the final direct illumination texture map. :

[0053] ;

[0054] in, For the first Dimensional features; and These represent the resolution of the image in the horizontal and vertical directions, respectively.

[0055] The process involves inverting and reconstructing the scene's scattering field distribution based on the full-frequency fringe information contained in the RGB full channels; and constructing the final multimodal imaging result based on the three-dimensional depth information, direct illumination texture map information, and scattering field distribution information, including:

[0056] Based on the fringe components of each of the R, G, and B channels, a two-dimensional scattered light intensity spectrum reflecting the spatial distribution of the scene's luminous flux is constructed.

[0057] Construct a two-dimensional anisotropic Gaussian model and optimize its parameter vectors to minimize the weighted sum of squared residuals;

[0058] Based on the optimized parameter vector, the key physical quantities used to describe the diffusion characteristics of the scattered field are obtained;

[0059] Based on the key physical quantities, principal component fusion is performed for dimensionality reduction, resulting in a single material reflectance coefficient. Key features obtained from Gaussian fitting are then selected to construct the observation data matrix. ,in Effective pixel count The feature dimension is defined; and the observed data matrix is ​​standardized to eliminate dimensional differences, resulting in a standardized matrix. :

[0060] ;

[0061] in, and The first The mean and standard deviation of the dimensional features;

[0062] Obtain the covariance matrix of the standardized matrix And perform eigenvalue decomposition on it:

[0063] ;

[0064] in, The eigenvector matrix, This is the diagonal matrix of eigenvalues; arrange the eigenvalues ​​in descending order, and the corresponding eigenvectors are the principal component directions.

[0065] Using the eigenvector matrix Standardized data Mapping to the principal component space yields the score matrix. And extract the first two principal components. Calculate their joint mode length as the original scattering response. :

[0066] ;

[0067] Set statistical threshold range and will Outliers exceeding the statistical threshold range are set as invalid values, and the remaining valid response values ​​are mapped back to the original image coordinate space, ultimately constructing a scattering field distribution map reflecting the subsurface scattering properties of the scene. :

[0068] ;

[0069] Based on the three-dimensional depth information, direct illumination texture map information, and scattered field distribution information, the final multimodal imaging result is constructed.

[0070] The construction of a two-dimensional scattered light intensity spectrum reflecting the spatial distribution of scene luminous flux based on the fringe components of each of the R, G, and B channels includes:

[0071] Constructing the distance field :

[0072] ;

[0073] Construct a Gaussian decay weight matrix Give the central region a higher fitting weight:

[0074] ;

[0075] in, Pixel; and These are the coordinates of any pixel in the two-dimensional scattered light intensity spectrum image and the pixel coordinates of the searched global maximum point, respectively.

[0076] use Pixel statistics estimation of background noise floor The input spectrum is then subjected to background subtraction and nonnegativity constraints.

[0077] .

[0078] in, The pure two-dimensional scattered light intensity spectrum after background subtraction and non-negativity constraint processing. This is the two-dimensional scattered light intensity spectrum after median filtering preprocessing.

[0079] The construction of a two-dimensional anisotropic Gaussian model and optimization of its parameter vectors to minimize the weighted sum of squared residuals includes:

[0080] Define a two-dimensional anisotropic Gaussian model , where the parameter vector These represent the center coordinates, axial standard deviation, and magnitude, respectively.

[0081] ;

[0082] Solving for the optimal parameter vector using the trust region reflection algorithm To minimize the weighted sum of squared residuals:

[0083] ;

[0084] in, The weight matrix is ​​a Gaussian decay matrix. ; The pure two-dimensional scattered light intensity spectrum after background subtraction and non-negative constraint processing; For distance field; Pixel.

[0085] Beneficial effects:

[0086] The beneficial effects of this disclosure include at least the following:

[0087] This disclosure employs a hybrid spectral modulation strategy that combines "multi-directional continuous features" and "sparse uniform reference" to divide the full-frequency sampling path into three specific frequency submanifolds: axially continuous, uniformly sparse, and diagonally coupled. These submanifolds are then encoded and mapped to the R, G, and B channels of a color image, constructing an RGB composite projection map. Simultaneously, a hierarchical depth solution strategy of "global locking-local optimization-epochal optimization" is used to collaboratively acquire high-precision 3D topography by leveraging the different frequency characteristics of each channel. Furthermore, direct illumination components are separated based on frequency domain response to reconstruct high-fidelity textures, and the subsurface scattering field distribution is inverted from the full-channel spectrum using noise-resistant Gaussian fitting and principal component analysis. Finally, the 3D topography, direct illumination texture, and scattering field data are fused to form a multimodal imaging result. This disclosure significantly improves sampling efficiency through spectral channel multiplexing, effectively solving the imaging interference problem of complex optical surfaces, such as highly reflective and semi-transparent surfaces, and is suitable for simultaneous high-precision measurement of geometric and physical properties in dynamic scenes. Attached Figure Description

[0088] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the services required in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0089] Figure 1 This is a schematic diagram of the structure of the computer system described in this disclosure;

[0090] Figure 2 This is a flowchart illustrating the multimodal imaging process performed by the computer system described in this disclosure.

[0091] Figure 3 Schematic diagrams of different set frequency stripe patterns and their corresponding composite patterns;

[0092] Figure 4 This is a schematic diagram illustrating the principle of three-dimensional depth calculation and measurement.

[0093] Figure 5 This is a schematic diagram of the reconstructed scattering field. Detailed Implementation

[0094] Various embodiments of this disclosure will be described more fully below. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0095] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a particular feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of features, numbers, steps, operations, elements, components, or combinations of the foregoing.

[0096] In various embodiments of this disclosure, the expression "or" or "at least one of A and / or B" includes any combination or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A and / or B" may include A, may include B, or may include both A and B.

[0097] The terms used in the various embodiments of this disclosure (such as "first," "second," etc.) may modify various components in the various embodiments, but do not limit the corresponding components. For example, the above terms do not limit the order and / or importance of the components. The above terms are only used for the purpose of distinguishing one component from others. For example, a first user device and a second user device refer to different user devices, although both are user devices. For example, a first component may be referred to as a second component without departing from the scope of the various embodiments of this disclosure, and similarly, a second component may also be referred to as a first component.

[0098] It should be noted that if a description is made of "connecting" one component to another, then the first component can be directly connected to the second component, and a third component can be "connected" between the first and second components. Conversely, when a component is "directly connected" to another component, it can be understood that there is no third component between the first and second components.

[0099] The term "user" as used in various embodiments of this disclosure may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0100] The terminology used in the various embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the various embodiments of this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this disclosure pertain. Terms (such as those defined in a generally used dictionary) are to be interpreted as having the same meaning as in the context of the relevant technical field and are not to be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this disclosure. Specific Implementation Example 1:

[0102] The present disclosure will be further described below with reference to specific embodiments:

[0103] This example uses a 1920 x 1080 pixel image as the projected image. Before that, the structure of a computer system based on a parallel single-pixel RGB composite stripe multimodal imaging model, as described in this embodiment, will be explained:

[0104] like Figure 1 As shown, a computer system based on a parallel single-pixel RGB composite stripe multimodal imaging model includes: a control terminal 1, a projector 2, a color industrial camera 3, a scene of the object under test 4, and projected stripes. Figure 5 The camera captures stripe patterns (6). The control unit 1, such as a computer, generates corresponding fused stripes. Figure 5 The data is transmitted to projector 2 and then projected onto the scene of the object being measured. Next, a color industrial camera 3 captures the corresponding color stripe pattern 6, and the image is transmitted to a computer for processing. The specific processing steps are as follows:

[0105] S1. Construct a stripe pattern with a specified frequency and modulate it to synthesize an RGB composite projection image.

[0106] S1-1 employs a hybrid spectrum modulation strategy that combines "multi-directional continuous features" and "sparse uniform reference" to divide the full-frequency sampling path into three frequency submanifolds with specific topological structures.

[0107] The generation of stripes includes constructing a first set. This serves as a continuous low-frequency set along the axial direction. Starting from the center of the frequency domain, proceed along the horizontal axis... with vertical axis A series of closely packed low-frequency components with a frequency number of L=5 are used to obtain the basic contour structure of the object and provide an initial phase reference in orthogonal directions. Their frequency sequence enumeration is as follows:

[0108] ;

[0109] Construct the second set This serves as a uniformly sparse sampling set. A set located along the axis and satisfying a fixed grid interval is selected. The high-frequency jump point with frequency number k=5, utilizing its sparse distribution characteristics and high signal-to-noise ratio, provides a refined phase constraint and positioning skeleton. Its frequency sequence enumeration is as follows:

[0110] ;

[0111] Construct the third set This serves as a continuous set of frequencies along the diagonal. Select frequencies along the frequency spectrum... and For continuous frequency points with a frequency count M=5 distributed along the diagonal, the coupling feature of the horizontal and vertical coordinates is introduced to eliminate periodic ambiguity along a single axis. The frequency sequence enumeration is as follows:

[0112] ;

[0113] Subsequently, a three-step phase-shift algorithm is used to perform calculations on the three frequency sets defined above, generating corresponding independent stripe image sequences: axially continuous stripes are generated for the first set. Generate uniform sparse stripes for the second set. Diagonal coupled stripes are generated for the third set. .

[0114] S1-2 performs the mapping and assembly operation of the spectral channels, embedding the generated independent stripe patterns into each channel of the RGB image according to the preset color encoding rules.

[0115] Specifically, Mapping to the R channel to establish the base coordinate index, Mapped to the G channel to utilize the high transmittance of the green light band to carry high-frequency details, Mapping to the B channel to introduce diagonal detail information. Finally, an RGB composite stripe image is constructed by linear overlay. :

[0116] ;

[0117] S2. Capture a color composite image modulated by the surface of the target object using an image acquisition device, and apply an RGB channel demultiplexing algorithm to separate and extract the stripe data of each independent frequency component.

[0118] S3. By jointly utilizing the uniform sampling frequency data carried by the G channel and the non-uniform sampling frequency data carried by the R channel, the three-dimensional depth information of the object is calculated and obtained.

[0119] S3-1 employs a hierarchical reconstruction strategy of "global maximum value locking region - local multi-peak arrangement and combination - epipolar geometric optimization". This process first uses the R channel (axially continuous low frequency) to determine a coarse confidence interval based on its response maximum value. Then, within this interval, all potential peaks of the G channel (uniformly sparse high frequency) are extracted and combined. Finally, epipolar constraints are used to lock the ground truth. First, for the R channel image, the corresponding first set is extracted. The frequency components are subjected to one-dimensional discrete Fourier inverse transforms along the horizontal and vertical directions, respectively, to generate axial light intensity distribution curves that include the basic contours of the object. and .

[0120] To eliminate background noise interference and quickly locate the target, the global maximum response value on these two curves is directly searched. and And set the radius as the center. Coarse position information interval of 20 pixels:

[0121] ;

[0122] in, , Similarly, this step strictly limits the subsequent calculation scope from the global domain to a local (40-pixel wide) region, significantly reducing the computational load.

[0123] S3-2 Further, for the G channel image, extract the corresponding second set. The frequency components are analyzed and an inverse transform is performed to generate a uniform sparse intensity distribution curve with high-frequency periodic characteristics. and .because The frequency intervals are large, and the curve exhibits multi-peak characteristics. This is only true within the aforementioned coarse positioning range. and Within the specified range, local peak detection algorithms are executed, with the minimum height threshold for peak detection set to 0.5 of the maximum value. All fine-grained peak points meeting the specified conditions within this region are extracted, forming a horizontal fine-grained peak set. With vertical fine peak set :

[0124] ;

[0125] ;

[0126] S3-3 To determine the unique two-dimensional projected coordinates, a Cartesian product operation is performed on the two refined peak sets mentioned above, pairing all horizontal and vertical peaks within the interval to construct a refined candidate point grid. Since the coarse localization interval typically contains several high-frequency periods, theoretically this set will contain a small number of high-confidence candidate points:

[0127] ;

[0128] S3-4 further utilizes polar geometry constraints from Eliminate spurious combinations. First, calculate the epipolar equation corresponding to the current camera pixel coordinates. Then, traverse Each two-dimensional candidate point The geometric distance from the epipolar line to the point is calculated. The point with the smallest distance is selected as the best matching point for an integer pixel. In this embodiment, the epipolar line distance threshold is set to 5.0 pixels. If the minimum distance is greater than this threshold, it is considered an invalid point, and the point with the smallest distance is selected as the best matching point for an integer pixel. :

[0129] ;

[0130] In order to overcome the limitations of the projector's physical resolution, S3-5 addresses the integer pixel coordinates locked in the fourth step. and The one-dimensional light intensity distribution curves in the G channel respectively and The response values ​​of three adjacent points are extracted and a local second-order parabolic model is constructed to calculate the peak offset.

[0131] set up , Subpixel offset The calculation formula is:

[0132] ;

[0133] Apply this formula to calculate the horizontal offset respectively. offset in the vertical direction This allows us to obtain the final high-precision sub-pixel projection coordinates:

[0134] ;

[0135] S3-6 Based on Locked Final Subpixel Projection Coordinates By combining calibration parameters and utilizing the principle of triangulation, the three-dimensional spatial coordinates of the target point can be directly calculated. This enables the final three-dimensional reconstruction.

[0136] S4. By jointly utilizing the fringe frequency data carried by the G and R channels, the direct lighting components are calculated and filtered to reconstruct the direct lighting texture map of the scene.

[0137] S4-1 Further, a two-dimensional spectrum is generated using the high-frequency fringe data of the G channel. The frequency domain response is extracted based on the projection coordinates locked in the above steps, and the direct illumination texture, which eliminates scattering interference, is solved through inverse transformation. The fringe image of the G channel separated above corresponds to the second set. A three-step phase-shift algorithm is used to construct a two-dimensional complex spectrum covering the entire field of view. Let... For the camera in pixels The corresponding frequency collected at the location The nth phase-shifted image, where The spectrum of Fourier coefficients was calculated by complex linear combination. Next, based on the final sub-pixel projection coordinates calculated in step S3... Addressing is performed within the generated two-dimensional spectrum H. Since the direct illumination component is mainly concentrated at the conjugate position of the projected frequency, the spectral response value corresponding to that coordinate is extracted. This is used as a coefficient characterizing the direct reflected energy. Then, the corresponding fringe inverse multiplication is combined with the texture to extract the spectral response value. The light transmission coefficients of the object's surface are reconstructed by multiplying the basis functions corresponding to the frequency, i.e., the fringe pattern projected onto the G channel, and then performing an inverse discrete Fourier transform to demodulate the texture information. That is, the final direct lighting texture map :

[0138] ;

[0139] S5. Based on the full-frequency stripe information contained in the RGB full channels, the scattering field distribution of the scene is inverted and reconstructed. Finally, based on the above three-dimensional topography, direct illumination texture and scattering field reconstruction data, the final multimodal imaging result is constructed.

[0140] S5-1 constructs a two-dimensional scattering spectrum based on the high-dimensional frequency domain information reconstructed from the RGB full-channel fringe. Then, using an algorithm combining "noise-resistant weighted Gaussian fitting" and "principal component analysis feature fusion," it solves the scalar field characterizing the subsurface scattering properties of the object. By comprehensively utilizing the R, G, and B channel fringe components extracted in step S2, and through multi-channel spectral superposition or demodulation, it constructs a two-dimensional scattered light intensity spectrum reflecting the spatial distribution of the scene's luminous flux. To suppress sensor thermal noise and shot noise, the original spectrum was first processed with a window size of [size missing]. Median filtering preprocessing was performed, followed by parametric modeling of the scattering spectrum using a confidence-weighted nonlinear least squares method to accurately quantify the morphological characteristics of the scattering spot. Global maximum point in As the scattering center, the radius of the region of interest is adaptively determined based on the image size. =30 pixels. Construct a distance field. :

[0141] ;

[0142] To reduce the impact of background noise far from the scattering center on the fitting accuracy, a Gaussian attenuation weight matrix is ​​constructed. Give the central region a higher fitting weight:

[0143] ;

[0144] The corresponding Pixel.

[0145] At the same time, utilize the outer region of ROI Pixel statistics estimation of background noise floor The input spectrum is then subjected to background subtraction and nonnegativity constraints.

[0146] ;

[0147] Furthermore, a two-dimensional anisotropic Gaussian model is defined. , where the parameter vector These represent the center coordinates, axial standard deviation, and magnitude, respectively.

[0148] ;

[0149] Then, the optimal parameter vector is solved using the trust region reflection algorithm. To minimize the weighted sum of squared residuals:

[0150] ;

[0151] S5-1 Based on the optimized parameters This allows for the calculation of key physical quantities describing the diffusion characteristics of the scattered field, including parameters such as full width at half maximum (FWHM) and integral intensity. Principal component fusion is then performed on the calculated key physical quantities to reduce dimensionality, yielding a single material reflectance coefficient. Key features obtained through Gaussian fitting are then selected, such as… and Construct the observation data matrix ,in Effective pixel count The feature dimension is used. The matrix is ​​standardized to eliminate dimensional differences, resulting in a standardized matrix. :

[0152] ;

[0153] In the formula and The first The mean and standard deviation of the dimensional features.

[0154] Calculate the covariance matrix of the normalized matrix. And perform eigenvalue decomposition on it:

[0155] ;

[0156] in The eigenvector matrix, This is the eigenvalue diagonal matrix. Arranging the eigenvalues ​​in descending order, the corresponding eigenvectors are the principal component directions.

[0157] Using the calculated projection matrix Standardized data Mapping to the principal component space yields the score matrix. Extract the first two principal components. Calculate their joint mode length as the original scattering response. :

[0158] ;

[0159] To ensure the robustness of the results, a statistical threshold range is set. In this embodiment, the normalization threshold for the effective scattering response is set to... =-1, =1 will Outliers outside this range are set to invalid values, and the remaining valid response values ​​are mapped back to the original image coordinate space, ultimately constructing a scattering field distribution map reflecting the subsurface scattering properties of the scene. :

[0160] ;

[0161] Finally, based on the above-mentioned three-dimensional topography, direct illumination texture, and scattered field reconstruction data, the final multimodal imaging results are constructed.

[0162] In summary, compared with existing technologies, this disclosure achieves parallel acquisition of multi-dimensional frequency domain information in a single exposure through RGB spectral multiplexing technology, exhibiting high measurement efficiency and dynamic adaptability. Moreover, by utilizing frequency domain response separation technology, it can effectively separate mutual reflection and subsurface scattering components from the mixed light field, achieving high-fidelity imaging of semi-transparent media and demonstrating excellent anti-interference capabilities. Furthermore, by employing noise-resistant Gaussian fitting and PCA analysis, it can simultaneously invert three-dimensional morphology, direct illumination texture, and scattering field distribution, providing rich material physical property data and enabling simultaneous acquisition of multi-modal information. This makes it suitable for synchronous high-precision measurement of geometric and physical properties in dynamic scenes.

[0163] The above disclosures only cover a few specific implementation scenarios. However, this disclosure is not limited to these, and any variations that can be conceived by those skilled in the art should fall within the protection scope of this disclosure. The serial numbers in this disclosure are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenarios.

Claims

1. A computer system based on a parallel single-pixel composite stripe multimodal imaging model, characterized in that, include: Control unit; The projector is connected to the control unit; Color industrial cameras are used to acquire modulated color composite images of the surface of target objects for multimodal imaging. The control unit performs multimodal imaging using the following method: Construct at least one stripe pattern with a specified frequency, and combine the stripe patterns into an RGB composite stripe image; Acquire a color composite image modulated by the surface of the target object, and separate and extract stripe data of any independent frequency component based on the color composite image; The three-dimensional depth information of the target object is obtained by using the uniform sampling frequency data carried by the G channel and the non-uniform sampling frequency data carried by the R channel. By jointly utilizing the fringe frequency data carried by the G and R channels, the direct illumination components are filtered to reconstruct the direct illumination texture map of the scene; Based on the full-frequency fringe information contained in the RGB full channels, the scattering field distribution of the scene is inverted and reconstructed; and based on the three-dimensional depth information, direct illumination texture map information and scattering field distribution information, the final multimodal imaging result is obtained.

2. The computer system based on a parallel single-pixel composite stripe multimodal imaging model according to claim 1, characterized in that, The process of constructing at least one stripe pattern of a specified frequency and combining the stripe patterns into an RGB composite stripe image includes: Construct the first set, selecting continuous components closely arranged along the horizontal and vertical axes starting from the center of the frequency domain, to obtain the basic contour structure of the object and provide an initial phase reference in orthogonal directions; A second set is constructed, selecting jump points located along the axis and satisfying a fixed grid interval frequency. Utilizing their sparse distribution characteristics and high signal-to-noise ratio, a refined phase constraint and positioning skeleton are provided. A third set is constructed, selecting continuous frequency points distributed along the diagonal direction in the spectrum, and introducing the coupling characteristics of the horizontal and vertical coordinates to eliminate the periodic ambiguity of a single axis. Axially continuous stripes are generated for the first set, uniform sparse stripes are generated for the second set, and diagonally coupled stripes are generated for the third set to obtain a stripe pattern. The stripe pattern is embedded into each channel of the RGB image according to a preset color encoding rule, and an RGB composite stripe image is constructed by linear superposition. : ； Wherein, R, G, and B are the R channel, G channel, and B channel, respectively; This is the first set constructed by humans. A second set constructed by humans; A third set constructed by humans; Axial continuous stripes generated for the first set; Uniform sparse stripes generated for the second set; Diagonal coupled stripes generated for the third set.

3. The computer system based on a parallel single-pixel composite stripe multimodal imaging model according to claim 1, characterized in that, The method of obtaining the three-dimensional depth information of the target object using the uniform sampling frequency data carried by the G channel and the non-uniform sampling frequency data carried by the R channel includes: For the R channel image, the frequency components of the axial continuous stripes are extracted, and one-dimensional discrete Fourier inverse transforms are performed along the horizontal and vertical directions respectively to generate axial light intensity distribution curves containing the basic contours of the object. For the G channel image, the corresponding uniform sparse stripe frequency components are extracted and inverse transformation is performed to generate a uniform sparse light intensity distribution curve with high frequency periodic characteristics. Perform a Cartesian product operation on the peak sets in the axial light intensity distribution curve and the uniform sparse light intensity distribution curve, and pair all horizontal peaks with vertical peaks within the interval to construct a fine candidate point grid. ; Using epipolar geometry constraints to select from fine candidate point meshes Remove spurious combinations and traverse a fine-grained grid of candidate points. any two-dimensional candidate point Obtain the two-dimensional candidate points The geometric distance to the epipolar line is calculated, and the point with the smallest distance is selected as the best matching point for integer pixels. The coordinates are... ; For the optimal matching point of the integer pixel The one-dimensional light intensity distribution curves in the G channel respectively and The response values ​​of three adjacent points are extracted, a local second-order parabolic model is constructed, the peak offset is obtained, and the sub-pixel projection coordinates are obtained. Based on the sub-pixel projection coordinates Solve for the three-dimensional spatial coordinates of the target point This enables 3D reconstruction and yields 3D depth information.

4. The computer system based on a parallel single-pixel composite stripe multimodal imaging model according to claim 3, characterized in that, For the R channel image, the axial continuous fringe frequency components are extracted, and a one-dimensional discrete Fourier inverse transform is performed along the horizontal and vertical directions respectively to generate an axial light intensity distribution curve containing the basic contour of the object, including: Extract the first set constructed by the corresponding person. The frequency components are subjected to one-dimensional discrete Fourier inverse transforms along the horizontal and vertical directions, respectively, to generate axial light intensity distribution curves that include the basic contours of the object. and ; Search for the axial light intensity distribution curve containing the basic outline of the object. and Global maximum response value and And set the radius as the center. Coarse position information interval of 20 pixels: ； in, , .

5. The computer system based on a parallel single-pixel composite stripe multimodal imaging model according to claim 3, characterized in that, The step of extracting the frequency components corresponding to the uniform sparse stripes and performing an inverse transform to generate a uniform sparse light intensity distribution curve with high-frequency periodic characteristics includes: Extract the corresponding artificially constructed second set The frequency components are analyzed and an inverse transform is performed to generate a uniform sparse intensity distribution curve with high-frequency periodic characteristics. and ; Within the preset coarse positioning range and Within the specified range, a minimum height threshold for peak detection is set to 0.5 of the maximum value. All fine peak points meeting the criteria within this region are extracted to form a horizontal fine peak set. With vertical fine peak set : ； 。 6. The computer system based on a parallel single-pixel composite stripe multimodal imaging model according to claim 3, characterized in that, The method utilizes epipolar geometry constraints to select from a fine candidate point mesh Remove spurious combinations and traverse a fine-grained grid of candidate points. any two-dimensional candidate point Obtain the two-dimensional candidate points The geometric distance to the epipolar line is calculated, and the point with the smallest distance is selected as the best matching point for the integer pixel, including: Set an epipolar distance threshold. If the minimum distance is greater than this threshold, the point is considered invalid, and the point with the smallest distance is selected as the best matching point for the integer pixel. : ； in, The L2 norm of the coefficient vector of the polar equation is denoted as . for Transpose of; One-dimensional light intensity distribution curves in channel G and The response values ​​of three adjacent points are extracted, and it is assumed that... , Subpixel offset The calculation formula is: ;in, The one-dimensional light intensity distribution curves are respectively Above, using integer pixel peak coordinates The light intensity response values ​​are centered on the light source and corresponding to its left and right adjacent positions. The horizontal offsets were obtained respectively. offset in the vertical direction This allows us to obtain the final high-precision sub-pixel projection coordinates. : 。 7. The computer system based on a parallel single-pixel composite stripe multimodal imaging model according to claim 1, characterized in that, The method of jointly utilizing the fringe frequency data carried by the G and R channels to filter the direct illumination components and reconstruct the direct illumination texture map of the scene includes: set up For the camera in pixels The corresponding frequency collected at the location The nth phase-shifted image, where ; Calculate the Fourier coefficient spectrum through complex linear combination ; Based on subpixel projection coordinates Addressing is performed in the two-dimensional spectrum H, and the spectral response value is... , which serves as a coefficient characterizing the energy directly reflected; Using the spectral response value The corresponding fringe inverse multiplication is performed, and the spectral response value is adjusted. Multiplying the fringe pattern projected onto the G channel at the corresponding frequency, an inverse discrete Fourier transform is performed to demodulate the texture information, yielding the final direct illumination texture map. : ;in, For the first Dimensional features; and These represent the resolution of the image in the horizontal and vertical directions, respectively.

8. The computer system based on a parallel single-pixel composite stripe multimodal imaging model according to claim 1, characterized in that, The scattering field distribution of the scene is inverted and reconstructed based on the full-frequency domain stripe information contained in the RGB full channels; Based on the aforementioned three-dimensional depth information, direct illumination texture map information, and scattered field distribution information, the final multimodal imaging result is constructed, including: Based on the fringe components of each of the R, G, and B channels, a two-dimensional scattered light intensity spectrum reflecting the spatial distribution of the scene's luminous flux is constructed. Construct a two-dimensional anisotropic Gaussian model and optimize its parameter vectors to minimize the weighted sum of squared residuals; Based on the optimized parameter vector, the key physical quantities used to describe the diffusion characteristics of the scattered field are obtained; Based on the key physical quantities, principal component fusion is performed for dimensionality reduction, resulting in a single material reflectance coefficient. Key features obtained from Gaussian fitting are then selected to construct the observation data matrix. ,in Effective pixel count The feature dimension is defined; and the observed data matrix is ​​standardized to eliminate dimensional differences, resulting in a standardized matrix. : ;in, and The first The mean and standard deviation of the dimensional features; Obtain the covariance matrix of the standardized matrix And perform eigenvalue decomposition on it: ;in, The eigenvector matrix, This is the diagonal matrix of eigenvalues; arrange the eigenvalues ​​in descending order, and the corresponding eigenvectors are the principal component directions. Using the eigenvector matrix Standardized data Mapping to the principal component space yields the score matrix. And extract the first two principal components. Calculate their joint mode length as the original scattering response. : ； Set statistical threshold range and will Outliers exceeding the statistical threshold range are set as invalid values, and the remaining valid response values ​​are mapped back to the original image coordinate space, ultimately constructing a scattering field distribution map reflecting the subsurface scattering properties of the scene. : ； Based on the three-dimensional depth information, direct illumination texture map information, and scattered field distribution information, the final multimodal imaging result is constructed.

9. The computer system based on a parallel single-pixel composite stripe multimodal imaging model according to claim 8, characterized in that, The construction of a two-dimensional scattered light intensity spectrum reflecting the spatial distribution of scene luminous flux based on the fringe components of each of the R, G, and B channels includes: Constructing the distance field : Construct a Gaussian decay weight matrix Give the central region a higher fitting weight: ； in, Pixel; and These are the coordinates of any pixel in the two-dimensional scattered light intensity spectrum image and the pixel coordinates of the searched global maximum point, respectively. use Pixel statistics estimation of background noise floor The input spectrum is then subjected to background subtraction and nonnegativity constraints. ； in, The pure two-dimensional scattered light intensity spectrum after background subtraction and non-negativity constraint processing. This is the two-dimensional scattered light intensity spectrum after median filtering preprocessing.

10. The computer system based on a parallel single-pixel composite stripe multimodal imaging model according to claim 8, characterized in that, The construction of a two-dimensional anisotropic Gaussian model and optimization of its parameter vectors to minimize the weighted sum of squared residuals includes: Define a two-dimensional anisotropic Gaussian model , where the parameter vector These represent the center coordinates, axial standard deviation, and magnitude, respectively. ； Solving for the optimal parameter vector using the trust region reflection algorithm To minimize the weighted sum of squared residuals: ； in, The weight matrix is ​​a Gaussian decay matrix. ; The pure two-dimensional scattered light intensity spectrum after background subtraction and non-negative constraint processing; For distance field; Pixel.

Images