A high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning

Abstract

The application discloses a kind of high-precision three-dimensional measurement methods based on multi-frequency fusion coding and deep learning, including generating RGB three-channel color phase shift sinusoidal fringe diagram, load DLP light source and project to the surface of measured object;Acquisition has the color structure light fringe measured object fringe image, and obtain mask from it;Based on measured object mask, the RGB each channel in measured object fringe image is carried out target segmentation, and the fringe diagram of low resolution area and the fringe diagram of high resolution area are segmented out;The segmented out graph is simultaneously input into depth neural network, and the fringe super-resolution image and saturated fringe correction image are obtained;Phase extraction and phase-height mapping are carried out to fringe super-resolution image and saturated fringe correction image, and three-dimensional measurement is completed in combination with the two phase-height mapping results obtained.The application can effectively improve the measurement accuracy and robustness in large depth range through multi-channel information fusion and depth neural network processing, and realize high-resolution three-dimensional measurement.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional measurement, and in particular to a high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning. Background Art

[0002] With the rapid development of computer vision, three-dimensional measurement has become a core research area in computer vision. Three-dimensional measurement refers to the precise measurement of attributes such as size, shape, and position of certain three-dimensional objects or scenes. The measured data is convenient for computer representation and processing. In actual measurement, three-dimensional measurement is the process of describing the images of objects, scenes, and human bodies in three-dimensional space. By extracting information from two-dimensional images or sensor data, the precise geometric parameters of three-dimensional objects, scenes, and dynamic human bodies are obtained. Therefore, three-dimensional measurement is a key technology for creating virtual reality representations of the objective world in computers.

[0003] Three-dimensional measurement based on structured light has been widely used in modern industrial inspection, surveying and mapping, intelligent manufacturing, reverse engineering and other industries. Traditional structured light measurement methods face challenges in deep-depth measurement scenarios. The main reasons are twofold: 1) The resolution of distant objects in the field of view drops sharply, causing phase recovery to easily lose detail information and reduce measurement accuracy; 2) Close-range objects are easily affected by light intensity saturation, which causes the sinusoidality of the stripes to be destroyed and makes it difficult to recover high-quality phases. Existing methods usually use simple interpolation methods when processing low-resolution stripe images. Such methods have limited effectiveness when facing stripe images with complex morphology. In addition, when facing light intensity saturation, traditional methods often use multiple exposures or adaptive projection methods. Such methods require adding a large number of stripe images with different light intensities, which greatly reduces measurement efficiency. Therefore, it is necessary to study a simple and efficient method that overcomes the problems of low resolution and light intensity saturation at the same time, so as to promote the development of structured light three-dimensional measurement technology in deep-depth measurement scenarios. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning. Compared with the traditional phase shift method, this method requires fewer frames and can increase the speed of three-dimensional measurement. By restoring the details of the fringe image and correcting the distortion of saturated fringes, the robustness of three-dimensional measurement can be improved.

[0005] To achieve the above objectives, the technical solutions provided by the present invention are:

[0006] A high-precision 3D measurement method based on multi-frequency fusion coding and deep learning, including:

[0007] The computer generates three RGB three-channel color phase-shifted sinusoidal fringe images, which are loaded into the DLP light source and projected onto the surface of the object being measured;

[0008] The camera collects the stripe image of the object under test with the colored structured light stripes, and the computer obtains the mask M of the object under test from the stripe image of the object under test;

[0009] The mask M of the object to be tested is used to determine different targets, and the RGB channels in the fringe image of the object to be tested are segmented to obtain the low-resolution fringe image I. low_R and I low_B And high resolution regional fringe pattern I high_G ;

[0010] The low-resolution area fringe image I low_R and I low_B And high resolution regional fringe pattern I high_G At the same time, the deep neural network is input, and after processing by the deep neural network, the fringe super-resolution image and the saturated fringe correction image are output respectively;

[0011] Phase extraction and phase-height mapping are performed on the fringe super-resolution image and the saturated fringe correction image, and the two phase-height mapping results are combined to complete the three-dimensional measurement.

[0012] This technical solution can effectively improve the measurement accuracy and robustness within a large depth range through multi-channel information fusion and deep neural network processing, while achieving high-resolution three-dimensional measurement, especially with strong adaptability in complex scenarios.

[0013] Furthermore, the process of computer-generated RGB three-channel color phase-shifted sinusoidal fringe pattern includes:

[0014] First, two sets of single-channel phase-shifted fringe images with different frequencies and different step numbers are generated according to the following formula:

[0015]

[0016] Among them, the first group is the intermediate frequency three-step phase shift stripes {I m1 , I m2 , I m3}, the second group is a high-frequency six-step phase-shifted fringe sequence {I h1 , I h2 , I h3 , I h4 , I h5 , I h6}, A is the background light intensity of the image, B is the modulation degree of the stripes, f m is the intermediate frequency fringe frequency, f h is the high-frequency fringe frequency, i represents the i-th phase-shifted fringe pattern;

[0017] Then, the stripes of two frequencies and two step numbers are combined to generate three RGB three-channel color phase-shifted sinusoidal stripe images {Ic1 , I c2 , I c3}, where I c1 The three channels are placed in {I h1 , I m1 , I h2}, I c2 The three channels are placed in {I h3 , I m2 , I h4}, I c3 The three channels are placed in {I h5 , I m3 , I h6}.

[0018] Furthermore, the process of obtaining the mask M of the object to be measured from the fringe image of the object to be measured by the computer includes:

[0019] Using the three RGB three-channel color phase-shifted sinusoidal fringe images, the fringe I of the G channel m1 ~I m3 , calculate its background intensity map img_b and modulation map img_m according to the following formula:

[0020]

[0021] Set the thresholds respectively and get the intensity mask M according to the background intensity map img_b b , use the modulation map img_m to get the modulation mask M m , and finally take the union of the two masks as the final object mask M.

[0022] Furthermore, the process of determining different targets using the object mask M and performing target segmentation on each RGB channel in the fringe image of the object includes:

[0023] First, the connected region marking method is used to mark the connected regions in the mask M of the object to be measured, and the number and position of the targets are determined;

[0024] Then, count the number of pixels in each connected domain and set an appropriate pixel threshold. Define the connected domain smaller than the threshold as a low-resolution target. Take the corresponding position of the target in the mask M of the object to be tested to generate the mask M. low， And use the mask M and M low Subtract and get the mask M of the remaining area high ;

[0025] Next, M low Multiplying with the R and B channel color phase-shifted sinusoidal fringe patterns, we get the segmented low-resolution area fringe pattern I low_R and I low_B ; The mask M of the remaining areahigh Multiplying with the G channel color phase-shifted sinusoidal fringe image, we get the segmented high-resolution area fringe image I high_G .

[0026] Furthermore, the deep neural network includes an encoding part and a decoding part. In the encoding part, the low-resolution area fringe image I low_R and I low_B And high resolution regional fringe pattern I high_G The high-level features of the image are extracted through convolution and downsampling; in the decoding part, it is divided into two branches, in which the low-resolution area stripe image I low_R and I low_B Entering branch 1, after deconvolution and upsampling, the stripe super-resolution image is output; the high-resolution area stripe image I high_G Entering branch 2, after deconvolution and upsampling, the saturated stripe correction image is output; the weights of branch 1 and branch 2 are not shared.

[0027] Furthermore, the formula for phase extraction of the fringe super-resolution image and the saturated fringe correction image is as follows:

[0028]

[0029] Among them, I sup_u , Ф sup and k h are the super-resolution fringe, super-resolution phase and high-frequency fringe level output by the network, u is the u-th super-resolution fringe image; I cor_v , Ф cor and k m The saturation optimization fringe, saturation error correction phase and intermediate frequency fringe level of the network output are respectively, and v is the vth saturation optimization fringe image.

[0030] Furthermore, when performing phase-height mapping, the system calibration is used to map the extracted phase results to the actual physical size to obtain depth information:

[0031]

[0032] Among them, a sup 、b sup 、c sup is the parameter used to calibrate the system using high-frequency fringes, which is used to map the super-resolution phase to three-dimensional coordinates. cor 、b cor 、c cor Parameters used to calibrate the system using intermediate frequency stripes, used to map the saturation correction phase to three-dimensional coordinates;

[0033] Finally, the three-dimensional coordinates in the two cases are integrated into the same coordinate system to complete the three-dimensional measurement.

[0034] Compared with the existing technology, the principles and advantages of this technical solution are as follows:

[0035] This technical solution can effectively improve the measurement accuracy and robustness within a large depth range through multi-channel information fusion and deep neural network processing, while achieving high-resolution three-dimensional measurement, especially with strong adaptability in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the services required for use in the embodiments or the prior art descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A schematic diagram of a three-dimensional measurement system for implementing a high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning according to the present invention;

[0038] Figure 2 This is a flow chart of a high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning in the present invention;

[0039] Figure 3 Schematic diagram for computer-generated RGB three-channel color phase-shifted sinusoidal fringe pattern;

[0040] Figure 4 is a schematic diagram of the mask M of the object to be measured;

[0041] Figure 5 Fringe patterns of low-resolution area (left) and high-resolution area (right);

[0042] Figure 6 Diagram of the architecture of a deep neural network for simultaneous optimization of low-resolution and saturation stripes.

[0043] Reference numerals:

[0044] 1-Computer; 2-DLP light source; 3-Color CCD camera; 4-Object under test. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with specific embodiments:

[0046] The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning described in this embodiment is implemented using a three-dimensional measurement system, such as Figure 1As shown, the three-dimensional measurement system includes a computer 1, a DLP light source 2, a color CCD camera 3, and a measured object 4; the DLP light source 2 and the color CCD camera 3 are connected to the computer 1 via data cables respectively; the DLP light source 2 and the color CCD camera 3 are at the same height.

[0047] like Figure 2 As shown, the working principle of the 3D measurement system is as follows:

[0048] S1, computer 1 generates three RGB three-channel color phase-shifted sinusoidal fringe images, loads them into DLP light source 2 and projects them onto the surface of the object under test 4;

[0049] In this step, the process of the computer 1 generating three RGB three-channel color phase-shifted sinusoidal fringe images includes:

[0050] First, two sets of single-channel phase-shifted fringe images with different frequencies and different step numbers are generated according to the following formula:

[0051]

[0052] Among them, the first group is the intermediate frequency three-step phase shift stripes {I m1 , I m2 , I m3}, the second group is a high-frequency six-step phase-shifted fringe sequence {I h1 , I h2 , I h3 , I h4 , I h5 , I h6}; A is the background light intensity of the image, B is the modulation degree of the stripes, f m is the intermediate frequency fringe frequency, f h is the high-frequency fringe frequency, i represents the i-th phase-shifted fringe pattern;

[0053] Then, the stripes of two frequencies and two step numbers are combined to generate three RGB three-channel color phase-shifted sinusoidal stripe images {I c1 , I c2 , I c3}, where I c1 The three channels are placed in {I h1 , I m1 , I h2}, I c2 The three channels are placed in {I h3 , I m2 , I h4}, I c3 The three channels are placed in {I h5 , I m3 , I h6},like Figure 3 shown.

[0054] S2, collecting a fringe image of the object under test with color structured light fringes by a color CCD camera 3, and obtaining a mask M of the object under test from the fringe image of the object under test by a computer 1;

[0055] In this step, the process of obtaining the mask M of the object to be measured from the fringe image of the object to be measured by the computer includes:

[0056] Using the three RGB three-channel color phase-shifted sinusoidal fringe images, the fringe I of the G channel m1 ~I m3 , calculate its background intensity map img_b and modulation map img_m according to the following formula:

[0057]

[0058] Set appropriate thresholds respectively and obtain the intensity mask M according to the background intensity map img_b b , use the modulation map img_m to get the modulation mask M m , and finally take the union of the two masks as the final object mask M, as Figure 4 shown.

[0059] S3, using the mask M of the object to be measured to determine different targets, perform target segmentation on each RGB channel in the fringe image of the object to be measured, and segment the low-resolution area fringe image I low_R and I low_B And high resolution regional fringe pattern I high_G ;

[0060] The process of this step includes:

[0061] First, the connected region marking method is used to mark the connected regions in the mask M of the object to be measured, and the number and position of the targets are determined;

[0062] Then, count the number of pixels in each connected domain and set an appropriate pixel threshold. Define the connected domain smaller than the threshold as a low-resolution target. Take the corresponding position of the target in the mask M of the object to be tested to generate the mask M. low， And use the mask M and M low Subtract and get the mask M of the remaining area high ;

[0063] Next, M low Multiplying with the R and B channel color phase-shifted sinusoidal fringe patterns, we get the segmented low-resolution area fringe pattern I low_R and I low_B ; The mask M of the remaining area high Multiplying with the G channel color phase-shifted sinusoidal fringe image, we get the segmented high-resolution area fringe image I high_G ,like Figure 5 shown.

[0064] S4, the low-resolution area fringe image I low_R and I low_B And high resolution regional fringe pattern I high_G At the same time, input Figure 6 The deep neural network shown in the figure outputs a fringe super-resolution image and a saturated fringe correction image after being processed by the deep neural network;

[0065] The process of this step includes:

[0066] First, the three segmented fringe images are grouped into a set {I low_R , I low_B , I high_G Input to a deep neural network for simultaneous optimization of low-resolution stripes and saturation stripes.

[0067] Then, the deep neural network is divided into an encoding part and a decoding part. In the encoding part, the three images are convolved and downsampled to extract high-order features of the images;

[0068] Then, the downsampled fringe pattern is divided into two branches in the decoding part, where I low_R and I low_B Enter branch 1, and after deconvolution and upsampling, output the stripe super-resolution image; I high_G Entering branch 2, after deconvolution and upsampling, the saturated stripe correction image is output. The weights of the two branches are not shared.

[0069] S5. Perform phase extraction and phase-height mapping on the fringe super-resolution image and the saturated fringe correction image, and combine the two phase-height mapping results to complete the three-dimensional measurement.

[0070] In this step, the phase extraction formula for the fringe super-resolution image and the saturated fringe correction image is as follows:

[0071]

[0072] Among them, I sup_u , Ф sup and k h are the super-resolution fringe, super-resolution phase and high-frequency fringe level output by the network, u is the u-th super-resolution fringe image; I cor_v , Ф cor and k m The saturation optimization fringe, saturation error correction phase and intermediate frequency fringe level of the network output are respectively, and v is the vth saturation optimization fringe image.

[0073] When performing phase-height mapping, the system calibration is used to map the extracted phase results to the actual physical size to obtain depth information:

[0074]

[0075] Among them, a sup 、b sup 、c sup is the parameter used to calibrate the system using high-frequency fringes, which is used to map the super-resolution phase to three-dimensional coordinates. cor 、b cor 、c cor Parameters used to calibrate the system using intermediate frequency stripes, used to map the saturation correction phase to three-dimensional coordinates;

[0076] Finally, the three-dimensional coordinates in the two cases are integrated into the same coordinate system to complete the three-dimensional measurement.

[0077] The embodiments described above are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, any changes made based on the shape and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning, characterized in that: include: The computer generates three RGB three-channel color phase-shifted sinusoidal fringe images, which are loaded into the DLP light source and projected onto the surface of the object being measured; The camera collects the stripe image of the object under test with the colored structured light stripes, and the computer obtains the mask M of the object under test from the stripe image of the object under test; The mask M of the object to be tested is used to determine different targets, and the RGB channels in the fringe image of the object to be tested are segmented to obtain the low-resolution fringe image I. low_R and I low_B And high resolution regional fringe pattern I high_G ; The low-resolution area fringe image I low_R and I low_B And high resolution regional fringe pattern I high_G At the same time, the deep neural network is input, and after processing by the deep neural network, the fringe super-resolution image and the saturated fringe correction image are output respectively; Phase extraction and phase-height mapping are performed on the fringe super-resolution image and the saturated fringe correction image, and the two phase-height mapping results are combined to complete the three-dimensional measurement.

2. The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning according to claim 1 is characterized in that: The process of computer-generated RGB three-channel color phase-shifted sinusoidal fringe pattern includes: First, two sets of single-channel phase-shifted fringe images with different frequencies and different step numbers are generated according to the following formula: Among them, the first group is the intermediate frequency three-step phase shift stripes {I m1 , I m2 , I m3 }, the second group is a high-frequency six-step phase-shifted fringe sequence {I h1 , I h2 , I h3 , I h4 , I h5 , I h6 }, A is the background light intensity of the image, B is the modulation degree of the stripes, f m is the intermediate frequency fringe frequency, f h is the high-frequency fringe frequency, i represents the i-th phase-shifted fringe pattern; Then, the stripes of two frequencies and two step numbers are combined to generate three RGB three-channel color phase-shifted sinusoidal stripe images {I c1 , I c2 , I c3 }, where I c1 The three channels are placed in {I h1 , I m1 , I h2 }, I c2 The three channels are placed in {I h3 , I m2 , I h4 }, I c3 The three channels are placed in {I h5 , I m3 , I h6 }.

3. The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning according to claim 1 is characterized in that: The process of obtaining the mask M of the object to be measured from the fringe image of the object to be measured by a computer includes: Using the three RGB three-channel color phase-shifted sinusoidal fringe images, the fringe I of the G channel m1 ~I m3 , calculate its background intensity map img_b and modulation map img_m according to the following formula: Set the thresholds respectively and get the intensity mask M according to the background intensity map img_b b , use the modulation map img_m to get the modulation mask M m , and finally take the union of the two masks as the final object mask M.

4. The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning according to claim 1 is characterized in that: The process of using the object mask M to determine different targets and performing target segmentation on each RGB channel in the object stripe image includes: First, the connected region marking method is used to mark the connected regions in the mask M of the object to be measured, and the number and position of the targets are determined; Then, count the number of pixels in each connected domain and set an appropriate pixel threshold. Define the connected domain smaller than the threshold as a low-resolution target. Take the corresponding position of the target in the mask M of the object to be tested to generate the mask M. low， And use the mask M and M low Subtract and get the mask M of the remaining area high ; Next, M low Multiplying with the R and B channel color phase-shifted sinusoidal fringe patterns, we get the segmented low-resolution area fringe pattern I low_R and I low_B ; The mask M of the remaining area high Multiplying with the G channel color phase-shifted sinusoidal fringe image, we get the segmented high-resolution area fringe image I high_G .

5. The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning according to claim 1 is characterized in that: The deep neural network includes an encoding part and a decoding part. In the encoding part, the low-resolution area fringe image I low_R and I low_B And high resolution regional fringe pattern I high_G All the high-level features of the image are extracted through convolution and downsampling; The decoding part is divided into two branches, in which the low-resolution area fringe pattern I lowR and I low_B Entering branch 1, after deconvolution and upsampling, the stripe super-resolution image is output; the high-resolution area stripe image I high_G Entering branch 2, after deconvolution and upsampling, the saturated stripe correction image is output; the weights of branch 1 and branch 2 are not shared.

6. The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning according to claim 1 is characterized in that: The formula for phase extraction of fringe super-resolution image and saturation fringe correction image is as follows: Among them, I sup_u , Ф sup and k h are the super-resolution fringe, super-resolution phase and high-frequency fringe level output by the network, u is the u-th super-resolution fringe image; I cor_v , Ф cor and k m The saturation optimization fringe, saturation error correction phase and intermediate frequency fringe level of the network output are respectively, and v is the vth saturation optimization fringe image.

7. The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning according to claim 6, characterized in that: When performing phase-height mapping, the system calibration is used to map the extracted phase results to the actual physical size to obtain depth information: Among them, a sup 、b sup 、c sup is the parameter used to calibrate the system using high-frequency fringes, which is used to map the super-resolution phase to three-dimensional coordinates. cor 、b cor 、c cor Parameters used to calibrate the system using intermediate frequency stripes, used to map the saturation correction phase to three-dimensional coordinates; Finally, the three-dimensional coordinates in the two cases are integrated into the same coordinate system to complete the three-dimensional measurement.

Images