High-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning

Abstract

The invention discloses a high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning, and the method comprises the steps: generating an RGB three-channel color phase shift sine fringe pattern, loading a DLP light source, and projecting the DLP light source to the surface of a measured object; acquiring a stripe image of a measured object with color structured light stripes, and acquiring a mask from the stripe image; based on the mask of the measured object, performing target segmentation on RGB channels in the stripe image of the measured object to obtain a low-resolution region stripe pattern and a high-resolution region stripe pattern; inputting the segmented images into a deep neural network at the same time to obtain a stripe super-resolution image and a saturated stripe correction image; phase extraction and phase-height mapping are carried out on the stripe super-resolution image and the saturated stripe correction image, and three-dimensional measurement is completed by combining two obtained phase-height mapping results. According to the invention, through multi-channel information fusion and deep neural network processing, the measurement precision and robustness in a large depth range can be effectively improved, and high-resolution three-dimensional measurement is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional measurement, and in particular to a high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning. Background Art

[0002] With the rapid development of computer vision, three-dimensional measurement has become the core research direction of computer vision. Three-dimensional measurement refers to the precise measurement of the size, shape, position and other attributes of certain three-dimensional objects or three-dimensional scenes. The measured data is convenient for computer representation and processing. In the actual measurement process, three-dimensional measurement is a process of describing the images of objects, scenes and human bodies in three-dimensional space. By extracting information from two-dimensional images or sensor data, the precise geometric parameters of three-dimensional objects, scenes and dynamic human bodies are obtained. Therefore, three-dimensional measurement is a key technology for establishing virtual reality in computers to express the objective world.

[0003] Three-dimensional measurement based on structured light has been widely used in modern industrial inspection, surveying and mapping, intelligent manufacturing, reverse engineering and other industries. Traditional structured light measurement methods face challenges in deep measurement scenes. There are two main reasons: 1) The resolution of distant objects in the field of view drops sharply, which makes it easy for phase recovery to lose detail information and reduce measurement accuracy; 2) Close objects are easily affected by light intensity saturation, which causes the sinusoidal nature of the stripes to be destroyed and it is difficult to recover high-quality phases. Existing methods usually use simple interpolation methods when processing low-resolution stripe images. Such methods have limited effects when facing complex morphology stripe images. In addition, when facing light intensity saturation, traditional methods often use multiple exposures or adaptive projection methods. Such methods require adding a large number of stripe images with different light intensities, which greatly reduces the measurement efficiency. Therefore, it is necessary to study a simple and efficient method that overcomes the problems of low resolution and light intensity saturation at the same time, so as to promote the development of structured light three-dimensional measurement technology in deep measurement scenes. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning. Compared with the traditional phase shift method, this method requires fewer frames and can improve the three-dimensional measurement speed; by restoring the details of the fringe image and correcting the distortion of the saturated fringes, the robustness of the three-dimensional measurement can be improved.

[0005] To achieve the above purpose, the technical solution provided by the present invention is:

[0006] A high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning, comprising:

[0007] The computer generates three RGB three-channel color phase-shifted sinusoidal fringe images, which are loaded into the DLP light source and projected onto the surface of the object being measured;

[0008] The camera collects the stripe image of the object under test with the color structured light stripes, and the computer obtains the mask M of the object under test from the stripe image of the object under test;

[0009] The mask M of the object to be tested is used to determine different targets, and the RGB channels in the fringe image of the object to be tested are segmented to obtain the low-resolution fringe image I. low_R and I low_B And high resolution regional fringe pattern I high_G ;

[0010] The segmented low-resolution area fringe image I low_R and I low_B And high resolution regional fringe pattern I high_G At the same time, the deep neural network is input, and after being processed by the deep neural network, the fringe super-resolution image and the saturated fringe correction image are output respectively;

[0011] Phase extraction and phase-height mapping are performed on the fringe super-resolution image and the saturated fringe correction image, and the two phase-height mapping results are combined to complete the three-dimensional measurement.

[0012] This technical solution can effectively improve the measurement accuracy and robustness within a large depth range through multi-channel information fusion and deep neural network processing, while achieving high-resolution three-dimensional measurement, especially with strong adaptability in complex scenes.

[0013] Further, the process of computer generating RGB three-channel color phase-shifted sinusoidal fringe pattern includes:

[0014] First, two sets of single-channel phase-shifted fringe images with different frequencies and different step numbers are generated according to the following formula:

[0015]

[0016] Among them, the first group is the intermediate frequency three-step phase shift stripes {I m1 , I m2 , I m3}, the second group is a high-frequency six-step phase-shifted fringe sequence {I h1 , I h2 , I h3 , I h4 , I h5 , I h6}, A is the background light intensity of the image, B is the modulation degree of the stripes, f m is the intermediate frequency fringe frequency, f h is the high-frequency fringe frequency, i represents the i-th phase-shifted fringe pattern;

[0017] Then, the stripes of two frequencies and two step numbers are combined to generate three RGB three-channel color phase-shifted sinusoidal stripe images {Ic1 , I c2 , I c3}, where I c1 The three channels are placed in {I h1 , I m1 , I h2}, I c2 The three channels are placed in {I h3 , I m2 , I h4}, I c3 The three channels are placed in {I h5 , I m3 , I h6}.

[0018] Furthermore, the process of obtaining the mask M of the object to be measured from the fringe image of the object to be measured by the computer includes:

[0019] Using the three RGB three-channel color phase-shifted sinusoidal fringe images, the fringe I of the G channel m1 ~I m3 , calculate the background intensity map img_b and modulation map img_m according to the following formula:

[0020]

[0021] Set the thresholds respectively and get the intensity mask M according to the background intensity map img_b b , use the modulation map img_m to get the modulation mask M m , and finally take the union of the two masks as the final mask M of the object to be tested.

[0022] Furthermore, the process of determining different targets by using the mask M of the object to be tested and performing target segmentation on each RGB channel in the stripe image of the object to be tested includes:

[0023] Firstly, the connected region marking method is used to mark the connected regions in the mask M of the object to be tested, and the number and position of the targets are determined;

[0024] Then, the number of pixels in each connected domain is counted, and an appropriate pixel number threshold is set. The connected domains smaller than the threshold are defined as low-resolution targets. The corresponding position of the target in the mask M of the object to be tested is taken to generate the mask M. low， And use the mask M and M low Subtract and get the mask M of the remaining area high ;

[0025] Next, M low Multiply it with the color phase-shifted sinusoidal fringe pattern of the R and B channels to obtain the segmented low-resolution area fringe pattern I low_R and I low_B ; The mask M of the remaining areahigh Multiply it with the G channel color phase-shifted sinusoidal fringe image to obtain the segmented high-resolution regional fringe image I high_G .

[0026] Furthermore, the deep neural network includes an encoding part and a decoding part. In the encoding part, the low-resolution area fringe image I low_R and I low_B And high resolution regional fringe pattern I high_G The high-level features of the image are extracted through convolution and downsampling; in the decoding part, it is divided into two branches, in which the low-resolution area fringe image I low_R and I low_B Entering branch 1, after deconvolution and upsampling, the stripe super-resolution image is output; the high-resolution area stripe image I high_G Entering branch 2, after deconvolution and upsampling, the saturated stripe correction image is output; the weights of branch 1 and branch 2 are not shared.

[0027] Furthermore, the formula for phase extraction of the fringe super-resolution image and the saturated fringe correction image is as follows:

[0028]

[0029] Among them, I sup_u , Ф sup and k h are the super-resolution fringe, super-resolution phase and high-frequency fringe level output by the network, u is the u-th super-resolution fringe image; I cor_v , Ф cor and k m The saturation optimization fringe, saturation error correction phase and intermediate frequency fringe level of the network output are respectively, and v is the vth saturation optimization fringe image.

[0030] Furthermore, when performing phase-height mapping, the extracted phase results are mapped to the real physical size using system calibration to obtain depth information:

[0031]

[0032] Among them, a sup , b sup 、c sup is the parameter used to calibrate the system using high-frequency fringes, which is used to map the super-resolution phase to three-dimensional coordinates. cor , b cor 、c cor Parameters used to calibrate the system using intermediate frequency stripes, used to map the saturation correction phase to three-dimensional coordinates;

[0033] Finally, the three-dimensional coordinates in the two cases are merged into the same coordinate system to complete the three-dimensional measurement.

[0034] Compared with the prior art, the principles and advantages of this technical solution are as follows:

[0035] This technical solution can effectively improve the measurement accuracy and robustness within a large depth range through multi-channel information fusion and deep neural network processing, while achieving high-resolution three-dimensional measurement, especially with strong adaptability in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the services required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A schematic diagram of a three-dimensional measurement system for realizing a high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning of the present invention;

[0038] Figure 2 This is a flow chart of a high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning in the present invention;

[0039] Figure 3 A schematic diagram of a computer-generated RGB three-channel color phase-shifted sinusoidal fringe pattern;

[0040] Figure 4 is a schematic diagram of a mask M of the object to be measured;

[0041] Figure 5 Fringe patterns of low-resolution area (left) and high-resolution area (right);

[0042] Figure 6 Figure 3. The structure of a deep neural network for simultaneously optimizing low-resolution stripes and saturation stripes.

[0043] Reference numerals:

[0044] 1-computer; 2-DLP light source; 3-color CCD camera; 4-object to be measured. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with specific embodiments:

[0046] The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning described in this embodiment is implemented by a three-dimensional measurement system, such as Figure 1As shown, the three-dimensional measurement system includes a computer 1, a DLP light source 2, a color CCD camera 3, and a measured object 4; the DLP light source 2 and the color CCD camera 3 are connected to the computer 1 via data cables respectively; the DLP light source 2 and the color CCD camera 3 are at the same height.

[0047] like Figure 2 As shown, the working principle of the 3D measurement system is as follows:

[0048] S1, the computer 1 generates three RGB three-channel color phase-shifted sinusoidal fringe images, loads them into the DLP light source 2 and projects them onto the surface of the object 4 under test;

[0049] In this step, the process of the computer 1 generating three RGB three-channel color phase-shifted sinusoidal fringe images includes:

[0050] First, two sets of single-channel phase-shifted fringe images with different frequencies and different step numbers are generated according to the following formula:

[0051]

[0052] Among them, the first group is the intermediate frequency three-step phase shift stripes {I m1 , I m2 , I m3}, the second group is a high-frequency six-step phase-shifted fringe sequence {I h1 , I h2 , I h3 , I h4 , I h5 , I h6}; A is the background light intensity of the image, B is the modulation degree of the stripes, f m is the intermediate frequency fringe frequency, f h is the high-frequency fringe frequency, i represents the i-th phase-shifted fringe pattern;

[0053] Then, the stripes of two frequencies and two step numbers are combined to generate three RGB three-channel color phase-shifted sinusoidal stripe images {I c1 , I c2 , I c3}, where I c1 The three channels are placed in {I h1 , I m1 , I h2}, I c2 The three channels are placed in {I h3 , I m2 , I h4}, I c3 The three channels are placed in {I h5 , I m3 , I h6},like Figure 3 shown.

[0054] S2, collecting a stripe image of the object under test with color structured light stripes by a color CCD camera 3, and obtaining a mask M of the object under test from the stripe image of the object under test by a computer 1;

[0055] In this step, the process of obtaining the mask M of the object to be measured from the fringe image of the object to be measured by the computer includes:

[0056] Using the three RGB three-channel color phase-shifted sinusoidal fringe images, the fringe I of the G channel m1 ~I m3 , calculate the background intensity map img_b and modulation map img_m according to the following formula:

[0057]

[0058] Set appropriate thresholds respectively and obtain the intensity mask M according to the background intensity map img_b b , use the modulation map img_m to get the modulation mask M m , and finally take the union of the two masks as the final object mask M, as Figure 4 shown.

[0059] S3, using the mask M of the object to be tested to determine different targets, perform target segmentation on each RGB channel in the fringe image of the object to be tested, and segment out the low-resolution area fringe image I low_R and I low_B And high resolution regional fringe pattern I high_G ;

[0060] The process of this step includes:

[0061] First, the connected region marking method is used to mark the connected regions in the mask M of the object to be tested, and the number and position of the targets are determined;

[0062] Then, the number of pixels in each connected domain is counted, and an appropriate pixel number threshold is set. The connected domains smaller than the threshold are defined as low-resolution targets. The corresponding position of the target in the mask M of the object to be tested is taken to generate the mask M. low， And use the mask M and M low Subtract and get the mask M of the remaining area high ;

[0063] Next, M low Multiply it with the color phase-shifted sinusoidal fringe pattern of the R and B channels to obtain the segmented low-resolution area fringe pattern I low_R and I low_B ; The mask M of the remaining area high Multiply it with the G channel color phase-shifted sinusoidal fringe image to obtain the segmented high-resolution regional fringe image I high_G ,like Figure 5 shown.

[0064] S4, the segmented low-resolution area fringe image I low_R and I low_B And high resolution regional fringe pattern I high_G At the same time, input Figure 6 The deep neural network shown in the figure outputs a fringe super-resolution image and a saturated fringe correction image respectively after being processed by the deep neural network;

[0065] The process of this step includes:

[0066] First, the three segmented fringe images are grouped into a set {I low_R , I low_B , I high_G Input to a deep neural network that optimizes both low-resolution and saturation fringes.

[0067] Then, the deep neural network is divided into an encoding part and a decoding part. In the encoding part, the three images are convolved and downsampled to extract high-order features of the images;

[0068] Next, the downsampled fringe image is divided into two branches in the decoding part, where I low_R and I low_B Enter branch 1, and after deconvolution and upsampling, output the stripe super-resolution image; I high_G Entering branch 2, after deconvolution and upsampling, the saturated stripe correction image is output. The weights of the two branches are not shared.

[0069] S5. Perform phase extraction and phase-height mapping on the fringe super-resolution image and the saturated fringe correction image, and combine the two phase-height mapping results to complete the three-dimensional measurement.

[0070] In this step, the formula for phase extraction of the fringe super-resolution image and the saturated fringe correction image is as follows:

[0071]

[0072] Among them, I sup_u , Ф sup and k h are the super-resolution fringe, super-resolution phase and high-frequency fringe level output by the network, u is the u-th super-resolution fringe image; I cor_v , Ф cor and k m The saturation optimization fringe, saturation error correction phase and intermediate frequency fringe level of the network output are respectively, and v is the vth saturation optimization fringe image.

[0073] When performing phase-height mapping, the extracted phase results are mapped to the actual physical size using system calibration to obtain depth information:

[0074]

[0075] Among them, a sup , b sup 、c sup is the parameter used to calibrate the system using high-frequency fringes, which is used to map the super-resolution phase to three-dimensional coordinates. cor , b cor 、c cor Parameters used to calibrate the system using intermediate frequency stripes, used to map the saturation correction phase to three-dimensional coordinates;

[0076] Finally, the three-dimensional coordinates in the two cases are merged into the same coordinate system to complete the three-dimensional measurement.

[0077] The embodiments described above are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made according to the shape and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning, characterized in that: include: The computer generates three RGB three-channel color phase-shifted sinusoidal fringe images, which are loaded into the DLP light source and projected onto the surface of the object being measured; The camera collects the stripe image of the object under test with the color structured light stripes, and the computer obtains the mask M of the object under test from the stripe image of the object under test; The mask M of the object to be tested is used to determine different targets, and the RGB channels in the fringe image of the object to be tested are segmented to obtain the low-resolution fringe image I. low_R and I low_B And high resolution regional fringe pattern I high_G ; The low-resolution area fringe image I low_R and I low_B And high resolution regional fringe pattern I high_G At the same time, the deep neural network is input, and after being processed by the deep neural network, the fringe super-resolution image and the saturated fringe correction image are output respectively; Phase extraction and phase-height mapping are performed on the fringe super-resolution image and the saturated fringe correction image, and the two phase-height mapping results are combined to complete the three-dimensional measurement.

2. According to claim 1, a high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning is characterized in that: The process of computer-generated RGB three-channel color phase-shifted sinusoidal fringe pattern includes: First, two sets of single-channel phase-shifted fringe images with different frequencies and different step numbers are generated according to the following formula: Among them, the first group is the intermediate frequency three-step phase shift stripes {I m1 , I m2 , I m3 }, the second group is a high-frequency six-step phase-shifted fringe sequence {I h1 , I h2 , I h3 , I h4 , I h5 , I h6 }, A is the background light intensity of the image, B is the modulation degree of the stripes, f m is the intermediate frequency fringe frequency, f h is the high-frequency fringe frequency, i represents the i-th phase-shifted fringe pattern; Then, the stripes of two frequencies and two step numbers are combined to generate three RGB three-channel color phase-shifted sinusoidal stripe images {I c1 , I c2 , I c3 }, where I c1 The three channels are placed in {I h1 , I m1 , I h2 }, I c2 The three channels are placed in {I h3 , I m2 , I h4 }, I c3 The three channels are placed in {I h5 , I m3 , I h6 }.

3. The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning according to claim 1 is characterized in that: The process of obtaining the mask M of the object to be measured from the fringe image of the object to be measured by a computer includes: Using the three RGB three-channel color phase-shifted sinusoidal fringe images, the fringe I of the G channel m1 ~I m3 , calculate the background intensity map img_b and modulation map img_m according to the following formula: Set the thresholds respectively and get the intensity mask M according to the background intensity map img_b b , use the modulation map img_m to get the modulation mask M m , and finally take the union of the two masks as the final mask M of the object to be tested.

4. The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning according to claim 1 is characterized in that: The process of determining different targets by using the mask M of the object to be tested and performing target segmentation on each RGB channel in the stripe image of the object to be tested includes: Firstly, the connected region marking method is used to mark the connected regions in the mask M of the object to be tested, and the number and position of the targets are determined; Then, the number of pixels in each connected domain is counted, and an appropriate pixel number threshold is set. The connected domains smaller than the threshold are defined as low-resolution targets. The corresponding position of the target in the mask M of the object to be tested is taken to generate the mask M. low， And use the mask M and M low Subtract and get the mask M of the remaining area high ; Next, M low Multiply it with the color phase-shifted sinusoidal fringe pattern of the R and B channels to obtain the segmented low-resolution area fringe pattern I low_R and I low_B ; The mask M of the remaining area high Multiply it with the G channel color phase-shifted sinusoidal fringe image to obtain the segmented high-resolution regional fringe image I high_G .

5. The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning according to claim 1 is characterized in that: The deep neural network includes an encoding part and a decoding part. In the encoding part, the low-resolution area fringe image I low_R and I low_B And high resolution regional fringe pattern I high_G All of them are convolved and downsampled to extract high-level features of the image; The decoding part is divided into two branches, in which the low-resolution area fringe pattern I lowR and I low_B Enter branch 1, after deconvolution and upsampling, output stripe super-resolution image; high-resolution area stripe image I high_G Entering branch 2, after deconvolution and upsampling, the saturated stripe correction image is output; the weights of branch 1 and branch 2 are not shared.

6. The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning according to claim 1 is characterized in that: The formula for phase extraction of fringe super-resolution image and saturated fringe correction image is as follows: Among them, I sup_u , Ф sup and k h are the super-resolution fringe, super-resolution phase and high-frequency fringe level output by the network, u is the u-th super-resolution fringe image; I cor_v , Ф cor and k m The saturation optimization fringe, saturation error correction phase and intermediate frequency fringe level of the network output are respectively, and v is the vth saturation optimization fringe image.

7. The high-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning according to claim 6 is characterized in that: When performing phase-height mapping, the extracted phase results are mapped to the actual physical size using system calibration to obtain depth information: Among them, a sup , b sup 、c sup is the parameter used to calibrate the system using high-frequency fringes, which is used to map the super-resolution phase to three-dimensional coordinates. cor , b cor 、c cor Parameters used to calibrate the system using intermediate frequency stripes, used to map the saturation correction phase to three-dimensional coordinates; Finally, the three-dimensional coordinates in the two cases are merged into the same coordinate system to complete the three-dimensional measurement.

Images