Structured light three-dimensional measurement method and device based on deep learning and storage medium
Through the structured light three-dimensional measurement method based on deep learning, the optimal projection pattern is determined using pre-trained neural networks, which solves the problem of low three-dimensional reconstruction efficiency of high-reflective objects, and realizes efficient and accurate three-dimensional reconstruction, reducing system complexity.
Patent Information
- Application Number
- CN202410828095.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-25
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art has low efficiency in three-dimensional reconstruction of objects with a large range of high reflectivity or surface reflectivity variations, and common methods are difficult to achieve pixel-level brightness control, increasing system complexity.
Using the three-dimensional structured light measurement method based on deep learning, the uniform brightness image and coded stripe image of the object surface are obtained through the pre-trained target deep neural network, and the optimal projection pattern is determined, so as to realize adaptive projection in high dynamic range to avoid pixel saturation areas. Improve the efficiency of three-dimensional reconstruction.
It realizes efficient three-dimensional reconstruction of objects with high reflectance or large surface reflectivity changes, reduces system complexity, improves the efficiency and accuracy of three-dimensional reconstruction, and avoids the problem of large-area missing of three-dimensional point clouds.
Smart Images

Figure CN118941703A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a structured light three-dimensional measurement method, device and storage medium based on deep learning. Background Art
[0002] 3D reconstruction technology is an important means of obtaining information for perceiving the 3D shape of the real world, and structured light vision technology has become the mainstream optical method in 3D sensing technology due to its non-contact, high-precision, and high-efficiency characteristics. However, in the face of actual measurement scenarios, when the active optical measurement method is encoding the surface shape of the measured object, the projected light beam is easily saturated in the camera that collects fringe modulation information after being reflected by the surface of the object with high reflectivity, resulting in the inability to resolve the height information of the object in this area and the loss of a large area of point cloud.
[0003] At present, the three-dimensional reconstruction of objects with high reflectivity or large surface reflectivity variations mainly includes methods such as changing camera exposure parameters, changing projection brightness, and using additional hardware devices. However, the method of changing camera exposure parameters is difficult to achieve pixel-level brightness control and it is difficult to select the best exposure parameters; the method of changing projection brightness requires projecting a large number of stripe sequences with different brightness to obtain an adaptive projection pattern, which has the disadvantage of low measurement efficiency; the use of additional hardware devices, such as polarizers, usually reduces the signal-to-noise ratio of the image, and multiple cameras increase the complexity of the system. Summary of the invention
[0004] The present invention provides a structured light three-dimensional measurement method, device and storage medium based on deep learning, which are used to solve the technical problem of low efficiency of three-dimensional reconstruction of objects with high reflective characteristics or a large range of surface reflectivity changes in the prior art.
[0005] In a first aspect, the present invention provides a structured light three-dimensional measurement method based on deep learning, comprising: Obtain uniform brightness image and coded fringe image modulated by the surface of the object; Using a pre-trained target deep neural network and based on the uniform brightness image and the coded fringe image, an optimal projection pattern of the projector plane is obtained; the target deep neural network is used to classify pixels of the image and determine the optimal projection grayscale value corresponding to each type of pixel; A target coded stripe image modulated by the surface of the object is acquired by projecting the optimal projection pattern of the projector plane, and a three-dimensional point cloud of the surface of the object is acquired based on the target coded stripe image.
[0006] In some embodiments, the method of using a pre-trained target deep neural network and obtaining an optimal projection pattern of a projector plane based on the uniform brightness image and the coded fringe image includes: The uniform brightness image is input into a pre-trained target deep neural network to obtain the optimal projection pattern of the camera plane; based on the optimal projection pattern of the camera plane, the optimal projection pattern of the projector plane is obtained by utilizing the homography relationship between the pixel saturation area of the camera image and the corresponding area of the projector projection plane; the homography relationship is obtained based on the solution of the coded fringe image.
[0007] In some embodiments, the target deep neural network includes a pixel classification network and an optimal brightness prediction network; wherein the input of the pixel classification network is a uniform brightness image captured under different projection brightness sequences, and the output is the category of each pixel of the uniform brightness image; the input of the optimal brightness prediction network is a uniform brightness image captured under different projection brightness sequences, and the output is an optimal brightness vector, and the optimal brightness prediction network includes a feature extraction backbone network and a fully connected network.
[0008] In some embodiments, the target deep neural network is trained by the following steps: Acquire a training data set; the training data set includes a simulated data set and a real data set; Pre-training a backbone network in the target deep neural network using a public data set; the backbone network includes a backbone network of the pixel classification network and a backbone network of the optimal brightness prediction network; Training the target deep neural network using the simulated data set; The target deep neural network is fine-tuned using the real dataset.
[0009] In some embodiments, obtaining a simulation data set includes: Establish a relationship model between camera image intensity, projected image intensity, and object surface reflectivity; The reflectivity of the surface of a randomly shaped object is simulated and a series of projection images with uniform brightness are projected onto the surface of the randomly shaped object. The camera image intensity corresponding to the projection image and the optimal projection grayscale image of the camera plane are obtained using the relationship model to obtain a simulated data set.
[0010] In some embodiments, the real data set is a uniform brightness image and a corresponding camera plane optimal projection grayscale image obtained by actual measurement using a variety of reflective objects.
[0011] In some embodiments, inputting the uniform brightness image into a pre-trained target deep neural network to obtain an optimal projection pattern on a camera plane includes: Using a pixel classification network in the target deep neural network to classify pixels in the uniform brightness image, to obtain a pixel classification result; Based on the pixel classification result and using the optimal brightness prediction network in the target deep neural network, an optimal projection pattern of the camera plane is obtained.
[0012] In a second aspect, the present invention provides a structured light three-dimensional measurement device based on deep learning, comprising the following modules.
[0013] A first acquisition module is used to acquire a uniform brightness image and a coded fringe image modulated by the surface of the object; A second acquisition module is used to obtain the optimal projection pattern of the projector plane based on the uniform brightness image and the coded fringe image by using a pre-trained target deep neural network; the target deep neural network is used to classify the pixels of the image and determine the optimal projection grayscale value corresponding to each type of pixel; The third acquisition module is used to acquire a target coded stripe image modulated by the surface of the object by projecting the optimal projection pattern of the projector plane, and acquire a three-dimensional point cloud of the surface of the object based on the target coded stripe image.
[0014] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the structured light three-dimensional measurement method based on deep learning as described in the first aspect above is implemented.
[0015] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the structured light three-dimensional measurement method based on deep learning as described in the first aspect above.
[0016] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the structured light three-dimensional measurement method based on deep learning as described in the first aspect above.
[0017] The deep learning-based structured light three-dimensional measurement method, device and storage medium provided by the present invention obtain a uniform brightness image and a coded stripe image modulated by the surface of an object; use a pre-trained target deep neural network and obtain the best projection pattern of the projector plane based on the uniform brightness image and the coded stripe image; the target deep neural network is used to classify the image pixels and determine the best projection grayscale value corresponding to each type of pixel; obtain a target coded stripe pattern modulated by the surface of the object by projecting the best projection pattern of the projector plane, and obtain a three-dimensional point cloud on the surface of the object based on the target coded stripe pattern, and use the trained target deep neural network to accurately and efficiently obtain the best projection pattern with a high dynamic range, thereby realizing efficient three-dimensional reconstruction of the surface of objects with a large reflectivity variation range or high reflectivity, solving the problem of large-area missing of three-dimensional point clouds that is prone to occur when the measured object has the characteristics of high reflectivity or large surface reflectivity variation, and improving the efficiency of three-dimensional reconstruction of objects with high reflectivity or large surface reflectivity variation. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 This is one of the flow charts of the structured light three-dimensional measurement method based on deep learning provided by the present invention.
[0020] Figure 2 It is a schematic diagram of the structure of the target deep neural network provided by the present invention.
[0021] Figure 3 It is a schematic diagram of the structure of the structured light three-dimensional measurement device based on deep learning provided by the present invention.
[0022] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0024] Figure 1is one of the flow charts of the structured light three-dimensional measurement method based on deep learning provided by the present invention, such as Figure 1 As shown, the present invention provides a structured light three-dimensional measurement method based on deep learning. The method comprises the following steps: Step 101: Acquire a uniform brightness image and a coded fringe image modulated by the surface of an object.
[0025] Specifically, for the object under test, a digital light processing (DLP) projector is used to project a uniform brightness image sequence and a coded fringe image sequence, and a camera is synchronously triggered to collect the above-mentioned uniform brightness image and coded fringe image modulated by the object surface.
[0026] In an embodiment of the present application, when building the system, the camera and the projector are placed side by side, and the angle between the two is adjusted so that the projection format is exactly located at the imaging center of the camera and occupies most of the camera's field of view, and they are calibrated to obtain the intrinsic and extrinsic parameters of the two; further, when the system starts working, the computer transmits the pre-generated encoded image to the projector via HDMI in a certain sequence. Every time the projector projects an image, it simultaneously triggers the camera to start collecting the surface information of the current object under test. Finally, the camera transmits the collected image sequence back to the computer, completing a closed-loop process of obtaining information on the object under test.
[0027] Optionally, the coded stripe image / pattern may be a Gray coding pattern, a phase shift coding pattern, a Gray and phase shift mixed coding pattern, a Gray and line shift mixed coding pattern, or the like.
[0028] Step 102: Utilize a pre-trained target deep neural network and obtain the optimal projection pattern of the projector plane based on the uniform brightness image and the coded fringe image; the target deep neural network is used to classify pixels of the image and determine the optimal projection grayscale value corresponding to each type of pixel.
[0029] Specifically, the target deep neural network is pre-trained to classify the pixels of the image and determine the optimal projected grayscale value corresponding to each type of pixel. The input is a series of images captured by the camera under different projection brightness sequences, and the output is the optimal projection pattern on the camera plane.
[0030] For example, a series of uniform brightness images captured by the camera under different projection brightness sequences are input into the pre-trained target deep neural network to obtain the optimal projection pattern of the camera plane. Then, the correlation between the camera image and the corresponding area of the projector plane is used to obtain the optimal projection pattern of the projector plane based on the optimal projection pattern of the camera plane.
[0031] Step 103: Acquire a target coded stripe image modulated by the surface of the object by projecting the optimal projection pattern on the projector plane, and acquire a three-dimensional point cloud of the surface of the object based on the target coded stripe image.
[0032] Specifically, after obtaining the best projection pattern on the projector plane, the best projection pattern on the projector plane is projected to obtain a target coded stripe image modulated by the object surface, and then the target coded stripe image is decoded and triangulated to obtain a complete three-dimensional point cloud of the object surface with high reflective properties.
[0033] Among them, the stripe coding method of the target coded stripe image can adopt Gray coding, phase shift coding, Gray and phase shift mixed coding, Gray and line shift mixed coding, etc.; at the same time, sub-pixel decoding accuracy can be achieved by projecting positive and negative stripes to improve the ability to resist surface reflection.
[0034] The deep learning-based structured light 3D measurement method provided in the embodiment of the present application uses a trained target deep neural network to accurately and efficiently obtain the optimal projection pattern with a high dynamic range, thereby achieving efficient 3D reconstruction of the surface of objects with a large reflectivity variation range or high reflectivity, solving the problem of large-area missing of 3D point clouds that is easily caused when the measured object has the characteristics of high reflectivity or large variations in surface reflectivity, improving the efficiency and accuracy of 3D reconstruction of objects with high reflectivity or large variations in surface reflectivity, and reducing system complexity.
[0035] In some embodiments, the target deep neural network includes a pixel classification network and an optimal brightness prediction network; wherein the input of the pixel classification network is a uniform brightness image captured under different projection brightness sequences, and the output is the category of each pixel of the uniform brightness image; the input of the optimal brightness prediction network is a uniform brightness image captured under different projection brightness sequences, and the output is an optimal brightness vector, and the optimal brightness prediction network includes a feature extraction backbone network and a fully connected network.
[0036] Specifically, the target deep neural network is used to classify pixels in the image and determine the optimal projected grayscale value corresponding to each type of pixel. That is, the target deep neural network has two goals: the first is to achieve accurate classification of each pixel to correspond to the correct pixel value interval; the second is to obtain the optimal projected grayscale value for each of the above intervals. The deep neural network model includes two branches, the pixel classification network and the optimal brightness prediction network.
[0037] The pixel classification network input is the uniform brightness image captured by the camera under different projection brightness sequences, and the output is the category of each pixel.
[0038] For the first goal, pixel classification can be achieved using classic image segmentation networks, such as U-Net, SegNet, PsPNet, and Deeplab-v3. The present application embodiment proposes a pixel classification network (PCN) to complete pixel classification, which has a network structure with a high-resolution feature extraction backbone. Figure 2 is a schematic diagram of the structure of the target deep neural network provided by the present invention, such as Figure 2 As shown in the figure, the PCN structure is shown in the upper branch in the figure. In the network diagram, convolution represents conv(k, s, p) as (kernel size, convolution step size, zero padding size), cbam represents the convolutional block attention module (Convolutional Block Attention Module), and resize is the size adjustment operation. After 4 layers of convolution for preliminary feature extraction, the size of the feature map becomes 120×120×64, and it will enter the subsequent PCN backbone for feature extraction. Enter the 4 BottleneckBlocks in turn for further feature extraction (Bottleneck is the basic structure of residual convolution used by ResNet50 and 101), and obtain a 256-layer feature map. Taking this as the starting point, multi-scale feature extraction is performed on this multi-layer feature map.
[0039] Specifically, Figure 2 As shown in the figure, the first branch will maintain a high resolution of 120×120×32 during the convolution process, and try to learn the high-frequency features of the image; it will downsample in sequence through convolution, and use Basic Block for continuous feature extraction (Basic Block is the basic structure of residual convolution used by ResNet18 and ResNet32), accompanied by feature map downsampling (higher resolution features pass through the (3,2,1) downsampling convolution layer and BN, and are added to the lower resolution convolution layer feature values and then pass through the ReLu activation function) and upsampling (lower resolution features pass through the (1,1,0) convolution layer and BN, and the nearest neighbor is upsampled to the size of the higher resolution feature map and then added to it and passed through BN) to initially merge feature values of different scales with each other, and continuously generate feature representations of multi-level sizes of 60×60×64, 30×30×128, and 15×15×256 to obtain feature information of different scales in the image. Finally, the feature maps of different scales are upsampled to 120×120 and stacked together. After feature integration by (1,1,0) convolution, the number of channels is adjusted again by (1,1,0) convolution to unify the number of predicted categories ( Figure 2The dashed arrows in the figure indicate the flow direction. As can be seen from the structure, the backbone network in PCN does not perform high-width compression of feature maps with the depth of convolution like common convolutional neural networks (such as VGG, MobileNet, ResNet, etc.) to extract features from top to bottom. Instead, it changes the working mode of the traditional convolutional neural network model so that the network always maintains high-resolution feature maps when working, which improves the preservation of high-frequency information and the accuracy of pixel classification.
[0040] Secondly, in order to make the network pay more attention to the pixel saturation areas caused by reflections and the feature image channels corresponding to these areas during the learning and prediction process, the CBAM (Convolutional Block Attention Module) attention mechanism, which is very suitable for this requirement, is introduced in PCN. It has dual attention degrees of freedom in space and channels, and can adaptively learn and increase the weights of key pixel position areas and image channels in feature maps, improving the learning ability of the network and the final online prediction performance. Finally, the PCN network outputs a 21-dimensional one-hot vector for each pixel category, and the number of positions with a value of 1 is the corresponding classification category.
[0041] For the second goal, we only need to consider which feature extraction backbone to use to accurately extract the interval optimal grayscale value information from the image. To meet this requirement, a brightness prediction network (BPN) is proposed to realize the information conversion from multi-channel images to optimal grayscale vectors, such as Figure 2 As shown in the figure, the structure of BPN is shown in the lower branch in the figure. After the initial information extraction by PCN, the feature layer is adjusted to 240×240×64 through the large convolution kernel convolution layer and then enters the classic ResNet-50 backbone network for feature extraction. Since BPN needs to output the best gray value vector information, the highly concentrated feature map information is finally transformed into a vector output through three fully connected network layers with layers of 1024, 128 and 20 respectively. Since the output at this time is a 20-dimensional vector with values between 0 and 254 (i.e., the vector in the figure), the 255 pixel value as the background must be added to the last dimension of this vector, that is, BPN finally outputs a 21-dimensional vector, which is consistent with the number of pixel categories output by PCN. Finally, the predicted category of each pixel of PCN is matched with the corresponding position of the vector obtained by BPN (through vector dot product) to generate the final adaptive best projection grayscale image as the final output of the deep neural network.
[0042] The deep learning-based structured light three-dimensional measurement method provided in the embodiment of the present application utilizes a target deep neural network to realize pixel classification and optimal brightness prediction, realizes the automatic acquisition of an adaptive optimal projection grayscale image with a high dynamic range, and realizes optimal pixel-level projection intensity prediction while having high computational efficiency, thereby avoiding complex matrix operations in traditional methods and improving the efficiency of three-dimensional reconstruction.
[0043] In some embodiments, the target deep neural network is trained by the following steps: Acquire a training data set; the training data set includes a simulated data set and a real data set; Pre-training a backbone network in the target deep neural network using a public data set; the backbone network includes a backbone network of the pixel classification network and a backbone network of the optimal brightness prediction network; Training the target deep neural network using the simulated data set; The target deep neural network is fine-tuned using the real dataset.
[0044] Specifically, a training data set is obtained, wherein the training data set includes a simulated data set and a real data set. The simulated data set is a uniform brightness image and a corresponding camera plane optimal projection grayscale image obtained by simulating imaging rules by establishing a physical model, and the real data set is a uniform brightness image and a corresponding camera plane optimal projection grayscale image obtained by actual measurement using a variety of reflective objects.
[0045] In some embodiments, obtaining a simulation data set includes: Establish a relationship model between camera image intensity, projected image intensity, and object surface reflectivity; The reflectivity of the surface of a randomly shaped object is simulated and a series of projection images with uniform brightness are projected onto the surface of the randomly shaped object. The camera image intensity corresponding to the projection image and the optimal projection grayscale image of the camera plane are obtained using the relationship model to obtain a simulated data set.
[0046] Specifically, the training dataset for training the target deep neural network includes a simulated dataset and a real dataset.
[0047] Among them, the simulation data set is a uniform brightness image and the corresponding camera plane optimal projection grayscale image obtained by establishing a physical model to simulate the imaging law.
[0048] First, a relationship model is established between the camera image intensity, the projected image intensity, and the object surface reflectivity. Assume that the projection intensity of the projector at a certain point is I p , the position of a pixel on the object surface in the camera coordinate system (x c ,y c ) has a reflectivity of α(xc ,y c ), the mutual reflection intensity I at this point i , the ambient light β1(x c ,y c ) and the ambient light β2(x c ,y c ), camera gain k, exposure time t and noise I n (x c ,y c ). That is, for a certain pixel value I(x c ,y c ) is expressed as: I(x c ,y c )=kt·α(β1+I p +I i )+kt·β2+I n .(1)
[0049] In order to estimate the values of parameters α and β, different brightness sequences can be projected onto the target. (i=1,2,…,n) uniform pattern, and obtain n sets of equations (1) from the image (I=(I1,I2,…,I n )). Let us assume that the sensor noise I n It conforms to the Gaussian distribution, that is, I n ~N(0,σ 2 ), and at the same time x i2 = kt, X = (x 11 x 12 , x 21 x 22 , …, x n1 x n2 ) T , and β=α(β1+I i )+β2, we can solve for the estimated values of α and β:
[0050] That is, at this time, it is estimated that the pixel position (x c ,y c ) corresponds to the object surface reflectivity and related to ambient light If the expected gray value at this pixel position is set to I exp , fix the camera parameters to k exp and t exp , then we can inversely solve the optimal intensity that the projector should project under the condition of the expected grayscale value of the camera imaging:
[0051] According to formula (3), the projector can adjust the projection intensity according to the actual reflectivity of the object surface. The range is 0 to 255, which enables pixel-level adaptive high dynamic range brightness control.
[0052] In order to generate a simulated data set, formula (1) is rewritten as: I(x c ,y c )=kt·(αI P +β)+I n .(4)
[0053] Where β=α(β1+I i )+β2 is a constant. This part mainly includes the mutual reflection intensity I related to the surface structure of the object being measured. i and ambient light intensity β1, β2. But for a specific pixel position (x c ,y c ) i The specific values of β1 and β2 are not available. After analyzing a large amount of real data, the value of αIp is about 10 times that of β. Therefore, β (i.e. β1 and β2) is temporarily ignored when making the simulated data set. At this time, for the simulated data set, the mathematical model followed is simplified to: I sim (x c ,y c )=kt·αI p +I n .(5)
[0054] In formula (5), the camera-related parameters k and t are generally fixed constants. For a set of projection brightness sequences, the projector projection intensity I p is also a constant, so we only need to consider the reflectivity α(x c ,y c ) and camera noise I n (x c ,y c ).
[0055] In order to simulate the reflectivity α of the surface of a random shape object, its intensity and distribution at the pixel position are mainly considered. A two-dimensional normal distribution α with a random number n is used i To simulate the reflectivity value in (x c ,y c ), namely And the intensity ω i Set to a random value within a certain range, that is:
[0056] In order to make the two-dimensional normal distribution able to form any shape, the parameters with randomness in formula (6) are: intensity ω i , the number of normal distributions n, the center position of the distribution And the covariance matrix ∑i, where ω is i The maximum value range of and α is limited to 3.5 to 4.5. In addition, the number of n in each image is limited to 1 to 30, the distribution center is located in the image pixel range h×w, the element value on the main diagonal of the covariance matrix is limited to 1 to 100, the anti-diagonal element value is limited to -100 to 100, and the positive definiteness of ∑i is guaranteed. Under the above conditions, the simulated reflectivity α(x c ,y c ) has strong randomness and arbitrariness.
[0057] Since it is difficult to n To accurately model the image noise, we can approximate it as follows based on experience:
[0058] in, is a Gaussian white noise with an intensity of 8dBW. Furthermore, the brightness output of the DLP projector is affected by the Gamma effect, so the linearity of the grayscale value of the projected pattern is better between 40 and 220, so the grayscale value is selected to be projected onto the object. The sequence is 40, 70, 100, 130, 160, 190 and 220 (a total of 7 images are projected). Finally, the generated α, Substitute the Ip and Ip sequences into equation (5), and fix the parameters k to 1 and t to 4ms to obtain the final simulated 7-channel image sequence. Substituting into formula (3), the optimal projection grayscale value corresponding to this pixel position can be calculated: Thus, the camera plane adaptive optimal projection grayscale image is obtained.
[0059] For high dynamic range For the neural network to be constructed, it is ideal that the network can achieve 256 classifications of 256 pixel values from 0 to 255. However, the task of super-multiple classification of 256 categories has strict computing requirements, and it is even difficult to run the algorithm under general hardware conditions. Too many classifications will lead to difficulties in training and convergence of the neural network. Therefore, the pixel values from 0 to 254 are divided into 20 intervals (255 is the exclusive category for background), and then an optimal grayscale value is selected for each interval. To reduce the number of categories. The weighted tie-breaker method is used to select the best grayscale value in each pixel interval, that is:
[0060] Where m is the pixel value of the pixel interval i to j, and Λ is the number of pixels with pixel value m. This process is called classification, which classifies all pixel values from 0 to 254 into the corresponding interval and uniformly assigns the pixel values in the interval to At the same time, a vector containing the best grayscale value of each interval is obtained. At this time, the simulated input 7-channel image sequence and its corresponding true value data are obtained, which will be used for network training from scratch to enable it to learn the physical laws implied by formula (3).
[0061] Specifically, the real data set is a uniform brightness image obtained by actual measurement of various reflective objects and the corresponding camera plane optimal projection grayscale image.
[0062] Certain physical laws are followed when generating simulated data sets, but there are undeniable gaps between the simulated data and the actual measurement scenes. For example, the ambient light reflected from the object surface to the camera and the ambient light directly entering the camera are omitted, and the sensor noise is empirically approximated. Therefore, it is very necessary to introduce real data sets into the training of neural networks.
[0063] The dataset was created using a variety of reflective objects and their combinations, including a variety of metal workpieces and smooth plastic workpieces, and the same method was used to obtain the classified true value images and vectors. Finally, the dataset was augmented by image processing methods such as magnification, splicing, and rotation. The real dataset will be used for fine-tuning the overall parameters of the model in the final stage to enhance the model's adaptability to real scenes.
[0064] After obtaining the training data set, the training of the target neural network consists of three stages: In the first stage, the idea of transfer learning pre-training is used to ensure the convergence of the model. Public datasets such as the ImgNet dataset are used to pre-train the network backbones that have a major impact on the model performance, so that the network can initially learn to extract correct feature information from the image and converge the network loss to a certain range.
[0065] In the second stage, the simulated data set is used to fully train the neural network as a whole to ensure the generalization ability of the model. Since the generated images are easy to obtain, sufficient data can be used to train the network as needed in this stage. The purpose is to introduce physical constraints to the network so that it can learn the mathematical laws described by equation (5) to get closer to the task goal. During training, the backbone layer of the network is first frozen, and the other layers are trained until convergence (these layers are not pre-trained in the first stage). Then the backbone layer is unfrozen and the entire network is fully trained. This can shorten the training time required for model convergence.
[0066] In the third stage, the network is fine-tuned as a whole using real data sets to ensure the accuracy of model testing. When the simulated data is generated, it is inevitable that there will be some differences from the real data. In order to overcome the gap between them and improve the accuracy of the model in actual real measurement scenarios, it is necessary to use real data to fine-tune the model as a whole.
[0067] The structured light 3D measurement method based on deep learning provided in the embodiment of the present application uses a simulated data set and a real data set as training data sets to train the target deep neural network, and proposes a simulated data set generation method based on random reflectivity, which introduces the a priori of physical model constraints into the neural network, thereby improving the robustness of the algorithm. The backbone network of the target deep neural network is trained using a simulated data set, and then the target deep neural network is tested and fine-tuned using the real data set obtained from actual measurements, thereby improving the accuracy and generalization ability of the target deep neural network, thereby improving the accuracy of obtaining the best projection pattern using the trained target deep neural network, and improving the accuracy and efficiency of 3D reconstruction.
[0068] In some embodiments, inputting the uniform brightness image into a pre-trained target deep neural network to obtain an optimal projection pattern on a camera plane includes: Using a pixel classification network in the target deep neural network to classify pixels in the uniform brightness image, to obtain a pixel classification result; Based on the pixel classification result and using the optimal brightness prediction network in the target deep neural network, an optimal projection pattern of the camera plane is obtained.
[0069] Specifically, after completing the training of the target deep neural network, the pixel classification network in the target deep neural network is used to classify the pixels in the uniform brightness image to obtain the pixel classification results, and then the optimal brightness prediction network is used to obtain the corresponding optimal projection grayscale value for each pixel classification result to obtain the optimal projection pattern of the camera plane.
[0070] For example, the projector projects a uniform brightness image sequence, which is a uniform brightness pattern of 40, 70, 100, 130, 160, 190 and 220 (a total of 7 images are projected); then the camera collects the corresponding 7 images and inputs them into the trained target deep neural network. The output of the target deep neural network is the predicted category of each pixel and an optimal brightness vector. The predicted category of each pixel is matched with the corresponding position of the optimal brightness vector to generate the final camera plane optimal projection grayscale image.
[0071] In some embodiments, the method of using a pre-trained target deep neural network and obtaining an optimal projection pattern of a projector plane based on the uniform brightness image and the coded fringe image includes: The uniform brightness image is input into a pre-trained target deep neural network to obtain the optimal projection pattern of the camera plane; based on the optimal projection pattern of the camera plane, the optimal projection pattern of the projector plane is obtained by utilizing the homography relationship between the pixel saturation area of the camera image and the corresponding area of the projector projection plane; the homography relationship is obtained based on the solution of the coded fringe image.
[0072] Specifically, after obtaining the optimal projection pattern of the camera plane through the uniform brightness image and the pre-trained target deep neural network, the homography relationship between the pixel saturation area of the camera image and the corresponding area of the projector projection plane is combined to further obtain the optimal projection pattern of the projector plane.
[0073] In the embodiment of the present application, the homography relationship between the saturated area of the camera image pixels and the corresponding area of the projector projection plane can be solved based on the coded fringe image. The homography relationship only uses the coded fringe image in a single direction (i.e., the coded fringe image), and uses the three-dimensional coordinates of the edge pixels of the saturated area as the calculation intermediate value to obtain a one-to-one correspondence between the edge pixel position of the saturated area of the camera image and the pixel position of the projector projection plane, and finally uses the RANSAC algorithm to calculate the local homography matrix; using the local homography relationship, the best projection pattern of the camera plane can be mapped to the best projection pattern of the projector plane.
[0074] Among them, the principle of solving the one-to-one correspondence between the above-mentioned regions and plane pixels by using only a single-direction coded fringe image is: if the projector is regarded as an inverse camera model, for the pinhole imaging model, a point in space (X w , Y w , Z w ) at a point (x) in the camera c ,y c ) imaging satisfies the following formula (recorded as the first formula): s c (x c ,y c , 1) T =Kc [R c |t c ](X w , T w , Z w ) T
[0075] Among them, K c is the camera internal parameter, [R c |t c ] is the camera external parameter, s c is the zoom factor corresponding to the camera. w , Y w , Z w ) at a point (x) in the projector p ,y p ) imaging satisfies the following formula (recorded as the second formula): s p (x p ,y p , 1) T =K p [R p |t p ](X w , Y w , Z w ) T
[0076] Among them, K p is the internal parameter of the projector, [R p |t p ] is the external parameter of the projector, s p is the scaling factor corresponding to the projector. If the vertical stripes (x p The direction encoding value is known, that is, x p Known), then for a point (x c ,y c ) can be obtained by decoding the code value and x p Correspondingly, the goal is to simultaneously find the y corresponding to this pixel point in the projector plane p , to form a point-to-point relationship; assuming M c 、M p for:
[0077] (X w , Y w , Z w ) as the intermediate value of the calculation process, combined with M c 、M p The expression of can be solved by the first and second equations above:
[0078] At this time, for the second formula, s p (x p ,y p , 1) T =K p [R p |t p ](X w , Y w , Z w ) T Currently, only s p ,y p Unknown, but we can directly construct two equations to solve these two unknowns respectively:
[0079] That is, using (X w , Y w , Z w ) is an intermediate value, and the reverse solution successfully obtains (x c ,y c ) and (x p ,y p )’s correspondence.
[0080] Assuming that the homography relationship is expressed as H, the principle of using the robust estimation algorithm to obtain a robust homography relationship is as follows: after the coordinate correspondence relationship of the saturated region is established, the homography is calculated for each region. The homography relationship is expressed as: (x p ,y p , 1) T =H·(x c ,y c , 1) T
[0081] The correct calculation of the local homography matrix H requires at least four sets of corresponding points (any three points are not collinear), so the sampling method without replacement is used to randomly select two points from the points with the same chain code value (i.e., the same straight line) as candidate points to participate in the calculation of the matrix H, avoiding the problem of singular solution caused by the selected points being collinear. At the same time, robust estimation algorithms such as RANSAC (Random Sample Consensus) or MSAC (M-estimator Sample Consensus) are used for all candidate points in each region to find the correct data (Inliers) for the calculation of the homography matrix, so as to obtain an accurate and robust local mapping relationship from the pixel saturated area of each uniform brightness image to the projector plane.
[0082] The deep learning-based structured light three-dimensional measurement method provided in the embodiment of the present application proposes an efficient and high-dynamic range structured light three-dimensional measurement method and system based on deep learning. The method has the advantages of high measurement efficiency and good robustness, and can accurately obtain a complete three-dimensional point cloud of the surface of the object being measured. In addition, the hardware system used in the embodiment of the present application is simple, including only a projector and a camera, without any additional hardware equipment, thereby reducing hardware costs.
[0083] In order to demonstrate the superiority of the high-efficiency and high-dynamic range structured light 3D measurement method based on deep learning proposed in the present invention, white smooth plastic propellers, metal propellers and aircraft engine blades were selected as targets. The reconstruction results of the adaptive projection brightness algorithm proposed in the present invention have complete and excellent reconstruction performance on all types of surfaces of the workpiece, and the point cloud reconstruction density is high.
[0084] Figure 3 is a schematic diagram of the structure of the structured light three-dimensional measurement device based on deep learning provided by the present invention, such as Figure 3 As shown, the present invention provides a structured light three-dimensional measurement device based on deep learning, including a first acquisition module 301, a second acquisition module 302 and a third acquisition module 303.
[0085] The first acquisition module 301 is used to acquire a uniform brightness image and a coded fringe image modulated by the surface of an object.
[0086] The second acquisition module 302 is used to use a pre-trained target deep neural network and obtain the optimal projection pattern of the projector plane based on the uniform brightness image and the encoded fringe image; the target deep neural network is used to classify pixels of the image and determine the optimal projection grayscale value corresponding to each type of pixel.
[0087] The third acquisition module 303 is used to acquire a target coded fringe image modulated by the object surface by projecting the optimal projection pattern of the projector plane, and acquire a three-dimensional point cloud of the object surface based on the target coded fringe image.
[0088] In some embodiments, the second acquisition module includes: A camera plane optimal projection pattern acquisition unit, used for inputting the uniform brightness image into a pre-trained target deep neural network to obtain a camera plane optimal projection pattern; The projector plane best projection pattern acquisition unit is used to obtain the projector plane best projection pattern based on the camera plane best projection pattern and using the homography relationship between the camera image pixel saturation area and the corresponding area of the projector projection plane; the homography relationship is obtained based on the coded fringe image.
[0089] In some embodiments, the target deep neural network includes a pixel classification network and an optimal brightness prediction network; wherein the input of the pixel classification network is a uniform brightness image captured under different projection brightness sequences, and the output is the category of each pixel of the uniform brightness image; the input of the optimal brightness prediction network is a uniform brightness image captured under different projection brightness sequences, and the output is an optimal brightness vector, and the optimal brightness prediction network includes a feature extraction backbone network and a fully connected network.
[0090] In some embodiments, the target deep neural network is trained by the following steps: Acquire a training data set; the training data set includes a simulated data set and a real data set; Pre-training a backbone network in the target deep neural network using a public data set; the backbone network includes a backbone network of the pixel classification network and a backbone network of the optimal brightness prediction network; Training the target deep neural network using the simulated data set; The target deep neural network is fine-tuned using the real dataset.
[0091] In some embodiments, obtaining a simulation data set includes: Establish a relationship model between camera image intensity, projected image intensity, and object surface reflectivity; The reflectivity of the surface of a randomly shaped object is simulated and a series of projection images with uniform brightness are projected onto the surface of the randomly shaped object. The camera image intensity corresponding to the projection image and the optimal projection grayscale image of the camera plane are obtained using the relationship model to obtain a simulated data set.
[0092] In some embodiments, the real data set is a uniform brightness image and a corresponding camera plane optimal projection grayscale image obtained by actual measurement using a variety of reflective objects.
[0093] In some embodiments, the camera plane optimal projection pattern acquisition unit includes: A pixel classification subunit, used to classify pixels in the uniform brightness image using a pixel classification network in the target deep neural network to obtain a pixel classification result; The optimal brightness prediction subunit is used to obtain the optimal projection pattern of the camera plane based on the pixel classification result and using the optimal brightness prediction network in the target deep neural network.
[0094] Specifically, the above-mentioned deep learning-based structured light three-dimensional measurement device provided by the present invention can implement all the method steps implemented by the above-mentioned deep learning-based structured light three-dimensional measurement method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0095] It should be noted that the division of units / modules in the above-mentioned embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.
[0096] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor (processor) 401, a communication interface (Communications Interface) 402, a memory (memory) 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404. The processor 401 may call the logic instructions in the memory 403 to execute the structured light three-dimensional measurement method based on deep learning, and the method includes: Obtain uniform brightness image and coded fringe image modulated by the surface of the object; Using a pre-trained target deep neural network and based on the uniform brightness image and the coded fringe image, an optimal projection pattern of the projector plane is obtained; the target deep neural network is used to classify pixels of the image and determine the optimal projection grayscale value corresponding to each type of pixel; A target coded stripe image modulated by the surface of the object is acquired by projecting the optimal projection pattern of the projector plane, and a three-dimensional point cloud of the surface of the object is acquired based on the target coded stripe image.
[0097] Specifically, the processor 401 can be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD), and the processor can also adopt a multi-core architecture.
[0098] The logic instructions in the memory 403 can be implemented in the form of software functional units and can be stored in a processor-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.
[0099] In some embodiments, a computer program product is further provided, the computer program product comprising a computer program, the computer program may be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the structured light three-dimensional measurement method based on deep learning provided by the above-mentioned method embodiments, the method comprising: Obtain uniform brightness image and coded fringe image modulated by the surface of the object; Using a pre-trained target deep neural network and based on the uniform brightness image and the coded fringe image, an optimal projection pattern of the projector plane is obtained; the target deep neural network is used to classify pixels of the image and determine the optimal projection grayscale value corresponding to each type of pixel; A target coded stripe image modulated by the surface of the object is acquired by projecting the optimal projection pattern of the projector plane, and a three-dimensional point cloud of the surface of the object is acquired based on the target coded stripe image.
[0100] Specifically, the above-mentioned computer program product provided in the embodiment of the present application can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0101] In some embodiments, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program is used to enable a computer to execute the structured light three-dimensional measurement method based on deep learning provided by the above-mentioned method embodiments, the method comprising: Obtain uniform brightness image and coded fringe image modulated by the surface of the object; Using a pre-trained target deep neural network and based on the uniform brightness image and the coded fringe image, an optimal projection pattern of the projector plane is obtained; the target deep neural network is used to classify pixels of the image and determine the optimal projection grayscale value corresponding to each type of pixel; A target coded stripe image modulated by the surface of the object is acquired by projecting the optimal projection pattern of the projector plane, and a three-dimensional point cloud of the surface of the object is acquired based on the target coded stripe image.
[0102] Specifically, the above-mentioned computer-readable storage medium provided by the present invention can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effects. The parts and beneficial effects that are the same as the method embodiments in this embodiment will not be described in detail here.
[0103] It should be noted that the computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.
[0104] It should also be noted that the terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are usually of the same type, and the number of objects is not limited. For example, the first object can be one or more.
[0105] In the present invention, the term "plurality" refers to two or more than two, and other quantifiers are similar to them.
[0106] In the present invention, "determine B based on A" means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "determine B based on A and C", "determine B based on A, C and E", "determine C based on A, and further determine B based on C", etc. In addition, it can also include taking A as a condition for determining B, for example, "when A meets the first condition, use the first method to determine B"; for another example, "when A meets the second condition, determine B", etc.; for another example, "when A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "when A meets the first condition, use the first method to determine C, and further determine B based on C", etc.
[0107] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.
[0108] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer executable instructions. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0109] These processor executable instructions may also be stored in a processor readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the processor readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0110] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0111] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A structured light three-dimensional measurement method based on deep learning, characterized in that: include: Obtain uniform brightness image and coded fringe image modulated by the surface of the object; Using a pre-trained target deep neural network and based on the uniform brightness image and the coded fringe image, an optimal projection pattern of the projector plane is obtained; the target deep neural network is used to classify pixels of the image and determine the optimal projection grayscale value corresponding to each type of pixel; A target coded stripe image modulated by the surface of the object is acquired by projecting the optimal projection pattern of the projector plane, and a three-dimensional point cloud of the surface of the object is acquired based on the target coded stripe image.
2. The structured light three-dimensional measurement method based on deep learning according to claim 1, characterized in that: The method of using a pre-trained target deep neural network and obtaining an optimal projection pattern of a projector plane based on the uniform brightness image and the coded fringe image includes: Inputting the uniform brightness image into a pre-trained target deep neural network to obtain an optimal projection pattern on the camera plane; Based on the optimal projection pattern of the camera plane, the optimal projection pattern of the projector plane is obtained by utilizing the homography between the pixel saturation area of the camera image and the corresponding area of the projector projection plane; the homography is obtained by solving the coded fringe image.
3. The structured light three-dimensional measurement method based on deep learning according to claim 1 or 2, characterized in that: The target deep neural network includes a pixel classification network and an optimal brightness prediction network; Among them, the input of the pixel classification network is a uniform brightness image collected under different projection brightness sequences, and the output is the category of each pixel of the uniform brightness image; the input of the optimal brightness prediction network is a uniform brightness image collected under different projection brightness sequences, and the output is an optimal brightness vector, and the optimal brightness prediction network includes a feature extraction backbone network and a fully connected network.
4. The structured light three-dimensional measurement method based on deep learning according to claim 3 is characterized in that: The target deep neural network is trained by the following steps: Acquire a training data set; the training data set includes a simulated data set and a real data set; Pre-training a backbone network in the target deep neural network using a public data set; the backbone network includes a backbone network of the pixel classification network and a backbone network of the optimal brightness prediction network; Training the target deep neural network using the simulated data set; The target deep neural network is fine-tuned using the real dataset.
5. The structured light three-dimensional measurement method based on deep learning according to claim 4, characterized in that: The obtaining of the simulation data set comprises: Establish a relationship model between camera image intensity, projected image intensity and object surface reflectivity; The reflectivity of the surface of a randomly shaped object is simulated and a series of projection images with uniform brightness are projected onto the surface of the randomly shaped object. The camera image intensity corresponding to the projection image and the optimal projection grayscale image of the camera plane are obtained using the relationship model to obtain a simulated data set.
6. The structured light three-dimensional measurement method based on deep learning according to claim 4, characterized in that: The real data set is a uniform brightness image obtained by actual measurement using a variety of reflective objects and a corresponding camera plane optimal projection grayscale image.
7. The structured light three-dimensional measurement method based on deep learning according to claim 3, characterized in that: The step of inputting the uniform brightness image into a pre-trained target deep neural network to obtain an optimal projection pattern of a camera plane includes: using a pixel classification network in the target deep neural network to classify pixels in the uniform brightness image to obtain a pixel classification result; Based on the pixel classification result and using the optimal brightness prediction network in the target deep neural network, an optimal projection pattern of the camera plane is obtained.
8. A structured light three-dimensional measurement device based on deep learning, characterized in that: include: A first acquisition module is used to acquire a uniform brightness image and a coded fringe image modulated by the surface of the object; A second acquisition module is used to obtain the optimal projection pattern of the projector plane based on the uniform brightness image and the coded fringe image by using a pre-trained target deep neural network; the target deep neural network is used to classify the pixels of the image and determine the optimal projection grayscale value corresponding to each type of pixel; The third acquisition module is used to acquire a target coded stripe image modulated by the surface of the object by projecting the optimal projection pattern of the projector plane, and acquire a three-dimensional point cloud of the surface of the object based on the target coded stripe image.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the structured light three-dimensional measurement method based on deep learning as described in any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the structured light three-dimensional measurement method based on deep learning as described in any one of claims 1 to 7 is implemented.
Citation Information
Cited By
High-precision three-dimensional measurement method based on multi-frequency fusion coding and deep learning
CN119935009A
