A three-dimensional imaging method and device based on special coding

Abstract

The present invention discloses a three-dimensional imaging method and device based on special coding, which mainly solves the problems existing in the prior art, such as large limitations, limiting the progress of three-dimensional imaging, high cost, difficulty in miniaturization, and the need for high-precision calibration and equipment synchronization. The three-dimensional imaging method based on special coding is to obtain a truncated phase image of a target object, input the truncated phase image and a pair of special coding level images into a trained deep convolutional neural network model for processing to obtain a corresponding absolute fringe level image; obtain the three-dimensional information of the target object according to the truncated phase image and the absolute fringe level image, and image the three-dimensional information; the present invention combines single-frequency phase-shifted fringe images with deep learning technology to achieve absolute expansion of single-frequency phase, thereby achieving rapid three-dimensional reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional imaging, and in particular to a three-dimensional imaging method and device based on special coding. Background Art

[0002] Structured light 3D imaging is a commonly used 3D measurement and reconstruction technology. Stripe structured light is widely used due to its high precision and other characteristics. Figure 1 As shown, the projector projects one, two or more groups of stripe patterns onto the surface of the target object, and the camera collects images of the target object surface at different orientations; the collected images mainly include deformed stripes modulated by the object surface height information, and the object surface height information can be obtained based on the deformed stripe information combined with the known system parameters of the imaging system.

[0003] After acquiring the deformed fringe image, the phase needs to be unfolded in the process of calculating the truncated phase map. Commonly used methods of phase unfolding and their problems are: (1) The spatial phase unfolding method only requires a truncated phase map of one frequency, which has a great limitation; (2) The temporal phase unfolding method requires the object to remain still before the projection of all fringe patterns is completed; (3) The stereoscopic vision geometric constraint phase unfolding method limits the progress of three-dimensional imaging; (4) The multi-eye phase unfolding method has the disadvantages of high cost, difficulty in miniaturization, and the need for high-precision calibration and equipment synchronization; (5) The phase unfolding method based on deep learning has the disadvantages of slow single-shot three-dimensional imaging speed and high system cost. Summary of the invention

[0004] The purpose of the present invention is to provide a three-dimensional imaging method and device based on special encoding, which can realize the absolute unfolding of single-frequency phase by combining single-frequency phase-shifted fringe images and deep learning technology, thereby realizing rapid three-dimensional reconstruction; this method is mainly used for the absolute phase unfolding task of truncated phase images of monocular or binocular stripe structured light three-dimensional imaging systems.

[0005] On the one hand, a three-dimensional imaging method based on special coding is characterized in that

[0006] Obtain a truncated phase image of the target object, and input the truncated phase image and a pair of specially encoded level images into a trained deep convolutional neural network model for processing to obtain a corresponding absolute fringe level image;

[0007] The three-dimensional information of the target object is obtained according to the truncated phase image and the absolute fringe order image, and the three-dimensional information is imaged.

[0008] In a possible implementation manner of the embodiment of the first aspect, the special coding level image is an image obtained after a special binary image is projected onto the surface of the object to be measured, and a camera captures the corresponding image and performs image binarization processing.

[0009] In a possible implementation manner combined with the first aspect embodiment, the special binary image projected onto the surface of the object to be measured is obtained after selecting the first type of special sub-level area and the second type of special sub-level area according to the level sequence number in the digital level map before projection.

[0010] In a possible implementation manner combined with the first aspect embodiment, the first type of special level area is a level area with a level number near the median of the projection stripe number, and the second type of special level area is a level area that is spaced a certain number of levels on the left and right sides of the first type of special level area.

[0011] In a possible implementation manner of the embodiment of the first aspect, the first type of special level area and the second type of special level area are divided into black and white binary images with other areas in the digital level image to obtain a special level binary image.

[0012] In a possible implementation manner of the embodiment of the first aspect, before inputting the trained deep convolutional neural network model for processing to obtain the corresponding absolute fringe order map, the method further includes: acquiring a plurality of training sample data, and training an initial deep convolutional neural network model through the training sample data to obtain a trained deep convolutional neural network model;

[0013] The training sample data includes a single-frequency truncated phase image, a special coded order image, and an absolute fringe order image label corresponding to the single-frequency truncated phase image.

[0014] In a possible implementation manner in combination with the first aspect embodiment, the truncated phase image in the training sample data is after removing the background and invalid areas. The removal method includes setting a threshold according to the modulation map in the camera, and the pixels with modulation values ​​lower than the threshold are the background or invalid areas and need to be removed.

[0015] In a possible implementation of the embodiment of the first aspect, a method for generating an absolute fringe order image label corresponding to a single-frequency truncated phase image is to project multiple groups of phase-shifted fringe patterns with different spatial frequencies on a structured light three-dimensional imaging device, and use a time phase unwrapping method to process the collected fringe image to obtain an absolute phase image.

[0016] In a possible implementation of the embodiment of the first aspect, the deep convolutional neural network model is a neural network model in the field of image semantic segmentation.

[0017] On the other hand, a three-dimensional imaging device based on special coding includes a memory for storing executable instructions; a processor for executing the executable instructions stored in the memory to implement a three-dimensional imaging method based on special coding.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] The present invention adds a projection pattern, but provides key constraint information, which ultimately supports the successful implementation of the task. The special coding level method makes several fixed levels become absolute reference features. Compared with traditional methods, it is extremely easy for neural networks to learn this obvious and special feature. Once this constraint is determined in a single-frequency truncated phase image, that is, a special level area is found, other level areas can be used as a reference, combined with their own spatial position characteristics and the number of level areas within the range of distance in the image space, to uniquely determine an absolute stripe level number. This makes absolute phase unfolding based on single-frequency stripes possible and greatly improves its accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation or the prior art description are briefly introduced below. Obviously, the drawings described below are some implementations of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0021] Figure 1 This is the principle of stripe structured light three-dimensional imaging.

[0022] Figure 2 This is a process diagram of the three-step phase shift technique.

[0023] Figure 3 This is a truncated phase diagram of the 64th frequency.

[0024] Figure 4 for Figure 5 The data curve of a row in the truncated phase diagram.

[0025] Figure 5 It is a 64-channel special encoding level image.

[0026] Figure 6 The workflow of a single-frequency absolute phase unwrapping network based on a special encoding order.

[0027] Figure 7(a) is the input fringe pattern.

[0028] Figure 7(b) is the special encoding diagram of the input.

[0029] Figure 7(c) is the truncated phase diagram of the input.

[0030] FIG. 7( d ) is a graph showing the error between the order predicted using the present invention and the order obtained using time phase unwrapping.

[0031] Figure 8 It is the error diagram between the order obtained by inputting only the truncated phase image of Figure 7(c) into the network expansion and the order obtained by using time phase expansion.

[0032] Fig. 9 This is a portrait model reconstructed by the present invention that has not participated in network training. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention clearer, the following will be combined with Figures 1 to 9 The present invention is further described in detail. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0034] The present invention proposes a special coding level method, which combines single-frequency phase-shifted fringe images and deep learning technology to achieve absolute unwrapping of single-frequency phase, thereby achieving rapid 3D reconstruction. This method is mainly used for the absolute phase unwrapping task of truncated phase images of monocular or binocular fringe structured light 3D imaging systems.

[0035] After the deformed fringe image is collected, the truncated phase map is calculated. There are two common methods: the first is to perform Fourier transform, spectrum filtering and inverse Fourier transform on the image; the second is to use phase shift technology (phase shift), the projector projects a series of phase shifted fringe patterns with the same frequency, the camera collects a series of corresponding deformed fringe patterns, and then calculates the phase through formula (1-1). In formula (1-1), I n (x, y) is the deformed fringe pattern, and N is the number of deformed fringe patterns, also known as the number of phase shift steps. The phases obtained by these two methods are usually called truncated phases. Figure 2 The figure shows the main process of the second method. The rightmost image is the truncated phase, where the number of phase shift steps is 3.

[0036]

[0037] The value of each pixel in the truncated phase image varies between [-pi, pi), so phase unwrapping is required to obtain true continuous phase information. The phase unwrapping task can be regarded as finding an integer value for each pixel in the truncated phase image, which is often called the fringe order. The truncated phase plus the order multiple 2pi gives the continuous phase, also called the unwrapped phase. It is worth noting that if this fringe order corresponds to the fringe order of the projected fringe pattern, then this order is the absolute fringe order, and phase unwrapping is also absolute phase unwrapping. The continuous phase obtained is also absolute phase, which has real physical meaning. The absolute phase combined with the system parameters can easily obtain the final object surface information. The relationship between the truncated phase, fringe order and continuous phase is shown below

[0038]

[0039] like Figure 3 In the single-frequency truncated phase diagram (single-frequency phase shift fringe diagram obtained by formula 1-1), the phase value of each pixel point is in the range of [-pi, pi), and each point has a unique fringe order number. The range of these order numbers depends on the fringe frequency (such as 16-frequency, 64-frequency, etc., Figure 3 is 64 frequencies), and the maximum value is the number of projected fringes (also called the fringing period). Figure 4 Displayed Figure 3 The truncated phase value of a row in . Figure 4 and Figure 3 It can be seen that the changes in different order number areas in the truncated phase image are roughly the same, and the distribution of the order increases from left to right. In fact, if the surface shape of the object changes slightly, the order of the same pixel before and after is likely to differ by 1 or more. At this time, the order of each pixel cannot be uniquely determined from the single-frequency truncated phase. Therefore, the spatial phase unwrapping method based on the single-frequency truncated phase image cannot obtain the absolute fringe order (only the relative order can be obtained). Although the temporal phase unwrapping method based on the multi-frequency truncated phase image can achieve absolute phase unwrapping, the need for more projected images greatly limits the application scenarios.

[0040] Based on the above analysis, we propose a special coding level method. On the basis of the single-frequency truncated phase image, we only need to add a special coding level image to assist the model in determining the unique fringe level of the pixel. Compared with the spatial phase unwrapping method, although there is an additional projection pattern required, it solves the problem of its inability to perform absolute phase unwrapping; compared with the temporal phase unwrapping method, only one special coding level pattern is added to the single-frequency fringe pattern, which reduces the time required for a three-dimensional imaging. Figure 5That is the special coded level image (64 frequencies) we designed. In the digital pattern before projection, the three middle levels (31, 32, 33) are selected as the first type of special level area, and then the second type of special level area is selected every 3 levels to the left and right. The selected special level areas and other areas are divided into black and white binary values, and finally the special coded level image is obtained. If the highest frequency is different (such as 32 frequencies), corresponding adjustments can be made. Because the three middle level areas are significantly different from other level areas, this feature can help us determine the absolute three levels. With this as a reference, combined with the special level changes with an interval of 3, it is greatly convenient to determine the overall absolute level. The characteristics of special levels can be learned using deep learning technology. Therefore, combined with deep learning, we have achieved absolute phase unfolding on single-frequency phase-shifted fringe images and special coded levels.

[0041] In the deep learning phase unwrapping method, the network model generally inputs a truncated phase image and outputs a fringe order image or a continuous phase image, but these methods do not explain in detail their working principles and generalization. The main reason is that the interpretability of neural networks is still a thorny issue, and many studies are only at the stage of using this technology. We found that the existence of special coded order images is extremely important for the single-frequency phase absolute unwrapping task based on deep learning, which can explain the feasibility of convolutional neural networks for absolute phase unwrapping to a certain extent, and verified our findings through experiments.

[0042] In the truncated phase image, the fringe-level sub-region has simple, consistent and obvious data variation characteristics, such as Figure 4As shown in the figure, it is not difficult for a neural network to learn such features and separate them, but it is difficult to give each area a unique serial number (i.e., the absolute fringe order corresponding to the projected fringe pattern). Because in the same structured light imaging environment, the absolute fringe orders of the order areas with roughly the same spatial positions in two truncated phase images are likely to differ by 1 or 2 or even more, which is due to the fact that the surface shapes of different objects are variable. At this time, the consistent data change characteristics of the image spatial position and the order area alone are not enough to support the network model to achieve sufficient accuracy of absolute phase unwrapping. More constraints are required, but for the single-frequency phase unwrapping task, constraint information is also lacking. Therefore, we propose a special coding order method. Although a projection pattern is added, the key constraint information is given, which ultimately supports the successful implementation of the task. The special coding order method makes several fixed orders become absolute reference features. Compared with traditional methods, it is extremely easy for neural networks to learn such obvious and special features. Once this constraint is determined in a single-frequency truncated phase image, that is, a special order region is found, other order regions can be used as a reference, combined with their own spatial position characteristics and the number of order regions within the range of distance in the image space, to uniquely determine an absolute fringe order number. This makes absolute phase unfolding based on single-frequency fringes possible and greatly improves its accuracy.

[0043] Special coding patterns, such as Figure 5 The picture shown is a black and white binary image. Although the image captured by the camera after being projected by the projector is grayscale, it still appears black and white visually. At this time, this image is compared with the average grayscale image obtained from the phase-shifted image. The average of several phase-shifted images is the average grayscale image, which can be understood as projecting a completely white image onto an object and then taking a picture with a camera. The special coding level image is projecting a black and white image onto an object and then taking a picture with a camera. Pixels with obvious grayscale value changes must belong to non-special coding. In this way, the special coding level image can be divided into black and white binary.

[0044] In general, the special coded level image obtained by the camera is compared with the average grayscale image. The pixels with obvious changes in grayscale values ​​and the pixels with slight changes belong to the non-special level area and the special level area, respectively. This can be distinguished, which gives the network model special information and important features, and the process is simple and easy to calculate.

[0045] Supervised training of deep convolutional neural networks requires a large number of image pairs. Figure 6 Single frequency truncated phase diagram and special encoding order on the left Figure 1 The absolute fringe level map on the right is the label, and they constitute an image pair. Figure 6 and Figure 3The truncated phase image is obtained by removing the background and invalid areas. Because the truncated phase information of the background and invalid areas is not useful for training and can be easily removed, the method is to set a threshold according to the corresponding modulation map. Pixels with modulation values ​​below the threshold can be considered as background and invalid areas. The modulation index can be calculated from the phase shift fringe image collected by the camera, just like the truncated phase. The formula 1-3 is as follows

[0046]

[0047]

[0048] The method for generating the absolute fringe order map used as a label during training is as follows: on a custom-developed structured light three-dimensional imaging device, multiple sets of phase-shifted fringe patterns with different spatial frequencies are projected, and the collected fringe images are processed using a temporal phase unwrapping method to obtain an absolute phase map.

[0049] The cutoff phase and modulation are obtained from the phase shift fringe pattern at the highest frequency according to formulas 1-1 and 1-3. With the absolute phase, cutoff phase and modulation, combined with formula 1-2 and the modulation threshold, we can get Figure 8 The target absolute fringe level map shown on the right, therefore no manual annotation is required.

[0050] The deep convolutional neural network for training can adopt the existing neural network model in the field of image semantic segmentation, and the loss function used for training can adopt the L1 loss function or the smoothL1 loss function.

[0051] After the training, the phase unwrapping network can recover the absolute phase by simply inputting a truncated phase map with the highest fringe frequency and a special encoding order map. It also has the following characteristics: (1) The number of fringe maps required is only 2 / 3 or 4 / 9 of the traditional time phase unwrapping method, which is more conducive to the realization of dynamic three-dimensional imaging, such as Figure 3 In the example shown, we only need 3 phase shift patterns of a single frequency and 1 special coded level image. In this case, temporal phase unwrapping requires at least 6 (generally 9) phase shift patterns; (2) It can output absolute phase. Traditional spatial phase unwrapping methods only input a truncated phase map of a fringe frequency, but can only recover relative phase. (3) It can be used for binocular structured light (2 cameras, 1 projector); it can also only require one projection unit and one camera. In this way, the method of the present invention is lower in cost, smaller in system size, and does not have the synchronization and calibration problems of multiple cameras (projectors).

[0052] Figure 7(a) to Figure 7(d)It is the result of phase unwrapping using a trained network based on special coding levels. Figure 7(a) is a fringe image taken at three different angles, Figure 7(b) and Figure 7(c) are three special coding level images and truncated phase images corresponding to Figure 7(a), and Figure 7(d) is an error diagram of the absolute level image predicted by the method of the present invention and the absolute level image label obtained by time phase unwrapping (black indicates absolute level error >= 1, and white indicates correct prediction of absolute level). It can be found that the locations where the network predicts errors are almost all at the edges of the object contours and the places where the levels jump, and the contour edges have little effect on the three-dimensional reconstruction results (the error is large and is generally removed during deep filtering), and the error at the level jump is usually unavoidable in the true value, so the final result of the network absolute phase unwrapping meets practical applications.

[0053] Figure 8 The error diagram between the absolute level predicted by the network and the absolute level obtained by using time phase unwrapping is shown when the network input data does not include the special coding level image but only includes the truncated phase image. It can be seen that once the special coding level image is lost, the key constraint information is lost, and the reference feature is lost, the prediction effect of the network is affected and the absolute phase unwrapping cannot be performed accurately, which shows the importance of the special coding level method we proposed for the single-frequency phase absolute unwrapping task based on deep learning.

[0054] Fig. 9 We demonstrate how to use a trained network based on a special encoding level to obtain the absolute phase and finally reconstruct a portrait model from three perspectives. This portrait model does not participate in the network training process, which further verifies the feasibility of our method.

[0055] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0056] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0057] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention. It should be noted that similar numbers and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0059] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A three-dimensional imaging method based on special coding, characterized in that: include: Obtain a truncated phase image of the target object, and input the truncated phase image and a pair of specially encoded level images into a trained deep convolutional neural network model for processing to obtain a corresponding absolute fringe level image; The three-dimensional information of the target object is obtained according to the truncated phase image and the absolute fringe order image, and the three-dimensional information is imaged; The special coding level image is an image obtained by projecting a special binary image onto the surface of the object to be measured, and then taking a corresponding image with a camera and performing image binarization processing; The special binary image projected onto the surface of the object to be measured is obtained after selecting the first special level area and the second special level area according to the level sequence number in the digital level map before projection; The first type of special level area is the level area whose level number is near the median of the projection fringe number, and the second type of special level area is the level area after a certain level interval on the left and right sides of the first type of special level area. The first and second special level areas and other areas in the digital level map are divided into black and white binary images to obtain a special level binary image.

2. A three-dimensional imaging method based on special coding according to claim 1, characterized in that: Before inputting the trained deep convolutional neural network model for processing to obtain the corresponding absolute fringe order map, the method further includes: acquiring a plurality of training sample data, and training the initial deep convolutional neural network model with the training sample data to obtain the trained deep convolutional neural network model; The training sample data includes a single-frequency truncated phase image, a special coded order image, and an absolute fringe order image label corresponding to the single-frequency truncated phase image.

3. A three-dimensional imaging method based on special coding according to claim 2, characterized in that: The truncated phase image in the training sample data is obtained by removing the background and invalid areas. The removal method includes setting a threshold according to the modulation map in the camera. Pixels with modulation values ​​lower than the threshold are background or invalid areas and need to be removed.

4. A three-dimensional imaging method based on special coding according to claim 2, characterized in that: The method for generating the absolute fringe order map label corresponding to the single-frequency truncated phase image is to project multiple sets of phase-shifted fringe patterns with different spatial frequencies on a structured light three-dimensional imaging device, and use the time phase unwrapping method to process the collected fringe image to obtain the absolute phase map.

5. The three-dimensional imaging method based on special coding according to claim 1, characterized in that: The deep convolutional neural network model is a neural network model in the field of image semantic segmentation.

6. A three-dimensional imaging device based on special coding, characterized in that: include Memory: used to store executable instructions; Processor: used to execute the executable instructions stored in the memory to implement a three-dimensional imaging method based on special coding as described in any one of claims 1-5.

Images