Three-dimensional imaging method, device and equipment based on deep learning, medium and product

Abstract

The invention provides a three-dimensional imaging method and device based on deep learning, equipment, a medium and a product, and relates to the technical field of image processing. The method comprises the steps that a fringe pattern set is acquired, and fringe patterns in the fringe pattern set are sequentially projected to an object to be measured through a projection device; in the projection process, shooting the to-be-measured object through a shooting device to obtain a shot image set; obtaining a high-frequency truncation phase diagram according to the plurality of high-frequency phase shift shot images in the shot image set; combining the high-frequency truncation phase diagram and at least two special-level coding shot images in the shot image set into a multi-channel composite image; based on the trained network model, extracting boundary information corresponding to the high-frequency truncated phase diagram, extracting feature information corresponding to the multi-channel composite diagram, and obtaining a corresponding level diagram according to the boundary information and the feature information; and constructing a three-dimensional model of the to-be-measured object according to the level diagram. According to the method provided by the invention, the technical effect of improving the three-dimensional reconstruction accuracy is realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-dimensional imaging method, device, equipment, medium and product based on deep learning. Background Art

[0002] 3D object reconstruction technology can not only be widely used in 3D face recognition, but also in the digital modeling of other 3D objects, playing an important role in the fields of virtual banking environment and product display. At present, Fringe Projection Profilometry (FPP), as a mainstream structured light 3D imaging method, strengthens the security of face recognition and improves the user experience in remote banking.

[0003] Phase unwrapping, as a key step in fringe projection profilometry, directly determines the rate and accuracy of three-dimensional imaging. Generally speaking, phase unwrapping methods include spatial phase unwrapping, temporal phase unwrapping, and deep learning phase unwrapping. Among them, spatial phase unwrapping is suitable for continuous surfaces, but has difficulties with discontinuous surfaces or isolated objects; temporal phase unwrapping can measure discontinuous surfaces with high precision, but requires multiple frames of patterns and has low efficiency; deep learning phase unwrapping uses neural networks to learn the mapping relationship between truncated phase and absolute phase, overcoming the shortcomings of phase unwrapping and temporal phase unwrapping. However, the existing deep learning phase unwrapping technology still has the technical problem of low three-dimensional reconstruction accuracy. Summary of the invention

[0004] The present application provides a three-dimensional imaging method, device, equipment, medium and product based on deep learning, which is used to solve the technical problem that the existing deep learning phase unwrapping technology still has low three-dimensional reconstruction accuracy.

[0005] In a first aspect, the present application provides a three-dimensional imaging method based on deep learning, comprising:

[0006] Acquire a fringe pattern set, and project the fringe patterns in the fringe pattern set to the object to be measured in sequence through a projection device; wherein the fringe pattern set includes a plurality of high-frequency phase-shifted fringe patterns and at least two special-order coded fringe patterns;

[0007] During the projection process, the object to be measured is photographed by a photographing device to obtain a photographing image set, wherein the photographing image set includes high-frequency phase-shift photographing images corresponding to a plurality of high-frequency phase-shift fringe patterns and special-order coded photographing images corresponding to at least two special-order coded fringe patterns;

[0008] A high-frequency truncated phase image is obtained according to a plurality of high-frequency phase-shifted images in the image set;

[0009] Combining the high frequency truncated phase image and at least two specially coded images in the image set into a multi-channel composite image;

[0010] Based on the trained network model, the boundary information corresponding to the high-frequency truncated phase image is extracted, the feature information corresponding to the multi-channel composite image is extracted, and the corresponding order image is obtained according to the boundary information and the feature information;

[0011] Construct a three-dimensional model of the object to be tested according to the hierarchy diagram.

[0012] In a possible implementation, the network model includes: a phase enhancement module and a plurality of processing layers connected in sequence, the plurality of processing layers including: N encoding layers, a convolutional mixing layer, and N-1 decoding layers; the phase enhancement module is connected to the N-1th decoding layer;

[0013] Extract the boundary information corresponding to the high-frequency truncated phase image, extract the feature information corresponding to the multi-channel composite image, and obtain the corresponding order image according to the boundary information and feature information, including:

[0014] The high-frequency truncated phase image is input into the phase enhancement module to obtain the corresponding boundary information;

[0015] The multi-channel composite image is input to the first encoding layer. After being processed by N encoding layers, convolutional mixing layers and the first N-2 decoding layers in sequence, the feature information output by the N-2 decoding layer is obtained.

[0016] The feature information and boundary information output by the N-2th decoding layer are concatenated and input into the N-1th decoding layer to obtain the output information of the N-1th decoding layer. The output information is passed through the activation function to obtain the corresponding level map.

[0017] In a possible implementation, the network model further includes: N-1 attention modules; wherein the i-th encoding layer is also connected to the Ni-th decoding layer through the i-th attention module; wherein i is a positive integer less than N;

[0018] The feature information and boundary information output by the N-2th decoding layer are concatenated and input into the N-1th decoding layer to obtain the output information of the N-1th decoding layer, including:

[0019] The output information of the first attention module, the feature information output by the N-2 decoding layer, and the boundary information are concatenated and input into the N-1 decoding layer to obtain the output information of the N-1 decoding layer.

[0020] In a possible implementation, the special level coding fringe pattern is determined by:

[0021] Acquire a first fringe pattern, merge a plurality of fringe orders corresponding to a first region in the first fringe pattern, and encode the merged result to obtain a first encoding result; the frequency corresponding to the first fringe pattern is consistent with the frequency corresponding to the high-frequency phase-shifted fringe pattern;

[0022] Performing a cyclic encoding process of a preset encoding on the area other than the first area in the first fringe pattern to obtain a second encoding result;

[0023] According to the first coding result and the second coding result, a corresponding special level coding fringe pattern is obtained;

[0024] Among them, the first area of ​​each special-order coded stripe pattern in at least two special-order coded stripe patterns contains stripe orders that are different.

[0025] In a possible implementation manner, the network model is obtained by training a training sample set, and the training samples in the training sample set are determined by the following method:

[0026] Acquire a sample fringe pattern set, and project the fringe patterns in the sample fringe pattern set to any three-dimensional object model in sequence through a projection device; wherein the sample fringe pattern set includes a plurality of sample high-frequency phase-shifted fringe patterns, at least two sample special-order coded fringe patterns, and a plurality of single-period phase-shifted fringe patterns;

[0027] During the projection process, any three-dimensional object model is photographed by a photographing device to obtain a set of sample photographed images, wherein the set of sample photographed images includes sample high-frequency phase-shifted photographed images corresponding to a plurality of sample high-frequency phase-shifted fringe images, sample special-order coded photographed images corresponding to at least two sample special-order coded fringe images, and single-period phase-shifted photographed images corresponding to a plurality of single-period phase-shifted fringe images;

[0028] Obtaining a single-cycle phase image according to a plurality of single-cycle phase-shifted images in the sample image set;

[0029] A sample high-frequency truncated phase image is obtained according to a plurality of sample high-frequency phase-shifted images in a sample image set, and the sample high-frequency truncated phase image and at least two sample special-order coded images in the sample image set are combined into a sample multi-channel composite image.

[0030] In a possible implementation, the network model is trained in the following manner:

[0031] Input any sample multi-channel composite image in the training sample set into the network model to be trained, and obtain the prediction level image corresponding to any sample multi-channel composite image;

[0032] Processing the predicted order map to obtain a predicted high-frequency truncated phase map and a predicted single-cycle phase map corresponding to the predicted order map;

[0033] The sample high-frequency truncated phase image and the predicted high-frequency truncated phase image are calculated to obtain a first loss value, and the single-cycle phase image and the predicted single-cycle phase image are calculated to obtain a second loss value, wherein the first loss value is used to indicate the loss of the self-supervisory signal, and the second loss value is used to indicate the loss of the weak supervision signal;

[0034] The first loss value and the second loss value are calculated to obtain a loss result, and the network model to be trained is trained according to the loss result until the loss result converges to obtain a trained network model.

[0035] In a second aspect, the present application provides a three-dimensional imaging device based on deep learning, comprising:

[0036] An acquisition module is used to acquire a fringe pattern set, and sequentially project the fringe patterns in the fringe pattern set onto the object to be measured through a projection device; wherein the fringe pattern set includes a plurality of high-frequency phase-shifted fringe patterns and at least two special-order coded fringe patterns;

[0037] A first processing module is used to obtain a set of photographed images by photographing the object to be measured through a photographing device during the projection process, wherein the set of photographed images includes high-frequency phase-shifted photographed images corresponding to a plurality of high-frequency phase-shifted fringe patterns and special-order coded photographed images corresponding to at least two special-order coded fringe patterns;

[0038] A second processing module is used to obtain a high-frequency truncated phase image according to a plurality of high-frequency phase-shifted images in the image set;

[0039] A combining module, used for combining the high-frequency truncated phase image and at least two special-order coded captured images in the captured image set into a multi-channel composite image;

[0040] The third processing module is used to extract the boundary information corresponding to the high-frequency truncated phase image based on the trained network model, extract the feature information corresponding to the multi-channel composite image, and obtain the corresponding level image according to the boundary information and the feature information;

[0041] The fourth processing module is used to construct a three-dimensional model of the object to be measured according to the level map.

[0042] In a possible implementation manner, the third processing module is further configured to:

[0043] The network model includes: a phase enhancement module and a plurality of processing layers connected in sequence, wherein the plurality of processing layers include: N encoding layers, a convolutional mixing layer, and N-1 decoding layers; the phase enhancement module is connected to the N-1th decoding layer;

[0044] The high-frequency truncated phase image is input into the phase enhancement module to obtain the corresponding boundary information;

[0045] The multi-channel composite image is input to the first encoding layer. After being processed by N encoding layers, convolutional mixing layers and the first N-2 decoding layers in sequence, the feature information output by the N-2 decoding layer is obtained.

[0046] The feature information and boundary information output by the N-2th decoding layer are concatenated and input into the N-1th decoding layer to obtain the output information of the N-1th decoding layer. The output information is passed through the activation function to obtain the corresponding level map.

[0047] In a possible implementation manner, the third processing module is further configured to:

[0048] The network model also includes: N-1 attention modules; wherein the i-th encoding layer is also connected to the Ni-th decoding layer through the i-th attention module; wherein i is a positive integer less than N;

[0049] The output information of the first attention module, the feature information output by the N-2 decoding layer, and the boundary information are concatenated and input into the N-1 decoding layer to obtain the output information of the N-1 decoding layer.

[0050] In a possible implementation, the acquisition module is further used to:

[0051] Acquire a first fringe pattern, merge a plurality of fringe orders corresponding to a first region in the first fringe pattern, and encode the merged result to obtain a first encoding result; the frequency corresponding to the first fringe pattern is consistent with the frequency corresponding to the high-frequency phase-shifted fringe pattern;

[0052] Performing a cyclic encoding process of a preset encoding on the area other than the first area in the first fringe pattern to obtain a second encoding result;

[0053] According to the first coding result and the second coding result, a corresponding special level coding fringe pattern is obtained;

[0054] Among them, the first area of ​​each special-order coded stripe pattern in at least two special-order coded stripe patterns contains stripe orders that are different.

[0055] In a possible implementation, the acquisition module is further used to:

[0056] Acquire a sample fringe pattern set, and project the fringe patterns in the sample fringe pattern set to any three-dimensional object model in sequence through a projection device; wherein the sample fringe pattern set includes a plurality of sample high-frequency phase-shifted fringe patterns, at least two sample special-order coded fringe patterns, and a plurality of single-period phase-shifted fringe patterns;

[0057] During the projection process, any three-dimensional object model is photographed by a photographing device to obtain a set of sample photographed images, wherein the set of sample photographed images includes sample high-frequency phase-shifted photographed images corresponding to a plurality of sample high-frequency phase-shifted fringe images, sample special-order coded photographed images corresponding to at least two sample special-order coded fringe images, and single-period phase-shifted photographed images corresponding to a plurality of single-period phase-shifted fringe images;

[0058] Obtaining a single-cycle phase image according to a plurality of single-cycle phase-shifted images in the sample image set;

[0059] A sample high-frequency truncated phase image is obtained according to a plurality of sample high-frequency phase-shifted images in a sample image set, and the sample high-frequency truncated phase image and at least two sample special-order coded images in the sample image set are combined into a sample multi-channel composite image.

[0060] In a possible implementation, the acquisition module is further used to:

[0061] Input any sample multi-channel composite image in the training sample set into the network model to be trained, and obtain the prediction level image corresponding to any sample multi-channel composite image;

[0062] Processing the predicted order map to obtain a predicted high-frequency truncated phase map and a predicted single-cycle phase map corresponding to the predicted order map;

[0063] The sample high-frequency truncated phase image and the predicted high-frequency truncated phase image are calculated to obtain a first loss value, and the single-cycle phase image and the predicted single-cycle phase image are calculated to obtain a second loss value, wherein the first loss value is used to indicate the loss of the self-supervisory signal, and the second loss value is used to indicate the loss of the weak supervision signal;

[0064] The first loss value and the second loss value are calculated to obtain a loss result, and the network model to be trained is trained according to the loss result until the loss result converges to obtain a trained network model.

[0065] In a third aspect, the present application provides a three-dimensional imaging device based on deep learning, comprising: a memory, a processor;

[0066] Memory stores computer-executable instructions;

[0067] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0068] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement the above first aspect and / or various possible implementations of the first aspect.

[0069] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0070] The present application provides a three-dimensional imaging method, device, equipment, medium and product based on deep learning. A fringe pattern set is projected onto an object to be measured by a projection device, and a shooting device is used to capture the reflected high-frequency phase shift image and secondary coding image, which are used to capture the details of the object surface and provide additional fringe level information, respectively. A high-frequency truncated phase image is extracted from the high-frequency phase shift image, and the high-frequency truncated phase image is combined with at least two special level coding shooting images into a multi-channel composite image. Using a trained network model, boundary information is extracted from the high-frequency truncated phase image, and feature information is extracted from the multi-channel composite image. Combining boundary information and feature information effectively strengthens the boundaries of each level area in the level map output by the network model, and improves the accuracy of the level map output by the network model. Based on the generated more accurate level map, the technical effect of improving the accuracy of three-dimensional reconstruction is further achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0072] Figure 1 A schematic diagram of a three-dimensional imaging method based on deep learning provided in an embodiment of the present application Figure 1 ;

[0073] Figure 2 A schematic diagram of the result of the first special level coded fringe pattern provided in an embodiment of the present application;

[0074] Figure 3 A schematic diagram of the result of the second special level coded fringe pattern provided in an embodiment of the present application;

[0075] Figure 4 A schematic diagram of a result of a three-channel composite image provided in an embodiment of the present application;

[0076] Figure 5 A schematic diagram of the structure of a network model provided in an embodiment of the present application;

[0077] Figure 6 A schematic diagram of the structure of a multi-scale attention gate module provided in an embodiment of the present application;

[0078] Figure 7 A schematic diagram of the structure of a convolutional mixing layer provided in an embodiment of the present application;

[0079] Figure 8 A schematic diagram of the structure of a phase enhancement module provided in an embodiment of the present application;

[0080] Fig. 9 A schematic diagram of a three-dimensional imaging method based on deep learning provided in an embodiment of the present application Figure 2 ;

[0081] Fig.10 A schematic diagram of a result of a single-cycle phase diagram provided in an embodiment of the present application;

[0082] Fig.11 A schematic diagram of a result of predicting a depth map provided in an embodiment of the present application;

[0083] Fig.12 A schematic diagram of a result of a depth truth map provided in an embodiment of the present application;

[0084] Fig.13 A schematic diagram of the result of reconstructing a point cloud image corresponding to a predicted depth image provided in an embodiment of the present application;

[0085] Fig.14 A schematic diagram of the result of reconstructing a point cloud image corresponding to the depth truth image provided in an embodiment of the present application;

[0086] Fig.15 A schematic diagram of the result of the reconstruction model corresponding to the predicted depth map provided in the embodiment of the present application;

[0087] Fig.16 A schematic diagram of the result of the reconstruction model corresponding to the depth truth map provided in the embodiment of the present application;

[0088] Fig.17 A schematic diagram of the results of the deviation diagram provided in the embodiments of the present application;

[0089] Fig.18 A schematic diagram of a three-dimensional imaging method based on deep learning provided in an embodiment of the present application Figure 3 ;

[0090] Fig.19 A schematic diagram of the structure of a three-dimensional imaging device based on deep learning provided in this application;

[0091] Fig. 20 A schematic diagram of the structure of a deep learning-based three-dimensional imaging device provided in this application.

[0092] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0093] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0094] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0095] In addition, this application involves conducting big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and using artificial intelligence technology to make automated decisions, and making technical solutions that have a significant impact on personal rights and interests based on the results of automated decisions. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decisions; if the user chooses to reject, the expert decision-making process will be entered.

[0096] With the continuous acceleration of the digitalization process, remote banking has become one of the important directions of the digital transformation of the banking industry. By adopting technologies such as remote audio and video and biometrics, remote banking can identify the identity of customers and verify their intentions. However, traditional two-dimensional face recognition technology mainly relies on image information, which has a certain degree of security risks and is sensitive to lighting conditions, which in turn affects the user experience. In comparison, three-dimensional object reconstruction technology has significant advantages in dealing with lighting changes, posture changes, makeup attacks, etc. This technology is widely used in three-dimensional face recognition and digital modeling of other three-dimensional objects, such as virtual banking environments and product displays.

[0097] As an active 3D imaging technology, the monocular structured light projection 3D measurement method generally includes a camera and one or more projection devices. The projection device is used to project the fringe pattern onto the surface of the object, and the camera is used to capture the deformed fringe image modulated by the surface of the object. Fringe projection profilometry is a mainstream structured light 3D imaging technology, which is favored for its non-contact, high-density point cloud and high measurement accuracy. In remote banking, in order to improve the security of face recognition and optimize the user experience, it is particularly important to achieve fast and accurate fringe projection profilometry 3D imaging.

[0098] In fringe projection profilometry, phase unwrapping is the core link that determines the speed and accuracy of three-dimensional imaging. Common phase unwrapping techniques include spatial phase unwrapping, temporal phase unwrapping, and deep learning phase unwrapping. Spatial phase unwrapping is suitable for processing continuous surfaces, but it encounters challenges when facing discontinuous or isolated objects; temporal phase unwrapping can measure discontinuous surfaces with high precision, but its process requires the support of multiple frames of patterns, so it is not efficient enough; deep learning phase unwrapping uses a neural network to learn the mapping relationship between the truncated phase and the absolute phase, thereby solving the problems faced by spatial and temporal phase unwrapping. Despite this, the current deep learning phase unwrapping technology still needs to be improved in terms of the accuracy of three-dimensional reconstruction.

[0099] The present application provides a three-dimensional imaging method, device, equipment, medium and product based on deep learning, which projects a set of fringe patterns onto an object to be measured through a projection device, and uses a shooting device to capture the reflected fringe image. These images include a high-frequency phase shift image and a secondary coding image, which are used to capture the details of the object surface and provide additional fringe level information, respectively. A high-frequency truncated phase image is extracted from the captured high-frequency phase shift image, and the high-frequency truncated phase image is combined with at least two special level coding images into a multi-channel composite image. Using a trained network model, boundary information is extracted from the high-frequency truncated phase image, and feature information is extracted from the multi-channel composite image. Combined with the boundary information and feature information, the boundaries of each level area in the level map output by the network model are effectively strengthened, and the accuracy of the level map output by the network model is improved. Based on the generated more accurate level map, the technical effect of improving the accuracy of three-dimensional reconstruction is further achieved.

[0100] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0101] Figure 1A schematic diagram of a three-dimensional imaging method based on deep learning provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, the method includes:

[0102] S101, obtaining a fringe pattern set, and sequentially projecting fringe patterns in the fringe pattern set onto an object to be measured through a projection device.

[0103] In this embodiment, the fringe pattern set includes multiple high-frequency phase-shifted fringe patterns and at least two special-order coded fringe patterns. The difference between each high-frequency phase-shifted fringe pattern in the multiple high-frequency phase-shifted fringe patterns is the difference in the phase shift amount. The different phase shift amounts lead to differences in the spatial phase distribution of the fringe. This difference is used to obtain more information about the surface features of the object to be measured, such as surface morphology, height, etc. in subsequent data processing and analysis. The change in the phase shift amount can, for example, increase or decrease according to a certain rule to achieve more comprehensive and accurate measurement. The at least two special-order coded fringe patterns can divide the order of the fringe into different multiple regions, provide more feature information for the network model, and help improve the accuracy and robustness of subsequent decoding using the network model.

[0104] Optionally, a special level coded stripe pattern is determined in the following manner: a first stripe pattern is obtained, a plurality of stripe levels corresponding to a first region in the first stripe pattern are merged, and a merged result is encoded to obtain a first encoding result; the frequency corresponding to the first stripe pattern is consistent with the frequency corresponding to the high-frequency phase-shift stripe pattern; a region other than the first region in the first stripe pattern is cyclically encoded with a preset encoding to obtain a second encoding result; a corresponding special level coded stripe pattern is obtained based on the first encoding result and the second encoding result; wherein, the first region in each of at least two special level coded stripe patterns contains different stripe levels.

[0105] In this embodiment, by merging and encoding multiple stripe levels in the first area, decoding errors caused by unclear stripe levels can be reduced. This merging process provides a clearer encoding result, which helps to improve the accuracy of decoding. Since the stripe levels contained in the first area of ​​each special-level encoded stripe image are different, this method can adjust the encoding strategy according to specific application requirements and can adapt to different measurement scenarios and conditions.

[0106] It should be noted that the frequency of the special order coded fringe pattern is consistent with the frequency of the high frequency phase shift fringe pattern, which helps to provide consistent information in the phase unwrapping process, thereby improving the accuracy of phase unwrapping. For example: In some measurements, the obtained phase value is usually limited to a specific range (such as ), which will lead to phase discontinuity and phase unwrapping. Phase unwrapping is to restore the real and continuous phase distribution.

[0107] Optionally, the special level coded stripe pattern is a stripe pattern encoded using black and white grayscale values ​​(usually represented as 0 and 1), and this encoding method can represent the levels of different stripes. Take the determination method of two special level coded stripe patterns corresponding to the case where the high frequency is determined to be 64 as an example, Figure 2 and Figure 3 As shown, the special level coded stripe pattern is a binary coded stripe using only two grayscale values ​​of black and white, and the absolute stripe level under the viewing angle of the projection device is divided into three areas, namely the left, middle and right areas. Figure 2 The red box in the middle shows the middle area of ​​the level corresponding to the first special level coding stripe image. Figure 3 The red boxes in the middle show the left and right areas corresponding to the levels of the second special level-coded fringe image. Figure 2 The middle three levels (23rd, 24th, and 25th levels) of the absolute stripe levels are merged and coded as 1, and the remaining areas are coded in a regular cycle from the middle level area to the left and right areas as 1 and 0. Figure 3 The left side (10th, 11th, 12th level) and the right side (40th, 41st, 42nd level) of the absolute fringe levels are merged and encoded as 1 respectively; the 34th-39th levels are merged and encoded as 0, and the remaining levels are cyclically encoded as 1, 1, 0, where the remaining level encoding example is that assuming the 13th level is the first remaining level, then the 13th level is encoded as 1, the 14th level is encoded as 1, the 15th level is encoded as 0, the 16th level is encoded as 1 again, the 17th level is encoded as 1, and the 18th level is encoded as 0, and so on. By additionally projecting two special level-coded fringe images, the levels can be divided into three different areas, providing more feature information for the network model, which helps to improve the accuracy and robustness of subsequent decoding using the network model. At the same time, by merging levels and using binary encoding, the subsequent phase unwrapping process can be simplified. Merging levels reduces the number of different levels that need to be processed, thereby reducing the complexity of phase unwrapping.

[0108] S102: During the projection process, the object to be measured is photographed by a photographing device to obtain a set of photographed images.

[0109] In this embodiment, during the projection process, when the fringe pattern in the fringe pattern set is projected onto the object to be measured, the fringe pattern will be deformed due to the inconsistency of the surface morphology and height of the object to be measured. Therefore, the set of photographed images captured by the photographing device are all deformed fringe patterns. Among them, the set of photographed images includes high-frequency phase-shifted photographed images corresponding to multiple high-frequency phase-shifted fringe patterns and special-order coded photographed images corresponding to at least two special-order coded fringe patterns. The high-frequency phase-shifted photographed images and the special-order coded photographed images are both deformed fringe patterns.

[0110] Optionally, multiple high-frequency phase-shifted fringe images can be processed. For example, the threshold of the modulation intensity can be set first to remove most of the noise and invalid points; then morphological operations can be used to fill the holes and breaks in the fringe image, which helps to restore the continuity of the fringe and ensure the integrity of the phase information. Small areas can be deleted. By deleting these small areas, the image can be further purified to ensure that only valid information related to the fringe is retained. When processing special level coded images, a reasonable lightness threshold needs to be set. , used to distinguish foreground from background. For pixels with brightness greater than the threshold, set them to 1 (indicating white), and for pixels with brightness less than the threshold, set them to 0 (indicating black):

[0111]

[0112] Where n represents the pixel point, Indicates the luminance corresponding to the nth pixel. This converts the special level coded image into a binary image with only black and white colors.

[0113] S103, obtaining a high-frequency truncated phase image according to a plurality of high-frequency phase-shifted images in the image set.

[0114] In this embodiment, the fringe pattern set includes a plurality of high-frequency phase-shifted fringe patterns which can be expressed as:

[0115]

[0116] Where f represents the frequency of the high-frequency phase-shift fringe pattern, N represents the number of phase-shift steps (n=0,…,N-1), Represents the nth high-frequency phase-shifted fringe pattern projected by the projection device.

[0117] The multiple high-frequency phase-shifted images in the image collection are represented as:

[0118]

[0119] in, is the surface reflectivity of the object to be measured, represents the background intensity, represents intensity modulation, is the absolute phase (continuous phase) to be recovered, which is highly correlated with the surface of the object to be measured. Indicates the nth high-frequency phase-shifted image captured by the camera. Based on the captured high-frequency phase-shifted image, the high-frequency truncated phase image is calculated using the inverse trigonometric function:

[0120]

[0121] S104: Combine the high-frequency truncated phase image and at least two special-order coded captured images in the captured image set into a multi-channel composite image.

[0122] In this embodiment, the high-frequency truncated phase image and the special level coded image are combined into a multi-channel composite image. Each image serves as a channel of the multi-channel composite image. For example, if the RGB image format is used, the high-frequency truncated phase image can be assigned to one channel (such as the R channel), and at least two special level coded images can be assigned to other channels (such as the G and B channels). In the multi-channel composite image, the information of each channel is combined for fusion to improve the feature expression ability of the overall image. The fusion process can be achieved by simple channel superposition or by feature extraction and fusion algorithms. Figure 4 As shown, the left figure represents, from top to bottom, a high-frequency truncated phase image, a first special level coded image, and a second special level coded image. These three images are combined to obtain a three-channel composite image corresponding to the right figure.

[0123] S105. Based on the trained network model, extract boundary information corresponding to the high-frequency truncated phase image, extract feature information corresponding to the multi-channel composite image, and obtain a corresponding order image according to the boundary information and the feature information.

[0124] In this embodiment, the network model includes a phase enhancement module and multiple processing layers connected in sequence, and N-1 attention modules. The multiple processing layers include: N encoding layers, convolutional mixing layers, and N-1 decoding layers; the phase enhancement module is connected to the N-1th decoding layer; wherein the i-th encoding layer is also connected to the Ni-th decoding layer through the i-th attention module; wherein i is a positive integer less than N.

[0125] In one possible implementation, the high-frequency truncated phase image is input into a phase enhancement module to obtain corresponding boundary information; the multi-channel composite image is input into the first encoding layer, and after being processed in sequence by N encoding layers, convolutional mixing layers and the first N-2 decoding layers, feature information output by the N-2 decoding layer is obtained; the output information of the first attention module, the feature information output by the N-2 decoding layer, and the boundary information are concatenated and input into the N-1 decoding layer to obtain the output information of the N-1 decoding layer, and the output information is subjected to an activation function to obtain the corresponding level map.

[0126] In this embodiment, the high-frequency truncated phase image is input into the phase enhancement block (PFB) module to obtain boundary information, which helps to more accurately extract the boundary features of the object to be tested. The multi-channel composite image is processed by multiple encoding layers, convolutional mixing layers and decoding layers in turn to gradually extract and integrate features. Information from different sources (such as attention module output, feature information, boundary information) is fused through a specific splicing operation and input into the subsequent decoding layer, and finally the level map is obtained through the activation function. The phase enhancement module can enhance the extraction of boundary information and make the model understand the boundary of the object more clearly. The architecture of multiple encoding layers and decoding layers can gradually analyze and extract features in depth, and the convolutional mixing layer helps to better integrate different features. The introduction of the attention module can make the network model more focused on important information. By splicing and fusing different information, it is possible to integrate various features and details, so that the obtained level map is more accurate and comprehensive, and the fusion of phase information and stripe level information is promoted from the channel and spatial dimensions, thereby further improving the quality and accuracy of constructing a three-dimensional model.

[0127] Optional, Figure 5 A schematic diagram of a network model is provided for the embodiment of the present application, such as Figure 5As shown in Figure 1, the network model consists of a symmetrical encoder and decoder. The encoder on the left includes five encoding layers. The first encoding layer includes two convolutional layers. Each convolutional layer consists of a 5×5 convolution operation, batch normalization (BN), and a rectified linear unit (Relu). The remaining four encoding layers include a maximum pooling layer and two convolutional layers. The right part of the network model is the decoder. The decoder contains four decoding layers. Each decoding layer consists of a transposed convolutional layer and two convolutional layers. The first encoding layer is skip-connected to the fourth decoding layer, the second encoding layer is skip-connected to the third decoding layer, the third encoding layer is skip-connected to the second decoding layer, and the fourth encoding layer is skip-connected to the first decoding layer. Attention modules such as the MultiScale Attention Gate (MSAG) module are introduced into the corresponding skip connections of the encoding layer and the decoding layer to strengthen the fusion of encoder-decoder features. The part connected between the last encoding layer and the first decoding layer is the ConvMixer layer. The first feature output by the last encoding layer has rich detail information. It is input into the convolutional mixing layer to obtain the global information of the fusion space and position of the multi-channel composite image, which can effectively extract the global context information corresponding to the multi-channel composite image. The phase enhancement module is connected to the fourth decoding layer. The phase enhancement module is used to extract the boundary information corresponding to the high-frequency truncated phase image. The boundary information corresponding to the high-frequency truncated phase image is spliced ​​together with the features processed by the multi-scale attention gate module, and the last decoding layer is used for processing to obtain the order map. The introduction of the phase enhancement module strengthens the importance of the truncated phase and extracts the key features, thereby further improving the prediction accuracy of the order map. The output layer at the end of the network model is a Sigmoid activation function, which controls the output of the network model within a preset range so that it can be represented by the differentiable stripe order required to ensure the directional propagation of the network.

[0128] Optional, Figure 6 A schematic diagram of the structure of a multi-scale attention gate module provided in an embodiment of the present application is shown in FIG. Figure 6As shown in the figure, in order to adaptively extract features of different scales, three types of convolutions with different receptive fields are used to operate on the encoder features, namely pointwise convolution, with a convolution kernel size of 1×1, ordinary convolution, with a convolution kernel size of 3×3, and dilation rate of 2. Each convolution is followed by a batch normalization (BatchNorm, BN) layer, which outputs features with the same resolution and number of channels. These features are concatenated together and then processed by the Relu activation function. Next, pointwise convolution is used to extract valuable feature values, and the Sigmoid activation function is applied. Finally, the output feature formula is obtained by multiplying the input features and adding them together:

[0129]

[0130]

[0131] in, is the encoder feature, is the feature output by the multi-scale attention gate module, and Same size. It is the feature after concatenating the three convolution outputs, and BN is batch normalization.

[0132] The multi-scale attention gate module takes a small amount of computation and optimizes the skip connections between the encoder and decoder of the network model, which can suppress irrelevant features and effectively fuse the phase information and level information of the multi-channel composite image.

[0133] Optional, Figure 7 A schematic diagram of the structure of a convolutional mixing layer provided in an embodiment of the present application is shown in FIG. Figure 7 As shown in the figure, the convolutional mixing layer consists of L convolutional mixing modules, each of which consists of three parts: depthwise convolution, pointwise convolution, and Gaussian Error Linear Unit (GELU) function. First, the input features are deep convolution (convolution kernel size is 7×7) to extract the feature information of the spatial dimension. After concatenating the input features, pointwise convolution operation (convolution kernel size is 1×1) is performed to fuse the information of the channel dimension. Both convolution operations are followed by GELU activation function and BatchNorm. The output features of each convolutional mixing module are expressed as follows:

[0134]

[0135]

[0136] in, It is the output feature of each convolution mixing module. Preferably, in this embodiment, the number of convolution mixing modules L is set to 7. The larger the convolution kernel of the depth convolution in the convolution mixing module, the farther the spatial information distance allowed to be mixed, the stronger the performance improvement of its network model, but the greater the amount of calculation. It has been proved by experiments that after the convolution kernel size is greater than 7, the increase in model accuracy becomes smaller and smaller. Therefore, preferably, the convolution kernel size of the depth convolution is 7×7.

[0137] Among them, the GELU activation function performs well in the Transformer model. The GELU activation function has smoother nonlinear characteristics than other commonly used activation functions such as the Relu function. Nonlinear activation functions such as Relu are usually used with random regularization to avoid network overfitting and improve generalization ability, while the GELU function provides random regularization with the probability statistics of the input neurons. Adding regularization to the activation function can effectively avoid the gradient vanishing problem and increase nonlinear complexity. The calculation formula of the GELU activation function is as follows:

[0138]

[0139] Where x is the input value, X is a Gaussian random variable with zero mean and unit variance, It represents the probability that X is less than or equal to a given value x. When the function conforms to the standard normal distribution, the GELU activation function can be approximately expressed as:

[0140]

[0141] Optional, Figure 8 A schematic diagram of the structure of a phase enhancement module provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the input of the phase enhancement module is the high-frequency truncated phase image , perform a 3×3 convolution operation on it, obtain the convolution result, and perform a linear transformation on the convolution result. Among them, the convolution kernel of the convolution operation is the Laplacian operator, and the parameters of the convolution kernel are fixed. The Laplacian operator is a convolution kernel commonly used to extract image boundaries in image processing. The input feature size of the phase enhancement module is consistent with the input feature, and the output feature can be expressed by the following formula:

[0142]

[0143] In the formula, represents the output characteristics of the phase enhancement module, is the Laplacian convolution kernel with fixed parameters. Represents the input features Do is the convolution operation of the convolution kernel. r and b represent the learnable scale parameter and bias parameter, which are initialized to 1 and 0, respectively.

[0144] S106: construct a three-dimensional model of the object to be measured according to the hierarchy diagram.

[0145] In this embodiment, the order map is phase-unwrapped to obtain an absolute phase map. Combined with the positional relationship between the projection device and the camera, matching points are found based on the absolute phase and epipolar constraint method, and the phase is converted into depth information using triangulation, thereby reconstructing a three-dimensional model of the object to be measured.

[0146] Optionally, since the calculation of the high-frequency truncated phase map uses inverse trigonometric functions, the extracted truncated phase is truncated at Within the range, there are 2 Therefore, it is necessary to restore these truncated phase values ​​to the true continuous phase distribution through the phase unwrapping algorithm. An appropriate integer multiple of , i.e., the fringe order, needs to be added to each pixel of the truncated phase. The formula of the phase unwrapping algorithm is as follows:

[0147]

[0148] In the formula, is a continuous phase, For the stripe level, is the high frequency cutoff phase.

[0149] Optionally, for a pixel point of a photograph taken by a photographing device , according to the epipolar constraint and the absolute phase value, the corresponding matching points of the imaging surface of the projection device are solved After that, the depth information can be obtained by triangulation. The R and t matrices of the camera can be expressed as the relative pose matrix Taking the coordinate system of the camera as reference, the projection matrix of the camera and the projection device is and , pixel coordinates , And the three-dimensional coordinates of the point to be measured The relationships can be expressed as:

[0150]

[0151]

[0152]

[0153]

[0154] in, is the homogeneous coordinate of P, use the cross product to eliminate the homogeneous factors, and and Expanding by row, we get:

[0155]

[0156] The superscript of P represents the number of rows. At this time, the singular value decomposition (SVD) can be used to decompose the A matrix into:

[0157]

[0158] in, and are all orthogonal matrices, is a diagonal matrix. Homogeneous coordinates is the singular vector of the SVD decomposition of A , and finally get the test point The three-dimensional space coordinates, that is, the point cloud:

[0159]

[0160]

[0161] In the formula, Indicates the depth information of the object to be measured.

[0162] A three-dimensional imaging method based on deep learning is provided in an embodiment of the present application. A fringe pattern set is projected onto an object to be tested by a projection device, and a reflected fringe image is captured by a shooting device. These images include a high-frequency phase shift image and a secondary coding image, which are used to capture the details of the object surface and provide additional fringe level information, respectively. A high-frequency truncated phase image is extracted from the captured high-frequency phase shift image, and the high-frequency truncated phase image is combined with at least two special level coding images to form a multi-channel composite image. The multi-channel composite image provides rich input features, which provides a basis for subsequent feature extraction and level map generation. Using a trained network model, boundary information is extracted from the high-frequency truncated phase image, and feature information is extracted from the multi-channel composite image. Combining boundary information and feature information effectively strengthens the boundaries of each level region in the level map output by the network model, and improves the accuracy of the level map output by the network model. Based on the generated more accurate level map, the accuracy of constructing a three-dimensional model of the object to be tested is further improved.

[0163] Fig. 9 A schematic diagram of a three-dimensional imaging method based on deep learning provided in an embodiment of the present application Figure 2 ,like Fig. 9As shown, in this embodiment Figure 1 Based on the embodiment, a possible implementation of a three-dimensional imaging method based on deep learning is described in detail, and the method includes:

[0164] S901, obtaining a sample fringe pattern set, and sequentially projecting fringe patterns in the sample fringe pattern set onto any three-dimensional object model through a projection device.

[0165] In this embodiment, the sample fringe pattern set includes a plurality of sample high-frequency phase-shifted fringe patterns, at least two sample special-order coded fringe patterns, and a plurality of single-period phase-shifted fringe patterns.

[0166] It should be noted that the sample high-frequency phase-shifted fringe pattern has a high spatial frequency and is used to accurately measure the details of the surface of a three-dimensional object model. Since the use of the sample high-frequency phase-shifted fringe pattern will lead to phase ambiguity, the sample special-order coded fringe pattern provides additional fringe order information to unfold the phase, thereby obtaining the absolute phase value. The single-cycle phase-shifted fringe pattern is a fringe with a single cycle, which is used to roughly measure the surface shape of a three-dimensional object model.

[0167] Optionally, a fringe pattern in a sample fringe pattern set is sequentially projected onto any three-dimensional object model at a specific position and angle by a projection device to obtain a training sample. A plurality of three-dimensional object models of different shapes and sizes are selected, and a projection device is used to project the fringe patterns onto the plurality of three-dimensional object models at specific positions and angles to obtain a training sample set. Alternatively, the positions and angles of the plurality of three-dimensional object models are changed, and the fringe patterns are projected onto the plurality of three-dimensional object models by a projection device to obtain a training sample set.

[0168] S902: During the projection process, any three-dimensional object model is photographed by a photographing device to obtain a set of sample photographed images.

[0169] In this embodiment, the sample shot image set includes sample high-frequency phase-shifted shots corresponding to multiple sample high-frequency phase-shifted fringe images, sample special-level coded shots corresponding to at least two sample special-level coded fringe images, and single-cycle phase-shifted shots corresponding to multiple single-cycle phase-shifted fringe images.

[0170] It should be noted that for each projected sample high-frequency phase-shifted fringe pattern, the shooting device captures its reflection image on the surface of any three-dimensional object model to form a sample high-frequency phase-shifted photographic image. These photographic images are used to calculate the phase information of the object surface. For each sample special-order coded fringe pattern, the shooting device captures its reflection image to form a sample special-order coded photographic image. These images provide additional fringe order information required for phase unfolding. For each single-cycle phase-shifted fringe pattern, the shooting device captures its reflection image to form a single-cycle phase-shifted photographic image. These images provide a reference for the high-frequency phase-shifted fringe pattern.

[0171] S903 , obtaining a single-cycle phase image according to a plurality of single-cycle phase-shifted images in the sample image set.

[0172] In this embodiment, a plurality of single-cycle phase-shifted images are calculated using a phase shift algorithm, such as a three-step phase shift method, a four-step phase shift method, etc., to obtain the phase value of each pixel in each single-cycle phase-shifted image, and the calculated phase value is calculated to obtain a single-cycle phase map. The single-cycle phase map is used as a weak supervisory signal for the network model to be trained. Compared with obtaining fully annotated three-dimensional shape data, the process of obtaining the single-cycle phase map is relatively simple and highly automated. Using a single-cycle phase map as a weak supervisory signal can reduce the reliance on accurately annotated data, thereby reducing the cost of data preparation.

[0173] Optional, such as Fig.10 As shown, based on multiple single-cycle phase-shifted images in the sample image set, an image corresponding to the single-cycle phase image is obtained.

[0174] S904. Obtain a sample high-frequency truncated phase image according to a plurality of sample high-frequency phase-shifted images in the sample image set, and combine the sample high-frequency truncated phase image and at least two sample special-level coded images in the sample image set into a sample multi-channel composite image.

[0175] Step S904 is similar to steps S103 and S104 and will not be described again here.

[0176] S905 , input any sample multi-channel composite image in the training sample set into the network model to be trained, and obtain a prediction level image corresponding to any sample multi-channel composite image.

[0177] In this embodiment, the model architecture of the network model to be trained can be referred to, for example, Figure 4 , input any sample multi-channel composite image in the training sample set into the network model to be trained, the network model to be trained can extract the boundary information corresponding to the sample high-frequency truncated phase image, and can extract the feature information corresponding to any sample multi-channel composite image, and obtain the corresponding prediction level image based on the boundary information and feature information.

[0178] S906 , processing the predicted order map to obtain a predicted high-frequency truncated phase map and a predicted single-cycle phase map corresponding to the predicted order map.

[0179] In this embodiment, the predicted order diagram is used for phase unwrapping. High-frequency phase information is extracted from the unfolded absolute phase, and the high-frequency phase information is truncated, and the truncated high-frequency phase information is combined into a predicted high-frequency truncated phase diagram. Single-cycle information is extracted from the absolute phase. The single-cycle phase diagram is usually used to represent the phase change within a complete cycle. The phase is periodically processed to ensure that it changes within one cycle. The processed single-cycle phase information is combined into a predicted single-cycle phase diagram.

[0180] S907. Calculate the sample high-frequency truncated phase image and the predicted high-frequency truncated phase image to obtain a first loss value, and calculate the single-cycle phase image and the predicted single-cycle phase image to obtain a second loss value, and calculate the first loss value and the second loss value to obtain a loss result, and train the network model to be trained according to the loss result until the loss result converges to obtain a trained network model.

[0181] In this embodiment, the actual sample high-frequency truncated phase map and the predicted high-frequency truncated phase map are compared and calculated to obtain a first loss value to measure the difference between the two, thereby generating a self-supervisory signal loss. Similarly, the second loss value is calculated for the single-cycle phase map and the predicted single-cycle phase map to generate a weak supervision signal loss. Although the single-cycle phase map does not provide absolute depth information, their relative changes can be used as weak supervision signals to help the network model to be trained learn the periodic characteristics of the surface shape of the three-dimensional object model. The two loss values ​​are combined to obtain the total loss result. The parameters of the network model are adjusted according to this loss result, and this process is continuously iterated until the loss result gradually converges.

[0182] Optionally, the loss function corresponding to the network model to be trained is defined as follows:

[0183]

[0184] in, express Guidelines, and yes and The weight coefficient can be determined empirically, for example is 95%, It is 5%. is the first loss value, used to indicate the loss of the self-supervisory signal, and They are the sample high-frequency truncated phase map and the predicted high-frequency truncated phase map respectively. is the second loss value, used to indicate the loss of the weak supervisory signal, and They are single-cycle phase diagram and predicted single-cycle phase diagram respectively.

[0185] The embodiment of the present application provides a three-dimensional imaging method based on deep learning, which outputs a predicted level map by inputting a sample multi-channel composite image into a network model to be trained, and extracts a predicted high-frequency truncated phase map and a predicted single-cycle phase map from it. The predicted high-frequency truncated phase map and the sample high-frequency truncated phase map are used to calculate the loss of the self-supervised signal, and the single-cycle phase map and the predicted single-cycle phase map are used to calculate the loss of the weakly supervised signal. By combining self-supervised and weakly supervised signals, the need for traditional label data is eliminated, and the cost and time of data preparation are reduced.

[0186] Optionally, the evaluation indicators of the network model can be the mean absolute error (MAE), root mean square error (RMSE), and phase unwrapping accuracy (AU) of the average depth map. The mathematical formula of the AU indicator is expressed as follows:

[0187]

[0188] Where X and Y represent the number of rows and columns of the depth image calculated after phase unwrapping, is a binary error map (BEM), and the calculation formula is as follows:

[0189]

[0190] in, To predict the depth map, is the depth truth map obtained by the four-frequency six-step temporal phase unwrapping algorithm. The operator min(·) represents the minimum value. BEM can calculate the correctly unwrapped pixels (CUPs) in the depth map. A pixel point and its corresponding , the absolute difference exceeds When the relative depth is 5%, it is set to 0 and is determined to be a point with phase unwrapping error. Otherwise, it is set to 1 and the pixel is considered to be a CUP.

[0191] Optionally, in order to verify the impact of different modules of the network model used in this application on the performance, an ablation experiment was performed. First, the improved network model (UNet) was used as the first ablation experiment #A-1. Then, the ConvMixer and MSAG modules were added as the second #A-2 and third #A-3 experiments, respectively, and finally the ConvMixer, MSAG and PEB modules were added at the same time as the fourth experiment #A-4. All experiments #A-1 to #A-4 are configured with the same batch size, learning rate and other hyperparameters, and are trained and tested on the same dataset. The analysis of the phase unwrapping results of 409 test datasets from 6 plaster models is taken as an example, as shown in Table 1.

[0192] Table 1

[0193]

[0194] As shown in Table 1, the AU value of #A-2 is lower than that of #A-1. This is because the existence of the ConvMixer layer enables the network model to converge faster. Compared with #A-1, #A-2 and #A-3, the depth map MAE and RMSE values ​​of #A-4 are lower, and the accuracy AU is higher, which shows that the ConvMixer, MSAG and PEB modules can enhance the phase unwrapping precision and accuracy of the network model.

[0195] Optionally, after the network model training is completed, a three-channel composite image consisting of a 64-frequency truncated phase image and two special level-coded images is input into the network model. The network model will output a level image. Based on the level image, the depth image is further calculated. Fig.11 and Fig.12 The predicted depth map of the object under test obtained by the network and the true depth map obtained by the traditional four-frequency three-step time phase unwrapping method are respectively shown. Among them, the mean square error (MAE) of the depth difference is 0.0936, the root mean square error (RMSE) is 0.4332, and the phase unwrapping accuracy (AU) is 99.24%. Fig.13 and Fig.14 The corresponding point cloud images generated by the predicted depth map and the depth truth map are shown respectively. Fig.15 and Fig.16 are the reconstruction models of the predicted depth map and the depth truth map, respectively. Then, the deviation analysis is performed on the reconstruction model of the predicted depth map and the reconstruction model of the depth truth map to obtain the deviation map, such as Fig.17 As shown, it can be seen that the standard deviation is 0.1761mm.

[0196] Optional, Fig.18 A schematic diagram of a three-dimensional imaging method based on deep learning provided in an embodiment of the present application Figure 3 .like Fig.18As shown, the projection device projects three or more phase-shifted sinusoidal wave fringe patterns onto the object in sequence, and the camera captures the deformed phase-shifted fringe pattern generated after being modulated by the surface of the object. By calculating the deformed phase-shifted fringe pattern, the truncated phase information is solved to obtain the corresponding truncated phase map. The truncated phase map is input into the trained network model to map the truncated phase to the order, and the order map is output. The order map is then phase-unwrapped to obtain the absolute phase map. Combined with the positional relationship between the camera and the projection device, matching points are found based on the absolute phase and epipolar constraints, and the phase is converted into depth information using triangulation, thereby reconstructing a three-dimensional model of the object. Finally, the depth information can be used to reconstruct the three-dimensional model and perform three-dimensional face recognition.

[0197] Fig.19 A schematic diagram of the structure of a three-dimensional imaging device based on deep learning provided in this application, such as Fig.19 As shown, the three-dimensional imaging method and device 1900 based on deep learning provided in this embodiment includes:

[0198] The acquisition module 1901 is used to acquire a fringe pattern set, and sequentially project the fringe patterns in the fringe pattern set onto the object to be measured through a projection device; wherein the fringe pattern set includes a plurality of high-frequency phase-shifted fringe patterns and at least two special-order coded fringe patterns;

[0199] The first processing module 1902 is used to obtain a set of photographed images by photographing the object to be measured through a photographing device during the projection process, wherein the set of photographed images includes high-frequency phase-shifted photographed images corresponding to a plurality of high-frequency phase-shifted fringe patterns and special-order coded photographed images corresponding to at least two special-order coded fringe patterns;

[0200] The second processing module 1903 is used to obtain a high-frequency truncated phase image according to a plurality of high-frequency phase-shifted images in the image set;

[0201] A combining module 1904 is used to combine the high-frequency truncated phase image and at least two special-order coded photographic images in the photographic image set into a multi-channel composite image;

[0202] The third processing module 1905 is used to extract the boundary information corresponding to the high-frequency truncated phase image based on the trained network model, extract the feature information corresponding to the multi-channel composite image, and obtain the corresponding level image according to the boundary information and the feature information;

[0203] The fourth processing module 1906 is used to construct a three-dimensional model of the object to be measured according to the level map.

[0204] In a possible implementation, the third processing module 1905 is further configured to:

[0205] The network model includes: a phase enhancement module and a plurality of processing layers connected in sequence, wherein the plurality of processing layers include: N encoding layers, a convolutional mixing layer, and N-1 decoding layers; the phase enhancement module is connected to the N-1th decoding layer;

[0206] The high-frequency truncated phase image is input into the phase enhancement module to obtain the corresponding boundary information;

[0207] The multi-channel composite image is input to the first encoding layer. After being processed by N encoding layers, convolutional mixing layers and the first N-2 decoding layers in sequence, the feature information output by the N-2 decoding layer is obtained.

[0208] The feature information and boundary information output by the N-2th decoding layer are concatenated and input into the N-1th decoding layer to obtain the output information of the N-1th decoding layer. The output information is passed through the activation function to obtain the corresponding level map.

[0209] In a possible implementation, the third processing module 1905 is further configured to:

[0210] The network model also includes: N-1 attention modules; wherein the i-th encoding layer is also connected to the Ni-th decoding layer through the i-th attention module; wherein i is a positive integer less than N;

[0211] The output information of the first attention module, the feature information output by the N-2 decoding layer, and the boundary information are concatenated and input into the N-1 decoding layer to obtain the output information of the N-1 decoding layer.

[0212] In a possible implementation, the acquisition module 1901 is further configured to:

[0213] Acquire a first fringe pattern, merge a plurality of fringe orders corresponding to a first region in the first fringe pattern, and encode the merged result to obtain a first encoding result; the frequency corresponding to the first fringe pattern is consistent with the frequency corresponding to the high-frequency phase-shifted fringe pattern;

[0214] Performing a cyclic encoding process of a preset encoding on the area other than the first area in the first fringe pattern to obtain a second encoding result;

[0215] According to the first coding result and the second coding result, a corresponding special level coding fringe pattern is obtained;

[0216] Among them, the first area of ​​each special-order coded stripe pattern in at least two special-order coded stripe patterns contains stripe orders that are different.

[0217] In a possible implementation, the acquisition module 1901 is further configured to:

[0218] Acquire a first fringe pattern set, and sequentially project a second fringe pattern in the first fringe pattern set to any three-dimensional object model at any position and angle through a projection device; wherein the first fringe pattern set includes a plurality of first high-frequency phase-shifted fringe patterns, at least two first special-order coded fringe patterns, and a plurality of single-period phase-shifted fringe patterns;

[0219] During the projection process, a first set of photographed images is obtained by photographing any three-dimensional object model at any position and angle through a photographing device, wherein the first set of photographed images includes first high-frequency phase-shifted photographed images corresponding to a plurality of first high-frequency phase-shifted fringe patterns, first special-order coded photographed images corresponding to at least two first special-order coded fringe patterns, and single-period phase-shifted photographed images corresponding to a plurality of single-period phase-shifted fringe patterns;

[0220] Obtaining a single-cycle phase image according to a plurality of single-cycle phase-shifted images in the first image set;

[0221] Obtaining a first high-frequency truncated phase image according to a plurality of first high-frequency phase-shifted images in the first image set, and combining the first high-frequency truncated phase image and at least two first special-order coded images in the first image set into a first multi-channel composite image;

[0222] According to the first multi-channel composite image and the single-cycle phase image corresponding to any three-dimensional object model at any position and angle, a plurality of three-dimensional object models are obtained, or a plurality of second multi-channel composite images corresponding to any three-dimensional object model at different positions and angles and a first single-cycle phase image corresponding to each second multi-channel composite image;

[0223] The first network model is trained according to the plurality of second multi-channel composite images and the first single-cycle phase image corresponding to each second multi-channel composite image to obtain a trained network model.

[0224] In a possible implementation, the acquisition module 1901 is further configured to:

[0225] Inputting each second multi-channel composite image into the first network model to obtain each predicted level image corresponding to each second multi-channel composite image;

[0226] Processing each predicted order map to obtain a predicted high-frequency truncated phase map and a predicted single-cycle phase map corresponding to each predicted order map;

[0227] The first high-frequency truncated phase image and the predicted high-frequency truncated phase image are calculated to obtain a first loss value, and the first single-cycle phase image and the predicted single-cycle phase image are calculated to obtain a second loss value, wherein the first loss value is used to indicate the loss of the self-supervisory signal, and the second loss value is used to indicate the loss of the weak supervision signal;

[0228] The first loss value and the second loss value are calculated to obtain a loss result, and the first network model is trained according to the loss result until the loss result converges to obtain a trained network model.

[0229] The deep learning-based three-dimensional imaging method and device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0230] Fig. 20 This is a schematic diagram of the structure of a three-dimensional imaging device based on deep learning provided in this application. Fig. 20 As shown, the deep learning-based three-dimensional imaging method device 2000 provided in this embodiment includes: at least one processor 2001 and a memory 2002. Optionally, the device 2000 also includes a communication component 2003. The processor 2001, the memory 2002 and the communication component 2003 are connected via a bus 2004.

[0231] In the specific implementation process, at least one processor 2001 executes the computer execution instructions stored in the memory 2002, so that at least one processor 2001 executes the above method.

[0232] The specific implementation process of the processor 2001 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0233] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0234] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0235] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0236] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0237] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0238] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0239] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0240] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0241] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0242] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0243] It should be noted that the terms "first", "second", etc. in the claims, the specification and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, products or devices.

[0244] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A three-dimensional imaging method based on deep learning, characterized in that: include: Acquire a fringe pattern set, and sequentially project the fringe patterns in the fringe pattern set onto the object to be tested through a projection device; wherein the fringe pattern set includes a plurality of high-frequency phase-shifted fringe patterns and at least two special-order coded fringe patterns; During the projection process, the object to be measured is photographed by a photographing device to obtain a photographing image set, wherein the photographing image set includes high-frequency phase-shift photographing images corresponding to the multiple high-frequency phase-shift fringe patterns and special-order coded photographing images corresponding to the at least two special-order coded fringe patterns; Obtaining a high-frequency truncated phase image according to a plurality of high-frequency phase-shifted images in the image set; Combining the high-frequency truncated phase image and at least two special-order coded images in the image set into a multi-channel composite image; Based on the trained network model, extracting boundary information corresponding to the high-frequency truncated phase image, extracting feature information corresponding to the multi-channel composite image, and obtaining a corresponding order image according to the boundary information and feature information; A three-dimensional model of the object to be measured is constructed according to the level map.

2. The method according to claim 1, characterized in that The network model includes: a phase enhancement module and a plurality of processing layers connected in sequence, wherein the plurality of processing layers include: N encoding layers, a convolutional mixing layer, and N-1 decoding layers; the phase enhancement module is connected to the N-1th decoding layer; Extracting boundary information corresponding to the high-frequency truncated phase image, extracting feature information corresponding to the multi-channel composite image, and obtaining a corresponding order image according to the boundary information and feature information, including: Inputting the high frequency truncated phase image into the phase enhancement module to obtain corresponding boundary information; Inputting the multi-channel composite image into the first coding layer, and processing the N coding layers, the convolutional mixing layer and the first N-2 decoding layers in sequence, to obtain feature information output by the N-2 decoding layer; The feature information output by the N-2th decoding layer and the boundary information are concatenated and input into the N-1th decoding layer to obtain the output information of the N-1th decoding layer. The output information is subjected to an activation function to obtain a corresponding level graph.

3. The method according to claim 2, characterized in that The network model further includes: N-1 attention modules; wherein the i-th encoding layer is also connected to the Ni-th decoding layer through the i-th attention module; wherein i is a positive integer less than N; The feature information output by the N-2th decoding layer and the boundary information are concatenated and inputted into the N-1th decoding layer to obtain the output information of the N-1th decoding layer, including: The output information of the first attention module, the feature information output by the N-2 decoding layer, and the boundary information are spliced ​​and input into the N-1 decoding layer to obtain the output information of the N-1 decoding layer.

4. The method according to claim 1, characterized in that The special level coding fringe pattern is determined by: Acquire a first fringe image, merge a plurality of fringe levels corresponding to a first region in the first fringe image, and encode the merged result to obtain a first encoding result; The frequency corresponding to the first fringe pattern is consistent with the frequency corresponding to the high-frequency phase-shifted fringe pattern; Performing a cyclic encoding process of a preset encoding on the area except the first area in the first fringe pattern to obtain a second encoding result; According to the first encoding result and the second encoding result, a corresponding special level encoding fringe pattern is obtained; Wherein, the first area of ​​each special-order coded stripe pattern in the at least two special-order coded stripe patterns contains stripe orders that are different.

5. The method according to claim 1, characterized in that The network model is obtained by training a training sample set, and the training samples in the training sample set are determined in the following manner: Acquire a sample fringe pattern set, and project the fringe patterns in the sample fringe pattern set to any three-dimensional object model in sequence through a projection device; wherein the sample fringe pattern set includes a plurality of sample high-frequency phase-shifted fringe patterns, at least two sample special-order coded fringe patterns, and a plurality of single-period phase-shifted fringe patterns; During the projection process, the shooting device is used to shoot any three-dimensional object model to obtain a sample shooting image set, wherein the sample shooting image set includes sample high-frequency phase-shifted shooting images corresponding to the multiple sample high-frequency phase-shifted fringe images, sample special-order coded shooting images corresponding to the at least two sample special-order coded fringe images, and single-period phase-shifted shooting images corresponding to the multiple single-period phase-shifted fringe images; Obtaining a single-cycle phase image according to a plurality of single-cycle phase-shifted images in the sample image set; A sample high-frequency truncated phase image is obtained according to a plurality of sample high-frequency phase-shifted images in the sample image set, and the sample high-frequency truncated phase image and at least two sample special-order coded images in the sample image set are combined into a sample multi-channel composite image.

6. The method according to claim 5, characterized in that The network model is trained in the following way: Inputting any sample multi-channel composite image in the training sample set into the network model to be trained, and obtaining a prediction level image corresponding to any sample multi-channel composite image; Processing the predicted order map to obtain a predicted high-frequency truncated phase map and a predicted single-cycle phase map corresponding to the predicted order map; Calculating the sample high-frequency truncated phase image and the predicted high-frequency truncated phase image to obtain a first loss value, and calculating the single-cycle phase image and the predicted single-cycle phase image to obtain a second loss value, wherein the first loss value is used to indicate the loss of the self-supervisory signal, and the second loss value is used to indicate the loss of the weak supervisory signal; The first loss value and the second loss value are calculated to obtain a loss result, and the network model to be trained is trained according to the loss result until the loss result converges to obtain a trained network model.

7. A three-dimensional imaging device based on deep learning, characterized in that: include: An acquisition module is used to acquire a fringe pattern set, and sequentially project the fringe patterns in the fringe pattern set onto the object to be measured through a projection device; wherein the fringe pattern set includes a plurality of high-frequency phase-shifted fringe patterns and at least two special-order coded fringe patterns; A first processing module is used for photographing the object to be measured by a photographing device to obtain a set of photographed images during the projection process, wherein the set of photographed images includes high-frequency phase-shifted photographed images corresponding to the multiple high-frequency phase-shifted fringe patterns and special-order coded photographed images corresponding to the at least two special-order coded fringe patterns; A second processing module, configured to obtain a high-frequency truncated phase image according to a plurality of high-frequency phase-shifted images in the image set; A combining module, used for combining the high-frequency truncated phase image and at least two special-order coded captured images in the captured image set into a multi-channel composite image; A third processing module is used to extract boundary information corresponding to the high-frequency truncated phase image and feature information corresponding to the multi-channel composite image based on the trained network model, and obtain a corresponding level image according to the boundary information and feature information; The fourth processing module is used to construct a three-dimensional model of the object to be measured according to the level map.

8. A three-dimensional imaging device based on deep learning, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images