Non-vision-field target three-dimensional reconstruction method and system based on TOF camera
Through the non-sight target three-dimensional reconstruction method based on TOF camera, the problems of poor imaging quality and high system cost in non-sight imaging technology are solved, and the low-cost and efficient three-dimensional reconstruction effect is achieved, which is suitable for interfaces with different scattering characteristics. .
Patent Information
- Application Number
- CN202411023312.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-22
AI Technical Summary
The existing non-sight imaging technology has poor imaging quality under random scattering of the target light signal, high system cost and complex maintenance, slow transient imaging method, unknown reverse process of passive imaging method, and great limitations.
Using the TOF camera-based three-dimensional reconstruction method to build a non-sighted imaging scene, the non-sighted data set is collected through the TOF camera, and image distortion is eliminated. The MRS based on variable-scale encoding-decoding architecture is used. -Net network model is used for three-dimensional reconstruction.
Three-dimensional reconstruction of non-sight targets with low cost, fast speed and excellent imaging results is achieved. The contour of the imaging target can be reconstructed from the situation where the depth degradation map is completely blurred, with few shadows, strong adaptability and good reconstruction quality.
Smart Images

Figure CN119006557A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of three-dimensional imaging technology, and in particular to a method and system for three-dimensional reconstruction of a non-viewing area target based on a TOF camera. Background Art
[0002] Non-line-of-sight imaging technology refers to the computational imaging technology of targets outside the field of view by the detector with the help of a reflective intermediary surface. The purpose of non-line-of-sight imaging is to achieve imaging or three-dimensional reconstruction of non-line-of-sight targets by performing optical inverse operations on the speckle light information on the intermediary surface or collecting the target return light information. This technology can play a key role in information collection in scenarios such as anti-terrorism reconnaissance, cave detection, and fire rescue, and has important research significance and application value. Since different materials in nature have different scattering properties as reflective intermediary surfaces, the target light signal presents different random scattering results, and the speckle information of the imaging target collected directly is seriously degraded. At present, the methods of non-line-of-sight imaging technology usually include transient imaging methods and passive imaging methods. The transient imaging method uses high-speed detectors and high-speed lighting to coordinate work, collect the target's return light information for reconstruction, and has high system cost and maintenance cost, slow imaging speed, and poor quality; the passive imaging method uses the speckle of the intermediary surface for optical inverse operations, but since the inverse process is unknown, this method has great limitations and poor imaging results. Summary of the invention
[0003] The main purpose of the embodiments of the present invention is to provide a method and system for three-dimensional reconstruction of non-viewing area targets based on a TOF camera with low cost, high speed and excellent imaging results.
[0004] To achieve the above object, an embodiment of the present invention provides a method for 3D reconstruction of a non-viewing area target based on a TOF camera, comprising the following steps:
[0005] Build a non-line-of-sight imaging scene; wherein the non-line-of-sight imaging scene consists of three parts: a TOF camera, an intermediate surface, and an imaging target;
[0006] Collecting a non-line-of-sight data set from the non-line-of-sight imaging scene by a TOF camera; wherein the non-line-of-sight data set includes an original depth-degraded image and a corresponding depth-clear image;
[0007] Preprocessing the non-viewing area data set to eliminate the image distortion of the TOF camera and retain the scattering degradation characteristics of the intermediary surface;
[0008] The preprocessed non-viewing area data set is used to train the constructed three-dimensional reconstruction model MRS-Ne to obtain the MRS-Net network model; wherein the three-dimensional reconstruction model MRS-Ne is constructed using a variable scale encoding-decoding architecture based on deep learning;
[0009] The depth-degraded image to be reconstructed is input into the MRS-Net network model, and a depth-reconstructed image of the non-viewing scene is output.
[0010] In some embodiments, the step of setting up a non-line-of-sight imaging scene comprises the following steps:
[0011] Selecting complex human portrait plaster as the imaging target;
[0012] A diffuse reflection plate with a reflectivity of 90% is selected as the intermediate surface;
[0013] The light source of the TOF camera's illumination module consists of four infrared laser diodes with a wavelength of 850nm, each of which is covered with a layer of frosted glass; the receiving module is used to receive the reflected light signal;
[0014] Building a non-viewing-area imaging scene according to the imaging target, the intermediate surface and the TOF camera;
[0015] Among them, during non-field of view imaging, the illumination light source directly illuminates the imaging target, and the lens receives the return light signal to the photoelectric sensor. By calculating the time difference caused by the propagation between the photoelectric sensor and the imaging target, the distance between the TOF camera and the imaging target is calculated.
[0016] In some embodiments, the collecting of the non-line-of-sight data set from the non-line-of-sight imaging scene by a TOF camera comprises the following steps:
[0017] Place a rotating table under the imaging target and obtain the features of the imaging target at different angles by rotating it one circle.
[0018] Use a TOF camera to image the target, obtain the depth information of each pixel of the target, and then construct a depth image;
[0019] Performing expansion processing on the acquired depth image to obtain the non-viewing area data set;
[0020] When the intermediate surface is a mirror, the depth image collected under the intermediate surface is used as the label data corresponding to the training data;
[0021] When the intermediate surface is a diffuse reflection plate, the depth image collected under the intermediate surface is used as training data.
[0022] In some embodiments, the preprocessing of the non-viewing area dataset comprises the following steps:
[0023] Use a standard scattering plane as the imaging target to fill the entire non-viewing field with light;
[0024] Place the imaging target at a distance d;
[0025] Collect the distorted depth map map (d, x, y), subtract the distorted depth data from the depth data to be corrected, and obtain the depth image with the image distortion eliminated. The expression for depth image correction is:
[0026] I d (x,y)=I′ d (x,y)-(map(d,x,y)-d x )
[0027] Among them, I d is the corrected depth image; I′ d is the natural scene before correction, map(d,x,y) is the depth distortion feature matrix, d x is the distance feature value.
[0028] In some embodiments, the preprocessed non-viewing area data set is used to train the constructed three-dimensional reconstruction model MRS-Ne to obtain the MRS-Net network model, including the following steps:
[0029] ResNet-50 is used as a feature extraction network to process the non-viewing area data set, and a residual relationship between input and output is constructed on the encoder side to obtain a feature map of the non-viewing area data set; wherein a bottleneck residual module is embedded in the feature extraction network;
[0030] The feature map is processed through the channel attention module to obtain a deep reconstructed image, completing the construction of the MRS-Net network model.
[0031] In some embodiments, the processing of the feature extraction network includes five stages, wherein:
[0032] In the first stage, the preprocessed deep degraded image is input into the feature extraction network to extract multi-layer features;
[0033] The last four stages contain 3, 4, 6, and 3 bottleneck residual modules respectively. The channel dimensions of each stage are 64, 128, 256, and 512 respectively. After the bottleneck residual modules of the last four stages, the feature maps of the corresponding shapes are obtained.
[0034] In some embodiments, the processing of the channel attention module includes:
[0035] The effective features obtained at each stage of the feature extraction network are processed by the channel attention module respectively. The features extracted in the previous stage are upsampled after passing through the channel attention module, and finally added with the features extracted in the next stage as the input of the corresponding channel attention module in the next stage. Skip connections are used to achieve multi-scale effective feature fusion. At the end of the decoder, the features decoded by channel attention in each stage are upsampled, and then the features are fused to output the final deep reconstructed image.
[0036] Another aspect of the embodiment of the present invention further provides a non-viewing area target 3D reconstruction system based on a TOF camera, comprising:
[0037] The first module is used to build a non-line-of-sight imaging scene; wherein the non-line-of-sight imaging scene consists of three parts: a TOF camera, an intermediate surface, and an imaging target;
[0038] The second module is used to collect a non-line-of-sight data set from the non-line-of-sight imaging scene through a TOF camera; wherein the non-line-of-sight data set includes an original depth-degraded image and a corresponding depth-clear image;
[0039] The third module is used to preprocess the non-viewing area data set to eliminate the image distortion of the TOF camera and retain the scattering degradation characteristics of the intermediate surface;
[0040] The fourth module is used to train the constructed three-dimensional reconstruction model MRS-Ne using the preprocessed non-viewing area data set to obtain the MRS-Net network model; wherein the three-dimensional reconstruction model MRS-Ne is constructed using a variable scale encoding-decoding architecture based on deep learning;
[0041] The fifth module is used to input the depth-degraded image to be reconstructed into the MRS-Net network model and output the depth-reconstructed image of the non-viewing scene.
[0042] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;
[0043] The memory is used to store programs;
[0044] The processor executes the program to implement the method described above.
[0045] Another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.
[0046] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.
[0047] The embodiment of the present invention includes at least the following beneficial effects: the present invention provides a method and system for three-dimensional reconstruction of non-line-of-sight targets based on a TOF camera, wherein the scheme first builds a non-line-of-sight imaging scene; wherein the non-line-of-sight imaging scene consists of three parts: a TOF camera, an intermediate surface, and an imaging target; then, a non-line-of-sight data set is collected from the non-line-of-sight imaging scene by a TOF camera; wherein the non-line-of-sight data set includes an original depth-degraded image and a corresponding depth-clear image; then, the non-line-of-sight data set is preprocessed to eliminate the image distortion of the TOF camera and retain the scattering degradation characteristics of the intermediate surface; the preprocessed non-line-of-sight data set is used to train the constructed three-dimensional reconstruction model MRS-Ne to obtain an MRS-Net network model; wherein the three-dimensional reconstruction model MRS-Ne is constructed using a variable-scale encoding-decoding architecture based on deep learning; finally, the depth-degraded image to be reconstructed is input into the MRS-Net network model, and a depth-reconstructed image of the non-line-of-sight scene is output. The present invention has low cost, high speed, and excellent imaging results, and can reconstruct the contour of the imaging target from a completely blurred depth degradation image, and there are fewer shadows. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;
[0049] Figure 2 is a flow chart of the overall steps provided by an embodiment of the present invention;
[0050] Figure 3 It is a flowchart of implementation steps in a specific scenario provided by an embodiment of the present invention;
[0051] Figure 4 is a schematic diagram of a non-viewing-area imaging scene provided by an embodiment of the present invention;
[0052] Figure 5 is a schematic diagram of the results before and after the depth image distortion correction provided by an embodiment of the present invention;
[0053] Figure 6 It is a schematic diagram of the structure of the neural network MRS-Net model provided by an embodiment of the present invention;
[0054] Figure 7is a schematic diagram of reconstruction results of a diffuse reflection plate with an intermediate surface having a reflectivity of 90% provided by an embodiment of the present invention;
[0055] Figure 8 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.
[0057] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0058] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.
[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.
[0060] Before describing the embodiments of the present invention in detail, some related technologies involved in the embodiments of the present invention are first described as follows:
[0061] TOF Camera: Photon Flight Time Ranging Camera (Time of Fight Camera);
[0062] Non-line-of-sight imaging: imaging targets that are not within the detector’s field of view (non-line-of-sight);
[0063] Intermediary surface: the reflective surface used for non-line-of-sight imaging;
[0064] ResNet-50: Deep Residual Networks.
[0065] The embodiment of the present invention provides a method and system for three-dimensional reconstruction of non-line-of-sight targets based on a TOF camera, which relates to the field of three-dimensional imaging technology. The method for three-dimensional reconstruction of non-line-of-sight targets based on a TOF camera provided in an embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a three-dimensional reconstruction method for non-line-of-sight targets based on a TOF camera, etc., but is not limited to the above forms.
[0066] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0067] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.
[0068] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.
[0069] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.
[0070] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.
[0071] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a method for three-dimensional reconstruction of non-line-of-sight targets based on a TOF camera. The following is explained using the application of the method for three-dimensional reconstruction of non-line-of-sight targets based on a TOF camera in a server 101 as an example. It can be understood that the method can also be applied to a terminal 102.
[0072] Reference Figure 2 , Figure 2 The flowchart of the method for 3D reconstruction of non-line-of-sight targets based on a TOF camera and applied to a server according to an embodiment of the present invention is provided. The execution subject of the method can be any of the aforementioned computer devices (including a server or a terminal). Figure 2 , the method may include the following steps:
[0073] Build a non-line-of-sight imaging scene; wherein the non-line-of-sight imaging scene consists of three parts: a TOF camera, an intermediate surface, and an imaging target;
[0074] Collecting a non-line-of-sight data set from the non-line-of-sight imaging scene by a TOF camera; wherein the non-line-of-sight data set includes an original depth-degraded image and a corresponding depth-clear image;
[0075] Preprocessing the non-viewing area data set to eliminate the image distortion of the TOF camera and retain the scattering degradation characteristics of the intermediary surface;
[0076] The preprocessed non-viewing area data set is used to train the constructed three-dimensional reconstruction model MRS-Ne to obtain the MRS-Net network model; wherein the three-dimensional reconstruction model MRS-Ne is constructed using a variable scale encoding-decoding architecture based on deep learning;
[0077] The depth-degraded image to be reconstructed is input into the MRS-Net network model, and a depth-reconstructed image of the non-viewing scene is output.
[0078] The implementation process of the method of the present invention in a specific application scenario is described in detail below in conjunction with the accompanying drawings of the specification:
[0079] The embodiment of the present invention provides a method for three-dimensional reconstruction of non-field of view targets based on a TOF camera to solve the problems existing in the aforementioned background technology. Based on its imaging principle, the TOF camera can obtain three types of data, namely, intensity, phase, and depth distance of the target light signal, and has the characteristics of fast imaging speed, low cost, and real-time three-dimensional imaging. However, the TOF camera has problems such as poor imaging accuracy and large lens distortion when performing depth imaging. The present invention can achieve high-quality non-field of view three-dimensional reconstruction of severely degraded depth images from the scattered information on the intermediate surface under intermediate surfaces with different scattering characteristics, and has the characteristics of strong adaptability and good reconstruction quality.
[0080] like Figure 3 As shown, the 3D reconstruction method of a non-viewing area target based on a TOF camera according to an embodiment of the present invention comprises the following steps:
[0081] Step 1: Build a non-line-of-sight imaging scene, which consists of three parts: a TOF camera, an intermediate surface, and an imaging target.
[0082] Step 2: Collect a non-viewing area dataset, which consists of the unprocessed original depth-degraded images and the corresponding depth-clear images.
[0083] Step 3: Preprocess the depth data collected by the TOF camera to eliminate the image distortion caused by the TOF camera imaging and only retain the intermediate surface scattering degradation characteristics.
[0084] Step 4: Build a 3D reconstruction model MRS-Net based on a deep learning variable scale encoding-decoding architecture to reconstruct non-viewing targets, and use the preprocessed depth data to complete the model training.
[0085] Step 5: Train the MRS-Net network model designed by the present invention. After the training converges, input the tested deep degraded image into the network for 3D reconstruction, and finally output the deep reconstructed image of the non-viewing scene.
[0086] Specifically, for the above five steps, the implementation process is as follows:
[0087] Step 1: Build a non-line-of-sight imaging scene, which consists of three parts: a TOF camera, an intermediate surface, and an imaging target. Figure 4 The figure shows a real non-field of view scene constructed by the present invention, and the camera field of view completely covers the imaging target. The light source of the lighting module of the TOF camera is composed of 4 infrared laser diodes with a wavelength of 850nm, each diode is covered with a layer of frosted glass, and the receiving module is used to receive the reflected light signal. The imaging target uses complex portrait plaster, and the intermediate surface uses a diffuse reflection plate with a reflectivity of 90%. In non-field of view imaging, the illumination light source directly illuminates the imaging target, and the lens receives the return light signal to the photoelectric sensor. By calculating the time difference T generated by the propagation between the sensor and the target, the speed of light is C, and the distance L between the TOF camera and the target can be expressed as:
[0088]
[0089] Step 2, collect a non-field of view data set, which consists of an unprocessed original depth-degraded image and a corresponding deep clear image. Use a TOF camera to image the imaging target and obtain the depth information of each pixel of the target. Since the neural network requires a large amount of data to learn the intrinsic relationship between input and output, the depth image data is collected in the non-field of view imaging scene of step 1. The resolution of the image is 240×320. In the process of collecting data, only the intermediate surface and the imaging target need to be replaced. When the intermediate surface is a mirror, the light signal can be effectively reflected to achieve point-to-point clear imaging. Therefore, the depth image collected under the intermediate surface is regarded as the label data corresponding to the training data. When the intermediate surface is a diffuse reflection plate, the light signal becomes blurred due to the scattering effect of diffuse reflection, so the depth image collected under the intermediate surface is regarded as the training data. In order to increase the diversity of the data, that is, the imaging target needs to have different shapes, angles and quantities, etc., a rotating table is placed under the imaging target, and the features of the imaging target at different angles are obtained by rotating one circle. The rotating platform can obtain 850 depth images at different angles after one rotation. There are 12 imaging targets in total, and a total of 850×12=10200 depth maps are obtained. In order to simulate various interferences in real application scenarios, the obtained depth maps are rotated 90°, 180°, and 270° respectively, and then the depth maps are expanded to 40,800 depth maps, which meets the requirements of deep learning training data.
[0090] Step 3: Preprocess the depth data collected by the TOF camera to eliminate the image distortion caused by the TOF camera imaging and only retain the intermediate surface scattering degradation characteristics. Since the TOF camera has image distortion and photoelectric conversion noise when imaging in the non-field of view, a numerical correction method is used to process the depth distortion image. Figure 5It is represented as the result of the depth image correction before and after the method. This method uses a standard scattering plane as the imaging target, the light source fills the entire field of view, and the target is placed at a distance d. Secondly, the distorted depth map map (d, x, y) is collected, and the depth data to be corrected minus the distorted depth data can be obtained to obtain the depth image with the image distortion eliminated. The expression of depth image correction is as follows:
[0091] I d (x,y)=I' d (x,y)-(map(d,x,y)-d x ) (2)
[0092] Among them, I' d is the natural scene before correction, map(d,x,y) is the depth distortion feature matrix, which represents the characteristics of the distortion, d x i is the distance characteristic value, which can be the center pixel value or the value of an external distance measuring device.
[0093] Step 4, build a 3D reconstruction model MRS-Net based on a deep learning variable scale coding-decoding architecture for reconstructing non-viewing targets, and use the preprocessed depth data to complete the model training. The intermediate surface selected in the present invention is a diffuse reflection plate with a reflectivity of 90%. The effective light information of the target is distributed in an infinite space due to its scattering effect. Therefore, a wide range of input is selected in the present invention to obtain more effective information in the deep degraded image. Figure 6 It is represented as the overall structure of the MRS-Net model. The specific implementation steps are as follows:
[0094] Step 41, at the encoder end, in order to more effectively extract effective information from the deep degraded image, the feature extraction network used in the present invention is ResNet-50, which introduces a bottleneck residual module composition, constructs a residual relationship between input and output, optimizes parameters more effectively, and solves the overfitting problem of the network while deepening the network depth. The feature extraction network is divided into 5 stages. In the first stage, the preprocessed deep degraded image is input into the feature extraction network to extract multi-layer features. The last four stages contain 3, 4, 6, and 3 bottleneck residual modules, respectively, and the channel dimensions of each stage are 64, 128, 256, and 512, respectively. In the embodiment of the present invention, the input size is 192×192 deep degraded image, and the feature map shapes obtained by the last four stages of ResNet-50 are: (64, 48, 48), (128, 24, 24), (256, 12, 12), (512, 6, 6).
[0095] Step 42: At the decoder end, the channel attention module is applied to the effective features obtained in each stage of step 41. The features extracted in the previous stage are upsampled after the channel attention module, and finally added to the features extracted in the next stage as the input of the corresponding channel attention module in the next stage. The jump connection is used to achieve multi-scale effective feature fusion. At the end of the decoder, the features decoded by the channel attention in each stage are upsampled, and then the features are fused to output the final deep reconstructed image of size 96×96.
[0096] Step 5, training the improved network designed by the present invention, inputting the tested deep degraded image into the network for reconstruction after the training converges, and finally outputting the deep reconstructed image of the non-field of view scene. Before training the MRS-Net model, the deep data preprocessing of the non-field of view imaging data set is performed according to the method of step 3, the input is a deep degraded image of size 192×192, and the output is a deep reconstructed image of size 96×96. The method of the present invention uses the MATLAB programming language to implement the non-field of view 3D reconstruction model. The model training is carried out on the Windows 11 operating system and NVIDIA GeForce RTX4090, and the training time is 10 hours. The data set is randomly divided into a training set, a validation set, and a test set in a ratio of 8:1:1. The network is trained with the training set to obtain a trained model, the batch size of the training is set to 128, and a total of 150 rounds of training are performed. The loss function is set to mean square error (MSE), and the loss function is back-propagated using the Adam optimizer to minimize the loss function value. Finally, the deep degraded image of the test set is sent to the trained model for 3D reconstruction, and high-quality 3D reconstruction in data-driven non-field of view scenes is achieved, such as Figure 7 Shown are the reconstruction results of the MRS-Net model.
[0097] In summary, the present invention has built a non-line-of-sight imaging scene to construct a depth data set, and the imaging intermediate surface uses a diffuse reflection plate with a reflectivity of 90%. The MRS-Net neural network model with a variable-scale encoding-decoding structure is designed. The model uses an encoder-decoder neural network that combines a deep residual network with channel attention to extract effective information from the depth-degraded image, and finally realizes the three-dimensional reconstruction of the non-line-of-sight target through multi-scale feature fusion. The model can reconstruct the outline of the imaging target from the completely blurred depth degradation map, and there are fewer shadows. The non-line-of-sight three-dimensional reconstruction method proposed in the present invention has the characteristics of good reconstruction effect, strong applicability, and low cost, and provides a new solution to the reconstruction problem of depth image degradation caused by intermediate surface scattering in non-line-of-sight scenes.
[0098] The embodiment of the present invention further provides a non-viewing area target 3D reconstruction system based on a TOF camera, comprising:
[0099] The first module is used to build a non-line-of-sight imaging scene; wherein the non-line-of-sight imaging scene consists of three parts: a TOF camera, an intermediate surface, and an imaging target;
[0100] The second module is used to collect a non-line-of-sight data set from the non-line-of-sight imaging scene through a TOF camera; wherein the non-line-of-sight data set includes an original depth-degraded image and a corresponding depth-clear image;
[0101] The third module is used to preprocess the non-viewing area data set to eliminate the image distortion of the TOF camera and retain the scattering degradation characteristics of the intermediate surface;
[0102] The fourth module is used to train the constructed three-dimensional reconstruction model MRS-Ne using the preprocessed non-viewing area data set to obtain the MRS-Net network model; wherein the three-dimensional reconstruction model MRS-Ne is constructed using a variable scale encoding-decoding architecture based on deep learning;
[0103] The fifth module is used to input the depth-degraded image to be reconstructed into the MRS-Net network model and output the depth-reconstructed image of the non-viewing scene.
[0104] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0105] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned non-line-of-sight target three-dimensional reconstruction method based on a TOF camera when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0106] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0107] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0108] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.
[0109] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the non-viewing area target three-dimensional reconstruction method based on a TOF camera according to the embodiment of the present invention;
[0110] Input / output interface 803, used to implement information input and output;
[0111] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);
[0112] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);
[0113] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .
[0114] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned non-viewing area target three-dimensional reconstruction method based on a TOF camera is implemented.
[0115] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0116] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0117] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.
[0118] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0119] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0120] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0121] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0122] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0123] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0124] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0125] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.
[0127] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.
Claims
1. A 3D reconstruction method for non-viewing targets based on TOF camera. It is characterized in that The following steps are involved: Build a non-line-of-sight imaging scene; wherein the non-line-of-sight imaging scene consists of three parts: a TOF camera, an intermediate surface, and an imaging target; Collecting a non-line-of-sight data set from the non-line-of-sight imaging scene by a TOF camera; wherein the non-line-of-sight data set includes an original depth-degraded image and a corresponding depth-clear image; Preprocessing the non-viewing area data set to eliminate the image distortion of the TOF camera and retain the scattering degradation characteristics of the intermediary surface; The preprocessed non-viewing area data set is used to train the constructed three-dimensional reconstruction model MRS-Ne to obtain the MRS-Net network model; wherein the three-dimensional reconstruction model MRS-Ne is constructed using a variable scale encoding-decoding architecture based on deep learning; The depth-degraded image to be reconstructed is input into the MRS-Net network model, and a depth-reconstructed image of the non-viewing scene is output.
2. According to claim 1, a method for three-dimensional reconstruction of non-viewing area targets based on a TOF camera, It is characterized in that The process of constructing a non-viewing area imaging scene comprises the following steps: Selecting complex human portrait plaster as the imaging target; A diffuse reflection plate with a reflectivity of 90% is selected as the intermediate surface; The light source of the TOF camera's illumination module consists of four infrared laser diodes with a wavelength of 850nm, each of which is covered with a layer of frosted glass; the receiving module is used to receive the reflected light signal; Building a non-viewing-area imaging scene according to the imaging target, the intermediate surface and the TOF camera; Among them, during non-field of view imaging, the illumination light source directly illuminates the imaging target, and the lens receives the return light signal to the photoelectric sensor. By calculating the time difference caused by the propagation between the photoelectric sensor and the imaging target, the distance between the TOF camera and the imaging target is calculated.
3. According to claim 1, a method for three-dimensional reconstruction of non-viewing area targets based on a TOF camera, It is characterized in that The method of collecting a non-line-of-sight data set from the non-line-of-sight imaging scene by using a TOF camera comprises the following steps: Place a rotating table under the imaging target and obtain the features of the imaging target at different angles by rotating it one circle. Use a TOF camera to image the target, obtain the depth information of each pixel of the target, and then construct a depth image; Performing expansion processing on the acquired depth image to obtain the non-viewing area data set; When the intermediate surface is a mirror, the depth image collected under the intermediate surface is used as the label data corresponding to the training data; When the intermediate surface is a diffuse reflection plate, the depth image collected under the intermediate surface is used as training data.
4. According to claim 1, a method for three-dimensional reconstruction of non-viewing area targets based on a TOF camera, It is characterized in that The preprocessing of the non-viewing area data set comprises the following steps: Use a standard scattering plane as the imaging target to fill the entire non-viewing field with light; Place the imaging target at a distance d; Collect the distorted depth map map (d, x, y), subtract the distorted depth data from the depth data to be corrected, and obtain the depth image with the image distortion eliminated. The expression for depth image correction is: I d (x,y)=I′ d (x,y)-(map(d,x,y)-d x ) Among them, I d is the corrected depth image; I′ d is the natural scene before correction, map(d,x,y) is the depth distortion feature matrix, d x is the distance feature value.
5. According to claim 1, a method for three-dimensional reconstruction of non-viewing area targets based on a TOF camera, It is characterized in that The method uses the preprocessed non-viewing area data set to train the constructed three-dimensional reconstruction model MRS-Ne to obtain the MRS-Net network model, including the following steps: ResNet-50 is used as a feature extraction network to process the non-viewing area data set, and a residual relationship between input and output is constructed on the encoder side to obtain a feature map of the non-viewing area data set; wherein a bottleneck residual module is embedded in the feature extraction network; The feature map is processed through the channel attention module to obtain a deep reconstructed image, completing the construction of the MRS-Net network model.
6. The method for three-dimensional reconstruction of non-viewing area targets based on a TOF camera according to claim 5, It is characterized in that The processing of the feature extraction network includes five stages, among which: In the first stage, the preprocessed deep degraded image is input into the feature extraction network to extract multi-layer features; The last four stages contain 3, 4, 6, and 3 bottleneck residual modules respectively. The channel dimensions of each stage are 64, 128, 256, and 512 respectively. After the bottleneck residual modules of the last four stages, the feature maps of the corresponding shapes are obtained.
7. The method for three-dimensional reconstruction of non-viewing area targets based on a TOF camera according to claim 6, It is characterized in that The processing process of the channel attention module includes: The effective features obtained at each stage of the feature extraction network are processed by the channel attention module respectively. The features extracted in the previous stage are upsampled after passing through the channel attention module, and finally added with the features extracted in the next stage as the input of the corresponding channel attention module in the next stage. Skip connections are used to achieve multi-scale effective feature fusion. At the end of the decoder, the features decoded by channel attention in each stage are upsampled, and then the features are fused to output the final deep reconstructed image.
8. A 3D reconstruction system for non-line-of-sight targets based on TOF cameras. It is characterized in that include: The first module is used to build a non-line-of-sight imaging scene; wherein the non-line-of-sight imaging scene consists of three parts: a TOF camera, an intermediate surface, and an imaging target; The second module is used to collect a non-line-of-sight data set from the non-line-of-sight imaging scene through a TOF camera; wherein the non-line-of-sight data set includes an original depth-degraded image and a corresponding depth-clear image; The third module is used to preprocess the non-viewing area data set to eliminate the image distortion of the TOF camera and retain the scattering degradation characteristics of the intermediate surface; The fourth module is used to train the constructed three-dimensional reconstruction model MRS-Ne using the preprocessed non-viewing area data set to obtain the MRS-Net network model; wherein the three-dimensional reconstruction model MRS-Ne is constructed using a variable scale encoding-decoding architecture based on deep learning; The fifth module is used to input the depth-degraded image to be reconstructed into the MRS-Net network model and output the depth-reconstructed image of the non-viewing scene.
9. An electronic device, It is characterized in that including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, It is characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.
Citation Information
Cited By
Two-stage active steady-state non-vision-field imaging method based on data-model driving
CN119383481A
Two-stage active steady-state non-line-of-sight imaging method based on data-model driven
CN119383481B
Rapid non-vision field imaging system and method of area array time-of-flight camera
CN119716894A
