A non-line-of-sight imaging method based on deep convolutional inverse graph network
Through deep convolutional reverse graph network, the neural network is trained using reflective images, and fast non-view imaging is achieved under fixed perspective, solving the problems of complex equipment and slow imaging speed in the existing technology. It is suitable for monitoring security scenarios and has general Adaptability and efficient imaging capabilities.
Patent Information
- Application Number
- CN202110665529.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-16
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-06-16
AI Technical Summary
Existing non-field-of-sight imaging methods have limitations in device complexity and imaging speed, especially when imaging in passive imaging, the signal-to-noise ratio decreases and relies on advanced algorithms, limiting its popularity in outdoor laboratories.
Using a non-sight imaging method based on a deep convolutional inverse graph network, two fixed viewing angle cameras and upper computer control systems are used to train a neural network to achieve fast non-sight imaging with ordinary cameras. The network structure includes an encoder and a decoder , use reflected images for preprocessing and training, and output non-sight scene images.
It realizes fast non-view imaging at a fixed perspective, simple equipment, suitable for monitoring security scenarios, with good universality and fast imaging speed.
Smart Images

Figure CN113344774B_ABST
Abstract
Description
Technical Field
[0001] The invention discloses a non-viewing-area imaging method based on a deep convolutional inverse graph network, and belongs to the field of computational imaging. Background Art
[0002] The current mainstream imaging methods of non-line-of-sight imaging include active and passive imaging. Active imaging is a field with earlier research and wider research methods. Its imaging equipment mainly includes modulated pulse lasers and high-speed imaging equipment such as streak cameras. The time resolution of high-speed cameras can generally reach picoseconds or even femtoseconds. Compared with the propagation speed of light, it can achieve the acquisition of optical information under transient light transmission. Although active imaging equipment is advanced, the equipment price and imaging complexity are relatively high, which has certain limitations on its application outside the laboratory.
[0003] Passive non-line-of-sight imaging does not require the use of active light sources, but only requires the self-luminescence of the imaging scene or the reflection of ambient light by objects. However, when imaging using this method, the imaging is subject to the interference of ambient light, which will cause the signal-to-noise ratio to decrease during imaging. Current passive non-line-of-sight imaging methods include the use of light intensity information, polarization information, and coherence information. Although traditional passive non-line-of-sight imaging devices are not as complex as active imaging devices, they rely on advanced algorithms, which will cause the imaging speed to decrease. The present invention is based on the principle of deep learning. For certain scenes under fixed viewing angles, such as monitoring security scenes, non-line-of-sight scenes (such as monitoring blind spots) and scenes within the field of view (such as monitoring scenes) are used to pre-train the neural network. When used later, only the field of view scene image needs to be input into the network, and the network can output the non-line-of-sight scene image, which has the advantages of fast imaging speed and simple equipment. Summary of the invention
[0004] Technical problem: The purpose of the present invention is to address the problems and shortcomings of the above-mentioned prior art and propose a non-field of view imaging method based on a deep convolutional inverse graph network. The imaging hardware device only requires an ordinary camera, and the software uses a deep convolutional inverse graph network to achieve fast non-field of view imaging in certain fixed-viewing angle scenes.
[0005] Technical solution: A non-viewing-area imaging method based on a deep convolutional inverse graph network is provided in the present invention. The method is based on two cameras with fixed viewing angles and a host computer for controlling the two cameras to shoot and train a neural network. The method comprises the following steps:
[0006] Step 1: Use camera 1 to obtain a non-viewing scene image, and camera 2 to obtain a reflected image generated by the non-viewing scene;
[0007] Step 2: Build a deep convolutional inverse graph network according to the usage scenario;
[0008] Step 3: After preprocessing the non-viewing scene image obtained in step 1 and the reflected image generated by it, the reflected image is used as input and the corresponding scene image is used as output as a data set for training the network;
[0009] Step 4: Use the data set obtained in step 3 to train the deep convolutional inverse graph network established in step 2, and save the trained network parameters for subsequent use;
[0010] Step 5: Use camera 2 to obtain the reflected image generated by the non-viewing scene, and after preprocessing, send it to the neural network trained in step 4 to output the corresponding non-viewing scene image;
[0011] in:
[0012] When executing step 1, the host computer is required to control camera 1 and camera 2 to shoot simultaneously to ensure that the acquired non-viewing scene image corresponds to the reflected image generated by the non-viewing scene.
[0013] The deep convolutional inverse graph network has an encoder part based on a convolutional layer, a decoder based on a transposed convolutional layer, and the number of network layers is adjusted according to the complexity of the scene.
[0014] The preprocessing described in step 3 and step 5 includes extracting ROI (Region of Interest) from the image, resampling the image according to the input dimension of the network, i.e. upsampling or downsampling the image, and removing the background of the image.
[0015] The inverse perspective transformation is used when extracting the ROI and resampling the image, and the image difference is used when the background is eliminated, that is, the difference between the reflected image and the background image is performed when there is no content in the non-viewing area scene.
[0016] The input dimension of the deep convolutional inverse graph network needs to match the number of channels of the reflection image generated by the non-viewing scene, that is, the corresponding network input dimensions are different when the reflection image generated by the non-viewing scene is in color or grayscale.
[0017] The loss function used in step 4 to train the deep convolutional inverse graph network is:
[0018] L(θ,φ)=D KL (q φ (z|x)|p(z))-E z~q [logp θ (y|z)] (1)
[0019] Among them, θ and φ represent the network parameters of the decoder and encoder respectively, x represents the input image, z represents the latent variable, and D KL (q φ(z|x)|p(z)) represents the decoder output distribution q φ KL divergence between (z|x) and the prior distribution p(z), E z~q [logp θ (y|z)] represents the likelihood probability of the decoder. During the network training process, the training is performed by minimizing the loss function. The network training status is checked through the output of the test set image. The network parameters are saved once every 10 trainings.
[0020] The reflection image generated by the non-viewing scene obtained in step 5 needs to be sent to the trained network after the same preprocessing as step 3.
[0021] Beneficial effects: The non-viewing-area imaging method based on the deep convolutional inverse graph network of the present invention has the characteristics of fast speed and simple imaging equipment for non-viewing-area imaging under fixed viewing angle. It is applicable to monitoring and security scenes under fixed viewing angle, and only needs to train the network for the scene in advance. For different scenes, the network structure can be easily changed, and it has good universality. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A stereogram of a non-viewing scene according to an embodiment of the present invention;
[0023] Figure 2 A top view of a non-viewing scene according to an embodiment of the present invention;
[0024] Figure 3 It is a schematic diagram of non-viewing scene changes;
[0025] Figure 4 This is the deep convolutional inverse graph network model diagram;
[0026] Figure 5 The figure is a deep convolutional reverse graph network structure diagram. Among them, I is the convolution layer, II is the BN layer, III is the Relu layer, IV is the pooling layer, V is the fully connected layer, and VI is the transposed convolution layer. DETAILED DESCRIPTION
[0027] The technical solution of the present invention is now further described in conjunction with the accompanying drawings and embodiments.
[0028] The basic idea of the present invention is that non-line-of-sight imaging is to obtain information from the non-line-of-sight scene based on the reflection image generated at a certain place by the non-line-of-sight scene acquired by the camera, and to infer the image of the non-line-of-sight scene. The general structure of the deep convolutional inverse graph network is to input an image and generate another image after network calculation. This structure is suitable for non-line-of-sight imaging scenes. For some scenes with fixed viewing angles, such as monitoring security scenes, the position of the camera is fixed, so the source of the information it captures is fixed. For some monitoring blind spots, they can be regarded as non-line-of-sight scenes, but the reflection information generated by them can be monitored and obtained. Therefore, if a large number of non-line-of-sight scene images and their corresponding reflection images are used to train the deep convolutional inverse graph network, then after the network training is completed, when the reflection information generated by the non-line-of-sight scene is input into the network, the network can calculate the relevant non-line-of-sight scene image according to the pre-trained network model. The specific operation principle is as follows.
[0029] like Figure 1 In the non-field of view scene shown, the information of the non-field of view scene is constantly changing, so the reflected image generated on the reflective surface also changes accordingly. The reason why there are occluders in the scene is that the existence of the occluders will encode the information of the non-field of view scene to a certain extent. The reflection information generated on the reflective surface at this time is called penumbra information. This penumbra information is particularly important when the reflective surface is a diffuse reflective surface. If there are no occluders, the information of each point on the reflective surface comes from all points on the non-field of view scene. This will cause the reflection information to be too redundant and retain very little useful information. When the reflective surface is a high-gloss material or other material, the effect of the occluder will be weakened. Because the background of the image taken by the camera in fields such as monitoring and security is basically fixed, any movement of any object in the scene will directly or indirectly affect the image taken by the camera. In non-field of view imaging, indirect influence is used. For example Figure 3 As shown, although the change in the position of the object in the non-viewing scene will not directly affect the picture taken by camera 2, the movement of the object's position will affect the distribution of light in the space, and the corresponding reflection information generated on the reflection surface will also be affected. This influence can be used by the network to extract the corresponding features for generating images of the non-viewing scene.
[0030] The non-line-of-sight imaging method based on the deep convolutional inverse graph network first needs to produce a data set for training the network. The data set mainly consists of non-line-of-sight scene images and reflection images generated by non-line-of-sight scenes on reflective surfaces. The reflection images generated by non-line-of-sight scenes on reflective surfaces are used as the input of the deep convolutional inverse graph network, and the non-line-of-sight scene images are used as the output of the network. The data set is obtained by two cameras controlled by the host computer, and the two cameras need to shoot at the same time to ensure that the non-line-of-sight scene corresponds to the reflection image it generates. During shooting, the objects in the non-line-of-sight scene are generally constantly changing, but the background is generally fixed. Training the network generally requires a data set of tens of thousands.
[0031] After obtaining the above dataset, it cannot be directly used for neural network training. For reflected images, the camera has a certain angle deviation when shooting, and the perspective effect will cause the image to be deformed to a certain extent, such as Figure 1 The ROI area shown needs to be transformed into a rectangular image after inverse perspective transformation. And because the input of the neural network is generally not suitable for using data of larger dimensions, the image after inverse perspective transformation generally needs to be resampled to meet the input dimension of the neural network. In the case of ambient light interference in the scene, the background of the reflected image needs to be eliminated. In this step, the background image can be taken in advance. When preprocessing the reflected image, the reflected image and the background image can be directly used for difference operation. For non-field of view scene images, since the image is only obtained in the training stage, the camera position can be aligned with the non-field of view scene, and generally only resampling is required without other transformations.
[0032] After basic preprocessing of the dataset metadata, the dataset needs to be divided into training set, test set and validation set. The division ratio is generally 60:20:20. If the number of datasets used is large, a ratio of 98:1:1 can be used. Since neural network training is generally assisted by deep learning frameworks such as Tensorflow, the dataset also needs to be converted into a tensor that can be read by Tensorflow. In this step, all datasets can be packaged as tensors in advance, but considering that images occupy a large amount of memory, a batch reading method can be used.
[0033] The deep convolutional inverse graph network can be built to adjust the number of layers or structure of the network according to the complexity of the non-viewing scene image, but the basic structure of the network remains unchanged, such as Figure 4 As shown in Figure 1, the network consists of an encoder and a decoder. The encoder is responsible for reading the preprocessed reflection image generated by the non-viewing scene, and is responsible for reducing the image dimension and extracting the probability distribution. The decoder is responsible for sampling the probability distribution and further increasing the image dimension to draw the non-viewing scene image based on the sampled features. The detailed structure of the network is shown in Figure 1. Figure 5As shown in the figure, the decoder is mainly composed of a stack of convolutional layers, BatchNormalization layers (BN layers), Relu layers, and pooling layers. It reduces the dimension and extracts features at the same time. The number of stacked layers can be adjusted accordingly according to the complexity of the imaging scene. The last layer of the encoder is a fully connected layer, which is used to output the probability distribution, and the latent variable z is obtained by resampling. The decoder reads in the latent variables and stacks transposed convolutional layers to increase the image dimension while drawing the non-viewing scene image. The number of transposed convolutional layers can also be adjusted according to the imaging needs, and finally outputs the non-viewing scene image. The loss function used when training the network is
[0034] L(θ,φ)=D KL (q φ (z|x)|p(z))-E z~q [logp θ (y|z)] (2)
[0035] Among them, θ and φ represent the network parameters of the decoder and encoder respectively, x represents the input image, z represents the latent variable, and D KL (q φ (z|x)|p(z)) represents the decoder output distribution q φ The KL divergence between (z|x) and the prior distribution p(z), which is usually a standard normal distribution. It represents the likelihood probability of the decoder, which is generally expressed by the cross entropy loss function between the network output and the reference value in the training set. When actually training the network, it is generally assumed that the decoder output distribution q φ (z|x) and the prior distribution p(z) are both normal distributions, where p(z) is the standard normal distribution q φ (z|x) is but
[0036]
[0037] The likelihood probability of the decoder is expressed using the cross entropy loss function, i.e.
[0038]
[0039] Where x and y represent the input and output of the network respectively, and f represents the nonlinear relationship represented by the network.
[0040] After the network is trained in a supervised learning manner according to the above loss function, the verification set can be used to verify the actual effect of the network, and the reflection image taken by the subsequent camera 1 can be pre-processed as input to output the non-viewing area scene image.
Claims
1. A non-viewing area imaging method based on a deep convolutional inverse graph network, characterized in that: The method is based on two cameras with fixed viewing angles and a host computer that controls the two cameras to shoot and train a neural network; the method includes the following steps: Step 1: Use camera 1 to obtain a non-viewing scene image, and camera 2 to obtain a reflected image generated by the non-viewing scene; Step 2: Build a deep convolutional inverse graph network according to the usage scenario; the network consists of an encoder and a decoder. The encoder is responsible for reading in the reflection image generated by the preprocessed non-viewing scene, and is responsible for reducing the image dimension and extracting the probability distribution, while the decoder is responsible for sampling the probability distribution and further increasing the image dimension to draw the non-viewing scene image according to the sampled features; the decoder is stacked by convolutional layers, BN layers, Relu layers, and pooling layers, which can reduce the dimension and extract features at the same time. The number of stacked layers can be adjusted according to the complexity of the imaging scene. The last layer of the encoder is a fully connected layer, which is used for the output of the probability distribution, and the latent variable z is obtained by resampling. The decoder reads in the latent variable and stacks the transposed convolutional layers to increase the image dimension while drawing the non-viewing scene image. The number of transposed convolutional layers can also be adjusted according to the imaging needs, and finally outputs the non-viewing scene image; Step 3: After preprocessing the non-viewing scene image obtained in step 1 and the reflection image generated by it, the reflection image is used as input and the corresponding scene image is used as output as a data set for training the network; Step 4: Use the data set obtained in step 3 to perform supervised training on the deep convolutional inverse graph network established in step 2, and save the trained network parameters for subsequent use; Step 5: Use camera 2 to obtain the reflected image generated by the non-viewing scene, and after preprocessing, send it to the neural network trained in step 4 to output the corresponding non-viewing scene image; The preprocessing in step 3 and step 5 includes extracting ROI from the image, resampling the image according to the input dimension of the network, i.e. upsampling or downsampling the image, and removing the background of the image; The inverse perspective transformation is used when extracting the ROI and resampling the image, and the image difference is used when eliminating the background, that is, the difference between the reflected image and the background image is used.
2. The non-viewing area imaging method based on deep convolutional inverse graph network according to claim 1, characterized in that: When executing step 1, the host computer is required to control camera 1 and camera 2 to shoot simultaneously to ensure that the acquired non-viewing scene image corresponds to the reflected image generated by the non-viewing scene.
3. The non-viewing area imaging method based on deep convolutional inverse graph network according to claim 1, characterized in that: The deep convolutional inverse graph network has an encoder part based on a convolutional layer, a decoder based on a transposed convolutional layer, and the number of network layers is adjusted according to the complexity of the scene.
4. The non-viewing area imaging method based on deep convolutional inverse graph network according to claim 1, characterized in that: The input dimension of the deep convolutional inverse graph network needs to match the number of channels of the reflection image generated by the non-viewing scene, that is, the corresponding network input dimensions are different when the reflection image generated by the non-viewing scene is in color or grayscale.
5. The non-viewing area imaging method based on deep convolutional inverse graph network according to claim 1, characterized in that: The loss function used in step 4 to train the deep convolutional inverse graph network is: L(θ,ϕ)=D KL (q φ (z|x)|p(z))-E z~q [log p θ (y|z)] (1) Among them, θ and φ represent the network parameters of the DCIGN decoder and encoder respectively, x represents the input image, y represents the output of the network, z represents the latent variable, and D KL (q φ (z|x)|p(z)) represents the decoder output distribution q φ The KL divergence between (z|x) and the prior distribution p(z), which is usually a standard normal distribution, E z~q [log p θ (y|z)] represents the likelihood probability of the decoder, which is represented by the cross entropy loss function between the network output and the reference value in the training set. During the network training process, the loss function is minimized for training, and the network training status is checked through the output of the test set images. The network parameters are saved once every 10 trainings.
6. The non-viewing area imaging method based on deep convolutional inverse graph network according to claim 1, characterized in that: The reflection image generated by the non-viewing scene obtained in step 5 needs to be sent to the trained network after the same preprocessing as step 3.
