Passive non-vision field imaging method and device based on deep learning

Abstract

The invention provides a passive non-vision field imaging method and device based on deep learning, and belongs to the technical field of optical non-vision field imaging. The method comprises the steps that in a passive non-vision field imaging scene, a camera is used for collecting a scattering image, generated on a medium wall surface, of a projection image projected by a self-luminous object; inputting the scattering image into a preset imaging neural network, wherein the imaging neural network outputs an imaging result of the projection image; wherein the imaging neural network is composed of a pair of encoder and decoder with symmetrical structures; the encoder is composed of nine layers of down-sampling units which are connected in sequence, and the decoder is composed of nine layers of up-sampling units which are connected in sequence. According to the passive non-vision field imaging method, passive non-vision field imaging is achieved through deep learning, no special limitation exists on an imaging scene, the imaging distance can be increased, and therefore the universality of passive non-vision field imaging is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical non-line-of-sight imaging, and in particular proposes a passive non-line-of-sight imaging method and device based on deep learning. Background Art

[0002] Non-line-of-sight imaging technology can perform optical detection and imaging of areas outside the linear field of view. This technology obtains information by using the relay reflective surface in the scene, and solves it through a specific algorithm to achieve imaging that bypasses obstacles. This technology expands the optical imaging range from the full field of view to the super field of view, and has great application potential in the fields of navigation and autonomous driving, disaster relief, endoscopic medical diagnosis, etc.

[0003] Non-line-of-sight imaging technology is divided into two categories: active and passive, depending on whether active light sources are used to controllably illuminate hidden scenes. Active methods have the disadvantages of high hardware equipment costs, complex system construction, long imaging acquisition time, high risk, and poor concealment; passive methods can effectively avoid the above disadvantages, but due to the small amount of information obtained during the detection phase, the imaging calculation process has a strong instability, resulting in poor imaging quality and generalization, and often has special requirements or restrictions on imaging scenes. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a method and device for passive non-line-of-sight imaging based on deep learning. The present invention uses deep learning to achieve passive non-line-of-sight imaging, has no special restrictions on the imaging scene, and can increase the imaging distance, thereby greatly improving the versatility of passive non-line-of-sight imaging.

[0005] The first aspect of the present invention provides a passive non-line-of-sight imaging method based on deep learning, comprising:

[0006] In the passive non-line-of-sight imaging scenario, a camera is used to collect the scattered image generated by the projection image projected by the self-luminous object on the intermediate wall;

[0007] The scattering image is input into a preset imaging neural network, and the imaging neural network outputs the imaging result of the projection image; wherein the imaging neural network is composed of a pair of encoders and decoders with symmetrical structures; the encoder is composed of 9 layers of downsampling units connected in sequence, and the decoder is composed of 9 layers of upsampling units connected in sequence.

[0008] In a specific embodiment of the present invention, the projected image is a square binary image with a pixel number less than or equal to the maximum pixel number of the self-luminous object.

[0009] In a specific embodiment of the present invention, before inputting the scattering image into a preset imaging neural network, the method further includes:

[0010] The scatter image is sampled to an input size set by the imaging neural network.

[0011] In a specific embodiment of the present invention, each downsampling unit of the encoder includes a two-dimensional convolution layer for downsampling, a batch normalization layer for controlling the stability of internal parameters of the model, and an activation function for introducing nonlinearity; each upsampling unit of the decoder includes a two-dimensional deconvolution layer for upsampling, a batch normalization layer for controlling the stability of internal parameters of the model, a jump connection for enhancing feature fusion, and an activation function for introducing nonlinearity.

[0012] In a specific embodiment of the present invention, it also includes:

[0013] The imaging neural network initially inputs a scattering image of size 512×512, and after passing through the first downsampling unit, the output image of size 256×256 and feature number 4 enters the second downsampling unit; after passing through the second downsampling unit, the output image of size 128×128 and feature number 4 enters the third downsampling unit; after passing through the third downsampling unit, the output image of size 64×64 and feature number 8 enters the fourth downsampling unit; after passing through the fourth downsampling unit, the output image of size 32×32 and feature number 16 enters the fifth downsampling unit; after passing through the fifth downsampling unit, the output image of size 16×16 and feature number 32 enters the sixth downsampling unit; after passing through the sixth downsampling unit, the output image of size 8 ×8, the image with a feature number of 32 enters the seventh down-sampling unit; after passing through the seventh down-sampling unit, the image with a size of 4×4 and a feature number of 32 enters the eighth down-sampling unit; after passing through the eighth down-sampling unit, the image with a size of 2×2 and a feature number of 32 enters the ninth down-sampling unit; after passing through the ninth down-sampling unit, the image with a size of 1×1 and a feature number of 32 enters the first up-sampling unit; after passing through the first up-sampling unit, the image with a size of 2×2 and a feature number of 32 is output, which is jump-connected with the image output by the eighth down-sampling unit, and is spliced ​​along the feature dimension into an image with a size of 2×2 and a feature number of 64, which enters the second up-sampling unit; after passing through the second up-sampling unit, the image with a size of 4×4 and a feature number of 32 is output. The image is jumped connected with the output image of the seventh down-sampling unit, and is spliced ​​along the feature dimension into an image with a size of 4×4 and a feature number of 64 to enter the third up-sampling unit; after the third up-sampling unit, an image with a size of 8×8 and a feature number of 32 is output, and the image is jumped connected with the output image of the sixth down-sampling unit, and is spliced ​​along the feature dimension into an image with a size of 8×8 and a feature number of 64 to enter the fourth up-sampling unit; after the fourth up-sampling unit, an image with a size of 16×16 and a feature number of 32 is output, and the image is jumped connected with the output image of the fifth down-sampling unit, and is spliced ​​along the feature dimension into an image with a size of 16×16 and a feature number of 64 to enter the fifth up-sampling unit; after the fifth up-sampling unit, an image with a size of 32 is output ×32, an image with a feature number of 16, this image is jump-connected with the output image of the fourth down-sampling unit, and is spliced ​​along the feature dimension into an image with a size of 32×32 and a feature number of 32 to enter the sixth up-sampling unit; after the sixth up-sampling unit, an image with a size of 64×64 and a feature number of 8 is output, this image is jump-connected with the output image of the third down-sampling unit, and is spliced ​​along the feature dimension into an image with a size of 64×64 and a feature number of 16 to enter the seventh up-sampling unit; after the seventh up-sampling unit, an image with a size of 128×128 and a feature number of 4 is output, this image is jump-connected with the output image of the second down-sampling unit, and is spliced ​​along the feature dimension into an image with a size of 128×128 and a feature number of 8 to enter the eighth up-sampling unit;After the eighth upsampling unit, an image with a size of 256×256 and a feature number of 4 is output, which is jump-connected with the image output by the first downsampling unit, and spliced ​​along the feature dimension into an image with a size of 256×256 and a feature number of 8 to enter the ninth upsampling unit; after the ninth upsampling unit, an image with a size of 512×512 and a feature number of 4 is output and enters the readout layer of the imaging neural network; the readout layer outputs an image with a size of 512×512 and a feature number of 1, which is the final imaging result. ;

[0014] In a specific embodiment of the present invention, it also includes:

[0015] Each downsampling unit includes a convolution layer with a convolution kernel size of 4×4 and a sliding step size of 2 and a LeakyRELU activation function; wherein the second downsampling unit to the eighth downsampling unit also include a batch normalization layer;

[0016] Each upsampling unit layer includes a deconvolution layer with a convolution kernel size of 4×4 and a sliding step size of 2, a batch normalization layer, and a RELU activation function; wherein, the first upsampling unit to the third upsampling unit also use a dropout with a probability of 0.5;

[0017] The readout layer includes a convolution layer with a convolution kernel size of 1×1 and a sigmoid function.

[0018] In a specific embodiment of the present invention, before inputting the scattering image into a preset imaging neural network, the method further includes:

[0019] training the imaging neural network;

[0020] The training of the imaging neural network comprises:

[0021] 1) In the passive non-line-of-sight imaging scene, the self-luminous object is used to project a plurality of true value images, and a scattered image generated by each true value image on the intermediate wall surface is acquired through the camera; wherein the plurality of true value images have the same features;

[0022] 2) sampling each true value image and the corresponding scattering image to a size set by the imaging neural network to form a training sample, wherein the input of the training sample is the scattering image after the size adjustment, and the true value label is the corresponding true value image after the size adjustment; all training samples constitute a training set;

[0023] 3) constructing the imaging neural network;

[0024] 4) Using the training set obtained in step 2) to train the imaging neural network, to obtain the trained imaging neural network.

[0025] A second aspect of the present invention provides a passive non-line-of-sight imaging device based on deep learning, comprising:

[0026] A scattered image acquisition module is used to use a camera to acquire a scattered image generated by a projection image projected by a self-luminous object on an intermediate wall in a passive non-line-of-sight imaging scene;

[0027] An imaging module is used to input the scattered image into a preset imaging neural network, and the imaging neural network outputs the imaging result of the projection image; wherein the imaging neural network is composed of a pair of encoders and decoders with symmetrical structures; the encoder is composed of 9 layers of downsampling units connected in sequence, and the decoder is composed of 9 layers of upsampling units connected in sequence.

[0028] A third aspect of the present invention provides an electronic device, including:

[0029] at least one processor; and a memory communicatively coupled to the at least one processor;

[0030] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above-mentioned passive non-line-of-sight imaging method based on deep learning.

[0031] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned passive non-line-of-sight imaging method based on deep learning.

[0032] Features and beneficial effects of the present invention:

[0033] The present invention does not have any special restrictions on the hidden space that is inaccessible in the non-line-of-sight imaging task (from the hidden target to the relay wall). It only uses a data-driven method to reduce the ill-posedness of the problem, greatly improving the practicality and versatility of passive non-line-of-sight imaging. In addition, the distance from the camera to the relay wall is increased from 1-2m in the traditional method to 10m in the present invention, further expanding the application scenarios of non-line-of-sight imaging. Under high-precision timing control, the present invention can realize high-speed automatic data acquisition, which greatly facilitates the acquisition of a large amount of experimental data required for neural network training; at the same time, after the network training is completed, real-time high-speed imaging can be achieved at the same speed, and the imaging effect is excellent.

[0034] The main advantages of the present invention are as follows:

[0035] Traditional passive non-line-of-sight imaging methods introduce occlusions between hidden targets and intervening walls to reduce the pathological degree of imaging problems. However, high-quality imaging requires occlusions to be as complex as possible, while high generalization requires occlusions to be as simple, arbitrary, or even non-occluded as possible. There is an unavoidable trade-off between the two. In contrast, the present invention uses a deep learning method to effectively circumvent this problem.

[0036] The imaging method using deep learning has no additional restrictions on the non-field of view area, and has no specific requirements for the placement distance and angle of the screen (hidden target), whether there is occlusion or scattering during the transmission process, whether the fixed background light is uniform, whether the observation relay wall is flat, etc. The present invention can fully extract and utilize these implicit environmental information. In fact, the more complex the environment, the more information can be used, the less pathological the problem is, and the more conducive to imaging. Since the hidden area behind the relay wall is often unknown or difficult to enter in the actual application of non-field of view detection, it is not reasonable to limit the area to meet specific requirements. The present invention does not restrict it and is more in line with the actual scene, which can effectively improve the application scope and practical value of the technology;

[0037] The present invention adopts an end-to-end neural network that includes additional scene prior information, and there is no need to explicitly solve and express this information. The prior effect can be directly utilized without specific knowledge of the prior. Compared with traditional methods, it is more direct and efficient and can avoid errors caused by approximate modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is an overall flow chart of a passive non-line-of-sight imaging method based on deep learning according to an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of a passive non-line-of-sight imaging scene according to a specific embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of the imaging neural network structure of a specific embodiment of the present invention;

[0041] Figure 4 This is a diagram showing the effect of imaging neural network training in a specific embodiment of the present invention;

[0042] Figure 5 This is a graph of imaging results of an imaging neural network test according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention proposes a passive non-line-of-sight imaging method and device based on deep learning, which is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] The first aspect of the present invention provides a passive non-line-of-sight imaging method based on deep learning, comprising:

[0045] In the passive non-line-of-sight imaging scenario, a camera is used to collect the scattered image generated by the projection image projected by the self-luminous object on the intermediate wall;

[0046] The scattering image is input into a preset imaging neural network, and the imaging neural network outputs the imaging result of the projection image; wherein the imaging neural network is composed of a pair of encoders and decoders with symmetrical structures; the encoder is composed of 9 layers of downsampling units connected in sequence, and the decoder is composed of 9 layers of upsampling units connected in sequence.

[0047] In a specific implementation of the present invention, the passive non-line-of-sight imaging method based on deep learning has an overall process as follows: Figure 1 As shown, it is divided into a training phase and a testing phase, including the following steps:

[0048] 1) Training phase.

[0049] 1-1) Build a passive non-line-of-sight imaging scene.

[0050] In a specific embodiment of the present invention, the passive non-line-of-sight imaging scene schematic diagram is as follows Figure 2As shown. In this scene, the imaging target is an image displayed by a self-luminous object. The shape of the self-luminous object is close to a two-dimensional plane. It can be an object of any form that meets the brightness requirements. A screen is generally used. In this embodiment, an LED screen is selected, and an LCD screen that meets the brightness requirements can also be used. In this embodiment, the self-luminous object selects a monochrome high-brightness LED screen, the average brightness of the screen is greater than 2000nit, and the spacing between the LED lamp beads on the screen is greater than 5mm; the camera uses a CMOS industrial camera with a pixel count of more than 300,000 and an output accuracy of 16 bits; the imaging lens parameters used with the camera are determined according to the specific imaging scene, so that the scattered image to be collected on the intermediate wall can completely fill the camera frame; the reflectivity of the monochromatic light emitted by the intermediate wall to the LED screen is greater than 20%. In this imaging scene, the distance between the LED screen and the intermediate wall is less than or equal to 60cm, and the distance between the intermediate wall and the camera is less than or equal to 10m. There is an obstruction between the camera and the LED screen, so that the camera cannot directly observe the image displayed on the LED screen, but can observe the scattered image transmitted to the intermediate wall. Specifically, in this embodiment, the size of the LED screen is 488mm×244mm, the number of pixels is 64×32, the size of a single pixel is 7.62mm×7.62mm, the average brightness is about 2500nit, the screen emits monochromatic green light, the central wavelength is 516nm, and the spectral line width is about 33nm. The camera uses a sCMOS scientific camera PCO panda4.2bi mono, with a maximum resolution of 2048×2048, a pixel size of 6.5μm×6.5μm, a photosensitive surface size of 13.3mm×13.3mm, a dynamic range of 87Db, and a maximum quantum efficiency of more than 80%. The imaging lens selects the Zhonglian Science and Technology 100mm fixed focus lens. The intermediate wall uses a whiteboard with rough texture wallpaper to simulate the common brick walls in life. The distance between the LED screen and the intermediate wall is about 60cm, and the distance between the intermediate wall and the camera is about 10m.

[0051] 1-2) Use the scene built in step 1-1) to build a training set.

[0052] In this embodiment, different true value images are displayed on the self-luminous object, and then the camera is used to collect the corresponding scattered images on the intermediate wall. Among them, the true value image should select a square binary image with a number of pixels less than or equal to the maximum number of pixels of the self-luminous object. In this embodiment, in order to cooperate with the imaging network structure, the input and output of the network are both 512×512 size images. Therefore, when constructing the training set, it is necessary to re-interpolate and sample the true value image displayed on the self-luminous object and the scattered image collected by the camera to a size of 512×512, and then use the interpolated scattered image as input, and the corresponding interpolated true value image as the true value label to form a set of training samples. In this embodiment, no less than 5000 groups of training samples are used to form a training set. The images displayed by the self-luminous objects in the training set have common features (for example, they are all handwritten numbers, handwritten letters, human body movements, etc.). In the process of constructing the training set, the imaging scene is kept fixed and free of interference.

[0053] In a specific embodiment of the present invention, a public data set MNIST handwritten digits is selected as the image displayed on the LED screen to construct a training set, and the size of the original image of the data set is 28×28. In this embodiment, the original image is preprocessed, converted into a binary image of 512×512 size, and then input into the LED screen for display, and then the corresponding scattered image is collected by the camera in the constructed passive non-field of view imaging scene, and the size of the collected scattered image is 800×800. The scattered image is preprocessed and converted into 512×512 size. This embodiment uses a total of 5500 different handwritten digit images, and the training set finally constructed has a total of 5500 sets of training samples.

[0054] 1-3) Construct an imaging neural network.

[0055] In this embodiment, the imaging neural network is based on the classic convolutional neural network Unet structure, and is composed of a pair of symmetrical encoders and decoders; wherein the encoder and the decoder are respectively a contraction path and an expansion path, and the encoder is composed of 9 layers of downsampling units connected in sequence, and each layer of downsampling unit contains a two-dimensional convolution layer for downsampling, a batch normalization layer for controlling the stability of the model's internal parameters, and an activation function for introducing nonlinearity; the encoder is directly connected to the decoder, and the encoder output is the decoder input; the decoder is composed of 9 layers of upsampling units connected in sequence, and each layer of upsampling unit contains a two-dimensional deconvolution layer for upsampling, a batch normalization layer for controlling the stability of the model's internal parameters, a jump connection for enhancing feature fusion, and an activation function for introducing nonlinearity. The encoder can gradually reduce the spatial dimension of the input image to achieve feature extraction, and the decoder gradually restores the features to the original image dimension output.

[0056] In this embodiment, the overall structure of the imaging neural network is as follows: Figure 3As shown in the figure, the imaging neural network initially inputs a scattering image of size 512×512. After passing through downsampling unit 1 (including a convolution layer with a convolution kernel size of 4×4, a sliding step of 2, and a LeakyRELU activation function), the image with an output size of 256×256 and a feature number of 4 enters downsampling unit 2; after passing through downsampling unit 2 (including a convolution layer with a convolution kernel size of 4×4, a sliding step of 2, a batch normalization layer, and a LeakyRELU activation function), the image with an output size of 128×128 and a feature number of 4 enters downsampling unit 3; after passing through downsampling unit 3 (including a convolution layer with a convolution kernel size of 4×4, a sliding step of 2, a batch normalization layer, and a LeakyRELU activation function), the image with an output size of 128×128 and a feature number of 4 enters downsampling unit 3. After the downsampling unit 4 (including the convolution layer with a convolution kernel size of 4×4 and a sliding step size of 2, the batch normalization layer and the LeakyRELU activation function), the image with an output size of 32×32 and a feature number of 16 enters the downsampling unit 5; after the downsampling unit 5 (including the convolution layer with a convolution kernel size of 4×4 and a sliding step size of 2, the batch normalization layer and the LeakyRELU activation function), the image with an output size of 16×16 and a feature number of 32 enters the downsampling unit 6; after the downsampling unit 6 (including the convolution layer with a convolution kernel size of 4×4 and a sliding step size of 2, the batch normalization layer and the LeakyRELU activation function), the image with an output size of 16×16 and a feature number of 32 enters the downsampling unit 6. Normalization layer and LeakyRELU activation function), the output size is 8×8 and the number of features is 32, which enters the downsampling unit 7; after the downsampling unit 7 (including the convolution layer with a convolution kernel size of 4×4 and a sliding step of 2, the batch normalization layer and the LeakyRELU activation function), the output size is 4×4 and the number of features is 32, which enters the downsampling unit 8; after the downsampling unit 8 (including the convolution layer with a convolution kernel size of 4×4 and a sliding step of 2, the batch normalization layer and the LeakyRELU activation function), the output size is 2×2 and the number of features is 32, which enters the downsampling unit 9; after the downsampling unit 9 (including the convolution layer with a convolution kernel size of 4×4 and a sliding step of 2 After the convolution layer and RELU activation function), the image with the output size of 1×1 and the number of features of 32 enters the upsampling unit 1; after the upsampling unit 1 (including the deconvolution layer with the convolution kernel size of 4×4 and the sliding step size of 2, the batch normalization layer, the RELU activation function, and the dropout with a probability of 0.5), the image with the output size of 2×2 and the number of features of 32 is output. This image is jump-connected with the output image of the downsampling unit 8, and is spliced ​​along the feature dimension into an image with the size of 2×2 and the number of features of 64, and enters the upsampling unit 2; after the upsampling unit 2 (including the deconvolution layer with the convolution kernel size of 4×4 and the sliding step size of 2, the batch normalization layer, the RELU activation function, and the dropout with a probability of 0.5), the image with the output size of 2×2 and the number of features of 32 is output.5 dropout) after the output size is 4 × 4, the number of features of the image is 32, the image and the down-sampling unit 7 output image jump connection, along the feature dimension splicing into a size of 4 × 4, the number of features of 64 image into the up-sampling unit 3; after the up-sampling unit 3 (including the convolution kernel size of 4 × 4, the sliding step size of 2 deconvolution layer, batch normalization layer, RELU activation function, and the probability of 0.5 dropout), the output size is 8 × 8, the number of features of the image is 32, the image and the down-sampling unit 6 output image jump connection, along the feature dimension splicing into a size of 8 × 8, the number of features of the image is 64, and the up-sampling unit 4; after the up-sampling unit 4 (including the convolution kernel size of 4 × 4, the sliding step size of 2 deconvolution layer, batch normalization layer, RELU activation function, and the probability of 0.5 dropout), the output size is 8 × 8, the number of features of the image is 32, the image and the down-sampling unit 6 output image jump connection, along the feature dimension splicing into a size of 8 × 8, the number of features of the image is 64, and the up-sampling unit 4 After passing through the upsampling unit 5 (including the deconvolution layer with a convolution kernel size of 4×4 and a sliding step size of 2, the batch normalization layer and the RELU activation function), the output image is 16×16 in size and 32 in feature number, which is jump-connected with the output image of the downsampling unit 5 and concatenated along the feature dimension into an image of 16×16 in size and 64 in feature number, and enters the upsampling unit 5; after passing through the upsampling unit 5 (including the deconvolution layer with a convolution kernel size of 4×4 and a sliding step size of 2, the batch normalization layer and the RELU activation function), the output image is 32×32 in size and 16 in feature number, which is jump-connected with the output image of the downsampling unit 4 and concatenated along the feature dimension into an image of 32×32 in size and 32 in feature number, and enters the upsampling unit 6; after passing through the upsampling unit 6 (including the deconvolution layer with a convolution kernel size of 4×4 and a sliding step size of 2 After passing through the upsampling unit 7 (including the deconvolution layer with a convolution kernel size of 4×4 and a sliding step size of 2, the batch normalization layer and the RELU activation function), the output image is 64×64 in size and 8 in feature number, which is jump-connected with the output image of the downsampling unit 3 and concatenated along the feature dimension into an image of 64×64 in size and 16 in feature number, which enters the upsampling unit 7; after passing through the upsampling unit 7 (including the deconvolution layer with a convolution kernel size of 4×4 and a sliding step size of 2, the batch normalization layer and the RELU activation function), the output image is 128×128 in size and 4 in feature number, which is jump-connected with the output image of the downsampling unit 2 and concatenated along the feature dimension into an image of 128×128 in size and 8 in feature number, which enters the upsampling unit 8; after passing through the upsampling unit 8 (including the convolution kernel size of 4×4 and a sliding step size of 2 After the deconvolution layer, batch normalization layer and RELU activation function), the image with a size of 256×256 and a feature number of 4 is output. This image is jump-connected with the output image of the downsampling unit 1, and is spliced ​​along the feature dimension into an image with a size of 256×256 and a feature number of 8, which enters the upsampling unit 9; after the upsampling unit 9 (including the deconvolution layer with a convolution kernel size of 4×4 and a sliding step of 2, the batch normalization layer and the RELU activation function), the image with a size of 512×512 and a feature number of 4 enters the readout layer; the readout layer (including the convolution layer with a convolution kernel size of 1×1 and the sigmoid function) finally outputs an image with a size of 512×512 and a feature number of 1, which is the imaging result finally output by the imaging neural network. .

[0057] It should be noted that the imaging neural network of this embodiment has made several optimizations based on the characteristics of the passive non-line-of-sight imaging problem. The main innovations are as follows:

[0058] a) Based on the traditional U-net network, the depth is increased, and 9 layers of downsampling units and 9 layers of upsampling units are used in the encoder and decoder respectively, increasing the model complexity and parameter space. According to the physical characteristics of the imaging problem, the input, output dimensions and feature number of each layer in the neural network are adjusted, while ensuring that the neural network has high imaging quality and fast training speed.

[0059] b) Different from the traditional U-net network that uses the maximum pooling layer for downsampling, the imaging neural network in the present invention directly achieves downsampling through a convolutional layer with a sliding step size of 2, and uses a convolutional layer with learnable parameters for downsampling to achieve simultaneous feature extraction and translation.

[0060] c) LeakyRELU is used as the activation function in the encoder to replace the RELU function to improve the gradient flow, which is suitable for training deeper networks while still maintaining its advantage of simple gradient operations. The RELU activation function is still used in the decoder, and a certain degree of neuron death is deliberately introduced to keep the expansion path network sparse, thereby suppressing overfitting and making the model have faster calculation speed and generalization ability.

[0061] d) Before each layer of convolutional network output passes through the activation function, a batch normalization operation is performed. This operation first translates and scales the network output to a normalized form with a mean of 0 and a variance of 1, and then introduces two learnable parameters to translate and scale the normalized output in a small range. This operation can improve the stability of the neural network training process, help reduce the network's sensitivity to initialization, and increase the allowable range of the learning rate, thereby accelerating model convergence and improving training speed.

[0062] e) Use Dropout regularization in some upsampling units. Dropout is a regularization method used to suppress overfitting of neural networks. By randomly dropping neurons with a certain probability during training, the neurons are not involved in the current calculation and parameter update, thereby reducing the network's dependence on specific neurons, improving the generalization ability of the network at this layer, and making the training process more robust.

[0063] 1-4) Using the training set obtained in step 1-2), train the imaging neural network established in step 1-3).

[0064] In this embodiment, the Adam optimizer is used to train the imaging neural network. The parameters of the Adam optimizer are β1=0.9, β2=0.99, and the weight decay rate is 2×10-5 , select mean square error MSE as the loss function in training, and the initial learning rate is 2.5×10 -4 , the training cycle is not less than 100, and the imaging quality improves with the increase of the training cycle. After the training is completed, the final imaging neural network is obtained.

[0065] Figure 4 This is a diagram showing the effect of imaging neural network training according to a specific embodiment of the present invention. Figure 4 The peak signal-to-noise ratio (PSNR) between the output of the imaging neural network and the target true value varies with the number of training cycles. After training, the highest PSNR of the imaging neural network test set reaches more than 18 dB, indicating that the imaging result in this embodiment is very close to the true value image.

[0066] 2)Testing phase.

[0067] 2-1) Obtain a test image.

[0068] In this embodiment, the test image and the true value image of the training set have the same features (handwritten numbers, handwritten letters, human body movements, etc.).

[0069] In a specific embodiment of the present invention, the test image is an MNIST handwritten digital image with a size of 28×28. 2-2) In the imaging scene constructed in step 1), the LED screen displays the test image of step 2-1), and the light is transmitted to the intermediate wall to form a corresponding original scattered image, which is captured by a camera.

[0070] The collected original scattering image is sampled to the input size set by the imaging neural network, which is 512×512 in this embodiment, to obtain the final scattering image.

[0071] 2-3) The final scattering image obtained in step 2-2) is input into the trained imaging neural network, and the network outputs the imaging result corresponding to the scattering image, thereby realizing the passive non-line-of-sight imaging process.

[0072] In a specific embodiment of the present invention, the output result of the imaging neural network is compared and analyzed with the true value image displayed on the LED screen. The imaging result of this embodiment is as follows: Figure 5 As shown, Figure 5 (a) and Figure 5In (b), the left column is the scattered images on the intermediate wall directly captured by the camera, the middle column is the imaging result output after the scattered images are input into the imaging neural network, and the right column is the true value image displayed on the LED screen. It can be seen that the method described in this embodiment can restore the scattered images to true value images with high quality, achieving the passive non-field of view imaging target. After specific testing, it was found that the resolution of the two-dimensional image imaged by the method described in this embodiment was less than 1 cm, the imaging speed was greater than 5 frames per second, the peak signal-to-noise ratio was greater than 15 dB, the recognition accuracy was more than 90%, and the imaging effect was excellent.

[0073] To implement the above embodiment, a second aspect of the present invention provides a passive non-line-of-sight imaging device based on deep learning, comprising:

[0074] A scattered image acquisition module is used to use a camera to acquire a scattered image generated by a projection image projected by a self-luminous object on an intermediate wall in a passive non-line-of-sight imaging scene;

[0075] An imaging module is used to input the scattered image into a preset imaging neural network, and the imaging neural network outputs the imaging result of the projection image; wherein the imaging neural network is composed of a pair of encoders and decoders with symmetrical structures; the encoder is composed of 9 layers of downsampling units connected in sequence, and the decoder is composed of 9 layers of upsampling units connected in sequence.

[0076] It should be noted that the aforementioned explanation of the embodiment of a passive non-line-of-sight imaging method based on deep learning is also applicable to a passive non-line-of-sight imaging device based on deep learning in this embodiment, and will not be repeated here. A passive non-line-of-sight imaging device based on deep learning proposed in an embodiment of the present invention uses a camera to capture a scattered image generated by a projection image projected by a self-luminous object on an intermediate wall in a passive non-line-of-sight imaging scene; the scattered image is input into a preset imaging neural network, and the imaging neural network outputs an imaging result of the projection image; wherein the imaging neural network is composed of a pair of encoders and decoders with a symmetrical structure; the encoder is composed of 9 layers of downsampling units connected in sequence, and the decoder is composed of 9 layers of upsampling units connected in sequence.

[0077] This makes it possible to use deep learning to achieve passive non-line-of-sight imaging without any special restrictions on the imaging scene, and to increase the imaging distance, thereby greatly improving the versatility of passive non-line-of-sight imaging.

[0078] To implement the above embodiment, a third aspect of the present invention provides an electronic device, including:

[0079] at least one processor; and a memory communicatively coupled to the at least one processor;

[0080] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above-mentioned passive non-line-of-sight imaging method based on deep learning.

[0081] To implement the above-mentioned embodiment, the fourth aspect of the present invention proposes a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned passive non-line-of-sight imaging method based on deep learning.

[0082] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0083] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device executes a deep learning-based passive non-line-of-sight imaging method of the above embodiment.

[0084] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0085] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0086] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0087] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0088] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0089] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0090] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0091] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0092] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A passive non-line-of-sight imaging method based on deep learning, characterized in that: include: In the passive non-line-of-sight imaging scenario, a camera is used to collect the scattered image generated by the projection image projected by the self-luminous object on the intermediate wall; The scattering image is input into a preset imaging neural network, and the imaging neural network outputs the imaging result of the projection image; wherein the imaging neural network is composed of a pair of encoders and decoders with symmetrical structures; the encoder is composed of 9 layers of downsampling units connected in sequence, and the decoder is composed of 9 layers of upsampling units connected in sequence.

2. The method according to claim 1, characterized in that The projected image is a square binary image with a pixel number less than or equal to the maximum pixel number of the self-luminous object.

3. The method according to claim 1, characterized in that Before inputting the scattering image into a preset imaging neural network, the method further includes: The scatter image is sampled to an input size set by the imaging neural network.

4. The method according to claim 1, characterized in that: Each downsampling unit of the encoder includes a two-dimensional convolution layer for downsampling, a batch normalization layer for controlling the stability of internal parameters of the model, and an activation function for introducing nonlinearity; each upsampling unit of the decoder includes a two-dimensional deconvolution layer for upsampling, a batch normalization layer for controlling the stability of internal parameters of the model, a jump connection for enhancing feature fusion, and an activation function for introducing nonlinearity.

5. The method according to claim 4, characterized in that Also includes: The imaging neural network initially inputs a scattering image of size 512×512, and after passing through the first downsampling unit, an image of size 256×256 and feature number 4 is output and enters the second downsampling unit; after passing through the second downsampling unit, an image of size 128×128 and feature number 4 is output and enters the third downsampling unit; after passing through the third downsampling unit, an image of size 64×64 and feature number 8 is output and enters the fourth downsampling unit; after passing through the fourth downsampling unit, an image of size 32×32 and feature number 16 is output and enters the fifth downsampling unit; after passing through the fifth downsampling unit, an image of size 16×16 and feature number 32 is output and enters the sixth downsampling unit; after passing through the sixth downsampling unit, an image of size 8×8 and feature number 32 is output and enters the seventh downsampling unit; after passing through the seventh downsampling unit, an image of size 4×4 and feature number 32 is output and enters the eighth downsampling unit; after passing through the eighth downsampling unit, an image of size 2×2 and feature number 32 is output and enters the ninth downsampling unit; after passing through the ninth downsampling unit, an image of size 1 ×1, the image with a feature number of 32 enters the first upsampling unit; after passing through the first upsampling unit, an image with a size of 2×2 and a feature number of 32 is output, and the image is jump-connected with the image output by the eighth downsampling unit, and spliced ​​along the feature dimension into an image with a size of 2×2 and a feature number of 64 to enter the second upsampling unit; after passing through the second upsampling unit, an image with a size of 4×4 and a feature number of 32 is output, and the image is jump-connected with the image output by the seventh downsampling unit, and spliced ​​along the feature dimension into an image with a size of 4×4 and a feature number of 64 to enter the third upsampling unit; after passing through the third upsampling unit, an image with a size of 8×8 and a feature number of 32 is output, and the image is jump-connected with the image output by the sixth downsampling unit, and spliced ​​along the feature dimension into an image with a size of 8×8 and a feature number of 64 to enter the fourth upsampling unit; after passing through the fourth upsampling unit, an image with a size of 16×16 and a feature number of 32 is output, and the image is jump-connected with the image output by the fifth downsampling unit, and spliced ​​along the feature dimension into an image with a size of 16×16 and a feature number of 64 to enter the fifth upsampling unit; After passing through the fifth upsampling unit, an image with a size of 32×32 and a feature number of 16 is output, and this image is jump-connected with the image output from the fourth downsampling unit, and is spliced ​​along the feature dimension into an image with a size of 32×32 and a feature number of 32 to enter the sixth upsampling unit; after passing through the sixth upsampling unit, an image with a size of 64×64 and a feature number of 8 is output, and this image is jump-connected with the image output from the third downsampling unit, and is spliced ​​along the feature dimension into an image with a size of 64×64 and a feature number of 16 to enter the seventh upsampling unit; after passing through the seventh upsampling unit, an image with a size of 128×128 and a feature number of 4 is output, and this image is jump-connected with the image output from the second downsampling unit, and is spliced ​​along the feature dimension into an image with a size of 64×64 and a feature number of 16 to enter the seventh upsampling unit. The output image is jump-connected and spliced ​​along the feature dimension into an image with a size of 128×128 and a feature number of 8 to enter the eighth up-sampling unit; after passing through the eighth up-sampling unit, an image with a size of 256×256 and a feature number of 4 is output, and this image is jump-connected with the image output by the first down-sampling unit, and spliced ​​along the feature dimension into an image with a size of 256×256 and a feature number of 8 to enter the ninth up-sampling unit; after passing through the ninth up-sampling unit, an image with a size of 512×512 and a feature number of 4 is output and enters the readout layer of the imaging neural network; the readout layer outputs an image with a size of 512×512 and a feature number of 1, which is the final imaging result.

6. The method according to claim 5, characterized in that Also includes: Each downsampling unit includes a convolution layer with a convolution kernel size of 4×4 and a sliding step size of 2 and a LeakyRELU activation function; wherein the second downsampling unit to the eighth downsampling unit also include a batch normalization layer; Each upsampling unit layer includes a deconvolution layer with a convolution kernel size of 4×4 and a sliding step size of 2, a batch normalization layer, and a RELU activation function; wherein, the first upsampling unit to the third upsampling unit also use a dropout with a probability of 0.5; The readout layer includes a convolution layer with a convolution kernel size of 1×1 and a sigmoid function.

7. The method according to claim 6, characterized in that Before inputting the scattering image into a preset imaging neural network, the method further includes: training the imaging neural network; The training of the imaging neural network comprises: 1) In the passive non-line-of-sight imaging scene, the self-luminous object is used to project a plurality of true value images, and a scattered image generated by each true value image on the intermediate wall surface is acquired through the camera; wherein the plurality of true value images have the same features; 2) sampling each true value image and the corresponding scattering image to a size set by the imaging neural network to form a training sample, wherein the input of the training sample is the scattering image after the size adjustment, and the true value label is the corresponding true value image after the size adjustment; all training samples constitute a training set; 3) constructing the imaging neural network; 4) Using the training set obtained in step 2) to train the imaging neural network, to obtain the trained imaging neural network.

8. A passive non-line-of-sight imaging device based on deep learning, characterized in that: include: A scattered image acquisition module is used to use a camera to acquire a scattered image generated by a projection image projected by a self-luminous object on an intermediate wall in a passive non-line-of-sight imaging scene; An imaging module is used to input the scattered image into a preset imaging neural network, and the imaging neural network outputs the imaging result of the projection image; wherein the imaging neural network is composed of a pair of encoders and decoders with symmetrical structures; the encoder is composed of 9 layers of downsampling units connected in sequence, and the decoder is composed of 9 layers of upsampling units connected in sequence.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 7.

Images