Active steady-state non-vision field imaging method based on conditional generative adversarial network

Abstract

The invention discloses an active steady-state non-vision field imaging method based on a conditional generative adversarial network. The method comprises the following steps: constructing an active steady-state non-vision field imaging device; establishing an improved conditional generative adversarial network and an overall loss function thereof; non-vision field images and specular reflection images of different hidden objects under different poses are obtained through the device, and after preprocessing, the images are input into the network for training optimization until the total loss function converges to obtain an optimized network; and inputting the non-vision-field image of the hidden object to be imaged into the optimization network for processing to obtain a reconstructed image, thereby realizing active steady-state non-vision-field imaging. The method promotes the capture of position-related intensity information in the non-vision-field image, improves the spatial perception capability, relieves information degradation in the image coding process, retains key features in the original non-vision-field information, improves the visual quality and detail representation of the reconstructed image, and has the advantages of simple imaging setting, convenient deployment, high resolution and the like. The reconstruction speed is high; and the generalization performance is good.

Description

Technical Field

[0001] The present invention relates to an active steady-state non-line-of-sight imaging method, and in particular to an active steady-state non-line-of-sight imaging method based on a conditional generative adversarial network. Background Art

[0002] Non-line-of-sight imaging technology overcomes the difficulty of traditional optical imaging in effectively detecting targets outside the line of sight, such as those hidden behind corners or scatterers, and effectively expands the detection range of traditional imaging systems. It is expected to be applied in fields such as autonomous driving, disaster relief, and medical examinations. In recent years, many researchers have explored non-line-of-sight imaging methods based on different detection principles and have made positive progress.

[0003] The transient non-line-of-sight imaging technology based on time resolution uses the flight time information of three-bounce photons, and has obvious advantages in the three-dimensional reconstruction of hidden scenes; the non-line-of-sight imaging technology based on optical speckle coherence can achieve the reconstruction of small objects in the non-line-of-sight scene or the tracking of small displacements by analyzing the speckle pattern generated by the coherent light source illuminating the hidden scene. However, the above technologies have certain limitations. For example, the transient non-line-of-sight imaging technology based on time resolution has the disadvantages of complex hardware system, high cost, long acquisition time, and low photon efficiency; the non-line-of-sight imaging technology based on optical speckle coherence uses the memory effect and has a limited viewing angle, which limits its application in macroscopic scenes such as autonomous driving.

[0004] In contrast, steady-state non-line-of-sight imaging systems use continuous active light sources or ambient light as the main optical excitation and conventional steady-state optical sensors for image acquisition. This approach has the advantages of simple system composition, high sampling efficiency, low cost, and easy implementation and deployment. However, compared with transient non-line-of-sight imaging, steady-state non-line-of-sight imaging cannot obtain the flight time information of photons, which brings challenges to recovering hidden targets based only on intensity information.

[0005] Recently, deep learning has demonstrated excellent performance in the field of steady-state non-line-of-sight imaging. Driven by data, deep learning models the mapping from non-line-of-sight measurements to the hidden scene to be reconstructed and generates prior information for the solution of the inverse problem. It has been proven that it can be used to locate, reconstruct and identify hidden targets. However, most of the existing deep learning-based steady-state non-line-of-sight imaging technologies are concentrated in the field of passive imaging, because passive imaging either relies on ambient light or scene self-luminescence, the scene setting is simple and easy to implement, and the signal attenuation is relatively low due to the small number of photon bounces, which reduces the difficulty of solving to a certain extent. Active imaging usually requires the use of active excitation light sources. The light needs to undergo at least three diffuse reflections from emission to reception by the sensor, and the signal attenuation is severe. Unlike passive imaging, it is challenging to obtain the true value suitable for supervised neural network learning. Although non-line-of-sight imaging technology has shown great potential in expanding the detection of targets beyond the line of sight and has made certain progress in different application scenarios, the existing methods still have significant shortcomings in terms of system complexity, cost, imaging accuracy and applicability. Especially in active steady-state imaging scenarios, how to obtain reliable supervision data and perform high-quality reconstruction based on intensity information remains a difficult problem that needs to be solved urgently. Summary of the invention

[0006] In order to solve the problems existing in the background technology, the present invention provides an active steady-state non-viewing-area imaging method based on conditional generative adversarial network. The method of the present invention has the advantages of simple imaging setting, convenient deployment, fast reconstruction speed and good generalization performance.

[0007] The technical solution adopted by the present invention is:

[0008] The active steady-state non-viewing-area imaging method based on a conditional generative adversarial network of the present invention comprises:

[0009] 1) Build an active steady-state non-line-of-sight imaging device.

[0010] 2) The optical transmission characteristics and data features of active steady-state non-line-of-sight imaging are used to establish an improved conditional generative adversarial network and its overall loss function and install it in an active steady-state non-line-of-sight imaging device.

[0011] 3) Obtain non-field of view images and mirror reflection images of different hidden objects in different postures through an active steady-state non-field of view imaging device, and input them into the improved conditional generative adversarial network for training and optimization after preprocessing until the overall loss function converges to obtain a trained optimized improved conditional generative adversarial network.

[0012] 4) The non-line-of-sight image of the hidden object to be imaged is input into the optimized and improved conditional generative adversarial network for processing to obtain a reconstructed image and display it, thereby realizing active steady-state non-line-of-sight imaging of the hidden object. The optimized and improved conditional generative adversarial network can quickly reconstruct the binary image of the corresponding target for the non-line-of-sight signal input of an unknown shape target or an unknown posture target.

[0013] In the step 1), the active steady-state non-line-of-sight imaging device includes a relay wall, a rotating table, a robotic arm, an obstruction, a camera, a computer, a laser and a controller. The relay wall is vertically installed on the rotating table. The rotating table rotates so that the relay wall rotates around its own vertical central axis. The robotic arm, the obstruction and the camera are arranged in sequence on one side of the relay wall. The robotic arm grabs a hidden object. The obstruction is located between the hidden object and the camera and blocks the hidden object so that the hidden object is invisible. The camera used to capture the non-line-of-sight image and the mirror reflection image of the hidden object faces the wall of the relay wall on its side, and the laser used to provide active lighting faces the wall of the relay wall on the side of the hidden object. The rotating table is connected to the controller, the camera and the controller are electrically connected to the computer used to control the rotational movement of the relay wall and the camera image acquisition, the robotic arm is wirelessly connected to the computer, and the improved conditional generative adversarial network is installed in the computer.

[0014] In the step 2), the improved conditional generative adversarial network includes an improved generator and a discriminator, the improved generator includes a basic network and an initial feature extraction branch IFEB (Initial Feature Extraction Branch), the discriminator is the original discriminator of the conditional generative adversarial network, the preprocessed non-viewing area image Y of the hidden object is respectively input into the basic network and the initial feature extraction branch IFEB for processing, the output after processing by the initial feature extraction branch IFEB is then input into the basic network for processing, and the basic network outputs the reconstructed image G(Y) and the complementary image after processing The reconstructed image G(Y) is the binary morphology estimation of the hidden object. The improved generator adopts a U-shaped structure and introduces an initial feature extraction flow outside the basic network, which alleviates the information degradation in the image encoding process and retains the key features of the original non-viewing information. The preprocessed mirror reflection image X is used as a supervision signal of the improved generator to guide the improved generator to generate a reconstructed image G(Y) as close to X as possible. The preprocessed mirror reflection image X and the preprocessed non-viewing image Y are spliced ​​in the channel dimension to obtain a first splicing result. The preprocessed non-viewing image Y and the reconstructed image G(Y) are spliced ​​in the channel dimension to obtain a second splicing result. The first splicing result and the second splicing result are then input into the discriminator for processing and identification, and the improved conditional generative adversarial network is trained through the overall loss function.

[0015] The basic network includes a first 3×3 convolution layer Conv, a first batch of normalized BN layers, a first leaky linear rectifier unit LReLU activation function, six downsampling blocks DSB (Downsampling Block), a first dense block DB (Dense Block), a first coordinate attention block CAB (Coordinate Attention Block), a first transition block TB (Transport Block), six upsampling blocks USB (Upsampling Block), a second 1×1 convolution layer Conv and a binary classification Softmax activation function, each downsampling block DSB includes a second dense block DB, a second coordinate attention block CAB, a second transition block TB and a first maximum pooling MP (MaxPool) layer connected in sequence, and each upsampling block USB includes a deep convolution layer DeConv, a third transition block TB and a third dense block DB connected in sequence; the initial feature extraction branch IFEB includes five pooling feature MF (MP&FEB) modules connected in sequence, and the pooling feature MF module includes a second maximum pooling MP layer and a feature extraction block FEB (Feature Extraction Block); the input of the generator is respectively input into the first 3×3 convolutional layer Conv of the basic network and the second maximum pooling MP layer of the first pooling feature MF module of the initial feature extraction branch IFEB for processing; the output after processing by the second maximum pooling MP layer of each pooling feature MF module is respectively input into the feature extraction block FEB of the current pooling feature MF module and the second maximum pooling MP layer of the next pooling feature MF module for processing; the output after processing by the feature extraction block FEB of the five pooling feature MF modules is respectively concatenated with the output after processing by the second coordinate attention block CAB in the second to sixth down-sampling blocks DSB in the channel dimension and then input into the second transition block TB of the current down-sampling block DSB for processing; the output after processing by the second coordinate attention block CAB of the sixth to the next down-sampling blocks DSB is respectively concatenated with the output after processing by the deep convolution layer DeConv of the first to sixth up-sampling blocks USB in the channel dimension and then input into the third transition block TB of the current up-sampling block USB for processing; and finally the output after processing by the binary classification Softmax activation function is used as the output of the generator.

[0016] The dense block DB includes three dense layers connected in sequence, each dense layer includes a second batch normalization BN layer, a second leaky linear rectifier unit LReLU activation function and a third 3×3 convolutional layer Conv connected in sequence, the input of the dense block DB is first input into the first dense layer for processing, the output after processing by the first dense layer and the input of the dense block DB are input into the second dense layer for processing together, the input of the dense block DB, the output of the first dense layer and the second dense layer are input into the third dense layer for processing together, and the input of the dense block DB and the output of the three dense layers are output together as the output of the dense block DB; the transition block TB includes a third batch normalization BN layer, a third leaky linear rectifier unit LReLU activation function and a fourth 1×1 convolutional layer Conv connected in sequence.

[0017] The feature extraction block FEB includes a fifth 3×3 convolutional layer Conv, a fourth batch normalization BN layer, a fourth leaky linear rectifier unit LReLU activation function, a sixth 3×3 convolutional layer Conv, a fifth batch normalization BN layer and a third coordinate attention block CAB, which are connected in sequence.

[0018] The activation function in the coordinate attention block CAB adopts the leaky linear rectifier unit LReLU activation function.

[0019] In the step 2), the overall loss function L is as follows:

[0020]

[0021] Among them, G and D represent the improved generator and discriminator respectively; L cGAN () represents the joint confrontation loss term, and the joint confrontation loss L cGAN The improved generator is forced to continuously improve its output to ensure that the reconstructed image G(Y) is as similar to its true value as possible visually. By improving the adversarial learning between the generator and the discriminator, the visual quality and detail performance of the non-viewing domain reconstructed image G(Y) are effectively improved; λ1 and λ2 represent the binary cross entropy loss term L bce () and the region overlap loss term L dice () weight parameter, binary cross entropy loss L bce It helps to improve the generator to adjust details at the local pixel level, thereby improving the structure and detail accuracy of the reconstruction results; regional overlap loss L dice By introducing regional consistency constraints, the structural features of the generated image are improved, more spatial details are restored, and the accuracy of reconstruction is enhanced; It represents the expectation that the discriminator judges it as a real sample under the joint distribution of the preprocessed specular reflection image X and its corresponding non-viewing image Y; It means that the improved generator generates the reconstructed image G(Y) by inputting the preprocessed non-viewing image Y. The discriminator judges it as a false expectation, and the preprocessed specular reflection image X, that is, the specular reflection binary image of the hidden object, is the true value image. The overall loss function L enables the improved generator to have both overall perception ability and local detail recovery ability.

[0022] In the step 3), for each position of each hidden object, the side of the relay wall facing the hidden object and the camera is used as a reflection surface, and the hidden object on one side of the occluder is moved to the current position by the computer-controlled mechanical arm of the active steady-state non-line-of-sight imaging device. The light beam emitted by the laser is reflected to the hidden object through one side of the relay wall, and a second reflection occurs on the surface of the hidden object, and then returns to the relay wall again and reflects for the third time on one side of the relay wall. Finally, it is captured by the camera on the other side of the occluder to obtain a non-line-of-sight image, and then transmitted to the computer for preprocessing; the computer controls the rotation of the rotating table through the controller, so that the relay wall rotates 180° around its own vertical central axis. At this time, the other side of the relay wall facing the hidden object and the camera is used as a reflection surface, and the third reflection occurrence area of ​​the other side of the relay wall is a mirror surface, and then the mirror reflection image is captured by the camera. The non-line-of-sight image is captured when the third reflection occurrence area is a conventional diffuse reflection plane, and the mirror reflection image is captured when the third reflection occurrence area is replaced by a mirror surface.

[0023] In the step 3), the non-viewing image and the mirror reflection image of the hidden object are preprocessed, specifically, the non-viewing image is subjected to background difference, mean filtering denoising, cropping and resizing operations in sequence to obtain a denoised non-viewing image, and the mirror reflection image is subjected to background difference, cropping, resizing and threshold segmentation operations in sequence to obtain a binary pattern of the denoised mirror reflection image, and the cropping is specifically uniformly cropped into a number of identical square images.

[0024] The beneficial effects of the present invention are:

[0025] 1) Simple imaging setup: Compared with the traditional transient non-line-of-sight imaging technology, the present invention adopts a steady-state imaging method, avoids complex and expensive hardware systems, simplifies imaging setup, and reduces system costs.

[0026] 2) Fast reconstruction speed: Through adversarial training of the generative network, the present invention can quickly reconstruct the two-dimensional image of the hidden object and achieve efficient non-field of view imaging.

[0027] 3) Excellent generalization performance: By utilizing adversarial learning between the generator and the discriminator, the model of the present invention can maintain good reconstruction effects in non-visual scenes with different target shapes and postures, showing good generalization ability.

[0028] 4) Accurate detail restoration: By introducing initial feature extraction, position-sensitive coordinate attention mechanism and region overlap loss, the present invention can more accurately restore the details and structural features of hidden targets, especially in terms of target contour and posture, thereby improving the visual quality and spatial perception ability of non-field of view imaging.

[0029] 5) Easy to deploy: The system of the present invention is simple in design and suitable for a variety of application scenarios.

[0030] In summary, the improved conditional generative adversarial network of the method of the present invention promotes the capture of position-related intensity information in non-field of view images and improves spatial perception ability by incorporating a position-sensitive coordinate attention mechanism; by introducing an initial feature extraction branch, it alleviates information degradation in the image encoding process and retains key features in the original non-field of view information; through adversarial learning between the generator and the discriminator, it effectively improves the visual quality and detail expression of the non-field of view reconstructed image. The method has the advantages of simple imaging setup, easy deployment, fast reconstruction speed and good generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a schematic diagram of an active steady-state non-line-of-sight imaging device and method in an embodiment of the present invention;

[0032] Figure 2 : is a schematic diagram of the setting of the relay wall in an embodiment of the present invention, wherein: Figure 2 (a) is a schematic diagram of the arrangement of a conventional diffuse reflection surface on one side of a relay wall when collecting non-line-of-sight signals in an embodiment of the present invention, Figure 2 (b) is a schematic diagram of the arrangement of a conventional diffuse reflection surface and a mirror surface spliced ​​on the other side of the relay wall when collecting the mirror reflection (true value) signal in an embodiment of the present invention;

[0033] Figure 3 2 is a schematic diagram of the structure of an improved conditional generative adversarial network in an embodiment of the present invention;

[0034] Figure 4 : is a graph of non-viewing area image reconstruction results on three test sets by the method of the present invention, the scattering imaging method based on U-Net, and the Pix2pix GAN (Generative Adversarial Networks) method for image conversion in an embodiment of the present invention, wherein: Figure 4 (a) is the non-viewing image reconstruction result of the three methods on dataset I. Figure 4 (b) is the non-viewing image reconstruction result of the three methods on dataset II. Figure 4 (c) is the non-viewing area image reconstruction result of the three methods on dataset III;

[0035] In the figure: 1. Relay wall, 2. Rotating table, 3. Robotic arm, 4. Hidden object, 5. Obstruction, 6. Camera, 7. Computer, 8. Laser, 9. Controller. DETAILED DESCRIPTION

[0036] The technical solution provided by the present invention is described in detail below in conjunction with specific embodiments. The following embodiments are limited to explaining the present invention, but are not intended to be limiting of the present invention. The protection scope of the invention should include the entire contents of the claims. All similar implementations that achieve the same effects by substantially the same methods fall within the protection scope of the present invention. Through the following embodiments, those skilled in the art can realize the entire contents of the claims of the present invention.

[0037] The embodiment of the present invention uses a complementary metal oxide semiconductor CMOS (Complementary Metal Oxide Semiconductor) camera 6 with a resolution of 1920×1080 to capture the non-viewing signal of the hidden object 4 and its mirror reflection signal, and the horizontal field of view angle is 76 degrees. Active illumination uses a laser 8 with a wavelength of 532.22nm, a power fluctuation of 0.437%, and a 74mW laser source. At the same time, 38 common characters are produced using 3D printing technology, including #, $, (, +, 1, 2, 3, 4, 5, 6, 7, a, A, b, B, C, d, D, e, E, f, g, G, I, J, K, L, N, O, R, S, T, U, V, X, y, Y and Z, which are used to obtain training and test data, and the robot arm 3 is controlled to grab the characters and realize spatial transformation in 50 predefined postures.

[0038] like Figure 1 As shown, the active steady-state non-viewing-area imaging method based on conditional generative adversarial network of the present invention is specifically as follows:

[0039] 1) Building an active steady-state non-line-of-sight imaging device, the active steady-state non-line-of-sight imaging device includes a relay wall 1, a rotating platform 2, a robotic arm 3, an obstruction 5, a camera 6, a computer 7, a laser 8 and a controller 9. The relay wall 1 is vertically installed on the rotating platform 2. The rotating platform 2 rotates so that the relay wall 1 rotates around its own vertical central axis. The robotic arm 3, the obstruction 5 and the camera 6 are arranged on one side of the relay wall 1 in sequence. The robotic arm 3 grabs a hidden object 4. The obstruction 5 is located between the hidden object 4 and the camera 6 and blocks the hidden object 4 so that the hidden object 4 is invisible. The camera 6 for capturing the non-line-of-sight image and the mirror reflection image of the hidden object 4 faces the wall of the relay wall 1 on its side, and the laser 8 for providing active lighting faces the wall of the relay wall 1 on the side of the hidden object 4; the rotating platform 2 is connected to the controller 9, the camera 6 and the controller 9 are electrically connected to the computer 7 for controlling the rotation of the relay wall 1 and the image acquisition of the camera 6, the robotic arm 3 is wirelessly connected to the computer 7, and the improved conditional generative adversarial network is installed in the computer 7.

[0040] like Figure 1 , Figure 2 (a) and Figure 2 As shown in (b), one side of the relay wall 1 in the embodiment of the present invention is made of polymethyl methacrylate using a frosting process, which serves as a conventional diffuse reflection surface; the other side is made of polymethyl methacrylate using a frosting process and polymethyl methacrylate using a coating process, wherein the part using the frosting process serves as a conventional diffuse reflection surface, which is the laser illumination area, and the part using the coating process serves as a mirror, which is the receiving area for the third reflection; by controlling the relay wall 1 to flip 180° along the vertical central axis, the non-field of view image and the mirror reflection image of the hidden target can be collected respectively, the non-field of view image is captured when the third reflection receiving area is a conventional diffuse reflection plane, and the corresponding mirror reflection image is captured when the third reflection receiving area is a mirror.

[0041] In active steady-state non-line-of-sight imaging, the change in the relative position between the small blocks on the relay wall 1 (corresponding to each pixel and the hidden object 4) will lead to a difference in intensity distribution. Based on this feature, the physical prior of spatially dependent intensity change is incorporated into the network design. By introducing the coordinate attention CA mechanism, the position-related intensity information is encoded into the extracted features, thereby enhancing the network's ability to capture position-related intensity information, enabling the network to accurately capture the specific information of the region of interest, and effectively improving the detail fidelity and quality of non-line-of-sight reconstruction.

[0042] 2) Optical transmission characteristics and data characteristics of active steady-state non-line-of-sight imaging Establish an improved conditional generative adversarial network and its overall loss function and install it in the active steady-state non-line-of-sight imaging device. Figure 3As shown, the improved conditional generative adversarial network includes an improved generator and a discriminator. The improved generator includes a basic network and an initial feature extraction branch IFEB. The discriminator is the original discriminator of the conditional generative adversarial network. The preprocessed non-viewing area image Y of the hidden object 4 with a size of 256×256×1 is respectively input into the basic network and the initial feature extraction branch IFEB for processing. The output after processing by the initial feature extraction branch IFEB is then input into the basic network for processing. After processing, the basic network outputs a reconstructed image G(Y) with a size of 256×256×1 and a complementary image with a size of 256×256×1. The reconstructed image G(Y) is the binary morphology estimation of the hidden object 4. The improved generator adopts a U-shaped structure and introduces an initial feature extraction flow outside the basic network, which alleviates the information degradation in the image encoding process and retains the key features in the original non-viewing information. The preprocessed mirror reflection image X is used as a supervision signal of the improved generator to guide the improved generator to generate a reconstructed image G(Y) as close to X as possible. The preprocessed mirror reflection image X and the preprocessed non-viewing image Y are spliced ​​in the channel dimension to obtain a first splicing result. The preprocessed non-viewing image Y and the reconstructed image G(Y) are spliced ​​in the channel dimension to obtain a second splicing result. The first splicing result and the second splicing result are then input into the discriminator for processing and identification, and the improved conditional generative adversarial network is trained through the overall loss function.

[0043] The discriminator includes a 3×3 convolutional layer Conv, a leaky linear rectifier unit LReLU (LeakyRectified Linear Unit) activation function, a 3×3 convolutional layer Conv, a batch normalization BN (Batch Normalization) layer, a leaky linear rectifier unit LReLU activation function, a 3×3 convolutional layer Conv, a batch normalization BN layer, a dropout layer, a 3×3 convolutional layer Conv and a logical sigmoid activation function. The output of the discriminator is a probability map of size 256×256×1. Each probability map corresponds to a small block in the reconstructed image G(Y). It can discriminate and evaluate the local area in the reconstructed image G(Y), so that the improved generator pays more attention to the local consistency and high-frequency details of the reconstructed image G(Y), thereby improving the accuracy of the hidden object 4 detail recovery. The input size of the discriminator is 256×256×2, and the output size becomes 256×256×64, 256×256×128, 256×256×256 and 256×256×1 after being processed by two leaky linear rectifier units LReLU, a dropout layer and a logistic sigmoid activation function, respectively.

[0044] The basic network includes a first 3×3 convolutional layer Conv, a first batch of normalized BN layers, a first leaky linear rectifier unit LReLU activation function, six downsampling blocks DSB, a first dense block DB, a first coordinate attention block CAB, a first transition block TB, six upsampling blocks USB, a second 1×1 convolutional layer Conv and a binary classification Softmax activation function, each downsampling block DSB includes a second dense block DB, a second coordinate attention block CAB, a second transition block TB and a first maximum pooling MP layer connected in sequence, and each upsampling block USB includes a deep convolutional layer DeConv, a third transition block TB and a third dense block DB connected in sequence; the initial feature extraction branch IFEB includes five pooling feature MF modules connected in sequence, and the pooling feature MF module includes a second maximum pooling MP layer and a feature extraction block FEB connected in sequence; the input of the generator is respectively input to the first 3×3 convolutional layer Conv and the initial feature extraction branch IFEB of the basic network The outputs processed by the second maximum pooling MP layer of the first pooling feature MF module are respectively input into the feature extraction block FEB of the current pooling feature MF module and the second maximum pooling MP layer of the next pooling feature MF module for processing. The outputs processed by the feature extraction block FEB of the five pooling feature MF modules are respectively spliced ​​with the outputs processed by the second coordinate attention blocks CAB in the second to sixth downsampling blocks DSB in the channel dimension and then input into the second transition block TB of the current downsampling block DSB for processing. The outputs processed by the second coordinate attention blocks CAB of the sixth to sixth downsampling blocks DSB are respectively spliced ​​with the outputs processed by the deep convolution layer DeConv of the first to sixth upsampling blocks USB in the channel dimension and then input into the third transition block TB of the current upsampling block USB for processing. Finally, the output processed by the binary classification Softmax activation function is used as the output of the generator. The input of the generator is processed by the first downsampling block DSB, the second downsampling block DSB, the fifth downsampling block DSB, the sixth downsampling block DSB, the first transition block TB, the first upsampling block USB, the fourth upsampling block USB, the fifth upsampling block USB, the sixth upsampling block USB and the binary classification Softmax activation function, and the output size becomes 128×128×72, 64×64×108, 8×8×216, 4×4×252, 4×4×288, 8×8×252, 64×64×144, 128×128×108, 256×256×72 and 256×256×2.

[0045] The characteristics of non-line-of-sight optical transmission and data are that the value of a single pixel, i.e., the light intensity, has significant spatial dependence during non-line-of-sight imaging. This phenomenon can be described by the equation of the last reflection:

[0046]

[0047] Wherein, Φ(W) represents the light intensity corresponding to a small block W on the relay wall 1, and w represents the unit area on a small block W on the relay wall 1; Ω l and Ω v They represent the solid angles formed by the incident light and the outgoing light respectively; Represents the bidirectional reflectance distribution BRDF (Bidirectional Reflectance Distribution Function) function, and Represent the incident, outgoing and normal unit vectors of light respectively; represents the radiance of the incident light. From the reflection equation, it can be seen that the change in the relative position between each pixel and the hidden object 4 corresponding to the small block on the relay wall 1 will lead to a difference in intensity distribution. According to this feature, the physical prior of spatially dependent intensity change is incorporated into the network design. By introducing the coordinate attention CA mechanism, the position-related intensity information is encoded into the extracted features, thereby enhancing the model's ability to capture position-related intensity information, enabling the model to accurately capture the specific information of the area of ​​interest, and effectively improving the detail fidelity and quality of non-field of view reconstruction.

[0048] The dense block DB includes three dense layers connected in sequence, each dense layer includes a second batch normalization BN layer, a second leaky linear rectifier unit LReLU activation function and a third 3×3 convolutional layer Conv connected in sequence, the input of the dense block DB is first input into the first dense layer for processing, the output after processing by the first dense layer and the input of the dense block DB are input into the second dense layer for processing together, the input of the dense block DB, the output of the first dense layer and the second dense layer are input into the third dense layer for processing together, and the input of the dense block DB and the output of the three dense layers are output together as the output of the dense block DB; the transition block TB includes a third batch normalization BN layer, a third leaky linear rectifier unit LReLU activation function and a fourth 1×1 convolutional layer Conv connected in sequence.

[0049] The feature extraction block FEB includes a fifth 3×3 convolutional layer Conv, a fourth batch normalization BN layer, a fourth leaky linear rectifier unit LReLU activation function, a sixth 3×3 convolutional layer Conv, a fifth batch normalization BN layer and a third coordinate attention block CAB, which are connected in sequence.

[0050] The activation function in the coordinate attention block CAB uses the leaky linear rectifier unit LReLU activation function. The coordinate attention block CAB includes an X-axis average pooling layer AvgPool, a Y-axis average pooling layer AvgPool, a 1×1 convolution layer Conv, a batch normalization BN layer, a leaky linear rectifier unit LReLU activation function, and two convolution activation modules. The convolution activation module includes a 1×1 convolution layer Conv and a logical Sigmoid activation function connected in sequence. The coordinate attention block CAB first inputs the X-axis average pooling layer AvgPool and the Y-axis average pooling layer AvgPool for processing, and the X-axis average pooling layer AvgPool and the Y-axis average pooling layer AvgPool are processed. The output processed by vgPool is concatenated in the length / width dimensions and then input into the 1×1 convolution layer Conv, the batch normalization BN layer, and the leaky linear rectifier unit LReLU activation function in sequence for processing. The output processed by the leaky linear rectifier unit LReLU activation function is divided and split and then input into two convolution activation modules for processing. The output processed by the two convolution activation modules re-weights the initial input of the coordinate attention block CAB, and the re-weighted output is used as the output of the coordinate attention block CAB.

[0051] The overall loss function L is as follows:

[0052]

[0053] Among them, G and D represent the improved generator and discriminator respectively; L cGAN () represents the joint confrontation loss term, and the joint confrontation loss L cGAN The improved generator is forced to continuously improve its output to ensure that the reconstructed image G(Y) is as similar to its true value as possible visually. By improving the adversarial learning between the generator and the discriminator, the visual quality and detail performance of the non-viewing domain reconstructed image G(Y) are effectively improved; λ1 and λ2 represent the binary cross entropy loss term L bce () and the region overlap loss term L dice () weight parameter, binary cross entropy loss L bce It helps to improve the generator to adjust details at the local pixel level, thereby improving the structure and detail accuracy of the reconstruction results; regional overlap loss L dice By introducing regional consistency constraints, the structural features of the generated image are improved, more spatial details are restored, and the accuracy of reconstruction is enhanced; It represents the expectation that the discriminator judges it as a real sample under the joint distribution of the preprocessed specular reflection image X and its corresponding non-viewing image Y; It means that the improved generator generates the reconstructed image G(Y) by inputting the preprocessed non-viewing image Y, and the discriminator judges it as a false expectation; the preprocessed specular reflection image X is the specular reflection binary image of the hidden object 4, which is the true value image. The overall loss function L enables the improved generator to have both overall perception ability and local detail recovery ability.

[0054] 3) Obtain non-viewing images and mirror reflection images of different hidden objects 4 at different positions through an active steady-state non-viewing imaging device, and input them into an improved conditional generative adversarial network for training optimization after preprocessing, until the overall loss function converges, and obtain a trained optimized improved conditional generative adversarial network; for each position of each hidden object 4, at this time, a side of the relay wall 1 facing the hidden object 4 and the camera 6 is used as a reflecting surface, and the computer 7 of the active steady-state non-viewing imaging device controls the mechanical arm 3 to move the hidden object 4 on the side of the obstruction 5 to the current position, and the light beam emitted by the laser 8 passes through one side of the relay wall 1. The light is reflected from the side to the hidden object 4, and then reflected for the second time on the surface of the hidden object 4. Then, it returns to the relay wall 1 again and is reflected for the third time on one side of the relay wall 1. Finally, it is captured by the camera 6 on the other side of the obstruction 5 to obtain a non-viewing image, which is then transmitted to the computer 7 for preprocessing; the computer 7 controls the rotation of the rotating table 2 through the controller 9, so that the relay wall 1 rotates 180° around its own vertical central axis. At this time, the other side of the relay wall 1 facing the hidden object 4 and the camera 6 is used as a reflection surface, and the third reflection area on the other side of the relay wall 1 is a mirror surface, and then the mirror reflection image is captured by the camera 6. The non-viewing image is captured when the third reflection area is a conventional diffuse reflection plane, and the mirror reflection image is captured when the third reflection area is replaced by a mirror surface.

[0055] The non-viewing area image and the mirror reflection image of the hidden object 4 are preprocessed, specifically, the non-viewing area image is subjected to background difference, mean filtering denoising, cropping and resizing operations in sequence to obtain a denoised non-viewing area image, and the mirror reflection image is subjected to background difference, cropping, resizing and threshold segmentation operations in sequence to obtain a binary mode of the denoised mirror reflection image, and the cropping is specifically uniformly cropped into several identical square images. The embodiment of the present invention preprocesses the captured image, and the preprocessed non-viewing area image Y is a grayscale image of 256×256 size, and the preprocessed mirror reflection image X is a binary image of 256×256 size.

[0056] 4) The non-line-of-sight image of the hidden object 4 to be imaged is input into the optimized and improved conditional generative adversarial network for processing to obtain a reconstructed image and display it, thereby realizing active steady-state non-line-of-sight imaging of the hidden object 4. The optimized and improved conditional generative adversarial network can quickly reconstruct a binary image of the corresponding target for the non-line-of-sight signal input of an unknown shape target or an unknown posture target.

[0057] In addition, the embodiment of the present invention divides the data set into three types (i.e., unknown shape, unknown posture, unknown shape and unknown posture) for evaluation respectively, thereby verifying the performance of the method of the present invention under different conditions. The specific division of the data set is shown in Table 1:

[0058] Table 1

[0059]

[0060] The neural network training in the embodiment of the present invention is implemented based on the PyTorch framework. The neural network is trained for 50 rounds and the batch size is set to 8. For the improved generator, the adaptive moment estimation Adam (Adaptive Moment Estimation) optimizer with β1=0.5 and β2=0.999 is used to update the weights; for the discriminator, the stochastic gradient descent SGD (Stochastic Gradient Descent) optimizer with a momentum of 0.95 is used to update the weights. The learning rate of the improved generator is set to 2×10 -4 , the decay rate is 0.5 every 10 rounds; the learning rate of the discriminator is set to 1×10 -3 , the decay rate is 0.5 every 5 rounds.

[0061] like Figure 4 (a) Figure 4 (b) and Figure 4As shown in (c), the reconstruction results of the non-viewing image Y of the improved conditional generative adversarial network, the scattered imaging method based on U-Net and the Pix2pix GAN method for image conversion proposed in the present invention on datasets I, II and III are shown, wherein the first row represents the non-viewing image Y, the second row represents the corresponding specular reflection (true value) image X, the third row represents the reconstruction effect of the method of the present invention, the fourth row represents the reconstruction effect of the scattered imaging method based on U-Net, and the fifth row represents the reconstruction effect of the Pix2pix GAN method for image conversion. The results show that compared with the comparison method, the method of the present invention achieves more accurate restoration, and the reconstruction result is more consistent with the hidden object 4, especially in terms of the contour and posture of the target. In addition, we also quantitatively evaluate the reconstruction effects of different methods based on the Pearson Correlation Coefficient PCC (Pearson Correlation Coefficient), Jaccard Index JI (Jaccard Index) and Hausdorff Distance HD (Hausdorff Distance) indicators. Among them, the Pearson correlation coefficient PCC is used to measure the similarity between the reconstructed image and the real image. The closer the value is to 1, the stronger the positive correlation between the two images. The Jaccard coefficient JI is used to evaluate the overlap between the reconstructed target and the real target. The closer the value is to 1, the higher the overlap. The Hausdorff distance HD is used to measure the maximum difference between the boundary or contour of the reconstructed target and the real target. The smaller the value is, the closer the boundary of the reconstructed target is to the boundary of the real target. The specific quantitative evaluation results are shown in Table 2:

[0062] Table 2

[0063]

[0064] Larger Pearson correlation coefficient PCC and Jaccard index JI values ​​indicate improved performance, while smaller Hausdorff distance HD values ​​indicate enhanced performance. Compared with the U-Net-based scatter imaging and pix2pix GAN methods, the proposed method achieves the best results in all indicators.

[0065] In addition, the embodiments of the present invention respectively conduct a comprehensive quantitative analysis of the influence of the position-sensitive coordinate attention mechanism CA, the initial feature extraction branch IFEB and the conditional generative adversarial model on the quality of non-view reconstruction. The specific evaluation results are shown in Table 3:

[0066] Table 3

[0067]

[0068]

[0069] It can be seen from the results in Table 3 that the method of the present invention has achieved the best results in all the above indicators, which further verifies the excellent performance of the method.

[0070] The method of the present invention promotes the capture of position-related intensity information in the non-viewing field image Y by incorporating the position-sensitive coordinate attention CA mechanism, thereby improving the spatial perception ability; by introducing the initial feature extraction branch IFEB, the information degradation in the image encoding process is alleviated, and the key features in the original non-viewing field information are retained; through the adversarial learning between the generator and the discriminator, the visual quality and detail performance of the non-viewing field reconstructed image are effectively improved. The method has the advantages of simple imaging setting, convenient deployment, fast reconstruction speed and good generalization performance.

Claims

1. An active steady-state non-line-of-sight imaging method based on conditional generative adversarial network, characterized in that: include: 1) Build an active steady-state non-line-of-sight imaging device; 2) Establish an improved conditional generative adversarial network and its overall loss function and install it in an active steady-state non-line-of-sight imaging device; 3) Obtaining non-viewing images and specular reflection images of different hidden objects (4) at different positions through an active steady-state non-viewing imaging device, and inputting them into an improved conditional generative adversarial network for training and optimization after preprocessing until the overall loss function converges, thereby obtaining a trained optimized improved conditional generative adversarial network; 4) The non-viewing-field image of the hidden object (4) to be imaged is input into the optimized improved conditional generative adversarial network for processing, and a reconstructed image is obtained and displayed, thereby realizing active steady-state non-viewing-field imaging of the hidden object (4).

2. The active steady-state non-line-of-sight imaging method based on conditional generative adversarial network according to claim 1, characterized in that: In the step 1), the active steady-state non-line-of-sight imaging device comprises a relay wall (1), a rotating platform (2), a mechanical arm (3), a shielding object (5), a camera (6), a computer (7), a laser (8) and a controller (9), wherein the relay wall (1) is vertically mounted on the rotating platform (2), and the rotating platform (2) rotates so that the relay wall (1) rotates around its own vertical central axis, the mechanical arm (3), the shielding object (5) and the camera (6) are arranged in sequence on one side of the relay wall (1), the mechanical arm (3) grasps a hidden object (4), and the shielding object (5) is located between the hidden object (4) and the camera (6) and shields the hidden object (4). The hidden object (4) is made invisible by the hidden object (4); the camera (6) for capturing the non-visual field image and the mirror reflection image of the hidden object (4) is facing the wall surface of the relay wall (1) on the side of the hidden object (4); the laser (8) for providing active lighting is facing the wall surface of the relay wall (1) on the side of the hidden object (4); the rotating platform (2) is connected to the controller (9); the camera (6) and the controller (9) are electrically connected to a computer (7) for controlling the rotation movement of the relay wall (1) and the image acquisition of the camera (6); the mechanical arm (3) is wirelessly connected to the computer (7); and the improved conditional generative adversarial network is installed in the computer (7).

3. The active steady-state non-line-of-sight imaging method based on conditional generative adversarial network according to claim 1, characterized in that: In the step 2), the improved conditional generative adversarial network includes an improved generator and a discriminator, the improved generator includes a basic network and an initial feature extraction branch IFEB, the discriminator is the original discriminator of the conditional generative adversarial network, the preprocessed non-field of view image Y of the hidden object (4) is respectively input into the basic network and the initial feature extraction branch IFEB for processing, the output after processing by the initial feature extraction branch IFEB is then input into the basic network for processing, the basic network outputs a reconstructed image G(Y) after processing, the preprocessed mirror reflection image X is used as a supervisory signal for training the improved generator, the preprocessed mirror reflection image X and the preprocessed non-field of view image Y are spliced ​​in the channel dimension to obtain a first splicing result, the preprocessed non-field of view image Y and the reconstructed image G(Y) are spliced ​​in the channel dimension to obtain a second splicing result, and then the first splicing result and the second splicing result are input into the discriminator in turn for processing and identification, and the improved conditional generative adversarial network is trained by the overall loss function.

4. The active steady-state non-line-of-sight imaging method based on conditional generative adversarial network according to claim 3, characterized in that: The basic network includes a first convolutional layer Conv, a first batch of normalized BN layers, a first leaky linear rectifier unit LReLU activation function, six downsampling blocks DSB, a first dense block DB, a first coordinate attention block CAB, a first transition block TB, six upsampling blocks USB, a second convolutional layer Conv and a binary classification Softmax activation function connected in sequence, each downsampling block DSB includes a second dense block DB, a second coordinate attention block CAB, a second transition block TB and a first maximum pooling MP layer connected in sequence, each upsampling block USB includes a deep convolutional layer DeConv, a third transition block TB and a third dense block DB connected in sequence; the initial feature extraction branch IFEB includes five pooling feature MF modules connected in sequence, and the pooling feature MF module includes a second maximum pooling MP layer and a feature extraction block FEB connected in sequence; the input of the generator is respectively input to the first convolutional layer Conv of the basic network and the first pool of the initial feature extraction branch IFEB The outputs processed by the second maximum pooling MP layer of the pooling feature MF module are respectively input into the feature extraction block FEB of the current pooling feature MF module and the second maximum pooling MP layer of the next pooling feature MF module for processing. The outputs processed by the feature extraction block FEB of the five pooling feature MF modules are respectively spliced ​​with the outputs processed by the second coordinate attention blocks CAB in the second to sixth downsampling blocks DSB in the channel dimension and then input into the second transition block TB of the current downsampling block DSB for processing. The outputs processed by the second coordinate attention blocks CAB of the sixth to sixth downsampling blocks DSB are respectively spliced ​​with the outputs processed by the deep convolution layers DeConv of the first to sixth upsampling blocks USB in the channel dimension and then input into the third transition block TB of the current upsampling block USB for processing. Finally, the output processed by the binary Softmax activation function is used as the output of the generator.

5. The active steady-state non-line-of-sight imaging method based on conditional generative adversarial network according to claim 4, characterized in that: The dense block DB includes three dense layers connected in sequence, each dense layer includes a second batch normalization BN layer, a second leaky linear rectifier unit LReLU activation function and a third convolutional layer Conv connected in sequence, the input of the dense block DB is first input into the first dense layer for processing, the output after processing by the first dense layer and the input of the dense block DB are input into the second dense layer for processing together, the input of the dense block DB, the output of the first dense layer and the second dense layer are input into the third dense layer for processing together, and the input of the dense block DB and the output of the three dense layers are output together as the output of the dense block DB; the transition block TB includes a third batch normalization BN layer, a third leaky linear rectifier unit LReLU activation function and a fourth convolutional layer Conv connected in sequence.

6. The active steady-state non-line-of-sight imaging method based on conditional generative adversarial network according to claim 4, characterized in that: The feature extraction block FEB includes a fifth convolutional layer Conv, a fourth batch normalization BN layer, a fourth leaky linear rectifier unit LReLU activation function, a sixth convolutional layer Conv, a fifth batch normalization BN layer and a third coordinate attention block CAB, which are connected in sequence.

7. The active steady-state non-line-of-sight imaging method based on conditional generative adversarial network according to claim 4, characterized in that: The activation function in the coordinate attention block CAB adopts the leaky linear rectifier unit LReLU activation function.

8. The active steady-state non-line-of-sight imaging method based on conditional generative adversarial network according to claim 1, characterized in that: In the step 2), the overall loss function L is as follows: L bce (G)=-XlogG(Y)-(1-X)log(1-G(Y)) Among them, G and D represent the improved generator and discriminator respectively; L cGAN () represents the joint adversarial loss term; λ1 and λ2 represent the binary cross entropy loss term L bce () and the region overlap loss term L dice () weight parameter; It represents the expectation that the discriminator judges it as a real sample under the joint distribution of the preprocessed specular reflection image X and its corresponding non-viewing image Y; It represents the expectation that the improved generator generates the reconstructed image G(Y) by inputting the preprocessed non-viewing image Y, and the discriminator judges it to be false.

9. The active steady-state non-line-of-sight imaging method based on conditional generative adversarial network according to claim 2, characterized in that: In the step 3), for each position of each hidden object (4), the side of the relay wall (1) facing the hidden object (4) and the camera (6) is used as a reflection surface, and the computer (7) of the active steady-state non-field of view imaging device controls the mechanical arm (3) to move the hidden object (4) on the side of the obstruction (5) to the current position, and the light beam emitted by the laser (8) is reflected to the hidden object (4) through one side of the relay wall (1), and is reflected for the second time on the surface of the hidden object (4), and then returns to the relay wall (1) and is reflected on the surface of the relay wall (1). After the third reflection occurs on one side, the non-viewing image is finally captured by the camera (6) on the other side of the obstruction (5), and then transmitted to the computer (7) for preprocessing; the computer (7) controls the rotation of the rotating table (2) through the controller (9), so that the relay wall (1) rotates 180 degrees around its own vertical center axis. At this time, the other side of the relay wall (1) facing the hidden object (4) and the camera (6) is used as a reflection surface, and the area where the third reflection occurs on the other side of the relay wall (1) is a mirror surface, and then the mirror reflection image is captured by the camera (6).

10. The active steady-state non-line-of-sight imaging method based on conditional generative adversarial network according to claim 1, characterized in that: In the step 3), the non-viewing image and the mirror reflection image of the hidden object (4) are preprocessed, specifically, the non-viewing image is subjected to background difference, mean filtering denoising, cropping and resizing operations in sequence to obtain a denoised non-viewing image, and the mirror reflection image is subjected to background difference, cropping, resizing and threshold segmentation operations in sequence to obtain a binary mode of the denoised mirror reflection image.

Images