Non-vision-field real-time imaging system and imaging method based on multiband information fusion

Abstract

The invention discloses a non-vision-field real-time imaging system and method based on multiband information fusion. The non-vision-field real-time imaging system comprises an information collection unit, a calculation processing unit and an application terminal. The information acquisition unit is used for controlling an acquisition position, acquiring optical information of a hidden imaging target on a relay surface, and pushing the optical information to the calculation processing unit in a video stream mode; the calculation processing unit is used for receiving the video stream, processing the video stream and outputting a real-time reconstruction result of the hidden imaging target; and the application terminal checks and analyzes the real-time reconstruction result through the online interactive application. According to the real-time imaging system and the imaging method, the infrared light image and the visible light image can be processed at the same time, information of the infrared light image and the visible light image is effectively fused, a real-time reconstruction result with high correlation with a hidden imaging target is obtained, the hidden imaging target in a non-vision field can be accurately restored, and a high-quality imaging result is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of non-line-of-sight imaging, and in particular, to a non-line-of-sight real-time imaging system and imaging method based on multi-band information fusion. Background Art

[0002] In reconnaissance missions, due to the complex spatial conditions and various obstacles in the environment, or in anti-terrorism and rescue missions where rescue personnel cannot directly observe the target to be rescued, it brings certain difficulties and risks to the members carrying out the mission. If the images of the targets and scenes behind the obstacles can be obtained quickly and accurately, the mission success rate can be greatly improved and the safety of personnel can be guaranteed. Therefore, there is an urgent need for a method that can "bend around corners" to image, so as to expand the sensing range and image hidden imaging targets and scenes.

[0003] Non-line-of-sight imaging technology (Non-Line-of-Sight Imaging) can use the weak and sparse signals on the relay plane to reconstruct hidden targets, and can break through the limit of traditional visual perception to realize the detection and reconstruction of targets behind obstacles. The existing non-line-of-sight imaging technologies are divided into active non-line-of-sight imaging technology and passive non-line-of-sight imaging technology according to whether there is an encodable active light source.

[0004] The existing active non-line-of-sight imaging technologies mainly rely on high-frequency modulated pulsed lasers and ultrafast imaging devices, such as intensified charge coupled devices (ICCD), single photon avalanche diodes (SPAD), streak cameras, and time-of-flight (ToF) cameras. These technologies can not only obtain the intensity information of light, but also obtain the information in the time dimension.

[0005] Compared with the active non-line-of-sight imaging technology, most of the existing passive non-line-of-sight imaging technologies use consumer-grade visible light cameras and only rely on the scene light source or the ambient light reflected by the object. There are also a small number of them that use infrared cameras to achieve "bending around corners" imaging through the thermal infrared information of the target. Since it does not require collecting photon information, the passive non-line-of-sight imaging technology has stronger real-time performance and is easier to be applied in practice.

[0006] However, whether it is an active or passive method, existing non-line-of-sight imaging technologies are limited to using information sources in a single band. However, due to the limitations of a single data source, current imaging methods face problems such as difficult parameter estimation, high computational time complexity, high cost of optical components, few constraints, and low signal-to-noise ratio. Moreover, the complexity of the real physical world and the dynamics of obstacles make the single-band perception data have problems such as high information interference and feature omission, making it even more difficult to be actually put into use. Summary of the Invention

[0007] The main problem to be solved by the present invention is to provide a non-line-of-sight real-time imaging system and imaging method based on multi-band information fusion, which solves the problems in existing non-line-of-sight imaging technologies, such as short imaging distance, low imaging resolution, long imaging time, and high cost of optical components due to a single visible light band, making it difficult to be actually put into use.

[0008] To solve the above technical problems, a technical solution adopted by the present invention is to provide a non-line-of-sight real-time imaging system based on multi-band information fusion, including: an information acquisition unit, a calculation and processing unit, and an application terminal;

[0009] The information acquisition unit includes an unmanned platform carrying a multi-band sensor cluster. The unmanned platform is used to control the acquisition position of the multi-band sensor cluster, and the multi-band sensor cluster is used to collect the optical information of a hidden imaging target on the relay surface and push the optical information to the calculation and processing unit in the form of a video stream;

[0010] The calculation and processing unit is used to receive the video stream, process the video stream, and output the real-time reconstruction result of the hidden imaging target;

[0011] The application terminal views and analyzes the real-time reconstruction result through an online interactive application.

[0012] In some embodiments, the multi-band sensor cluster includes an infrared light sensor and a visible light sensor;

[0013] The infrared light sensor is used to regard the hidden imaging target as a heat source, and the infrared light sensor captures the heat source through the surface reflection of the relay surface;

[0014] The visible light sensor is used to capture the weak light signal reflected by the hidden imaging target through the relay surface. The weak light signal is generated by the self-luminescence of the hidden imaging target or the reflection of ambient light.

[0015] In some embodiments, the calculation and processing unit includes a preprocessing module and a reconstruction processing module;

[0016] The preprocessing module uses a spatio-temporal registration method to align and match the data in different bands of the video stream, and then inputs the preprocessing result obtained after processing into the reconstruction processing module;

[0017] The reconstruction processing module receives the preprocessing result for processing and outputs the real-time reconstruction result.

[0018] In some embodiments, the spatio-temporal registration method includes:

[0019] First, frame alignment is performed on the video stream, and format and resolution processing are performed on the distortion and data calibration problems existing in the synchronized multi-band video stream based on the diverse frame rates of the video streams in different bands;

[0020] Then, the data in the infrared and visible light bands are respectively subjected to infrared frame preprocessing and visible light preprocessing to obtain the preprocessing result.

[0021] In some embodiments, the reconstruction processing module includes a passive non-line-of-sight imaging network for multi-band information fusion, and the passive non-line-of-sight imaging network includes an encoder, an attention fusion module, and a decoder;

[0022] The encoder receives the preprocessing result, and the preprocessing result includes a visible light image and an infrared light image; there are two groups of encoders, which respectively process the visible light image and the infrared light image in multiple sizes, and send the two groups of processing results to the attention fusion module and the decoder;

[0023] The attention fusion module fuses and splices the two groups of processing results and inputs them into the decoder;

[0024] The decoder fuses the reconstruction results of the visible light image and the infrared light image into the real-time reconstruction result.

[0025] In some embodiments, a hybrid loss function is adopted in the passive non-line-of-sight imaging network, which is expressed as:

[0026] Loss total =Loss pixel +λLoss SSIM ,

[0027] wherein, it includes a content loss function Loss pixel and a contour loss function Loss SSIM ;

[0028] The content loss function Loss pixel is:

[0029] The contour loss function Loss SSIMis: Loss SSIM = 1 - SSIM(I, Out);

[0030] where V i is the visible light image, Out is the real-time reconstruction result, H and W are the height and width of the visible light image respectively; I is the infrared light image, and SSIM is the structural similarity function between the real-time reconstruction result and the infrared light image; λ is a hyperparameter used to reduce the interference of the infrared loss on the color loss.

[0031] In some embodiments, in the hybrid loss function, λ = 0.1.

[0032] The present invention also provides a non-line-of-sight real-time imaging method based on multi-band information fusion, including the following steps:

[0033] Collect the optical information of the hidden imaging target on the relay plane;

[0034] Convert the collected optical information into a video stream;

[0035] Process the video stream to obtain the real-time reconstruction result of the hidden imaging target.

[0036] In some embodiments, in the step of collecting the optical information of the hidden imaging target on the relay plane, it includes simultaneously capturing the infrared signal and the visible light signal reflected by the relay plane.

[0037] In some embodiments, it further includes obtaining the transformation relationship between the hidden imaging target and the projection image:

[0038] Represent the projection image as a first transformation formula:

[0039]

[0040] where I(p y ) is the light intensity of the projection image; I(p f ) is the light intensity of the hidden imaging target; A(p f , p y ) is the light transmission matrix from the hidden imaging target to the projection image; F represents all point light sources of the reflecting surface; is the ambient light and the noise of the multi-band sensor cluster;

[0041] Discretize the first transformation formula into a second transformation formula:

[0042] y = A(μ, r, c)f + n b+d ,

[0043] Among them, f is the hidden imaging target, y is the projected image, and A is determined by the bidirectional reflection distribution function μ of the relay surface, the distance r between the hidden imaging target and the relay surface, and the position c of the multi-band sensor cluster;

[0044] Let the corresponding transformation from the hidden imaging target to the projected image be T μ,r,c,n , then the third transformation formula is obtained:

[0045] Among them, is the inverse of T μ,r,c,n ;

[0046] Combining the second transformation formula and the third transformation formula, T μ,r,c,n is obtained.

[0047] The beneficial effects of the present invention are as follows: The present invention discloses a non-line-of-sight real-time imaging system and imaging method based on multi-band information fusion, including: an information acquisition unit, a calculation and processing unit, and an application terminal; the information acquisition unit includes an unmanned carrier platform and a multi-band sensor cluster, the unmanned carrier platform carries the multi-band sensor cluster, the unmanned carrier platform is used to control the acquisition position of the multi-band sensor cluster, and the multi-band sensor cluster is used to collect the optical information of the hidden imaging target on the relay surface and push the optical information to the calculation and processing unit in the form of a video stream; the calculation and processing unit is used to receive the video stream, process the video stream, and output the real-time reconstruction result of the hidden imaging target; the application terminal views and analyzes the real-time reconstruction result through an online interactive application. This real-time imaging system and imaging method can simultaneously process infrared light images and visible light images, effectively fuse the information of the two, obtain a real-time reconstruction result with high correlation with the hidden imaging target, and can accurately restore the hidden imaging target within the non-line-of-sight, providing high-quality imaging results. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a block diagram of an embodiment of a non-line-of-sight real-time imaging system based on multi-band information fusion according to the present invention;

[0049] Figure 2 is a schematic diagram of an embodiment of a non-line-of-sight real-time imaging system based on multi-band information fusion according to the present invention;

[0050] Figure 3 is a schematic diagram of the composition of the calculation and processing unit in an embodiment of a non-line-of-sight real-time imaging system based on multi-band information fusion according to the present invention;

[0051] Figure 4 is a schematic diagram of the structure of a passive non-line-of-sight imaging network in an embodiment of a non-line-of-sight real-time imaging system based on multi-band information fusion according to the present invention;

[0052] Figure 5 It is the data flow diagram of the first attention fusion module in an embodiment of a non-line-of-sight real-time imaging system based on multi-band information fusion according to the present invention;

[0053] Figure 6 It is the flow chart of an embodiment of a non-line-of-sight real-time imaging method based on multi-band information fusion according to the present invention. Detailed implementation manners

[0054] To facilitate the understanding of the present invention, the present invention will be described in more detail below in conjunction with the accompanying drawings and specific embodiments. The preferred embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure content of the present invention more thorough and comprehensive.

[0055] It should be noted that unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not used to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0056] As Figure 1 shown, a non-line-of-sight real-time imaging system based on multi-band information fusion according to the present invention is shown, including: an information acquisition unit 1 and a calculation and processing unit 2.

[0057] The information acquisition unit 1 is used to collect the optical information of the hidden imaging target on the relay surface in real time and convert the optical information into a video stream. Among them, the relay surface is used to reflect the optical information of the hidden imaging target into the field of view acceptance range of the information acquisition unit 1, and at the same time, the real-time projection image of the hidden imaging target on the relay surface can be displayed. The above optical information is contained in the real-time projection image, and there is a specific correspondence between the real-time projection image and the hidden imaging target.

[0058] Specifically, the information acquisition unit 1 includes a multi-band sensor cluster, and the multi-band sensor cluster can collect the corresponding real-time projection image of the hidden imaging target in real time. The real-time projection image is multi-band optical imaging, including infrared light imaging and visible light imaging.

[0059] Furthermore, the multi-band sensor cluster can be carried on an unmanned platform, and the unmanned platform is used to control the acquisition position of the multi-band sensor cluster to adapt to different scene requirements.

[0060] The computing and processing unit 2 is used to receive a video stream, process the video stream, and output the real-time reconstruction result of the hidden imaging target.

[0061] The imaging system may further include an application terminal 3 for viewing and analyzing the real-time reconstruction result through an online interactive application. Such a real-time imaging system can simultaneously process infrared light imaging and visible light imaging, effectively fuse the information of both, and obtain a real-time reconstruction result with high correlation to the hidden imaging target.

[0062] As Figure 2 shown, an application embodiment may be: the unmanned platform is a robot dog, and other devices such as drones can also be used. A binocular camera is mounted on the robot dog. The binocular camera serves as a multi-band sensor cluster to identify the real-time projection image on the relay surface and collect the optical information of the hidden imaging target contained therein, so that the collected optical information is received by the computing and processing unit in the form of a video stream. After the computing and processing unit performs corresponding processing, a real-time reconstruction result is obtained. Application terminals such as mobile phones and other mobile devices can be used to view and analyze the real-time reconstruction result through an online interactive application.

[0063] Furthermore, the multi-band sensor cluster includes an infrared light sensor and a visible light sensor. The infrared light sensor takes the hidden imaging target as a heat source and captures it through the surface reflection of the relay surface by the infrared light sensor. The visible light sensor is used to capture the weak light signal reflected by the hidden imaging target through the relay surface. The weak light signal is generated by the self-luminescence of the hidden imaging target or by reflecting ambient light. These light signals obtained by the multi-band sensor cluster after being reflected by the relay surface are the collected real-time projection images, including infrared light imaging captured by the infrared light sensor and visible light imaging captured by the visible light sensor.

[0064] As Figure 3 shown, it shows the module composition and data transmission relationship inside the computing and processing unit 2.

[0065] Furthermore, the computing and processing unit 2 includes a preprocessing module 21 and a reconstruction processing module 22. The preprocessing module 21 uses a spatio-temporal registration method to align and match the data of different bands in the video stream, and then inputs the preprocessing result obtained after processing into the reconstruction processing module 22. The reconstruction processing module 22 receives the preprocessing result for processing and outputs the real-time reconstruction result.

[0066] Specifically, the spatio-temporal registration method includes:

[0067] First, perform frame alignment on the video stream, and perform format and resolution processing on the distortion and data calibration problems existing in the synchronized multi-band video stream based on the diverse frame rates of the video streams of different bands.

[0068] Then, the data in the infrared and visible light bands are respectively subjected to infrared frame preprocessing and visible light frame preprocessing to obtain preprocessing results.

[0069] Specifically, the infrared light video output by the infrared light sensor is 25 frames per second, and the pixels of a single-frame image are 640*512. The visible light video output by the visible light sensor is 30 frames per second, and the pixels of a single-frame image are 1280*720. Frame alignment sets the frame rates of both to 20 frames per second. Since the resolutions are different, the fields of view and imaging sizes of the two sensors are also different. The larger field of view is cropped and calibrated to make them coincide. This step limits the frame rates of both to 20 frames per second so that each frame of the two corresponds respectively. Then, infrared frame preprocessing and visible light frame preprocessing operations are performed on the infrared light video and the visible light video.

[0070] Among them, infrared frame preprocessing and visible light frame preprocessing respectively perform alignment calibration on each frame of the infrared light video and the visible light video, and then perform some processing before inputting to the reconstruction processing module, such as cropping, scaling, and normalizing the data types of the formats of each frame of the video, etc., so that the infrared light video and the visible light video can correspond and coincide. Among them, each frame is a picture. Each frame of the infrared light video is an infrared light image, and each frame of the visible light video is a visible light image. For example: Take the first frame of the infrared light video and the visible light video respectively. Because the camera parameters are different, the angles are slightly deviated, resulting in differences in the size and position of the two images; then perform some processing on the two images so that the two images can correspond and coincide.

[0071] In the present invention, the preprocessing module 21 aligns and matches the data from different bands so that further non-line-of-sight target reconstruction and restoration can be performed. In this step, the spatio-temporal registration method plays a key role in ensuring that the data from different bands are consistent with each other in time and space. Because there may be time offsets and position differences in the data sources of different bands, by aligning and matching the data, the system can more accurately restore non-line-of-sight targets and provide high-quality imaging results.

[0072] Furthermore, the reconstruction processing module 22 includes a passive non-line-of-sight imaging network for multi-band information fusion. The passive non-line-of-sight imaging network receives the preprocessing results for processing and outputs real-time reconstruction results.

[0073] As Figure 4 shown, the passive non-line-of-sight imaging network adopts a network based on deep learning for reconstructing hidden imaging targets. The passive non-line-of-sight imaging network includes three layers of encoders, three layers of decoders, and three layers of attention fusion modules.

[0074] Specifically, the three-layer encoder receives the preprocessing results, which include the visible light image 6 and the infrared light image 5. The three-layer encoder consists of two groups, which process the visible light image 6 and the infrared light image 5 respectively in three sizes, and send the two groups of processing results to the three-layer decoder and the three-layer attention fusion module respectively. The three-layer attention fusion module fuses and splices the two groups of processing results and inputs them into the three-layer decoder. The three-layer decoder fuses the reconstruction results of the visible light image 6 and the infrared light image 5 into the real-time reconstruction result 7 through the same decoder set and upsampling operation.

[0075] As Figure 4 shown, the two groups of three-layer encoders are respectively an infrared light encoder group and a visible light encoder group. The infrared light encoder group includes the first infrared light encoder 51, the second infrared light encoder 52, and the third infrared light encoder 53. The visible light encoder group includes the first visible light encoder 61, the second visible light encoder 62, and the third visible light encoder 63. The three-layer decoder includes the first decoder 71, the second decoder 72, and the third decoder 73. The three-layer attention fusion module includes the first attention fusion module 81, the second attention fusion module 82, and the third attention fusion module 83.

[0076] During the encoding process, by stacking multi-scale inputs, the model receptive field is gradually increased through continuous downsampling. Taking the infrared light image 5 as an example, its features are mainly extracted by the infrared light encoder. In the first infrared light encoder 51, first, the first-scale infrared features with a pixel size of 256*256*32 are convolved by the first infrared light convolution module 50 with a 3*3 convolution; then the first shallow features are extracted by the first infrared light residual block 54; and then the extracted first shallow features are output in three paths. Among them, the first path is output to the third decoder 73 to participate in the decoding process of the third decoder 73. The second path is output to the second infrared light encoder 52 to participate in the encoding process of the second infrared light encoder 52, while retaining the high-scale detailed information and combining the low-scale to better obtain the global information. The third path is output to the first attention fusion module 81.

[0077] In the second infrared light encoder 52, first, the second-scale infrared features with a pixel size of 128*128*64 in the first shallow features are convolved by the second infrared light convolution module 55 with a 3*3 convolution; then the first infrared light feature attention module 56 is used to emphasize or suppress the features output after convolution and highlight the target regions of interest; then the second infrared light residual block 57 is used to perform deep feature extraction to obtain the first deep features; and then the first deep features are output in three paths. Among them, the first path is output to the second decoder 72 to participate in the decoding process of the second decoder 72; the second path is output to the third infrared light encoder 53 to participate in the encoding process of the third infrared light encoder 53; the third path is output to the second attention fusion module 82.

[0078] In the third infrared light encoder 53, first, the third-scale infrared feature with a pixel size of 64*64*128 in the first deep feature is convolved by a 3*3 convolution through the third infrared light convolution module 58; then the second infrared light feature attention module 59 is used to emphasize or suppress the output feature after convolution and highlight the target area of interest; then the third infrared light residual block 90 is used to perform deep feature extraction to obtain the third deep feature; and then the third deep feature is output to the third attention fusion module 83.

[0079] Meanwhile, the visible light image 6 is mainly subjected to feature extraction through the visible light encoder, and its feature extraction process is synchronized with that of the infrared light image 5.

[0080] In the first visible light encoder 61, first, the first-scale visible light feature with a pixel size of 256*256*32 is convolved by a 3*3 convolution through the first visible light convolution module 60; then the second shallow feature is extracted through the first visible light residual block 64; and then the extracted second shallow feature is output in three paths. Among them, the first path is output to the third decoder 73 to participate in the decoding process of the third decoder 73. The second path is output to the second visible light encoder 62 to participate in the encoding process of the second visible light encoder 62, while retaining high-scale detail information and better obtaining global information by combining low scales. The third path is output to the first attention fusion module 81 to be fused with the first shallow feature obtained through the first infrared light encoder 51.

[0081] In the second visible light encoder 62, first, the second-scale visible light feature with a pixel size of 128*128*64 in the second shallow feature is convolved by a 3*3 convolution through the second visible light convolution module 65; then the first visible light feature attention module 66 is used to emphasize or suppress the output feature after convolution and highlight the target area of interest; then the second visible light residual block 67 is used to perform deep feature extraction to obtain the second deep feature; and then the second deep feature is output in three paths. Among them, the first path is output to the second decoder 72 to participate in the decoding process of the second decoder 72; the second path is output to the third visible light encoder 63 to participate in the encoding process of the third visible light encoder 63; the third path is output to the second attention fusion module 82 to be fused with the first deep feature obtained through the second infrared light encoder 52.

[0082] In the third visible light encoder 63, first, the third-scale visible light feature with a pixel size of 64*64*128 in the second deep feature is convolved by a 3*3 convolution through the third visible light convolution module 68; then the second visible light feature attention module 69 is used to emphasize or suppress the output feature after convolution and highlight the target area of interest; then the third visible light residual block 91 is used to perform deep feature extraction to obtain the fourth deep feature; and then the fourth deep feature is output to the third attention fusion module 83 to be fused with the third deep feature obtained through the third infrared light encoder 53.

[0083] As Figure 5 shown, the working principle of the first attention fusion module 81 is shown. According to the basic strategy of providing color information for the visible light image 6 and contour information for the infrared light image 5, the features of the two groups of encoders are fused and stitched together.

[0084] Furthermore, both the first shallow feature A1 and the second shallow feature B1 include 32 channels, and the features within each channel can be represented as a matrix with a height and width of 256, that is, the feature size of each channel is 256*256*1. In the first attention fusion module 81, the features of the same channels in the first shallow feature A1 and the second shallow feature B1 are fused to obtain 32-channel pre-fusion results C1(1), C1(2)...C1(32). Among them, the size of the pre-fusion results of each channel is 256*256*2. Then, three-layer convolution operations are respectively performed on the pre-fusion results of each channel. The convolution kernels in the first convolution 11, the second convolution 12, and the third convolution 13 are set to 3*3*2, 3*3*2, and 3*3*1 respectively. After three-layer convolution, the third feature attention module is used to perform the fourth convolution 14 and fuse it with the result after the third convolution 13, so that the size of the image remains unchanged, but the number of channels changes from 2 to 1, thereby obtaining 32 first intermediate fusion results D1(1), D1(2)...D1(32) with a size of 256*256*1. The 32 first intermediate fusion results are concatenated to obtain a second intermediate fusion result D1 with a size of 256*256*32. Then, the first shallow feature A1 and the second shallow feature B1 are weighted using a long connection to obtain the first fusion result E1. The specific formula is as follows:

[0085] E1 = A1 + B1 + D1

[0086] D1 = For(Cat(D1(i)))

[0087] D1(i) = Conv1(Cat(A1(i),B1(i))) × Conv2(A1(i),B1(i))

[0088] Among them, A1(i) and B1(i) are the features corresponding to the i-th channel in the first shallow feature A1 and the second shallow feature B1 respectively, where i is the number of channels and takes an integer in [1, 32]; For() is a loop function, Cat refers to concatenating two vectors in the channel dimension, Conv1() is a three-layer convolution, and Conv2() is the convolution function in the third feature attention module.

[0089] In this embodiment, the per-channel fusion operation is completed through the splitting and combination of features, and then the original features are weighted using skip connections to prevent feature degradation.

[0090] Furthermore, in the second attention fusion module 82, the features of the same channel in the first deep feature and the second deep feature are fused. The specific operation is similar to that of the first attention fusion module 81 and will not be elaborated here. At the same time, the second fusion result output by the second attention fusion module 82 is added to the first fusion result output by the first attention fusion module 81 to obtain the first-layer feature extraction result, preventing feature degradation. Among them, before adding the first fusion result and the second fusion result, the first fusion result is first passed through a 1*1 convolution to halve the size of the first fusion result and double the number of channels, making it have the same size and number of channels as the second fusion result for convenient addition of corresponding features.

[0091] In the third attention fusion module 83, the features of the same channel in the third deep feature and the fourth deep feature are fused. The specific operation is similar to that of the first attention fusion module 81 and will not be elaborated here. At the same time, the third fusion result output by the third attention fusion module 83 is added to the second fusion result output by the second attention fusion module 82 to obtain the second-layer feature extraction result. Then, the first-layer feature extraction result and the second-layer feature extraction result are added to obtain the pre-reconstruction result, which is input to the first decoder 71. Among them, before adding the second fusion result and the third fusion result, the second fusion result is first passed through a 1*1 convolution to halve the size of the second fusion result and double the number of channels, making it have the same size and number of channels as the third fusion result for convenient addition of corresponding features.

[0092] In the feature fusion part, the output features of each encoder are fused through multiple per-channel attention fusion modules, and per-channel feature fusion is performed on the result of each feature extraction during the encoding process, thereby retaining the linear correlation between different channels, reducing feature degradation, increasing the non-linear expression ability and information utilization rate of the network, and improving the generalization ability of the model.

[0093] During the decoding process, feature reconstruction is performed on the input fused features through downlink channel operations. At the same time, to prevent feature degradation and gradient vanishing problems during the training process, the multi-scale features of the two encoder modules are summed with the corresponding scale features of the decoder module, and finally the reconstruction result is output in an end-to-end manner.

[0094] Specifically, in combination with Figure 4 , the first decoder 71 performs a concat operation on the pre-reconstruction result obtained by adding the first-layer feature extraction result and the second-layer feature extraction result, keeping the size of the pre-reconstruction result unchanged and doubling the number of channels. Then, feature reconstruction is performed through the first reconstruction residual block 70 and the first transposed convolution module 74. At the same time, the first deep feature, the second deep feature, the third deep feature, and the fourth deep feature are respectively summed with the features of the corresponding scales to obtain the first decoding result and input it into the second decoder 72. The second decoder 72 performs convolution, residual, and transposed convolution operations through the fourth convolution module 75, the second reconstruction residual block 76, and the second transposed convolution module 77 for feature reconstruction. At the same time, the first shallow feature, the second shallow feature, the first deep feature, and the second deep feature are respectively summed with the features of the corresponding scales to obtain the second decoding result and input it into the third decoder 73. The number of channels of the third decoder 73 is halved, and convolution and residual operations are performed through the fifth convolution module 78 and the third reconstruction residual block 79 for feature reconstruction. At the same time, the first shallow feature and the second shallow feature are respectively summed with the features of the corresponding scales, changing the number of channels to 3 and restoring the size of the image to obtain the real-time reconstruction feature 92 and further obtain the real-time reconstruction result 7.

[0095] In this embodiment, through the same decoder set and upsampling operation, the real-time reconstruction result 7 of the fusion of the visible light image 6 and the infrared light image 5 is finally output. The task of the decoder is to reconstruct the image from the features extracted by the encoder. In this network, the decoder receives the output of the corresponding encoder and superimposes it with its own input. This design allows the decoder to capture different features of the image at different scales and effectively fuse these features together. During this process, the results of different scales can be combined in different ways according to the task requirements, such as through weighted average, concatenation, or parallel connection, etc.

[0096] This network adopts an innovative structure based on a deep learning network, forming a Y-shaped network. The non-line-of-sight target reconstruction task is performed in a multi-input and multi-scale manner, simultaneously processing the infrared light image 5 and the visible light image 6, and dividing them into multiple different sizes to better capture information at different scales. To effectively fuse this information, the network uses multiple attention fusion modules, which helps to integrate multiple features and improve the network performance.

[0097] Meanwhile, the network structure includes three layers of encoders and three layers of decoders. This symmetric structure helps to maintain information consistency. Each encoder and decoder layer contains residual blocks, which are used to gradually expand the receptive field (i.e., the range of the network's understanding of the input data) through deep feature extraction. The purpose of this structural design is to make more full use of the features and information of multi-band data, especially in complex environments, in order to obtain more accurate and high-quality target images. This deep network structure helps the network learn how to automatically extract and fuse relevant features of multi-band data, thereby improving the performance and quality of non-line-of-sight imaging.

[0098] Furthermore, a hybrid loss function is adopted in the passive non-line-of-sight imaging network, which can be expressed as:

[0099] Loss total = Loss pixel + λLoss SSIM ,

[0100] wherein, it includes content loss function Loss pixel and contour loss function Loss SSIM .

[0101] The content loss function Loss pixel is:

[0102] The contour loss function Loss SSIM is: Loss SSIM = 1 - SSIM(I, Out);

[0103] wherein, V i is the visible light image, Out is the real-time reconstruction result, H and W are respectively the height and width of the visible light image; I is the infrared light image, SSIM is the structural similarity function between the real-time reconstruction result and the infrared light image; λ is a hyperparameter used to reduce the interference of infrared loss on color loss.

[0104] Furthermore, in the hybrid loss function, λ = 0.1.

[0105] It can be seen from the definition of the SSIM formula that in the loss calculation, the SSIM formula is also processed at the pixel level. Therefore, the hyperparameter λ is introduced in the hybrid loss function to reduce the interference of infrared loss on color loss. Through repeated experiments, it is proved that the network performance is the best when λ = 0.1.

[0106] Based on the same inventive concept, Figure 6 is a flowchart of an embodiment of a non-line-of-sight real-time imaging method based on multi-band information fusion according to the present invention, including the following steps:

[0107] S1: Collect the optical information of the hidden imaging target on the relay surface.

[0108] Among them, it includes simultaneously capturing the infrared signal and the visible light signal reflected by the relay surface.

[0109] Specifically, take the hidden imaging target as a heat source and capture the infrared signal reflected by the surface of the relay surface. Capture the weak light signal reflected by the hidden imaging target through the relay surface. The weak light signal is generated by the self-luminescence of the hidden imaging target or by reflecting ambient light, and belongs to the visible light signal.

[0110] S2: Convert the collected optical information into a video stream.

[0111] Among them, the video stream includes the infrared light video corresponding to the infrared signal and the visible light video corresponding to the weak light signal.

[0112] S3: Process the video stream to obtain the real-time reconstruction result of the hidden imaging target.

[0113] Among them, perform feature extraction processing on each frame of the infrared light video and the visible light video respectively, and then perform fusion reconstruction to obtain a real-time reconstruction result with high correlation with the hidden imaging target.

[0114] Furthermore, this real-time imaging method further includes the step of obtaining the transformation relationship from the hidden imaging target to the projection image, where the projection image is multi-band light imaging, including the infrared light image in the infrared light video and the visible light image in the visible light video. Specifically, it includes:

[0115] Express the projection image as the first transformation formula:

[0116]

[0117] where, I(p y ) is the light intensity of the projection image; I(p f ) is the light intensity of the hidden imaging target; A(p f ,p y ) is the light transmission matrix from the hidden imaging target to the projection image; F represents all point light sources on the entire reflection surface, which is a rectangular area with two spatial dimensions, corresponding to the double integral; is the ambient light and the noise of the multi-band sensor cluster.

[0118] Discretize the first transformation formula into the second transformation formula:

[0119] y = A(μ, r, c)f + n b+d ,

[0120] where, f is the hidden imaging target, which can be regarded as I(p f) vector; y is the projected image, which can be regarded as I(p y ) vector; A is determined by the bidirectional reflection distribution function μ of the relay surface, the distance r between the hidden imaging target and the relay surface, and the position c of the multi-band sensor cluster.

[0121] Let the corresponding transformation from the hidden imaging target to the projected image be T μ,r,c,n , then the third transformation formula is obtained: where is the inverse of T μ,r,c,n .

[0122] Combining the second transformation formula and the third transformation formula, T μ,r,c,n is obtained, that is, the transformation relationship from the hidden imaging target to the projected image is obtained.

[0123] It can be seen that the present invention discloses a non-line-of-sight real-time imaging system and imaging method based on multi-band information fusion, including: an information acquisition unit, a calculation and processing unit, and an application terminal; the information acquisition unit includes an unmanned carrier platform and a multi-band sensor cluster, the unmanned carrier platform carries the multi-band sensor cluster, the unmanned carrier platform is used to control the acquisition position of the multi-band sensor cluster, and the multi-band sensor cluster is used to collect the optical information of the hidden imaging target on the relay surface and push the optical information to the calculation and processing unit in the form of a video stream; the calculation and processing unit is used to receive the video stream, process the video stream, and output the real-time reconstruction result of the hidden imaging target; the application terminal views and analyzes the real-time reconstruction result through an online interactive application. Such a real-time imaging system and imaging method can simultaneously process infrared light images and visible light images, effectively fuse the information of the two, obtain a real-time reconstruction result with high correlation with the hidden imaging target, can accurately restore the hidden imaging target in the non-line-of-sight area, and provide high-quality imaging results.

[0124] The above are only embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structural transformation made by using the specification and drawings of the present invention, or directly or indirectly applied to other related technical fields, is included in the patent protection scope of the present invention.

Claims

1. A non-line-of-sight real-time imaging system based on multi-band information fusion, characterized in that Comprising: An information acquisition unit, a calculation and processing unit, and an application terminal; The information acquisition unit includes an unmanned platform carrying a multi-band sensor cluster. The unmanned platform is used to control the acquisition position of the multi-band sensor cluster, and the multi-band sensor cluster is used to collect the optical information of a hidden imaging target on the relay surface and push the optical information to the calculation and processing unit in the form of a video stream; The calculation and processing unit is used to receive the video stream, process the video stream, and output the real-time reconstruction result of the hidden imaging target; The application terminal views and analyzes the real-time reconstruction result through an online interaction application.

2. The non-line-of-sight real-time imaging system based on multi-band information fusion according to claim 1, wherein The multi-band sensor cluster includes an infrared light sensor and a visible light sensor; The infrared light sensor uses the hidden imaging target as a heat source, and is captured by the infrared light sensor through the surface reflection of the relay surface; The visible light sensor is used to capture the weak optical signal reflected by the hidden imaging target through the relay surface. The weak optical signal is generated by the self-luminescence of the hidden imaging target or the reflection of ambient light.

3. The non-line-of-sight real-time imaging system based on multi-band information fusion according to claim 1 or 2, wherein The calculation and processing unit includes a preprocessing module and a reconstruction processing module; The preprocessing module uses a spatio-temporal registration method to align and match the data of different bands in the video stream, and then inputs the preprocessing result obtained after processing into the reconstruction processing module; The reconstruction processing module receives the preprocessing result for processing and outputs the real-time reconstruction result.

4. The non-line-of-sight real-time imaging system based on multi-band information fusion according to claim 3, wherein The spatio-temporal registration method includes: First, perform frame alignment on the video stream. Based on the diverse frame rates of the video streams of different bands, perform format and resolution processing on the distortion and data calibration problems existing in the synchronized multi-band video streams; Then, perform infrared frame preprocessing and visible light preprocessing on the data of the infrared and visible light bands respectively to obtain the preprocessing result.

5. The non-line-of-sight real-time imaging system based on multi-band information fusion according to claim 4, wherein The reconstruction processing module includes a passive non-line-of-sight imaging network for multi-band information fusion. The passive non-line-of-sight imaging network includes an encoder, an attention fusion module, and a decoder; The encoder receives the preprocessing result, and the preprocessing result includes a visible light image and an infrared light image; The encoder includes two groups, which process the visible light image and the infrared light image in multiple sizes respectively, and send the processing results of the two groups to the attention fusion module and the decoder; The attention fusion module fuses and splices the two groups of processing results and inputs them into the decoder; The decoder fuses the reconstruction results of the visible light image and the infrared light image into the real-time reconstruction result.

6. The non-line-of-sight real-time imaging system based on multi-band information fusion according to claim 5, wherein A hybrid loss function is adopted in the passive non-line-of-sight imaging network, expressed as: Loss total = Loss pixel + λLoss SSIM , Among them, it includes the content loss function Loss pixel and the contour loss function Loss SSIM ; The content loss function Loss pixel is as follows: The contour loss function Loss SSIM is: Loss SSIM = 1 - SSIM(I, Out); where V i is the visible light image, Out is the real-time reconstruction result, H and W are the height and width of the visible light image respectively; I is the infrared light image, SSIM is the structural similarity function between the real-time reconstruction result and the infrared light image; λ is a hyperparameter used to reduce the interference of infrared loss on color loss.

7. The non-line-of-sight real-time imaging system based on multi-band information fusion according to claim 6, characterized in that, In the hybrid loss function, λ = 0.

1.

8. A non-line-of-sight real-time imaging method based on multi-band information fusion, characterized in that, Including the following steps: Collect the optical information of the hidden imaging target on the relay surface; Convert the collected optical information into a video stream; Process the video stream to obtain the real-time reconstruction result of the hidden imaging target.

9. The non-line-of-sight real-time imaging method based on multi-band information fusion according to claim 8, wherein, In the step of collecting the optical information of the hidden imaging target on the relay surface, it includes simultaneously capturing the infrared signal and the visible light signal reflected by the relay surface.

10. The non-line-of-sight real-time imaging method based on multi-band information fusion according to claim 9, wherein It also includes obtaining the transformation relationship from the hidden imaging target to the projection image: Express the projection image as a first transformation formula: where I(p y ) is the light intensity of the projection image; I(p f ) is the light intensity of the hidden imaging target; A(p f , p y ) is the light transmission matrix from the hidden imaging target to the projection image; F represents all point light sources on the reflection surface; is the noise of the ambient light and the multi-band sensor cluster; Discretize the first transformation formula into a second transformation formula: y = A(μ, r, c)f + n b+d , Where, f is the hidden imaging target, y is the projection image, and A is determined by the bidirectional reflection distribution function μ of the relay surface, the distance r between the hidden imaging target and the relay surface, and the position c of the multi-band sensor cluster; Let the corresponding transformation from the hidden imaging target to the projected image be T μ,r,c,n , then the third transformation formula is obtained: Among them, is the inverse of T μ,r,c,n ; Combining the second transformation formula and the third transformation formula, T is obtained μ,r,c,n .

Images