Device and method for ai-based nlos (non-line-of-sight) imaging reconstruction

Abstract

The present invention relates to an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device, comprising an input processing unit that samples a partial image from an original image, a neural network processing unit that generates a frequency-converted image for the partial image and inputs the frequency-converted image into time and frequency domain networks to generate a prediction pacer field, and a scene reconstruction unit that reconstructs a hidden scene based on the prediction pacer field.

Description

Technology Field

[0001] The present invention relates to an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction technology, and more specifically, to an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction apparatus and method capable of reconstructing a hidden scene based on a prediction pacer field generated by inputting a frequency-converted image of a partial image generated through a neural network into time and frequency domain networks. Background Technology

[0003] AI-based image reconstruction technology refers to a technology that utilizes artificial intelligence (AI) and machine learning (ML) algorithms to restore damaged or missing images or to convert low-resolution images into high-resolution ones. This technology is used to fill in missing parts of an image or to transform and reconstruct images of better quality. To this end, deep learning algorithms, particularly generative models and convolutional neural networks (CNNs), are primarily utilized.

[0004] Korean Published Patent No. 10-2022-0180535 (December 21, 2022) includes the steps of: setting a mask for a plurality of regions based on an original image; outputting a latent code for generating a first image reconstructed for each of the plurality of regions from the original image based on a first artificial intelligence learning model; generating the first images reconstructed from the latent code for each of the plurality of regions based on a second artificial intelligence learning model; and applying the mask to each of the first images generated for each of the plurality of regions, and combining the first images to which the mask is applied to generate a second image reconstructed for the original image. Prior art literature

[0006] Korean Patent Publication No. 10-2022-0180535 (December 21, 2022) The problem to be solved

[0007] One embodiment of the present invention aims to provide an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction apparatus and method capable of generating a frequency-converted image for a partial image through a neural network and generating a prediction pacer field by inputting the frequency-converted image into time and frequency domain networks.

[0008] One embodiment of the present invention aims to provide an artificial intelligence-based Non-Line-of-Sight (NLOS) imaging reconstruction device and method capable of reconstructing a hidden scene based on a predicted pacer field.

[0009] One embodiment of the present invention aims to provide an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction apparatus and method capable of generating a noise-removed partial image by performing noise removal through sensor noise simulation in a partial image. means of solving the problem

[0011] Among the embodiments, an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device includes an input processing unit that samples a partial image from an original image, a neural network processing unit that generates a frequency-converted image for the partial image and inputs the frequency-converted image into time and frequency domain networks to generate a prediction pacer field, and a scene reconstruction unit that reconstructs a hidden scene based on the prediction pacer field.

[0012] The above input processing unit can generate a noise-removed partial image by performing noise removal through sensor noise simulation in the above partial image.

[0013] The neural network processing unit performs input phasor convolution to generate the frequency-transformed image by performing an FFT transformation, applying an illumination function, and an IFFT transformation on the partial image, and the illumination function can extract a frequency band of interest by passing a specific frequency band through the frequency band of the partial image.

[0014] The above neural network processing unit can generate a time information preservation image by inputting the frequency-converted image into the time domain network implemented as a residual block for temporal information processing.

[0015] The above neural network processing unit can generate a frequency information processing image by inputting it into the frequency domain network implemented as a convolution layer for processing the frequency components of the above temporal information preservation image.

[0016] The above neural network processing unit can generate the above prediction pacer field that predicts hidden objects by extracting the frequency band of interest from the above frequency information processing image through target training.

[0017] The above neural network processing unit can implement the target training as a loss function for controlling outliers of the hidden object.

[0018] The scene reconstruction unit can restore the hidden scene by determining the location and shape of the object through Rayleigh-Sommerfeld Diffraction (RSD) operations on the frequency band constituting the prediction pacer field.

[0019] Among the embodiments, an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction method performed in an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device comprises an input processing step of sampling a partial image from an original image, a neural network processing step of generating a frequency-converted image for the partial image and inputting the frequency-converted image into time and frequency domain networks to generate a prediction pacer field, and a scene reconstruction step of reconstructing a hidden scene based on the prediction pacer field. Effects of the invention

[0021] The disclosed technology may have the following effects. However, this does not mean that a specific embodiment must include all of the following effects or only the following effects; therefore, the scope of the rights of the disclosed technology should not be understood as being limited by this.

[0022] An artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device and method according to one embodiment of the present invention can generate a frequency-converted image of a partial image through a neural network and input the frequency-converted image into time and frequency domain networks to generate a prediction pacer field.

[0023] An artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device and method according to one embodiment of the present invention can reconstruct a hidden scene based on a predicted pacer field.

[0024] An artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device and method according to one embodiment of the present invention can generate a noise-removed partial image by performing noise removal through sensor noise simulation in a partial image. Brief explanation of the drawing

[0026] FIG. 1 is a diagram illustrating an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device according to one embodiment of the present invention. Figure 2 is a diagram illustrating the functional configuration of the artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device of Figure 1. Figure 3 is a diagram illustrating the system configuration of the artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device of Figure 1. FIG. 4 is a flowchart illustrating an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction method according to the present invention. Figure 5 is a diagram of the reconstruction result of FK (right) for the illumination function in the frequency domain (left) and wavenumber filtered measurements. Figure 6 is a diagram of the qualitative results for Bike and Dragon in the Stanford real dataset. Figure 7 is a qualitative result diagram related to resolution for non-confocal 16 × 16 sparse sampling in an actual dataset. Figure 8 is a diagram showing the results of qualitative ablation for the denoising criterion. Figure 9 shows the qualitative ablation result for frequency filtering. Specific details for implementing the invention

[0027] The description of the present invention is merely an example for structural or functional explanation, and therefore the scope of the present invention should not be interpreted as being limited by the examples described in the text. That is, since the examples are subject to various modifications and may take various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical concept. Furthermore, the objectives or effects presented in the present invention do not imply that a specific example must include all of them or only such effects; therefore, the scope of the present invention should not be understood as being limited by them.

[0028] Meanwhile, the meaning of the terms described in this application should be understood as follows.

[0029] Terms such as "first," "second," etc., are intended to distinguish one component from another, and the scope of rights shall not be limited by these terms. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0030] When it is stated that one component is "connected" to another component, it should be understood that it may be directly connected to that other component, or that there may be other components in between. Conversely, when it is stated that one component is "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationships between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0031] A singular expression should be understood to include a plural expression unless the context clearly indicates otherwise, and terms such as "include" or "have" are intended to specify the existence of the implemented features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood not to preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0032] In each step, identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of the steps; the steps may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, the steps may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0033] The present invention may be implemented as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, the computer-readable recording medium may be distributed across networked computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0034] Unless otherwise defined, all terms used herein have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the context of the relevant technology and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this application.

[0035] FIG. 1 is a diagram illustrating an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device according to one embodiment of the present invention.

[0036] Referring to FIG. 1, an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device (100) may include an input processing unit (110), a neural network processing unit (120), and a scene reconstruction unit (130), and the neural network processing unit (120) may include an input pacer convolution module (122), a time domain network module (124), a prediction pacer field generation module (126), and a target training module (128).

[0037] The input processing unit (110) can sample a partial image from the original image.

[0038] More specifically, the operation of the input processing unit (110) is as follows.

[0039] Since processing the entire data of the original image consumes a lot of computational resources, the input processing unit (110) can reduce the amount of data by selectively sampling a partial region containing important information from the entire image, thereby increasing the processing speed and processing in real time.

[0040] The input processing unit (110) can exclude unnecessary areas of the original image or parts that may contain noise from the processing target so as to improve the quality of the signal and focus on important information.

[0041] Sampling methods include random sampling, which involves randomly selecting and sampling various parts of the original image to obtain data representing the entire image; importance-based sampling, which involves prioritizing the sampling of areas containing important information (e.g., parts containing objects) in the original image to enable subsequent processing based on higher quality data; and region-based sampling, which involves selectively extracting and sampling specific parts of the image (e.g., the center, the boundary of a specific object), which is advantageous for intensively analyzing the area where an object is located in NLOS imaging.

[0042] The neural network processing unit (120) can generate a frequency-converted image for a partial image and input the frequency-converted image into time and frequency domain networks to generate a prediction pacer field.

[0043] More specifically, the operation of the neural network processing unit (120) is as follows.

[0044] The neural network processing unit (120) can extract components of a specific frequency band from an image through a frequency transform, which is a process of converting an input partial image from the time domain to the frequency domain, and analyze the features of an object based on this, and in this process, reduce noise and emphasize important information.

[0045] The neural network processing unit (120) can input the image obtained through frequency transformation into two networks, a Time Domain Network and a Frequency Domain Network, which can learn important features in their respective domains and combine them to derive prediction results. Here, the Time Domain Network analyzes temporal changes in the image, for example, by learning the movement of an object or changes in signals over time to estimate the position of the object. The Frequency Domain Network filters frequency components of the image in the frequency domain, improves signal quality by emphasizing specific frequency bands or removing noise components, and can predict the shape of the object more accurately.

[0046] The neural network processing unit (120) can generate a prediction phasor field containing phase and amplitude information of a wave reflected from an object as the final result generated by the neural network processing unit. Here, the prediction phasor field contains important information indicating the position and shape of the object, and in a subsequent step, the hidden scene can be restored through Rayleigh-Sommerfeld Diffraction (RSD) operations, etc.

[0047] The scene reconstruction unit (130) can reconstruct a hidden scene based on a predicted pacer field.

[0048] More specifically, the operation of the scene reconstruction unit (130) is as follows.

[0049] The scene reconstruction unit (130) can visually indicate what shape the hidden object actually has and where it is located by analyzing the way the waves reflected from the object propagated based on the predicted pacer field and restoring the position and shape of the object.

[0050] The scene reconstruction unit (130) can calculate the diffraction phenomenon that occurs as waves reflected from an object propagate around an obstacle using Rayleigh-Sommerfeld Diffraction (RSD), which is the main mathematical model used in scene reconstruction. Here, the RSD calculation can accurately restore information about hidden objects by mathematically modeling how waves reflected from an object propagate and how those waves affect the position or shape of the object. The RSD formula is as follows.

[0051]

[0052] u(P) represents how a wave is diffracted and reflected at a specific point PPP, λ is the wavelength, r is the distance traveled by the wave, and θ is the angle of incidence.

[0053] The scene reconstruction unit (130) can estimate the exact location of a hidden object by analyzing the path of a wave reflected from an object using a predicted pacer field.

[0054] The scene reconstruction unit (130) can provide information including the size, boundaries, and surface shape of the object by analyzing the diffraction and reflection patterns of the waves and restoring the shape of the object, since the reflected waves of the object affect the external shape of the object.

[0055] The scene reconstruction unit (130) not only estimates the position and shape of the object but also reconstructs a high-resolution image of the object, thereby providing a very precise image even in situations where the object is not directly visible.

[0056] The scene reconstruction unit (130) can provide an overall understanding of the hidden scene through a reconstructed image that includes not only the appearance of the object but also information about the surrounding environment.

[0057] Figure 2 is a diagram illustrating the functional configuration of the artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device of Figure 1.

[0058] Referring to FIG. 2, an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device (100) may include an input processing unit (110), a neural network processing unit (120), and a scene reconstruction unit (130), and the neural network processing unit (120) may include an input pacer convolution module (122), a time domain network module (124), a prediction pacer field generation module (126), and a target training module (128).

[0059] The input processing unit (110) can generate a noise-removed partial image by performing noise removal through sensor noise simulation in the partial image.

[0060] More specifically, the input processing unit (110) can simulate sensor noise that may occur in the actual environment and learn the patterns and characteristics of the noise that may occur by applying a noise removal algorithm that takes this into account. Such sensor noise is an irregular signal that occurs during the process of collecting image data and can generally appear as Gaussian noise, Poisson noise, etc., and can occur for various reasons such as sensor performance, environmental factors, and signal variability.

[0061] The input processing unit (110) can process data containing simulated noise to remove unnecessary noise components and leave only important signal components. To this end, the input processing unit (110) can use various filtering techniques or neural networks. Representative methods include a median filter that removes noise using the values ​​of surrounding pixels and maintains the clarity of the original image, a Gaussian filter that suppresses noise based on a Gaussian distribution and generates a smoother image, and a method of learning and removing noise through a deep learning model, such as neural network-based removal which is particularly effective for complex noise patterns.

[0062] The input processing unit (110) generates a noise-removed portion image, which is a clean data to be used for subsequent processing, and through the generation of the noise-removed portion image, the influence of sensor noise is reduced and important signal components are more clearly revealed, so that, as a result, more accurate data can be provided to the neural network during the learning and inference process.

[0063] The neural network processing unit (120) may include an input pacer convolution module (122), a time domain network module (124), a prediction pacer field generation module (126), and a target training module (128).

[0064] The input phasor convolution module (122) performs input phasor convolution to generate a frequency-transformed image by performing an FFT transformation on a partial image, applying a lighting function, and performing an IFFT transformation, and the lighting function can extract a frequency band of interest by passing a specific frequency band from the frequency band of the partial image.

[0065] More specifically, the input phasor convolution module (122) can separate the signal in the time domain into frequency components using the FFT (Fast Fourier Transform) and thereby extract important frequency components within the image.

[0066] Additionally, the input phasor convolution module (122) acts as a filter on the frequency-transformed data obtained through the FFT and can apply an illumination function to allow only specific frequency bands to pass through while suppressing components of the remaining bands. Here, the illumination function acts as a filter that allows only specific frequency bands of the signal to pass through in the frequency domain, thereby blocking unnecessary frequency components of the signal or bands containing a lot of noise, and leaving only frequency bands containing useful information. This frequency band of interest may contain important information related to the location or shape of an object.

[0067] Additionally, the input phasor convolution module (122) can restore the signal filtered in the frequency domain back to the time domain through an inverse Fourier transform (IFFT) to provide a more precise signal.

[0068] The time domain network module (124) can input the frequency-converted image into a time domain network implemented with a residual block for processing temporal information to generate a temporal information-preserving image, and input it into a frequency domain network implemented with a convolution layer for processing the frequency components of the temporal information-preserving image to generate a frequency information-processed image.

[0069] More specifically, the time domain network module (124) can receive a frequency-converted image into the time domain network and convert it into an image that reflects the temporal change of an object through temporal information processing.

[0070] The residual block is a neural network structure designed to solve the vanishing gradient problem that occurs as the neural network deepens and to preserve important features of the signal. By passing the input information as is to the output through the Skip Connection, while processing the signal in the intermediate convolution layer to learn important temporal features, the neural network can preserve and process important information about changes in the signal in the time domain.

[0071] A time domain network is a network focused on processing and learning temporal information of a signal. By learning signals that change over time or when an object moves, it can track dynamic objects or reconstruct images that reflect changes over time. The time domain network module (124) can generate a temporal information-preserving image by learning changes in the signal over time through a neural network via a residual block and reflecting these changes in the image.

[0072] Temporal information-preserving images are images that reflect how an object changes over time or how a signal changes over time, enabling accurate reconstruction in NLOS imaging even when the object is not in a static state but is moving or changing over time. Temporal information-preserving images can visually represent not only the position of an object but also how it moves and changes over time.

[0073] Additionally, the time domain network module (124) can input the time information preservation image into a frequency domain network implemented with a convolution layer to filter out frequency components and generate a frequency information processing image in which only important information remains, enabling more precise object reconstruction in subsequent neural network processing.

[0074] Frequency domain networks composed of convolution layers can extract important information existing in specific frequency bands or remove unnecessary noise by analyzing the frequency components of an input signal as frequency components rather than analyzing them as variations over time.

[0075] Convolutional layers process the frequency components of an input signal, can extract specific patterns, emphasize information in important frequency bands, and suppress noise components. They also act as filters in the frequency domain to remove unnecessary frequency bands, leaving only the frequency of interest required for object reconstruction.

[0076] Through this, the neural network processing unit (120) learns more precise information about the position and shape of the object and can derive more accurate results during the reconstruction process.

[0077] The prediction pacer field generation module (126) can generate a prediction pacer field that predicts hidden objects by extracting a frequency band of interest from a frequency information processing image through target training.

[0078] Target training is a learning method based on a loss function used in the learning process to reduce the difference between the result predicted by the neural network and the actual result. In this process, the prediction pacer field generation module (126) learns important frequency components through frequency information processing images and extracts frequency bands of interest to exclude unnecessary noise or frequency components and emphasize only the necessary information.

[0079] The prediction pacer field generation module (126) can train the network by using a loss function, mainly an L1 loss function or a Huber loss function, to measure how well the value predicted by the network matches the actual location and shape of the object.

[0080] The prediction pacer field generation module (126) can more accurately reconstruct an object by excluding unnecessary components from the frequency band and leaving only important components, by using a neural network trained through target training to accurately extract the frequency band of interest, which is a frequency range containing important information about the position and shape of an object, from a frequency information processing image in which only important frequency components remain from a signal filtered through a convolution layer in the previous step.

[0081] The prediction phasor field generation module (126) can learn the frequency band of interest from the frequency information processing image through target training and generate a prediction phasor field that predicts the position and shape of an object based on this. The prediction phasor field is a field containing phase and amplitude information of a wave reflected from an object, and contains important information about the position and shape of the object, so it is used to estimate the accurate position and shape of the object in subsequent steps and can play an important role in restoring hidden objects in high resolution.

[0082] The neural network processing unit (120) can predict the position and shape of an object more precisely by the prediction pacer field generation module (126) extracting information that more accurately reflects the features of the hidden object from the frequency information processing image.

[0083] The target training module (128) can implement target training with a loss function for controlling outliers of hidden objects.

[0084] Outliers are values ​​that deviate significantly from other normal patterns in the data. In NLOS imaging, outliers can occur due to sensor noise, measurement errors, environmental factors, etc. Since these outliers can increase errors in the prediction process of neural networks and reduce the accuracy of reconstructed objects, it is important to handle them effectively.

[0085] A loss function is used to minimize the difference between predicted and actual values ​​during neural network training. In particular, outlier control loss functions can help ensure more stable training by reducing the network's sensitivity to outliers. Commonly used outlier control loss functions include the L1 loss function and the Huber loss function. The L1 loss function calculates the absolute error between the predicted and actual values ​​and is less sensitive to outliers, meaning the network may not be significantly affected even in the presence of outliers. The Huber loss function combines the advantages of both L1 and L2 loss functions; it operates like an L2 loss function for small errors and like an L1 loss function for large errors (outliers), allowing the network to maintain its training performance while minimizing the impact of outliers.

[0086]

[0087] a can represent the difference between the predicted value and the actual value, and δ can represent the threshold value.

[0088] Target training is the process of minimizing the difference between the values ​​predicted by the network and the actual values, and it is important to reduce unnecessary errors that may arise from outliers.

[0089] An outlier control loss function can enhance the stability of learning and enable more sophisticated reconstruction by preventing the network from becoming overly sensitive to errors even when outliers occur during neural network training.

[0090] In NLOS imaging, outliers are errors that may occur during the process of reconstructing hidden objects, and if the target training module (128) fails to handle them, the location or shape of the object may be incorrectly predicted.

[0091] Through the outlier control loss function, the target training module (128) learns important frequency components and signals while minimizing the influence of outliers, so that the location and shape of the hidden object can be accurately reconstructed.

[0092] The scene reconstruction unit (130) can restore the hidden scene by determining the location and shape of the object through Rayleigh-Sommerfeld Diffraction (RSD) operations on the frequency bands constituting the prediction pacer field.

[0093] More specifically, the scene reconstruction unit (130) can reconstruct the scene and visualize it as a high-resolution image even when the object is not visible, based on the position and shape of the object extracted through RSD calculation.

[0094] Figure 3 is a diagram illustrating the system configuration of the artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device of Figure 1.

[0095] Referring to FIG. 3, an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device (100) may include a processor (210), memory (230), user input / output unit (250), network input / output unit (270), and communication port unit (290).

[0096] The processor (210) can receive a question composed of images and text through a text-only language model and a vision-language model, generate a text response and a multimodal response to the question, manage memory (230) that is read or written during this process, and schedule the synchronization time between volatile memory and non-volatile memory in memory (230). The processor (210) can control the overall operation of the visual-language model-based multimodal language processing device (100) and is electrically connected to the memory (230), user input / output unit (250), network input / output unit (270), and communication port unit (290) to control the data flow between them. The processor (210) can be implemented as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the visual-language model-based multimodal language processing device (100).

[0097] The memory (230) may include an auxiliary storage device implemented as non-volatile memory such as an SSD (Solid State Disk) or HDD (Hard Disk Drive) and used to store all data required for the visual-language model-based multimodal language processing device (100), and may include a main memory implemented as volatile memory such as RAM (Random Access Memory). Additionally, the memory (230) may store a set of instructions that perform the role of the visual-language model-based multimodal language processing device (100) according to the present invention by being executed by an electrically connected processor (210).

[0098] The user input / output unit (250) includes an environment for receiving user input and an environment for outputting specific information to the user, and may include an input device including an adapter such as a touch pad, touch screen, virtual keyboard, or pointing device, and an output device including an adapter such as a monitor or touch screen. In one embodiment, the user input / output unit (250) may correspond to a computing device connected via remote access, and in such case, the visual-language model-based multimodal language processing device (100) may be performed as an independent server.

[0099] The network input / output unit (270) provides a communication environment for connecting to an attack IP terminal or a test IP terminal through a network, and may include an adapter for communication such as a LAN (Local Area Network), MAN (Metropolitan Area Network), WAN (Wide Area Network), and VAN (Value Added Network). Additionally, the network input / output unit (270) may be implemented to provide short-range communication functions such as WiFi or Bluetooth, or wireless communication functions of 4G or higher for wireless transmission of data.

[0100] The communication port section (290) is a hardware interface for connecting to external hardware, for example, the external hardware may include a printer, a mouse, or USB hardware. The communication port section (290) can detect the connection of specific USB hardware and perform the role of a CTI enhancement device (130).

[0101] FIG. 4 is a flowchart illustrating an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction method according to the present invention.

[0102] In FIG. 4, an artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device (100) performs an input processing step (step S310) of sampling a partial image from an original image, a neural network processing step (step S330) of generating a frequency-converted image for the partial image and inputting the frequency-converted image into time and frequency domain networks to generate a prediction pacer field, and a scene reconstruction step (step S350) of reconstructing a hidden scene based on the prediction pacer field.

[0103] In step S310, the input processing unit (110) may sample only the parts containing important information, without using the entire original image.

[0104] More specifically, the input processing unit (110) can improve the quality of the data by sampling only the area containing important information from the original image containing noise or unnecessary information.

[0105] In step S330, the neural network processing unit (120) can generate a frequency-transformed image and input it into a time and frequency domain network to generate a prediction pacr field. Through this process, the neural network processing unit (120) can accurately predict the location and shape of an object in NLOS imaging and play an important role in restoring hidden objects.

[0106] More specifically, the neural network processing unit (120) can learn not only the static characteristics of an object but also changes over time by processing the time domain and the frequency domain simultaneously, thereby enabling the derivation of more precise results that reflect the movement or changes of the object. The neural network processing unit (120) can significantly improve the accuracy of object reconstruction by extracting important information in the frequency domain and removing noise components to increase the quality of the final pacer field.

[0107] In step S350, the scene reconstruction unit (130) can reconstruct the location and shape of the hidden object based on the predicted pacer field and the RSD calculation result.

[0108] More specifically, the scene reconstruction unit (130) can restore a high-resolution image of an object to visually reconstruct what a hidden object looks like and where it is located in real-time or near-real-time, and thus contribute to accurately restoring an object in a complex environment.

[0109] 1. Proposed method

[0110] 1.1. Overview of Pacer Field NLOS Imaging

[0111] The goal of NLOS (Non-Line-of-Sight) imaging is to reconstruct a hidden scene through indirect multi-reflectance measurements, where short laser pulses at multiple points of relay wall P When illuminated, light scatters toward the hidden object, and some photons strike the object and return to the relay wall. Various points of relay wall C When scanning, impulse response You can obtain.

[0112] Recent phasor field NLOS methods view NLOS imaging as a diffraction wave propagation problem using a virtual camera, demonstrating that this can be solved using directly observable diffraction operators. The phasor wavefront at a virtual aperture is It can be calculated from.

[0113]

[0114] Here is the wavefront of a virtual light source, and is the convolution operator in time. The hidden scenes are It can be reconstructed using the wave propagation operator Φ.

[0115]

[0116] Here is a point in the imaged hidden scenes. The propagation operator Φ is typically formulated using the Rayleigh-Sommerfeld Diffraction (RSD). While diffraction-based NLOS methods have shown notable results, the reconstruction quality of these methods depends heavily on the quality of the measurement data.

[0117] Resolution limit

[0118] The spatial resolution of a phasor camera is defined as follows.

[0119] 0.61λL / d

[0120] Here, λ represents the wavelength, L represents the imaging distance, and d represents the diameter of the virtual aperture. The minimum achievable wavelength λ is determined by the sampling distance Δp, where λ > 2Δp holds. Therefore, theoretically, increasing Δp or decreasing d limits the spatial resolution of the system.

[0121] The illumination phasor field in NLOS imaging is primarily implemented using a virtual transient camera and utilizes a short Gaussian-shaped flash. The corresponding illumination function is expressed as follows.

[0122] P(xp,t)=δ(xpxls)(eiΩCtet22σ2)

[0123] Here,

[0124] · δ(xpxls) is the Dirac delta function, representing the difference between the illumination position xls and the sampling position xp.

[0125] · eiΩCt is a complex periodic function that represents the time dependence at a specific frequency ΩC.

[0126] · et22σ2 is a Gaussian function that represents a Gaussian pulse shape with respect to time t, and σ represents the standard deviation of the Gaussian.

[0127] The Fourier domain representation of this function can be expressed as follows.

[0128] P^(xp,ω)=δ(xpxls)eσ2(ωΩC)2

[0129] This shows that the illumination function in the time domain appears in a Gaussian form in the frequency domain as well.

[0130]

[0131] Here, xls represents the position of a virtual light source, and ΩC is the central frequency, which is determined by the wavelength (λ). The illumination phasor field PF(xp,Ω) in the frequency domain is defined in a Gaussian form (Fig. 5, top-left), and acts as a band-pass filter.

[0132] Therefore, the phasor wavefront calculated from the aperture is a band-limited signal, meaning that only signals within a specific frequency range are required to reconstruct hidden scenes. In other words, to reconstruct hidden scenes, only signals within a certain frequency band are used, and unnecessary frequency components are eliminated.

[0133] 1.2. Noise Removal and Frequency of Interest

[0134] NLOS imaging can result in degraded measurements due to a very low signal-to-noise ratio (SNR). Reducing the number of scan points decreases the total number of detected photons, further amplifying the impact of noise. Since sensor noise is typically modeled as a Poisson distribution, criteria for removing Poisson noise are applied to the signal recovery problem.

[0135] However, because sensor noise affects the entire frequency spectrum, this training method induces the network to recover all frequency components. As a result, the trained network often loses detail and produces over-smoothed results.

[0136] Analysis of the influence of noise on frequency components

[0137] To analyze the effect of noise on frequency components, frequency components are visualized by separating the rendered measurements of Stanford Bunny into cases with and without noise. Additionally, the reconstructed scene is visualized after using the FK method and a band-pass filter to retain only components within a specific frequency range and remove the rest.

[0138] result

[0139] As shown in Fig. 5, most of the significant signal is observed in the center frequency range (range B). When Poisson noise is added, artifacts appear in the low-frequency band, and high-frequency components become difficult to distinguish from the noise. On the other hand, the signal near the center frequency still contains a clear object shape, is more robust against noise, and is easier to recover.

[0140] These results provide an incentive to limit the network spectrum and use the aperture's phasor field as band-limited input and output.

[0141] 1.3. Learning to Enhance Aperture Phasor Field

[0142] Based on the observations described earlier, we will explain Learning to Enhance Aperture Phasor Field (LEAP), a phasor-based neural network. LEAP can predict clean and complete measurements from noisy partial observations. Assuming a single virtual illumination point xls, it eliminates the integral over the illumination function P.

[0143] Method Overview (Refer to Fig. 5)

[0144] First, partial inputs are sampled from the total measurement H(xp→xc,t) and contaminated with Poisson noise; then, the enhancement network receives these noise-containing partial inputs and predicts the optimal phasor field in the frequency domain that includes a complete scan and a clean signal at the aperture.

[0145] Learning process

[0146] The network is trained by minimizing the L1 distance between the predicted pacer field at the aperture and the optimal pacer field. Once training is complete, hidden scenes are reconstructed by propagating the predicted pacer field using the RSD algorithm.

[0147] Description of each component

[0148] Sensor noise simulation

[0149] To simulate the powerful effects of noise in NLOS imaging, a Poisson distribution is used to model sensor noise following the computational model of a SPAD (Single-Photon Avalanche Diode). Sensor noise is modeled at multiple exposure levels by considering the process of calculating the accumulated number of photons.

[0150]

[0151] Here, H′ is the noise-added measurement, and H~ is a measurement partially sampled from the original measurement H. η is the photon detection efficiency, and g models time jitter. c controls the exposure time, and d models background noise, which includes ambient light and dark counts. After the measurements are partially sampled and corrupted by noise, these values ​​are provided as input to the network.

[0152] Input Phasor Field Convolution

[0153] The noisy partial input is subsequently processed by convolution with various illumination functions. A set of illumination wavefronts with multiple wavelengths is used, and their frequency ranges are selected to be close to the target frequency range.

[0154] The convolution output F={f1,f2,...,fi} for various wavelengths {λ1,λ2,...,λi} is calculated using the convolution theorem in the frequency domain. This can be explained as follows.

[0155] Fi=F(H′)F(Pλi)

[0156] Here, F represents the Fourier transform, H′ is a noisy partial measurement, and Pλi is the illumination function for a specific wavelength λi. According to the convolution theorem in the frequency domain, the convolution between the input signal and the illumination function for multiple wavelengths can be processed efficiently.

[0157]

[0158] Here, PFi(xp,Ω) is the illumination phasor field corresponding to wavelength λi in the frequency domain. Both the real and imaginary components are concatenated with F and passed to an enhancement network for feature extraction.

[0159] Enhancement Network

[0160] A 3D residual convolutional neural network (CNN) is used as the enhancement network. This network extracts feature volumes from F through multiple 3D residual blocks and transforms them into the frequency domain. Subsequently, three convolutional layers extract additional features from these frequency volumes to predict residuals. Residuals include both real and imaginary parts in the frequency domain.

[0161] These residuals are added to the upsampled (and zero-padded for small apertures) input values ​​to predict a clean and complete measurement H^(xp→xc,Ω). The model finally calculates the aperture phasor field.

[0162] P^F(xp→xc,Ω)

[0163] This network accepts partial input values ​​containing noise, extracts features in the frequency domain, and predicts the complete signal with the noise removed.

[0164]

[0165] Here, PF(xp,Ω) represents the target illumination phasor wavefront and has wavelength λT. This is also used to calculate the ground truth phasor field.

[0166] Compared to Spatially Shifted Networks (SSN), the network is composed only of 3D convolution blocks and does not include computationally expensive attention branches, which improves the efficiency of the model.

[0167] Training Objective and Reconstruction

[0168] To train the network, the L1 distance between the predicted pacer field P^F(xc,Ω) and the target pacer field PF(xc,Ω) is minimized. The target aperture wavefront is calculated by convolving the optimal measurements with the target illumination function PF(xp,t).

[0169] Since the significant frequency components required to reconstruct the hidden scene are determined by the wavelength λT, loss is minimized only for such components. The learning objective of this method can be described as follows.

[0170] L=∑Ω|P^F(xc,Ω)PF(xc,Ω)

[0171] This loss function calculates the difference between the predicted pacer field and the target pacer field, and trains the network to predict a more accurate pacer field.

[0172]

[0173] Here, S represents the frequency range and is defined as follows.

[0174] S=[ΩCΔΩ,ΩC+ΔΩ]

[0175] This range is the frequency region where the coefficients of the input wavefront are greater than the maximum ratio γ. The center frequency ΩC is determined by the target wavelength λT. When the network is trained based on the aperture wavefront, the network's training objective is limited to the frequency range of interest.

[0176] After the network predicts the enhanced phasor field from the aperture, the hidden scenes can be reconstructed using conventional wave propagation operators. The reconstruction task is performed using an RSD algorithm based on the 2D Fast Fourier Transform (FFT).

[0177] This process estimates the location and shape of hidden objects through meaningful signals within a frequency range and enables accurate reconstruction in an NLOS environment.

[0178] 2. Experiment

[0179] To demonstrate the efficiency of the proposed method, experiments are conducted in two practical acquisition scenarios: (1) sparse sampling and (2) small aperture scanning.

[0180] Evaluation Scenarios

[0181] The acquisition scenario is divided into a total of 4 settings.

[0182] · Conf-16: Confocal sparse scanning with 16 × 16 sampling.

[0183] · Conf-8: Confocal sparse scan with 8 × 8 sampling.

[0184] · Conf-small: Confocal scanning with 16 × 16 sampling in a small area of ​​1 m × 1 m.

[0185] · Non-16: Non-confocal sparse scanning at 16 × 16 points.

[0186] Following previous research, performance is evaluated in partial sampling scenarios, where sampling points are sampled evenly while maintaining appropriate spatial spacing across the entire measurement (center-crop method is used for small apertures). The goal is to recover measurements with a 2 m × 2 m aperture, a temporal resolution of 32 ps, and 64 × 64 sampling. This reconstructs hidden volumes of size 64 × 64 × 64.

[0187] Baselines

[0188] The proposed method is compared with the following representative baseline.

[0189] · FK, LCT, RSD: Uses nearest interpolation and trilinear interpolation methods.

[0190] · SSCR: Optimization-based few-shot NLOS method.

[0191] · LFE, USM, SSN: Learning-based methods. SSN is a method for recovering missing signals from partial measurements, and USM extends the LFE architecture to consist of a signal recovery network, a feature propagator, and a volume refinement module.

[0192] Since the code for SSN is not publicly available, SSN is reproduced based on the original paper. The results of the learning-based method are reproduced using a synthetic dataset. For a fair comparison, LFE and USM are modified to use RSD as the propagation operator, trained with noise augmentation, and supervised by 2D labels generated by projecting the RSD output using the optimal measure.

[0193] Implementation Details

[0194] The model was implemented in PyTorch and trained for 160 epochs on an RTX A5000 GPU. The training process was completed within one day in the environment.

[0195] · 7 wavelengths are used for input phasor field convolution.

[0196] · γ was set to 0.1 and the target wavelength λT was set to 9.375 cm.

[0197] · The 2D projection results of all methods (models and baselines) are obtained using maximum intensity projection.

[0198] 2.1 Synthetic Dataset Evaluation

[0199] To train and validate the model, a synthetic NLOS dataset was generated using data provided by ShapeNet. The data was generated using the NLOS renderer, and a total of 15,000 objects were used. Of these, 11,000 were used for training and 4,000 were used for validation.

[0200] The generated synthetic dataset has the following characteristics.

[0201] · Scan area: 2 m × 2 m

[0202] · Sampling points: 64 × 64

[0203] · Temporal resolution: 32 ps (including temporal jitter)

[0204] Quantitative comparison

[0205] To evaluate the visual quality of the model, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) were measured, and the root mean square error (RMSE) was used to evaluate the reconstructed geometric accuracy. The 2D projection result using the optimal RSD measurement was used as the ground truth intensity images.

[0206] Due to differences in albedo reconstruction values, it was compared with RSD-based methods. It was compared with nearest neighbor interpolation (RSDNearest), triple linear interpolation (RSDLinear), and learning-based methods, and an evaluation was conducted using a sensor noise model.

[0207] Results

[0208] Table 1 shows the quantitative results for four evaluation scenarios on the synthetic dataset. The proposed model demonstrates superior performance compared to all other methods in terms of visual quality and geometric accuracy. RSD exhibits the lowest performance among both interpolation methods.

[0209] · SSN fails to learn robust representations from noisy measurements, leading to inaccurate results.

[0210] · LFE failed to yield meaningful results, indicating that it struggled to utilize inaccurately propagated feature volumes.

[0211] · USM improves performance by adding a signal recovery network to LFE and provides the second-best results.

[0212] Nevertheless, the performance gap between USM and the proposed model demonstrates that the pacer-based approach guides the model better to effectively extract meaningful signals from noisy partial inputs.

[0213] [Table 1]

[0214]

[0215] 2.2 Confocal Real-World Evaluation

[0216] To evaluate the generalization ability of the proposed model to real-world measurements, results were compared on the Stanford confocal real-world dataset. The Bike and Dragon instances were selected for evaluation, as these two instances were more difficult to reconstruct compared to other instances due to their low photon counts and signal-to-noise ratio (SNR).

[0217] The original measurements from the Stanford dataset were captured at 512 × 512 sampling, a 2 m × 2 m aperture, and a 32 ps time resolution. The measurements were first downsampled by 2x to slightly increase the exposure time per pixel, after which partial measurements were sampled using spatial spacing and center cropping. Measurements with an exposure time of approximately 55 ms were used, which corresponds to a 60-minute exposure time of the original measurements.

[0218] Trilinear interpolation and LFE results were omitted due to space constraints.

[0219] Sparse Sampling Results

[0220] The results for the 16 × 16 sparse sampling scenario are shown in Figure 6 (first and fourth rows). The proposed model demonstrated superior performance compared to other baselines and produced clean results with detail. Inverse NLOS methods using nearest interpolation reconstruct only rough shapes, and in most cases, it is difficult to identify objects due to artifacts. SSCR reconstructs only parts of the objects, and SSN produced results containing severe artifacts due to the lack of noise removal capabilities. USM produced generally plausible results, but some artifacts were observed.

[0221] In contrast, the proposed model successfully reconstructed a clean form, including details such as the bike's rear wheel and the dragon's head.

[0222] In the 8 × 8 sparse sampling scenario (scan time four times shorter than 16 × 16), other baselines failed to produce plausible results (see the second and fifth rows of Fig. 6). On the other hand, the proposed method provided results in which various object parts, such as the wheels of the bike and the legs of the dragon, were clearly visible. The results of the proposed model in the sparse sampling scenario demonstrate the efficiency of the pacer-based network in real-world noisy environments and led to a significant improvement compared to other baselines.

[0223] Smaller Aperture Results

[0224] The proposed model achieved high-quality results even in small aperture scenarios (see the third and sixth rows of Fig. 6). Nearest interpolation only captures the rough shape of the object near the aperture, includes severe artifacts, SSCR misses much of the object, USM generates a distorted shape of the object, and SSN is affected by noise.

[0225] On the other hand, the proposed method demonstrates the applicability of the proposed model even under limited aperture conditions by reconstructing a large portion of an object, such as a bike wheel, placed outside the aperture.

[0226] 2.3 Non-Confocal Real-World Evaluation

[0227] Finally, the proposed method is evaluated in a non-focal 16 × 16 sparse sampling scenario. The results were compared using the measurements provided by Resolution. These measurements were captured at an aperture of 1.8 m × 1.3 m, a sampling interval of 1 cm, a temporal resolution of 4 ps, and an exposure time of 1 second. First, spatial zero-pad and temporal averaging were applied to adjust the measurements to a 2 m × 2 m aperture and a temporal resolution of 32 ps, after which partial inputs were sampled using the spatial interval.

[0228] Results

[0229] As shown in Fig. 7, the proposed model shows results close to the full RSD measurement and significantly improved performance, whereas the RSD results using the interpolation method are blurry and contain artifacts. LFE only reconstructed the rough shape of the object.

[0230] Interestingly, all learning-based methods using signal recovery networks (USM, SSN, and the proposed method) appear to produce convincing results under low-noise conditions. These conditions apply when the exposure time per pixel is sufficient (about 20 times longer than the Stanford dataset) and the target object has high reflectivity.

[0231] 2.4 Ablation Study

[0232] To verify the effectiveness of the proposed concept, cancellation experiments were performed in confocal and non-confocal 16 × 16 sparse sampling scenarios.

[0233] Denoising and the Phasor-Based Network

[0234] First, we conducted denoising experiments on the denoising criterion for the signal recovery problem and compared various signal recovery network variations with the proposed model.

[0235] · SSN

[0236] · SSN (SSN+) trained with added noise removal criteria

[0237] · Enhancement Network in the Time Domain (excluding pacer-based methods, Ourstime) - Two versions depending on whether noise removal criteria are applied

[0238] Reported results for Bike and Resolution instances in real-world evaluation.

[0239] The results of noise reduction for the synthetic dataset and real-world measurements can be seen in Table 2 and Figure 8. Adding noise reduction criteria greatly helps the network remove background noise and learn more robust representations. However, as shown in Figure 8, applying noise reduction criteria often results in over-resolution of the results and loss of object details (e.g., the rear wheel of a bike).

[0240] By applying a phasor-based scheme, the proposed model achieves both quantitative and qualitative improvements, generating clean results that include detail. In particular, the attention branches used in SSN do not provide significant improvements compared to the proposed model; instead, they degrade output and incur high computational costs. Based on these results, the application of simple noise removal can often lead to undesirable outcomes, highlighting the efficiency of phasor-based networks.

[0241] [Table 2]

[0243] *

[0244] Frequency Filtering

[0245] Next, the effect of frequency filtering was explored by modifying the target illumination function through comparison with two illumination functions.

[0246] · A low-pass filter that passes all frequencies smaller than the center frequency ΩC.

[0247] · A high-pass filter that passes all frequencies greater than the center frequency ΩC.

[0248] The shape of each illumination function is shown in Fig. 9, and their frequency ranges are closely related to Fig. 5. These models use additional wavelengths in the input phasor field convolution to adequately cover the target frequency range.

[0249] As reported in Table 3 and Figure 9, the low-pass and high-pass models show performance degradation compared to the proposed model. Interestingly, the low-pass model performs worse than the high-pass model and failed to reconstruct details of objects in real-world data (e.g., a dragon's head). This is also consistent with the spectral bias of neural networks toward low-frequency signals revealed in previous studies.

[0250] These results demonstrate that supervising the network using phasor wavefronts with band-limited signals helps the neural network effectively avoid spectral bias.

[0251] [Table 3]

[0252]

[0253] A learning-based method called LEAP can enhance noisy partial measurements and enable Non-Line-of-Sight (NLOS) imaging with less sampling and smaller apertures. Enhancement networks using a phased-based approach have proven their effectiveness by demonstrating high accuracy while being robust against noise across various scan scenarios. This method can be an effective solution for addressing excessive scanning procedures in NLOS imaging.

[0254] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims. Explanation of the symbols

[0256] 100: AI-based NLOS (Non-Line-of-Sight) imaging reconstruction 110: Input processing unit 120: Neural network processing unit 130: Scene Reconstruction Section 122: Input Pacer Convolution Module 124: Time Domain Network Module 126: Prediction Pacer Field Generation Module 128: Goal Training Module

Claims

Claim 1 An artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device comprising: an input processing unit that samples a partial image from an original image; a neural network processing unit that generates a frequency-converted image for the partial image, inputs the frequency-converted image into time and frequency domain networks to generate a prediction pacer field, and inputs the frequency-converted image into the time domain network implemented as a residual block for temporal information processing to generate a temporal information-preserving image; and a scene reconstruction unit that reconstructs a hidden scene based on the prediction pacer field. Claim 2 An artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device according to claim 1, wherein the input processing unit performs noise removal through sensor noise simulation in the partial image to generate a noise-removed partial image. Claim 3 An artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device according to claim 1, wherein the neural network processing unit performs input phasor convolution to generate the frequency-transformed image by performing FFT transformation, application of an illumination function, and IFFT transformation on the partial image, and the illumination function extracts a frequency band of interest by passing a specific frequency band through the frequency band of the partial image. Claim 4 delete Claim 5 An artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device according to claim 1, wherein the neural network processing unit inputs the frequency information processing image into the frequency domain network implemented as a convolution layer for processing the frequency components of the temporal information preservation image to generate a frequency information processing image. Claim 6 An artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device according to claim 5, wherein the neural network processing unit generates a prediction pacer field that predicts a hidden object by extracting a frequency band of interest from the frequency information processing image through target training. Claim 7 An artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device according to claim 6, wherein the neural network processing unit implements the target training as a loss function for controlling outliers of the hidden object. Claim 8 An artificial intelligence-based NLOS (Non-Line-of-Sight) imaging reconstruction device according to claim 1, wherein the scene reconstruction unit determines the location and shape of an object through Rayleigh-Sommerfeld Diffraction (RSD) operations on the frequency band constituting the prediction pacer field to restore the hidden scene. Claim 9 An AI-based NLOS (Non-Line-of-Sight) imaging reconstruction method performed in an AI-based NLOS (Non-Line-of-Sight) imaging reconstruction device, comprising: an input processing step of sampling a partial image from an original image; a neural network processing step of generating a frequency-converted image for the partial image, inputting the frequency-converted image into time and frequency domain networks to generate a prediction phasor field, and inputting the frequency-converted image into the time domain network implemented as a residual block for temporal information processing to generate a temporal information-preserving image; and a scene reconstruction step of reconstructing a hidden scene based on the prediction phasor field.

Images