Method for creating indoor environment structure map by using radar and deep learning and device therefor

Abstract

Disclosed are a method for creating an indoor environment structure map by using a radar and deep learning, and a device therefor. The method for creating an indoor environment structure map by using a radar and deep learning comprises the steps of: (a) transmitting a radar transmission signal in an indoor environment, receiving a reception signal reflected from a target, and signal-processing the reception signal to generate an initial indoor environment map for the indoor environment; and (b) applying the initial indoor environment map to a trained deep learning model to remove ghost targets, and generating an indoor environment map in which undetected areas are interpolated.

Description

Method and apparatus for creating an indoor environment structure map using radar and deep learning

[0001] The present invention relates to a method for creating an indoor environment structure map using radar and deep learning, and an apparatus thereof.

[0002]

[0003] Recently, autonomous driving has been attracting attention not only in the automotive sector but also in the field of indoor robotics. The core technology of autonomous driving involves using cameras, LiDAR, and radar sensors to perceive the environment surrounding vehicles or robots. However, camera and LiDAR sensors do not demonstrate consistent performance. For example, LiDAR struggles to handle obstacles such as long corridors, fog, smoke, dust, and debris, and its application in various fields is limited due to its relatively high cost. Additionally, while cameras can be used to extract indoor structures, they require a light source and may raise potential privacy infringement issues.

[0004] On the other hand, radar sensors have the advantage of maintaining relatively stable detection performance even in such environments. Due to this advantage, several studies have been conducted to generate environmental maps using radar sensors. While using radar sensors offers the advantage of enabling indoor environment mapping even in foggy weather conditions, smoky conditions, or visually challenging environments, it has the disadvantage of generating ghost targets due to multiple reflections, and these ghost targets pose a problem in that they hinder the accurate extraction of the indoor environment structure.

[0005]

[0006] The present invention is intended to provide a method for creating an indoor environment structure map using radar and deep learning, and an apparatus thereof.

[0007] In addition, the present invention aims to provide a method and apparatus for creating an indoor environment structure map using radar and deep learning, which can extract an indoor environment structure with ghost targets removed in various environments and simultaneously interpolate unmeasured areas to create an indoor environment map.

[0008]

[0009] According to one aspect of the present invention, a method for creating an indoor environment structure map using radar and deep learning is provided.

[0010] According to one embodiment of the present invention, a method for creating an indoor environment structure map using radar and deep learning may be provided, comprising: (a) transmitting a radar transmission signal in an indoor environment and receiving a reception signal reflected from a target, and processing the reception signal to generate an initial indoor environment map for the indoor environment; and (b) applying the initial indoor environment map to a pre-trained deep learning model to remove ghost targets and generate an indoor environment map in which undetected areas are interpolated.

[0011] The above deep learning model can be pre-trained using conditional data, which is structural information about the indoor environment.

[0012] The above deep learning model is a CGAN (Conditional Generative Adversarial Network) based model.

[0013] The above step (a) comprises: a step of processing the received signal to obtain a detection result of a target in the indoor environment; and a step of generating an initial indoor environment map including a ghost target based on the detection result of the target, wherein the detection result of the target is distance, azimuth, and elevation angle information for the target.

[0014]

[0015] According to another aspect of the present invention, an apparatus and system for performing a method of creating an indoor environment structure map using radar and deep learning are provided.

[0016] According to one embodiment of the present invention, a computing device may be provided comprising: a memory for storing at least one instruction; and a processor for executing the instruction stored in the memory, wherein the instruction executed by the processor comprises: (a) a step of transmitting a radar transmission signal in an indoor environment and then receiving a reception signal reflected from a target, and signal processing the reception signal to generate an initial indoor environment map for the indoor environment; and (b) a step of applying the initial indoor environment map to a pre-trained deep learning model to remove ghost targets and generate an indoor environment map in which the undetected area is interpolated.

[0017]

[0018] According to another embodiment of the present invention, a system may be provided comprising: a radar system that transmits a radar transmission signal in an indoor environment, receives a reception signal reflected from a target, and processes the reception signal to generate an initial indoor environment map of the indoor environment; and a computing device that applies the initial indoor environment map to a learned deep learning model to remove ghost targets and generate an indoor environment map in which the undetected area is interpolated.

[0019]

[0020] By providing a method and apparatus for creating an indoor environment structure map using radar and deep learning according to one embodiment of the present invention, it is possible to extract an indoor environment structure with ghost targets removed in various environments without changing separate parameters, and simultaneously interpolate unmeasured areas to create an indoor environment map.

[0021]

[0022] FIG. 1 is a flowchart illustrating a method for generating an initial indoor environment map according to an embodiment of the present invention.

[0023] FIG. 2 is a diagram illustrating a transmission signal according to an embodiment of the present invention.

[0024] FIG. 3 is a diagram illustrating a method for estimating the distance and velocity of a target from a sampled signal according to an embodiment of the present invention.

[0025] FIG. 4 is a diagram illustrating a target signal extraction method for target angle estimation according to an embodiment of the present invention.

[0026] FIG. 5 is a diagram illustrating the location of a target from a radar according to an embodiment of the present invention.

[0027] FIG. 6 is a drawing illustrating an actual environment according to an embodiment of the present invention.

[0028] FIG. 7 is a drawing illustrating an initial indoor environment map according to an embodiment of the present invention.

[0029] FIG. 8 is a flowchart illustrating a method for creating an indoor environment structure map using radar and deep learning according to an embodiment of the present invention.

[0030] FIG. 9 is a drawing illustrating training data according to an embodiment of the present invention.

[0031] FIG. 10 is a drawing illustrating the overall architecture of a deep learning model according to an embodiment of the present invention.

[0032] FIGS. 11 to 13 are drawings illustrating an indoor environment map generated through a deep learning model according to an embodiment of the present invention.

[0033] FIG. 14 is a schematic diagram illustrating a system configuration according to one embodiment of the present invention.

[0034] FIG. 15 is a drawing illustrating the architecture of a radar system according to one embodiment of the present invention.

[0035]

[0036] As used in this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "composed" or "comprising" should not be interpreted as necessarily including all of the various components or steps described in the specification, and should be interpreted as meaning that some of the components or steps may be excluded, or that additional components or steps may be included. Furthermore, terms such as "...part," "module," etc., as used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0037] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0038]

[0039] FIG. 1 is a flowchart illustrating a method for generating an initial indoor environment map according to an embodiment of the present invention, FIG. 2 is a diagram illustrating a transmitted signal according to an embodiment of the present invention, FIG. 3 is a diagram illustrating a method for estimating the distance and velocity of a target from a sampled signal according to an embodiment of the present invention, FIG. 4 is a diagram illustrating a target signal extraction method for estimating a target angle according to an embodiment of the present invention, FIG. 5 is a diagram illustrating the position of a target from a radar according to an embodiment of the present invention, FIG. 6 is a diagram illustrating an actual environment according to an embodiment of the present invention, and FIG. 7 is a diagram illustrating an initial indoor environment map according to an embodiment of the present invention.

[0040] In step 110, the radar system (1210) transmits a radar signal in which the frequency changes linearly over time. An example of a transmitted signal is shown in FIG. 2.

[0041] In Fig. 2 and represents the duration, bandwidth, and center frequency of the q-th chirp, respectively. The transmitted signal of the q-th chirp can be expressed as Equation 1.

[0042]

[0043] In step 115, the radar system (1210) receives a signal (received signal) reflected from a target through a receiving antenna. This received signal includes a Doppler shift caused by the relative velocity of the target, a delay time due to the distance to the target, and signal attenuation.

[0044] This is the relative velocity, and In the case of this distance to the k-th target, the received signal of the q-th chirp can be expressed as Equation 2.

[0045]

[0046] Here, represents the amplitude of the attenuated signal.

[0047] In step 120, the radar system (1210) converts the received signal into a baseband signal through a frequency mixer and a low-pass filter. That is, the radar system (1210) can convert the transmitted signal and the received signal into a baseband signal by multiplying them through a frequency mixer and then passing them through a low-pass filter to remove the high-frequency components of the mixed signal.

[0048] In step 125, the radar system (1210) samples the baseband signal through an analog-to-digital converter (ADC).

[0049] The sampled signal for the q-th chirp can be represented as Equation 3.

[0050]

[0051] Here, represents the number of targets, and represent the indices of the time sample and the chirp, respectively. Also, represents the amplitude of the signal, represents the sampling time.

[0052] In step 130, the radar system (1210) processes the sampled signal to generate a target detection result and uses the target detection result to generate an initial indoor environment map.

[0053] The time-sampled signal can be arranged into a two-dimensional matrix as shown in Fig. 3. In addition, to estimate both the distance and velocity of the target, Fourier transforms are applied along the sampling axis (p-axis) and chirp axis (q-axis) of the matrix, respectively, and then peak detection is performed.

[0054] The result of the 2D Fourier transform for the sampled signal can be expressed as Equation 4.

[0055]

[0056] Here, and represents the index in the frequency domain.

[0057] In order to accurately estimate the location of a target, it is essential to estimate not only the distance but also the angle between the radar system and the target. In one embodiment of the present invention, an azimuth and elevation angle can be estimated using a MIMO antenna array. This will be explained in more detail below.

[0058] The elevation and azimuth angles between the center of the antenna and the k-th target, respectively and Assuming this, the sampled baseband signal (Equation 3) can be extended as shown in Equation 5.

[0059]

[0060] Here, and represents the distance between the transmitting antenna elements in the elevation and azimuth directions, respectively. Likewise, and represents the distance between the receiving antenna elements in the elevation and azimuth directions, respectively. Also, and represents the element index of the transmitting antenna and the receiving antenna, and Wow represents the number of transmitting antennas and receiving antennas, respectively. Therefore, the total number of virtual receiving channels is It can increase up to. Therefore, when using a MIMO antenna, the size of the data cube It can be expanded into a 3D data cube.

[0061] To estimate the target's azimuth, a subset of virtual receiving channels aligned in the azimuth direction must be selected. The data cube is It consists of (see Fig. 4(a)). Here, represents the number of channels in the azimuth direction.

[0062] The index of the k-th target obtained through mathematical formula 4 and If represented as such, the total as shown in Fig. 4(b) Can sample values ​​be extracted. Channel A signal vector composed of the sampling values ​​can be represented as in Equation 6.

[0063]

[0064] Next, if a correlation matrix is ​​generated using signal vectors, it can be represented as Equation 7.

[0065]

[0066] In mathematical formula 7 represents the number of snapshots, which indicates the number of independent data sets collected at different time intervals. Also, represents the conjugate prefix operator.

[0067] Angle estimation can be performed using digital beamforming techniques. In one embodiment of the present invention, it is assumed that the angle is estimated using the Bartlett method. The normalized pseudo-spectrum of the Bartlett beamformer can be expressed as Equation 8.

[0068]

[0069] Here, represents a steering vector considering the distance in the azimuth direction, and the estimated azimuth for the k-th target maximizes the value of the normalized pseudospectrum am.

[0070] Elevation angle estimation is similar to azimuth angle estimation. The difference in estimating the elevation angle is that a subset of virtual receiving channels arranged in the elevation direction is selected, and the steering vector considers the distance in the elevation direction. Since everything else is identical, redundant explanations will be omitted.

[0071] According to one embodiment of the present invention, the radar system (1210) can generate an initial indoor environment map using target detection results (target distance, azimuth information, elevation information) as described above. Based on the target information, the position of the target in the xy plane It can be represented as shown (see Fig. 5).

[0072] In Fig. 7, blue dots represent detected targets, and yellow lines represent the locations of actual walls in an indoor environment (see Fig. 6). As shown in Fig. 8, targets in all indoor environments were commonly detected at locations similar to actual walls. However, some targets were detected at locations where there were no actual walls (i.e., areas without physical structures). In indoor environments, radio waves in the millimeter wave band are reflected or scattered, resulting in multipath reflection. As a result of these multipath reflections, ghost targets are generated, which are targets that do not exist.

[0073] Therefore, maps generated using information containing such ghost targets do not accurately represent the structure of the actual environment. Below, we will explain a method for extracting the indoor environment structure with ghost targets removed.

[0074]

[0075] FIG. 8 is a flowchart illustrating a method for creating an indoor environment structure map using radar and deep learning according to an embodiment of the present invention, FIG. 9 is a diagram illustrating training data according to an embodiment of the present invention, FIG. 10 is a diagram illustrating the overall architecture of a deep learning model according to an embodiment of the present invention, and FIG. 11 is a diagram illustrating an indoor environment map generated through a deep learning model according to an embodiment of the present invention. Hereinafter, it is assumed that an indoor environment map of a target (hereinafter referred to as the initial indoor environment map) is generated using radar as described in FIG. 1.

[0076] In step 810, the computing device (1220) obtains an initial indoor environment map. The initial indoor environment map may include the result of detecting a target in the indoor environment using radar reception signals (distance, azimuth information, and elevation angle information for the target).

[0077] Referring to Fig. 9, the following explanation will be provided. Fig. 9(a) shows a correct structure image for an indoor environment, and Fig. 9(b) shows a training dataset generated through simulation of radar data including a ghost target.

[0078] In step 815, the computing device (1220) applies the initial indoor environment map to a pre-trained deep learning model to remove ghost targets and interpolate undetected areas to output a final indoor environment map.

[0079] It is assumed that the deep learning model is trained based on simulation data. As previously mentioned, it is assumed that the deep learning model is pre-trained using conditional data, which is structural information about the indoor environment obtained through simulation.

[0080] I will explain this in more detail.

[0081] According to one embodiment of the present invention, the deep learning model may be a Conditional Generative Adversarial Networks (CGAN) based model. For the convenience of understanding and explanation, the GAN model will be briefly described.

[0082] A GAN consists of two networks: a generative model and a discriminator model. These models can improve their respective performance by continuously updating their parameters through training.

[0083] During the training process, the loss function of a GAN is designed to minimize the loss of the generative model and maximize the loss of the discriminative model, which can be expressed using a minimax strategy.

[0084] The loss function of such a GAN can be expressed as Equation 9.

[0085]

[0086] Here, represents the loss function of the GAN. These It can be expressed as in mathematical formula 10.

[0087]

[0088] Here, X and Z represent the actual image and the noise vector, respectively. At the start of training, the generative model produces results similar to the noise, but as training progresses, the model can be trained to generate synthetic images that are increasingly difficult to distinguish from the actual image.

[0089] However, these GAN models have limitations in generating desired image types or styles. This is because conventional GANs are trained to generate images from random noise vectors without specific instructions or constraints for image generation. Consequently, while conventional GANs can generate a variety of images, they are somewhat lacking in the ability to consistently target a specific style or type.

[0090] To overcome these shortcomings, interest is growing in Conditional GANs (CGANs), which generate images in a desired form by receiving conditional data such as image class and style. In the case of CGANs, the generative model can be trained to generate fake images using conditional data, and the discriminative model can be trained to distinguish between real and fake images using conditional data. In other words, CGANs can use conditional data in both the generative and discriminative models.

[0091] Therefore, the loss function of CGAN can be expressed as Equation 11.

[0092]

[0093] Here, Y represents conditional data. In Equation 11, CGAN can apply a minimax strategy using conditional data. Additionally, an L1 loss function is used during the training process, which can be expressed as in Equation 12.

[0094]

[0095] The integrated loss function can be expressed as Equation 13.

[0096]

[0097] Here, represents the weights of the loss function.

[0098] In one embodiment of the present invention, it is assumed that the deep learning model is a CGAN-based model, and the explanation will focus on this. The deep learning model can be pre-trained using conditional data, which is structural information about an indoor environment.

[0099] The detailed architecture of a deep learning model according to one embodiment of the present invention is as shown in FIG. 10.

[0100] As shown in FIG. 10, deep learning models may include a U-Net-based generative model and a patch-GAN-based discriminative model.

[0101] A U-Net-based generative model may include 8 downsampling layers (encoders) and 7 upsampling layers (decoders). In the encoder-decoder structure of the generative model, skip connections between layers may be applied to recover information lost during downsampling operations.

[0102] In addition, the discriminant model can be trained to distinguish between real and fake images at the local patch level by learning the structure of a specific patch scale using a patch-GAN architecture. The discriminant model can be composed of four downsampling layers.

[0103] Consequently, deep learning models can be pre-trained based on conditional data, which is structural information about an indoor environment generated through simulation. Conditional data may be image data regarding the structural information of the indoor environment. Such conditional data can be generated by considering the multipath conditions that occur when measuring the structure of an indoor environment using a radar system. Both generative and discriminant models can use ReLU as the activation function.

[0104] FIG. 11 illustrates the results of generating indoor environment maps after changing the training epochs to 5, 10, and 100, respectively, for different indoor environments. As shown in FIG. 11, it can be seen that as the training epochs progress, accurate ghost targets are removed and an indoor environment map with interpolated undetected areas can be generated.

[0105]

[0106] FIG. 14 is a schematic diagram illustrating a system configuration according to one embodiment of the present invention, and FIG. 15 is a diagram illustrating the architecture of a radar system according to one embodiment of the present invention.

[0107] Referring to FIG. 14, a system (1400) according to one embodiment of the present invention is configured to include a radar system (1410) and a computing device (1420).

[0108] The radar system (1410) is an FMCW radar system and can generate an initial indoor environment map based on the results of target detection using radar signals in an indoor environment.

[0109] The overall architecture of such a radar system (1410) is as shown in FIG. 15. Also, regarding the method for generating an initial indoor environment map of the radar system (1410), since it is the same as described with reference to FIG. 1 to FIG. 7, a redundant description will be omitted.

[0110] A computing device (1420) may be configured to include memory and a processor. The memory may store instructions for performing a method for creating an indoor environment structure map using radar and deep learning according to an embodiment of the present invention. Additionally, the processor may execute instructions stored in the memory. The instructions executed by the processor may each apply an initial indoor environment map to a pre-trained deep learning model to remove ghost targets from an initial indoor environment map generated based on radar signals for a specific indoor environment and generate an indoor environment map in which undetected areas are interpolated.

[0111]

[0112] An apparatus and method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable medium may be those specifically designed and configured for the present invention, or they may be those known and available to a person skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0113] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

[0114] The present invention has been described above with reference to its embodiments. Those skilled in the art will understand that the present invention may be implemented in modified forms without departing from the essential characteristics of the invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention.

Claims

1. (a) transmitting a radar transmission signal in an indoor environment, receiving a reception signal reflected from a target, and processing the reception signal to generate an initial indoor environment map for the indoor environment; and (b) A method for creating an indoor environment structure map using radar and deep learning, comprising the step of applying the above initial indoor environment map to a pre-trained deep learning model to remove ghost targets and generate an indoor environment map in which undetected areas are interpolated.

2. In Paragraph 1, A method for creating an indoor environment structure map using radar and deep learning, characterized in that the deep learning model is pre-trained using conditional data which is structural information about the indoor environment.

3. In Paragraph 1, A method for creating an indoor environment structure map using radar and deep learning, characterized in that the deep learning model is a CGAN (Conditional Generative Adversarial Network) based model.

4. In Paragraph 1, The above step (a) is, A step of processing the received signal to obtain a target detection result in the indoor environment; and The method includes the step of generating an initial indoor environment map containing a ghost target based on the detection results of the above target, A method for creating an indoor environment structure map using radar and deep learning, characterized in that the detection result of the above target is distance, azimuth, and elevation angle information for the target.

5. A computer-readable recording medium having program code for performing a method according to any one of claims 1 to 4.

6. Memory storing at least one instruction; and It includes a processor that executes instructions stored in the memory above, The instructions executed by the above processor are, respectively, (a) transmitting a radar transmission signal in an indoor environment and receiving a reception signal reflected from a target, and processing the reception signal to generate an initial indoor environment map for the indoor environment; and (b) A computing device characterized by performing the step of applying the above initial indoor environment map to a pre-trained deep learning model to remove ghost targets and generate an indoor environment map in which undetected areas are interpolated.

7. In Paragraph 6, A computing device characterized by the deep learning model being pre-trained using conditional data, which is structural information about the indoor environment.

8. A radar system that transmits a radar transmission signal in an indoor environment, receives a reception signal reflected from a target, and processes the reception signal to generate an initial indoor environment map of the indoor environment; and A system comprising a computing device that applies the above initial indoor environment map to a pre-trained deep learning model to remove ghost targets and generate an indoor environment map in which undetected areas are interpolated.

Images