Radar Data Processing with Deep Neural Networks (DNN)

Abstract

1. A computer-implemented method for detecting an object (1) in a monitoring area (200) of a monitoring device (100) having a first predetermined number of receivers (21) and a second predetermined number of transmitters (22), comprising: - dividing the raw signal data to generate divided signal data corresponding to each receiver (21) associated with each transmitter (22), to obtain a third predetermined number of divided sets of divided signal data equal to the first predetermined number multiplied by the second predetermined number; - inputting the segmented set into a deep neural network, the deep neural network having a receiving layer with a number of feature extraction branches equal to the first predetermined number and a transmitting layer with a number of feature extraction branches equal to the second predetermined number; - outputting the detection of said at least one object (1).

Description

[Technical Field]

[0001] The present invention relates to the field of methods for detecting objects in a surveillance area. [Background technology]

[0002] Methods for detecting objects in a surveillance area, particularly using neural networks, are known in the art. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent Application Publication No. 2021 / 396843 [Patent Document 2] U.S. Patent Application Publication No. 2021 / 318427 [Patent Document 3] U.S. Patent Application Publication No. 2021 / 255304 [Non-patent literature]

[0004] [Non-Patent Document 1] Deep Learning-Based Object Classification on Automotive Radar Spectra, Kanil Patel et al., Bosch Center for Artificial Intelligence, Renningen, Germany and Institute of Signal Processing and System Theory, University of Stuttgart, Stuttgart, Germany Summary of the Invention [Problem to be solved by the invention]

[0005] U.S. Patent Application Publication No. 2021 / 396843 (Patent Document 1) discloses inputting filtered data from millimeter-wave radar sensor signals into a deep neural network. This document discloses the use of filters such as band-pass filters and low-pass filters as preprocessing. This method is not unique because, as will be described later, radar signal processing research uses Fourier transforms (a type of filtering) to create distance maps (see the fourth prior art). The difference from distance map methods is the number of filters. Creating a distance map requires applying Fourier transforms three times, while Patent Document 1 discloses only a single filtering. Here, the lack of performance due to the small number of filters is compensated for by adding a convolutional layer to the neural network portion.

[0006] That is, the drawback of this prior art is that it cannot achieve better performance compared to other conventional methods due to the lack of a convolutional layer to reduce noise. Also, another problem in radar processing, namely, unclean data due to mixing signals from multiple transmitters, still exists. In the situation of dealing with training data that mixes signals from multiple transmitters, training to achieve better performance becomes more difficult.

[0007] U.S. Patent Application Publication No. 2021 / 318427 (Patent Document 2) discloses the idea of ​​filtering and centering (normalizing) radar data before inputting it into a neural network. While similar in methodology to the prior art discussed above, this second prior art document includes several applications, such as verifying and identifying users by their gestures. For filtering, this second prior art uses a frequency filter, specifically an 8th-order Butterworth high-pass filter with a cutoff frequency of 31,250 Hz, to remove low-frequency noise.

[0008] The second drawback of the prior art is that the filter must be designed by the person skilled in the art, and the person using the filter must specify the algorithm and each parameter (frequency) based on his / her own knowledge and experience. This method requires trial and error to change the algorithm and each parameter depending on the required application, which is time-consuming and laborious.

[0009] U.S. Patent Application Publication No. 2021 / 255304 (Patent Document 3) discloses a method for processing radar data without creating a distance map. However, what distinguishes this method from other prior art (the first and second prior arts described above) is that it converts radar signals from polar coordinate space to Cartesian coordinate space. For example, in an application where the final output is displayed on a map, Cartesian coordinate information is required, but raw radar data is represented in polar coordinate space (complex numbers). Therefore, this prior art uses conversion from a polar coordinate system to a Cartesian coordinate system as a preprocessing step for a neural network.

[0010] However, this third prior art has the following drawbacks: the Cartesian coordinate system is intuitively understandable because it is closer to human visual perception than the polar coordinate system, yet the Cartesian coordinate system is complex and time-consuming (although the process is simpler than creating a distance map, as described in the fourth prior art below).

[0011] "Deep Learning-Based Object Classification on Automotive Radar Spectra," Kanil Patel et al., Bosch Center for Artificial Intelligence (Renningen, Germany) and Institute of Signal Processing and System Theory, University of Stuttgart (Stuttgart, Germany) (Non-Patent Document 1) discloses the following. This article describes a representative technique that uses a distance map obtained from radar sensor data. In this article and other studies, the distance map is input into a deep neural network to perform object detection and classification. In practice, this method is convenient for handling and performance measurement, since the detection and classification results can be confirmed by visually inspecting the map (the map is consistent with human visual recognition). This advantage also makes it convenient to create training datasets (annotations). Therefore, this method is used when using machine learning methods.

[0012] However, while distance maps are easier to work with than raw data, a lot of information is lost in the process of creating the map. The most common way to create a map from radar data is to apply a Fourier transform several times (for example, to create a map using the object's velocity, perform a Fourier transform (FFT) three times). In this process, the Fourier transform limits the bandwidth from which the data is extracted, eliminating information such as reflected radar waves. Therefore, if you want to use all the information contained in the raw data, you need to work with the raw data without filtering.

[0013] In summary, in view of the prior art, there is a need to overcome the mentioned drawbacks and improve the accuracy of recognition or detection while simplifying and limiting the effort required to do so, i.e. reducing the computational effort and time and limiting the resources used.

[0014] Therefore, one of the objectives of the present invention is to overcome the drawbacks of the prior art mentioned above, in particular to improve the accuracy of recognition or detection while simplifying and limiting the effort required to achieve this. Radar signals (raw data) contain a lot of useful information, and there is a desire to make the most of it. While the Fourier transform (FFT) is a convenient way to extract meaningful information, it also removes a lot of information that could be useful for detection or recognition. This is why the present invention handles radar signals without filtering them at an early stage. [Means for solving the problem]

[0015] A first aspect of the present invention is 1. A computer-implemented method for detecting at least one object in a monitoring field of a monitoring device having a first predetermined number of receivers and a second predetermined number of transmitters, comprising: - receiving raw signal data from said monitoring device; - dividing the raw signal data to generate divided signal data corresponding to each receiver associated with each transmitter, thereby obtaining a third predetermined number of divided sets of divided signal data equal to the first predetermined number multiplied by the second predetermined number; - inputting the segmented set into a deep neural network (DNN), the deep neural network including at least one receiving layer having a number of feature extraction branches equal to the first predetermined number, and at least one transmitting layer having a number of feature extraction branches equal to the second predetermined number; - outputting the detection of the at least one object in the surveillance area based on the results of the processing step; The present invention relates to a computer-implemented method, including: [Effects of the Invention]

[0016] This provides an improved method that is more accurate and faster, yet simple to implement.

[0017] In other words, DNN transforms input data, extracts important data, reduces redundant data, and outputs the required results. Therefore, preparing a branch for each input from each receiver / transmitter is equivalent to performing feature extraction specialized for each receiver / transmitter.

[0018] In addition to object detection, the method can also classify images (e.g., scenes) and pixels, meaning that the method can be used for other applications besides object detection, such as classification and segmentation.

[0019] Raw data is the signal obtained from the instrument without any transformation or filtering.

[0020] Other features and advantages of the present invention will become more apparent from the following detailed description of specific, non-limiting examples of the invention, as illustrated in the accompanying drawings. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 shows a schematic diagram of a monitoring device and the functions of a computer-implemented method according to the invention for detecting at least one object. [Figure 2] A diagram showing a deep neural network DNN according to the present invention. [Figure 3] 1 shows raw signal data processed by the method according to the invention; [Figure 4] 1 shows raw signal data being segmented by the method according to the invention; [Figure 5] FIG. 1 shows the functioning of the method according to the invention. [Figure 6] Schematic diagram showing the phase difference between receivers in a monitoring device DETAILED DESCRIPTION OF THE INVENTION

[0022] FIG. 1 shows a schematic diagram of a monitoring device 100 and the functions of a computer-implemented method according to the present invention for detecting at least one object.

[0023] The monitoring device 100 is configured to monitor a monitoring area 200 in which at least one object 1 to be detected is (or may be) located.

[0024] The monitoring device 100 may be located in a vehicle. The object 1 may be, for example, a person, a child, an animal (e.g., any kind of animal such as a dog or a cat), a child seat, or any type of object, whether animate or physical.

[0025] The monitored area 200 may be the interior of a vehicle. The result (i.e., output) of the method may be to detect the presence of an object 1 (such as a person, animal, or object). The result may also be to identify the object 1, i.e., to state that the object 1 is a person, animal, or object, preferably even to state its type. The result may also be to identify where the object 1 is located within the monitored area 200. As a (non-limiting) example, the result may be: "There is an adult in the right seat of the first row of the car." "There is a child in the right seat of the second row of the car." "There was a child in the center seat of the second row of the car, sitting in a child car seat." "A child car seat was located in the left seat of the second row of the car, and no one was in the car seat." "There is a dog in the left seat of the third row of the car."

[0026] The monitoring area 200 may be outside the vehicle. Thus, the (non-limiting) results may be as follows: "There is a cyclist stopped 100 meters to the left of the road." "Pedestrians are moving 25 meters down the street." "The pedestrian's speed is 5km / h." "The pedestrian's speed is abnormally fast" (compared to normal walking speed) "A car is moving on the right side of the road" "A car is moving forward on the right side of the road"

[0027] The monitoring device 100 may comprise receivers (Rx or 21), in particular a first predetermined number of receivers, for example four receivers Rx1, Rx2, Rx3, Rx4, and transmitters Tx, in particular a second predetermined number of transmitters, for example three transmitters Tx1, Tx2, Tx3.

[0028] Each transmitter 22 (or Tx) may be spaced apart by a distance lambda (symbol λ), e.g., 5 mm. Each receiver Rx may be spaced apart by a distance lambda / 2, e.g.,

[0029] Each transmitter Tx emits one or more transmitted waves 11. The transmitted waves 11 may be reflected by the object 1 as one or more reflected waves 10. The reflected waves 10 may be received (i.e., captured) by each receiver Rx. The types of signals and raw signal data for each transmitter Tx and each receiver Rx are described in more detail below.

[0030] FIG. 2 shows a deep neural network DNN according to the present invention.

[0031] A deep neural network (DNN) includes at least one receiving layer (A layer) and at least one transmitting layer (B layer). The DNN may further include input nodes and output nodes. The DNN may optionally have additional layers (C layers) for any purpose.

[0032] The receiving layer has a number of feature extraction branches equal to a first predetermined number, and the transmitting layer has a number of feature extraction branches equal to a second predetermined number.

[0033] Each input node is configured to divide raw signal data from the monitoring device, in particular from each receiver, to generate divided signal data. That is, each input node is configured to divide the raw signal data to generate divided signal data corresponding to each receiver associated with each transmitter, thereby obtaining a third predetermined number of divided sets of divided signal data equal to the first predetermined number multiplied by the second predetermined number. The division of the raw signal data is further described below with reference to Figures 3 and 4. The divided sets (of raw signal data) are input to the receiving layer for processing.

[0034] The receive layer processes the split set of raw signal data, particularly for noise removal, while simultaneously performing feature extraction (FE) with training. The transmit layer processes the processed data from the receive layer while simultaneously performing feature extraction to correct the angle calculation by correctly grouping the signals from each transmitter. The processed data from the transmit layer is sent to the next layer (C layer), which is a fully connected layer, and then finally to the output node.

[0035] Each output node may have, for example, two branches (but is not limited to this number) and determine the result (e.g., class 1, class 2, class 3, passenger in seat number 1, etc.) output by the DNN. This may be, for example, result cleaning. For example, the transmission layer (B layer) outputs several possible results, and each output node determines the result.

[0036] FIG. 3 represents the raw signal data processed by the method according to the invention.

[0037] The raw signal data (top of the figure) is represented diagrammatically as a chirp signal within a (general) frame.

[0038] Each input node allows the division of the frame into identified chirp signals, each of which has a reference to both the associated receiver and transmitter. For example, it is possible to identify the first chirp signal as belonging to the first receiver Rx1 and the first transmitter Tx1 (i.e., the signal emitted from the first transmitter Tx1 and reflected by the target 1 towards the first receiver Rx1). The same applies to the other chirp signals, e.g., the seventh chirp signal from the left belongs to the third receiver Rx3 and the second transmitter Tx2 (i.e., the signal emitted from the second transmitter Tx2 and reflected by the target 1 towards the third receiver Rx3). That is, the number of chirp signals is equal to a third predetermined number (i.e., the first predetermined number multiplied by a second predetermined number). In this example, there are 12 chirp signals (3, the number of transmitters Tx1 to Tx3, multiplied by 4, the number of receivers Rx1 to Rx4).

[0039] FIG. 4 represents raw signal data being segmented by the method according to the invention.

[0040] That is, each input node allows for the individualization of each chirp signal, i.e., splitting the frame into single chirp signals, as mentioned above. The idea is to separate each chirp signal so that it can be sent to a corresponding branch in the receiving layer, as further explained in Figure 5.

[0041] FIG. 5 shows the functioning of the method according to the invention with each feature extraction (FE) branch.

[0042] The divided sets of chirp signals corresponding to each receiver and each transmitter are input to the receiving layer at the level of a corresponding branch. The processed data from the receiving layer is then input to the transmitting layer at the level of the corresponding branch. The output from the transmitting layer is then sent to the output node to output a clear result.

[0043] At each input node (shown in FIG. 2 or FIG. 5), as described above, each piece of divided data (chirp signal) is input to each branch.

[0044] From an implementation standpoint, it is necessary to prepare one or more nodes at the end of each branch, since each chirp signal consists of 128, 256, or other number of numeric data, depending on the sensor specifications or preprocessing.

[0045] In the preferred embodiment, each signal passes through both layers, which means that each signal has two class attributes.

[0046] That is, the first point is to avoid mixing signals from multiple transmitters. This means that it is preferable to use the attribute information of each transmitter Tx. The second point is to accurately calculate the angle by handling the signal from each receiver Rx, as shown in Figure 6.

[0047] In other words, handling two pieces of attribute information is necessary for angle calculation, as shown in FIG. 6 and the like below.

[0048] That is, in the prior art, all data are treated as the same and all data are input at once without clear separation. The present invention distinguishes each layer depending on the hardware structure (such as the number of receivers and the number of transmitters), and has better performance by making the number of feature extraction branches in the receiving layer equal to the number of receivers, and the number of feature extraction branches in the transmitting layer equal to the number of transmitters.

[0049] FIG. 6 shows a schematic diagram of the phase difference between receivers in a monitoring device.

[0050] This shows the phase difference between the signal reflected towards the first receiver Rx1 and the signal reflected towards the second receiver Rx2. The phase difference is calculated by using signals from the same transmitter at different receivers. Therefore, when considering this calculation using a DNN, two attribute information (Tx information and Rx information) is required.

[0051] It will, of course, be understood that improvements and / or modifications obvious to those skilled in the art can be made within the scope of the present invention as defined in the appended claims.

[0052] A brief summary of this embodiment of the computer-implemented method described above follows.

[0053] A first aspect of the present invention is 1. A computer-implemented method for detecting at least one object (1) in a monitoring area (200) of a monitoring device (100) having a first predetermined number of receivers (21) and a second predetermined number of transmitters (22), comprising: - receiving raw signal data from said monitoring device (100); - dividing the raw signal data to generate divided signal data corresponding to each receiver (21) associated with each transmitter (22), thereby obtaining a third predetermined number of divided sets of divided signal data equal to the first predetermined number multiplied by the second predetermined number; - inputting the segmented set into a deep neural network (DNN), the deep neural network including at least one receiving layer having a number of feature extraction branches equal to the first predetermined number, and at least one transmitting layer having a number of feature extraction branches equal to the second predetermined number; - outputting the detection of said at least one object (1) in said monitoring area (200) based on the result of said processing step; The present invention relates to a computer-implemented method, including:

[0054] This provides an improved method that is more accurate and faster, yet simple to implement.

[0055] In other words, the DNN transforms input data, extracts important data, reduces redundant data, and outputs the required results. Therefore, preparing each branch for input from each receiver (21) / transmitter (22) is equivalent to performing feature extraction specialized for each receiver (21) / transmitter (22).

[0056] In addition to detecting objects (1), it is also possible to classify images (scenes, etc.) and pixels. In other words, it can be used not only for object detection but also for classification and segmentation.

[0057] Raw data is the signal obtained from the instrument without any transformation or filtering.

[0058] Advantageously, the DNN has an input layer or input nodes.

[0059] This allows data to be fed into the DNN.

[0060] Advantageously, the raw signal is divided into several groups and the divided sets are sent to corresponding branches.

[0061] Beneficially, the DNN has an output layer or output node, which preferably has a number of neurons or branches equal to the number of seats (or available locations) in the vehicle.

[0062] It is beneficial if the DNN has both a branch part and a fully connected part, i.e. branches are used in the input part and then fully connected layers are used in the output part.

[0063] Advantageously, the transmitter (22) is a radar, preferably a radar transmitting millimeter waves.

[0064] Advantageously, the transmitter 22 is a wireless device that emits radio waves, which may be a sonar, LiDAR, light emitting element, radio transmitter, or any suitable device.

[0065] Advantageously, the receiver (21) is configured to detect the radiation by the transmitter (22) and / or re-radiation reflected by the object (1) to be detected (or any other object (1) within the monitoring area (200)).

[0066] Advantageously, the monitoring device (100) is located in a vehicle.

[0067] Advantageously, the surveillance area (200) is located within the vehicle.

[0068] Advantageously, said object (1) is a vehicle occupant, such as a driver, a passenger, an animal, or an object, such as a child seat.

[0069] This allows the detection of objects (1) inside the vehicle, and in particular, helps to prevent children or animals from being left in the vehicle while parking for a long time in a parking lot, etc. It also makes it possible to confirm the presence of children (babies) regardless of whether a child seat is installed or not.

[0070] Advantageously, the DNN, in particular the receiving layer and / or the transmitting layer, is trained, preferably by supervised learning.

[0071] It is possible to use a training dataset (radar data + labels (i.e., corrections, e.g., there are occupants in seats A and C)). Then the DNN is trained by following the labels.

[0072] This can improve the performance of the method in terms of accuracy and speed.

[0073] Radar uses millimeter waves, but after reception, the data is converted into digital data, regardless of the transmission method.

[0074] Millimeter waves are high frequency waves.

[0075] In other words, it can send large amounts of data, so it is used for data transfer in mobile phones, etc. The higher the frequency, the higher the resolution, but the shorter the transfer distance.

[0076] Advantageously, the monitoring device (100) is adapted to monitor outdoors.

[0077] Advantageously, the monitoring device (100) is adapted to monitor on the road (for example, seat occupancy at a bus stop).

[0078] Advantageously, the surveillance area (200) is outside the vehicle.

[0079] Advantageously, the monitoring device (100) is adapted to monitor a room.

[0080] Advantageously, each of said branches consists of one or more nodes (neurons) in a DNN architecture.

[0081] Advantageously, each branch of the receiving layer and each branch of the transmitting layer are paired (assigned) for the divided signal data corresponding to each receiver (21) associated with each transmitter (22).

[0082] The divided sets are input to the DNN as follows: divided signal data corresponding to each receiver (21) associated with each transmitter (22) is input to a pair of branches in the receiving and transmitting layers.

[0083] DNNs, especially feature extraction, transform input data to extract important data, reduce redundant data, and output the required results.

[0084] Therefore, preparing each branch for input from each receiver (21) / transmitter (22) is equivalent to performing feature extraction specifically for each receiver (21) / transmitter (22). [Industrial Applicability]

[0085] The present invention can be applied to detecting objects in a surveillance area. [Explanation of symbols]

[0086] 1. Target 21 Receiver 22 Transmitter 100 Monitoring equipment 200 monitoring area DNN Deep Neural Network

Claims

1. 1. A computer-implemented method for detecting at least one object in a monitoring field of a monitoring device having a first predetermined number of receivers and a second predetermined number of transmitters, comprising: - receiving raw signal data from said monitoring device; - dividing the raw signal data to generate divided signal data corresponding to each receiver associated with each transmitter, to obtain a third predetermined number of divided sets of divided signal data equal to the first predetermined number multiplied by the second predetermined number; - inputting said segmented set into a deep neural network (DNN), said deep neural network comprising at least one receiving layer having a number of feature extraction branches equal to said first predetermined number, and at least one transmitting layer having a number of feature extraction branches equal to said second predetermined number; - outputting the detection of said at least one object in said surveillance area based on the result of said processing step; A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the transmitter is a radar.

3. The computer-implemented method of claim 1 or 2, wherein the monitoring device is located in a vehicle.

4. The computer-implemented method of claim 1 , wherein the monitoring area is within the vehicle.

5. The computer-implemented method of claim 1 , wherein the object is a vehicle occupant, such as a driver, a passenger, an animal, or an object, such as a child car seat.

6. 6. The computer-implemented method according to claim 1, wherein the DNN, in particular the receiving layer and / or the transmitting layer, is trained, preferably by supervised learning.

Images