Deep neural network (DNN)-based radar data processing

Abstract

A computer-implemented method for detecting an object (1) in a monitoring area (200) of a monitoring device (100) having a first predetermined number of receivers (21) and a second predetermined number of transmitters (22), the method comprising the following steps: for each receiver (21) linked to each transmitter (22), segmenting raw signal data into segmented signal data, obtaining a third predetermined number of segmented sets of the segmented signal data, the third predetermined number being equal to the first predetermined number multiplied by the second predetermined number; inputting the segmentation set to a deep neural network having a receiver layer having a number equal to a first predetermined number of extraction branches and a transmitter layer having a number equal to a second predetermined number of feature extraction branches; a detection of the at least one object (1) is output.

Description

Technical Field

[0001] The present invention relates to the field of methods for detecting objects in a surveillance area. Background Art

[0002] Methods for detecting objects in a surveillance area are known in the art, in particular methods using neural networks.

[0003] List of citations

[0004] Patent Literature

[0005] Patent Document 1: US2021396843A1

[0006] Patent Document 2: US2021318427A1

[0007] Patent Document 3: US2021255304A1

[0008] Non-patent literature

[0009] Non-Patent Literature 1: Deep Learning-based Object Classification on Automotive Radar Spectra, Kanil Patel et al., Bosch Center for Artificial Intelligence, Renningen, Germany, and Institute for Signal Processing and Systems Theory, University of Stuttgart, Stuttgart, Germany.

[0010] Technical issues

[0011] Document US2021396843A1 discloses the input of filtered data of millimeter-wave radar sensor signals to a deep neural network. It discloses the use of filtering such as bandpass filtering and low-pass filtering as preprocessing. This method is not clear because, as explained later, research on radar signal processing uses Fourier transform (similar to filtering) to produce distance maps (see the fourth prior art). The difference from the distance map method is the number of filtering times. Although three Fourier transforms need to be applied to produce a distance map, the document only discloses one type of filtering. The performance deficiencies caused by less filtering are then compensated by adding convolutional layers to the neural network part.

[0012] In other words, the disadvantage of this existing technology is that, due to the lack of convolutional layers to reduce noise, it does not achieve better performance than other traditional methods. Furthermore, another problem in radar processing remains: unclean data caused by mixed signals from different transmitters. When processing training data containing mixed signals from multiple transmitters, achieving better training performance becomes even more difficult.

[0013] Document US2002318427A1 discloses a concept for filtering and centering (normalizing) radar data before inputting it into a neural network. While similar in methodology to the aforementioned prior art, this second prior art document includes several applications, such as user authentication and recognition through gestures. Regarding filtering, this second prior art document uses a frequency filter, specifically an eighth-order Butterworth high-pass filter with a cutoff frequency of 31250 Hz, to remove low-frequency noise.

[0014] The second disadvantage of the prior art is that the filtering must be designed manually. For example, those skilled in the art who use the technology should determine the algorithm and parameters (frequency) based on their own knowledge and experience. In this method, it is necessary to change the algorithm and parameters according to the application through trial and error, which consumes a lot of time and energy.

[0015] Document US2021255304A1 discloses a method for processing radar data without making a range map. However, the difference from other prior arts (the first prior art and the second prior art mentioned above) is that the radar signal is converted from polar coordinate space to Cartesian coordinate space. For example, although the original radar data is in polar coordinate space (complex number), Cartesian coordinate information is required in applications where the final output is displayed on a map. This is why the prior art uses polar to Cartesian coordinate conversion as a preprocessing of the neural network.

[0016] However, the third prior art has the following disadvantages: Cartesian coordinates are intuitively understandable because they are closer to human vision than polar coordinates (however, processing is simpler than creating a distance map described in the fourth prior art below), but are still complex and time-consuming.

[0017] The paper "Object Classification in Automotive Radar Spectra Based on Deep Learning" by Karnil Patel et al. from the Bosch Center for Artificial Intelligence in Lenningen, Germany, and the Institute for Signal Processing and Systems Theory at the University of Stuttgart in Stuttgart, Germany, describes a technique that uses distance maps derived from radar sensor data. This paper and other research inputs distance maps into deep neural networks for object detection and classification. This approach is simple to process and measure performance, as detection or classification results can be confirmed by visual inspection on a map (which is synchronized with human vision). This advantage also facilitates the creation of training datasets (annotations), making this approach suitable for use with machine learning methods.

[0018] However, while range maps are easier to process than raw data, much information is lost during the mapping process. The most common method for creating maps from radar data is to apply several Fourier transforms (for example, three Fast Fourier Transforms (FFTs) are used to create maps based on object velocity). This process removes information such as reflected radar waves because the Fourier transform limits the frequency bands extracted. Therefore, if you want to use all the information contained in the raw data, you need to process the raw data without any filtering.

[0019] That is, given the existing technology, there is a need to overcome the above shortcomings and improve recognition or detection accuracy while simplifying and limiting the workload to achieve the same goal. That is, there is a need to reduce the computational workload and time and limit resources. Summary of the Invention

[0020] Therefore, the present invention aims to overcome the shortcomings of the aforementioned prior art, specifically to improve recognition or detection accuracy while simplifying and limiting the workload required to achieve the same goal. In other words, radar signals (raw data) contain a wealth of useful information, and it is desirable to fully utilize this information. The Fourier transform (FFT) is a convenient method for extracting meaningful information, but it also removes much information that could be useful for detection or recognition. This is why the present invention processes radar signals without any filtering at the initial stage.

[0021] Solutions to the problem

[0022] A first aspect of the present invention relates to a computer-implemented method for detecting at least one object in a monitoring area of ​​a monitoring device, the monitoring device having a first predetermined number of receivers and a second predetermined number of transmitters, the method comprising the following steps:

[0023] receiving raw signal data from the monitoring device,

[0024] for each receiver linked to each transmitter, splitting the original signal data into split signal data to have a third predetermined number of split sets of split signal data, the third predetermined number being equal to the first predetermined number multiplied by the second predetermined number,

[0025] Inputting the segmented set into a deep neural network (DNN), wherein the deep neural network includes at least one receiver layer and at least one transmitter layer, the receiver layer having a number of feature extraction branches equal to the first predetermined number, and the transmitter layer having a number of feature extraction branches equal to the second predetermined number,

[0026] Based on a result of the processing step, a detection of the at least one object in the surveillance area is output.

[0027] Advantageous Effects of the Invention

[0028] The present invention allows an improved method to be provided which has a higher accuracy and a higher speed and has a simplified structure.

[0029] That is, DNN transforms the input data to extract important data and reduce redundant data to output the desired result. Therefore, preparing each branch for the input from each receiver / transmitter is equivalent to performing feature extraction for each receiver / transmitter.

[0030] This allows not only object detection but also classification of images (scenes, etc.) or pixels. In other words, this method can be used not only for object detection but also for classification, segmentation, and other purposes.

[0031] Raw data is the signal from the device without any conversion / filtering. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Other features and advantages of the present invention will appear more clearly from the following detailed description of certain non-limiting examples of the invention illustrated by the accompanying drawings.

[0033] Figure 1 The diagram shows a schematic arrangement of a monitoring device according to the invention and the functionality of a computer-implemented method for detecting at least one object.

[0034] Figure 2 Denotes a deep neural network DNN according to the present invention.

[0035] Figure 3 represents the raw signal data to be processed by the method according to the present invention.

[0036] Figure 4 represents the original signal data segmented by the method according to the present invention.

[0037] Figure 5 Represents the functionality of the method according to the invention.

[0038] Figure 6 Schematic diagram showing the phase difference between the receivers of monitoring equipment. DETAILED DESCRIPTION

[0039] Figure 1 Schematic arrangement of a monitoring device 100 according to the present invention and the functionality of a computer-implemented method for detecting at least one object are shown.

[0040] The monitoring device 100 is arranged to monitor a monitoring area 200 in which at least one object 1 to be detected is (or can be) placed.

[0041] The monitoring device 100 may be placed in a vehicle, and the object 1 may be, for example, a person, a child, an animal (such as a dog, a cat, or any kind of animal), a child seat, or any type of living object or item.

[0042] The monitoring area 200 may be the interior of a vehicle, and the result (or output) of the method may be the detection of the presence of an object 1 (a person, animal, object, etc.). In addition, the result of the method may be the identification of the object 1, that is, the statement of whether the object 1 is a person, an animal, or an object, preferably indicating which type it is. Furthermore, the result of the method may be the identification of the position of the object 1 in the monitoring area 200. For example (and not limitation), the result may be "an adult is in the right seat of the first row of the car", or "a child is in the right seat of the second row of the car", or "a child is in the center seat of the second row of the car and is in a child seat", or "a child seat is placed on the left seat of the second row of the car, and the child seat is empty", or "a dog is in the left seat of the third row of the car".

[0043] Monitoring area 200 may also be outside the vehicle. Thus, the results (not limited to) may be "a cyclist is stopped 100 meters to the left of the road," or "a pedestrian is moving 25 meters in the middle of the street," or "the pedestrian's speed is 5 km / h," or "the pedestrian's speed is abnormally high (compared to a typical walking speed)," or "a car is moving on the right side of the road," or "a car is moving in a forward direction on the right side of the road."

[0044] The monitoring device 100 can be equipped with receivers (Rx or 21), specifically equipped with a first specified number of receivers, such as four receivers Rx1, Rx2, Rx3, Rx4, and the monitoring device 100 can be equipped with transmitters Tx, specifically a second predetermined number of transmitters, such as three transmitters Tx1, Tx2, Tx3.

[0045] The transmitters 22 (or Tx) may be spaced apart by a distance λ (symbol λ). The distance λ is, for example, 5 mm. The receivers Rx may be spaced apart by, for example, λ divided by 2.

[0046] The transmitter Tx transmits one or more transmission waves 11, which can be reflected by the object 1 as one or more reflection waves 10. The reflection waves 10 can be received (or captured) by the receiver Rx. The signal types and raw signal data of the transmitter Tx and the receiver Rx are explained in more detail below.

[0047] Figure 2 Denotes a deep neural network DNN according to the present invention.

[0048] A deep neural network (DNN) has at least one receiver layer (layer A) and at least one transmitter layer (layer B). The DNN may also include input nodes and output nodes. The DNN may optionally have additional layers (layer C) for any purpose.

[0049] The receiver layer has a number of feature extraction branches equal to a first predetermined number. The transmitter layer has a number of feature extraction branches equal to a second predetermined number.

[0050] The input node is arranged to split the original signal data from the monitoring device, in particular from the receiver, into split signal data. That is, the input node is arranged to split the original signal data into split signal data for each receiver and link to each transmitter to have a split set of a third predetermined number of split signals, the third predetermined number being equal to the first predetermined number multiplied by the second predetermined number. Figure 3 and Figure 4 Further explaining the segmentation of the original signal data, the segmented set (of the original signal data) is input to the receiver layer for processing.

[0051] The receiver layer performs feature extraction (FE) while training to process segmented sets of raw signal data, specifically denoising the data. The transmitter layer performs feature extraction while processing the processed data from the receiver layer to correct the angle calculation by correctly grouping the signals from the transmitter. The processed data from the transmitter layer is fed into the next layer (Layer C), which is a fully connected layer, and then ultimately fed to the output node.

[0052] The output node can have, for example, two branches (but is not limited to this number), and ultimately determines the result to be output by the DNN (e.g., category 1, category 2, category 3, the passenger in seat 1, etc.). This can be an example of cleaning the results. For example, the transmitter layer (layer B) outputs the probabilities of some results, and the output node finally gives the results.

[0053] Figure 3 represents the raw signal data to be processed by the method according to the present invention.

[0054] The original signal data (upper part of the figure) is schematically represented as a chirp signal in a (common) frame.

[0055] The input node allows the frame to be segmented into identified chirp signals by making a double reference to the receivers and transmitters involved. For example, it can then be identified that the first chirp signal belongs to the first receiver Rx1 and the first transmitter Tx1 (that is, the signal emitted from the first transmitter Tx2 and reflected by the object 1 to the first receiver Rx2). The same applies to the other chirp signals, for example, the seventh chirp signal from the left belongs to the third receiver Rx3 and the second transmitter Tx2 (that is, the signal emitted from the second transmitter Tx2 and reflected by the object 1 to the third receiver Rx3). That is, the number of chirp signals is equal to the third predetermined number (that is, equal to the first predetermined number multiplied by the second predetermined number). In the given example, there are twelve chirp signals (three transmitters Tx1 to Tx3 multiplied by four receivers Rx1 to Rx4).

[0056] Figure 4 represents the original signal data segmented by the method according to the present invention.

[0057] That is, the input node allows to personalize the chirp signal, which means that the frame is divided into individual chirps as described above. The idea is to separate each chirp signal so that each chirp signal can be sent to the corresponding branch of the receiver layer, as in Figure 5 As further explained in .

[0058] Figure 5 ] represents the functionality of the feature extraction (FE) branch of the method according to the invention.

[0059] The segmented set of chirp signals corresponding to each receiver and each transmitter is input to the receiver layer at the level of the corresponding branch. The processed data from the receiver layer is then input to the transmitter layer at the level of the corresponding branch. The output of the transmitter layer is then sent to the output node to output the clean result.

[0060] In the input node (such as Figure 2 or Figure 5 As shown), as described above, each segmented data (chirp signal) is input to each branch.

[0061] From an implementation perspective, it is necessary to prepare one or more nodes at the edge of each branch because each chirp signal consists of numerical data of 128, 256 or other numbers, depending on sensor specifications or preprocessing.

[0062] In a preferred embodiment, each signal will pass through two layers, which means that each signal must have two types of properties.

[0063] That is, the first point is to avoid mixing signals from different transmitters, which means that it is desirable to use the properties of the transmitter Tx. In addition, the second point is to process the signal in each receiver Rx to perform Figure 6 The exact angle calculation is shown.

[0064] That is, processing two attributes is necessary for the following Figure 6 That angle calculation is necessary.

[0065] In other words, the prior art processes all data in the same way and inputs all data at once without clear separation. The present invention achieves better performance by differentiating layers based on hardware structure (such as the number of receivers and the number of transmitters), where the hardware structure has a receiver layer with a number of feature extraction branches equal to the number of receivers and a transmitter layer with a number of feature extraction branches equal to the number of transmitters.

[0066] Figure 6 Schematic diagram showing the phase difference between monitoring equipment receivers.

[0067] This is used to illustrate the phase difference between the reflected signals of the first receiver Rx1 and the second receiver Rx2. The phase difference is calculated by using signals from the same transmitter and at different receivers. Therefore, if this calculation is considered in a DNN, two attributes (Tx and Rx information) are required.

[0068] It will of course be appreciated that obvious modifications and / or alterations may be applied by those skilled in the art, which will still fall within the scope of the present invention as defined by the appended claims.

[0069] The following briefly describes an overview of this embodiment of the above-mentioned computer-implemented method.

[0070] A first aspect of the present invention relates to a computer-implemented method for detecting at least one object 1 in a monitoring area 200 of a monitoring device 100, the monitoring device 100 having a first predetermined number of receivers 21 and a second predetermined number of transmitters 22, the method comprising the following steps:

[0071] Receive raw signal data from the monitoring device 100,

[0072] for each receiver 21 linked to each transmitter 22, splitting the original signal data into split signal data to have a third predetermined number of split sets of split signals, the third predetermined number being equal to the first predetermined number multiplied by the second predetermined number,

[0073] Inputting the segmented set into a deep neural network DNN, wherein the deep neural network has at least one receiver layer and at least one transmitter layer, the receiver layer has a number of feature extraction branches equal to the first predetermined number, and the transmitter layer has a number of feature extraction branches equal to the second predetermined number,

[0074] Based on the results of the processing steps, a detection of at least one of the objects 1 in the monitoring area 200 is output.

[0075] This allows an improved method to be provided with higher accuracy and higher speed, and with a simplified structure.

[0076] That is, the DNN transforms the input data to extract important data and reduce redundant data to output the desired result. Therefore, preparing each branch for the input from each receiver 21 / transmitter 22 is equivalent to performing feature extraction for each receiver 21 / transmitter 22.

[0077] This not only allows detection of objects 1 but also allows classification of images (scenes, etc.) or pixels. That is, this method can be used not only for object detection but also for classification, segmentation, and other purposes.

[0078] Raw data is the signal from the device without any conversion / filtering.

[0079] Advantageously, the DNN has an input layer or input nodes.

[0080] This allows for input of data into the DNN.

[0081] Advantageously, the original signal is divided into a plurality of groups, and the divided sets are sent to corresponding branches.

[0082] Advantageously, the DNN has an output layer or output nodes, preferably with a number of neurons or branches equal to the number of seats (or available positions) in the vehicle.

[0083] Advantageously, the DNN has both a branching part and a fully connected part. That is, branches are used in the input part and then a fully connected layer is used in the output part.

[0084] Advantageously, the transmitter 22 is a radar, preferably transmitting millimeter waves.

[0085] Advantageously, the transmitter 22 is a wireless device that transmits waves. It can be a sonar, a lidar, a light transmitter or a radio transmitter, or any other suitable device.

[0086] Advantageously, the receiver 21 is arranged to detect emissions of the transmitter 22 and / or re-emissions reflected by the object 1 to be detected (or by any other object 1 in the surveillance area 200 ).

[0087] Advantageously, the monitoring device 100 is placed in a vehicle.

[0088] Advantageously, the monitoring area 200 is inside the vehicle.

[0089] Advantageously, the object 1 is an occupant of the vehicle, such as a driver, a passenger, an animal, or an object, such as a child seat.

[0090] This allows detection of objects 1 in vehicles, in particular, for example, to avoid leaving children or animals in cars parked in parking lots for long periods of time. In addition, this allows confirmation of the presence of children (infants) regardless of whether a child seat is installed.

[0091] Advantageously, the DNN, in particular the receiver layer and / or the transmitter layer, is preferably trained by supervised learning.

[0092] You can use a training dataset (radar data + labels (i.e. corrections, such as whether seat A and seat C have passengers)) and then train a DNN based on the labels.

[0093] This allows improving the performance of the method in terms of, for example, accuracy and speed.

[0094] Radar uses millimeter waves, but upon reception, the data is converted to digital data regardless of the transmission method.

[0095] Millimeter waves are high-frequency waves.

[0096] This means that it can transmit large amounts of data, so it's used, for example, for data transmission on mobile devices. The higher the frequency, the higher the resolution. Conversely, the higher the frequency, the shorter the transmission distance.

[0097] Advantageously, the monitoring device 100 is arranged to monitor outdoors.

[0098] Advantageously, the monitoring device 100 is arranged to monitor a road (eg bus stop seat occupancy).

[0099] Advantageously, monitoring area 200 is outside the vehicle.

[0100] Advantageously, the monitoring device 100 is arranged to monitor the interior of a room.

[0101] Advantageously, each of the above branches consists of one or more nodes (neurons) in the DNN architecture.

[0102] Advantageously, each branch of the receiver layer and each branch of the transmitter layer is paired (assigned) with corresponding split signal data for each receiver 21 linked to each transmitter 22 .

[0103] Input of the segmented set to the DNN is done as follows: the segmented signal data corresponding to each receiver 21 linked to each transmitter 22 is input to the paired branches of the receiver layer and the transmitter layer.

[0104] DNN, especially feature extraction, transforms input data to extract important data and reduces redundant data to output the desired results.

[0105] Therefore, preparing individual branches for input from each receiver 21 / transmitter 22 is equivalent to performing feature extraction for each receiver 21 / transmitter 22 .

[0106] Industrial Availability

[0107] The present invention can be applied to detecting objects in a surveillance area.

[0108] Explanation of symbols

[0109] 1 object, 21 receivers, 22 transmitters, 100 monitoring devices, 200 monitoring areas, DNN deep neural network.

Claims

1. A computer-implemented method for detecting at least one object in a monitoring area of ​​a monitoring device, the monitoring device having a first predetermined number of receivers and a second predetermined number of transmitters, wherein: The method comprises the following steps: receiving raw signal data from the monitoring device, for each receiver linked to each transmitter, splitting the original signal data into split signal data to have a third predetermined number of split sets of split signal data, the third predetermined number being equal to the first predetermined number multiplied by the second predetermined number, Inputting the segmented set into a deep neural network (DNN), wherein the deep neural network includes at least one receiver layer and at least one transmitter layer, the receiver layer having a number of feature extraction branches equal to the first predetermined number, and the transmitter layer having a number of feature extraction branches equal to the second predetermined number, Based on a result of the processing step, a detection of the at least one object in the surveillance area is output.

2. The computer-implemented method of claim 1 , wherein: The transmitter is a radar.

3. The computer-implemented method according to claim 1 or 2, wherein: The monitoring device is placed in the vehicle.

4. The computer-implemented method according to any one of claims 1 to 3, wherein: The monitoring area is the interior of the vehicle.

5. The computer-implemented method according to any one of claims 1 to 4, wherein: The object is a passenger or an object of the vehicle. The passenger is, for example, a driver, a passenger, or an animal. The object is, for example, a child seat.

6. The computer-implemented method according to any one of claims 1 to 5, wherein: The deep neural network, in particular the receiver layer and / or the transmitter layer, is preferably trained by supervised learning.

Images