Method for training neural network for detecting object and method for detecting object by means of neural network

Abstract

The invention relates to a method of training a neural network (7) for detecting an object, obtaining geometric dimensions of a test object from a class of objects, images of the test object being generated by a plurality of cameras (21) over a period of time; generating an occupancy map according to the obtained geometric dimension and the generated image; transmitting a radar signal by the radar device (25) and receiving a radar signal reflected by the test object; mixing the transmitted radar signal and the received radar signal in a complex baseband to form a mixed signal; calculating a complex four-dimensional mixed frequency spectrum of the mixed signal; calculating a first complex two-dimensional sub-spectrum and a second complex two-dimensional sub-spectrum according to the complex four-dimensional mixed spectrum; fusing the occupation map and the sub-spectrum into training data; and inputting the training data into the neural network (7). The invention also relates to a method for detecting an object by means of a neural network (7) to which training data has been previously input.

Description

Technical Field

[0001] The invention relates to a method for training a neural network for detecting an object. In the process, training data is generated and input into the neural network. The invention also relates to a method for detecting an object by means of a neural network. Background Art

[0002] In the image processing of camera systems, it has become common practice to use neural networks to identify objects in images, and excellent results have been achieved. In the field of neural networks for locating objects in images, convolutional neural networks (CNN for short) with autoencoder structures are particularly known in the prior art. In order to successfully locate and classify objects using neural networks, the prerequisite is to train the neural networks accordingly.

[0003] Convolutional networks contain different convolutional layers, which together represent the intelligence of the neural network. They include input layers and output layers. These layers are connected to each other through mathematical convolution operations. In image processing, an image is input to the input layer, and a map with the locations of objects on the image is output at the output layer.

[0004] To identify objects from radar data, algorithms such as the Constant False Alarm Rate (CFAR) algorithm are often used, but they give suboptimal results under many conditions. Algorithms like CFAR follow a strict pattern to generate their output and rely on pre-set parameters. If these parameters are not chosen correctly, the results can be much worse than originally expected. Current techniques for image processing using neural networks have shown that neural networks are able to locate and classify objects more independently of the state of the existing image.

[0005] A method for training a neural network is known from European Patent EP 3 690 727 A1. In this method, a camera and a radar are used together.

[0006] An evaluation device, a training system and a training method for segmenting radar images of an environment are known from German patent DE 10 2018 203 684 A1.

[0007] From German patent DE 11 2021 000 135 T5 a system and a method are known which relate to machine learning-based sensor fusion for autonomous machine applications.

[0008] A device and a method for generating validated training data for a self-learning system are known from German patent DE 10 2019 219 894 A1.

[0009] From European Patent EP 3 832 341 A1 a neural network for detecting obstacles is known which can be used in autonomous vehicles and which uses radar sensors. Summary of the invention

[0010] The technical problem to be solved by the present invention is to further improve a method for training a neural network for detecting an object, and a method for detecting an object by means of a neural network.

[0011] This problem is solved by a method for training a neural network for detecting an object having the features of claim 1. Advantageous embodiments and improvements are the subject matter of the dependent claims. This problem is also solved by a method for detecting an object by means of a neural network having the features of claim 12. Advantageous embodiments and improvements are the subject matter of the dependent claims.

[0012] The present invention proposes a method for training a neural network for detecting an object. Here, the geometric dimensions of a test object from an object category are obtained. Within a time period, (photographed) images of the test object are generated by multiple cameras. An occupancy map is generated based on the acquired geometric dimensions and the generated images. A radar signal is emitted by a radar device, and a radar signal reflected by the test object is received. The emitted radar signal and the received radar signal are mixed into a complex baseband to become a mixed signal, and a complex four-dimensional mixed spectrum of the mixed signal is calculated. A first complex two-dimensional sub-spectrum and a second complex two-dimensional sub-spectrum are calculated from the complex four-dimensional mixed spectrum. The occupancy map and the sub-spectrum are fused into training data, and the training data is input into the neural network.

[0013] The training data contains enough mixed spectra that contain radar images and are associated with information about the location of the test object and the object category to which the test object belongs. This information is called the "ground truth". When generating an occupancy map, cells containing test objects are assigned higher values, while free cells are assigned lower values. From the corresponding occupancy map, it is possible to know the probability that a test object of a specific object category occupies a position in the field of view of the radar device. The most likely test object can be extracted through a hard decision threshold.

[0014] According to an advantageous embodiment of the invention, the mixed spectrum contains information about the distance, azimuth, elevation and radial velocity of the test object. The radial velocity is determined by the frequency shift between the transmitted radar signal and the reflected radar signal. The frequency shift is caused by the Doppler effect of the moving object.

[0015] According to an advantageous embodiment of the invention, the first sub-spectrum contains information about the distance and azimuth of the test object, and the second sub-spectrum contains information about the distance and radial velocity of the test object.

[0016] According to an advantageous embodiment of the invention, the first sub-spectrum contains a first radar image with information about the magnitude (size) of the distance and azimuth of the test object. The first sub-spectrum also contains a second radar image with information about the distance and azimuth phase of the test object. The second sub-spectrum contains a third radar image with information about the distance and radial velocity of the test object. The second sub-spectrum also contains a fourth radar image with information about the distance and radial velocity phase of the test object. The radar images are preferably presented in polar coordinates.

[0017] According to an advantageous embodiment of the invention, the test object moves during the time period. Therefore, a plurality of different images of the test object at different positions and with different orientations are generated, and the corresponding mixed spectra are calculated. The quality of the training data can be improved by a larger number of different images and the corresponding mixed spectra.

[0018] According to a preferred embodiment of the present invention, the occupancy map is first generated in a Cartesian coordinate system, and then the Cartesian coordinates are transformed into polar coordinates. Here, before the occupancy map and the sub-spectrum are fused into training data, the Cartesian coordinates are transformed into polar coordinates. Thus, the compatibility of the occupancy map and the sub-spectrum (whose radar image is also presented in polar coordinates) is achieved.

[0019] According to an advantageous improvement of the invention, before generating the image, a mark is set on the test object so that the mark is visible in the generated image. If such a mark is set on the test object, the six-dimensional pose of the test object can be calculated, which includes the position of the test object and the attitude / orientation of the test object.

[0020] According to an advantageous embodiment of the invention, the camera is designed as an infrared camera and / or the marker is designed as an infrared marker. The camera and the marker are part of a position detection system. In particular, providing such a marker on the test object allows the six-dimensional pose of the test object to be accurately calculated, which includes the position of the test object and the pose of the test object.

[0021] According to a preferred embodiment of the present invention, the pose of the test object is calculated from the image, and the pose includes the position of the test object and the pose of the test object. The calculated pose is discretized and integrated into the occupancy map.

[0022] According to a preferred development of the invention, these method steps are repeated for at least one further test object from another object class. In the process, the occupancy map and / or the training data are assigned to the respective object class. Such object classes are, for example, persons, forklifts or autonomous transport vehicles.

[0023] According to a preferred embodiment of the present invention, the neural network is designed as a convolutional network, which has an input layer, an output layer and a plurality of convolutional layers. In particular, the neural network is preferably designed as a convolutional neural network (CNN) of an autoencoder structure. These layers are arranged in sequence and connected to each other through mathematical convolution operations. A convolution operation is performed once from one layer to the next layer. The convolution operator used to perform these convolution operations is determined by processing the training data input to the neural network.

[0024] A method for detecting an object by means of a neural network to which training data has previously been input is also proposed. Here, training data has been input into the neural network using the method for training a neural network according to the present invention. A radar signal is emitted by a radar sensor, and a radar signal reflected by an object is received. The emitted radar signal and the received radar signal are mixed into a mixed signal, and a mixed spectrum of the mixed signal is calculated. Input data including the mixed spectrum is input into the neural network. The input data are processed in the neural network. The object and the position of the object are detected by the neural network. The object class of the detected object and the position of the detected object are output by the neural network as output data.

[0025] The more independent dimensions that are provided as input data to the neural network, the more effective the classification of objects can be by means of clear / univocal features. Clear features / signatures in the mixed spectrum enable reliable classification.

[0026] According to a preferred embodiment of the present invention, the neural network is designed as a convolutional network, which has an input layer, an output layer and a plurality of convolutional layers. A convolution operation is performed from one layer to the next layer. The layers are arranged in sequence and connected to each other through mathematical convolution operations. A convolution operation is performed from one layer to the next layer.

[0027] According to a preferred embodiment of the present invention, the calculated mixed spectrum includes at least the range and azimuth measured by the radar. The first input data including the range measured by the radar is input to the neural network. The second input data including the azimuth measured by the radar is input to the neural network. The first input data and the second input data constitute a first complex image composed of complex data. Therefore, the first complex image includes two real images / simple images including magnitude and phase.

[0028] According to an advantageous improvement of the present invention, the calculated mixed spectrum includes at least the range and radial velocity measured by the radar. The neural network is input with third input data including the range measured by the radar. The neural network is input with fourth input data including the radial velocity measured by the radar. The third input data and the fourth input data form a second complex image composed of complex data. Therefore, the second complex image includes two real images including magnitude and phase.

[0029] The present invention is not limited to the feature combinations described in the claims. For a person skilled in the art, other meaningful claim combination possibilities can be derived from the features of the claims and / or the features of the individual claims and / or the features of the specification and / or the drawings, in particular from the inventive task and / or the task proposed by comparison with the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The invention is described in more detail below with the aid of the accompanying drawings. The invention is not limited to the embodiments shown in the accompanying drawings. These drawings only illustrate the subject matter of the invention in a schematic manner. Among them:

[0031] Figure 1 A schematic diagram showing an apparatus for acquiring training data;

[0032] Figure 2 A schematic diagram showing a neural network;

[0033] Figure 3 A schematic diagram showing input data to a neural network;

[0034] Figure 4 Schematic diagram showing the output data of a neural network. DETAILED DESCRIPTION

[0035] Figure 1A schematic diagram of a system for obtaining training data for a neural network 7 is shown. The system has a measuring region 40 and a radar region 42. The measuring region 40 and the radar region 42 substantially overlap. A test object, not shown here, is located in the measuring region 40 and in the radar region 42.

[0036] The system includes a radar device 25. The radar device 25 is arranged so that a test object located in a radar area 42 can be detected by the radar device 25. The radar area 42 is designed in a sector shape. The radar device 25 is arranged at the vertex / center of the sector.

[0037] The radar device 25 transmits a radar signal and receives a radar signal reflected by a test object. The radar device 25 has a multiplier. The transmitted radar signal and the received radar signal are mixed into a complex baseband by the multiplier to become a mixed signal. The radar device 25 also calculates a complex four-dimensional mixed spectrum of the mixed signal. The mixed spectrum is calculated by discrete Fourier transform of the sampled raw data of the mixed signal.

[0038] The radar device 25 has a two-dimensional MIMO antenna array. The transmitted radar signal has a frequency-modulated continuous wave (FMCW) modulation device. Therefore, the calculated mixed spectrum is four-dimensional and contains information about the distance, azimuth, elevation and radial velocity of the test object reflecting the radar signal.

[0039] A first complex two-dimensional sub-spectrum and a second complex two-dimensional sub-spectrum are calculated from the complex four-dimensional mixed spectrum. The first sub-spectrum contains information about the distance and azimuth of the test object, and the second sub-spectrum contains information about the distance and radial velocity of the test object.

[0040] The first sub-spectrum contains a first radar image with information about the magnitude (magnitude) of the distance and azimuth of the test object. The first sub-spectrum also contains a second radar image with information about the distance and azimuth phase of the test object. The second sub-spectrum contains a third radar image with information about the distance and radial velocity of the test object. The second sub-spectrum also contains a fourth radar image with information about the distance and radial velocity phase of the test object. These radar images are presented in the form of polar coordinates.

[0041] The system includes a plurality of cameras 21 for generating images. Six cameras 21 are provided here. The cameras 21 are arranged so that the test object located in the measuring area 40 can be detected by all cameras 21. The measuring area 40 is designed as a rectangle. The cameras 21 are arranged at the corners and on the side lines of the rectangle. The cameras 21 are designed as infrared cameras and are part of the position detection system.

[0042] The system further includes a digital calculator 32 and a processing unit 34. The camera 21 is connected to the processing unit 34 and transmits the generated image to the processing unit 34. The radar device 25 is also connected to the processing unit 34 and transmits data to the processing unit 34. The processing unit 34 is connected to the digital calculator 32 and transmits data to the digital calculator 32.

[0043] In order to obtain training data for the neural network 7, a test object is first selected from an object category. Object categories include, for example, a person, a forklift, or an autonomous transport vehicle. Therefore, the selected test object is, for example, a person, a forklift, or an autonomous transport vehicle.

[0044] First, the geometric dimensions of the test object are obtained. In particular, the length, width and height of the test object are measured. In addition, marks are set on the test object. These marks are designed as infrared marks. As mentioned above, the camera 21 is designed as an infrared camera. These marks are set on the test object so that they are visible in the image subsequently generated by the camera 21.

[0045] The acquisition of training data for the neural network 7 with the aid of the selected test object takes place within a predefined time period. Within this time period, the test object is moved in an area that is within the measuring area 40 and within the radar area 42. If necessary, the test object moves autonomously within this area within this time period.

[0046] During this time period, images of the test object are generated by the camera 21. The pose of the test object is calculated from these images. The pose is six-dimensional and includes the position and attitude of the test object.

[0047] Based on the previously acquired geometry of the test object and the generated image, an occupancy map is generated. The calculated pose is integrated into the occupancy map. The occupancy map is assigned to the object class of the selected test object. The occupancy map is first generated in Cartesian coordinates, and then the Cartesian coordinates are transformed into polar coordinates.

[0048] During the time period, a radar signal is simultaneously transmitted by the radar device 25 and a radar signal reflected by the test object is received. The transmitted radar signal and the received radar signal are mixed into a mixed signal. A complex four-dimensional mixed spectrum of the mixed signal is also calculated. A first complex two-dimensional sub-spectrum and a second complex two-dimensional sub-spectrum are calculated from the complex four-dimensional mixed spectrum. The sub-spectrum contains a radar image presented in polar coordinate form.

[0049] The occupancy map and the sub-spectra are then fused to form training data. The training data are assigned to the corresponding object class of the selected test object. The training data thus obtained are input into the neural network 7.

[0050] The above-described method steps for obtaining training data for the neural network 7 are repeated for other test objects from other object classes. In the process, test objects are selected from other object classes. The above-described method steps for obtaining training data for the neural network 7 are also performed without real test objects, but in free space. In this case, the occupancy map and the training data are assigned to the corresponding object class or to free space.

[0051] Figure 2 A schematic diagram of a neural network 7 is shown. The neural network 7 is designed as a convolutional network. Currently, the neural network 7 has an input layer 6, a first convolutional layer 11, a second convolutional layer 12, a third convolutional layer 13, a fourth convolutional layer 14, a fifth convolutional layer 15, a sixth convolutional layer 16, a seventh convolutional layer 17 and an output layer 9.

[0052] Input data 1, 2, 3, 4 are input to the input layer 6 of the neural network 7. The input layer 6, the convolutional layers 11, 12, 13, 14, 15, 16, 17 and the output layer 9 are arranged in sequence. A convolution operation is performed once from one layer to the next layer. Output data 51, 52, 53, 54 are output from the output layer 9 of the neural network 7.

[0053] Furthermore, an intermediate connection 8 is provided between the first convolutional layer 11 and the seventh convolutional layer 17. An intermediate connection 8 is also provided between the second convolutional layer 12 and the sixth convolutional layer 16. An intermediate connection 8 is also provided between the third convolutional layer 13 and the fifth convolutional layer 15. The intermediate connection 8 forms a direct connection between two layers, and no convolution operation is performed via the intermediate connection 8. The intermediate connection 8 is used to speed up the training phase. This is a heuristic method.

[0054] Each layer forms a three-dimensional matrix of individual pixels. Here, the size of the input layer 6 is 4×128×128 pixels. The size of the first convolutional layer 11 is 16×64×64 pixels. The size of the second convolutional layer 12 is 32×32×32 pixels. The size of the third convolutional layer 13 is 64×16×16 pixels. The size of the fourth convolutional layer 14 is 128×8×8 pixels. The size of the fifth convolutional layer 15 is 64×16×16 pixels. The size of the sixth convolutional layer 16 is 32×32×32 pixels. The size of the seventh convolutional layer 17 is 16×64×64 pixels. The size of the output layer 9 is 4×64×64 pixels.

[0055] In order to detect an object with the aid of a neural network 7 into which training data has previously been input, a radar measurement is performed. Here, a radar signal is emitted by the radar sensor and a radar signal reflected by an object is received. The emitted radar signal and the received radar signal are mixed to form a mixed signal. A mixed spectrum of the mixed signal is calculated.

[0056] The calculated mixed spectrum includes the range measured by the radar and the azimuth measured by the radar. The calculated mixed spectrum also includes the range measured by the radar and the radial velocity measured by the radar.

[0057] Input data 1, 2, 3, 4 containing mixed frequency spectra are input to the input layer 6 of the neural network 7. Figure 3 A schematic diagram of input data 1 , 2 , 3 , 4 of a neural network 7 is shown.

[0058] First input data 1 containing the distance measured by the radar is input to the input layer 6 of the neural network 7. The first input data 1 forms a two-dimensional matrix composed of individual pixels. Here, the size of the first input data 1 is 128×128 pixels.

[0059] Second input data 2 containing the azimuth angle measured by the radar are input to the input layer 6 of the neural network 7. The second input data 2 form a two-dimensional matrix composed of individual pixels. Here, the size of the second input data 2 is 128×128 pixels.

[0060] The third input data 3 containing the distance measured by the radar is input to the input layer 6 of the neural network 7. The third input data 3 constitutes a two-dimensional matrix composed of individual pixels. Here, the size of the third input data 3 is 128×128 pixels.

[0061] The fourth input data 4 containing the radial velocity measured by the radar is input to the input layer 6 of the neural network 7. The fourth input data 4 constitutes a two-dimensional matrix composed of individual pixels. Here, the size of the fourth input data 4 is 128×128 pixels.

[0062] The input data 1, 2, 3, 4 are processed in the neural network 7. In this process, convolution operations are performed from one layer to the next. The neural network 7 detects the object and its position through the convolution operations performed successively. The neural network 7 can also detect the object class of the object through the convolution operations performed successively.

[0063] Output data 51 , 52 , 53 , and 54 are output from the output layer 9 of the neural network 7 . Figure 4 A schematic diagram of output data 51 , 52 , 53 , 54 of the neural network 7 is shown.

[0064] Here, the first output data 51 is assigned to an object from a first object class (eg a person). The first output data 51 contains the detected object position. The first output data 51 forms a two-dimensional matrix of individual pixels. Here, the size of the first output data 51 is 64×64 pixels.

[0065] Here, the second output data 52 are assigned to an object from a second object class (e.g., a forklift). The second output data 52 contain the detected object position. The second output data 52 form a two-dimensional matrix of individual pixels. Here, the size of the second output data 52 is 64×64 pixels.

[0066] Here, the third output data 53 are assigned to objects from a third object class (e.g., autonomous transport vehicles). The third output data 53 contain the detected object positions. The third output data 53 form a two-dimensional matrix of individual pixels. Here, the size of the third output data 53 is 64×64 pixels.

[0067] Here, the fourth output data 54 is assigned to an object from a fourth object class (eg, free space). The fourth output data 54 contains the detected object position. The fourth output data 54 forms a two-dimensional matrix of individual pixels. Here, the size of the fourth output data 54 is 64×64 pixels.

[0068] Reference numerals list

[0069] 1First input data

[0070] 2 Second input data

[0071] 3 Third input data

[0072] 4 Fourth input data

[0073] 6 Input Layer

[0074] 7. Neural Networks

[0075] 8Intermediate connection

[0076] 9Output Layer

[0077] 11 The first convolutional layer

[0078] 12 The second convolutional layer

[0079] 13 The third convolutional layer

[0080] 14 The fourth convolutional layer

[0081] 15 The fifth convolutional layer

[0082] 16 Sixth convolutional layer

[0083] 17 Seventh convolutional layer

[0084] 21 cameras

[0085] 25 Radar equipment

[0086] 32-digit calculator

[0087] 34 processing units

[0088] 40 measurement areas

[0089] 42 radar area

[0090] 51 First output data

[0091] 52 Second output data

[0092] 53 Third output data

[0093] 54 Fourth output data

Claims

1. A method for training a neural network (7) for detecting an object, wherein: Get the geometric dimensions of the test object from the object category; as well as In a period of time: generating images of the test object by a plurality of cameras (21); generating an occupancy map based on the acquired geometry and the generated image; The radar device (25) transmits a radar signal and receives the radar signal reflected by the test object; mixing the transmitted radar signal and the received radar signal into a mixed signal in complex baseband; Calculating a complex four-dimensional mixed spectrum of the mixed signal; The complex four-dimensional mixed spectrum is calculated The first complex two-dimensional sub-spectrum and a second complex two-dimensional sub-spectrum; fusing the occupancy map and sub-spectrum into training data; and The training data is fed into the neural network (7).

2. The method according to any one of the preceding claims, characterized in that The mixed spectrum contains information about the distance, azimuth, elevation and radial velocity of the test object.

3. The method according to any one of the preceding claims, characterized in that The first sub-spectrum contains information about the distance and azimuth of the test object, and the second sub-spectrum contains information about the distance and radial velocity of the test object.

4. The method according to claim 3, characterized in that The first sub-spectrum contains a first radar image with information about the magnitude of the distance and azimuth of the test object; The first sub-spectrum contains a second radar image with information about the phase of the distance and azimuth of the test object; The second sub-spectrum contains a third radar image with information about the magnitude of the distance and radial velocity of the test object; The second sub-spectrum contains a fourth radar image with information about the phase of the distance and radial velocity of the test object.

5. The method according to any one of the preceding claims, characterized in that The test subject moves during the time period.

6. The method according to any one of the preceding claims, characterized in that Generate occupancy maps in Cartesian coordinates and transform Cartesian coordinates to polar coordinates.

7. The method according to any one of the preceding claims, characterized in that Prior to generating the image, a marker is placed on the test object such that the marker is visible in the generated image.

8. The method according to claim 7, characterized in that The camera (21) is designed as an infrared camera and / or the marker is designed as an infrared marker.

9. The method according to any one of the preceding claims, characterized in that Calculating the poses of the test object from the images, respectively, the poses respectively including the position and posture of the test object; Integrate the computed pose into the occupancy map.

10. The method according to any one of the preceding claims, characterized in that repeating the method steps for at least one additional test object from another object class; Assign occupancy maps and / or training data to corresponding object classes.

11. The method according to any one of the preceding claims, characterized in that The neural network (7) is designed as a convolutional network having an input layer (6), an output layer (9), and a plurality of convolutional layers (11, 12, 13, 14, 15, 16, 17).

12. A method for detecting an object by means of a neural network (7), the neural network having previously been fed with training data using a method according to any of the preceding claims, wherein: A radar sensor transmits a radar signal and receives the radar signal reflected by an object; mixing the transmitted radar signal and the received radar signal into a mixed signal; Calculating a mixed spectrum of the mixed signal; Inputting the input data (1, 2, 3, 4) including the mixed spectrum into the neural network (7); Processing input data (1, 2, 3, 4) in a neural network (7); The object and its location are detected by a neural network (7); The object class of the detected object and the detected object position are output by the neural network (7) as output data (51, 52, 53, 54).

13. The method according to claim 12, characterized in that The neural network (7) is designed as a convolutional network. The convolutional network has an input layer (6), an output layer (9), and multiple convolutional layers (11, 12, 13, 14, 15, 16, 17). A convolution operation is performed from one layer to the next.

14. The method according to any one of claims 12 to 13, characterized in that The calculated mixed spectrum includes at least the range and azimuth measured by the radar, and the first input data (1) including the range measured by the radar is input to the neural network (7), and the second input data (2) including the azimuth measured by the radar is input to the neural network (7).

15. The method according to any one of claims 12 to 14, characterized in that The calculated mixed spectrum includes at least the range and radial velocity measured by the radar, and the third input data (3) including the range measured by the radar is input to the neural network (7), and the fourth input data (4) including the radial velocity measured by the radar is input to the neural network (7).

Images