ASSISTANCE SYSTEM FOR DISCOURAGING OBJECTS IN A VEHICLE'S ENVIRONMENT USING A FUSION OF RADAR DATA WITH CAMERA DATA

Abstract

An assistance system (100) for distinguishing objects (61, 62) in the environment of a vehicle (40) is disclosed, wherein the assistance system (100) comprises an evaluation unit (4) with a trained neural network (1) and the evaluation unit (4) is configured, - to generate power spectra (10) as a function of received signals (110) generated by means of receiving antennas (11) of a radar system (3), - to generate an angular spectrum (2001) depending on the power spectra (10), - to determine pixel values ​​of pixels of an image (1001) as a function of intensity values ​​and / or the phase information of angle pairs of the angular spectrum (2001) and as a function of pixel values ​​of pixels of a camera image (3001) of the objects (61, 62) generated using a camera (30), - using the image (1001) as input for the neural network (1), compute an output (800) of the neural network (1) using the neural network (1), the output comprising information for distinguishing the objects (61, 62).

Description

AREA OF TECHNOLOGY

[0001] The invention relates to a radar system for detecting objects. Furthermore, the invention relates to a method for detecting objects using a radar system. STATE OF THE ART

[0002] DE 10 2020 201 025 A1 describes a radar sensor with multiple receiving antennas and multiple transmitting antennas arranged on a printed circuit board. A large proportion of the transmitting antennas are arranged at different positions with respect to a first axis and at the same position with respect to a second axis that is orthogonal to the first axis. A portion of the receiving antennas are arranged at different positions with respect to the first axis and at the same position with respect to the second axis. SUMMARY

[0003] The object is to provide an improved radar system for detecting objects and an improved method for detecting objects using a radar system. The problems underlying the invention are solved by the features of the independent claims.

[0004] An assistance system for distinguishing objects in the vicinity of a vehicle is proposed. The assistance system comprises an evaluation unit with a trained neural network (NN). The evaluation unit is configured to generate power spectra based on received signals generated by the receiving antennas of a radar system. The first frequencies of each power spectrum represent the distances of the objects relative to the receiving antennas. The second frequencies of each power spectrum represent the relative velocities of the objects relative to the receiving antennas. Furthermore, the evaluation unit is configured to generate an angular spectrum based on the power spectra. Values ​​of the first dimension of the angular spectrum represent azimuth angles relative to the receiving antennas. Values ​​of the second dimension of the angular spectrum represent elevation angles relative to the receiving antennas.The angular spectrum assigns intensity information and / or phase information to each pair of angles, which includes a value of the first dimension and a value of the second dimension.

[0005] Furthermore, the evaluation unit is set up to determine pixel values ​​of pixels of an image depending on the intensity values ​​and / or the phase information of the angle pairs and depending on pixel values ​​of pixels of a camera image of the objects generated using a camera.

[0006] Furthermore, the evaluation unit is configured to use the image as input for the neural network and, with the help of the neural network, to calculate an output. The output includes information for distinguishing the objects.

[0007] In another aspect, a method for distinguishing objects in a vehicle's environment using an assistance system with an evaluation unit using a trained neural network is disclosed.

[0008] The method involves generating power spectra depending on received signals generated by receiving antennas of a radar system, wherein first frequencies of the respective power spectrum represent distances of the objects relative to the receiving antennas and second frequencies of the respective power spectrum represent relative velocities of the objects relative to the receiving antennas.

[0009] Furthermore, the method includes generating an angular spectrum depending on the power spectra, wherein values ​​of a first dimension of the angular spectrum represent values ​​of an azimuth angle relative to the receiving antennas and values ​​of a second dimension of the angular spectrum represent values ​​of an elevation angle relative to the receiving antennas, and the angular spectrum assigns intensity information and / or phase information to each pair of angles, which includes a value of the first dimension and a value of the second dimension.

[0010] Furthermore, the method includes determining pixel values ​​of pixels of an image depending on the intensity values ​​and / or phase information of the angle pairs and depending on pixel values ​​of pixels of a camera image of the objects generated using a camera.

[0011] Furthermore, the procedure includes calculating an output of the neural network using the neural network with the image as input for the neural network, where the output includes information for distinguishing the objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The following examples are explained in more detail using the drawings. They show: Fig. 1. An assistance system for a vehicle with an evaluation unit with a neural network, Fig. 2. A flowchart to illustrate the generation of an output of the in Fig. 1 shown neural network depending on an image with phase information of received signals, Fig. 3. A flowchart illustrating the generation of power spectra as a function of received signals generated using receiving antennas. Fig. 4 an exemplary range of services offered by the Fig. 3 performance spectra shown, Fig. 5 a vehicle with the in Fig. 1. Assistance system shown, a radar system, a camera, and two objects in the vehicle's environment. Fig. 6. A flowchart illustrating the calculation of an angular spectrum as a function of phase information from the Fig. 3 performance spectra shown, Fig. 7 one depending on the phase information of the in Fig. The angular spectrum generated by the power spectra shown in the 3 is a Fig. 8 using the in Fig. 5 camera image shown, Fig. 9 one depending on the in Fig. 7 shown angular spectrum and the one in Fig. The variant of the camera image shown in 8 was generated in Fig. 2 of the shown image, Fig. 10 a variant of the in Fig. 2 shown edition of the in Fig. 2 shown neural network, Fig. 11. A flowchart to illustrate a training of the Fig. 2 shown neural network. DETAILED DESCRIPTION

[0013] In the following, similar elements are marked with the same reference symbols.

[0014] Because the evaluation unit is configured to determine the pixel values ​​of the image based on the intensity values ​​and / or phase information of the angle pairs and the pixel values ​​of the objects in the camera image, initial information from the received signals can be combined with secondary information from the camera image. Since this increases the amount of information about the objects in the vehicle's vicinity that is contained in the image, the resolution of the assistance system could be improved. This means that the minimum distance between objects at which they can still be distinguished using the assistance system could be reduced.This would allow the proposed assistance system to more accurately detect the traffic situation in which the vehicle is located, thereby increasing vehicle safety. It goes without saying that, depending on the potential application, the assistance system could be integrated into the vehicle.

[0015] Furthermore, because the image contains the first information, objects can be distinguished from each other better in bad weather conditions, where the second information from the camera image is usually less reliable, than if only the second information were used to distinguish the objects.

[0016] The first piece of information contained in the received signals can be specified in the form of the angular spectrum. The second piece of information contained in the camera image can be specified using the camera image itself. Furthermore, the proposed assistance system allows the first and second pieces of information to be combined and fused in the form of the pixel values ​​of the image pixels. This allows the first and second pieces of information, in the form of the image, to be processed as a single input to the neural network (NN). This offers the advantage of using a neural network structure known from the field of computer vision, particularly pattern recognition using images with neural networks. For example, the structure of the neural network, i.e.,A predefined number of layers of the neural network (NN), a predefined number of neurons in each layer, and a predefined type of neuron in each layer, exhibiting the structure of ResNet18 or ResNet50. For example, the ResNet18 structure has a 7x7 convolutional layer with 64 filters and a Max Pooling layer as its input layer. Furthermore, ResNet18 comprises a first residual block with two 3x3 convolutional layers, each with 64 filters; a second residual block with two 3x3 convolutional layers, each with 128 filters; a third residual block with two 3x3 convolutional layers, each with 256 filters; a fourth residual block with two 3x3 convolutional layers, each with 512 filters; a subsequent layer configured to perform global pooling; and a final layer configured to perform classification based on the pooling results.

[0017] The ResNet18 or ResNet50 is more compact and faster than larger models like the ResNet101, yet still offers comparable performance. The ResNet18 or ResNet50 can be used as a base architecture in image classification tasks because it offers a good balance between complexity and accuracy. In the proposed assistance system, for example, the neural network (NN) can be implemented as a ResNet18, ResNet50, or ResNet101, with the ResNet18, ResNet50, or ResNet101 adapted to a specific image format. The number of input channels of the ResNet18, ResNet50, or ResNet101 for image processing can be matched to the number of pixel values ​​in the image. Conveniently, the number of input channels that the ResNet18, ResNet50, or ResNet101 provides for each pixel in the image is equal to the number of pixel values ​​in the image.

[0018] In the proposed radar system, the evaluation unit contains the neural network (NN) in a trained state. During training, the connection weights of the NN are modified based on training data to adapt them to the data. To generate the NN, one variant allows the aforementioned ResNet50 structure to be used as the NN structure, with the connection weights adjusted according to the training data.

[0019] The trained neural network (NN) can function internally by weighting different areas of the camera image differently depending on the intensity and / or phase values ​​of the angle pairs, thus processing them differently within the NN. The intensity and / or phase values ​​of the angle pairs can be used as triggers for filters within the NN. These filters can be created during the NN's training. Similarly, functions that activate or deactivate a specific filter based on the intensity and / or phase values ​​of the angle pairs can be adapted to the training data. Furthermore, the NN can be configured to internally determine the relative velocities and / or distances of objects based on the intensity and / or phase values ​​of the angle pairs.This can be explained by the fact that the intensity values ​​and / or the phase values ​​of the angle pairs are calculated as a function of the power spectra, and therefore information from the power spectra is incorporated into the image.

[0020] By calculating the pixel values ​​of the image based on the phase information of the angle pairs, as suggested by one variant of the proposed assistance system, additional information in the form of the phase information of the angle pairs could be used as input for the neural network (NN) when creating the image. This is in contrast to a variant that uses only the intensity values ​​of the angle pairs to calculate the pixel values. Using this additional information could increase the NN's resolution in distinguishing objects. In particular, relative changes in the phase information of the angle pairs can often be significantly larger than relative changes in the intensity values ​​of the angle pairs, especially when dealing with small changes in distances or velocities.In other words, the phase information of the angular pairs can be more sensitive to small changes in distance or velocity than the intensity values ​​of the angular pairs in many applications. Using the phase information of the angular pairs to distinguish objects could be particularly advantageous when the objects are positioned very close to each other and have similar velocities and distances from the assistance system. For example, it has been observed that a slight movement, such as one centimeter, of one of the objects can cause a large change, such as 180 degrees, in the phase information of one of the angular pairs.

[0021] The information used to differentiate the objects can, for example, provide information about the relative position of each object in relation to the assistance system, in particular to the receiving antennas or the camera. According to one possible embodiment, the receiving antennas and the camera are part of the assistance system.

[0022] For example, the output of the neural network can include probability values ​​for sub-areas of a region surrounding the vehicle. The probability values ​​of the sub-areas that exceed a predefined threshold can each indicate the probability that a given presumed object is located in that sub-area. In this case, the objects can be part of a set of presumed objects. For instance, the objects could be those presumed objects located in the sub-areas with the highest probability values.

[0023] In one example, the output can be considered as an output from the evaluation unit. The evaluation unit can be coupled to a vehicle control unit and configured to send the output to the vehicle control unit. The vehicle control unit can be configured to control a vehicle component, such as a steering system or a drive system, depending on the output from the evaluation unit. In another example, the evaluation unit itself can be configured to control the vehicle component depending on the output. In this case, the evaluation unit can be implemented as the vehicle control unit.

[0024] According to one variant, the assistance system can include an output unit that makes information about the relative positions of objects in relation to the assistance system, particularly the vehicle, perceptible to the driver. For example, the output unit could be a display showing the relative positions of the objects, perhaps as rectangles or circles. With this variant, the driver can assess the traffic situation based on the relative positions of the objects. In most applications, the objects are located in front of the vehicle. Information about the location of these objects could enhance driver safety.

[0025] In most applications, the evaluation unit is configured to generate the respective power spectrum in the form of a two-dimensional power spectrum. The first dimension of this power spectrum, hereinafter referred to simply as the spectrum, represents the first frequencies, which correspond to the distances of the objects mentioned above. The second dimension represents the second frequencies, which correspond to the relative velocities of the objects mentioned above. These second frequencies can be considered Doppler frequencies. The spectrum can therefore be viewed as a distance Doppler spectrum. Each frequency pair within the spectrum, containing one of the first frequencies and one of the second frequencies, can be considered a distance Doppler pair.The respective spectrum assigns a power and / or intensity value and phase information to each frequency pair of the respective spectrum.

[0026] Although the receiving antennas are spatially separated, for the purpose of calculating the spectra, the distance of each object to its respective receiving antenna is assumed to be approximately the same. This is because the distances of the objects to the receiving antennas are much greater than the distance between the receiving antennas. The same applies to the relative velocities of the objects. The relative velocities of the objects with respect to their respective receiving antennas can be considered approximately the same. Therefore, the power spectra can be similar or identical, except for the phase information of the respective frequency pair within each spectrum.

[0027] In principle, the evaluation unit can be configured to process, store, and / or provide the image in the form of an image data set containing the pixel values ​​of the image's pixels. The mapping of pixel values ​​to the image pixels can be achieved by ordering the pixel values ​​within the image data set. Similarly, the evaluation unit can be configured to process, store, and / or provide the camera image in the form of a camera image data set containing the pixel values ​​of the image's pixels. The mapping of pixel values ​​to the image pixels can again be achieved by ordering the pixel values ​​within the camera image data set.

[0028] According to one possible configuration, the evaluation unit is set up to select a first frequency pair for the respective power spectrum. The first selected frequency pairs of the power spectra each comprise one frequency of the first frequencies lying in a first frequency range and one frequency of the second frequencies lying in a second frequency range. The first frequency range represents a range of distances, and the second frequency range a range of relative velocities. Thus, the first frequency pairs can be considered the first distance Doppler pairs.

[0029] Furthermore, this configuration can include a system in which the evaluation unit is set up to generate the angular spectrum based on the phase information of the first selected frequency pairs, and in particular, based solely on the phase information of the first selected frequency pairs. This allows the information base used to generate the angular spectrum to be limited to the phase information of the first selected frequency pairs. As a result, the radar system only incorporates information about objects whose distances and relative velocities are equal or similar within a tolerance into the image generation. This allows the neural network to be used to distinguish between objects that are indistinguishable or only barely distinguishable based on their distances and velocities.The proposed assistance system could be particularly helpful in this use case, allowing the objects to be distinguished from one another.

[0030] The selection of the first frequency pair for the respective spectrum mentioned above can initially involve selecting a first frequency pair from a selected spectrum of spectra, which will subsequently be referred to as the first selected frequency pair. Alternatively, the selected spectrum can be chosen randomly.

[0031] According to one possible configuration, the evaluation unit can be set up to determine the first selected frequency pair as the frequency pair of the selected spectrum that exhibits the highest power or intensity value. According to another possible variant, the first selected frequency pair of the respective spectrum can be the same as the first selected frequency pair of the selected spectrum.

[0032] It is possible that the first selected frequency pair of the respective spectrum is the one whose power or intensity value is highest among all frequency pairs in that spectrum, particularly when considering a given tolerance. This can be due to the fact that the first selected frequency pair is chosen based on the power or intensity values ​​of the frequency pairs in the selected spectrum, and that the spectra may be similar or identical to each other except for the phase information of the frequency pairs, as described above. If the first selected frequency pair of the respective spectrum is the same as the first selected frequency pair of the selected spectrum, then an upper limit of the first frequency range is the same as a lower limit of the first frequency range, and an upper limit of the second frequency range is the same as a lower limit of the second frequency range.

[0033] According to an alternative variant, the evaluation unit can be configured to determine a two-dimensional tolerance range, starting from the first selected frequency pair of the selected spectrum, within which the first selected frequency pairs of the remaining spectra should lie. The remaining spectra comprise the aforementioned spectra with the exception of the selected spectrum. The two-dimensional tolerance range, hereinafter referred to simply as the tolerance range, can be specified by a lower and an upper limit for the first frequencies and a lower and an upper limit for the second frequencies.The evaluation unit can be configured to determine the lower and upper limits of the first frequencies and the lower and upper limits of the second frequencies using a predefined tolerance, such as 1 percent, depending on the first and second frequencies of the first selected frequency pair of the selected spectrum. In this case, the lower and upper limits of the first frequencies can be the limits of the aforementioned first frequency range. Similarly, the lower and upper limits of the second frequencies can be the limits of the aforementioned second frequency range.

[0034] According to one possible configuration, the evaluation unit can be set up to determine the first selected frequency pair of the respective spectrum of the remaining spectra as the frequency pair of the respective spectrum which lies within the tolerance range and has the highest power or intensity value.

[0035] Because the first frequencies represent the distances between the objects and the second frequencies represent the relative velocities of the objects, the first frequency range represents a range of distances in which the distances between the objects lie, and the second frequency range represents a range of relative velocities in which the relative velocities of the objects lie.

[0036] The tolerance range is often referred to as the "range-Doppler bin", since the first frequency range specifies a range of possible distances between the objects and the second frequency range specifies a range of possible relative velocities of the objects, which can be calculated from determined Doppler velocities.

[0037] The following describes one possible way in which the evaluation unit can generate the spectra depending on the received signals.

[0038] The radar system can include a control unit for managing the system's transmitting antennas to send radar signals in response to transmitted signals. Each transmitting antenna can then emit its respective radar signal based on the transmitted signal. For example, the transmitted signals can differ from one another, such as in their phase.

[0039] The radar system can include the aforementioned receiving antennas. Each receiving antenna can be configured to generate a corresponding received signal in response to received reflected radar signals. The reflected radar signals can result from reflections of the transmitted radar signals off objects. For example, the reflected radar signal can result from reflections of the radar signals off the respective object.

[0040] The control unit can be configured to operate the transmitting and receiving antennas in MIMO (Multiple Input / Multiple Output) mode. Operating the transmitting and receiving antennas in MIMO mode can involve each receiving antenna receiving the reflected radar signals and the evaluation unit being able to distinguish between these reflected radar signals. To achieve this, the control unit can be configured to generate the transmitted signals as coded signals. For example, the control unit can be configured to perform time-division multiplexing and / or binary phase modulation to encode the transmitted signals and thus the radar signals. This encoding enables the evaluation unit to differentiate between the various received signals and, consequently, the reflected radar signals when processing them.

[0041] Using the MIMO method, a virtual antenna array with virtual receiving antennas, hereinafter also referred to as virtual channels, can be generated. The distances between the virtual receiving antennas within the virtual antenna array can be determined by convolving the relative distances between the transmitting antennas and the receiving antennas.

[0042] As an example, the radar system can include mixing modules. These modules can be configured to mix each transmitted signal with each received signal to generate mixed signals within an operating time interval. Each mixed signal can be assigned to one of the virtual channels. For example, if the radar system has four transmitting antennas and four receiving antennas, the mixing modules can generate 16 mixed signals, depending on the received signals from the four receiving antennas and the transmitted signals from the four transmitting antennas. The mixing modules can be generally referred to as IQ demodulators, which can be implemented in analog or digital form.

[0043] The control unit can be configured to generate the transmission signals in such a way that the transmission signals contain several chirps within the operating time interval. During each chirp, the frequency of the respective transmission signal can increase from a starting frequency to operate the radar system as an FMCW radar.

[0044] Furthermore, the evaluation unit can be configured to generate the respective spectrum depending on the specific mixed signal. The evaluation unit can be set up to generate the respective spectrum by performing a Fourier transform, in particular a fast Fourier transform (FFT), on the respective mixed signal. As described above, the respective spectrum assigns the power or intensity value and the phase information to each frequency pair within the spectrum.

[0045] In a further embodiment, the evaluation unit is configured to assign at least one of the angle pairs and at least one of the pixels of the camera image to each pixel of the image. Furthermore, in this embodiment, the evaluation unit is configured to determine a first set of pixel values ​​for each pixel of the image. The pixel values ​​of this first set depend on the pixel values ​​of the corresponding pixel in the camera image. The evaluation unit can also be configured to determine a second set of pixel values. The pixel values ​​of this second set depend on the phase information and / or the intensity information of the angle pair in the angular spectrum that corresponds to the respective pixel in the image.Furthermore, in this configuration, the evaluation unit is set up to carry out the assignment of the respective angle pair to the respective pixel of the image depending on the value of the first dimension and the value of the second dimension of the respective angle pair and, in particular, depending on a relative position of the camera to the receiving antennas.

[0046] By mapping the pixels of the image to the angular pairs and pixels of the camera image, and by determining the first and second sets of pixel values ​​for the pixels of the image, the phase information and / or the intensity information can be transferred into the combined image along with the pixel values ​​of the pixels of the camera image. The first set of pixel values ​​for each pixel of the image can, for example, comprise three color values, such as a red value, a green value, and a blue value, of the pixel in the camera image that corresponds to that pixel in the image. The second set of pixel values ​​for each pixel in the image can, for example, comprise the phase information and / or the intensity information of that angular pair of the angular spectrum that corresponds to that pixel in the image.

[0047] By assigning the respective pair of angles to the respective pixel of the image depending on the value of the first dimension and the value of the second dimension of the respective pair of angles, information about the relative positions of the objects to the receiving antennas, which is contained in the received signals, can be used when generating the image.

[0048] According to one variant, the evaluation unit can be configured to assign the respective angle pair to the respective pixel of the image, depending on the relative position of the camera to the receiving antennas, using a displacement vector whose entries can specify a distance of the camera to the receiving antennas in three dimensions, or a rotation matrix whose entries can specify a position of the camera rotated relative to the receiving antennas. This could increase the accuracy of the assistance system.

[0049] In a further development process, the evaluation unit can be configured to assign each angular pair to the respective pixel of the image based on the value of the first frequency of at least one of the first selected frequency pairs. The value of the first frequency of at least one of the first selected frequency pairs represents the distance of one of the objects relative to the receiving antennas. In the variant described above, where the evaluation unit generates the angular spectrum based on the phase information of the first selected frequency pairs, the first frequencies of the selected frequency pairs lie within the first frequency range. Therefore, the distances of the objects lie within the aforementioned range of distances represented by the first frequency range.Therefore, the distance between one of the objects, taking into account the aforementioned tolerance range, can be considered a common distance between the objects in relation to the receiving antennas.

[0050] According to one variant, the evaluation unit is configured to calculate the common distance as a function of the value of the first frequency of the first selected frequency pair of the selected spectrum. For this purpose, the evaluation unit can determine a difference between a frequency of the received signals and a frequency of the transmitted signals at selected sampling times within the operating time interval. According to another variant, the evaluation unit can then calculate the common distance from these differences and the slope of the respective chirp in which the sampling times lie.

[0051] In advanced training, the evaluation unit can be configured to generate a point cloud containing points in a spherical coordinate system, based on the values ​​of the first and second dimensions of the angle pairs and their common distance. Each point in the spherical coordinate system is specified by a value in the first dimension of the respective angle pair, which indicates the azimuth angle of the point, and a value in the second dimension of the respective angle pair, which indicates the elevation angle of the point, as well as by their common distance. The evaluation unit can be configured to assign the angle pairs to the pixels of the image based on the points in the point cloud.

[0052] In one possible configuration, the evaluation unit can be set up to project the points of the point cloud onto a projection plane. In many applications, the projection plane can run parallel to the camera's image plane, in which the camera image is generated. In another variant, the projection plane can coincide with the camera's image plane. Furthermore, the pixels of the image can lie within the projection plane.

[0053] According to one possible configuration, the evaluation unit can assign the pixels of the image to the pixels of the camera image in such a way that a portion of the image pixels are identical to the pixels of the camera image. In one variant, all pixels of the image can be identical to the pixels of the camera image. In this case, the camera can have a fisheye lens, and its field of view can be equal to or larger than that of the radar system. The term "identical" here refers to the position of the respective pixel in the camera image or in the image. The pixel values ​​of the image pixels differ from those of the camera image in cases of identical pixels in that the image pixels have a higher number of pixel values ​​compared to the pixels of the camera image that are identical to the image pixels.

[0054] According to a second variant, the image pixels can include additional pixels beyond those of the camera image. In this case, the radar system's field of view can be larger than the camera's field of view. The camera can be a pinhole camera. The additional pixels of the image cannot be assigned to pairs of angles within the angular spectrum. The pixel values ​​of the first set of these additional pixels can be zero.

[0055] If the radar system's field of view is smaller than the camera's field of view, the evaluation unit can crop the camera image at its edges to create the image. In this case, the pixels of the image can be those pixels of the camera image that lie within the cropped area. Instead of cropping the camera image at the edges, the evaluation unit can set the pixel values ​​of a second set of outer pixels to zero if these pixels are not associated with an angle pair. In this variant, the pixels of the outer set are only associated with an outer set of pixels in the image.

[0056] In one possible configuration, the evaluation unit can be configured to map the image pixels onto a global two-dimensional coordinate system using camera specifications, such as the lens focal length and the camera's aperture angle. Each image pixel can be assigned a point in the global coordinate system, specified by a value on one of the x-axis and a value on one of the y-axis. Conveniently, the origin of the global coordinate system can coincide with the origin of the spherical coordinate system. For simplified image generation, the origin of the global coordinate system, its x-axis and y-axis, and the origin of the spherical coordinate system can all lie in the projection plane.

[0057] Furthermore, the evaluation unit can be configured to assign each point of the point cloud projected onto the projection plane to the pixel of the image whose corresponding point in the global coordinate system within the projection plane has the shortest distance to the respective projected point. Since each point of the point cloud is specified by the value of the first and second dimensions of the respective pair of angles, such an assignment between the projected points and the pixels of the image provides a mapping between the respective pair of angles that specifies each point of the point cloud and the respective pixel of the image to which that point of the point cloud is assigned.

[0058] In most applications, the number of projected points is less than the number of pixels in the image. Furthermore, in most cases, the distances between the projected points of the point cloud are greater than the distances between the points assigned to the pixels of the image in the global coordinate system. Therefore, according to one approach, the evaluation unit can assign several pixels of the image to each point of the point cloud projected onto the projection plane, for example, those pixels of the image whose assigned points in the projection plane lie within a predefined radius around the respective projected point.

[0059] To map the points of the point cloud to the pixels of the image, one approach involves the evaluation unit multiplying the coordinate values ​​of the point cloud points by a reduction factor to convert them into the global coordinate system. The coordinate values ​​of the point cloud points are specified by their azimuth angles, elevation angles, and common distance. This multiplication by the reduction factor can be performed, for example, before projecting the point cloud points onto the projection plane.

[0060] Alternatively, the aforementioned mapping of points in the global coordinate system to the pixels of the image can be performed in such a way that the coordinate values ​​of the points in the point cloud do not need to be changed. Starting with a local coordinate system, which includes the coordinates of image sensors, such as photodiodes, of the camera that are assigned to the pixels of the camera image, the coordinates of the image sensors can be multiplied by a magnification factor to calculate the coordinates of those points in the global coordinate system that are assigned to the pixels of the image.

[0061] In the event that the radar system's field of view is larger than the camera's field of view, the evaluation unit can only consider those projected points of the point cloud for creating the image that lie within a predetermined maximum distance of the pixels of the camera image that are assigned to the points.

[0062] In a further embodiment, the evaluation unit can be configured to select at least the first and a second frequency pair for the respective power spectrum. The first and second selected frequency pairs of the respective power spectrum comprise two different frequencies of the first frequencies and / or two different frequencies of the second frequencies. The second frequency pairs can be considered as second distance Doppler pairs.

[0063] Furthermore, in this configuration, the evaluation unit can be set up to generate the angular spectrum based on the phase information of the first and second selected frequency pairs. In this case, the angular spectrum, using the intensity and phase information of the angular pairs, could also provide information about the objects if they have different speeds and / or distances relative to the assistance system. Additionally, the information about the objects, which can be obtained from the received signals and the camera image, could be combined in the image and processed as a single input for the neural network. This could reduce the speed at which the objects can be distinguished from one another using the assistance system.In some applications, this could ensure real-time capability of the assistance system in distinguishing between objects.

[0064] Analogous to the previously described configuration, in a further embodiment the evaluation unit can be set up to select at least the first and second frequency pairs for the respective power spectrum, wherein the respective two selected frequency pairs of the power spectra each comprise two different frequencies of the first frequencies and / or two different frequencies of the second frequencies.

[0065] As an alternative to the previously described configuration, in this further configuration the evaluation unit can be set up to generate the angular spectrum based on the phase information of the first selected frequency pairs of the power spectra and to generate a further angular spectrum based on the phase information of the second selected frequency pairs of the power spectra. Analogous to the angular spectrum, values ​​of a first dimension of the further angular spectrum represent values ​​of an azimuth angle relative to the receiving antennas, and values ​​of a second dimension of the further angular spectrum represent values ​​of an elevation angle relative to the receiving antennas. Furthermore, the further angular spectrum assigns intensity information and / or phase information to each angular pair of the further angular spectrum, which comprises a value of the first dimension and a value of the second dimension of the further angular spectrum.

[0066] Furthermore, in a further configuration, the evaluation unit can be set up to determine pixel values ​​of pixels in another image based on the intensity values ​​and / or phase information of the angular pairs of the additional angular spectrum and based on the pixel values ​​of the pixels in the camera image. The evaluation unit can generate the additional image analogously to the first image, whereby the evaluation unit processes the intensity values ​​and / or phase information of the angular pairs of the additional angular spectrum instead of the intensity values ​​and / or phase information of the angular pairs of the angular spectrum.

[0067] Furthermore, in a further configuration, the evaluation unit can be set up to use the additional image as input for the neural network and, with the help of the neural network, to calculate a further output. This further output includes additional information for distinguishing the objects. The further output can be in the same format as the output. In a further configuration, the evaluation unit can also be set up to distinguish the objects based on the output and the further output. According to one variant, the output and the further output can contain information about the speeds and / or distances of the objects.

[0068] The further configuration described last describes a variant of the assistance system in which the evaluation unit can be used to process several images sequentially, including the initial image and subsequent images, using the neural network. Information about objects that can be assigned to the same distance Doppler pair can be processed using each image. This allows information about objects in the vehicle's vicinity to be processed sorted according to the values ​​of the distance Doppler pairs. This could enable the prioritization of information about the objects depending on their distances and / or speeds.

[0069] According to one possible approach, the evaluation unit can prioritize the processing of distance Doppler pairs differently depending on how low or high the values ​​of the distance Doppler pairs are. For example, distance Doppler pairs with low distance values ​​can be prioritized over distance Doppler pairs with high distance values. Another approach allows the evaluation unit to ignore distance Doppler pairs with distance values ​​exceeding a predefined distance threshold for calculating the output and / or subsequent output. This could ensure real-time object differentiation capabilities of the assistance system in certain applications.

[0070] In a training program, the evaluation unit can be configured to combine the output and subsequent output into a common feature space. For example, a first set of vectors in the feature space could contain values ​​from the output, and a second set of vectors in the feature space could contain values ​​from the subsequent output. In this training program, the evaluation unit can be configured to differentiate the objects using this common feature space. For example, the evaluation unit could represent the first set of vectors and the second set of vectors as points in the feature space. On the display, those points representing objects that are farther away from the assistance system could be shown smaller than the other points.

[0071] According to one variant, the spectrum can provide the power or intensity value and phase information of the first or second selected frequency pair in the form of a complex number. An FFT result can include the complex number associated with the first or second selected frequency pair. The complex number of the first or second selected frequency pair can be specified by a real part and an imaginary part.In one example, the evaluation unit can be configured to calculate the power or intensity value and the phase information of the first or second selected frequency pair as a function of the real and imaginary parts of the complex number of the respective first or second selected frequency pair, for example, by calculating the arctangent of the quotient of the imaginary and real parts of the complex number of the respective first or second selected frequency pair. In this case, the phase information corresponds to a phase value. Alternatively, the power or intensity value and the phase information of the respective first or second selected frequency pair can be provided indirectly using the complex number associated with the respective first or second selected frequency pair.

[0072] In another example, the phase information of the respective first or second selected frequency pair can be equal to the imaginary part of the complex number of the respective first or second selected frequency pair. In this case, it is assumed that the imaginary part indirectly provides the phase information along with the power or intensity value of the respective first or second selected frequency pair. In one example, the power or intensity value and the phase information of the respective first or second selected frequency pair can be indirectly provided via the real and imaginary parts of the complex number of the respective first or second selected frequency pair.

[0073] In another example, the evaluation unit can be configured to calculate the power or intensity value of the respective first or second selected frequency pair as a function of the real part and the imaginary part of the complex number of the respective first or second selected frequency pair.

[0074] According to one variant, the phase information of the respective first or second selected frequency pair of the respective spectrum can specify a respective phase shift between a frequency of the respective transmitted signal, from which the respective mixed signal is generated, and a phase of the frequencies of the respective first or second selected frequency pair, or be related to the respective phase shift.

[0075] According to one possible configuration, the evaluation unit is configured to generate the angular spectrum based on the phase information of the first selected frequency pairs. The evaluation unit can be configured to generate the angular spectrum by performing a Fourier transform on the power or intensity values ​​and / or the phase information of the first selected frequency pairs, hereinafter also referred to as a spatial Fourier transform. Thus, the phase information of the angular pairs in the angular spectrum is derived, among other things, from the phase information of the first selected frequency pairs. The values ​​of the first dimension of the angular spectrum can represent values ​​of first spatial frequencies, each representing a value of the azimuth angle. The values ​​of the second dimension of the angular spectrum can represent values ​​of second spatial frequencies, each representing a value of the elevation angle.

[0076] Similarly, the evaluation unit can be configured to generate the extended angular spectrum based on the phase information of the second selected frequency pairs. The evaluation unit can be configured to generate the extended angular spectrum by performing a Fourier transform on the power or intensity values ​​and / or the phase information of the second selected frequency pairs. In this way, the phase information of the angular pairs of the extended angular spectrum is derived, among other things, from the phase information of the second selected frequency pairs. Analogous to the angular spectrum, values ​​of the first and second dimensions of the extended angular spectrum can specify values ​​of the first and second spatial frequencies, respectively, representing values ​​of the azimuth angle and elevation angle.

[0077] The following describes how the intensity and phase information of the angular pairs of the angular spectrum or the wider angular spectrum can be formed. The angular pairs of the angular spectrum or the wider angular spectrum are generally referred to as angular pairs.

[0078] The intensity information for each angle pair can be expressed as a power or intensity value. The phase information for each angle pair can be expressed as a phase value, according to one variant. Alternatively, the phase information can be provided as a complex number within the angular spectrum associated with that angle pair. In this case, the phase value can be calculated by relating the real and imaginary parts of the complex number, specifically by calculating the arctangent of the quotient of the imaginary and real parts of that complex number.Since the phase value for each pair of angles can be calculated from the imaginary part using the real part, according to another variant, the phase information of each pair of angles can be formed in the form of the imaginary part of that complex number of the angle spectrum which is assigned to the respective pair of angles.

[0079] The angular spectrum and the extended angular spectrum can each be viewed as a 2D function, where different frequencies of the first frequencies of the angular spectrum or the extended angular spectrum can represent different values ​​of a first argument of the 2D function, and different frequencies of the second frequencies of the angular spectrum or the extended angular spectrum can represent different values ​​of a second argument of the 2D function. Thus, each pair of values ​​of the first and second arguments can represent the respective pair of angles. The 2D function assigns a corresponding set of function values ​​to each pair of values ​​of the first and second arguments. This set of function values ​​can include the intensity information and the phase information of the respective pair of angles represented by that pair of values ​​of the first and second arguments.

[0080] In a further development, the evaluation unit can be configured to provide the intensity and phase information of each angular pair of the angular spectrum using the aforementioned complex number, where the complex number comprises the real and imaginary parts. In this further development, the second set of pixel values ​​for each pixel of the image comprises a first pixel value that depends on the real part of the complex number of the respective angular pair, and a second pixel value that depends on the imaginary part of the complex number of the respective angular pair of the angular spectrum.

[0081] By calculating the pixel values ​​of the image pixels in this way, the respective information provided by the real and imaginary parts of the complex number of the angular spectrum can be represented by each pixel. This makes it possible to represent the associated information, defined by the relationship between the real and imaginary parts in the form of the complex number of the angular spectrum, using the image. Because this associated information can be represented by the image, the information density of the image regarding distances between objects could be increased. Increasing the information density of the image could improve the accuracy of object differentiation using the neural network.

[0082] The first frequencies of the angular spectrum can indicate an azimuth angle of presumed sub-areas of the vehicle's surroundings where the objects might be located. The second frequencies of the angular spectrum can indicate an elevation angle of the presumed sub-areas where the objects might be located.

[0083] Each sub-area can be assigned at least one of the angle pairs. The intensity information, such as the power or intensity value, assigned to each angle pair can indicate the probability that one of the objects is located in the sub-area assigned to that angle pair. The higher the power or intensity value of the respective angle pair, the higher the probability that one of the objects is located in that sub-area assigned to that angle pair.

[0084] If, as described above, the intensity and / or phase information of the angular spectrum is integrated into the image in the form of pixel values, this could simplify object discrimination using the neural network (NN), especially if the NN's structure is based on, or exactly resembles, the structure of a neural network designed for image recognition tasks. Because the pixel values ​​can contain phase information in addition to intensity information, the increased information density described above could improve object discrimination accuracy.

[0085] According to a further embodiment, the second set of pixel values ​​for each pixel can include a third pixel value, which is equal to a phase value of the angular pair of the angular spectrum assigned to that pixel. In this embodiment, the phase information provided by the respective complex number of the angular pair can be weighted more heavily, as it is integrated into the image in two ways: indirectly through the imaginary part and directly through the phase value. This could further improve the accuracy of object discrimination in some applications.

[0086] According to a further embodiment, the second set of pixel values ​​for each pixel includes a fourth pixel value equal to the magnitude of the complex number corresponding to the pair of angles in the angular spectrum associated with that pixel. Such additional weighting of the power or intensity value used in calculating the magnitude could, in some applications, further improve the accuracy of object discrimination.

[0087] In general, using the phase information of the first or second selected frequency pairs could have a further advantage, which is described below.

[0088] In general, as described above, the respective reflected radar signal can result from the reflections of the transmitted radar signals off the respective object. Due to different surface properties of the objects and / or different environmental conditions near the objects, there can be a different phase shift in the reflection of the radar signals off the respective object. For example, the reflection of an electromagnetic wave off a metal plate can cause the electromagnetic field to be phase-shifted relative to the magnetic field compared to the reflection of an electromagnetic wave off a non-metallic part.

[0089] Such differing phase shifts due to reflections from the objects can influence the processing of the received signals and manifest themselves as differing phase information for the selected frequency pairs. These differing phase shifts due to reflections can affect the phase information of the spectra and thus also the angular spectrum. This can lead to more precise object differentiation based on the image. In many cases, it has been shown that the neural network is capable of performing object differentiation internally.

[0090] In general, one or more mathematical formulas relating the physical laws of radar signal phase shift to surface properties and / or environmental conditions can be very complex and sensitive to variations in those conditions. Furthermore, there is a risk of overlooking a relevant physical effect when expressing these laws through formulas. This difficulty can be overcome by equipping the evaluation unit with a neural network.

[0091] The neural network (NN) can be trained to distinguish objects based on the phase information of the angular pairs, particularly in the form of the image pixel values. For this purpose, the training data can include observations of different phase shifts of reflected training radar signals at at least two different training objects. By training the NN with such data, it can learn the physical effects that cause phase shifts in reflected radar signals at the various training objects. The trained NN, with its parameter values ​​adapted to the training data (such as the connection weights mentioned above), can represent at least some of the physical laws governing the phase shifts of reflected radar signals at the different training objects.The neural network (NN) can be treated as a black box, and its training can be performed instead of experiments on the phase shifts of radar signals reflected by different vehicle types or people under varying environmental conditions. Thus, the NN can reduce the effort required to develop the proposed assistance system, which can take phase shifts of reflected radar signals into account to distinguish between objects.

[0092] In a further embodiment, the neural network can have connection weights that indicate the strength of connections between neurons within the network, with at least some of these connection weights being complex values. The complex values ​​representing the connection weights each have a real and an imaginary part. Because the connection weights are complex values, coupled information—represented by the intensity and phase information of the respective angle pairs—can be propagated through computations within the neural network. This coupled information can be processed, for example, by multiplying it by one or more of the complex connection weights. This could further improve the accuracy in determining the number of objects.

[0093] Training a neural network (NN) can involve generating training data. This data can include training images and target datasets. The training images can be generated based on training reception signals and training camera images. Furthermore, training the NN can involve inputting the training images into the neural network and receiving training output datasets generated by the network. Additionally, training the NN can involve calculating the value of a loss function based on the target and training output datasets. Finally, training the NN can involve adapting the values ​​of the neural network's parameters based on the value of the loss function. After training, the NN is in a trained state.

[0094] Fig. Figure 1 shows an assistance system 100 for distinguishing objects in an environment 140 of a vehicle 40, which is in Fig. Figure 5 shows that the objects can, for example, comprise a first object 61 and a second object 62. The assistance system can include an evaluation unit 4. The evaluation unit 4 has a trained neural network 1.

[0095] The evaluation unit 4 is configured to generate power spectra 10 as a function of received signals 110 generated by receiving antennas 11. First frequencies of the respective power spectrum represent distances of the objects relative to the receiving antennas 11. Second frequencies of the respective power spectrum represent relative velocities of the objects relative to the receiving antennas 11. The power spectra 10 are hereinafter referred to simply as spectra 10.

[0096] Furthermore, the evaluation unit 4 can be configured to select a first frequency pair for the respective spectrum. The first frequency pairs of the spectra 10 each comprise one frequency of the first frequencies, which lies in a first frequency range, and one frequency of the second frequencies, which lies in a second frequency range. The first frequency range represents a range of distances. The second frequency range represents a range of relative velocities. The respective spectrum includes at least intensity information and / or phase information of the selected first frequency pair of the respective spectrum.

[0097] Furthermore, the evaluation unit 4 is configured to generate an angular spectrum based on the spectra 10. Values ​​of a first dimension of the angular spectrum represent values ​​of an azimuth angle relative to the receiving antennas 11. Values ​​of a second dimension of the angular spectrum represent values ​​of an elevation angle relative to the receiving antennas 11. As described above, the angular spectrum assigns intensity information and / or phase information to each pair of angles, which comprises a value of the first dimension and a value of the second dimension.

[0098] Furthermore, evaluation unit 4 is set up to process pixel values ​​of pixels of a [unclear] in [unclear] Fig. to determine the intensity values ​​and / or phase information of the angle pairs shown in image 1001 as a function of the pixel values ​​of pixels of a camera image 3001 of objects 61, 62 generated using a camera 30.

[0099] Furthermore, the evaluation unit 4 is set up to calculate an output 800 of the neural network 1 using the image 1001 as input for the neural network 1, the output of which includes information for distinguishing the objects.

[0100] Evaluation unit 4 can be configured to generate the spectra 10 using a spectrum module 210 of evaluation unit 4, as described in Fig. 3 is shown. Fig. Figure 3 shows, as examples, a first spectrum 101, a second spectrum 102, and an nth spectrum 10. n of the spectra 10. The spectra 10 are generated depending on the received signals 110. The received signals 110 are generated using a set of receiving antennas 11 of a radar system 3. The radar system 3 can be part of the assistance system 100.

[0101] The first Spectrum 101 is an example. Fig. Figure 4 illustrates this. Along a first dimension 311 of the first spectrum 101, short lines are drawn, symbolically representing values ​​of the first frequencies of the first spectrum 101. A second dimension 312 of the first spectrum 101 represents values ​​of the second frequencies of the first spectrum 101. A third dimension 313 of the first spectrum 101 can represent an intensity or a power value.

[0102] Fig. Figure 5 shows an application example of the radar system 3. The assistance system 100 can be part of the vehicle 40 and, for example, be located at the front of the vehicle 40. The environment 140 of the vehicle 40 can be a three-dimensional space between the dashed lines 150.

[0103] The radar system 3 can comprise the transmitting antennas mentioned above and the control unit (not shown in the figures) for transmitting radar signals 401 depending on the aforementioned transmitting signals. The radar signals 401 can be considered an example of the radar signals mentioned above. In particular, each transmitting antenna can transmit the respective radar signal of the radar signals 401 depending on the respective transmitting signal.

[0104] The receiving antennas 11 can generate the received signals 110 in response to the reception of reflected radar signals 501. The reflected radar signals 501 can result from reflections of the radar signals 401 off the objects, for example, from reflections off the first object 61 and the second object 62. For example, a first radar signal of the reflected radar signals 501 can result from reflections of the radar signals 401 off the first object 61, and a second radar signal of the reflected radar signals 501 can result from reflections of the radar signals 401 off the second object 62. The evaluation unit 4 indirectly generates the spectra 10 as a function of the reflected radar signals 501, since the received signals 110 are generated as a function of the reflected radar signals 501.

[0105] The control unit can be configured to perform time-division multiplexing and / or binary phase modulation to encode the transmitted signals. This allows the evaluation unit 4 to distinguish the reflected radar signals 501 from one another based on the encoding during the processing of the received signals 110.

[0106] Furthermore, the evaluation unit 4 can be configured to generate the spectra 10 as a function of the transmitted signals and the received signals 110, as described above. For example, the evaluation unit 4 can generate the mixed signals by mixing the transmitted signals with the received signals 110 and perform a Fourier transform on the respective mixed signal to obtain the respective spectrum of the spectra 10.

[0107] For example, evaluation unit 4 can select a frequency pair 300 from the first spectrum 101. The selected frequency pair 300 can comprise a frequency 301 from the first frequencies of the first spectrum 101, hereinafter referred to as the first selected frequency 301, and a frequency 302 from the second frequencies of the first spectrum 101, hereinafter referred to as the second selected frequency 302. Evaluation unit 4 can select the frequency pair 300 as the frequency pair from the first spectrum 101 that has the highest power or intensity value.

[0108] Regarding the in Fig. In the application example 5 of the assistance system 100, the first selected frequency 301 can represent the distance of objects 61, 62 relative to radar system 3, and the second selected frequency 302 can represent the relative velocity of objects 61, 62 with respect to radar system 3. It can be assumed that the distance of the first object 61 to radar system 3 is similar to the distance of the second object 62 to radar system 3. The same can be assumed for the relative velocity of objects 61, 62.

[0109] According to one variant, the evaluation unit 4 can process the spectra 10 and search for the selected frequency pair 300 in each spectrum of the spectra 10. In this case, the selected frequency pair of the respective spectrum of the spectra 10 has the first selected frequency 301 and the second selected frequency 302. The respective spectrum of the spectra 10 can include phase information of the respective selected frequency pair of the respective spectrum.

[0110] According to an alternative variant, the evaluation unit 4 can determine the selected frequency pairs of the remaining spectra, as described above, depending on the aforementioned two-dimensional tolerance range and the selected frequency pair 300 of the first spectrum 101. In this case as well, the respective spectrum of the spectra 10 includes the phase information of the respective selected frequency pair of the respective spectrum.

[0111] According to an example, the evaluation unit 4 can write the phase information of the selected frequency pair of each spectrum of the spectra 10 into a phase information file 500.

[0112] The evaluation unit 4 can be configured to perform a spatial Fourier transform based on the phase information of the selected frequency pairs of the spectra 10, for example, depending on the phase information file 500, in order to generate a result of the spatial Fourier transform 600. The evaluation unit 4 can be configured to perform the spatial Fourier transform depending on the phase information file 500 and depending on the relative positions of the transmitting antennas to each other and to the receiving antennas 11.

[0113] The result of the Fourier transform 600 can include first spatial frequencies 601 and second spatial frequencies 602. The evaluation unit 4 can include a Fourier module 60 for calculating the first spatial frequencies 601 and the second spatial frequencies 602 as a function of the phase information file 500, as shown in Fig. Figure 6 illustrates this. The first spatial frequencies 601 can represent possible azimuth angles of the objects with respect to a longitudinal axis 400 of the vehicle 40. The second spatial frequencies 602 can represent possible elevation angles of the objects with respect to a horizontal plane containing the longitudinal axis 400.

[0114] The evaluation unit 4 can be configured to generate the angular spectrum as a function of the result of the spatial Fourier transform 600. In this process, the evaluation unit 4 can associate one of the first spatial frequencies 601 and one of the second spatial frequencies 602, in the form of a further frequency pair, with each of the angular pairs of the angular spectrum.

[0115] The intensity and phase information of each additional frequency pair can result from the Fourier transform of the phase information file 500. For example, the result of the spatial Fourier transform 600 can provide the intensity and phase information of each additional frequency pair in the form of a complex number.

[0116] Each complex number can comprise a real part and an imaginary part. For example, evaluation unit 4 can be configured to convert the result of the spatial Fourier transform 600 into the angular spectrum. In this case, the intensity and phase information, for example in the form of the complex number, of each additional frequency pair can be assigned to the respective angular pair associated with that additional frequency pair.

[0117] Fig. Figure 7 shows an example of the angular spectrum 2001, which can be considered an example of the angular spectrum mentioned above. The angular spectrum can be represented as a 2D image with an x-axis 2011 and a y-axis 2012. The pairs of angles in the angular spectrum 2001 can each be assigned to a corresponding pair of coordinate values ​​for rasterizing the angular spectrum 2001. Each pair of coordinate values ​​comprises one value on the x-axis 2011 and one value on the y-axis 2012 of the angular spectrum 2001. The values ​​on the x-axis 2011 represent the azimuth angle, and the values ​​on the y-axis 2012 represent the elevation angle of the angular pairs. Thus, each pair of coordinate values ​​can represent the respective pair of angles that has an azimuth value equal to the x-axis value 2011 of the respective pair of coordinate values ​​and an elevation angle equal to the y-axis value 2012 of the respective pair of coordinate values.

[0118] According to another example, the coordinate values ​​can represent the further frequency pairs. Here, the x-axis values ​​2011 represent the first spatial frequencies 601, and the y-axis values ​​2012 represent the second spatial frequencies 602 of the further frequency pairs. Thus, each pair of coordinates consisting of an x-axis value 2011 and a y-axis value 2012 can represent the respective further frequency pair, which includes the first spatial frequency, equal to the x-axis value 2011, and the second spatial frequency, equal to the y-axis value 2012. Since the azimuth angles and elevation angles can be calculated from the first and second spatial frequencies, these values ​​indirectly represent the azimuth angles and elevation angles of the angle pairs.

[0119] Fig. Figure 5 shows an exemplary application of the assistance system 100, in which the first object 61 is located at a first azimuth angle of zero degrees with respect to the longitudinal axis 400, and the second object 62 is located at a second azimuth angle 32 with respect to the longitudinal axis 400. The second azimuth angle 32 can be negative. With respect to this application, the angle spectrum 2001 can include a first angle pair 2101, which specifies a location of the first object 61, and a second angle pair 2102, which specifies a location of the second object 62. The angle pairs 2101 and 2102 represent values ​​of the azimuth angle, namely zero and the second azimuth angle 32, as indicated by dashed lines in Figure 5. Fig. Figure 7 is shown, and an elevation angle of 33° is assigned. The elevation angle of 33° of objects 21 and 22 is shown in Figure 7. Fig. 5 is not shown for the sake of simplicity.

[0120] According to one possible variant, the angular spectrum assigns to the first pair of angles 2101 a first complex number with a first real part 21011 and a first imaginary part 21012. Similarly, the angular spectrum assigns to the second pair of angles 2102 a second complex number with a second real part 21021 and a second imaginary part 21022.

[0121] Evaluation unit 4 can be configured to determine the aforementioned points of the point cloud by filtering those angle pairs of the angular spectrum 2001 whose intensity values ​​exceed a predefined intensity threshold. (Referring to the in) Fig. In the application shown in 5, the filtered points can include a first point whose coordinates in the above-mentioned spherical coordinate system are specified by the first pair of angles 2101 and the common distance of objects 61, 62 to the radar system, and a second point whose coordinates in the spherical coordinate system are specified by the first pair of angles 2101 and the common distance.

[0122] Furthermore, the evaluation unit 4 can project the points of the point cloud onto the aforementioned projection plane using the common distance between objects 61 and 62. Prior to this, according to one possible embodiment, the evaluation unit 4 can multiply the spherical coordinates of the first and second points, specified by the first angle pair 2101, the second angle pair 2102, and the common distance between objects 61 and 62, by the reduction factor.

[0123] Fig. Figure 8 shows camera image 3001, which depicts the first object 61 and the second object 62. The pixel values ​​of each pixel in camera image 3001 can represent different channels of that pixel. For example, if the pixels of the camera image each comprise three channels, the first pixel value of each pixel in camera image 3001 can represent a red value, the second pixel value of each pixel in camera image 3001 can represent a green value, and the third pixel value of each pixel in camera image 3001 can represent a blue value.

[0124] The pixels of the camera image 3001 can each be assigned to a corresponding pair of coordinate values ​​for rasterizing the camera image 3001, where each pair of coordinate values ​​comprises a value of an x-axis 3011 and a value of a y-axis 3012 of the camera image 3001.

[0125] Evaluation unit 4 can be configured to process, store, and / or provide the camera image 3001 in the form of a camera image data set. For this purpose, camera 30 is coupled to evaluation unit 4 for the transmission of the camera image data set and / or sensor signals from an image sensor of camera 30 to evaluation unit 4. The camera image data set comprises the pixel values ​​assigned to the pixels of camera image 3001. The assignment of each pixel of camera image 3001 to the respective coordinate value pair can be provided by a sequence of the pixel values ​​of the pixels of camera image 3001 in the camera image data set.

[0126] With regard to the in Fig. In the use case shown in Figure 5, and for the sake of simplicity, a first pixel 3101 of the camera image 3001 can represent the first object 61, and a second pixel 3102 of the camera image 3001 can represent the second object 62. The first pixel 3101 has a first pixel value 31011, a second pixel value 31012, and a third pixel value 31013. The second pixel 3102 has a first pixel value 31021, a second pixel value 31022, and a third pixel value 31023. It is understood that in practical applications, the objects can usually be represented by multiple pixels.

[0127] Fig. Figure 9 shows image 1001, which combines information from the angular spectrum 2001 with the pixel values ​​of camera image 3001. According to one possible variant, the evaluation unit 4 can be configured to generate image 1001 by adding additional pixel values ​​to the pixel values ​​of camera image 3001. In this variant, the pixels of image 1001 can be shaped like the pixels of camera image 3001. The pixels of image 1001 can each be assigned to a corresponding pair of coordinate values ​​for rasterizing image 1001, with each pair of coordinate values ​​comprising a value on an x-axis 1011 and a value on a y-axis 1012 of image 1001.

[0128] According to one variant, evaluation unit 4 can be configured to assign each point of the point cloud to at least one pixel of the camera image 3001. This can include mapping the pixels of camera image 3001 onto the global coordinate system described above and projecting the points of the point cloud onto the projection plane as described above.

[0129] The evaluation unit 4 can be configured to generate image 1001 such that the first set of pixel values ​​of each pixel in image 1001 depends on the pixel values ​​of the corresponding pixel in camera image 3001. With regard to the use case, this shows Fig. 9. A first pixel 1101 of image 1001, which is identical to the first pixel 3101 of camera image 3001. A first set of pixel values ​​of the first pixel 1101 of image 1001 comprises a first pixel value 11011, a second pixel value 11012, and a third pixel value 11013. Similarly, image 1001 has a second pixel 1102, which is identical to the second pixel 3102 of camera image 3001. A first set of pixel values ​​of the second pixel 1102 comprises a first pixel value 11021, a second pixel value 11022, and a third pixel value 11023.

[0130] According to one variant, the first pixel value 11011 of the first pixel 1101 of image 1001 can be equal to the first pixel value 31011 of the first pixel 3101 of the camera image 3001; the second pixel value 11012 of the first pixel 1101 of image 1001 can be equal to the second pixel value 31012 of the first pixel 3101 of the camera image 3001; and the third pixel value 11013 of the first pixel 1101 of image 1001 can be equal to the third pixel value 31013 of the first pixel 3101 of the camera image 3001.

[0131] Similarly, the first pixel value 11021 of the second pixel 1102 of image 1001 can be equal to the first pixel value 31021 of the second pixel 3102 of camera image 3001; the second pixel value 11022 of the second pixel 1102 of image 1001 can be equal to the second pixel value 31022 of the second pixel 3102 of camera image 3001; and the third pixel value 11023 of the second pixel 1102 of image 1001 can be equal to the third pixel value 31023 of the second pixel 3102 of camera image 3001.

[0132] Furthermore, evaluation unit 4 can be configured to generate image 1001 such that a second set of pixel values ​​for each pixel in image 1001 depends on the intensity and / or phase information of the angle pair assigned to that pixel. As described above, the assignment of points in the point cloud to pixels in image 1001 provides an assignment of angle pairs to pixels in image 1001, since each point in the point cloud is assigned to one of the angle pairs.

[0133] According to one variant, the second set of pixel values ​​for the first pixel 1101 of image 1001 can include a fourth pixel value 11014 and a fifth pixel value 11015 for the first pixel 1101. Similarly, the second set of pixel values ​​for the second pixel 1102 of image 1001 can include a fourth pixel value 11024 and a fifth pixel value 11025 for the second pixel 1102.

[0134] At the in Fig. In the use case shown in Figure 5, the first pair of angles 2101 is assigned to the first pixel 1101 of the image 1001 and the second pair of angles 2102 is assigned to the second pixel 1102 of the image 1001.

[0135] According to one variant, the evaluation unit 4 can be configured to generate image 1001 such that the fourth pixel value 11014 of the first pixel 1101 of image 1001 is equal to the first real part 21011 of the first angle pair 2101, and the fifth pixel value 11015 of the first pixel 1101 is equal to the first imaginary part 21012 of the first angle pair 2101. Similarly, the fourth pixel value 11024 of the second pixel 1102 of image 1001 can be equal to the second real part 21021 of the second angle pair 2102, and the fifth pixel value 11025 of the second pixel 1102 can be equal to the second imaginary part 21022 of the second angle pair 2102.

[0136] Evaluation unit 4 can be configured to process, store, and / or provide image 1001 in the form of an initial image data set containing the pixel values ​​assigned to the pixels of image 1001. The assignment of each pixel of image 1001 to its respective coordinate value pair can be provided by a sequence of the pixel values ​​of image 1001 in the initial image data set.

[0137] According to a second variant, the second set of the first pixel 1101 of image 1001 can include a sixth pixel value of the first pixel 1101, not shown in the figures, which is equal to a phase value or a magnitude of the first complex number of the first angle pair 2101. Similarly, the second set of the second pixel 1102 of image 1001 can include a sixth pixel value of the second pixel 1102, not shown in the figures, which is equal to a phase value or a magnitude of the second complex number of the second angle pair 2102.

[0138] According to a third variant, it may be provided that the second set of the first pixel 1101 of the image 1001 includes a sixth pixel value of the first pixel 1101, not shown in the figures, which is equal to the phase value of the first complex number of the first pair of angles 2101, and includes a seventh pixel value of the first pixel 1101, not shown in the figures, which is equal to the magnitude of the first complex number of the first pair of angles 2101.

[0139] Similarly, in this third variant, the second set of the second pixel 1102 of the image 1001 can include a sixth pixel value of the second pixel 1102, not shown in the figures, which is equal to the phase value of the second complex number of the second pair of angles 2102, and a seventh pixel value of the second pixel 1102, not shown in the figures, which is equal to the magnitude of the second complex number of the second pair of angles 2102.

[0140] Fig. Figure 10 shows a possible variant of output 800, in which output 800 is represented as a matrix with matrix elements 810. Each matrix element 810 can be specified by a row number (801) and a column number (802). Each element of the matrix can be assigned a sub-area from a set of sub-areas of the environment 140 of the vehicle 40. The sub-areas of the set of sub-areas can each have a distance from the vehicle 40 specified by the first frequency of the selected area Doppler pair 300, i.e., the first selected frequency 301. Furthermore, the sub-areas can lie within a corresponding azimuth angle range and a corresponding elevation angle range. The azimuth angle can be measured with respect to the longitudinal axis 400 of the vehicle 40.The elevation angle can be measured with respect to a horizontal plane encompassing the longitudinal axis 400. For example, the column number of each element of the matrix elements 810 can indicate the respective azimuth angle range in which the sub-area associated with that element of the matrix elements 810 lies. Additionally, the row number of each element of the matrix elements 810 can indicate the respective elevation angle range in which the sub-area associated with that element of the matrix elements 810 lies.

[0141] In one example, each element of the matrix elements 810 can include a respective probability or intensity value that indicates a probability that one of the objects 61, 62 is located in the sub-area that is assigned to the respective element of the matrix elements 810.

[0142] Taking into account the in Fig. In the application example shown, a first element 811 and a third element 813 of the matrix elements 810 can have the highest probability values ​​among the matrix elements 810. This can indicate that the first object 61 is located in a first sub-area that can be assigned to the first element 811, and that the second object 62 is located in a third sub-area that can be assigned to the third element 813.

[0143] According to another variant, the output can indicate the sub-areas in which objects 61 and 62 are located in the form of rectangles on an output image.

[0144] The following describes an exemplary procedure for training NN1. The training can involve generating in Fig.The training data 4001 shown in section 11 comprises the following: The training data 4001 can include input datasets and target datasets. Generating the training data 4001 can involve generating the input datasets of the training data 4001 based on training reception signals and training camera images.

[0145] The training receive signals can be generated using training radar signals via radar system 3. The training radar signals can comprise sets of training radar signals. Each set of training radar signals can be transmitted within a specific time interval using the transmitting antennas of radar system 3. Thus, the sets of training radar signals can be transmitted sequentially. During the transmission period of each set of training radar signals, each transmitting antenna can transmit one training radar signal from that set.

[0146] The training receive signals can comprise sets of training receive signals, with the antennas 11 being able to generate the respective set of training receive signals within the respective time period. The antennas 11 can generate the respective set of training receive signals in response to receiving a respective reflected set of training radar signals. The respective reflected set of training radar signals can result from reflections of the respective transmitted set of training radar signals from training objects in the vicinity 140 of the vehicle 40 within the respective time period. In an example, the training objects can include the first object 61 and the second object 62.

[0147] Similarly, the training camera images can be generated within the respective time period using camera 30. The training camera images depict the training objects for the respective time period.

[0148] As an example, each input data set of training data 4001 can be generated depending on the respective set of training received signals and the respective training camera image in the same way as image 1001 is generated depending on the received signals 110 and the camera image 3001. Each input data set comprises a respective training image, which is generated analogously to image 1001. However, instead of using the received signals 110 and the camera image 3001 to generate image 1001, the respective set of training received signals and the respective training camera image can be used to generate the respective training image.More precisely, the evaluation unit 4 can generate a respective set of training performance spectra 10, analogous to the performance spectra mentioned above, depending on the respective set of training reception signals, and depending on the respective set of training performance spectra, generate a respective training angle spectrum in order to generate the respective training image depending on the respective training angle spectrum and the respective training camera image.

[0149] Furthermore, the training can include inputting the training data records 4001, in particular the training images, into NN 1 and receiving training output records 4011 from NN 1 in response. In this case, NN 1 can generate a corresponding output record 4011 for each input training data record 4001, in particular for each training image.

[0150] Furthermore, the training can include calculating the value of a loss function that depends on the training output datasets 4011 and the target training datasets 4001. For example, the value of the loss function can be calculated as a function of a difference between the training output datasets 4011 and the target training datasets 4001. For instance, the value of the loss function can be calculated as a function of the sum of squares of differences, where each difference can be the difference between the respective output dataset of the training output datasets 4011 and the respective target training dataset 4001.

[0151] The training output datasets 4011 and the target training datasets 4001 can each have the same format as the output 800. For example, the relative positions of the training objects to the assistance system 100 can be manually determined to generate the respective target training dataset 4001.

[0152] Training NN1 can involve adjusting the values ​​of its parameters based on the value of the loss function. These parameters can include connection weights between neurons, activation function parameters, or pooling function parameters if NN1 is a CNN. Adjusting the parameter values ​​to the training data 4001 can be performed using machine learning techniques such as backpropagation or other learning methods. Once the parameter values ​​have been adjusted to the training data 4001, NN1 can be considered trained. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2020 201 025 A1

[0002]

Claims

[1] Assistance system (100) for distinguishing objects (61, 62) in the environment of a vehicle (40), wherein the assistance system (100) comprises an evaluation unit (4) with a trained neural network (1) and the evaluation unit (4) is configured, - to generate power spectra (10) as a function of received signals (110) generated by means of receiving antennas (11) of a radar system (3), wherein first frequencies of the respective power spectrum represent distances of the objects (61, 62) in relation to the receiving antennas (11) and second frequencies of the respective power spectrum represent relative velocities of the objects (61, 62) in relation to the receiving antennas (11), - to generate an angular spectrum (2001) as a function of the power spectra (10), wherein values ​​of a first dimension of the angular spectrum (2001) represent values ​​of an azimuth angle in relation to the receiving antennas (11) and values ​​of a second dimension of the angular spectrum (2001) represent values ​​of an elevation angle in relation to the receiving antennas (11) and the angular spectrum (2001) assigns intensity information and / or phase information to each pair of angles comprising a value of the first dimension and a value of the second dimension, - To determine pixel values ​​of pixels of an image (1001) depending on the intensity values ​​and / or phase information of the angle pairs and depending on pixel values ​​of pixels of a camera image (3001) of the objects (61, 62) generated using a camera (30), - using the image (1001) as input for the neural network (1), compute an output (800) of the neural network (1) using the neural network (1), the output comprising information for distinguishing the objects (61, 62). [2] Assistance system (100) according to claim 1, wherein the evaluation unit (4) is configured, - to select a first frequency pair for the respective power spectrum, wherein the selected first frequency pairs of the power spectra (10) each comprise a frequency of the first frequencies that lies in a first frequency range and a frequency of the second frequencies that lies in a second frequency range, and the first frequency range represents a range of distances and the second frequency range represents a range of relative velocities, and - to generate the angular spectrum (2001) depending on the phase information of the first selected frequency pairs. [3] Assistance system (100) according to claim 1 or 2, wherein the evaluation unit (4) is configured to assign at least one of the angle pairs and at least one of the pixels of the camera image (3001) to the respective pixel of the image (1001) and to determine a respective first set of pixel values ​​for the respective pixel of the image (1001), wherein the pixel values ​​of the respective first set depend on the pixel values ​​of that pixel of the camera image (3001) which is assigned to the respective pixel of the image (1001), and to determine a respective second set of pixel values, wherein the pixel values ​​of the respective second set depend on the phase information and / or the intensity information of that angle pair of the angular spectrum (2001) which is assigned to the respective pixel of the image (1001), wherein the evaluation unit (4) is configuredto perform the assignment of the respective angle pair to the respective pixel of the image (1001) depending on the value of the first dimension and the value of the second dimension of the respective angle pair and in particular depending on a relative position of the camera (30) to the receiving antennas, wherein in particular the evaluation unit (4) is configured to perform the assignment of the respective angle pair to the respective pixel of the image (1001) depending on a value of the first frequency of at least one of the first selected frequency pairs, wherein the value of the first frequency of one of the frequency pairs of the first selected frequency pairs represents a distance of one of the objects (61, 62) in relation to the receiving antennas. [4] Assistance system (100) according to one of the preceding claims, wherein the evaluation unit (4) is configured, - to select at least one first and one second frequency pair for the respective power spectrum, wherein the respective first and second selected frequency pair of the respective power spectrum comprise two different frequencies of the first frequencies and / or two different frequencies of the second frequencies, and - to generate the angular spectrum (2001) depending on the phase information of the first and second selected frequency pairs. [5] Assistance system (100) according to one of the preceding claims, wherein the evaluation unit (4) is configured, - to select at least one first and one second frequency pair for the respective power spectrum, wherein the respective two selected frequency pairs of the power spectra (10) each comprise two different frequencies of the first frequencies and / or two different frequencies of the second frequencies, - to generate the angular spectrum (2001) depending on the phase information of the first selected frequency pairs of the power spectra, - depending on the phase information of the second selected frequency pairs of the power spectra, to generate a further angular spectrum, wherein values ​​of a first dimension of the further angular spectrum represent values ​​of an azimuth angle relative to the receiving antennas (11) and values ​​of a second dimension of the further angular spectrum represent values ​​of an elevation angle relative to the receiving antennas (11) and the further angular spectrum assigns intensity information and / or phase information to a respective angular pair of the further angular spectrum, which includes a value of the first dimension and a value of the second dimension, - To determine pixel values ​​of pixels of another image depending on the intensity values ​​and / or phase information of the angle pairs of the further angle spectrum and depending on the pixel values ​​of the pixels of the camera image (3001), - using the further image as input for the neural network (1), to compute a further output of the neural network (1) using the neural network (1), wherein the further output includes further information for distinguishing the objects (61, 62), and - to distinguish the objects (61, 62) based on the output (800) and the further output. [6] Assistance system (100) according to one of the preceding claims, wherein the evaluation unit (4) is configured to provide the intensity information and the phase information of the respective angular pair of the angular spectrum (2001) using a complex number, wherein the complex number comprises a real part and an imaginary part, wherein the second set of pixel values ​​of the respective pixel of the image (1001) comprises a first pixel value which depends on the real part of the complex number of the respective angular pair, and a second pixel value which depends on the imaginary part of the complex number of the respective angular pair of the angular spectrum (2001). [7] Assistance system (100) according to claim 6, wherein the second set of pixel values ​​of the respective pixel comprises a third pixel value which is equal to a phase value of that pair of angles of the angular spectrum (2001) which is assigned to the respective pixel. [8] Assistance system (100) according to claim 6 or 7, wherein the second set of pixel values ​​of the respective pixel comprises a fourth pixel value which is equal to the magnitude of the complex number which is assigned to the pair of angles of the angular spectrum (2001) which is assigned to the respective pixel. [9] Vehicle (40) comprising an assistance system (100) according to any of the preceding claims. [10] Method for distinguishing objects (61, 62) in the environment of a vehicle (40) using an assistance system (3) with an evaluation unit (4) with a trained neural network (1), the method comprising: - Generating power spectra (10) depending on received signals (110) generated by receiving antennas (11) of a radar system (3), wherein first frequencies of the respective power spectrum represent distances of the objects (61, 62) in relation to the receiving antennas (11) and second frequencies of the respective power spectrum represent relative velocities of the objects (61, 62) in relation to the receiving antennas (11), - Generating an angular spectrum (2001) depending on the power spectra, wherein values ​​of a first dimension of the angular spectrum (2001) represent values ​​of an azimuth angle relative to the receiving antennas (11) and values ​​of a second dimension of the angular spectrum (2001) represent values ​​of an elevation angle relative to the receiving antennas (11) and the angular spectrum (2001) assigns intensity information and / or phase information to each pair of angles comprising a value of the first dimension and a value of the second dimension, - Determining pixel values ​​of pixels of an image (1001) depending on the intensity values ​​and / or phase information of the angle pairs and depending on pixel values ​​of pixels of a camera image (3001) of the objects (61, 62) generated using a camera (30), - Computing an output (800) of the neural network (1) using the neural network (1) using the image (1001) as input for the neural network (1), wherein the output includes information for distinguishing the objects (61, 62). [11] The method of claim 10, wherein the method further comprises: - Generating training data that includes training images and target datasets, with the training images being generated depending on training reception signals and training camera images, - Inputting the training images into the neural network and receiving training output datasets generated using the neural network (1), - Calculating the value of a loss function as a function of the target datasets and the training output datasets, - Adapting values ​​of parameters of the neural network (1) depending on the value of the loss function. [12] Computer program product comprising instructions executable by a processor, wherein the execution of the instructions causes the processor to carry out the method according to claim 10 or 11.

Images