Method and system for improving the accuracy of radar-based estimates of a vehicle's self-motion

Abstract

The invention relates to a method for improving the accuracy of radar-based estimates of the self-motion of a vehicle (200), comprising: - Acquiring radar data (350) in a detection area in the vicinity of the vehicle (200) in a temporal sequence, wherein the radar data (350) comprise at least first radar raw targets (370) at a first time (t1) and second radar raw targets (380) at a second time (t2); - Acquiring image data (450) for an image sequence (390) with at least a first image (370) and a second image (380); - Fusing the acquired radar raw targets (370) with the acquired first image (470) to form a first fusion image (570); - Fusing the acquired radar raw targets (380) with the acquired second image (480) to form a second fusion image (580);- Determining anchor points (750) from the radar raw targets (370, 380) for performing radar odometry, wherein a radar raw target (370, 380) is selected as an anchor point (750) only if the radar raw target (370, 380) can be assigned to an object that has not moved in the time interval ΔT = t2- t1.;

Description

The invention relates to a method, a system and a computer program product for improving the accuracy of radar-based estimates of the own movement of a vehicle.In the automotive industry, radar sensors are frequently used in modern vehicles to support important functions such as autonomous driving, adaptive cruise control, and lane keeping assist. These driver assistance systems contribute to increasing the safety and efficiency in road traffic by monitoring and reacting with the movement of the vehicle in its environment. In addition, the measurement results of the radar sensors can be used to estimate the own movement of the vehicle.By "vehicle's own movement" is meant the movement of a vehicle which is generated by itself, i.e. without the action of an external influence or actuation force. In an automobile, this is the motion of the vehicle due to the driving force of its engine. The vehicle has control over its movement and is capable of controlling the direction and speed of its movement.Estimating the own motion of a vehicle from the raw data of radar targets relates to the use of radar technology to determine the motion of a vehicle by detecting the reflections of the radar beams emitted by the vehicle. This method is frequently used in the automotive industry and in aerospace applications to monitor and control the motion and speed of vehicles. Radar stands for Radio Detection and Ranging, and is a technology that uses electromagnetic waves to detect objects in the environment. A radar apparatus emits radar beams and detects the reflected signals from objects including vehicles. The reflected signals from vehicles received from a radar are referred to as "raw targets.". These raw targets form the basis for the motion analysis. In this case, the Doppler effect which occurs when a source of electromagnetic waves or sound waves moves relative to an observer is to be taken into account. It causes a change in the frequency or wavelength of the reflected signals. In radar, the Doppler effect is used to determine the speed of a moving vehicle. If a vehicle approaches the radar, the reflected signals are compressed, if it moves away from the radar, the signals are stretched. Based on the frequency changes of the reflected radar beams, the radar device can estimate the speed, the position and the direction of movement of a vehicle. This information is important for driver assistance systems in motor vehicles, but also for flight protection and navigation of aircraft and ships. These estimates are frequently updated in real time to allow continuous monitoring of vehicle motion.Estimating the vehicle's own movement by evaluating raw radar targets is therefore an accurate method for detecting the vehicle movement. In contrast to wheel odometry, the radar doppler effect enables a direct and drift-minimized estimation of the movement of the vehicle.Wheel odometry is a method for determining the position and movement of a vehicle from the information about the wheel revolutions. This method is based on the assumption that the distance travelled by a vehicle is directly related to the number of revolutions of its wheels. The vehicle is equipped with wheels provided with sensors to measure the number of wheel revolutions. These sensors can be Hall sensors or optical sensors, for example. The distance travelled by the vehicle can be calculated from the data of the wheel sensors and the estimated vehicle position is updated by integrating the wheel revolutions over time. In addition, the speed and the direction of movement of the vehicle can be estimated by analyzing the changes in wheel revolutions over time. However, wheel odometry is prone to drift because small errors in the wheel sensors or bumps in the roadway can result in inaccuracies in the position estimates. These inaccuracies may accumulate over time and lead to significant errors.However, when using radar, selecting and identifying raw radar targets in complex environments may be problematic. When the targets are on moving objects or when there are many reflections from different objects, the estimation of vehicle motion by raw radar targets may become inaccurate. The radar may have difficulty identifying the correct targets and tracking their movements correctly.DE 10 2022 130 559 A1 discloses a system and method for assigning three-dimensional bounding boxes to tracked objects based on information collected from objects in a two-dimensional image for applications such as autonomous navigation. An object is tracked in three-dimensional space and a two-dimensional image is received from a vehicle sensor. The system generates a three-dimensional bounding box for the object in the two-dimensional image, determines a two-dimensional image property of the object, and assigns the three-dimensional bounding box to the tracked object based on the three-dimensional bounding box of the object and the two-dimensional image property of the object.DE 10 2020 127 216 A1 discloses a method for determining a trajectory for a vehicle. A neural network is set up to determine, using a digital environment map, the start position and the target position of the vehicle, at least one anchor point for a trajectory of the vehicle, which anchor point connects a start position of the vehicle to a target position of the vehicle.DE 10 2016 223 072 A1 discloses a method for determining a self-movement of a vehicle by capturing image data by means of sensor means of a vehicle and identifying stationary objects by means of a classifier on the basis of this image data. The classifier works with a prior knowledge about an appearance of stationary objects. The determination of the own movement takes place depending on these stationary objects.US 2019 / 0162842 A1 discloses an apparatus having one or more processors configured to translate a RADAR speed map into an image plane of a camera to generate a three-dimensional RADAR speed image. The 3D RADAR speed image includes a relative speed of each pixel, wherein the relative speed of each pixel is based on a RADAR speed estimate in the three-dimensional RADAR speed map. Based on the determined relative speed of each pixel, it is determined whether the visual features correspond to a moving object.The object on which the invention is based is now to provide possibilities for improving the accuracy of radar-based estimates of the vehicle's own movement, which are simple to implement, are distinguished by high reliability and require only low computing power.With the method according to the present invention, suitable anchor points for radar odometry in the environment of a vehicle can be identified. By fusing the raw radar targets with the images of the image sequence recorded by a camera device, it can be established whether the objects to be assigned to the raw radar targets, i.e. objects or other vehicles, are moving dynamically or are static. The comparison of the images of the fusion image sequence provides information about time sequences, movement patterns, changes and other important aspects that cannot be recognized from the static individual images alone. Only those raw radar targets that can be assigned to static objects within a time interval are selected as anchor points.These anchor points serve as reference points to estimate the position and movement of the vehicle and to enable reliable navigation. The selection of suitable anchor points is particularly important for driver assistance systems (ADAS), autonomous vehicles, and other systems that require accurate estimation of self-motion to make decisions for safe navigation and control of the vehicle. Accurate estimation of the own motion of a vehicle is therefore crucial to the safety and performance of driver assistance systems and autonomous vehicles and can be significantly improved by the present invention.This object is achieved according to the invention with respect to a method by the features of claim 1, with respect to a system by the features of claim 8, and with respect to a computer program product by the features of claim 15. The further claims relate to preferred embodiments of the invention.According to a first aspect, the invention provides a method for improving the accuracy of radar-based estimates of the own movement of a vehicle, wherein at least one radar sensor module having one or more radar sensors, at least one camera device having one or more cameras and a data evaluation module having a software application are used. The method comprises the method steps:acquiring radar data in an acquisition area in the environment of the vehicle from the radar sensor module in a temporal sequence, wherein the radar data comprise at least first raw radar targets at a first point in time t 1 and second raw radar targets at a second point in time t 2 ;capturing image data for an image sequence having at least a first image and a second image;fusing the raw radar targets detected at the first time t 1 with the first image recorded at this first time t 1 to form a first fusion image;fusing the raw radar targets detected at the second time t 2 with the second image recorded at this second time t 2 to form a second fusion image;classifying and identifying objects having first raw radar targets in the first fusion image and objects having second raw radar targets in the second fusion image;comparing the first fusion image and the second fusion image with respect to the movement of the classified and identified objects;determining anchor points from the raw radar targets for performing a radar odometry, wherein a raw radar target is selected as the anchor point only if the raw radar target can be assigned to an object that has not moved in the time interval ΔT=t 2- t 1.In a development, it is provided that if the density of anchor points in the surroundings of the vehicle is too low, mobile anchor points from raw radar targets, which are assigned to objects that have moved in the time interval ΔT=t 2 t 1 are additionally used for radar odometry and these mobile anchor points are provided with a weighting.In a further specific embodiment, it is provided that a radar sensor of the radar sensor module is designed as a short-range radar having a measurement range of up to approximately 30 m or as a medium-range radar having a measurement range of up to approximately 150 m or as a long-range radar having a measurement range of up to approximately 220 m.In particular, it is provided that a radar sensor is designed as a microwave radar.In an advantageous embodiment, it is provided that the data evaluation module comprises a software application for image fusion and image analysis, which uses artificial intelligence algorithms.Advantageously, neural networks and / or classification algorithms and / or optical flow algorithms and / or matching of feature points and / or tracking with Kalman filters are used as algorithms.In particular, the camera device comprises one or more cameras, wherein a camera is configured as a digital RGB camera, stereo camera, depth camera, wide-angle camera and / or infrared camera.In a development, it is provided that additional information from a lidar system and / or an ultrasound system is integrated into the first and the second image and / or the image sequence and / or the first and the second fusion image and / or the fusion image sequence.In a second aspect, the invention provides a system for improving the accuracy of radar-based estimates of the own motion of a vehicle. The system includes at least one radar sensor module having one or more radar sensors, at least one camera device having one or more cameras, and a data evaluation module having a software application. The radar sensor module is designed to acquire radar data in an acquisition range in the environment of the vehicle from the radar sensor module in a temporal sequence, wherein the radar data comprise at least first raw radar targets at a first point in time t 1 and second raw radar targets at a second point in time t 2. The camera device is configured to record image data for an image sequence having at least a first image and a second image. The software application of the data evaluation module is configured to merge the raw radar targets captured at the first time t 1 with the first image captured at this first time t 1 to form a first fusion image, to merge the raw radar targets captured at the second time t 2 with the second image captured at this second time t 2 to form a second fusion image, to classify and identify objects with first raw radar targets in the first fusion image and objects with second raw radar targets in the second fusion image, to compare the first fusion image and the second fusion image with respect to the movement of the classified and identified objects, and for determining anchor points from the raw radar targets for performing a radar odometry, wherein a raw radar target is selected as the anchor point only if the raw radar target can be assigned to an object that has not moved in the time interval ΔT=t 2- t 1.In a development, it is provided that if the density of anchor points in the surroundings of the vehicle is too low, mobile anchor points from raw radar targets, which are assigned to objects that have moved in the time interval ΔT=t 2- t 1 are additionally used for radar odometry and these mobile anchor points are provided with a weighting.In a further specific embodiment, it is provided that a radar sensor of the radar sensor module is designed as a short-range radar having a measurement range of up to approximately 30 m or as a medium-range radar having a measurement range of up to approximately 150 m or as a long-range radar having a measurement range of up to approximately 220 m.In particular, it is provided that a radar sensor is designed as a microwave radar.In an advantageous embodiment, it is provided that the data evaluation module comprises a software application for image fusion and image analysis, which uses artificial intelligence algorithms.It is advantageously provided to use neural networks and / or classification algorithms and / or optical flow algorithms and / or matching of feature points and / or tracking using Kalman filters as algorithms.In particular, the camera device comprises one or more cameras, wherein a camera is configured as a digital RGB camera, stereo camera, depth camera, wide-angle camera and / or infrared camera.In a development, it is provided that additional information from a lidar system and / or an ultrasound system is integrated into the first and the second image and / or the image sequence and / or the first and the second fusion image and / or the fusion image sequence.According to a third aspect, the invention provides a computer program product comprising executable program code configured to, when executed, perform the method according to the first aspect.The invention is explained in more detail below with reference to exemplary embodiments shown in the drawing.The following shows: FIG. 1 is a block diagram for explaining an embodiment of a system according to the present invention; FIG. 2 shows a flow chart for explaining the individual method steps of the method according to the invention; FIG. 3 shows a schematic representation of a computer program product.Additional characteristics, aspects and advantages of the invention or its embodiments will become apparent from the detailed description in connection with the claims.FIG. 1 shows a system 100 according to the invention for improving the accuracy of radar-based estimates of the own movement of a vehicle 200. The system 100 comprises at least one radar sensor module 300, a camera device 400 and a data evaluation module 500. The vehicle 200 can be designed as a motor vehicle, as an automated vehicle, as an agricultural vehicle such as a combine harvester, as a production or service robot, as a watercraft or as a flying object such as a drone. The radar sensor module 300, the camera device 400 and the data evaluation module 500 can each be equipped with a processor and / or a memory unit. The data evaluation module 500 can be integrated in the vehicle 200 or designed as a cloud-based solution which is connected to the vehicle 200 via a mobile radio connection. For this purpose, radar sensor module 300 and camera device 400, for example, or another communication module may be equipped with radio modules for transmitting data into the cloud. In particular, 5G radio modules or 6G radio modules may be used.In the context of the invention, a "processor" may be, for example, a machine or an electronic circuit. A processor can in particular be a main processor (or processor. Central processing unit, CPU), may be a microprocessor or a microcontroller, e.g. an application specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions. A processor may also be a virtualized processor, a virtual machine, or a soft CPU. It can also be a programmable processor, for example, which is equipped with configuration steps for executing the method according to the invention mentioned above, or is configured with configuration steps in such a way that the programmable processor implements the features according to the invention of the method, of the component, of the modules or of other aspects and / or partial aspects of the invention. In particular, the processor may include highly parallel computing units and powerful graphics modules.A "storage unit" or a "storage module" and the like can be used in connection with the invention to mean, for example, a volatile memory in the form of a working memory (or a "storage module". Random Access Memory, RAM) or a permanent memory such as a hard disk or a data carrier or, for example, a replaceable memory module. The storage module may also be a cloud-based storage solution.In the context of the invention, a "module" can be understood to mean, for example, a processor and / or a memory unit for storing program instructions. For example, the processor is specifically configured to execute or implement the program instructions of the method according to the invention or a step of the method according to the invention.Radar sensor module 300 contains at least one radar sensor, in particular in the form of a microwave radar. Microwave radars operate with electromagnetic waves in the frequency range of 1 to 100 GHz and are used for navigation, for detecting objects such as aircraft, ships or vehicles and in the industry for monitoring production environments and for automatizing working processes. The radar sensor can be a short-range radar (narrow radar). Short Range Radar, SRR) with a measurement range up to about 30 m, as a central range radar (narrow. Mid-range radar, MRR) with a measurement range up to about 150 m or as a long-range radar (narrow range. Long Range Radar, LRR) can be designed with a measurement range up to approximately 220 m. Short-range radar is used primarily for detecting obstacles in the short range of a vehicle, for example, during parking or lane change. Mid-range radar is used for detecting vehicles in adjacent lanes and for supporting driver assistance systems such as adaptive cruise control. Long-range radar is used primarily for detecting obstacles at a relatively long distance, for example other vehicles on the freeway.The radar sensor module 300 may also include beamforming antennas. Radar beamforming (radar beamforming) is a technique used in radar systems to steer radar energy in a particular direction while suppressing it in other directions. This is achieved by combining the signals of several antenna elements into a beam which can be electronically steered. This improves radar performance. For this purpose, the incoming radar signal is first received by each antenna element. Then, the signals are combined so that the waves of each element constructively add in the desired direction, while they cancel each other out in the other directions. For this purpose, the phase and amplitude of the individual signals are adjusted before the combination. After the signals are combined, the resulting beam can be controlled by adjusting the phase and amplitude of the signals in each antenna element. In this way, the radar system may be directed to a particular target or region of interest while minimizing interference from other sources.Beamforming allows the radar to generate a narrow beam that can capture and track targets with high accuracy, even if noise, interference, and interference are present. In addition, the radar can electronically control the beam in real time without requiring physical movement of the antennas. There are two main types of beamforming: analog and digital. In analog beamforming, the phase and amplitude of the signals from each antenna are adjusted using analog circuitry. In digital beamforming, the signals of the antennas are combined and manipulated by digital signal processing. Radar beamforming is used in various applications such as weather forecast, monitoring, navigation, and communication. Advantages include improved range, resolution, sensitivity, and target detection.In the vehicle 200, in particular for an Advanced Driver Assistance System (ADAS) or for autonomous driving, various types of cameras are used to monitor the environment, increase driving safety, and support autonomous driving functions. The camera device 400 can contain, for example, a simple single camera, which is used for capturing images and videos in a specific direction or for monitoring specific regions and is designed as a digital RGB camera for the three main colors red (R), green (G) and blue (B). This camera can be used for various purposes, for example for reading traffic signs, for recognizing roadway markings or for capturing surrounding images. The camera device 400 may also include a stereo camera and / or a depth camera for acquiring depth information used for calculating 3D information and important for functions such as distance measurement, obstacle detection, and avoidance. In addition, the camera device 400 may include a wide-angle camera for detecting a larger field of view. This is frequently used for parking aids and for identifying approaching vehicles in adjacent lanes. The wide-angle camera can be designed in particular as a fish-eye camera in order to capture a very large field of view of up to 360 degrees. Furthermore, the camera device 400 can have an infrared camera in order to be able to record images even in poor lighting conditions or at night. In particular, at least one camera of the camera device 400 has a high image quality and resolution and a high frame rate in order to enable precise detection of the environment of the vehicle 200. The camera device 400 can have, in particular, a combination of different camera types. In addition, the vehicle 200 may use further sensors, such as a lidar system and / or an ultrasonic system, which acquire data from the environment of the vehicle 200 in addition to the radar sensor module 300 and the camera device 400.Typically, the image rates of a camera of the camera device 400 are between 30 and 60 images per second (fps). Within the scope of the present invention, cameras with an image rate of 60 fps are used in particular in order to be able to capture faster movements and details of objects in the environment of the vehicle 200. This makes sense in object detection and tracking and in fast avoidance maneuvers. However, it may also be provided to use an image rate of 120 fps or more in a high speed range of the vehicle 200.The resolution of a camera of the camera device 400 can range from HD (high definition) to 4K ultra HD. Cameras with an HD resolution of 1280 x 720 pixels are frequently used in backup cameras and parking assist systems. Higher image quality and more details provide full HD resolution of 1920 x 1080 pixels. Cameras with a resolution of 2K (2560 x 1440 pixels) show even more detail and are used in some more sophisticated ADAS applications, particularly in cameras for object detection and tracking. The 4K resolution (3840 x 2160 pixels) provides the highest image quality and detail accuracy. This resolution is used in some high-quality ADAS systems and autonomous driving cameras to capture fine details of the environment.However, the selection of camera resolution depends on the specific requirements for the accuracy of the estimation of the own movement of the vehicle 200, e.g. whether it is a partially or fully autonomous driving vehicle. Higher resolutions provide more details, but are also more computation- and memory-intensive, since higher resolutions generate larger amounts of data. In selecting the camera, it is also necessary to take account of how the data are processed and transmitted.Overall, image quality depends not only on resolution, but also on factors such as image processing technology, the quality of the lens, and the ability of the camera to adapt to different lighting conditions.The camera device 400 records image data 450 of the environment of the moving vehicle 200, and this image data 450 is supplied to the data evaluation module 500. The image data 450 can be the image data of a single camera or the image data of a plurality of cameras of the camera device 400, which are superimposed to form a coherent image 470 either in the camera device 400 and / or in the data evaluation module 500. In order to combine the image data 450 of the various cameras to form the common image 470 of the environment, suitable image processing algorithms, such as in particular image calibration and image fusion, are used. In image calibration, the images of the various cameras are aligned on the same coordinate frame. This includes determining the intrinsic camera parameters (such as focal length and distortion) and the extrinsic parameters (the relative position and orientation of the cameras to each other). In image fusion, pixel values from different images may be combined into a single image. In feature-based fusion, features or objects are identified in the images and the information from different cameras is combined to more accurately determine the position and characteristics of those features.The image 470 generated from the image data 450 thus represents either the image of a single camera or the common image of a plurality of cameras. The image 470 is a first image, generated at a first point in time t 1 of an image sequence 490 which consists of a sequence of individual images F 1, F 2,..., F i,..., F n recorded one after the other. At a second time t 2 a second image 480 is generated, etc. Each frame F i of the image sequence 490 is a still image that was captured by one or more cameras of the camera device 400 at a specific time t i.In a development of the invention, data from sensors such as a lidar system and / or an ultrasonic system can also be added to the images of the image sequence 490 within the scope of a sensor fusion. These sensors provide additional information about the environment and can be included in the fusion to obtain more accurate and complete images of the image sequence 490.Radar data 350 in a detection range is likewise generated by radar sensor module 300 in a temporal sequence. The detection range covers the range that the radar is to monitor. The radar sensor module 300 repeatedly emits electromagnetic pulses and measures the time required to recover the reflected signals from objects in the sensing area. The temporal sequence allows radar sensor module 300 to detect changes in the position and motion of objects over time.As a result, the radar data 350 provides a point cloud of raw radar targets 370 having a particular signal strength in different range cells or angle cells. In radar technology, the signal strength is usually expressed in dBm (decibels milliwatts) or dB (decibels watts). Higher signal strength generally means better visibility of an object.In the context of the present invention, untreated or unfiltered measurement results are referred to as a raw radar target, which radar sensor module 300 receives after it has emitted electromagnetic waves and which are reflected by objects in its surroundings. These raw targets are the direct measurements of radar sensor module 300 before further signal processing steps are performed. These raw targets are usually stored in a special format containing information about the propagation time, amplitude and phase angle of the received signals.At a first point in time t 1 first raw radar targets 370 are thus detected, and at a second point in time t 2 second raw radar targets 380 are detected by radar sensor module 300. The first and second raw radar targets 370, 380 form the starting point for the further radar evaluation according to the invention. For this purpose, the radar data 350 with the first raw radar targets 370 and the second raw radar targets 380 are supplied to the data evaluation module 500.The data evaluation module 500 comprises a software application 550 with algorithms for evaluating the radar data 350 in order to estimate the own movement of the vehicle 200 therefrom. This radar odometry requires anchor points 750, which serve as reference points, on the basis of which the radar can calculate the position and movement changes of the vehicle 200. Anchor points 750 are characteristic and recognizable features in the environment that are detected by the radar to accurately determine the motion of the vehicle 200 and reduce drift in radar odometry. Typically, these are physical features or objects in the environment of the vehicle 200 that can be detected by the radar. These may be, for example, buildings, traffic signs, bridges, light towers, other vehicles or other objects reflecting the radar. Ideal anchor points 750 are stable and are not moved or changed during travel of the vehicle 200 to enable reliable time tracking. In addition, the anchor points 750 should be unique and easily identifiable. This facilitates tracking and mapping of anchor points 750 from one radar shot to the next.The use of anchor points 750 in radar odometry is important for accurate positioning and navigation, particularly in environments where GPS signals are not available or not reliable enough. By combining radar information with known anchor points 750, the vehicle 200 may determine its position and motion with higher accuracy, enabling more precise vehicle control and navigation. This is particularly important for vehicles with driver assistance systems, autonomous vehicles, and robotics applications.It is helpful if the positions of the anchor points 750 are known in advance or can be measured accurately. The radar can then accurately calculate the movements of the vehicle relative to these points. However, this is not always possible, so that reference points or reference objects that have not yet been measured must be used as anchor points 750. However, it is very important to choose the correct reference points as anchor points 750, since only in this way is it possible to estimate the own movement of the vehicle accurately.While a parked vehicle may be used as a stable reference point to track the motion of the own vehicle, moving vehicles, e.g., a 100 m preceding vehicle, are not usually suitable reference points for radar odometry. This is because the speed and the movement of the preceding vehicle may change. In addition, a moving vehicle whose reflections have been selected as anchor points for radar odometry has a Doppler velocity that no longer corresponds to the own movement of the vehicle. Therefore, these reflections affect the result of the vehicle's own motion estimation, making the use of this vehicle as a stable reference point problematic.For radar odometry, the use of stable and predictable reference points is thus of fundamental importance to obtain accurate motion estimates and minimize drift. Parked vehicles, lane markings, buildings, or other fixed objects in the environment may serve as suitable reference points as they do not move and their positions may be known in advance.The selection of suitable anchor points 750 thus plays a key role in improving the accuracy of estimating the own motion of a vehicle, particularly in applications such as autonomous navigation or localization in environments where GPS data is unreliable or unavailable. The selection of stable anchor points 750 that do not move or only slightly change position is important to avoid drift in the estimation of the own motion of the vehicle 200. Unstable or moving anchor points 750 may result in erroneous estimates. Anchor points 750 should also be visible and reliably detectable during the travel of the vehicle 200. If an anchor point 750 is not visible or obscured by obstacles, it should not be used for estimating vehicle own motion.According to the invention, it is therefore provided that the raw radar targets 370, 380 are analyzed with regard to their suitability as anchor points 750 for radar odometry. For this purpose, the first raw radar targets 370 of a first radar recording are projected into the first image 470 of the environment of the vehicle 200 recorded simultaneously by the camera device 450, so that a first fusion image 570 is produced. Subsequently, the second raw radar targets 380 of a second radar recording are projected into the simultaneously recorded second image 480 of the image sequence 490 of the environment of the vehicle 200, so that a second fusion image 580 is produced. This fusion of the corresponding raw radar targets acquired at a time t i can be carried out for each image F i or for selected images F k of the image sequence 490. Overall, a fusion image sequence 590 is produced. The software application 550 of the data evaluation module 500 comprises image processing algorithms for generating the first and second fusion images 570, 580 or the fusion image sequence 590 and for evaluating the same.The image processing algorithms include, in particular, artificial intelligence algorithms to extract and interpret information from the first and second fusion images 570, 580 and other images of the fusion image sequence 590, respectively. These include, in particular, neural networks such as convolutional neural networks (CNNs).Neural networks are often specified as a prototype of AI algorithms. Neural networks are also interesting because they can be combined so that the learning success can be increased. Neural networks coupled in this way are also referred to as deep learning. A neural network is composed of neurons which are arranged in a plurality of layers and are connected to one another differently. A neuron is capable of accepting information from the outside or from another neuron at its input, evaluating the information in a specific manner and forwarding it in modified form at the neuron output to another neuron or outputting it as a final result. Hidden neurons are arranged between the input neurons and output neurons. Depending on the network type, there may be multiple layers of hidden neurons. They provide for the forwarding and processing of the information. Output neurons ultimately deliver a result and output it to the outside world. The arrangement and the linking of the neurons result in different types of neural networks such as forward-directed networks (engl. Feedforward Neural Networks (FNN), Feedback Neural Networks (FNN). Recurrent Neural Networks (RNNs) or Convolutional Neural Networks (RNNs). Convolutional Neural Networks (CNN)).CNNs have been developed specifically for image processing and use layers of convolution operations, pooling, and activation functions to extract features from images. They are frequently used for tasks such as image classification, object recognition and segmentation.Furthermore, encoder-decoder architectures can also be used. Encoder-decoder networks are particularly suitable for tasks such as image segmentation and reconstruction. The encoder extracts features from the input image, while the decoder uses these features to generate an output.Moreover, classification algorithms can be used to classify and identify the objects in the first and second fusion images 570, 580 and the images of the fusion image sequence 590 into different categories or classes. Classification models may be based on CNN or other network architectures and use techniques such as softmax activation functions to generate probabilities for different classes.Optical flow algorithms can be used to track the movement of pixels or features in the image sequences 490, 590. The optical flow describes the apparent movement of pixels from one image to the next in an image sequence due to the actual movement of objects or camera movement. An optical flow algorithm attempts to estimate the apparent displacement of features (e.g., points or texture patterns) between two successive images in an image sequence. This allows the movement of objects in image sequences to be tracked. There are various algorithms and approaches for calculating the optical flow, including block-based, correlation-based, and variation-based methods. The selection depends on the specific application and the requirements for accuracy and speed.Moreover, tracking with Kalman filters can be used. The Kalman filter is a mathematical algorithm designed to estimate the state of a dynamic system based on noisy and incomplete measurements. In image processing, the Kalman filter is frequently used to track objects in images or image sequences. The Kalman filter first estimates the current state of the object to be tracked (e.g., its position and velocity). Using the system's physical equations of motion, the next state is predicted. In each time step, measurements from cameras or radar sensors are used to correct or update the current state of the object. The Kalman filter combines the prediction and measurement using weights to obtain an improved estimate of the current state of the object. The Kalman filter can track the motion of objects in images, particularly when the measurements are noisy or there is uncertainty about the exact position of the object.Another method is matching feature points, in which characteristic features or points in different images are identified and assigned in order to track the movement or position of objects. These features may be corners, edges, texture points, or other characteristic structures in the images. Characteristic features are extracted from each image for this purpose. For each extracted feature, a descriptive representation is created that allows this feature to be recognized in different images. This descriptive representation is referred to as a feature vector. The features in one image are compared to the features in another image to find matches. This can be done by comparing feature vectors and using distance metrics such as the Euclidean distance. The matches found between the images are used to track the movement or position of objects between the images.These algorithms and techniques can be used according to the invention in various combinations and modifications in order to select suitable anchor points 750 for radar odometry from the measured raw radar targets 370. If it is determined during the evaluation of the first and second fusion images 570, 580 or of the images of the fusion image sequence 590 that an object assigned to a radar raw target 370, 380 has moved in the time interval ΔT=t 2- t 1 then according to the invention this radar raw target 370, 380 is not taken into account as a static anchor point 750. By contrast, those radar raw targets 370, 380 which can be assigned to stationary objects are used as anchor points 750 for radar odometry.By selecting suitable anchor points 750 according to the present invention, the accuracy of estimating the own movement of vehicle 200 may be improved, since anchor points with inaccurate position data are avoided. The anchor points 750 selected from the raw radar targets 370, 380 have accurate and well known position data and thus serve as stable reference points for radar odometry. The positions of the unselected raw radar targets 370, 380, on the other hand, are uncertain, so they may impair the accuracy of estimation of the own motion of the vehicle 200.For a reliable estimation of the own movement of the vehicle 200 in all directions, it is also important to ensure a sufficient distribution and redundancy of the anchor points 750 in the environment of the vehicle 200. The redundancy of the anchor points 750 can increase the accuracy and reliability of the radar odometry.It may additionally be provided that the software application 550 classifies and identifies the type of objects, e.g., whether it is a parked vehicle, a traffic light, or a distinctive architectural feature such as a pillar on a building. The parked vehicle is an object that is at rest only for a moment, while the traffic light and the column are static objects. This distinction allows further differentiation of the anchor points 750.However, it may occur that there are not sufficient suitable anchor points 750 in the environment of the vehicle 200, e.g. because there are only a few stationary objects. This is the case, for example, in an open landscape, such as an agricultural landscape with cereal fields. Such landscapes are often characterized by a lack of solid buildings or structures that could serve as stable anchor points. In addition, the environment is often very homogeneous, with no distinctive features that can be used to estimate the own motion of the vehicle. In addition, objects such as trees or plants move, making their use as stable anchor points more difficult.In this case, it is necessary to also use the raw radar targets 370, 380 of moving objects such as a preceding vehicle as the anchor mobile points 780. Since the method according to the invention discloses which raw radar targets 370, 380 are to be assigned to moving objects, it is provided in a development of the invention to estimate the speed caused by the own movement of the moving object, for example by means of optical flow methods. The positions of the mobile anchor points 780 may then be corrected by a certain factor as if they were in the idle state. Furthermore, it is possible to weight the mobile anchor points 780 with respect to the static anchor points 750. This has the effect that the mobile anchor points 780 have a lower influence on the estimation of the own movement of the vehicle 200 than the anchor points 750 of selected raw radar targets 370, 380.The anchor points 750, 780 determined in this way are then output by an output module 700 or transmitted to corresponding control devices of the vehicle 200, such as in particular a driver assistance system (ADAS).FIG. 2 shows the method steps for improving the accuracy of radar-based estimates of the own movement of a vehicle 200.In a step S 10, radar data 350 in a detection range in the environment of the vehicle 200 are detected by the radar sensor module 300 in a temporal sequence, wherein the radar data 350 comprise at least first raw radar targets 370 at a first point in time t 1 and second raw radar targets 380 at a second point in time t 2.In a step S 20, image data 450 for an image sequence 390 comprising at least a first image 370 and a second image 380 are recorded.In a step S 30, the raw radar targets 370 detected at the first time t 1 are fused with the first image 470 recorded at this first time t 1 to form a first fusion image 570.In a step S 40, the raw radar targets 380 detected at the second time t 2 are fused with the second image 480 recorded at this second time t 2 to form a second fusion image 580.In a step S 50, objects with first raw radar targets 370 in the first fusion image 570 and objects with second raw radar targets 380 in the second fusion image 580 are classified and identified.In a step S 60, the first fusion image 570 and the second fusion image 580 are compared with respect to the movement of the classified and identified objects.In a step S 70, anchor points 750 are determined from the radar raw targets 370, 380 for carrying out a radar odometry, wherein a radar raw target 370, 380 is selected as anchor point 750 only if the radar raw target 370, 380 can be assigned to an object that has not moved in the time interval ΔT=t 2- t 1.FIG. 3 schematically illustrates a computer program product 900 comprising executable program code 950 configured to perform the method according to the first aspect of the present invention when executed.With the method according to the present invention, suitable anchor points for radar odometry in the environment of a vehicle can be identified. By fusing the raw radar targets with the images of the image sequence recorded by a camera device, it can be established whether the objects to be assigned to the raw radar targets are moving dynamically or are static. The comparison of the images of the fusion image sequence provides information about time sequences, movement patterns, changes and other important aspects that cannot be recognized from the static individual images alone. Only those raw radar targets that can be assigned to static objects within a time interval are selected as anchor points.These anchor points serve as reference points to estimate the position and movement of the vehicle and to enable reliable navigation. The selection of suitable anchor points is particularly important for driver assistance systems (ADAS), autonomous vehicles, and other systems that require accurate estimation of self-motion to make decisions for safe navigation and control of the vehicle. Accurate estimation of the own motion of a vehicle is therefore crucial to the safety and performance of driver assistance systems and autonomous vehicles and can be significantly improved by the present invention.Reference numerals denote reference numerals100 System 200 Vehicle 300 Radar sensor module 350 Radar data 370 First raw radar target 380 Second raw radar target 400 Camera device 450 Image data 470 First image 480 Second image 490 Image sequence 500 Data evaluation module 550 Software application 570 First fusion image 580 Second fusion image 590 Fusion image sequence 700 Output module 750 Anchor point 780 Mobile anchor point 900 Computer program product 950 Program code

Claims

Method for improving the accuracy of radar-based estimates of the own movement of a vehicle (200), wherein at least one radar sensor module (300) having one or more radar sensors, at least one camera device (400) having one or more cameras and a data evaluation module (500) having a software application (550) are used, comprising: - acquiring (S10), by the radar sensor module (300), radar data (350) in an acquisition region in the environment of the vehicle (200) in a temporal sequence, wherein the radar data (350) comprise at least first raw radar targets (370) at a first point in time (t 1) and second raw radar targets (380) at a second point in time (t 2) ; capturing (S 20) of image data (450) for an image sequence (390) having at least a first image (370) and a second image (380); - fusing (S 30) the raw radar targets (370) captured at the first point in time (t 1) with the first image (470) captured at this first point in time (t 1) to form a first fusion image (570); - fusing (S 40) the raw radar targets (380) captured at the second point in time (t 2) with the second image (480) captured at this second point in time (t 2) to form a second fusion image (580); classifying (S50) and identifying objects with first raw radar targets (370) in the first fusion image (570) and objects with second raw radar targets (380) in the second fusion image (580); comparing (S60) the first fusion image (570) and the second fusion image (580) with respect to the movement of the classified and identified objects; determining (S70) anchor points (750) from the raw radar targets (370, 380) for carrying out a radar odometry, wherein a raw radar target (370, 380) is selected as the anchor point (750) only if the raw radar target (370, 380) can be assigned to an object that has not moved in the time interval ΔT=t 2- t 1.Method according to Claim 1, wherein, in the event of too low a density of anchor points (750) in a surrounding area of the vehicle (200), mobile anchor points (780) from raw radar targets (370, 380) which are assigned to objects which have moved in the time interval ΔT = t 2- t 1 are additionally used for the radar odometry and these mobile anchor points (780) are provided with a weighting.Method according to Claim 1 or 2, wherein a radar sensor of the radar sensor module (300) is designed as a short-range radar having a measurement range of up to approximately 30 m or as a medium-range radar having a measurement range of up to approximately 150 m or as a long-range radar having a measurement range of up to approximately 220 m or as a microwave radar.The method according to any one of claims 1 to 3, wherein the data evaluation module (500) comprises an image fusion and image analysis software application (550) using artificial intelligence algorithms.Method according to Claim 4, wherein neural networks and / or classification algorithms and / or optical flow algorithms and / or matching of feature points and / or tracking using Kalman filters are used as algorithms.Method according to one of Claims 1 to 5, wherein the camera device (300) comprises one or more cameras, wherein a camera is configured as a digital RGB camera, stereo camera, depth camera, wide-angle camera and / or infrared camera.The method of any one of claims 1 to 6, wherein additional information from a lidar system and / or an ultrasound system is integrated into the first and second images (470, 480) and / or the image sequence (490) and / or the first and second fusion images (570, 580) and / or the fusion image sequence (590).System (100) for improving the accuracy of radar-based estimates of the own movement of a vehicle (200), comprising at least one radar sensor module (300) having one or more radar sensors, at least one camera device (400) having one or more cameras and a data evaluation module (500) having a software application (550), wherein the radar sensor module is designed to acquire radar data (350) in a detection region in the environment of the vehicle (200) from the radar sensor module (300) in a temporal sequence, wherein the radar data (350) comprise at least first raw radar targets (370) at a first point in time (t 1) and second raw radar targets (380) at a second point in time (t 2) ; wherein the camera device (400) is configured to record image data (450) for an image sequence (390) having at least a first image (370) and a second image (380); wherein the software application (550) of the data evaluation module (500) is configured to merge the radar raw targets (370) recorded at the first point in time (t 1) with the first image (470) recorded at this first point in time (t 1) to form a first fusion image (570), to merge the radar raw targets (380) recorded at the second point in time (t 2) with the second image (480) recorded at this second point in time (t 2) to form a second fusion image (580), classifying and identifying objects having first raw radar targets (370) in the first fusion image (570) and objects having second raw radar targets (380) in the second fusion image (580), comparing the first fusion image (570) and the second fusion image (580) with respect to the movement of the classified and identified objects, and determining anchor points (750) from the raw radar targets (370, 380) for performing radar odometry, wherein a raw radar target (370, 380) is selected as the anchor point (750) only if the raw radar target (370, 380) can be assigned to an object that has not moved in the time interval ΔT=t 2- t 1.The system (100) according to claim 8, wherein if the density of anchor points (750) in a surrounding area of the vehicle (200) is too low, mobile anchor points (780) from raw radar targets (370, 380) associated with objects that have moved in the time interval ΔT = t 2- t 1 are additionally used for radar odometry and these mobile anchor points (780) are provided with a weighting.The system (100) according to claim 8 or claim 9, wherein a radar sensor of the radar sensor module (300) is configured as a short-range radar having a measurement range of up to about 30 m or as a middle-range radar having a measurement range of up to about 150 m or as a long-range radar having a measurement range of up to about 220 m or as a microwave radar.The system (100) according to any one of claims 8 to 10, wherein the data evaluation module (500) comprises an image fusion and image analysis software application (550) using artificial intelligence algorithms.The system (100) according to claim 11, wherein neural networks and / or classification algorithms and / or optical flow algorithms and / or matching of feature points and / or tracking with Kalman filters are used as algorithms.The system (100) according to any one of claims 8 to 12, wherein the camera device (300) comprises one or more cameras, wherein a camera is configured as a digital RGB camera, stereo camera, depth camera, wide-angle camera and / or infrared camera.The system (100) of any of claims 8 to 13, wherein additional information from a lidar system and / or an ultrasound system is integrated into the first and second images (470, 480) and / or the image sequence (490) and / or the first and second fusion images (570, 580) and / or the fusion image sequence (590).A computer program product (900) comprising executable program code (950) configured to, when executed, perform the method of any one of claims 1 to 7.

Images