Method and motor vehicle for detecting moving objects in the environment surrounding a vehicle

Abstract

The present invention relates to a method for detecting moving objects in the vehicle surroundings and a motor vehicle designed accordingly. Here, camera data and radar echoes from the surroundings are acquired. By comparing the relevant azimuth angles and radar echoes with a specific distance of an object recognized based on the camera, an association is established between at least one radar echo and a bounding box generated based on the camera data of the object recognized based on the camera. If the assignment is successful, the distance assumed based on the camera is corrected according to the radar-determined distance of the correspondingly recognized surrounding object. Then, the corresponding bounding box with the corrected distance is output as an object data set representing a successful object detection.

Description

Technical Field

[0001] The present invention relates to a method for detecting moving objects in the surrounding environment of a vehicle and a motor vehicle designed to carry out such a method. Background Art

[0002] In the field of vehicle technology, for various driving assistance systems and driving functions, especially for highly automated or autonomous driving, detecting objects in the surrounding environment of the corresponding vehicle is particularly important. Such objects can in particular be other road users. Here, as fast and reliable detection as possible and precise localization especially of moving objects is particularly important, because dangerous situations can occur particularly frequently and rapidly due to these objects. Therefore, further improvements in this field are always desirable.

[0003] As a solution to this, WO 2017 / 207727 A1 describes an improved object detection and object motion state estimation. Here, relevant azimuth angles and Doppler velocity components are determined for radar detection. In addition, an object is captured by a camera, and an optical flow is determined from at least two camera images. Thereby, another velocity component in the vertical direction is determined. Then, a complete velocity vector of the corresponding object is determined based on the two velocity components.

[0004] As another solution, a system for real-time control of a device is known from US 2018 / 0259637 A1. In this case, an object that is inclined, i.e., in an inclined position relative to the device, is optically detected. The horizontal angle and the radial distance of the object relative to the device and the rate of change of the distance are detected by a radar device. Then, the device is controlled based on a 3D position matrix and a 3D rate-of-change matrix determined based on visual data and radar data.

[0005] As another solution, an object detection system is described in EP 3229041 A1, which includes a radar sensor and a camera. In this case, the radar signal is intended to indicate the distance and direction of an object in the surrounding environment of the vehicle. The image captured by the camera should overlap the field of view of the radar signal. Then, a distance map of the image should be determined based on the distance and direction of the object. Based on this, a detection area is defined in the image. Then, only this detection area of the image is processed to identify the object. Summary of the Invention

[0006] The object of the present invention is to be able to improve the detection of moving objects in the surrounding environment of a vehicle. According to the present invention, this object is achieved by the subject matter of the independent claims. Advantageous designs and improvements of the present invention are set forth in the dependent claims, the description, and the drawings.

[0007] The method according to the invention is used to detect objects moving in the surroundings of a motor vehicle. The surroundings can hereby be determined by the detection range or the sensor range of at least one camera and at least one radar device of the motor vehicle. In this method, camera data of the surroundings are acquired. Here, the cameras of the motor vehicle can be used to capture the camera data. Similarly, the acquisition of the camera data can include or mean detecting or receiving them by means of a data processing device which is set up to carry out this method, either by the camera or via a corresponding data interface. Thus, the camera data depict the surroundings of the motor vehicle. Furthermore, in the method according to the invention, radar echoes from the surroundings of the motor vehicle are acquired. The radar echoes can be acquired, i.e., measured, by means of a radar device. Similarly, the acquisition of the radar echoes can mean or include detecting or receiving them by means of a data processing device, either by the radar device or via a corresponding data interface. In particular, these are radar echoes which are caused by at least one radar pulse emitted by the radar device into the surroundings and reflected back to the radar device by an object located in the surroundings. Thus, the radar echoes represent objects in the surroundings of the motor vehicle and, in this sense, also depict the surroundings of the motor vehicle.

[0008] In a method step of the method according to the invention, objects imaged in the camera data are identified by means of the camera data, where, for the identified objects, the azimuth angle or the horizontal angle range in which the object is located is determined respectively from the perspective of the motor vehicle, and a bounding box (technical term in the art: bounding box) surrounding the respective object is generated, which has a supposed or estimated distance from the motor vehicle. In order to identify objects in the camera data, i.e., in particular in the respective camera images, image processing or object recognition means or algorithms known per se can be used. Thus, objects can be detected, for example, by means of edges, contours, connected surfaces, etc. The azimuth angle range is hereby defined in a plane which can, for example, extend parallel to the floor of the motor vehicle. In particular, this can be a possibly idealized or smooth plane of the ground on which the motor vehicle is traveling, i.e., for example, a roadway surface, etc. The azimuth angle range can hereby be defined with reference to a specified direction. For example, as a reference or zero direction, a fictitious direction extending forward from the motor vehicle, in particular from the camera, or in the driving direction (also referred to as the x-direction) can be defined.

[0009] The assumed or estimated distance of the corresponding object or the corresponding bounding box can be determined or fixed, for example, based on predefined assumptions about the width or length of different objects. The optical characteristics of the camera can be considered here, which, for example, determine how large an object with a specific actual size at a specific distance from the camera appears in the camera data or in the corresponding camera image. The estimated or predefined relative size relationships between different objects imaged in the camera data can also be considered here. For example, if a car appears as large as a building in the camera data, or if a pedestrian appears as wide as a car, for example, these objects can be considered to be at different distances from the camera and thus at different distances from the motor vehicle. This in turn allows conclusions to be drawn about the approximate distance of the corresponding object from the motor vehicle, possibly taking into account the predefined typical sizes of these objects. Similarly, for example, environmental features imaged and detected in the camera data between the corresponding object and the camera, such as road width or lane markings, etc., can be used to estimate the distance.

[0010] Particularly preferably, the corresponding bounding box can be projected, in particular, into a plane that is also used for the azimuth range based on the assumed distance in the 2D space. For this plane, a predefined coordinate system can be used here, in which the motor vehicle or the camera or the environmental sensor system of the motor vehicle including the camera and the radar device can be arranged, for example, at the zero point of the coordinate system. In particular, the predefined coordinate system can be the camera coordinate system, i.e., the coordinate system in which the camera data is detected or processed or managed, and the bounding box is also located in this coordinate system, for example.

[0011] In another method step of the method according to the invention, for the radar echo, a relevant Doppler velocity is determined with reference to the current position of the motor vehicle. The Doppler velocity in this sense is here the radial velocity of the respective object that generates the respective radar echo, which radial velocity has been corrected or compensated for the intrinsic velocity of the motor vehicle, i.e., reduced or increased according to the sign. The radar device or the motor vehicle or its current position can thus be regarded as the center point of a circle, and the respective object is arranged on the circumference of this circle. The Doppler velocity then indicates the velocity component of the object along the radius of the circle - towards the center point or away from the center point - independently of the self-movement of the motor vehicle, i.e., in a world-fixed coordinate system or with reference to this coordinate system. For some radar echoes, a non-vanishing, i.e., non-zero, Doppler velocity has been determined here, and these radar echoes are called moving radar echoes here. At least for the determined moving radar echoes, the relevant distance and the relevant azimuth angle from the motor vehicle are determined. These distances and azimuth angles are also, in particular only, determined with the aid of the measured radar echoes or the corresponding radar data, i.e., with the aid of the measurement data of the radar device. With the aid of the respective determined distance and the respective azimuth angle, a 2D position is generated for the respective radar echo. This 2D position, i.e., effectively the respective radar echo or the object that causes the respective radar echo, can then also be projected or input into the mentioned 2D space or the mentioned plane, i.e., into the coordinate system for the camera data. The radar echo can also be projected or transmitted into the corresponding camera image. The radar echo or the radar echo combined with the determined distance and azimuth angle can thus be understood as a point distribution or a point cloud in the respective 2D space or coordinate system. The azimuth angle determined based on radar may exhibit greater uncertainty or measurement inaccuracy due to the principle involved. To take this into account hereinafter, the azimuth angle range determined based on the camera can preferably be extended or the tolerance range increased on both sides or in two directions. This tolerance range can correspond to the predetermined or assumed uncertainty or measurement inaccuracy of the azimuth angle determination based on radar. Particularly preferably, the tolerance range can be determined as a multiple of the standard deviation, in particular the 3σ range, so that thus approximately 99.7% of all corresponding measured values are detected.

[0012] In another method step of the method according to the invention, an association is made between at least one moving radar echo and at least one bounding box. For this purpose, the respective azimuth angle is compared with the determined azimuth angle range of the bounding box, which azimuth angle ranges are used here in particular as a reference or ground truth, and the respective radar-determined distance (which is used here in particular as a reference or ground truth) is compared—directly or indirectly—with the camera-assumed distance. Here, preferably, the comparison of the distances can be performed based on the respective scaling error, which will be explained in more detail below. Then the comparison can be made particularly based on a percentage rather than absolutely, i.e., not in meters, for example. Thus, for example, the quotient of the respective radar-determined distance and the respective camera-assumed distance can be formed and evaluated, i.e., compared with a predetermined reference value or threshold value, for example.

[0013] In the sense of the present invention, making an association can particularly mean performing these comparisons, where, if the comparison is successful, i.e., for example, the respective predetermined criteria are met, the association can be successful, and if the comparison fails, i.e., for example, at least one predetermined criterion is not met, the association can fail or be discarded. For example, the comparison can be successful when the values or data being compared or contrasted with each other have a predetermined similarity to each other, i.e., for example, differ from each other by a maximum of a predetermined value or amount, etc. Other details regarding the comparison are explained in more detail below. Ultimately, it is determined here whether there is or can be found a combination that matches in the respective characteristics and consists of a bounding box and at least one moving radar echo. For example, this would be the case if one or more moving radar echoes are generated by an object and directly reflected back to the radar device and the radar echo is surrounded or represented by the respective bounding box. In different cases or designs of the present invention, here only one radar echo or several radar echoes can be assigned to the bounding box. Here, particularly preferably, for example, by means of a corresponding query or by removing the successfully assigned radar echoes from the set of radar echoes still available for assignment, the double assignment of a particular radar echo to different bounding boxes, i.e., the double assignment of the radar echo, is automatically prevented. If it is possible or reasonable to assign the radar echo to different bounding boxes based on the data or characteristics mentioned, then the respective probability that the radar echo belongs to one or the other bounding box or object can be determined. Then the assignment can be performed with a greater probability. The determination of such a probability will be explained in more detail below.

[0014] In another method step of the method according to the invention, at least one moving radar echo has been successfully assigned to some bounding boxes. For these bounding boxes, the correspondingly assumed distance is corrected according to the radar-determined distance of the respectively assigned radar echo. In this case, the assumed distance can be replaced, for example, by the distance of the assigned radar echo. If multiple radar echoes have been assigned to a bounding box, the assumed distance of the bounding box can be replaced, for example, by the average value of the distances determined for the assigned radar echoes. Similarly, when correcting the assumed distance, for example, a predetermined or estimated measurement error or measurement inaccuracy of the radar-determined distance, the inaccuracy or uncertainty of the assumed distance, etc. can be considered. According to the radar-determined distance, or the distance of the detected object from the motor vehicle, the corresponding bounding box can be moved in the 2D space or in the plane.

[0015] Then, the bounding box with the correspondingly corrected distance is output, provided, or further used as an object data set, which indicates a successful object detection. Then, the object data set can be used, for example, or be used as the basis for further calculations, such as the basis or input data of a Kalman filter, for the driving function or control process of the motor vehicle and / or the like. The object data set can be supplemented here by other data or characteristics in order to, for example, more precisely characterize the corresponding object. For example, the corresponding Doppler velocity, the azimuth of the corresponding object, its yaw rate, its type or category, one or more assigned probabilities or confidences, and / or the determined individual values or characteristics, etc. can be added to the object data set. For example, the object data set can be used for tracking, i.e., tracking the corresponding object, or for filling a dynamic map grid or raster and / or the like.

[0016] In the context of the present invention, a detection object can thus mean or include identifying the presence of an object in the surroundings, localizing the object, i.e., determining its position relative to the motor vehicle, and possibly determining the relative movement between the object and the motor vehicle, in particular determining the direction of movement of the object relative to the motor vehicle in the radial direction. For this purpose, camera data and radar data are combined or fused with each other in the present case. These two types of sensors or data advantageously have different advantages and disadvantages in terms of object detection, and thus advantageously complement each other in order to achieve particularly accurate and reliable object detection. Thus, the azimuth range in which the object is located can be determined particularly precisely by means of the camera or by means of the camera data, and the corresponding object can be classified, i.e., the type or category of the corresponding object can be determined, for example. In contrast, by means of the radar device or by means of the radar data or radar echoes, the distance and the Doppler velocity of the corresponding object can be determined particularly precisely relative to the radar device or the motor vehicle equipped with the radar device, respectively. A particular advantage of the method is that the raw data here can be combined or fused with each other. The radar echoes, which may include the associated Doppler velocity and distance as well as the associated azimuth, do not themselves represent a complete object detection in the traditional sense, because, for example, the radar data themselves have not been processed to the extent that complete objects can be identified based on them and corresponding bounding boxes can be generated. Rather, in the present invention, the fusion or combination of the camera data and the radar data takes place at an earlier point in the data processing chain, since here, before the object detection is completed, the camera data or the bounding boxes generated therein or generated based on it are associated with the raw radar data, i.e., the individual radar echoes. As a result, by using the present invention, objects can be detected with improved reliability and accuracy and more quickly and with less data processing effort compared to conventional detection methods. It is particularly advantageous here that the object detection according to the method of the present invention can be achieved based on a single radar measurement cycle and a single camera image. A single camera image here can be a single frame that has been acquired. A single radar measurement cycle can be, for example, the single emission of a radar pulse train, i.e., the single sampling or scanning of the surroundings or the detection area or field of view of the radar device, and the acquisition of the resulting radar echoes. Depending on the design or configuration of the radar device, for example 1024 signal ramps of different frequencies if necessary can be emitted in one radar measurement cycle. Such a measurement cycle can be performed, for example, every 20 milliseconds. Depending on the design of the camera, a camera image can be acquired, for example, every 50 milliseconds, or, for example, synchronously with the radar measurement cycle or the radar pulses.

[0017] With the present invention, it is advantageously possible not only to determine the position of an object in the surroundings of a motor vehicle with particular precision, but also to determine particularly precisely whether the corresponding object is relevant or highly relevant, i.e., whether it moves automatically relative to the motor vehicle, in particular towards the motor vehicle, and what type of object it is. Compared with conventional detection methods, the use of the present invention can advantageously achieve a significant reduction in the initialization time and a combined measurement or object detection based on two different sensor types. By combining the camera data and radar data proposed herein, it is advantageously possible to achieve a particularly high confidence level in the resulting object detection, since the successful object detection is based on different measurement principles and measuring devices. For example, compared with a purely radar-based object detection, it is possible to particularly reliably identify that a specific radar echo originates from a relevant object and not from clutter, i.e., not from interference data. Therefore, all possible subsequent processing or procedures can also be advantageously carried out with higher reliability and credibility, so that ultimately, for example, the motor vehicle can be operated more reliably and safely. The method according to the invention can preferably be used for autonomous driving functions in urban areas, i.e., in the city environment, and in non-urban areas.

[0018] In an advantageous refinement of the invention, the method is carried out for the radar echoes of each individual measurement cycle. If no camera data acquired simultaneously are available for the acquisition time of the measurement cycle, the available camera data are interpolated or extrapolated to the acquisition time of the measurement cycle. For example, interpolation can be carried out between a camera image acquired before the radar echo and immediately after the end of the measurement cycle, in which the radar echo is acquired. If at least one object has moved during the time between the acquisition time points of these two camera images, the position and / or orientation of the corresponding bounding box can be moved or adjusted, for example, by interpolation between these two camera images, such that it - at least approximately or on average - more precisely corresponds to the actual position or orientation of the corresponding object at the acquisition time of the radar echo. Conversely, if no camera images acquired either during the acquisition time of the radar echo (i.e., during the measurement cycle) or afterwards are available, the position and / or orientation of the corresponding object can be extrapolated from one or more camera images acquired before the measurement cycle. Due to the usually used acquisition frequencies for the camera data and the measurement cycle, as well as the actual speeds of the objects in road traffic, the improvement can be achieved with a sufficiently high degree of reliability compared to using non-interpolated or non-extrapolated camera data. The interpolation or extrapolation of the camera data at the acquisition time point of the measurement cycle can be carried out, for example, using the time stamps of the corresponding data, which can be automatically assigned or set when the data are acquired accordingly. By interpolating or extrapolating the camera data, a more precise and reliable association between the corresponding radar echo and one or more bounding boxes or the corresponding object can be advantageously determined. Here, linear interpolation or extrapolation can be particularly preferably used, which can be advantageously carried out with a particularly small amount of computation and thus particularly quickly, while having sufficient accuracy and reliability.

[0019] In another advantageous design of the invention, for comparing distances, the radar echo and the bounding box are embedded in a predetermined or common coordinate system, in particular the aforementioned camera coordinate system, the center or zero point of which corresponds to the position of the camera. For at least the moving radar echo, a corresponding scaling error or scaling factor is determined, which indicates the distance of the corresponding radar echo from the corresponding bounding box in the radial direction in the predetermined coordinate system with reference to the current position of the motor vehicle, in particular with reference to the current position of the camera. However, the Doppler velocity direction of the corresponding object can be determined in the radar coordinate system (the center or zero point of which corresponds to the current position of the radar device or its radar sensor, respectively), without having to be converted or transmitted to the camera coordinate system.

[0020] Then consider the correspondingly determined scaling error or scaling factor, as it must be less than a predetermined threshold or lie within a predetermined range or interval in order to successfully assign the corresponding radar echo to the corresponding bounding box. Here, the minimum scaling error and the maximum scaling error can be calculated for the corresponding radar echo. The minimum scaling error represents the distance of the corresponding radar echo or the position of the corresponding radar echo from the nearest point or edge of the corresponding bounding box in a predetermined coordinate system. Correspondingly, the maximum scaling error represents the distance of the corresponding radar echo or its position from the farthest point or edge of the bounding box. These distances can be determined, in particular, along the radius line extending from the corresponding radar echo to the motor vehicle, at least in those cases where such a radius line touches or intersects the bounding box. If the corresponding radius line extends close to the corresponding bounding box, then, for example, an auxiliary line extending perpendicular to the corresponding radius line and intersecting the bounding box can be used. Then the minimum scaling error is obtained as the distance between the radar echo and the nearest intersection point of the auxiliary line and the radius line, through which the auxiliary line intersects the bounding box. Similarly, the maximum scaling error can be used as the distance between the radar echo and the intersection point of the radius line and the auxiliary line, which is the farthest from the radar echo and still causes the auxiliary line to touch or intersect the bounding box. Additionally or alternatively, an average scaling error can also be calculated, which, for example, corresponds to the distance of the corresponding radar echo from the center point or from the midpoint between the points used for calculating the minimum and maximum scaling errors located in the bounding box.

[0021] For example, the error or inaccuracy of the distance estimated using camera data and the corresponding position of the bounding box in a predetermined coordinate system can be within a range of 30%. Thus, for example, ±30% or for example ±50% can be specified here as the threshold of the scaling error. The scaling error can be signed, depending on whether the bounding box is between the radar echo and the motor vehicle or on the side of the radar echo away from the motor vehicle. For a predetermined threshold or for example a corresponding gating of 50%, for the assignment, then those radar echoes with a scaling error, for example, in the range of -0.5 to +0.5 can be adopted or considered. An alternative design can also be used here. For example, if the radar echo is within the bounding box, the scaling error can be constantly set to 1. In the case of a predetermined threshold of, for example, 50%, those radar echoes with a scaling error in the range of 0.5 to 1.5 are considered. Radar echoes with a scaling error outside the so-determined range or higher than the predetermined threshold can be discarded or not used or considered for assignment to the corresponding bounding box. In a specific example, for an object, the camera-based assumed distance can be 60m, and the distance determined based on radar can be 40m. Thus, a scaling factor of 60m / 40m = 1.5 is obtained. For example, the scaling error here is 0.5. As a valid value for a successful assignment, for example, a range of 0.5 - 1.5 of the scaling factor and / or an upper threshold of 0.5 for the scaling error can be specified. So in this example, the assignment based on distance will be successful.

[0022] Taking into account the respective advantages and disadvantages of different basic measurement methods, using the scaling error described here as the criterion for assigning radar echoes to a specific bounding box advantageously enables a robust fusion of camera data and radar data.

[0023] In another advantageous design of the present invention, a probability function is specified for the azimuth angle and / or for the scaling error, and the corresponding assignment is evaluated with the aid of this probability function. Here, for the radar echo, the scaling error in a predefined or common coordinate system, in particular in the mentioned camera coordinate system, respectively indicates its distance from the corresponding bounding box, and can thus particularly correspond to the scaling error or scaling factor described elsewhere. The probability function can preferably have a continuously extending central region, from which the probability function or the corresponding probability decreases towards both sides. For example, if the scaling error of the corresponding radar echo is within a predefined range or interval or below a predefined threshold, a probability of 1 can be assigned according to the probability function of the scaling error of the association or the actual relevance between the radar echo and the bounding box. For radar echoes with a measured scaling error outside the predefined range or above the predefined threshold, their assignment to the bounding box or the correctness of such an assignment, i.e., ultimately the actual relevance between the radar echo and the bounding box or the corresponding object, can be assigned a probability that decreases with increasing distance according to the predefined probability function of the scaling error. If the minimum and maximum scaling errors have been calculated for the radar echo, the higher of the two corresponding probability values can be used, i.e., the one assigned to the radar echo or assigning it to the corresponding bounding box. Similarly, for example, if the radar echo is within the azimuth angle range covered by the corresponding bounding box in the predefined coordinate system, a probability or a probability value of 1 can be assigned to the radar echo or its association with the bounding box. Radar echoes located outside this azimuth angle range or their association with the corresponding bounding box can be assigned a probability that decreases with increasing distance from the corresponding azimuth angle range of the bounding box according to the probability function of the azimuth angle.

[0024] The relevant probability of the radar echo finally successfully assigned to the bounding box or the total probability value calculated therefrom - for example, the average value of the probabilities - can then be added as additional information to the corresponding object data group. Based on this, for example, the detection can be weighted. This can advantageously achieve, for example, a graded reaction or the prioritization of potential hazards or measures when operating or controlling a motor vehicle. Thus, for example, it can be stipulated that a certain measure, such as an avoidance maneuver or braking, is only executed when an object is detected in the path of the motor vehicle with a predefined minimum probability.

[0025] For example, a probability threshold can also be specified. Then, all radar echoes can be assigned to the corresponding bounding box for which, for their assignment or association with the bounding box, a probability of at least corresponding to the probability threshold is determined. In other words, gating or windowing can be performed based on the determined probability. Correspondingly, if one or more probability values determined for the corresponding radar echo are below the predefined probability threshold, the association between the radar echo and the bounding box can be discarded or failed accordingly.

[0026] By determining and taking into account probabilities as presented here, additional flexibility of the method according to the invention can be advantageously provided, and a corresponding graded or soft consideration of such an assignment or object detection can also be made in cases where further data processing steps cannot be clearly carried out. Ultimately, this contributes to a more reliable and safer control of the motor vehicle.

[0027] In another advantageous design of the invention, if at least three radar echoes have been assigned to a bounding box, a Doppler velocity distribution is determined with the aid of these at least three radar echoes, and as many as possible of the assigned radar echoes are consistent with this Doppler velocity distribution. The distance of the corresponding bounding box is then corrected only based on the determined distances of the radar echoes that are consistent with the Doppler velocity distribution. To determine a consistent set or sets of radar echoes, a robust scheme such as RANSAC (random sample consensus) can preferably be used. Since multiple radar echoes are assigned to the same bounding box, it can be assumed here that all of these radar echoes originate from the same extended object. For example, two radar echoes can then be randomly selected separately. A Doppler velocity distribution is then determined for each of the separately selected radar echoes. This is based on the recognition that the Doppler velocity as a function of the azimuth angle, at least in the theoretically ideal case, i.e., the expected Doppler velocity as a function of the azimuth angle of the object radar echo, can always be described as a cosine function with two degrees of freedom, namely amplitude and phase. Using these two radar echoes, the two degrees of freedom or parameters can then be determined analytically, i.e., the corresponding function can be solved analytically. The resulting cosine function then effectively represents the corresponding Doppler velocity distribution, and for the remaining radar echoes, it is then checked whether they lie on the corresponding cosine curve and thus are consistent with the corresponding Doppler velocity distribution. A predetermined deviation can be allowed here, i.e., if these radar echoes lie within a predetermined range around the corresponding cosine curve, these radar echoes can also be classified as being consistent with the corresponding Doppler velocity distribution. When determining the Doppler velocity of a radar echo to be, for example, 0.5 m / s, such a deviation can be given, for example, by a predetermined expected error. It is then determined how many radar echoes are consistent with the corresponding Doppler velocity distribution, thereby confirming the corresponding object hypothesis. This method can be carried out for each pair of radar echoes, a predetermined number of iterations, or until a predetermined minimum number of consistent radar echoes is reached.

[0028] Then, the set of radar echoes that is itself consistent or consistent with a common Doppler velocity distribution and includes the most radar echoes (best consensus set) can be assigned to the corresponding bounding box. If a set of radar echoes with a consistent Doppler distribution corresponds to at least a predetermined minimum number, this assignment can actually be carried out or be successful here. For example, the minimum number of radar echoes can be three, because in any case, a cosine curve of two radar echoes can always be found that precisely passes through these two radar echoes in the Doppler velocity / azimuth coordinate system. If the data is affected by a certain amount of noise, then, for example, the minimum number can be predetermined as five to obtain a more reliable result. If no group or combination of radar echoes reaches this minimum number of consistent radar echoes, the association of the radar echoes with the bounding box may be discarded or fail. Then, for example, the single-echo scheme can be continued for individual radar echoes, which will be further described below.

[0029] Particularly preferably, when forming a set of consistent radar echoes described here, the minimum and maximum or average scaling errors or scaling factors of these radar echoes can also be considered. Then, only those radar echoes can be included in the set, that is, these radar echoes are consistent with a common Doppler velocity distribution, and their scaling errors can be successfully assigned to the corresponding bounding box in the described manner, and / or at least basically have the same scaling error. For example, then, only those radar echoes whose scaling errors differ from each other by at most a predetermined value, such as at most 10% or at most 5%, can be included in a set of consistent radar echoes.

[0030] Through the measures described here, it can be advantageously and particularly reliably ensured that only those radar echoes that actually originate from the same real object are assigned to the corresponding bounding box or the corresponding object. Ultimately, this can advantageously contribute to particularly reliable object detection and improved and more consistent tracking of the corresponding object.

[0031] To reliably detect only moving objects, that is, particularly relevant objects, here a minimum Doppler velocity can also be specified as the threshold for considering radar echoes. Therefore, the radar echoes can be filtered according to this threshold, so that only those radar echoes are considered, or here for forming a set of consistent radar echoes, or for assignment to the corresponding bounding box, that is, for which a Doppler velocity higher than the predetermined threshold has been determined. As such a threshold, for example, a minimum Doppler velocity of 0.5 m / s can be specified.

[0032] Particularly preferably, a plurality of radar devices or a plurality of radar sensors may be used to detect radar echoes from the surroundings of a motor vehicle. In the case of an object moving linearly, the radar echoes or the corresponding results or data from different radar devices or radar sensors may then be consistent with the same Doppler velocity distribution. As a result, a more precise and reliable assignment can be achieved, and for example, the number of radar echoes in a finally determined set of consistent radar echoes can be increased.

[0033] In an advantageous refinement of the invention, the radar echoes are used to estimate the orientation of the relevant object. The estimated orientation is compared with the orientation of the corresponding bounding box. Then, the assignment is evaluated based on the hereby determined deviation between these orientations and / or the assignment is only carried out or maintained if this deviation is less than a predetermined threshold. Here, the orientation can be determined in each iteration, i.e., for each specific set of radar echoes consistent with the Doppler velocity distribution. Then, the orientation or its deviation from the orientation of the bounding box can be used as an additional criterion as to whether the corresponding set of radar echoes is actually assigned to the bounding box or classified as being consistent with the object. Similarly, for example, the orientation can be determined only for the best set of radar echoes determined at least based on the Doppler velocity distribution. The latter can advantageously save computational effort. If the deviation between the orientations is greater than the specified threshold, the assignment may be discarded or fail. Then, the single echo scheme can be continued for the individual radar echoes, which will be described further below. Similarly, the deviation between the orientations can be used as a basis for evaluating the assignment, for example, by specifying the corresponding association or association probability of the radar echo with the bounding box.

[0034] The orientation can be determined with the aid of the radar echoes, for example, as the regression line of the positions of the radar echoes entered in a coordinate system, possibly taking into account their Doppler velocity or the direction of movement determined thereby. Similarly, another or more complex model for determining the orientation can be specified, which can, for example, take into account the type or classification of the corresponding object determined with the aid of camera data and the form and / or the like specified therefor.

[0035] By proposing herein to consider the orientation as an additional criterion, the reliability of the assignment can be advantageously further increased, and thus ultimately a more reliable and safer operation of the motor vehicle can be achieved.

[0036] In an advantageous refinement of the invention, the orientation of the corresponding bounding box is determined and thus the orientation of the corresponding object enclosed by it. As described, the radar-based orientation and the moving speed of the object in the direction of the radar-determined orientation are estimated with the aid of radar echoes, in particular radar echoes consistent with a common Doppler velocity distribution. The radar-determined orientation and the moving speed can be combined into a radar-determined movement vector of the corresponding object or the corresponding estimated object. The radar-determined orientation is then compared with the orientation of the bounding box. If the deviation between these orientations exceeds a predetermined threshold, it is checked whether the corresponding data or values or results are consistent with a yawing object, i.e., an object rotating about a vertical axis perpendicular to the azimuth plane. A predetermined, in particular speed-dependent, yaw rate and / or a predetermined speed are used here as reasonable references. Particularly preferably, the object represented by the corresponding bounding box can be classified here with the aid of camera data, in particular identified in terms of its type or category, where different reasonable references or yaw rate reference values can be specified for different species, types or categories of objects. This classification of the object can also be used as a basis for estimating its speed or as a basis for a reasonable speed range. For example, a table or database with reasonable value ranges and / or reasonable combinations of values or value ranges can be specified for different categories or types of objects.

[0037] For example, if the vehicle has a non-zero yaw rate, i.e., is turning or, for example, driving through a bend, the Doppler velocity distribution of the relevant radar echoes of the object can also be described as a cosine or cosine function with two degrees of freedom. However, the yaw rate or yaw or turn of the object affects or distorts or offsets the radar-determined orientation and / or the moving speed, i.e., the movement vector of the object. In this case, it is generally not possible to solve the corresponding problem with three degrees of freedom, i.e., speed, orientation and yaw rate, with the aid of radar echo or radar data analysis. Therefore, it is proposed here to estimate the yaw rate based on the orientation of the bounding box and the estimated or determined speed. With the aid of a predetermined reasonable value or one or more predetermined thresholds, it can be checked whether the yaw rate and / or speed or their combination determined in this way is realistic. For example, a reasonable yaw rate for a motor vehicle may be less than 60° per second. In this case, the greater the speed of the vehicle, the lower the reasonable yaw rate. Another possibility is that if the number of radar echoes consistent with the velocity distribution is greater than a predetermined number threshold, the assignment of the radar echoes consistent with the common Doppler velocity distribution to the corresponding bounding box is accepted, i.e., maintained or carried out. As such a number threshold, for example, five can be specified. Thus, if - possibly despite detecting a deviation between the radar-determined orientation and the orientation of the corresponding bounding box - a set of radar echoes consistent with the common Doppler velocity distribution contains at least five radar echoes, the assignment of these radar echoes to the bounding box can be maintained or carried out and possibly marked or labeled as originating from a yawing or turning or orbiting object.

[0038] Particularly preferably, a plurality of radar devices or radar sensors can be used to detect radar echoes from the surroundings of a motor vehicle. In this case, the complete movement state of the respective object, i.e. its speed, azimuth and yaw rate, can be estimated. This is based on the recognition that up to four degrees of freedom or parameters can be determined or estimated using two separate radar devices or radar sensors. The reliability or confidence of the determined movement state can here particularly preferably be determined or estimated as a fourth parameter.

[0039] By the measures described here, it is advantageously possible to reliably achieve target detection even for yawing objects, or an improved flexibility of the method according to the invention can be achieved, which enables the detection of even yawing objects.

[0040] In an advantageous refinement of the invention, a scaling error of the radar echo is determined, which represents the distance of the respective radar echo from the respective bounding box in the radial direction with reference to the current position of the motor vehicle. In particular, this can be the scaling error already mentioned elsewhere. If several consistent moving radar echoes are successfully assigned to a bounding box, the stationary radar echoes detected or located in the vicinity or surroundings of the predetermined space of these radar echoes are analyzed. In the sense of the invention, stationary radar echoes are those radar echoes for which a vanishing Doppler velocity, i.e. a zero Doppler velocity, has been determined - at least up to a predetermined error. Then those stationary radar echoes that are analyzed are also assigned to the respective bounding box, which also agree with the Doppler velocity distribution and whose scaling error differs from the scaling error of the moving radar echoes already assigned to the respective bounding box by at most a predetermined amount. This approach is based on the recognition that such stationary radar echoes can also originate from the same real object as the moving radar echoes. For example, this is the case when an extended object passes at least substantially perpendicularly through the trajectory or direction of movement of the motor vehicle. In this case, for example, moving radar echoes can be received from the front and rear ends of the passing object, and stationary radar echoes can be received from the central region of the object located therebetween. By assigning such stationary radar echoes to the respective bounding box as proposed here, it is advantageously possible to characterize the respective object more completely or in more detail, which can advantageously, for example, enable more reliable tracking and a plausibility check of the data, characteristics or assumptions determined based on the moving radar echoes for the object.

[0041] In another advantageous design of the present invention - particularly for the case where fewer than three moving radar echoes have been assigned to a bounding box, or such an assignment has led to an error, i.e., for example, the assignment has been discarded or failed - a single echo scheme is executed. This single echo scheme is the single echo scheme already mentioned elsewhere and can be employed if the alternative described for the case of successfully assigning at least three radar echoes fails, or if, for example, only one or two radar echoes are acquired. For example, this may be the case for an object that is far from the motor vehicle and thus appears small. Similarly, the single echo scheme can be adopted from the start, which advantageously requires less computing time than a complete or hierarchical method, thereby advantageously saving computing effort. For example, the determination of the Doppler velocity distribution or the application or execution of the RANSAC method can be omitted. Experience has shown that the single echo scheme is particularly used for objects that do not vary or change much in terms of their - determined at different points of the object - Doppler velocity. For example, such excessive changes may occur in cross traffic or when the object orbits with a relatively high yaw rate.

[0042] As part of the single echo scheme, the orientation of the corresponding bounding box is determined, and thus the orientation of the object enclosed by it. Based on this orientation, a reasonable Doppler velocity of the object relative to the current position of the motor vehicle is estimated. For this purpose, for example, a reasonable velocity range can be specified for the object or different orientations of different objects. Such a specification can also depend on the surrounding environment, so that, for example, different velocities may be reasonable in an urban environment compared to, for example, on a highway. Obviously, for a vehicle whose orientation is radially directly towards the motor vehicle, a higher Doppler velocity is reasonable compared to a vehicle whose orientation is towards the motor vehicle's trajectory or direction of movement transversely. Similarly, in certain cases or depending on the corresponding surrounding environment, for example, a Doppler velocity towards or perpendicular to a specific orientation may be unreliable. For example, for a motor vehicle moving in an urban environment, a range from 1 m / s to 30 m / s can be specified as reasonable.

[0043] Furthermore, in the case of the single echo scheme, by comparing the estimated Doppler velocity with the Doppler velocity determined based on the radar for the corresponding radar echo, the probability that the corresponding radar echo originates from the object enclosed by the corresponding bounding box is determined. Then, only or at least the radar echo for which the highest probability has been determined is used as the basis for correcting the distance of the bounding box, i.e., for assigning it to the bounding box or associating it with the corresponding object. In this case, a corresponding probability threshold can also be specified, which must be reached at least for a successful assignment or association. In this way, moving objects that receive fewer than three radar echoes can also be reliably detected.

[0044] Particularly preferably, after the radar echo with the highest probability has been determined, additional radar echoes having Doppler velocities and / or scaling errors measured in a similar manner thereto in a predetermined way can be determined and possibly also assigned to the corresponding bounding boxes. In the single echo scenario, the described gating or the described windowing by means of azimuth angle and / or by means of scaling error can also be used in order to achieve a particularly reliable assignment of the radar echo to the bounding box.

[0045] Particularly preferably, the single echo scenario can be carried out for a plurality of individually, randomly selected radar echoes or for all radar echoes, for example for a predetermined number of radar echoes or until a predetermined result or criterion is reached, including the subsequent assignment of other similar radar echoes. For all runs, then the set of radar echoes having the highest probability and / or the largest number of radar echoes can be adopted, i.e., assigned to the corresponding bounding boxes. Thereby, if necessary, the reliability of the assignment can also be increased, and it is also possible to achieve a simplified or improved tracking of correspondingly smaller and / or correspondingly more distant objects from the motor vehicle.

[0046] In a simplified variant, instead of probability, predetermined discrete values can be used as reference or comparison values for the Doppler velocity and / or the scaling factor, for example. If the comparison of the estimated Doppler velocity with the Doppler velocity determined based on the radar results in a deviation of at most a predetermined amount, for example, the assignment can be made. If the scaling factor is taken into account, the radar echo that satisfies this condition and has the smallest scaling error can be used. In this way, it is possible to advantageously save computational effort if necessary, so that a faster execution of the method according to the invention can be achieved.

[0047] Another possibility is to determine all other detections for each radar echo (similar to RANASC), i.e., the following radar echoes: these radar echoes differ from the corresponding radar echo or its scaling factor or scaling error and its Doppler velocity by less than a predetermined amount or threshold in terms of their scaling factor or scaling error and their Doppler velocity. Here, preferably, all radar echoes can be examined sequentially or, for example, randomly selected, i.e., examined for other radar echoes that are similar in terms of scaling factor and Doppler velocity. The maximum number of other radar echoes, i.e., the so-called inliers, has been determined for the radar echo, and the radar echo is then assigned to the corresponding bounding box together with the other radar echoes determined for the radar echo.

[0048] In another advantageous design of the present invention, objects identified by means of camera data and for which bounding boxes have been generated or have already been generated are classified according to a predetermined category in terms of their class or their type. For example, an object can be classified as a vehicle, a motor vehicle, a car, a truck, a bicycle, a pedestrian, etc. For this purpose, for example, conventional object recognition algorithms based on image processing can be used. A reasonable Doppler velocity direction determined on the basis of radar is assigned to each predetermined category. If, for the type of object enclosed by the respective bounding box, the Doppler velocity direction determined for the respective radar echo differs from the reasonable Doppler velocity direction, the assignment of the radar echo to the bounding box is excluded. In other words, the so-called micro-Doppler effect can be taken into account here. In particular, the spatial distance between radar echoes with different Doppler velocity directions can be considered. Corresponding reasonable distances can also be specified for different categories. For example, for pedestrians, radar echoes with different Doppler velocities or different Doppler velocity directions can usually be received, because, for example, the arms and legs can move in different directions, especially in a direction opposite to the overall movement direction of the pedestrian, or can be stationary at least sometimes despite the movement of the pedestrian. Therefore, for pedestrians, radar echoes with different Doppler velocity directions can be expected and are thus reasonable. A similar effect occurs, for example, on the wheels of a motor vehicle. However, the spatial distance between the respective different radar echoes should at least substantially correspond to the typical or reasonable wheel diameter in order to be able to reasonably assign them to the motor vehicle. Considering or checking the reasonableness of the Doppler velocity direction can advantageously enable the assignment of radar echoes to bounding boxes or objects to be achieved more reliably.

[0049] In another advantageous design of the present invention, objects identified by means of camera data are likewise automatically classified according to their type on the basis of a predetermined category in the manner described. A reasonable Doppler velocity range is assigned to each predetermined category. If the Doppler velocity determined for the respective radar echo lies outside the reasonable Doppler velocity range for the type of object enclosed by the respective bounding box, the assignment of the respective radar echo to the bounding box is excluded. For example, different reasonable velocity ranges can be assigned to pedestrians, cyclists, and motor vehicles. Thus, a radar echo with a Doppler velocity of, for example, 20 m / s from a pedestrian may be implausible.

[0050] In other words, a plausibility check or filtering of the radar echoes suitable for assignment to a specific bounding box can be carried out on the basis of the determined Doppler velocity. This can also advantageously improve the reliability of the assignment of radar echoes to bounding boxes.

[0051] In another advantageous design of the present invention, the corresponding radar cross section, i.e., the so-called RSC value (RSC: Radar Cross Section), is determined at least for moving radar echoes. The objects identified by means of camera data are automatically classified in the described manner based on their class or their type according to a predetermined category. Here, a reasonable radar cross section or a reasonable range of radar cross sections is assigned to each predetermined category. If the radar cross section of the corresponding radar echo is outside the reasonable radar cross section range for the object type surrounded by the corresponding bounding box (i.e., for the category of the corresponding object), the assignment of the radar echo to the bounding box is excluded. For example, a larger RSC value can be assigned to a truck or a motor vehicle. If, by means of camera data, a specific object is classified as a truck or a motor vehicle, etc., i.e., as an object with a relatively large RSC value, then in order to assign the radar echo to the object or its bounding frame, radar echoes with a smaller radar cross section can be discarded or ignored, or used or considered with a correspondingly reduced weight. Similarly, for example, if a pedestrian is recognized by the camera, a correspondingly smaller radar cross section can be expected. Accordingly, when assigning the radar echo to the pedestrian or the corresponding bounding box, those radar echoes whose radar cross section or radar cross section value is greater than the expected or reasonable radar cross section range of the pedestrian can be ignored, discarded, or used or considered with a reduced weight or influencing factor. If such radar echoes are consistent with the object or the remaining radar echoes assigned to the object in terms of their spatial position (i.e., for example, their scaling error), and / or, for example, with the Doppler velocity distribution of these remaining radar echoes, then such ignoring, discarding, or minimal weighting of the radar echoes can also be carried out. For example, radar echoes from a metal post such as a traffic sign or a traffic signal may generate radar echoes with a relatively large radar cross section that are spatially consistent with a pedestrian or their radar echoes located nearby. By taking into account the radar cross section described here, the assignment of radar echoes to objects or bounding boxes can be advantageously further improved, particularly carried out with increased reliability and accuracy.

[0052] Another aspect of the present invention is a motor vehicle having an environmental sensor system, the environmental sensor system having at least one camera for acquiring camera data of the surroundings of the motor vehicle and at least one radar device or at least one radar sensor for emitting radar pulses into the surroundings and for acquiring radar echoes generated by the surroundings. The motor vehicle also has a data processing device connected to the environmental sensor system, wherein the motor vehicle according to the invention is configured to automatically execute at least one variant of the method according to the invention. For this purpose, the data processing device may, for example, have a data memory and a processor device connected thereto. Then a predetermined computer program may be stored in the data memory, which encodes or represents the method steps or sequences or corresponding control instructions and / or process steps of the corresponding method according to the invention. This computer program may then be executed by the processor device in order to cause or initiate an especially automatic execution of the corresponding method. The motor vehicle according to the invention may in particular be the motor vehicle mentioned in connection with the method according to the invention. Correspondingly, the motor vehicle according to the invention may have some or all of the components, devices, assemblies and / or features mentioned in connection with the method according to the invention, or be configured for the processes or measures mentioned there.

[0053] The data processing device may have a communication interface or a data interface through which camera data or radar echoes or radar data including them can be received. The data processing device may also be implemented as a controller for controlling the camera and / or the radar device. The corresponding control signals may then be sent from the data processing device to the camera or the radar device through the communication interface or the data interface. The data processing device or the combination of the data processing device and the environmental sensor system may form a driving assistance system of the motor vehicle.

[0054] Such an assistance system for a motor vehicle may itself be another independent aspect of the present invention.

[0055] Other features of the present invention can be derived from the claims, the drawings and the description of the drawings. The features and combinations of features mentioned above in the description and the features and combinations of features shown below in the description of the drawings and / or separately in the drawings can be used not only in the respectively given combinations, but also in other combinations or alone without departing from the scope of the present invention. Description of the Drawings

[0056] Figure 1 is an exemplary schematic flow chart of a method for detecting objects in the vehicle surroundings;

[0057] Figure 2 is an exemplary schematic flow chart of a first part of the method;

[0058] Figure 3is an exemplary and schematic flow chart of a second part method;

[0059] Figure 4 is a schematic overview for explaining the details of the method;

[0060] Figure 5 is a schematic diagram for explaining the probability function that can be used in the method; and

[0061] Figure 6 is a schematic diagram for explaining the Doppler velocity distribution that can be used in the method.

[0062] In the drawings, identical and functionally identical components are labeled with the same reference numerals. Detailed Description of the Invention

[0063] Today, different types of sensors for detecting the surrounding environment have been used in vehicle technology. However, it can be observed that different sensors have different advantages and disadvantages. So far, it often happens that the data or characteristics of surrounding environment objects are difficult to determine or inaccurately determined by a specific sensor, but the sensor is still used for determination. For example, this can involve pure optical distance determination or radar-based classification or determination of angular position. Therefore, it is advantageous to perform data fusion on sensor data from different sensors as early as possible or at a low level during the data processing process to obtain a combined measurement that determines or indicates all attributes of the corresponding surrounding environment object with good quality, that is, combines the advantages of different sensor types. In addition, it is also helpful to establish a direct relationship between the surrounding environment objects recognized by the camera and the radar detection. Then this combined detection can be used in downstream data processing steps or applications, for example, with a correspondingly greater weight compared to detections based only on sensor data from a single sensor or a single sensor type, for example, in a corresponding digital environment representation such as a dynamic map grid or grid.

[0064] Figure 1 For this purpose, for example, a schematic flow chart 10 of a corresponding method for detecting objects in the vehicle surrounding environment is shown. This method will also be explained below with reference to other drawings.

[0065] This method is performed herein by a motor vehicle 12, which is represented as Figure 4Part of the schematic overview shown therein. To this end, the motor vehicle 12 has an assistance system 14, which in turn includes cameras and radar devices for detecting or imaging the surroundings 16 of the motor vehicle 12. The cameras and radar devices can also be arranged at different positions on the motor vehicle 12, i.e., spatially distributed or spaced apart from each other. However, in any case, the respective positions and orientations of the cameras and radar devices in or relative to the motor vehicle 12 are predefined or known, especially fixed. Therefore, different coordinate systems can be used for the cameras and radar devices or the corresponding sensor data or measurement data obtained thereby, and however, conversions or transformations can be performed between them in a known manner based on specifications.

[0066] In the current case, there is an object 18 at an initially unknown position in the surroundings 16. In addition, there may be other moving and / or stationary objects in the surroundings 16 that are not detailed here. In method step S1, camera data of the surroundings 16 is acquired by the camera. In this or one or more corresponding camera images, the automatically imaged objects are identified and marked. In the current case, for example, the object 18 is identified as a vehicle, and a bounding box 20 is generated for the object 18, which surrounds the object 18 identified in the camera data. In addition, other attributes of the object 18, such as its type or category, size or extent, orientation, etc., can be determined or estimated with the help of the camera data. If necessary, predetermined assumptions or other data that can be obtained, for example, from another assistance system of the motor vehicle 12, can be considered.

[0067] In method step S2, which can be performed at least substantially in parallel, for example, the surroundings 16 are scanned or sampled using the radar device. A large number of radar echoes 22 are received from the surroundings 16. The relevant Doppler velocity V D can be automatically determined for these radar echoes 22, the relevant distance, the relevant azimuth angle Φ, for example, relative to the travel direction 24 of the motor vehicle 12, and / or other data or characteristics. Similarly, the radar echoes 22 or the corresponding radar data - like the camera data acquired in method step S1 - can be equipped with a timestamp that indicates the corresponding acquisition time.

[0068] If there is no camera data with the same timestamp after the radar data is acquired, the available camera data can be interpolated or extrapolated to the time point indicated by the timestamp of the radar data in method step S3. In this case, for example, the bounding box 20 can be moved accordingly.

[0069] In method step S4, if Doppler compensation has not occurred yet, the compensation can be performed. In this case, the Doppler velocity V of the radar echoes 22 Dcan be determined with the aid of radar data and the currently invoked speed of the motor vehicle 12. The Doppler speed V D specifies the speed component of the surrounding object that has generated the corresponding radar echo 22 in the radial direction, i.e., the speed component towards or away from the motor vehicle 12. To illustrate this, one of the radar echoes 22 is additionally labeled as radar echo 26, where a radius line is drawn that connects the position of the radar echo 26 in the radial direction to the motor vehicle 12. The Doppler speed V of the radar echo 26 D is the speed component of the surrounding object that causes the radar echo 26, and this speed component is along the radius line in the global coordinate system. Based on the Doppler speed V D , the radar echoes 22 can then be filtered to select those with vanishing or Doppler speeds V below a predetermined threshold D .

[0070] In method step S5, the radar echoes 22 can be transformed or projected into the coordinate system of the camera data, or the radar echoes 22 and the camera data can be transformed or projected into a common or predetermined coordinate system. This can in particular be the plane in which the motor vehicle 12 is traveling. This is shown in Figure 4 .

[0071] where the azimuth range Φ covered by the bounding box 20 is shown from the perspective of the motor vehicle 12 c,min -Φ c,max . To take into account the uncertainty σ when determining the azimuth Φ of the radar echo 22 Φ , the azimuth range can be expanded by the uncertainty σ on both sides Φ , so that the azimuth range finally considered is Φ c,min +σ Φ to Φ c,max -σ Φ . In addition, for the remaining, i.e., moving, radar echoes 22, a scaling error λ is determined, which indicates the distance of the respective radar echo 22 from the bounding box 20 in the radial direction. Here, the average scaling error λ or, for example, the respective minimum scaling error λ min and the respective maximum scaling error λ max can be determined separately.

[0072] In method step S6, the radar echoes 22 are further filtered based on the azimuth Φ and the scaling error λ. For example, the radar echoes 22 marked by crosses that are outside the extended azimuth range are filtered out, and only those radar echoes 22 that are within the extended azimuth range and whose scaling error falls within a predetermined interval are considered.

[0073] In addition, the respective probabilities of the scaling error λ and the azimuth Φ error can be determined for the radar echoes 22. For thisFigure 5 A coordinate system is schematically shown. For example, the scaling error λ or the error of the azimuth angle Φ can be plotted on the abscissa 28 of this coordinate system, and the corresponding probability values can be plotted on its ordinate 30. An exemplary predefined probability function 32 is plotted therein. According to the probability function 32, a constant probability can be assigned to the corresponding values centered on a predetermined quantity, while a correspondingly smaller probability can be assigned to larger or smaller values.

[0074] If at least three radar echoes 22 are retained after the filtering in method step S6, the method can continue with method step S7. Here, the hypothesis that the remaining radar echoes 22 originate from an extended object can be checked. If this hypothesis is successful, the method can continue in method step S10. If the hypothesis fails, i.e., is not successful, the method can continue in method step S8. To illustrate this, Figure 2 For this purpose, an exemplary schematic first detailed flowchart 34 is shown. In method step S7.1, the radar echoes 22 remaining after the filtering are provided as input data. In method step S7.2, two candidates are randomly selected from among them. In method step S7.3, for these selected candidates, the Doppler velocity distribution and the estimated azimuth of the hypothesized surrounding object are determined, as well as the deviation of this estimated azimuth from the object azimuth 36 determined based on the camera and assigned to the bounding box 20.

[0075] For this purpose, Figure 6 A coordinate system is schematically shown, whose x-axis represents the azimuth angle Φ of the radar echo 22 and whose y-axis represents the Doppler velocity V of the radar echo 22 D . Here, the expected cosine curves 40 are plotted for two candidate echoes 38 selected herein by way of example, which correspond to the Doppler velocity distribution expected for the two candidate echoes 38 according to the hypothesis. A set of radar echoes 22 consistent with this Doppler velocity distribution is determined herein. Two radar echoes 22 marked by cross signs deviate too much from the cosine curve 40 and are therefore classified as inconsistent with the Doppler velocity distribution and are thus not recorded in this set of consistent radar echoes 22.

[0076] In method step S7.4, corresponding comparisons are made based on the scaling error λ and the error of the azimuth angle Φ in order to determine the corresponding sets of radar echoes 22 that are also consistent with these criteria.

[0077] If the corresponding sets of radar echoes 22 have been determined, then in method step S7.5, these sets of radar echoes 22 are compared to select the best set. The best set in this sense can be, for example, the set of radar echoes 22 that best confirms the hypothesis, for example, the set with the highest probability or the largest number of consistent radar echoes 22. As shown here by the corresponding loop paths, method steps S7.2 to S7.5 can be run iteratively multiple times.

[0078] In method step S7.6, based on the selected best group of radar echoes 22, the speed of the underlying surrounding objects and their orientation can be estimated. If the selected group of consistent radar echoes 22 contains enough radar echoes 22 and the orientation and / or speed determined based on them are reasonable, for example compared with the camera data or the bounding box 20, then in method step S7.7, the corresponding hypothesis is classified or evaluated as successful. Otherwise, the hypothesis is classified as unsuccessful or failed, and the method continues with method step S8. There, the hypothesis that the radar echoes 22 originate from a yawed, i.e. rotating object can be tested. In method step S8, for this purpose, under this hypothesis or corresponding predetermined relevant conditions or criteria, the estimated speed and the estimated yaw rate are evaluated or checked for their plausibility. If it is shown here that the corresponding data or values ​​or the result of the assumption that the radar echoes 22 originate from a yawed object are successful, i.e. reasonable, i.e. the estimated speed and the estimated yaw rate are within the reasonable range for the yawed object, then the method also continues in method step S10. Otherwise, the method continues with method step S9. If fewer than three radar echoes 22 remain after filtering in method step S6 , method step S9 is also carried out.

[0079] In method step S9, a single echo scheme is performed. To explain this single echo scheme, Figure 3 A schematic second detailed flow chart 42 is shown by way of example in FIG. 1 . In method step S9.1, a radar echo 22 is provided as input data. In method step S9.2, based on the Doppler velocity V D The probability of the association or assignment of the respective radar echo 22 to the hypothetical object is determined. In method step S9.3, the probability is determined for the respective radar echo 22 based on the scaling error λ. In method step S9.4, the probability is determined for the respective radar echo 22 based on the azimuth angle Φ. If several radar echoes 22 are provided as input data, this can be performed for each or randomly selected radar echo 22. In this case, a corresponding predetermined probability function 32 can be used in each case.

[0080] In method step S9.5, the radar echo 22 with the highest probability is selected. It can then be checked whether this probability meets a predetermined threshold value. If this is not the case, the corresponding hypothesis, i.e. that the corresponding radar echo 22 belongs to the object 18 represented by the bounding box 20, can be discarded, i.e. classified as unsuccessful or failed. In this case, the method can end, or continue with the next bounding box 20, for which the method can then be run again, for example, starting from method step S6. Likewise, the object detection of the object 18 can then be outputted solely based on the camera data, possibly provided with a correspondingly reduced detection probability or weight.

[0081] On the other hand, if the probability of the best radar echo 22 selected in method step S9.5 is greater than the probability threshold, then in method step S9.7, radar echoes 22 similar to the radar echo 22 can be identified according to one or more predetermined criteria, so that in this case, if necessary, a set of radar echoes 22 can also be formed, which can be associated with the bounding box 20 and the object 18 respectively. Whether or not such similar radar echoes 22 can be identified, in method step S9.8, the corresponding hypothesis is classified as successful.

[0082] Then, in method step S10, the radar echo 22 successfully assigned to the bounding box 20 or the object 18 according to method steps S7, S8 or S9 is used to correct the previously only estimated distance of the bounding box 20 from the motor vehicle 12 based on the radar-determined distance thereof. In this case, the bounding box 20 can be moved, for example, by the scaling error λ of the successfully assigned radar echo 22 in the radial direction, in Figure 4 i.e., for example, to the position of the object 18 marked by the dashed line.

[0083] In method step S11, a corresponding measurement or a corresponding object detection or an object data set indicating a successful object detection is generated for the thus detected object 18. The object data set can include the bounding box 20 with the corrected distance, the successfully assigned, i.e., associated, radar echo 22, and sometimes also other variables or data determined during the process of this method.

[0084] In an optionally or application-dependent method step S12, if necessary, based on the object data set or the successful detection of the object 18, then, for example, the object 18 can be tracked or updated, the detection or the object 18 can be input into a dynamic map grid or grid, the motor vehicle 12 can be controlled, etc. In particular, the radar detections specified in the object data set can be marked as actually originating from real surrounding environment objects and can be used for other data processing with a correspondingly high confidence, because it can be assumed that they are not clutter echoes, for example.

[0085] In summary, it is proposed here to determine bounding boxes with orientation and size directly based on camera data in each individual measurement cycle and to assign the raw Doppler detections of the radar to these bounding boxes. For example, such an assignment is not trivial due to clutter echoes and multipath propagation of the radar radiation as well as a possible high object density (e.g., in urban areas or traffic jams). The proposed method first attempts to establish an association with the extended camera objects by combining Doppler detections with consistent Doppler profiles. The association can be evaluated here with the aid of the Doppler curve and the resulting azimuth estimate. In addition, a scaling error λ in the camera-determined distance of the corresponding bounding box can be corrected or rationalized based on the radar-determined distance. As a result, a direct association of the raw Doppler detections (i.e., the corresponding raw radar data) with the camera-identified objects can be obtained, as well as a correspondingly corrected object detection. Here, not only can surrounding objects located in front in the radial direction or moving towards the motor vehicle 12 be detected, but also, for example, with the aid of a plausibility check, surrounding objects entering the detection range of the surroundings 16 or the assistance system 14 from the side, i.e., at least substantially perpendicular to the travel direction 24, can be identified or extracted or detected particularly quickly using the Doppler curve. In this method, the grouping and association of radar detections to surrounding objects can be advantageously used for a particularly robust update (trajectory update) of the tracking of surrounding objects over time.

[0086] In summary, the described example thus shows how a radar-camera association based on Doppler detections and bounding boxes can be realized to enable improved detection of particularly moving objects in the vehicle's surroundings.

[0087] List of reference signs

[0088] 10 Flowchart

[0089] 12 Motor vehicle

[0090] 14 Assistance system

[0091] 16 Surroundings

[0092] 18 Object

[0093] 20 Bounding box

[0094] 22 Radar echo

[0095] 24 Travel direction

[0096] 26 Radar echo

[0097] 28 Abscissa

[0098] 30 Ordinate

[0099] 32 Probability function

[0100] 34 First detailed flowchart

[0101] 36 Object orientation

[0102] 38 Candidate echo

[0103] 40 Cosine curve

[0104] 42 Second detailed flowchart

[0105] S1 - S12 Method steps

[0106] Φ Azimuth angle

[0107] Φ c,min - Φ c,max Azimuth angle range

[0108] λ Scaling error

[0109] λ min Minimum scaling error

[0110] λ max Maximum scaling error

[0111] σ Φ Uncertainty

[0112] V D Doppler velocity

Claims

1. A method (10) for detecting an object (18) moving in the surroundings (16) of a motor vehicle (12) from the motor vehicle (12), wherein, - Obtain camera data of the surrounding environment (16); - Obtain radar echoes (22) from the surrounding environment (16); - Identify an object (18) imaged in the camera data by means of the camera data, wherein for the identified object (18), the azimuth range in which the object is located is determined respectively from the perspective of the motor vehicle (12), and a bounding box (20) surrounding the object (18) is generated, and the bounding box is at a presumed distance from the motor vehicle (12); - For the radar echo (22), a relevant Doppler velocity (V D ) is determined with reference to the current position of the motor vehicle (12), wherein, at least for the moving radar echo (22) thus determined, a relevant distance from the motor vehicle (12) and a relevant azimuth angle (Φ) are determined for which a non-vanishing Doppler velocity has been determined for the moving radar echo; - Perform an association between at least one of the moving radar echoes (22) and at least one of the bounding boxes (20) by comparing the respective azimuth (Φ) with the determined azimuth range and by comparing the respective radar-determined distance with the camera-presumed distance; - For the bounding box (20) to which at least one of the moving radar echoes (22) has been successfully assigned, correct the respective presumed distance according to the radar-determined distance of the respectively assigned radar echo (22); - Output the bounding box (20) with the respective corrected distance as an object data set indicating a successful object detection, characterized in that for the case where at least three radar echoes (22) have been assigned to a bounding box (20), determine a Doppler velocity distribution (40) by means of these radar echoes, as many of the radar echoes (22) as possible are consistent with the Doppler velocity distribution, and correct the distance of the respective bounding box (20) only based on the determined distances of the radar echoes (22) that are consistent with the Doppler velocity distribution (40).

2. The method (10) according to claim 1, characterized in that The method (10) is performed for the radar echoes (22) of each measurement cycle, wherein in the case where no simultaneously acquired camera data is available for the acquisition time of the measurement cycle, the available camera data is interpolated or extrapolated to the acquisition time of the measurement cycle.

3. The method (10) according to claim 1 or 2, characterized in that In order to compare the distances, the radar echo (22) and the bounding box (20) are embedded in a predetermined coordinate system, and a scaling error is determined in the coordinate system at least for the moving radar echo (22), the scaling error indicating the distance between the respective radar echo (22) and the bounding box (20) in the radial direction with reference to the current position of the motor vehicle (12), and the respective scaling error is considered in such a way that for a successful assignment, the scaling error of the respective radar echo (22) must be less than a predetermined threshold.

4. The method (10) according to claim 1 or 2, characterized in that Specify a probability function (32) for the azimuth and / or for the scaling error, for the radar echo (22), the azimuth and / or the scaling error indicating its distance from the respective bounding box (20) in a predetermined coordinate system, and evaluate the respective assignment by means of the probability function.

5. The method (10) according to claim 1, characterized in that - Estimate the azimuth of the relevant object (18) by means of the radar echo (22); - Compare the estimated azimuth with the azimuth (36) of the respective bounding box (20); and - Evaluate the allocation by means of the deviations determined in this case for these orientations, and / or only maintain the allocation if the deviation is less than a predetermined threshold.

6. The method (10) according to claim 1 or 5, characterized in that - Determine the orientation (36) of the respective bounding box (20); - Estimate the radar-based orientation and the moving speed of the object (18) in the direction of the orientation determined based on the radar, by means of the radar echoes (22) that are consistent with the Doppler velocity distribution (40); - When the deviation between these orientations exceeds a predetermined threshold, check whether the corresponding respective data is consistent with a yawed object (18), where a predetermined yaw rate and / or a predetermined speed is used as a reasonable reference.

7. The method (10) according to claim 1 or 5, characterized in that - Determine a scaling error for the radar echo (22), which scaling error indicates the distance of the respective radar echo (22) from the respective bounding box (20) in the radial direction with reference to the current position of the motor vehicle (12); - When a plurality of consistent moving radar echoes (22) are successfully allocated to a bounding box (20), analyze the stationary radar echoes (22) with vanishing Doppler velocity detected in the predetermined space adjacent to these radar echoes (22); and - Allocate those of the analyzed stationary radar echoes (22) to the respective bounding box (20) that are also consistent with the Doppler velocity distribution and whose scaling error differs from the scaling error of the moving radar echoes (22) already allocated to the respective bounding box (20, 22) by at most a predetermined amount.

8. The method (10) according to claim 1 or 2, characterized in that Execute a single echo scenario, where for each radar echo (22) allocated to the respective bounding box (20): - Determine the orientation of the respective bounding box (20); - Based on the orientation, estimate a reasonable Doppler velocity of the object (18) relative to the current position of the motor vehicle (12); - By comparing the estimated Doppler velocity with the Doppler velocity (V D ) determined based on the radar for the respective radar echo (22), determine the probability that the respective radar echo (22) originates from the object (18) surrounded by the respective bounding box (20); and - Use only the radar echo (22) for which the highest probability has been determined as the basis for correcting the distance of the bounding box (20), or at least use the radar echo (22) for which the highest probability has been determined as the basis for correcting the distance of the bounding box (20).

9. The method (10) according to claim 1 or 2, characterized in that Automatically classify the objects (18) identified by means of the camera data according to their type based on a predetermined category, where a reasonable Doppler velocity direction determined based on the radar is assigned to each category, and if the Doppler velocity direction determined for the respective radar echo (22) for the type of the object (18) enclosed by the respective bounding box (20) differs from the reasonable Doppler velocity direction, then exclude the allocation of the radar echo (22) to the bounding box (20).

10. The method (10) according to claim 1 or 2, characterized in that Automatically classify the object (18) identified from the camera data according to its type based on a predetermined category, where a reasonable Doppler velocity range is assigned to each category, and if the Doppler velocity determined for the corresponding radar echo (22) for the type of the object (18) surrounded by the corresponding bounding box (20) is outside the reasonable Doppler velocity range, exclude the assignment of the radar echo (22) to the bounding box (20).

11. The method (10) according to claim 1 or 2, characterized in that - Determine the corresponding radar cross-section for at least the moving radar echo (22); - Automatically classify the object (18) identified from the camera data according to its type based on a predetermined category, where a reasonable radar cross-section range is assigned to each category; and - If the radar cross-section of the corresponding radar echo (22) is outside the reasonable radar cross-section range for the type of the object (18) surrounded by the corresponding bounding box (20), exclude the assignment of the radar echo (22) to the bounding box (20).

12. The method (10) according to claim 4, characterized in that The probability function has a constant central region, and the probability decreases from this central region to both sides.

13. The method (10) according to claim 6, characterized in that The yaw rate is a velocity-dependent yaw rate.

14. The method (10) according to claim 8, characterized in that Execute the single echo scheme if fewer than three moving radar echoes (22) have been assigned to the bounding box (20) or if such an assignment has led to an error.

15. A motor vehicle (12) having an environmental sensor system and a data processing device connected to the environmental sensor system, the environmental sensor system having at least one camera for acquiring camera data of the surroundings (16) of the motor vehicle (12) and at least one radar device for emitting radar pulses into the surroundings (16) and for acquiring radar echoes (22) generated by the surroundings (16), wherein the motor vehicle (12) is designed to automatically perform the method (10) according to any one of claims 1 to 14.

Images