Method for calibrating a radar sensor inside a vehicle

Abstract

A method for calibrating a radar sensor within a vehicle by means of a calibration device (1) comprising a body (3) with a plurality of moving targets (2) arranged in a predefined relationship to one another, the method comprising the calibration device (1) placed at at least one predefined position within the vehicle: Operating the radar sensor to generate a point cloud, the point cloud comprising a plurality of points detected by the radar sensor and associated with at least the at least one predefined position; Operating the calibration device (1) to move at least one of the moving targets (2) such that the moving target is detected by the radar sensor to define dynamic points contained in the point cloud; Monitoring, by means of the radar sensor, at least one predefined position within the vehicle; and Determining a three-dimensional bounding box for the predefined position within the vehicle using the detected points resulting from the movement of the moving targets (2), wherein the detected points are assigned to the three-dimensional bounding box defining a space within the vehicle to be monitored.

Description

[0001] The invention relates to a method for configuring, in particular calibrating, a radar sensor that is placed inside a vehicle and is configured to monitor at least part of the interior of the vehicle, as well as a corresponding device for calibrating a radar sensor inside a vehicle.

[0002] Vehicles, such as cars, are known for their interior space being monitored for various reasons, particularly security. For example, the vehicle's interior can be scanned to detect criminal break-ins and trigger an alarm. In addition, systems capable of monitoring people inside the vehicle, particularly those in their seats, are becoming increasingly interesting, particularly for security reasons. Such seat occupancy detection can also be used (alternatively or additionally) to deactivate / activate an airbag for a specific seat or to inform the driver of the presence of small objects in the rear seats, such as a baby or pet.

[0003] Such functionality is typically provided by capacitive sensors built into the seats, such as sensor mats (in the seating area and / or the seat back), which can detect the absence or presence of an object, especially a (living) person. However, this requires that each seat incorporate an additional functional electronic component.

[0004] DE 10 2007 041 456 A1 discloses a device for simulating a moving radar target, preventing any interfering reflection of radar waves by objects located near the device and / or the radar. In the device, a tube is mounted in front of a recess, so that the field width of the radar waves emerging from the recess and reflected by the rotor is limited by the tube.

[0005] DE 10 2017 214 009 A1 discloses a method for detecting the presence and / or movement of a vehicle occupant. For this purpose, a signal is emitted in the interior of a vehicle, and a reflected signal is detected and evaluated. At least two operating modes are provided for evaluating the reflected signal. In a first operating mode, the reflected signal is used to determine whether a vehicle seat is occupied by a vehicle occupant. In a second operating mode, the reflected signal is used to detect a body part of a vehicle occupant, and a gesture is determined from the movement of the body part.

[0006] Alternatively, it is known to detect the presence of an object placed on a seat in the interior of a vehicle by means of other sensors that are not integrated into the seats but located elsewhere within the car, which are capable of monitoring at least a desired part of the vehicle interior. For example, one or more radar sensors can be used, which can be placed inside a vehicle and which can be hidden, for example, in the roof lining, the interior mirror, or behind the sun visors. A single radar sensor may be sufficient, but it is also possible to place more than one sensor. However, due to the position of the sensor independent of the seat, it may be necessary to perform a calibration, at least initially, as one of the following causes may occur.

[0007] An object of the present invention is to provide an improvement in the configuration, in particular a calibration of a radar sensor placed inside the vehicle to monitor the vehicle interior, in particular to detect the occupancy of seats.

[0008] This object is achieved according to the teaching of the independent claims. Various embodiments and further embodiments of the invention are the subject of the dependent claims.

[0009] A first aspect of the invention relates to a method for calibrating a radar sensor within a vehicle. The calibration device comprises a body with a plurality of moving targets arranged in a predefined relationship to one another. The relative relationship can also be referred to as a predetermined pattern of the moving targets. Once the calibration device is placed at at least one predefined position within the vehicle, the radar sensor is operated to generate a point cloud, wherein the point cloud comprises a plurality of points detected by the radar sensor and assigned to at least the at least one predefined position. The calibration device is operated to move at least one of the moving targets such that the moving target is detected by the radar sensor, in order to define dynamic points contained in the point cloud.The movement of the moving targets makes the target visible to the radar sensor, since—as one skilled in the art will recognize—radar sensors are more sensitive to moving targets, while fixed targets are less, barely, or not at all visible to the radar sensor. The radar sensor monitors at least one predefined position within the vehicle where the calibration device is placed. A three-dimensional bounding box is then calculated for the predefined position within the vehicle using the detected points resulting from the movement of the moving targets. The three-dimensional bounding box defines a space within the vehicle to be monitored.

[0010] According to the invention, a three-dimensional bounding box is defined for a seat to be monitored within the vehicle. This can reduce computational effort, as only points of the point cloud within the box are considered. Points detected by the radar sensor that lie outside the box (or boxes) are not relevant for determining whether a seat is occupied or not.

[0011] A "bounding box" in the sense of the present invention is a space in the interior of a vehicle in which a large part of a person sitting on a specific seat of the vehicle is located. The bounding box does not have to be exactly the same size as the respective person. The frame can be larger or smaller than the person; for example, it can only cover the space above a seating area of ​​a seat, which means that the legs of a person sitting on a specific seat of a vehicle may not be within the frame. It is understood that it is not necessary to also capture a person's legs if, for example, their upper body is within the bounding box. The term bounding box can be replaced by the term "frame," "volume," or the like. It is a three-dimensional section of the interior of the vehicle and can have a rectangular shape.

[0012] The term "inside," as used throughout this disclosure with reference to a vehicle, is to be understood as a sensor, such as a radar sensor, intended to be disposed "inside" the vehicle and configured to monitor at least a portion of the vehicle's interior. In contrast to a sensor "inside" the vehicle as defined in this disclosure, various other sensors, such as radar sensors, lidar sensors, cameras, etc., may be disposed in, on, or generally onboard a vehicle, but configured to sense the exterior, i.e., the surroundings of the vehicle.

[0013] The calibration device can also be referred to as a "marker." It is a mobile device that can be placed by a user inside the vehicle. At least the moving targets are visible or detectable by the radar sensor. Information obtained by placing the calibration device at known and predetermined positions within the vehicle and detecting these positions by the radar sensor can therefore be used for calibration.

[0014] A "point cloud" within the meaning of the invention is understood to be a set of points in a vector space (unless limited to specific dimensions in the following embodiments) of any dimension M > 1, which may, in particular, have an organized or unorganized spatial structure. A point cloud is described by the points it contains, each of which can be recorded, in particular, by their positions specified by spatial coordinates. In addition to the points, attributes such as geometric normals, color values, temperature values, recording times, measurement accuracies, or other information can be recorded.

[0015] The terms "comprises," "includes," "contains," "includes," "has," "with," or any other variation thereof, as may be used herein, are intended to cover non-exclusive inclusion. For example, a method or device that includes or has a list of elements is not necessarily limited to those elements, but may include other elements not specifically listed or inherent in such a method or device.

[0016] Unless explicitly stated otherwise, "or" refers to an inclusive "or" and not an exclusive "or." For example, a condition A or B is satisfied by one of the following: A is true (or present) and B is false (or absent); A is false (or absent) and B is true (or present); and both A and B are true (or present).

[0017] The terms "a" or "an" as used herein are defined to mean "one or more." The terms "other" and "different" and any other variations thereof are defined to mean "at least one more."

[0018] The term “plurality” as used here should be understood to mean “two or more”.

[0019] The term "configured" or "set up" to perform a specific function (and respective variations thereof) is to be understood within the meaning of the invention that the corresponding device is already present in a configuration or setting in which it can perform the function, or is at least adaptable - i.e. configurable - so that it can perform the function after appropriate adaptation. In this context, the configuration can be carried out, for example, by appropriately setting parameters of a process sequence or of switches or the like for activating or deactivating functionalities or settings. In particular, the device can have a plurality of predetermined configurations or operating modes, such that a configuration can be carried out by selecting one of these configurations or operating modes.

[0020] In the following, preferred embodiments of the method are described, which, unless expressly excluded or technically not possible, can be combined with each other and with the other aspects of the invention described below.

[0021] In some embodiments, the dynamic points are extracted from the point cloud to determine a position of the calibration device within the vehicle associated with the respective three-dimensional bounding box. During the calibration process, the moving target should be the only element within the vehicle's interior that is moving and therefore most visible or detectable by the radar sensor. All other elements within the vehicle do not move during the calibration process and are thus essentially invisible to the radar sensor.

[0022] In some embodiments, the calibration device is operated such that only one of the moving targets is moved at a time. In this way, the position of each of the moving targets can be determined independently, compared to an operating mode in which more than one or all of the moving targets would move. Moving only one of the moving targets at a time can therefore improve the accuracy of the calibration device's detection.

[0023] In some embodiments, the operation of the calibration device is repeated, placing the calibration device at a different location within the vehicle. By detecting more than one predetermined location within the vehicle, calibration can be improved. The information can be combined to calculate the positions and orientations of the bounding boxes associated with the vehicle's seats.

[0024] In some embodiments, the points contained in the point cloud are classified as static points if a Doppler value is zero or essentially zero, or at least below a certain threshold, and as dynamic points if the Doppler value is greater than zero or greater than a certain threshold. Assuming that the interior of the vehicle is not moving, all detected points in the point cloud that have a very low Doppler value, in particular essentially zero, are likely to refer to fixed points in the interior of the vehicle. In contrast, if a seat is occupied by a person, it is assumed that the person is moving at least slightly, resulting in a Doppler value greater than zero or greater than a certain threshold.To eliminate outliers, only dynamic points with a signal-to-noise ratio (SNR) greater than a predefined threshold are used to determine the bounding boxes.

[0025] In some embodiments, the calibration device comprises three moving targets, wherein the moving targets are moved in a predetermined order. Detecting three points related to the three moving targets easily allows for the calculation of a center of gravity of the corresponding triangle, which in turn allows for the calculation of a center of the corresponding bounding box. It is understood that more than three, such as four, five, six, or more moving targets, may be provided.

[0026] In some embodiments, the calibration device can be placed at at least three different positions within the vehicle. These positions can relate to three rear seats of the vehicle. By combining the information obtained by each determination, in particular taking into account the three-center-of-gravity triangle, a dimension can be calculated for each of the positions. It is understood that more than three, such as four, five, six, or more positions within the vehicle can be determined by placing the calibration device at the respective positions, in particular by placing the calibration device on the respective seat.

[0027] For example, any seat of the vehicle can be used as a position for placing the calibration device during the method of the present invention. In some embodiments, the calibration device is placed in the rear seats of the vehicle. The device can additionally be placed at a predetermined distance above a center rear seat of the vehicle to determine another dimension of the positions to be determined, particularly the centers of the bounding boxes of interest.

[0028] In some embodiments, it may be provided that a particular bounding box is larger than would be required for the current seat position. This can cover "dynamic" seat positions, e.g., when a seat position or its orientation can change. For example, it is well known that the front seats of a car in particular can be moved forward and backward, and the recline angle of the seatback can be changed based on a user's needs or preferences. The bounding box can be made sufficiently large to cover all possible seat positions. This makes it possible to determine whether a seat is occupied or not, even if a user changes the seat position.

[0029] A second aspect of the invention relates to a device for calibrating a radar sensor within a vehicle, which can be used in particular in the method according to the first aspect of the invention and the preferred embodiments of the method set out above. Reference is made to the method described above for how and where to place the calibration device within a vehicle to calibrate the radar sensor. The calibration device comprises a body with a plurality of movable targets arranged in a predefined relationship to one another, wherein the calibration device comprises a control unit configured to move at least one of the movable targets. The movable targets are associated with a three-dimensional bounding box defining a space to be monitored within the vehicle.

[0030] In some embodiments, the control unit is configured to control the movement of the at least one movable target such that only one of the movable targets is moved at a time.

[0031] In some embodiments, at least a portion of a surface of the moving targets is configured to reflect a signal transmitted by the radar sensor. This improves the detection of the moving targets by the radar sensor. For example, the moving targets may comprise a plastic material, while at least a portion of the surface comprises a metallic material or is coated with a metallic material.

[0032] In some embodiments, the moving targets are configured as rotating fans. The rotation can be easily detected by a radar sensor, especially if the fan blades are tilted. Of course, any other suitable configuration for the moving targets could be considered.

[0033] As also mentioned above, in some embodiments, the device comprises three movable targets. The predefined relationship between the movable targets may be an equilateral triangle. This facilitates the calculation of a center of gravity of the movable targets of the calibration device. In some embodiments, the calibration device comprises a plate-shaped body sized and dimensioned to be placed on a seat of a vehicle. Other shapes and sizes are possible, as long as the calibration device can be placed at the positions within the vehicle to be used for calibration.

[0034] A third aspect of the invention may relate to a data processing system comprising at least one processor configured to perform the method according to the first aspect of the invention, in particular using the device according to the second aspect of the invention. The system may, in particular, comprise one or more radar sensors within the vehicle, i.e., one or more radar sensors configured to monitor at least part of the vehicle's interior.

[0035] In particular, the system may be a computer or a controller for another or higher-level system, for example for a vehicle or for a production machine or line.

[0036] A further aspect of the invention relates to a computer program comprising instructions which, when executed on a system according to the second aspect, cause the system to carry out the method according to the first aspect.

[0037] In particular, the computer program can be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular a server, and can be downloadable via a data connection, such as the Internet or a dedicated data connection, such as a proprietary or local network. Furthermore, the computer program can comprise a plurality of interacting individual program modules.

[0038] Accordingly, the system according to the third aspect may comprise a program memory in which the computer program is stored. Alternatively, the system may also be configured to access an externally available computer program, for example, one or more servers or other data processing units, via a communication connection, in particular to exchange data therewith that is used during the execution of the method or computer program or represents outputs of the computer program.

[0039] The features and advantages explained with reference to the first aspect of the invention apply accordingly to the further aspects of the invention.

[0040] Those skilled in the art will recognize that additional sensors (e.g., capacitive sensors) may be provided in the seats for various purposes. For example, additional sensors, although not necessary for the present invention, may assist the functionality of the present invention with respect to seat occupancy detection. That said, the invention is not limited to calibrating only one radar sensor within the vehicle. There may be more than one radar sensor, such as two, e.g., one for the rear row of seats and one for the front row of seats, or even more, such as one sensor per seat.

[0041] Further advantages, features and possible applications of the present invention will become apparent from the following detailed description in conjunction with the figures. Fig. 1 shows a calibration device according to the invention. Fig. Figure 2 shows a flowchart of an algorithm for finding bounding boxes. Fig. 3 shows the following image shows the extracted triangle points and their centroid of the triangle marker placed at the location indicated above.

[0042] Fig. Figure 1 shows a calibration device 1 according to the present invention. In this embodiment, the device 1 is designed as a triangular marker reflector. It has three moving targets 2 arranged on a plate-shaped body 3. A motor (not shown) drives the moving targets 2, which are in the form of rotating fans or propellers. The propellers can be 3D-printed and made of plastic, which in this embodiment is covered with aluminum foil so that it becomes visible to the radar sensor (not shown). The resulting triangle from these three propellers is, in this case, an equilateral triangle. The edge length can be, for example, 10 to 30 cm, e.g., 20 cm. An LED light 5 can be provided at each corner of the triangle to indicate whether the corresponding motor is running or not. This optional feature allows a user to better control the calibration process.The three moving targets 2 of the triangle are controlled by a control unit 4, e.g., programmed with a Raspberry Pi. The control unit 4 is programmed so that only one motor of the moving targets 2 is running at any given time. A switch 6 can be attached to the control unit 4 to assist in selecting the appropriate motor.

[0043] The radar sensor (not shown) to be calibrated by the method according to the invention can be placed in the upper part of the vehicle, such as the roof lining, to monitor each seat. The radar sensor outputs a point cloud (x, y, and z coordinates), and each point in the point cloud is assigned a corresponding signal-to-noise ratio and an associated Doppler. There are two types of points in the point cloud: static and dynamic points. Static points are the points where the Doppler value is zero (or essentially zero), and dynamic points are points with a non-zero Doppler value. The method of the present invention uses only dynamic points with a signal-to-noise ratio higher than a predefined threshold.Since the propellers of the moving targets 2 are rotated, they are recognized as dynamic points and thus form the relevant points used for further calculation, in particular for determining the bounding boxes.

[0044] With reference to Fig. 2, an exemplary method of the invention will now be described. In particular, it will be described how the aforementioned calibration device can be used to calibrate one or more radar sensors within a vehicle. More specifically, the method aims to define bounding boxes that can then be used to determine whether a seat of a vehicle is occupied or not. This information can then be further processed, e.g., to issue a seatbelt warning, etc. The use of such bounding boxes, which describe a volume within which a person is likely to be located when sitting in a corresponding seat. In other words, when a person is sitting in a seat, most of their body is within the frame.This eliminates the need to monitor the entire interior of the vehicle during operation, as well as interference from outside the vehicle that can be detected by the radar sensor, i.e., interference that may be contained in the point cloud generated by the radar signal. All points outside the bounding boxes can be ignored or omitted, which significantly reduces the need for computing resources. Furthermore, the method according to the invention only takes moving objects or persons into account, i.e., only dynamic points within the boxes are evaluated to determine whether a seat is occupied or not. The method according to the invention relates to the calibration process, which may only be necessary once after vehicle manufacture.It may even be sufficient to use the same calibration data for a specific car model because the seats are always in the same position.

[0045] The radar sensor detects a sparse and unclassified point cloud (step 201). For calibration, i.e., to determine the position and orientation of the bounding boxes assigned to the car seats, particularly relative to the radar sensor inside the vehicle, the calibration device in this embodiment is placed at four different positions, namely on each of the three rear seats and at a predetermined distance, such as 50 cm above the center rear seat. For each position, the moving targets 2 are moved, resulting in four triangles that are determined by extracting the resulting dynamic points (step 202). To eliminate outliers, not only the Doppler value is considered, but also the signal-to-noise ratio (see steps 203 and 204). Once the relevant points in the point cloud have been found, these points are combined at step 205, which is necessary to calculate relative positions.For each detected triangle, the center of gravity is determined in step 206. This is shown in . Fig. 3, which shows exemplary points 21, 22, and 23 for the three moving targets, with the calibration device placed above the center rear seat, and the calculated centroids 11, 12, 13, and 14 for each of the resulting triangles from the different positions where the calibration device was placed. Then, in step 207, a rotation matrix is ​​constructed representing the orientation of each bounding box. The positions are determined at step 208, taking into account all measured and relevant dynamic points, more specifically, the calculated centroids.

[0046] The orientation of the frames is determined based on the measured data points, specifically the determined centers of gravity (CG) of the four triangles. The center of gravity for the lower three triangles in the three seats is used to find the x-axis vector. The difference between the lower left, middle, and right CG points is used to find the unit vector in the x-direction. Fig. Figure 3 shows the triangle corners of the center seat and also the CoG of the left, center, and right frames, indicated by a black dot. The crosses indicate the triangle corners corresponding to the upper, right, and left propellers of the pattern (see, for example, reference numbers 21, 22, and 23 for the position above the center rear seat). To find the y-axis vector, the difference between the upper and lower CoG of the center seat is considered. Fig.Figure 3 shows the triangle vertices for the middle seat and the CoG for the upper and lower patterns. The z-axis vector is calculated by taking the cross product of the x-axis and y-axis vectors.

[0047] After finding the 3-axis vectors, the rotation matrix is ​​created as follows. The rotation matrix is ​​denoted by Rt, where (X1, X2, X3) define the x-axis direction vector, (Y1, X2, X3) define the y-axis direction vector, and (Z1, Z2, Z3) define the z-axis direction vector. The resulting rotation matrix Rt can be denoted by: Rt=[X1Y1Z1X2Y2Z2X3Y3Z3]

[0048] The rotation angles are calculated using the rotation matrix Rt from the previous step. Finally, the frames can be set up using the calculated CoG and the rotation angles estimated from the previous step.

[0049] In view of the foregoing, according to the present invention, a method and system can be provided that assist in externally calibrating a radar sensor by using a marker (in the form of the calibration device) placed at a known location (inside the vehicle interior). This makes it possible to estimate the position and the three rotation angles of the 3D frame (i.e., six degrees of freedom) embodied by the triangular marker.

[0050] Those skilled in the art will understand that there are two main uses for the invention: extrinsic radar sensor calibration and object calibration. For extrinsic radar sensor calibration, the sensor position is unknown, and the sensor position is estimated using markers (of the calibration device described herein) at known locations. Conversely, in object calibration, the sensor position is known, but objects such as detection frames (position and orientation) must be calibrated. This means that the invention can be used in different variants for a wide variety of applications. Such applications include not only the configuration explained above, in particular calibration of a radar sensor in the vehicle interior.

[0051] Although at least one embodiment has been described above, it should be noted that a large number of variations thereof exist. It should also be noted that the described embodiments are only non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description provides those skilled in the art with guidance for implementing at least one embodiment, it being understood that various changes in the operation and arrangement of the elements described in an embodiment may be made without departing from the subject matter set forth in each of the appended claims and their legal equivalents.

Claims

[1] A method for calibrating a radar sensor within a vehicle by means of a calibration device (1) comprising a body (3) with a plurality of moving targets (2) arranged in a predefined relationship to one another, the method comprising placing the calibration device (1) at at least one predefined position within the vehicle, comprising: Operating the radar sensor to generate a point cloud, the point cloud comprising a plurality of points detected by the radar sensor and associated with at least the at least one predefined position; Operating the calibration device (1) to move at least one of the moving targets (2) such that the moving target is detected by the radar sensor to define dynamic points contained in the point cloud; Monitoring, by means of the radar sensor, at least one predefined position within the vehicle; and Determining a three-dimensional bounding box for the predefined position within the vehicle using the detected points resulting from the movement of the moving targets (2), wherein the detected points are assigned to the three-dimensional bounding box defining a space within the vehicle to be monitored. [2] The method of claim 1, further comprising extracting the dynamic points from the point cloud to determine a position of the calibration device (1) within the vehicle associated with the corresponding three-dimensional bounding box. [3] A method according to claim 1 or 2, wherein the calibration device is operated such that only one of the movable targets is moved at a time. [4] A method according to any one of the preceding claims, wherein the operation of the calibration device is repeated, the calibration device being placed at a different position within the vehicle. [5] A method according to any one of the preceding claims, wherein the points contained in the point cloud are classified as static points if a Doppler value is zero and as dynamic points if the Doppler value is greater than zero. [6] The method of claim 5, wherein only dynamic points with a signal-to-noise ratio higher than a predefined threshold are used to determine the bounding box. [7] A method according to any preceding claim, wherein the calibration device comprises three movable targets, the movable targets being moved in a predetermined order. [8] Method according to one of the preceding claims, wherein the calibration device is placed at at least three different positions within the vehicle. [9] The method of claim 8, wherein the calibration device is placed on the rear seats of the vehicle and at a predetermined distance above a center rear seat of the vehicle. [10] Calibration device for calibrating a radar sensor within a vehicle, wherein the radar sensor is configured to monitor at least one predefined position within the vehicle, wherein the calibration device comprises a body with a plurality of moving targets arranged in a predefined relationship to one another, wherein the calibration device comprises a control unit configured to move at least one of the moving targets. [11] The apparatus of claim 10, wherein the control unit is configured to control the movement of the at least one movable target such that only one of the movable targets is moved at a time. [12] The apparatus of claim 10 or 11, wherein at least a portion of a surface of the moving targets is configured to reflect a signal transmitted by the radar sensor. [13] Device according to one of claims 10 to 12, wherein the movable targets are designed as rotatable fans. [14] The apparatus of any one of claims 10 to 13, wherein the apparatus comprises three moving targets and wherein the predefined relationship between the moving targets is an equilateral triangle. [15] Device according to one of claims 10 to 14, wherein the calibration device comprises a plate-shaped body which is dimensioned and sized to be placed on a seat of a vehicle.

Images