Method and system for detecting a seat occupancy state of a seating arrangement on the basis of radar point clouds

Abstract

The invention relates to a method for automated detection of a seat occupancy state, in particular related to a seat, of a seating arrangement having at least one seat, the method comprising: receiving or generating measurement data representing, for each measuring frame of a sequence of a plurality of temporally successive measuring frames, an associated radar point cloud, so that the measurement data represent a sequence of radar point clouds corresponding to the sequence of measuring frames, wherein each radar point cloud in the sequence is or has been obtained on the basis of a radar scan, taking place at or during a measurement time or measurement period associated with the relevant measuring frame, of a spatial region at least partly surrounding the seating arrangement; accumulating a plurality of radar point clouds in the sequence in order to obtain an accumulated radar point cloud containing radar points from each of the individual radar point clouds combined as part of the accumulating process; determining a seat occupancy state of the seating arrangement on the basis of an evaluation model which provides, on the basis of the accumulated radar point cloud, one of a number of predefined possible seat occupancy states of the seating arrangement as an evaluation result; and outputting a piece of information defined on the basis of the evaluation result.

Description

[0001] METHOD AND SYSTEM FOR DETECTING A

[0002] OCCUPANCY STATE OF A SEATING ARRANGEMENT BASED ON RADAR DOT CLOUDS

[0003] The present invention relates to a method, a computer program and a system configured to carry out the method, each for the automated detection of a seat occupancy state of a seating arrangement with at least one seat.

[0004] In various situations, it may be necessary to automatically determine the current seat occupancy status of a seating arrangement with at least one seat. Such a situation can occur particularly in vehicles, for example, in motor vehicles where a vehicle configuration or the activation, deactivation, and / or control of one or more vehicle functions is to take place depending on a current seat occupancy status. For example, it is known in motor vehicles to issue an acoustic or visual indication to vehicle occupants to fasten seat belts or to control the activation or deactivation of airbags depending on a detected seat occupancy status.

[0005] For the automated detection of the current occupancy status of one or more seats, in particular of an arrangement of vehicle seats in a vehicle, so-called seat occupancy mats are known. These mats are integrated into the seats (usually one per seat) and use pressure-sensitive sensors to detect occupancy of the respective seat. The seat occupancy status is determined based on the sensor signals or data from these sensors, usually by means of a threshold test.

[0006] These known solutions therefore require the seats to be equipped with built-in sensors and are usually unable to distinguish between seat occupancy states beyond "occupied" and "unoccupied." Furthermore, the sensors typically have to be integrated into the seats at the factory, making retrofitting difficult or impossible, and eliminating the ability to detect seat occupancy for seats not equipped with such sensors.

[0007] It is an object of the present invention to provide an improved solution for the automated detection of a seat occupancy state of a seating arrangement with at least one seat. This object is achieved according to the teaching of the independent claims. Various embodiments and further developments of the invention are the subject of the dependent claims.

[0008] A first aspect of the solution presented here relates to a method, in particular a computer-implemented method, for the automated detection of a seat occupancy state, in particular a seat-related one, of a seating arrangement with at least one seat (or equivalently: seat), in particular a seat in or for a vehicle, such as an automobile (e.g. truck, car or bus). The method comprises: (i) receiving or generating measurement data which represent an assigned radar point cloud for each measurement frame of a sequence of several temporally successive measurement frames, such that the measurement data represent a sequence of radar point clouds corresponding to the sequence of measurement frames. Each radar point cloud of the sequence is or wasobtained during a radar scan of a spatial area surrounding the seating arrangement at least in part, carried out at a measurement time or measurement period assigned to the respective measurement frame; (ii) accumulating several of the radar point clouds of the sequence in order to obtain an accumulated radar point cloud containing radar points, in particular all radar points, from each of the individual radar point clouds combined during the accumulation; (iii) determining, in particular estimating, a seat occupancy state of the seating arrangement using an evaluation model which, depending on the accumulated radar point cloud, provides one of several predefined possible seat occupancy states of the seating arrangement as an evaluation result; and (iv) outputting information defined depending on the evaluation result.

[0009] The term "seat occupancy state" of a seating arrangement with at least one seat, as used herein, is to be understood in particular as information indicating whether or to what extent the seating arrangement or at least one of its seats is occupied by an object, in particular a thing or a person. In a simple example, the seat occupancy state can only indicate the presence or absence of an object, or in a more advanced example, in the case of the presence of at least one object on the seating arrangement or one or more of its seats, it can provide information about the type or other property, such as a spatial extent, of the object.

[0010] The term "radar point cloud," as used herein, refers in particular to a set of points in a vector space obtained by radar scanning of at least one object surface, which has a typically unorganized spatial structure ("cloud"). In the case of a radar point cloud, the points of the radar point cloud can be referred to as "radar points." A (radar) point cloud can be described in particular by the (radar) points it contains. The radar points, in turn, can each be described in particular by their spatial coordinates, which indicate, for each radar point, a location of the reflection of an emitted radar signal on an object surface, measured during the radar scanning. In addition to the radar points, attributes such as measured Doppler velocity or a signal-to-noise ratio (SNR) can also be recorded.

[0011] The term "accumulated radar point cloud," as used herein, refers to a radar point cloud resulting from the accumulation of a plurality of individual radar point clouds—in particular, in the mathematical sense of a union—such that the accumulated radar point cloud contains radar points, in particular all radar points, from each of the individual radar point clouds combined during the accumulation. An accumulated radar point cloud can, in particular, contain more radar points than each of the individual radar point clouds included in the accumulation. Accordingly, the term "accumulate" refers to such a combination of several individual radar point clouds into an accumulated radar point cloud.

[0012] The term "measurement frame," as used herein, refers in particular to a measurement time or measurement period in a temporal sequence of defined, in particular periodically successive, measurement times or measurement periods, at or during which a measurement, in this case a radar scan of at least one object surface, takes place or has taken place. Each measurement frame is assigned the result of its associated measurement, in this case the radar point cloud generated for the measurement frame by radar scanning.

[0013] The term “evaluation model,” as used herein, refers to a model, particularly a mathematical one, that uses a radar point cloud, particularly an accumulated one, or one or more parameters characterizing it as input variables in order to provide an evaluation result depending thereon, in this case one of several predefined possible seat occupancy states of the seating arrangement. The evaluation model can in particular be a mathematical estimation function, wherein the accumulated radar point cloud represents empirical data as a sample and the evaluation result represents an estimated value determined depending thereon. The evaluation model can in particular be a “machine learning model” (orA "machine learning model" can be a model (also known as a "machine learning model"), which here is understood in particular to mean a mathematical, in particular statistical, model for making predictions or decisions created using at least one machine learning algorithm on the basis of sample data referred to as training data, without the algorithm(s) being explicitly programmed to make such predictions or decisions. In particular, decision tree-based machine learning models ("decision trees") are machine learning models.

[0014] The information to be output can, in particular, represent the evaluation result itself. It can also be a detectable signal, particularly one that can be perceived by the human senses, such as a warning, a control signal for controlling a signal source, or a data signal carrying the information.

[0015] The terms "comprises," "includes," "includes," "has," "has," "with," or any other variation thereof, as used herein, are intended to cover non-exclusive inclusion. For example, a method or apparatus that includes or has a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or that are inherent in such a method or apparatus.

[0016] Furthermore, unless explicitly stated otherwise, "or" refers to an inclusive "or" and not an exclusive "or." For example, a condition A or B is satisfied by one of the following conditions: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).

[0017] The terms "a" or "an" as used herein are defined as "one or more." The terms "another" and "another," and any other variations thereof, are defined as "at least one other."

[0018] The term "plurality", as used here, is to be understood in the sense of "two or more". The terms "configured" or "set up" to fulfil a specific function (and respective variations thereof), as used herein, are to be understood, within the meaning of the invention, as meaning that the corresponding device is already in a configuration or setting in which it can carry out the function, or that it is at least adjustable - i.e. configurable - so that it can carry out the function after being set accordingly. The configuration can be carried out, for example, by setting parameters of a process sequence or switches or the like accordingly to activate or deactivate functionalities or settings. In particular, the device can have a plurality of predetermined configurations or operating modes, so that the configuration can be carried out by selecting one of these configurations oroperating modes.

[0019] Using the method according to the first aspect, an evaluation result characterizing a seat occupancy state of the seating arrangement (particularly in the sense of a prediction or classification) can be obtained based on a chronological sequence of radar point clouds obtained from an actual or simulated radar scan of a spatial area surrounding the seating arrangement. Thus, radar-based solutions can be implemented, particularly in the vehicle context (particularly for automobiles), that can reliably detect a seat occupancy state (particularly exclusively) based on radar and, based on this, can activate, deactivate, or control / regulate certain functionalities or systems, such as a seat belt warning system or an airbag system, entirely or selectively.

[0020] The number of radar points generated per radar point cloud depends on whether or how an object is moving in many radar measurement methods. The use of at least one accumulated radar point cloud, condensed by accumulating several consecutive radar point clouds, serves in particular to increase the number of radar points on the basis of which the evaluation to determine the seat occupancy status is carried out. This can improve the quality of the evaluation and, in particular, reduce its error rate. This is particularly important if the individual radar point clouds acquired for the individual measurement frames each contain only a small number of radar points (a "sparse point cloud"), so that without accumulation, the evaluation could become inaccurate or error-prone due to an insufficient data basis. This can be particularly the case with stationary or only slightly moving objects, e.g.when there is a stationary thing, a barely moving person or a very small object on a seat.

[0021] In the following, various exemplary embodiments of the method are described, each of which, unless expressly excluded or technically impossible, can be combined with each other as well as with the other aspects of the present solution described.

[0022] In some embodiments, the accumulation of several of the radar point clouds of the sequence occurs multiple times in order to obtain an accumulated radar point cloud in each case, which contains radar points, in particular all radar points, from each of the individual radar point clouds combined during the respective accumulation, wherein with each accumulation, only radar point clouds of an individual subset of the sequence assigned to it are accumulated, so that at least two of the subsets are different. The seat occupancy state of the seating arrangement is determined here as a function of at least two of the accumulated radar point clouds. In this way, in particular, a temporal profile of the radar scanning can be used to determine the seat occupancy state of the seating arrangement.Based on such a temporal progression, a movement or a specific movement pattern of one or more objects on the seating arrangement, and thus the presence of the object(s) themselves, can be detected with greater certainty than would typically be possible based on a single accumulated radar point cloud. This further improves the quality and reliability of the automated detection of a seat occupancy status in the seating arrangement.

[0023] In some embodiments, the method further comprises: for each of the at least two accumulated radar point clouds taken into account when determining the seat occupancy state, determining the respective value of at least one defined parameter for characterizing radar point clouds. The evaluation model is or will be defined such that, when determining the seat occupancy state of the seat arrangement, the evaluation result is determined as a function of the respective values ​​of the at least one parameter for the at least two accumulated radar point clouds taken into account when determining the seat occupancy state. In this way, the evaluation model can be determined and applied in a simplified manner, since instead of entire accumulated radar point clouds, only the values ​​of the at least one parameter need to be considered as input variables.In particular, the temporal progression of the values ​​can reflect a movement or a specific movement pattern of one or more objects on the seating arrangement, so that the evaluation model can determine the seat occupancy status of the seating arrangement in a particularly reliable manner on this basis.

[0024] In particular, in some embodiments, the subsets can be selected in a rolling manner to form an ordered series of subsets such that a first of the subsets has a number N of point sets that follow one another in the sequence, and each subsequent subset arises from the respective preceding subset by replacing the point sets of the preceding subset that are in the sequence M leading according to their order in the sequence by the M point sets that follow one another in the sequence, where 0 < M < N with N, M e N. In this way, a smoothing of the input variable(s) provided to the evaluation model can be achieved, which in turn can be used to increase quality.

[0025] In particular, it has been found that the following value ranges are favorable in terms of high quality of the method: N is or is chosen such that 4 < N < 8; and / or M is or is chosen such that M = 1 or M = 2.

[0026] A particularly suitable selection is N = 6 and M = 1 . In this case, the respective radar point clouds of N = 6 temporally successive measurement frames are combined to form a subset or, based on this, an accumulated radar point cloud. If, for example, the measurement frames are numbered 1, 2, 3, etc. according to their chronological order, then a first accumulated radar point cloud can be generated by accumulating the individual radar point clouds of measurement frames 1 to 6. The subsequent second accumulated radar point cloud can then be generated by accumulating the individual radar point clouds of measurement frames 2 to 7, etc. During this rolling, the leading measurement frame of the preceding subset is no longer contained in the immediately subsequent subset, but is replaced by a subsequent measurement frame.Overall, this results in a temporal sequence of accumulated radar point clouds, each defined by the accumulation of N individual radar point clouds. This allows a temporal progression of consecutive accumulated radar point clouds to be generated with the aforementioned advantages. In some embodiments, when determining the seat occupancy status of the seat arrangement, the evaluation result is determined as a function of the series of the respective values ​​of the parameter for the subsets of the series corresponding to each parameter for the series of subsets. This again reduces the complexity of the input variable(s) for the evaluation model. This allows, in particular, computational effort to be saved and high performance, especially real-time capability, to be achieved with weaker computing resources or lower computing resource utilization.

[0027] In some embodiments, for at least one of the parameters, the series of its respective values ​​is analyzed to determine whether a periodic course, in particular a course corresponding to a periodic breathing pattern, of the parameter is detected therein. The evaluation result is then determined depending on the result of the analysis. In particular, not only can the presence of any object on a seat of the seating arrangement be detected, but a distinction can even be made with high reliability between living objects present on the seating arrangement, in particular between people and mammals such as pets (e.g., dogs). In particular, it is possible to define the information to be output according to the method depending on the detection or non-detection of such a periodic course.In particular, one or more detected frequencies of the periodic profile can also be taken into account, in particular in such a way that the information is defined depending on whether the frequency(s) lies within a specific frequency range, such as a typical respiratory rate range. In particular, a seat belt warning function or an airbag system can be controlled depending on whether a respiratory rate, and thus with a high probability a person on the seating arrangement or a specific seat, has been detected.

[0028] In some embodiments, the or one of the characteristic variables is or is determined by the number of radar points in the respective accumulated radar point cloud or a variable dependent thereon. If, during radar scanning in a measurement frame, the number of radar points in the radar point cloud generated thereby depends on the extent to which the scanned object moves, the extent of the movement can be mapped in the characteristic variable(s), in particular with regard to the aforementioned detection of the object's breathing rate.

[0029] In some embodiments, the sequence of the measuring frames is or will be defined such that the temporally successive measuring frames therein follow one another periodically at a frequency f with 6 fps < f < 9 fps, in particular a frequency f with 7 fps < f < 8 fps has proven advantageous (fps = frames per second, ie measuring frames per second).

[0030] In some embodiments, the evaluation model comprises a trained machine learning model. Data representing at least one accumulated radar point cloud or values ​​of one or more parameters determined for this purpose are provided to the machine learning model as input data in order to obtain the evaluation result as its output. This allows for a particularly flexible and adaptable implementation of the evaluation model, whereby machine learning can be used to continuously improve the evaluation model and thus the quality and reliability of the seat occupancy detection. The evaluation result can, in particular, specify a class of a seat occupancy state classification. The machine learning model can, in particular, be a decision tree-based model or a model based on an artificial neural network.

[0031] In some embodiments (“first group”), the seating arrangement comprises a plurality of seats for which a seat occupancy status is to be determined individually or cumulatively within the scope of the method. For each (individual) radar point cloud, the set of its radar points is divided into several clusters, each containing a subset of the radar points, by means of cluster formation depending on the respective spatial position of the radar points in relation to the seats, in order to individually assign to each of the seats a cluster that is spatially closest to it (the clustering thus already takes place at the level of the individual radar point cloud(s)). The accumulation of radar point clouds of the sequence occurs cluster by cluster, with only the radar points belonging to this cluster from the respective radar point clouds to be accumulated being accumulated for each cluster in order to form a respective accumulated radar point cloud for each cluster.The seat occupancy state of the seating arrangement is determined individually for each of the seats as a function of the accumulated radar point cloud determined for the respective associated cluster in order to obtain an evaluation result, in particular a classification result, that characterizes the seat occupancy state of the respective seat. The information to be output is then defined as a function of the respective individual evaluation results for the various seats. In some embodiments of the first group, each radar point cloud is segmented into several clusters by assigning a subset of the radar points of the respective radar point cloud to each of the seats as a cluster depending on their respective position such that the radar points of the cluster lie in a defined closed spatial area, in particular a cuboid, in the vicinity of the seat.This enables particularly simple and low-computational-intensive clustering and thus seat-related seat occupancy detection, whereby the location (position and orientation) and the shape of the spatial area is or can be defined in such a way that it strongly overlaps the spatial area usually occupied by a typical object to be detected, in particular a person, on a seat of the seating arrangement.

[0032] In some other embodiments ("second group"), the seating arrangement also comprises a plurality of seats, for which a seat occupancy status is to be determined individually or cumulatively within the scope of the method. However, for each accumulated radar point cloud, the set of its respective radar points is subdivided into several clusters, each containing a subset of the radar points, by means of cluster formation, depending on the respective spatial position of the radar points in relation to the seats, in order to individually assign each of the seats a cluster that is spatially closest to it (clustering thus only takes place at the level of the already accumulated radar point cloud(s)).Determining the seat occupancy status of the seating arrangement involves individually determining a respective individual seat occupancy status for each of the seats based on the accumulated radar point cloud determined for the respective associated cluster in order to obtain an evaluation result characterizing a seat occupancy status of the respective seat. The information to be output is defined depending on the respective individual evaluation results for the various seats.

[0033] The embodiments of the second group thus represent alternatives to the embodiments of the first group. In both cases, in the case of a multi-seat seating arrangement, seat-related, i.e. individual for each seat, seat occupancy detection is made possible, which is particularly advantageous or even necessary if a seat-related reaction to the detected seat occupancy is to be carried out, for example by activating or deactivating or otherwise controlling a specific functionality or system for a specific seat depending on its detected seat occupancy, such as a seat-related airbag system, a seat-related seat belt warning or a seat-related seat heating.

[0034] In some embodiments of the second group, each accumulated radar point cloud is segmented into several clusters by assigning a subset of the radar points of the respective accumulated radar point cloud to each of the seats as a cluster, depending on their respective position, such that the radar points of the cluster are located in a defined, closed spatial area, in particular a cuboid, in the vicinity of the seat. This enables cluster formation that is also simplified compared to the first group and requires little computational effort, and thus enables seat-related seat occupancy detection, wherein the location (position and orientation) and shape of the spatial area is or can be defined such that it strongly overlaps the spatial area generally occupied by a typical object to be detected, in particular a person, on a seat of the seating arrangement.

[0035] The clusters thus each contain a subset, in particular a true subset, of the set of radar points of the respective entire (in the first group) individual or (in the second group) accumulated radar point cloud. In some embodiments, cluster formation can be performed in particular such that the clusters are disjoint, so that no radar point is assigned to two different clusters.

[0036] In some embodiments, the or each individual or accumulated radar point cloud is segmented into a plurality of clusters by assigning a subset of the radar points to each of the seats as a cluster depending on their respective position, in particular uniquely for each radar point, such that the radar points of the cluster are located in a defined, closed, in particular cuboid-shaped, spatial area in the vicinity of the seat. The assignment can in particular be carried out such that each radar point is assigned to the cluster of the seat closest to it. In this way, the radar point cloud can be divided into clusters, i.e. subsets of the radar point cloud localized in the vicinity of the respective seats, so that seat-specific seat occupancy states can be determined in a targeted manner and therefore with a high degree of reliability on the basis of the cluster assigned to the respective seat.

[0037] In some embodiments, outputting the information comprises controlling a signal source as a function of the information in order to cause the signal source to output a defined signal as a function of the control. The signal source can be, in particular, an audio source, an optical signal source, in particular a display device for images or text, and / or a haptic actuator, or a combination of at least two of the aforementioned signal sources. Thus, based on the signaling, the detected seat occupancy status can be communicated to a user or used to control another technical system, such as an airbag system.

[0038] In some of these embodiments, the signal source is controlled in dependence on the information such that it outputs a signal, in particular defined by the control, when the information results from an evaluation result, according to which at least one seat of the seat arrangement is occupied and / or a selected predetermined seat occupancy state exists.

[0039] In some embodiments, the method further comprises: (i) detecting a seat belt fastening state of at least one seat of the seat assembly or receiving seat belt information characterizing this seat belt fastening state; (ii) wherein the signal source is controlled as a function of the seat belt information and the information from the evaluation result such that it outputs a seat belt fastening indication signal if, according to the information, at least one seat of the seat assembly is occupied and / or a selected predetermined seat occupancy state exists and seat belt information indicates that the associated seat belt of the seat is not fastened. This makes it possible to achieve a radar-based seat belt fastening checking and warning system, particularly with regard to detection exclusively.

[0040] In some embodiments, the individual radar points of the radar point cloud are each represented by a position of the respective radar point in three-dimensional space and by at least one of the following parameters: (i) a Doppler shift value of the radar signal at the respective radar point; (iii) a signal-to-noise ratio value of the radar signal at the respective radar point. These parameters can be used, in particular, for pre-filtering the radar point cloud as part of a pre-processing step prior to feature determination.

[0041] In particular, in some of these embodiments, the determination of the seat occupancy status of the seat arrangement using the evaluation model can be carried out exclusively or at least predominantly on the basis of those radar points whose Doppler shift value is at or above a predetermined shift threshold different from zero. Thus, only or at least predominantly so-called dynamic radar points are used as the basis for the evaluation, i.e. those radar points that indicate a movement of the scanned object, whose Doppler shift value is at or above the shift threshold. This can be used in particular to further increase the quality, in particular the reliability of the method, because static, i.e.Essentially stationary points on the object surfaces, such as points on the surface of a seat, are not included in the analysis, or are included only in smaller numbers, than points that exhibit dynamic behavior and can therefore be attributed with a high degree of probability to a living being, in particular a person or an animal. This means that the information obtained and output by the analysis (e.g., for airbag control or a seat belt warning system) can be used depending on whether a living being or a static object surface was detected.

[0042] A second aspect of the present solution relates to a system, in particular a data processing device, for the automated detection of a, in particular respective, seat occupancy state of a seating arrangement with at least one seat, in particular with at least one vehicle seat in or for a vehicle. The system comprises a data processing device configured, in particular by means of a corresponding computer program, to execute the method according to the first aspect for detecting the seat occupancy state.

[0043] A third aspect of the present solution relates to a computer program or computer program product comprising instructions which, when executed on the data processing device of the system according to the second aspect, cause the system to carry out the method according to the first aspect.

[0044] The computer program can in particular be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloaded via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network. In addition, the computer program can have a plurality of interacting individual program modules. The modules can in particular be configured or at least be usable in such a way that they can be used in the sense of distributed computing (DC).“Distributed computing” is carried out on different devices (computers or processor units) that are geographically separated from each other and connected via a data network.

[0045] The system according to the second aspect can accordingly comprise a program memory in which the computer program is stored. Alternatively, the system can also be configured to access an external computer program, for example, available on one or more servers or other data processing units, via a communication connection, in particular to exchange data with it that is used during the execution of the method or computer program or that represents outputs of the computer program.

[0046] A fourth aspect of the present solution relates to a vehicle comprising: (i) a seating arrangement with at least one seat; (ii) a radar sensor for at least partially radar scanning the seating arrangement; and (iii) a system according to the second aspect for automatically detecting a, in particular respective, seat occupancy state of the seating arrangement as a function of an at least partially radar scan of the seating arrangement carried out by the radar sensor.

[0047] The features and advantages explained with regard to the first aspect of this solution also apply accordingly to the other aspects of the solution.

[0048] Further advantages, features and possible applications of the present solution will become apparent from the following detailed description in conjunction with the figures.

[0049] It shows:

[0050] Fig. 1 schematically shows an exemplary embodiment of a vehicle equipped with a system for automatically detecting a seat occupancy state of a seat arrangement in the vehicle;

[0051] Fig. 2 schematically shows the vehicle from Fig. 1, with a baby seat with a baby lying in it being the object to be detected on the passenger seat; Fig. 3A shows an exemplary two-dimensional representation of a (single) radar point cloud corresponding to a single measurement frame recorded by a radar sensor of the vehicle from Fig. 2;

[0052] Fig. 3B is an exemplary two-dimensional representation of an accumulated radar point cloud resulting from the accumulation of the respective individual radar point clouds of several consecutive measurement frames recorded according to Fig. 3A;

[0053] Fig. 3C is an exemplary representation of a clustering of the accumulated radar point cloud embodiment 3B according to the positions of the individual seats of the seating arrangement;

[0054] Fig. 4 is a flowchart illustrating an exemplary embodiment of a method for automatically detecting a seat occupancy state of a seat arrangement;

[0055] Fig. 5A is an exemplary representation of the course of the value of a characteristic parameter for individual radar point clouds for a sequence of measurement frames; and

[0056] Fig. 5B is an exemplary representation of the course of the value of a characteristic parameter for already accumulated radar point clouds for the same sequence of measurement frames as in Fig. 5A;

[0057] In the figures, like reference numerals designate like, similar, or corresponding elements. Elements shown in the figures are not necessarily drawn to scale. Rather, the various elements shown in the figures are depicted in such a way that their function and general purpose will be understood by those skilled in the art. Connections and couplings between functional units and elements shown in the figures can also be implemented as indirect connections or couplings, unless expressly stated otherwise. Functional units can, in particular, be implemented as hardware, software, or a combination of hardware and software.

[0058] The exemplary embodiment of a vehicle 100 schematically illustrated in Fig. 1 has a seating arrangement 105 with five individual seats or seating locations 105a to 105e. Each of the seats 105a to 105e is suitable for accommodating a person as a passenger of the vehicle 100. The vehicle 100 further has a radar sensor 110, which is mounted on the ceiling within the vehicle cabin and is configured to scan the seating arrangement 105, at least substantially, using radar beams. Accordingly, the seats 105a to 105e, in particular their seating surfaces, are each at least predominantly located within an observation field 110a that can be scanned by the radar sensor 110.In addition, the vehicle 100 has a system 115 for automatically detecting a seat occupancy state of the seat arrangement 105 as a function of a radar scan of the seat arrangement 105 carried out by the radar sensor 110, at least in sections with respect to the observation field 110a.

[0059] The system 115 comprises, in particular, a data processing unit 115a with at least one microprocessor and a memory 115b connected to the microprocessor for signal transmission, in which a computer program configured to carry out the method for automatically detecting a seat occupancy state of the seat arrangement 105, described below with reference to Fig. 4, is stored. Furthermore, the sensor data generated by the radar sensor 110 during radar scanning or information already obtained therefrom through further processing can be or will be stored in the memory 115b.

[0060] The vehicle 100 shown in Fig. 2 corresponds to the vehicle in Fig. 1, but here, a baby seat B with a baby lying therein is arranged on the passenger seat 105b as an object to be recognized for the purposes of a seat occupancy check. In the following discussion of Figs. 3A to 5B, reference is made to the configuration in Fig. 2.

[0061] Reference is now made below to Figures 3A to 3C, each of which represents a radar point cloud, wherein, for the purpose of representation, the respective, essentially three-dimensional radar point cloud has been reduced to two dimensions by projecting the positions of the radar points of the radar point cloud onto a plane spanned by two of its dimensions.

[0062] Fig. 3A illustrates an exemplary individual radar point cloud 300 as acquired within a single measurement frame, i.e., as a result of a radar scan of the seat assembly 105 by the radar sensor 110 during a specified time interval (measurement period). The position of the individual radar points within the radar point cloud 300 can be represented by spatial coordinates; for example, Cartesian coordinates X and Y can be assigned to the drawing plane and, accordingly, to each individual point. In reality, if the dimensionality reduction due to the drawing is ignored, a third coordinate Z for the third spatial dimension is also added.

[0063] If, during radar scanning, not only the spatial positions of the points where the radar beam is reflected by the scanned objects are recorded as coordinates, but also a respective Doppler shift is measured, then the individual radar points can be classified depending on the magnitude of this Doppler shift, specifically divided into two different classes. The latter can be achieved, for example, by comparing the Doppler shift with a predefined shift threshold corresponding to a specific shift velocity.Depending on the result of the comparison, those radar points 310 that, according to the value of their associated Doppler shift, exhibit no velocity or an object surface velocity at the reflection point that is below the displacement wave can be classified as "static" radar points (each represented by a filled black circle in Figures 3A-3C). Conversely, those radar points 315 that exhibit a Doppler shift above the displacement threshold can be classified as "dynamic" radar points 315 (each represented by a black ring in Figures 3A-3C).

[0064] The classification of radar points 310 and 315 according to their Doppler shift is not mandatory, but it can be used to process radar point cloud 300, particularly as part of a preprocessing step prior to its evaluation, particularly to filter it depending on the classification. For example, such filtering could be performed in such a way that only dynamic radar points 315 are considered for evaluation, for example, to detect only moving objects.

[0065] Fig. 3B shows an accumulated radar point cloud 305 which was generated by accumulating a plurality of individual radar point clouds, for example by accumulating six (i.e. N=6) individual radar point clouds from six, in particular consecutive, measurement frames. In the present example, the individual radar point cloud 300 from Fig. 3A is one of these radar point clouds combined by accumulation, so that its radar points 310 and 315 can also be found in Fig. 3B. By comparing Fig. 3A and Fig. 3B, it can easily be seen that, due to the accumulation, the average radar point density in Fig. 3B is significantly higher than in Fig. 3A. In particular, it can also be seen that in the areas 320 marked by circles, there is a particularly high radar point density which, in particular, has both static radar points 110 and dynamic radar points 115.This indicates that there is a high probability that an object is present (high radar point density) that is at least partially moving (high point density of dynamic radar points).

[0066] N = 6 represents a particularly advantageous choice for N here, because it covers both several periods of the periodic breathing process recognizable in Fig. 3B and the accumulation with regard to the number of individual radar point clouds summarized by accumulation is short enough to obtain all important data (such as the breathing pattern itself and the length of a breath) for the subsequent evaluation in the resulting accumulated radar point cloud 305.

[0067] Fig. 3C shows the same radar point cloud 305 as in Fig. 3B. However, selected spatial regions 325a to 325e are additionally depicted here, either cuboid-shaped (3D case) or rectangular in the present 2D representation. These spatial regions are spatially assigned to the respective location of the individual seats 105a to 105e. The definition of these spatial regions 325a to 325e can now be used to cluster the accumulated radar point cloud 305, with each radar point 310 or 315 being assigned, as far as possible, to the spatial region 325a to 325e in which it is located. All radar points not located in one of the spatial regions 325a to 325e can be disregarded in the following. In particular, it can be seen that the areas 320 with a particularly high radar point density are located in the area of ​​the passenger seat 105b, on which the baby seat with the baby B is located according to Fig. 2.

[0068] Figure 4 shows a flowchart illustrating an exemplary embodiment 400 of a method for automatically detecting a seat occupancy state of a seat arrangement. The method can be embodied, in particular, as a computer-implemented method. For this purpose, it can be stored, in particular, in the memory 115b of the system 115 as a computer program and be executable on the data processing unit 115a.

[0069] In method 400, an initialization process 405 is first carried out, by which initial values ​​are set for various parameters of the method. In particular, a counter i for measurement frames and a counter j for generated accumulated radar point clouds can be initialized, for example with i = 0 and j = 0. Furthermore, a parameter N can be initialized here, which specifies a number of individual radar point clouds (e.g., radar point cloud 300), each corresponding to a corresponding number of different measurement frames, which are to be combined to form an accumulated radar point cloud (e.g., accumulated radar point cloud 305). In addition, a further parameter M can be initialized, which, as part of a rolling accumulation, specifies how many new measurement frames are added in the next step of the rolling process, with a corresponding number of old measurement frames being discarded at the same time.In addition, a further parameter can be defined as a number L of measurement frames that must be included in the rolling accumulation before the generated sequence of accumulated radar point clouds is evaluated for detecting a seat occupancy state. In the present example, N = 6, M = 1, and L = 48 are set.

[0070] After the initialization process 405 is completed, a further phase of the method 400 begins, in which radar measurement data, in the present example from radar sensor 110 of the vehicle 100, is received and further processed to form one or more accumulated radar point clouds. For this purpose, in a process 410, the radar measurement data for a new measurement frame from a temporal, in particular periodic, sequence of several measurement frames is first received. At the same time, the counter i is incremented to indicate the receipt of the radar measurement data for a new measurement frame. In a further process 415, a check is then made as to whether radar measurement data for N measurement frames have already been received. If this is not the case (415 - no), the process 410 is repeated until the radar measurement data for N measurement frames, each representing an individual radar point cloud measured at a time or period corresponding to the respective measurement frame, have been received.

[0071] Then, in a further process 420, the first N received individual radar point clouds 300 are accumulated to form an accumulated radar point cloud 305. Optionally, filtering can be performed to filter out static radar points, i.e., radar points whose Doppler shift value is less than a predefined Doppler shift threshold. The Doppler shift threshold can, if necessary, also be initialized in the initialization process 405.

[0072] The now available accumulated radar point cloud 305 can then be clustered in a further process 425, in which each of its radar points is checked to determine whether it lies within one of the defined spatial areas 325a to 325e (see Fig. 3C) and, if so, in which one. Thus, each of the points can be assigned either to one of the spatial areas 325a to 325e or to the other observation field. All radar points that lie within the same spatial area 325a to 325e are combined into a respective cluster. As a result, each of the seats 105a to 105e is assigned a corresponding cluster of the accumulated radar point cloud 305. This forms the basis for an individual evaluation to be carried out for each seat 105a to 105e as to whether the respective seat 105a to 105e is or was occupied or not during the period covered by the N measurement frames on the basis of which the accumulated radar point cloud 305 was formed.

[0073] To facilitate the subsequent evaluation of the clustered accumulated radar point clouds 305, a corresponding characteristic K(i) can be determined for each of the clusters in a process 430, wherein this characteristic K(i) can be defined, in particular, as the number of radar points in the cluster. If no filtering according to the Doppler shift value has taken place, this can involve a joint count of both the static and the dynamic radar points 310 and 315, respectively. However, if the static radar points 310 were previously filtered out, for example, as mentioned, in process 420, only the dynamic radar points 315 are counted.

[0074] Since the subsequent evaluation is to be based on a progression of the characteristic K(i) over a value range of the counter i defined by the parameter L, a process 435 checks whether a total of L measurement frames have already been processed. If this is not the case (435 - no), the method returns to step 410 to receive another measurement frame and determine a new accumulated radar point cloud and its characteristic. Otherwise (435 - yes), the process can continue with determining a seat occupancy state in a process 440, as will be explained below.

[0075] First, however, Figures 5A and 5B will be discussed, which each illustrate an exemplary course of the value of the parameter K(i) for the cluster formed for the seat 105b or the spatial area 325b in Figures 5A and 5B.

[0076] For the purpose of comparison with Fig. 5B, the number K(i) of radar points to be counted for each individual measurement frame i is shown in Fig. 5A, without any accumulation taking place. The total number K(i) for the first L = 48 measurement frames is shown here. In contrast, Fig. 5B shows a corresponding representation of a curve 505 of the parameter K(i) according to method 400, with accumulation taking place over N = 6 measurement frames each. The counter i here corresponds to an index of the first measurement frame taken into account in the accumulation. The accumulated number of radar points resulting from accumulating the first N = 6 measurement frames 1 to 6 is thus assigned a counter value i = 1 in the diagram in Fig. 5B. The accumulated number of radar points from the subsequent accumulation of measurement frames 2 to 7 is then assigned the value i = 2, etc.

[0077] According to the initialization of N = 6 and M = 1 chosen in the present example, six measurement frames or their associated individual radar point clouds 300 are accumulated in each case. When moving from one accumulation to the next, the first measurement frame is removed from the next accumulation and replaced by a measurement frame that follows in the sequence of measurement frames and has not yet been taken into account. This is indicated in Fig. 5B by the brackets in the lower left area of ​​the image. The rolling accumulation takes place until a total of the third parameter L defined number of measurement frames has been processed, i.e. in the present example a total of L = 48 measurement frames were taken into account in the accumulation to form the various accumulated radar point clouds.

[0078] When comparing the curves of K(i) in Figures 5A and 5B, it is easy to see that the curve of K(i) in Figure 5B is smoothed compared to that in Figure 5A. However, the characteristic periodic pattern, which may be caused in particular by the breathing process of baby B on the seat 105b detected during the radar measurement, is retained due to the appropriate choice of N=6. However, the larger N, the greater the risk that such patterns will be flattened or disappear during the smoothing process, so that N should expediently be set here with a view to preserving such patterns.

[0079] Referring again to Fig. 4, the evaluation of the course of the parameter K(i) for the cluster for seat 105b can now take place (the same can be done analogously for the respective clusters for the other seats). For this purpose, in process 440, the course of the parameter K(i) from Fig. 5B is made available as an input to an evaluation model. This can in particular be a model based on machine learning, such as an artificial neural network or a decision tree-based model (“decision tree(s)”). The training and, if applicable, validation data used for the preceding training can be structured such that they each contain courses of the parameter K(i) corresponding to the type shown in Fig. 5B for a multitude of different radar point clouds or clusters thereof, as well as an assigned correct class of a classification of possible seat occupancy states for each course of K(i).In this way, the model can be trained and validated using supervised learning. In the simplest case, seat occupancy states indicate whether the seat is occupied or not. However, more sophisticated classifications are also conceivable, in which, in the case of the presence of an object, the respective class additionally indicates the type of object, for example, whether it is moving or stationary, and in the case of a moving object, in particular, whether it is a person (generally recognizable in particular based on a breathing pattern in the course of the characteristic K(i)).

[0080] As an alternative to a machine learning model, a simple comparison algorithm can also be used, in which the course of the parameter K(i) is used as an input in the sense of a mathematical function and compared with one or more comparison thresholds to obtain a classification result. It is also conceivable that a frequency analysis of K(i) is performed, for example, using a Fourier transform, particularly to detect a respiratory rate as the basis for classification.

[0081] If, in process 440, a seat occupancy state for the seat arrangement 105 was determined based on the evaluation model, in particular for one or more of its seats 105a to 105e individually, this result can be output as corresponding information in process 445, for example at a user interface of the vehicle or in the form of data for further processing by one or more other systems, in particular systems of the vehicle.

[0082] In the present example, this information is to be used in particular to check whether or not to issue a belt warning signal depending on the seat occupancy status of a respective seat 105a to 105e and the result of a check as to whether or not a corresponding seat belt has been fastened for this seat.

[0083] For this purpose, in process 450, it can be checked whether the seat belt for the relevant seat (here, for example, for seat 105b) is fastened, and in step 455, a functionality of the vehicle 100 can be controlled depending on the information on the seat occupancy state output in process 445 and the status of the seat belt determined in process 450. In particular, this can be done in such a way that in process 455, a signal source is activated to output a belt status signal, in particular a visual and / or acoustic one, in order to signal to one or more other occupants of the vehicle, if necessary, that a seat is occupied but the seat belt is not fastened there. The method then branches back to step 410 to start another loop run.

[0084] While at least one exemplary embodiment has been described above, it should be appreciated that a wide variety of variations exist. It should also be understood that the described exemplary embodiments are merely non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description will provide one skilled in the art with guidance for implementing at least one exemplary embodiment, it being understood that various changes in the operation and arrangement of the elements described in an exemplary embodiment may be made without departing from the subject matter as defined in the appended claims, as well as their legal equivalents.

[0085] LIST OF REFERENCE SYMBOLS

[0086] B Baby in baby seat

[0087] 100 vehicles

[0088] 105 Seating arrangement

[0089] 105a-e Seats or seating positions

[0090] 110 radar sensor

[0091] 110a Observation field of radar sensor 110

[0092] 115 System for automated detection of a seat occupancy status

[0093] 115a Data processing unit

[0094] 115b memory

[0095] 300 (individual) radar point clouds

[0096] 305 accumulated radar point cloud

[0097] 310 static radar points

[0098] 315 dynamic radar points

[0099] 320 areas of the accumulated radar point cloud 305 with high

[0100] Radar point density

[0101] 325a-e Spatial areas for cluster definition

[0102] 400 Methods for the automated detection of a seat occupancy status

[0103] 405-455 individual processes or procedural steps within the framework of procedure 400

[0104] 500 Course of the parameter K(i) for a single radar point cloud

[0105] 505 Course of the parameter K(i) for a cumulative radar point cloud

Claims

REQUIREMENTS 1. Method for automatically detecting the occupancy status of a seating arrangement (105) with at least one seat (105a-e), wherein the method comprises: Receiving or generating measurement data which, for each measurement frame of a sequence of several temporally successive measurement frames, represent an associated radar point cloud (300), such that the measurement data represent a sequence of radar point clouds corresponding to the sequence of measurement frames, wherein each radar point cloud (300) of the sequence was or is obtained on the basis of a radar scan of a spatial area surrounding the seating arrangement (105) at least partially, which takes place at or during a measurement time or measurement period associated with the respective measurement frame; Accumulating several of the radar point clouds of the sequence to obtain an accumulated radar point cloud (305) containing radar points (310, 315) from each of the individual radar point clouds aggregated in the accumulation; Determining a seat occupancy state of the seating arrangement (105) using an evaluation model that, depending on the accumulated radar point cloud, delivers one of several predefined possible seat occupancy states of the seating arrangement (105) as an evaluation result; and Output of information defined based on the evaluation result.

2. Method according to claim 1, wherein: the accumulation of several of the radar point clouds of the sequence is carried out multiple times in order to obtain an accumulated radar point cloud (305) which contains radar points (310, 315) from each of the individual radar point clouds combined in the respective accumulation, wherein in each accumulation only radar point clouds of an assigned individual subset of the sequence are accumulated, such that at least two of the subsets are different; and the determination of the seat occupancy state of the seat arrangement (105) is carried out as a function of at least two of the accumulated radar point clouds. The method according to claim 2, further comprising: for each of the at least two accumulated radar point clouds considered in the context of determining the seat occupancy state, determining the respective value of at least one defined characteristic parameter (K(i)) for characterizing radar point clouds; wherein the evaluation model is or is defined such that, when determining the seat occupancy state of the seat arrangement (105), the evaluation result is determined as a function of the respective values ​​of the at least one characteristic parameter (K(i)) for the at least two accumulated radar point clouds considered in the context of determining the seat occupancy state.The method of claim 3, wherein the subsets are selected in a rolling manner to form an ordered series of subsets such that a first of the subsets has a number N of point sets that are successive in the sequence, and each subsequent further subset is derived from the preceding subset by replacing the point sets of the preceding subset that are leading in the sequence M in the sequence with the point sets that are next M in the sequence, wherein 0 < M < N with N, M. N applies. Method according to claim 4, wherein: N is chosen or is chosen such that 4 < N < 8; and / or M is chosen or is chosen such that M = 1 or M = 2. A method according to claim 4 or 5, wherein, when determining the seat occupancy status of the seat arrangement (105), the evaluation result is determined as a function of the series of values ​​of the characteristic parameter (K(i)) corresponding to the series of subsets of the series for each characteristic parameter (K(i)). A method according to claim 5, wherein, for at least one of the characteristic parameters, the series of its respective values ​​is analyzed to determine whether a periodic trend of the characteristic parameter (K(i)) is detected therein; and the evaluation result is determined as a function of the result of the analysis.

8. Method according to one of claims 3 to 7, wherein the or one of the characteristic parameters is or is determined by the number of radar points (310, 315) in the respective accumulated radar point cloud or a parameter dependent thereon.

9. Method according to one of the preceding claims, wherein the sequence of measurement frames is or is defined such that the temporally successive measurement frames therein follow each other periodically with a frequency f of 6 fps < f < 9 fps.

10. Method according to one of the preceding claims, wherein: the evaluation model comprises a trained machine learning model; and data representing at least one accumulated radar point cloud (305) or values ​​of one or more specific parameters are provided as input data to the machine learning model in order to obtain the evaluation result as its output.

11. Method according to one of the preceding claims, wherein: the seating arrangement (105) has a plurality of seats (105a-e) for which a seat occupancy state is to be determined individually or cumulatively within the framework of the method; for each radar point cloud (300), the set of its radar points (310, 315) is subdivided by means of cluster formation into several clusters, each containing a subset of the radar points (310, 315), depending on the respective spatial location of the radar points (310, 315) with respect to the seats (105a-e), in order to individually assign to each of the seats (105a-e) one of the clusters that is spatially closest to it; The accumulation of radar point clouds of the sequence is carried out cluster by cluster, whereby for each cluster only the radar points (310, 315) belonging to this cluster of the radar point clouds to be accumulated are accumulated in order to form a respective accumulated radar point cloud (305) for each cluster;and determining the occupancy status of the seating arrangement (105) includes individually determining a respective individual occupancy status for each of the seats (105a-e) depending on the accumulated radar point cloud determined for the respective associated cluster, in order to obtain an evaluation result characterizing an occupancy status of the respective seat (105a-e); and; The information to be output is defined depending on the respective individual evaluation results for the different seats (105a-e). Method according to claim 1, wherein each radar point cloud (300) is segmented into several clusters by assigning to each of the seats (105a-e) as a cluster a subset of the radar points (310, 315) of the respective radar point cloud (300) depending on their respective position such that the radar points (310, 315) of the cluster lie in a defined closed spatial area in the vicinity of the seat (105a-e). Method according to any one of the preceding claims 1 to 10, wherein: the seating arrangement (105) has a plurality of seats (105a-e) for which a seat occupancy state is to be determined individually or cumulatively within the framework of the method;For each accumulated radar point cloud (305), the set of its respective radar points (310, 315) is subdivided into several clusters, each containing a subset of the radar points (310, 315), by means of cluster formation depending on the respective spatial location of the radar points (310, 315) in relation to the seats (105a-e), in order to individually assign to each of the seats (105a-e) a cluster that is spatially closest to it; and the determination of the seat occupancy state of the seating arrangement (105) comprises an individual determination of a respective individual seat occupancy state for each of the seats (105a-e) depending on the accumulated radar point cloud determined for the respective associated cluster, in order to obtain an evaluation result characterizing a seat occupancy state of the respective seat (105a-e);and the information to be output is defined depending on the respective individual evaluation results for the different seats (105a-e). Method according to claim 13, wherein each accumulated radar point cloud (305) is segmented into several clusters by assigning to each of the seats (105a-e) as a cluster a subset of the radar points (310, 315) of the respective accumulated radar point cloud depending on their respective position such that the radar points (310, 315) of the cluster lie in a defined closed spatial area in the vicinity of the seat (105a-e).

15. Method according to any of the preceding claims, wherein the output of the information comprises controlling a signal source depending on the information in order to cause the signal source to output a defined signal depending on the control.

16. Method according to claim 15, wherein the signal source is controlled in such a way as to depend on the information that it outputs a signal when the information results from an evaluation result according to which at least one seat of the seating arrangement (105) is occupied and / or a selected predetermined seat occupancy state exists.

17. The method of claim 16, further comprising: Detecting a seat belt fastening state of at least one seat of the seating arrangement (105) or receiving seat belt information characterizing this seat belt fastening state; wherein the signal source is controlled in such a way as to depend on the seat belt information and the information from the evaluation result, such that it outputs a seat belt fastening indicator signal when, according to the information, at least one seat of the seating arrangement (105) is occupied and / or a selected predetermined seat occupancy state is present and seat belt information indicates that the corresponding seat belt of the seat is not fastened.

18. Method according to one of the preceding claims, wherein the individual radar points (310, 315) of each radar point cloud (300) are each represented by a position of the respective radar point in three-dimensional space and by at least one of the following parameters: - A Doppler shift value of the radar signal relative to the respective radar point; - A signal-to-noise ratio value of the radar signal to the respective radar point.

19. Method according to claim 18, wherein: the individual radar points (310, 315) of each radar point cloud (300) are each represented by a position of the respective radar point in three-dimensional space and by the Doppler shift value of the radar signal to the respective radar point (310, 315); and the determination of the seat occupancy status of the seating arrangement (105) based on the evaluation model is carried out exclusively or at least numerically predominantly based on such radar points (310, 315) whose Doppler displacement value is at or above a predetermined displacement threshold other than zero. System (115) for automatically detecting the occupancy status of a seating arrangement (105) with at least one seat (105a-e), wherein the system (115) comprises a data processing device configured to execute the method according to one of the preceding claims for detecting the occupancy status. Computer program or computer program product comprising instructions which, when executed on the data processing device of the system (115) according to claim 20, cause the system (115) to execute the method according to one of claims 1 to 19.Vehicle (100) comprising: a seating arrangement (105) with at least one seat (105a-e); a radar sensor (110) for at least section-wise radar scanning of the seating arrangement (105); and a system (115) according to claim 20 for automatically detecting a seat occupancy state of the seating arrangement (105) depending on at least section-wise radar scanning of the seating arrangement (105) performed by the radar sensor (110).

Images