Method and system for training an evaluation model for detecting a seat occupancy status of a seating arrangement based on radar point clouds

Abstract

Method for training an evaluation model for automated detection of a seat occupancy state of a seating arrangement (105) with at least one seat (105a-e), the method comprising: - Acquiring measurement data representing an associated radar point cloud (305), wherein the radar point cloud (305) was or is obtained on the basis of a radar scan of a spatial region surrounding the seat arrangement (105) at least in sections and is associated with one of several predefined possible seat occupancy states of the seat arrangement (105); - generating augmented measurement data representing an associated augmented point cloud from the measurement data associated with the radar point cloud (305), wherein generating the augmented measurement data comprises: - Determining multiple anchor points within the spatial area; - determining a transformation with respect to each of the anchor points, wherein each of the transformations is applied to at least a portion of the points of the radar point cloud (305) to obtain a transformed point cloud associated with the respective anchor point; and - Merging the transformed point clouds associated with the anchor points to obtain the augmented point cloud associated with the augmented measurement data; and - generating training data from the measurement data and the augmented measurement data, wherein the training data are made available to the evaluation model as input data in order to obtain an evaluation result as its output, which is assigned to the seat occupancy state of the seat arrangement (105).

Description

[0001] The present invention relates to a method, a computer program, and a system configured to carry out the method, each for the automated detection of a seat occupancy state of a seating arrangement having at least one seat using a trained evaluation model. In particular, the invention also relates to a method for training such an evaluation model, in particular a machine learning model.

[0002] In various situations, it may be necessary to automatically determine the current seat occupancy status of a seating arrangement with at least one seat. Such a situation can occur particularly in vehicles, for example in motor vehicles, where a vehicle configuration or the activation, deactivation, and / or control of one or more vehicle functions is to take place depending on a current seat occupancy status. For example, it is known in motor vehicles to issue an acoustic or visual indication to vehicle occupants to fasten seat belts or to control the activation or deactivation of airbags depending on a detected seat occupancy status.

[0003] For the automated detection of the current occupancy status of one or more seats, in particular of an arrangement of vehicle seats in a vehicle, so-called seat occupancy mats are known. These mats are integrated into the seats (usually one per seat) and use pressure-sensitive sensors to detect occupancy of the respective seat. The seat occupancy status is determined based on the sensor signals or sensor data from these sensors, usually by means of a threshold test.

[0004] These known solutions therefore require the seats to be equipped with built-in sensors and are usually unable to distinguish between seat occupancy states beyond "occupied" and "unoccupied." Furthermore, the sensors typically have to be integrated into the seats at the factory, making retrofitting difficult or impossible, and eliminating the ability to detect seat occupancy for seats not equipped with such sensors.

[0005] Therefore, methods are also known that use radar technology to automatically detect current seat occupancy. Radar sensors scan the vehicle interior to generate measurement data in the form of radar point clouds. Based on the measured radar point cloud, current seat occupancy can be determined. A corresponding evaluation model, which may be a machine learning model, can be used for this purpose. This model can be trained with appropriate data before actual use. However, to obtain a reliable result, a considerable amount of training data is required, which in turn involves considerable effort.

[0006] It is an object of the present invention to provide an improved solution for the automated detection of a seat occupancy state of a seating arrangement having at least one seat. In particular, an improved solution for training an evaluation model for the automated detection of a seat occupancy state of a seating arrangement is to be provided.

[0007] This object is achieved according to the teaching of the independent claims. Various embodiments and developments of the invention are the subject of the dependent claims.

[0008] A first aspect of the solution presented here relates to a method, in particular a computer-implemented method, for training an evaluation model, in particular a machine learning model, for automated detection of a seat occupancy state of a seating arrangement with at least one seat. The method comprises: (i) capturing, in particular receiving or generating, measurement data representing an associated radar point cloud, wherein the radar point cloud was or is obtained based on a radar scan of a spatial region surrounding the seating arrangement at least in part.and is assigned to one of several predefined possible seat occupancy states of the seating arrangement; (ii) generating augmented measurement data representing an associated augmented point cloud from the measurement data assigned to the radar point cloud; and (iii) generating training data from the measurement data and the augmented measurement data, wherein the training data is made available as input data to the evaluation model in order to obtain as its output an evaluation result which is assigned to the seat occupancy state of the seating arrangement.Generating the augmented measurement data comprises: (i) determining a plurality of anchor points within the spatial domain; (ii) determining a transformation with respect to each of the anchor points, wherein each of the transformations is applied to at least a portion of the points of the radar point cloud to obtain a transformed point cloud associated with the respective anchor point; and (iii) merging the transformed point clouds associated with the anchor points to obtain the augmented point cloud associated with the augmented measurement data.

[0009] Using the method according to the first aspect, an evaluation model, which can in particular be a machine learning model, can be effectively trained so that the accuracy in detecting a seat occupancy state can be improved. Using data augmentation, an increased amount of training data can be easily generated without the need for new actual measurement data. The training method is particularly advantageous for the machine learning of a radar-based system for detecting seat occupancy. The method involves training machine learning using actually measured radar point clouds and augmented point clouds in parallel. By augmenting (also called "extending") the points from different anchor points or starting points, multiple sets of augmented (extended) point clouds can be easily obtained.These extended point clouds are merged to obtain the additional training data.

[0010] The term "seat occupancy state" of a seating arrangement with at least one seat, as used herein, is to be understood in particular as information indicating whether or to what extent the seating arrangement or at least one of its seats is occupied by an object, in particular a thing or a person. In a simple example, the seat occupancy state can only indicate the presence or absence of an object, or in a more advanced example, in the case of the presence of at least one object on the seating arrangement or one or more of its seats, it can provide information about the type or other property, such as a spatial extent, of the object.

[0011] The term "radar point cloud," as used herein, refers in particular to a set of points in a vector space obtained by radar scanning of at least one object surface, which has a typically unorganized spatial structure ("cloud"). In the case of a radar point cloud, the points of the radar point cloud can be referred to as "radar points." A (radar) point cloud can be described in particular by the (radar) points it contains. The radar points, in turn, can each be described in particular by their spatial coordinates, which indicate, for each radar point, a location of the reflection of an emitted radar signal on an object surface, measured during the radar scanning. In addition to the radar points, attributes such as measured Doppler velocity or a signal-to-noise ratio (SNR) can also be recorded.

[0012] The term "augmentation" or "data augmentation," as used herein, refers to the expansion or enlargement of a data set used for training purposes for an evaluation model, in particular a machine learning model. Data augmentation (or data expansion) achieves an increase in the data set by adding slightly modified copies of existing data or newly created synthetic data from existing data. It also acts as a regulator when training a machine learning model and helps reduce overfitting during training of a machine learning model.

[0013] The term “augmented point cloud” as used herein refers in particular to a point cloud obtained by data augmentation of a radar point cloud.

[0014] The term "anchor point," as used herein, refers in particular to a starting point or reference point used for the points of the point cloud to be transformed during a transformation. For example, an anchor point can serve as a reference point for a measurement, such as a distance measurement, or as a center for a transformation, such as a center of rotation for a rotation. The anchor point can thereby form a zero point of a coordinate system or space in which the transformation is performed. The anchor point can also itself be subject to the transformation.

[0015] The term “evaluation model,” as used herein, refers to a model, particularly a mathematical one, that uses a radar point cloud or one or more parameters characterizing it as input variables in order to provide an evaluation result depending thereon, in this case one of several predefined possible seat occupancy states of the seating arrangement. The evaluation model can in particular be a mathematical estimation function, wherein the radar point cloud represents empirical data as a sample and the evaluation result represents an estimated value determined depending thereon. The evaluation model can in particular be a “machine learning model” (orA machine learning model can be a "machine learning model" (synonymous with "machine learning model"), which here is understood in particular to mean a mathematical, in particular statistical, model for making predictions or decisions created by means of at least one machine learning algorithm on the basis of sample data referred to as training data, without the algorithm(s) being explicitly programmed to make such predictions or decisions. In particular, decision tree-based machine learning models (English "decision trees") are machine learning models.

[0016] The terms "comprises," "includes," "includes," "has," "has," "with," or any other variation thereof, as used herein, are intended to cover non-exclusive inclusion. For example, a method or apparatus that includes or has a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or that are inherent in such a method or apparatus.

[0017] Furthermore, unless explicitly stated to the contrary, "or" refers to an inclusive "or" and not an exclusive "or." For example, a condition A or B is satisfied by one of the following conditions: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).

[0018] As used herein, the terms "a" or "an" are defined to mean "one or more." The terms "another" and "another," and any other variations thereof, are defined to mean "at least one other."

[0019] The term “plurality” or “several” as used here shall be understood to mean “two or more”.

[0020] The terms “configured” or “set up” to perform a specific function (and respective variations thereof), as used herein, are to be understood within the meaning of the invention as meaning that the corresponding device is already in a configuration or setting in which it can perform the function or is at least adjustable - i.e. configurable - so that it can perform the function after being set accordingly. The configuration can be carried out, for example, by appropriately setting parameters of a process sequence or of switches or the like for activating or deactivating functionalities or settings. In particular, the device can have a plurality of predetermined configurations or operating modes, so that the configuration can be carried out by selecting one of these configurations or operating modes.

[0021] In the following, various exemplary embodiments of the method are described, each of which, unless expressly excluded or technically impossible, can be combined with each other as well as with the other aspects of the present solution described.

[0022] In some embodiments, the anchor points are points from the radar point cloud associated with the measurement data. This simplifies the determination of the anchor points, since no new points need to be created in the spatial region; instead, a selection can be made from the existing points in the radar point cloud.

[0023] In some embodiments, the anchor points are determined such that each of the anchor points within the spatial area has the greatest distance to the respective remaining anchor points. This allows the transformations to be distributed as optimally as possible across the spatial area. Furthermore, this allows different types of transformations to be carried out, in particular compared to anchor points that are located close to one another. The anchor points can be determined such that a first anchor point is determined randomly or, alternatively, manually. A second anchor point is then selected with the greatest possible distance from the first anchor point. All subsequent anchor points are then each determined with the greatest possible distance from the anchor points already determined. So-called “farthest point sampling” (FPS) can be used, in particular to select the anchor points from the set of radar points.Alternatively, all anchor points could be determined randomly, although this would no longer guarantee the greatest possible distance between the anchor points.

[0024] In some embodiments, each of the transformations comprises at least one of rotation, scaling, and translation (displacement) with respect to the respective anchor point. A transformation may comprise one of the aforementioned or a combination thereof. The values ​​of the rotation, scaling, and translation may be randomly determined. These transformations can be easily applied to the points of the radar point cloud, for example, using appropriate matrices. Optimized value ranges of the transformations can be determined using hyperparameter optimization, e.g., Bayesian optimization.

[0025] In some embodiments, each of the transformations is weighted point by point with a weighting factor, wherein the weighting factor depends in each case on the distance between a point in the radar point cloud to be transformed and the respective anchor point. The weighting can be used to influence the effect of the respective transformation on the individual points in the point cloud. “Point by point” here means that each point in the point cloud receives a weighting factor. The weighting factors are advantageously selected such that the transformation has a smaller effect on points that are further away from the corresponding anchor point than on points that are closer to the anchor point. For this purpose, a transformed point can be multiplied by the weighting factor or, if the weighting factors are selected accordingly, divided. In this way, local transformations are created with the respective anchor point as the center.For example, the weighting factors can be chosen according to a Gaussian distribution depending on the distance to the anchor point. In other words, this results in a "smoothing" with a Gaussian kernel. The weighting factor w can be defined, for example, as w=e−d2σ2 where d is the distance from a point to the anchor point. The value σ can be, for example, 0.5.

[0026] In some embodiments, the transformed point clouds assigned to the anchor points are combined by forming a point-wise average of the transformed point clouds assigned to the anchor points. In this way, an augmented point cloud can be easily generated from the transformed point clouds. Each point of the radar point cloud is assigned several transformed points, with the number corresponding to the number of anchor points, since each anchor point is assigned a transformed point cloud.

[0027] In some embodiments, the training data is generated by a weighted sum of the measurement data and the augmented measurement data. By linking the obtained augmented measurement data with the actual measurement data, it can be ensured that the training data is related to the actual measurement data and, for example, does not deviate too arbitrarily from the measurement data. Weighting can be performed with a parameter, for example, a difficulty parameter α, which can assume values ​​from 0 to 1. Here, the augmented measurement data can be weighted with α and the actual measurement data with (1 - α). The difficulty parameter can also be determined by appropriate hyperparameter optimization.

[0028] The seating arrangement may comprise a plurality of seats, wherein the seat occupancy state may be an individual or cumulative seat occupancy state of the seats.

[0029] In some embodiments, the set of radar points for the radar point cloud is divided into several clusters, each containing a subset of the radar points, by means of cluster formation depending on the respective spatial position of the radar points in relation to the seats, in order to individually assign to each of the seats a cluster that is spatially closest to it, wherein the augmented measurement data is generated individually for each cluster. The seat occupancy status of the seating arrangement is determined for each of the seats depending on the radar point cloud determined for the respective associated cluster in order to obtain an evaluation result, in particular a classification result, characterizing a seat occupancy status of the respective seat. The information to be output is then defined depending on the respective individual evaluation results for the various seats.If the augmented measurement data is generated individually for each cluster, the efficiency of the training data can be increased, since the entire radar point cloud does not need to be used for the training data. Alternatively, it is also possible to apply the augmentation to the entire radar point cloud.

[0030] In some embodiments, the radar point cloud is segmented into multiple clusters by assigning a subset of the radar points of the respective radar point cloud to each of the seats as a cluster, depending on their respective position, such that the radar points of the cluster are located in a defined, closed, in particular cuboid-shaped, spatial area in the vicinity of the seat. This enables particularly simple and low-computational-intensive cluster formation and thus seat-related seat occupancy detection, wherein the location (position and orientation) and shape of the spatial area is or can be defined such that it strongly overlaps the spatial area usually occupied by a typical object to be detected, in particular a person, on a seat of the seating arrangement.

[0031] In some embodiments, for the augmented point cloud, the set of its points is divided into several clusters, each containing a subset of the points, by means of cluster formation depending on the respective spatial position of the points in relation to the seats, in order to individually assign to each of the seats a cluster that is spatially closest to it. In other words, clustering takes place (also) after augmentation of the radar point cloud. This can further improve the result, since some points could lie outside an original cluster after augmentation, i.e., after the transformations. Such points are then no longer assigned to the original cluster during clustering after augmentation.In particular, as described above, a cluster can be created by means of corresponding spatial areas assigned to the individual seats, which are preferably cuboid-shaped and essentially cover the area in which a person is expected to be on the respective seat.

[0032] In some embodiments, the individual radar points of the radar point cloud are each represented by a position of the respective radar point in three-dimensional space and by at least one of the following parameters: (i) a Doppler shift value of the radar signal at the respective radar point; (iii) a signal-to-noise ratio value of the radar signal at the respective radar point. These parameters can be used, in particular, for pre-filtering the radar point cloud as part of a pre-processing step prior to feature determination.

[0033] A second aspect of the solution presented here relates to a method, in particular a computer-implemented method, for the automated detection of a seat occupancy state, in particular a seat-related one, of a seating arrangement with at least one seat (or equivalently: seat), in particular a seat in or for a vehicle, such as an automobile (e.g., truck, car, or bus). The method comprises: (i) acquiring, in particular receiving or generating, measurement data representing an associated radar point cloud.Each radar point cloud is or was obtained based on a radar scan of a spatial region surrounding the seating arrangement at least in part; (ii) determining, in particular estimating, a seat occupancy state of the seating arrangement using an evaluation model that, depending on the radar point cloud, provides one of several predefined possible seat occupancy states of the seating arrangement as an evaluation result; and (iii) outputting information defined depending on the evaluation result. The method according to the second aspect uses an evaluation model, in particular a machine learning model, which was trained using the method according to the first aspect.

[0034] Using the method according to the second aspect, an evaluation result characterizing the seat occupancy state of the seat arrangement (particularly in the sense of a prediction or classification) can be obtained based on a radar point cloud obtained by radar scanning of a spatial area surrounding the seat arrangement. This allows radar-based solutions to be implemented, particularly in the vehicle context (particularly for automobiles), which can reliably detect a seat occupancy state (particularly exclusively) using radar and, based on this, can activate, deactivate, or control / regulate certain functionalities or systems, such as a seat belt warning system or an airbag system, either entirely or selectively. Training with the augmented measurement data can improve the quality of the evaluation result when actually detecting a seat occupancy state, since a large amount of data can be used for training.

[0035] The information to be output can, in particular, represent the evaluation result itself. It can also be a detectable signal, particularly one that can be perceived by the human senses, such as a warning, a control signal for controlling a signal source, or a data signal carrying the information.

[0036] Various exemplary embodiments of the method according to the second aspect are described below. Each of these embodiments can be combined with one another and with the other described aspects of the present solution as desired, unless expressly excluded or technically impossible. Embodiments of the method according to the first aspect can also be applied to the method according to the second aspect, and vice versa. In other words, features of the method that are used in training the evaluation model can equally be applied in detecting a seat occupancy state, and vice versa.

[0037] In some embodiments, the seating arrangement has a plurality of seats for which a seat occupancy state is to be determined individually or cumulatively within the scope of the method. For each (individual) radar point cloud, the set of its radar points is divided into several clusters, each containing a subset of the radar points, by means of cluster formation depending on the respective spatial position of the radar points in relation to the seats, in order to individually assign to each of the seats a cluster that is spatially closest to it. The seat occupancy state of the seating arrangement is determined for each of the seats depending on the radar point cloud determined for the respectively associated cluster in order to obtain an evaluation result, in particular a classification result, characterizing a seat occupancy state of the respective seat.The information to be output is then defined depending on the respective individual evaluation results for the different seats.

[0038] In some associated embodiments, each radar point cloud is segmented into multiple clusters by assigning a subset of the radar points of the respective radar point cloud to each of the seats as a cluster, depending on their respective position, such that the radar points of the cluster are located in a defined, closed spatial area, in particular a cuboid, in the vicinity of the seat. This enables particularly simple and low-computational-intensive cluster formation and thus seat-related seat occupancy detection, wherein the location (position and orientation) and shape of the spatial area is or can be defined such that it strongly overlaps the spatial area generally occupied by a typical object to be detected, in particular a person, on a seat of the seating arrangement.

[0039] This enables seat-related, i.e. individual, seat occupancy detection for each seat in the case of a multi-seat seating arrangement, which is particularly advantageous or even necessary if a seat-related response to the detected seat occupancy is to be made, for example by activating or deactivating or otherwise controlling a specific functionality or system, such as a seat-related airbag system, a seat-related seat belt warning, or a seat-related seat heating system, for a specific seat depending on its detected seat occupancy. In some embodiments, cluster formation can be carried out in particular such that the clusters are disjoint, so that no radar point is assigned to two different clusters.

[0040] In some embodiments, the or each individual radar point cloud is segmented into a plurality of clusters by assigning a subset of the radar points to each of the seats as a cluster depending on their respective position, in particular uniquely for each radar point, such that the radar points of the cluster are located in a defined, closed, in particular cuboid-shaped, spatial area in the vicinity of the seat. The assignment can in particular be carried out such that each radar point is assigned to the cluster of the seat closest to it. In this way, the radar point cloud can be divided into clusters, i.e. subsets of the radar point cloud localized in the vicinity of the respective seats, so that seat-specific seat occupancy states can be determined in a targeted manner and therefore with a high degree of reliability on the basis of the cluster assigned to the respective seat.

[0041] In some embodiments, the method further comprises: determining the respective value of at least one defined parameter for characterizing radar point clouds. The evaluation model is or will be defined such that, when determining the seat occupancy state of the seating arrangement, the evaluation result is determined as a function of the respective values ​​of the at least one parameter for the radar point clouds. In this way, the evaluation model can be determined and applied in a simplified manner, since instead of entire radar point clouds, only the values ​​of the at least one parameter need to be taken into account as input variables. In particular, the temporal progression of the values ​​can thus reflect a movement or a specific movement pattern of one or more objects on the seating arrangement, such that the evaluation model can determine the seat occupancy state of the seating arrangement in a particularly reliable manner on this basis.

[0042] In some embodiments, for at least one of the parameters, the series of its respective values ​​is analyzed to determine whether a periodic pattern, in particular a pattern corresponding to a periodic breathing pattern, of the parameter is detected therein. The evaluation result is then determined depending on the result of the analysis. In particular, not only can the presence of any object on a seat of the seating arrangement be detected, but a distinction can even be made with high reliability between living objects present on the seating arrangement, in particular between people and mammals such as pets (e.g., dogs). In particular, it is possible to define the information to be output according to the method depending on the detection or non-detection of such a periodic pattern.In particular, one or more detected frequencies of the periodic curve can also be taken into account, in particular in such a way that the information is defined depending on whether the frequency(ies) lies within a specific frequency range, such as a typical respiratory rate range. In particular, a seat belt warning function or an airbag system can be controlled depending on whether a respiratory rate, and thus with a high probability a person on the seating arrangement or a specific seat, has been detected.

[0043] In some embodiments, the or one of the characteristic variables is or is determined by the number of radar points in the radar point cloud or a variable dependent thereon. If, during radar scanning, the number of radar points in the radar point cloud generated thereby depends on the extent to which the scanned object moves, the extent of the movement can be mapped in the characteristic variable(s), in particular with regard to the aforementioned detection of a breathing rate of the object. In addition to the number of radar points, other features of the point cloud (or of a cluster) can also be used as a characteristic variable, such as a position of the center of gravity of the point cloud (or of a cluster), the density of the points (of a cluster) and / or an average Doppler value.

[0044] In some embodiments, the evaluation model comprises a trained machine learning model. Data representing at least one radar point cloud or values ​​of one or more parameters determined for this purpose are provided to the machine learning model as input data in order to obtain the evaluation result as its output. This allows a particularly flexible and adaptable implementation of the evaluation model, whereby machine learning can be used to continuously improve the evaluation model and thus the quality and reliability of the seat occupancy detection. The evaluation result can, in particular, indicate a class of a seat occupancy state classification. The machine learning model can, in particular, be a decision tree-based model or a model based on an artificial neural network.As explained above in connection with the method according to the first aspect, the evaluation model is trained with training data before use in the method according to the second aspect, augmented data being used for this purpose in order to increase the amount of data that can be made available to the evaluation model as input data for training.

[0045] In some embodiments, outputting the information comprises controlling a signal source as a function of the information in order to cause the signal source to output a defined signal as a function of the control. The signal source can be, in particular, an audio source, an optical signal source, in particular a display device for images or text, and / or a haptic actuator, or a combination of at least two of the aforementioned signal sources. Thus, based on the signaling, the detected seat occupancy status can be communicated to a user or used to control another technical system, such as an airbag system.

[0046] In some of these embodiments, the signal source is controlled in dependence on the information such that it outputs a signal, in particular defined by the control, if the information results from an evaluation result, according to which at least one seat of the seat arrangement is occupied and / or a selected predetermined seat occupancy state exists.

[0047] In some embodiments, the method further comprises: (i) detecting a seat belt fastening state of at least one seat of the seat arrangement or receiving seat belt information characterizing this seat belt fastening state; (ii) wherein the signal source is controlled as a function of the seat belt information and the information from the evaluation result such that it outputs a seat belt fastening indication signal if, according to the information, at least one seat of the seat arrangement is occupied and / or a selected predetermined seat occupancy state exists and seat belt information indicates that the associated seat belt of the seat is not fastened. This makes it possible to achieve a radar-based seat belt fastening checking and warning system, in particular with regard to detection exclusively.

[0048] In some embodiments, the individual radar points of the radar point cloud are each represented by a position of the respective radar point in three-dimensional space and by at least one of the following parameters: (i) a Doppler shift value of the radar signal at the respective radar point; (iii) a signal-to-noise ratio value of the radar signal at the respective radar point. These parameters can be used, in particular, for pre-filtering the radar point cloud as part of a pre-processing step prior to feature determination.

[0049] In particular, in some of these embodiments, the determination of the seat occupancy status of the seat arrangement using the evaluation model can be carried out exclusively or at least predominantly on the basis of radar points whose Doppler shift value is at or above a predetermined shift threshold other than zero. Thus, only or at least predominantly so-called dynamic radar points are used as the basis for the evaluation, i.e. radar points that indicate a movement of the scanned object, whose Doppler shift value is at or above the shift threshold. This can be used in particular to further increase the quality, in particular the reliability of the method, because static, i.e.Essentially stationary points on the object surfaces, such as points on the surface of a seat, are not included in the analysis, or are included only in smaller numbers, than points that exhibit dynamic behavior and can therefore be attributed with a high degree of probability to a living being, in particular a person or an animal. This means that the information obtained and output by the analysis (e.g., for airbag control or a seatbelt warning system) can be used depending on whether a living being or a static object surface was detected.

[0050] A third aspect of the present solution relates to a system, in particular a data processing device, for the automated detection of a, in particular respective, seat occupancy state of a seating arrangement with at least one seat, in particular with at least one vehicle seat in or for a vehicle. The system comprises a data processing device configured, in particular by means of a corresponding computer program, to execute the method according to the second aspect for detecting the seat occupancy state.

[0051] A fourth aspect of the present solution relates to a computer program or computer program product comprising instructions which, when executed on the data processing device of the system according to the third aspect, cause the system to carry out the method according to the second aspect.

[0052] The computer program can in particular be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloaded via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network. In addition, the computer program can have a plurality of interacting individual program modules. The modules can in particular be configured or at least be usable in such a way that they can be used in the sense of distributed computing (DC).“Distributed computing” is carried out on different devices (computers or processor units) that are geographically separated from each other and connected via a data network.

[0053] The system according to the third aspect can accordingly comprise a program memory in which the computer program is stored. Alternatively, the system can also be configured to access an external computer program, for example, available on one or more servers or other data processing units, via a communication connection, in particular to exchange data with the program that is used during the execution of the method or computer program or that represents outputs of the computer program.

[0054] A fifth aspect of the present solution relates to a vehicle comprising: (i) a seating arrangement with at least one seat; (ii) a radar sensor for at least partially radar scanning the seating arrangement; and (iii) a system according to the third aspect for automatically detecting a, in particular respective, seat occupancy state of the seating arrangement as a function of an at least partially radar scan of the seating arrangement carried out by the radar sensor, in particular according to a method according to the second aspect.

[0055] The features and advantages explained with regard to the first and second aspects of the present solution also apply accordingly to the other aspects of the solution.

[0056] Further advantages, features and possible applications of this solution are evident from the following detailed description in conjunction with the drawings.

[0057] It shows: Fig. 1 schematically shows an exemplary embodiment of a vehicle equipped with a system for automatically detecting a seat occupancy state of a seat arrangement in the vehicle; Fig. 2 schematically shows the vehicle Fig. 1, where the passenger seat is occupied; Fig. 3A is an exemplary two-dimensional representation of a radar sensor of the vehicle Fig. 2 recorded radar point clouds; Fig. 3B shows an example of a clustering of the radar point cloud from Fig. 3A according to the positions of the individual seats in the seating arrangement; Fig. 4 is a flowchart illustrating an exemplary embodiment of a method for automatically detecting a seat occupancy state of a seat arrangement; Fig. 5 is a flowchart illustrating an exemplary embodiment of a method for training an evaluation model for the automated detection of a seat occupancy state of a seat arrangement; Fig. 6 an exemplary representation of four radar points during augmentation; and Fig. 7 an exemplary three-dimensional representation of a radar point cloud with an augmented point cloud obtained from it.

[0058] In the figures, like reference numerals designate like, similar, or corresponding elements. Elements shown in the figures are not necessarily drawn to scale. Rather, the various elements shown in the figures are depicted in such a way that their function and general purpose will be understood by those skilled in the art. Connections and couplings between functional units and elements shown in the figures can, unless expressly stated otherwise, also be implemented as indirect connections or couplings. Functional units can, in particular, be implemented as hardware, software, or a combination of hardware and software.

[0059] The Fig. An exemplary embodiment of a vehicle 100 schematically illustrated in Figure 1 comprises a seating arrangement 105 with five individual seats or seating locations 105a to 105e. Each of the seats 105a to 105e is suitable for accommodating a person as a passenger of the vehicle 100. The vehicle 100 further comprises a radar sensor 110 mounted within the vehicle cabin on the ceiling thereof and configured to scan the seating arrangement 105, at least substantially, using radar beams. Accordingly, the seats 105a to 105e, in particular their seating surfaces, are each located at least predominantly within an observation field 110a scannable by the radar sensor 110.In addition, the vehicle 100 has a system 115 for automatically detecting a seat occupancy state of the seat arrangement 105 as a function of a radar scan of the seat arrangement 105 carried out by the radar sensor 110, at least in sections with respect to the observation field 110a.

[0060] The system 115 comprises in particular a data processing unit 115a with at least one microprocessor and a memory 115b connected to the microprocessor, in which a memory 115b is stored for carrying out the method described below with reference to Fig. 4 for the automated detection of a seat occupancy state of the seat arrangement 105 is stored. Furthermore, the sensor data generated by the radar sensor 110 during radar scanning or information already obtained therefrom through further processing can be or will be stored in the memory 115b.

[0061] The Fig. The vehicle 100 shown in Figure 2 corresponds to the vehicle from Fig. 1, but here the passenger seat 105b is occupied by a person P. In the following discussion of the Fig. 3A and Fig. 3B will be based on the constellation Fig. 2 is referred to.

[0062] The following will now refer to the Fig. 3A and Fig. 3B, each of which represents a radar point cloud, wherein, for the purpose of representation, the respective, essentially three-dimensional radar point cloud was reduced to two dimensions by projecting the positions of the radar points of the radar point cloud onto a plane spanned by two of its dimensions.

[0063] In Fig. 3A illustrates an exemplary radar point cloud 305, as it was acquired as a result of a radar scan of the seating arrangement 105 by the radar sensor 110 during a defined time interval (measurement period). The position of the individual radar points within the radar point cloud 305 can be represented by spatial coordinates; for example, one can assign Cartesian coordinates X and Y to the drawing plane and, accordingly, to each individual point. In reality, if the dimensional reduction due to the drawing is ignored, a third coordinate Z for the third spatial dimension is also assigned (cf. the Fig. 7 shown coordinates X, Y and Z).

[0064] If, during radar scanning, not only the spatial positions of the points at which the radar beam is reflected by the scanned objects are recorded as coordinates, but also a respective Doppler shift is measured, then the individual radar points can be classified depending on the magnitude of this Doppler shift, in particular divided into two different classes. The latter can be achieved, for example, by comparing the Doppler shift with a predefined shift threshold corresponding to a specific shift velocity. Depending on the result of the comparison, those radar points 310 which, according to the value of their assigned Doppler shift, have no velocity or a velocity of the object surface at the reflection point that is below the displacement wave, can be classified as "static" radar points (in the Fig. 3A and Fig. 3B each with a filled black circle). Conversely, those radar points 315 that have a Doppler shift above the shift threshold can be classified as “dynamic” radar points 315 (in the Fig. 3A and Fig. 3B each shown with a black ring).

[0065] The classification of radar points 310 and 315 according to their Doppler shift is not mandatory, but it can be used to process radar point cloud 305, particularly as part of a preprocessing step prior to its evaluation, particularly to filter it depending on the classification. For example, such filtering could be performed in such a way that only dynamic radar points 315 are considered for evaluation, for example, to detect only moving objects.

[0066] In Fig. 3B is the same radar point cloud 305 as in Fig. 3A. In addition, however, cuboid-shaped (3D case) or, in the present 2D representation, rectangular, selected spatial regions 325a to 325e are shown here, which are spatially assigned to the respective location of the individual seats 105a to 105e. The definition of these spatial regions 325a to 325e can now be used to cluster the radar point cloud 305, whereby each radar point 310 or 315 is assigned, as far as possible, to the spatial region 325a to 325e in which it is located. All radar points not located in one of the spatial regions 325a to 325e can be disregarded in the following. In particular, it can be seen that the regions 320 with a particularly high radar point density are in the area of ​​the front passenger seat 105b, on which, according to Fig. 2 the person P is located.

[0067] Fig. 4 shows a flowchart illustrating an exemplary embodiment 400 of a method for automatically detecting a seat occupancy state of a seat arrangement. The method can be embodied, in particular, as a computer-implemented method. For this purpose, it can be stored, in particular, in the memory 115b of the system 115 as a computer program and be executable on the data processing unit 115a.

[0068] In the method 400, a radar point cloud 305 is acquired by receiving radar measurement data, in the present example from the radar sensor 110 of the vehicle 100, in a step 410 and further processing it to form one or more radar point clouds.

[0069] The radar point cloud 305 now present can then be clustered in a further process 420 by checking for each of its radar points whether it lies within one of the defined spatial areas 325a to 325e (cf. Fig. 3B) and, if applicable, in which one. Thus, each of all points can be assigned either to one of the spatial areas 325a to 325e or to the other observation field. All radar points that lie within the same spatial area 325a to 325e are combined into a respective cluster. As a result, each of the seats 105a to 105e is assigned a corresponding cluster of the radar point cloud 305. This forms the basis for an individual evaluation for each seat 105a to 105e as to whether the respective seat 105a to 105e is or was occupied or not while the radar point cloud 305 was being formed.

[0070] To facilitate the subsequent evaluation of the clustered radar point cloud 305, a corresponding characteristic K can be determined for each of the clusters in a process 430, wherein this characteristic K can be defined, in particular, as the number of radar points in the cluster. If no filtering according to the Doppler shift value has taken place, this can involve a joint count of both the static and the dynamic radar points 310 and 315, respectively. However, if the static radar points 310 were previously filtered out, this only involves a count of the dynamic radar points 315. Alternatively or additionally, the static radar points 310 can also be counted.

[0071] The evaluation of the characteristic K for the cluster for seat 105b can now take place (the same can be done analogously for the respective clusters for the other seats). For this purpose, in process 440, the characteristic K (or a temporal progression of the characteristic K) is made available as an input to an evaluation model. This can in particular be a model based on machine learning, such as an artificial neural network or a decision tree-based model (decision tree(s)). The training and, if applicable, validation data used for the preceding training can be structured such that they contain, according to the type of characteristic K, a plurality of different radar point clouds or clusters thereof, as well as an assigned correct class of a classification of possible seat occupancy states for each progression of K. The model can thus be trained and validated in the sense of supervised learning.In the simplest case, seat occupancy states indicate whether the seat is occupied or not. However, more sophisticated classifications are also conceivable, in which, in the case of the presence of an object, the respective class additionally indicates the type of object, for example, whether it is moving or stationary, and in the case of a moving object, in particular, whether it is a person (generally recognizable in particular based on a breathing pattern in the course of the parameter K).

[0072] If in process 440 a seat occupancy state for the seat arrangement 105 was determined on the basis of the evaluation model, in particular for one or more of its seats 105a to 105e individually, this result can be output as corresponding information in process 445, for example at a user interface of the vehicle or in the form of data for further processing by one or more other systems, in particular systems of the vehicle.

[0073] In the present example, this information is to be used in particular to check whether or not to issue a belt warning signal depending on the seat occupancy status of a respective seat 105a to 105e and the result of a check as to whether or not a corresponding seat belt has been fastened for this seat.

[0074] For this purpose, in process 450, it can be checked whether the seat belt for the relevant seat (here, for example, for seat 105b) is fastened, and in step 455, a functionality of the vehicle 100 can be controlled depending on the information on the seat occupancy state output in process 445 and the status of the seat belt determined in process 450. In particular, this can be done in such a way that in process 455, a signal source is activated to output a belt status signal, in particular an optical and / or acoustic one, in order to signal to one or more other occupants of the vehicle, if necessary, that a seat is occupied but the seat belt is not fastened there. The method then branches back to step 410 to start another loop run.

[0075] Fig. 5 shows a flowchart illustrating an exemplary embodiment 500 of a method for training an evaluation model, which can be used in the method 400 for automatically detecting a seat occupancy state of a seating arrangement. The method 500 is simultaneously described with reference to Fig. 6 using four exemplary points a, b, c and d.

[0076] First, in a step 510, measurement data are acquired which represent an associated radar point cloud, such as the radar point cloud 305. In Fig. To illustrate the method, four radar points a, b, c, and d are shown at the top of Figure 6. Anchor points are then determined from the set of radar points. For example, a first anchor point can be randomly determined, and all subsequent anchor points can be determined with the greatest distance from the previously determined anchor points. For example, 2, 3, 4, or 5 anchor points can be determined. The number of anchor points can vary depending on the application; in one example, four anchor points proved to be optimal. This can be determined by hyperparameter optimization, for example, Bayesian optimization. In the Fig. In the example shown in Figure 6, point a is selected as the anchor point. All points a, b, c, and d are then copied to obtain points a_a, b_a, c_a, and d_a.

[0077] In step 530, a random transformation of the points with respect to the anchor point a is performed. This includes a randomly selected rotation, a randomly selected scaling, and a randomly selected translation. This results in the points a_a_aug, b_a_aug, c_a_aug, and d_a_aug. The extent of each transformation can also be determined by hyperparameter optimization. After the transformation, smoothing is performed, for example, using a Gaussian kernel.

[0078] This ensures that points farther from the anchor point undergo a smaller transformation than points closer to the anchor point. Appropriate weights can be selected for this purpose. This results in the points a_a_aug_smooth, b_a_aug_smooth, c_a_aug_smooth, and d_a_aug_smooth.

[0079] The transformation and smoothing is repeated for each of the anchor points. For example, for anchor point c, the points a_c_aug_smooth, b_c_aug_smooth, c_c_aug_smooth, and d_c_aug_smooth would result. In step 550, all transformed and smoothed point clouds are then combined. This can be done, for example, by forming a point-wise average, for example, a_aug = (a_a_aug_smooth + a_c_aug_smooth) / 2, etc., to obtain points a_aug, b_aug, c_aug, and d_aug (not shown in Fig. 6).

[0080] The obtained augmented points a_aug, b_aug, c_aug, and d_aug are now adjusted in step 560 using a difficulty parameter α. Here, the obtained augmented points are coupled to the original points a, b, c, and d so that the resulting augmented point cloud does not deviate too significantly from the original radar point cloud. A result can be obtained by a_result = a_aug * α + a * (1-α), as well as for b_result, c_result, and d_result. In this way, a machine learning model can be obtained that, through training with the augmented measurement data, delivers high accuracy in detecting a seat occupancy state.

[0081] Since the positions of the obtained points a_result, b_result, c_result and d_result may differ from the positions of the original points a, b, c and d, the augmented point cloud is clustered (step 570) in order to then obtain suitable input data for training the evaluation model (step 580).

[0082] In Fig. Finally, Figure 7 shows another example of a radar point cloud 705. The illustration shows the radar point cloud 705 as a three-dimensional point cloud with the coordinates X, Y and Z. The view corresponds to a seat occupancy such as that in Fig. 2 shown seat occupancy with one person P in the front passenger seat. This is in Fig. 7 is recognizable by the concentration of radar points 715 in the upper right corner (which corresponds to the front right corner of the vehicle). The majority of the points in 715 are therefore located in a cluster associated with the passenger seat. Furthermore, Fig. 7 an augmented point cloud 710 with the points 720, which is represented by unfilled rings.

[0083] It can be seen that the augmented point cloud 710 deviates slightly from the actual radar point cloud 705. In particular, larger deviations are found within the cluster of the passenger seat, whereas the deviations in the edge areas are smaller (in the illustration, the points 720 of the augmented point cloud 710 are mostly covered by the points 715 of the radar point cloud 720 and are therefore in Fig. 7 not visible).

[0084] While at least one exemplary embodiment has been described above, it should be appreciated that a wide variety of variations exist. It should also be understood that the described exemplary embodiments are merely non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description will provide a guide to implementing at least one exemplary embodiment, with the understanding that various changes in the operation and arrangement of the elements described in an exemplary embodiment may be made without departing from the subject matter as defined in the appended claims, as well as their legal equivalents. LIST OF REFERENCE SYMBOLS P Person in the passenger seat 100 vehicles 105 Seating arrangement 105a-e Seats or seating positions 110 radar sensor 110a Observation field of radar sensor 110 115 System for automated detection of seat occupancy status 115a Data processing unit 115b memory 305 radar point cloud 310 static radar points 315 dynamic radar points 320 areas of the radar point cloud 305 with high radar point density 325a-e Spatial areas for cluster definition 400 Method for the automated detection of a seat occupancy status 410-455 individual processes or procedural steps within the framework of procedure 400 500 methods for training an evaluation model 510-580 individual processes or procedural steps within the framework of procedure 500 705 radar point cloud 710 augmented point clouds 715 point of the radar point cloud 705 720 point of the augmented point cloud 710

Claims

[1] Method for training an evaluation model for automated recognition of a seat occupancy state of a seating arrangement (105) with at least one seat (105a-e), the method comprising: - Acquiring measurement data representing an associated radar point cloud (305), wherein the radar point cloud (305) was or is obtained on the basis of a radar scan of a spatial region surrounding the seat arrangement (105) at least in sections and is associated with one of several predefined possible seat occupancy states of the seat arrangement (105); - generating augmented measurement data representing an associated augmented point cloud from the measurement data associated with the radar point cloud (305), wherein generating the augmented measurement data comprises: - Determining multiple anchor points within the spatial area; - determining a transformation with respect to each of the anchor points, wherein each of the transformations is applied to at least a portion of the points of the radar point cloud (305) to obtain a transformed point cloud associated with the respective anchor point; and - Merging the transformed point clouds associated with the anchor points to obtain the augmented point cloud associated with the augmented measurement data; and - generating training data from the measurement data and the augmented measurement data, wherein the training data are made available to the evaluation model as input data in order to obtain an evaluation result as its output, which is assigned to the seat occupancy state of the seat arrangement (105). [2] Method according to claim 1, wherein the anchor points are points from the radar point cloud (305) associated with the measurement data. [3] Method according to claim 1 or 2, wherein the anchor points are determined such that each of the anchor points within the spatial area has a greatest distance from the respective remaining anchor points. [4] Method according to one of the preceding claims, wherein each of the transformations comprises at least one of rotation, scaling and translation with respect to the respective anchor point. [5] Method according to one of the preceding claims, wherein each of the transformations is weighted point by point with a weighting factor, wherein the weighting factor depends in each case on a distance between a point of the radar point cloud to be transformed and the respective anchor point. [6] Method according to one of the preceding claims, wherein the merging of the transformed point clouds assigned to the anchor points is carried out by forming a point-wise average of the transformed point clouds assigned to the anchor points. [7] Method according to one of the preceding claims, wherein the training data are generated by a weighted sum of the measurement data and the augmented measurement data. [8] Method according to one of the preceding claims, wherein: the seating arrangement (105) comprises a plurality of seats (105a-e), wherein the seat occupancy state is an individual or cumulative seat occupancy state of the seats (105a-e); and for the radar point cloud (305), the set of its radar points (310, 315) is divided into a plurality of clusters, each containing a subset of the radar points (310, 315), by means of cluster formation depending on the respective spatial position of the radar points (310, 315) in relation to the seats (105a-e), in order to individually assign to each of the seats (105a-e) a cluster that is spatially closest to it, wherein the augmented measurement data are generated individually for each cluster. [9] Method according to claim 8, wherein the radar point cloud (305) is segmented into a plurality of clusters by assigning to each of the seats (105a-e) as a cluster a subset of the radar points (310, 315) of the respective radar point cloud (305) depending on their respective position such that the radar points (310, 315) of the cluster lie in a defined closed spatial area in the vicinity of the seat (105a-e). [10] Method according to one of the preceding claims, wherein: the seating arrangement (105) comprises a plurality of seats (105a-e), wherein the seat occupancy state is an individual or cumulative seat occupancy state of the seats (105a-e); and for the augmented point cloud (305), the set of its points (310, 315) is divided into several clusters, each containing a subset of the points (310, 315), by means of cluster formation depending on the respective spatial position of the points (310, 315) in relation to the seats (105a-e), in order to individually assign to each of the seats (105a-e) a cluster that is spatially closest to it. [11] Method according to one of the preceding claims, wherein the individual radar points (310, 315) of each radar point cloud (305) are each represented by a position of the respective radar point in three-dimensional space and by at least one of the following parameters: - A Doppler shift value of the radar signal at the respective radar point; - A signal-to-noise ratio value of the radar signal to the respective radar point. [12] Method for the automated detection of a seat occupancy state of a seat arrangement (105) with at least one seat (105a-e), the method comprising: - Acquiring measurement data representing an associated radar point cloud (305), wherein each radar point cloud (305) was or is obtained on the basis of a radar scan of a spatial region surrounding the seat arrangement (105) at least in sections; - Determining a seat occupancy state of the seat arrangement (105) based on an evaluation model which was created according to a method according to one of claims 1 to 11 and which, depending on the radar point cloud, provides one of several predefined possible seat occupancy states of the seat arrangement (105) as an evaluation result; and - Output of information defined depending on the evaluation result. [13] Method according to claim 12, wherein the outputting of the information comprises controlling a signal source in dependence on the information in order to cause the signal source to output a defined signal in dependence on the control, wherein the signal source is controlled in dependence on the information in such a way that it outputs a signal when the information results from an evaluation result, according to which at least one seat of the seat arrangement (105) is occupied and / or a selected predetermined seat occupancy state exists. [14] The method of claim 13, further comprising: Detecting a seat belt fastening state of at least one seat of the seat arrangement (105) or receiving seat belt information characterizing this seat belt fastening state; wherein the signal source is controlled in dependence on the seat belt information and the information from the evaluation result such that it outputs a seat belt fastening indication signal if, according to the information, at least one seat of the seat arrangement (105) is occupied and / or a selected predetermined seat occupancy state exists and seat belt information indicates that the associated seat belt of the seat is not fastened. [15] System (115) for the automated detection of a seat occupancy state of a seat arrangement (105) having at least one seat (105a-e), wherein the system (115) comprises a data processing device configured to carry out the method according to one of claims 12 to 14 for detecting the seat occupancy state. [16] A computer program or computer program product comprising instructions which, when executed on the data processing device of the system (115) according to claim 15, cause the system (115) to carry out the method according to any one of claims 12 to 14. [17] Vehicle (100), comprising: a seating arrangement (105) with at least one seat (105a-e); a radar sensor (110) for at least partially scanning the seat arrangement (105); and a system (115) according to claim 15 for the automated detection of a seat occupancy state of the seat arrangement (105) as a function of an at least section-wise radar scan of the seat arrangement (105) carried out by the radar sensor (110).

Images