Method for issuing an action signal to an occupant in a motor vehicle

Abstract

The invention relates to a method for outputting an action signal (HS) to an occupant in a motor vehicle, wherein the action signal (HS) is generated depending on an occupant state (IZ) determined by means of occupant parameters (IP). An occupant parameter (IP) is assigned a total confidence value (GC), which represents the sum of weighted individual confidence values ​​(EV), wherein the individual confidence values ​​(EV) are generated by processing an image data stream via a processing path (VP). Each processing path (VP) is trained to process a training image data stream. A training acquisition position (TP) is characterized by a position-specific camera perspective, whereby each processing path (VP) and a training acquisition position (TP) are mutually assigned.The image data stream is assigned to a processing path (VP) depending on the spatial distance between its acquisition position (EP) and a training acquisition position (TP). The individual confidences (EK) generated for each processing path (VP) are weighted with at least one processing path-specific relevance (R), which is determined depending on the spatial distance between the training acquisition position (TP) and the acquisition position (EP).

Description

[0001] The invention relates to a method for outputting an action signal to an occupant in a motor vehicle, wherein the action signal is generated as a function of an occupant state which is determined by means of or as a function of occupant parameters, wherein a respective occupant parameter is additionally assigned an overall confidence value which represents the sum of weighted individual confidences, wherein the individual confidences related to the respective occupant parameter are each generated by processing an image data stream through a processing path and, for example, quantify how reliably the occupant parameter could be detected.

[0002] In order to adapt a vehicle and its systems to vehicle occupants, attempts are made nowadays to determine the condition of the occupants, in particular the driver. Such an occupant condition can, for example, describe a physical condition such as tiredness or the direction of gaze of an occupant. As an example of an occupant parameter that can be used to determine an occupant condition, the blink frequency or the duration of an eyelid blink indicates whether an occupant, in particular the driver, is tired. An occupant condition that can be described as inattentive can, for example, be determined based on the direction of gaze of the driver. The occupant parameter here represents the direction of gaze or a frequently changing direction of gaze pointing away from the road or deviating from it. Likewise, an occupant parameter can be the position of extremities, such as hands or legs, for example of a front passenger.For example, the passenger placed his feet on the dashboard on an airbag.

[0003] Depending on or based on the determined current occupant condition, the action signal can include a recommendation to change or improve the occupant's condition. For example, if fatigue is detected, the action signal can include a recommendation to take a break. The action signal itself can be a request to one or more occupants, for example a request to correctly fasten a seat belt. Incorrect buckling can mean, for example, that the belt is in the buckle but the occupant is sitting in front of it. This means that the belt is behind the occupant's back and therefore weight sensors in the seat in conjunction with belt buckle sensors alone do not detect that the occupant is not properly buckled in.Likewise, the action signal can issue a warning, especially to the driver, if it has been determined that the driver is tired, for example. Additionally or alternatively, the action signal can prompt the driver to take a break if they are too emotionally aroused, for example, if they are upset or angry.

[0004] US 2021 / 0 056 306 A1 describes the determination of a viewing direction of a vehicle occupant independently of the detection angle of detection sensors using a neural network.

[0005] US 2019 / 0 206 084 A1 describes the calculation of a three-dimensional position and orientation of a camera in the interior of a vehicle using image data.

[0006] US 2023 / 0 244 941 A1 shows systems and methods for determining a viewing direction and for projecting this viewing direction onto an area of ​​a viewed object.

[0007] The publication "A Neural Network-Based Driver Gaze Classification System with Vehicle Signals" by S. Dari, N. Kadrileev, and E. Hüllermeier presents a method for detecting a driver's attention directed toward traffic based on their eye and head posture. This method involves evaluating video or image material captured by in-vehicle cameras using neural networks. The goal is to redirect the driver's attention toward the traffic in critical traffic situations.

[0008] An occupant state can be detected, for example, by feeding a respective image data stream from one or more cameras to a respective processing path for image data processing, where, for example, an artificial neural network or, more generally, a machine learning model maps image information from the image data stream to one of several possible occupant states. Such a machine learning model must be trained with a training image data stream.

[0009] A dependency identified by the inventors as crucial lies in the camera perspective of the camera that generated the training image data stream and the camera perspective of the available camera or cameras in a specific vehicle in which the occupant status is to be detected. The greater the deviation, the more error-prone the detection of an occupant status is. Typical camera perspectives or corresponding camera installation positions are available, on the one hand, in the instrument cluster behind the steering wheel (from the driver's perspective) and / or, on the other hand, in or on the rearview mirror, from where a camera can be directed specifically at the driver. However, it has been shown that differences in camera installation positions can arise between vehicle models and configuration variants within the vehicle models, which can result in variability in the range of up to 50 cm (e.g., left on the rearview mirror or right on the instrument cluster).For each installation region (e.g., instrument cluster and / or rearview mirror), there are perspective groups with multiple possible camera perspectives, which currently requires a correspondingly large number of training image data streams (namely, for all potentially possible installation positions and, in the case of multiple cameras, installation position combinations) to be made available when training the machine learning models. This leads to an undesirably high effort.

[0010] The invention is based on the object of issuing an action signal to an occupant in a motor vehicle depending on his and / or an occupant's condition.

[0011] The problem is solved by the subject matter of the independent patent claims. Advantageous developments of the invention are described by the dependent patent claims, the following description, and the figures.

[0012] The invention provides a method for outputting an action signal to an occupant in a motor vehicle, wherein the action signal is generated as a function of an occupant state which is determined by means of occupant parameters.

[0013] The invention provides a method which assumes that at least one occupant parameter (e.g., the viewing direction) is determined by processing a respective image data stream through a respective processing path. Each processing path is provided trained to process a respective training image data stream that has been acquired at only one of several predetermined training acquisition positions. A training acquisition position is characterized by a position-specific camera perspective, whereby each processing path and a respective training acquisition position are mutually associated. The method can now take into account that the currently used camera is not necessarily mounted at a training acquisition position, e.g., because this was not available in a given vehicle.

[0014] In order to compensate for the fact that the image data stream is not captured exactly from such a training capture position, but e.g. from between two such training capture positions of two processing paths, it is assigned to at least one processing path for processing depending on a spatial distance between its actual capture position and at least one training capture position.

[0015] The image data stream can thus be provided by a camera device whose actual capture position does not match, coincide, or is identical to any of the predetermined training capture positions. For example, if multiple processing paths are available, the image data stream can be fed to at least one or more processing paths whose training capture positions are closest to, or are closest to, the actual capture position.

[0016] Here, any distortion of the detection result, such as the deviation from the "ideal" detection positions (i.e., training detection positions) to the actual detection positions, should be taken into account. The method takes advantage of the fact that a processing path can signal not only the occupant state itself, but also a respective confidence value, such as that available from a machine learning model. This confidence value is referred to below as the "individual confidence."

[0017] In other words, an individual confidence, which indicates the certainty with which an occupant parameter is detected, is weighted or relativized when evaluating the image data stream by a processing path by the extent to which the actual detection position corresponds to one or more training detection positions of the processing paths to which the image data stream is fed and which generate the respective individual confidences.

[0018] Each detected occupant parameter is assigned an overall confidence value, which represents the sum of weighted individual confidences from the processing paths used for detection. The individual confidences relate to the respective occupant parameter in the manner described and are each generated by processing a respective image data stream through a respective processing path.

[0019] The overall confidence value describes how safely or reliably a respective occupant parameter is determined overall. The overall confidence value, in turn, is composed of a sum of weighted individual confidences. The individual confidences themselves also represent a reliability or degree of certainty with which an occupant parameter was detected. Individual confidences that are summed to form an overall confidence value relating to a respective occupant parameter refer to the same occupant parameter (e.g., "gaze direction") as the overall confidence value resulting from the summation. The individual confidences are generated by processing, assessing, and / or evaluating an image data stream through a processing path. The image data stream can be provided by a camera device or an optical recognition device, arranged, for example, in the vehicle interior.A processing path to which the image data stream is fed can be specified for the processing or evaluation with regard to the recognition of one occupant parameter at a time.

[0020] Each processing path is trained to process a training image data stream acquired at only one of several predetermined training acquisition positions. Each training acquisition position is characterized by a position-specific camera perspective, whereby each processing path and a respective training acquisition position are mutually associated. This specialized training generates processing paths that can distinguish particularly sharply between the different occupant states because there is no "blurring" due to different camera positions during training.

[0021] The said current image data stream can represent or comprise current or currently provided by, for example, the camera device. In contrast, the training image data stream represents image material or moving image material with which each processing path is configured or trained. Only after this training has been completed is the image data stream comprising current image data fed to a processing path. Each processing path is trained with a training image data stream to which a predetermined training acquisition position is assigned. As a result, each processing path is assigned a training acquisition position. A training acquisition position can be a mounting position of the camera device in the vehicle and / or a perspective in or onto the vehicle interior. Thus, each processing path can be assigned a position-specific camera perspective.

[0022] The individual confidences generated for each processing path are weighted with at least one processing path-specific relevance, which is determined depending on the spatial distance between the respective training acquisition position and the acquisition position.

[0023] In summary, an action signal should be issued to an occupant when an occupant state associated with the action signal is detected. The occupant state, in turn, is determined based on one or more occupant parameters, the respective accuracy of which is determined or evaluated based on their respective overall confidence values. This respective overall confidence value for each occupant parameter is composed of the summation of individual confidences. The individual confidences are generated by evaluating one or more image streams through respective processing paths.The certainty with which the respective individual confidences indicate the accuracy of an occupant parameter is relativized by how closely the actually mounted camera device in the vehicle interior, i.e. its actual detection position, corresponds to at least one or more training detection positions or comes close to those with which the respective processing paths were configured.

[0024] This has the advantage that the processing paths do not have to be configured with each actual acquisition position or to the perspective of each camera device actually installed in the vehicle, since the relevance takes into account the distance to the configuration positions or training acquisition positions and thus the certainty of the individual confidences can be relativized.

[0025] The action signal can be issued visually or acoustically to one or more occupants in the motor vehicle. For this purpose, a display device such as a screen, for example an LCD (LCD: Liquid Crystal Display) and / or an LED display (LED: Light Emitting Diode) and / or an OLED display (OLED: Organic Light Emitting Diode) and / or a TFT display (TFT: Thin Film Transistor) in the motor vehicle and / or an acoustic reproduction device such as a loudspeaker can be used. The action signal can be directed at each occupant in the motor vehicle individually or at several occupants at the same time. For this purpose, an occupant can be addressed by the action signal, for example, if the action signal includes their seat number or indicates a seating position from the driver’s perspective or the direction of vehicle travel, for example as driver or front passenger or rear right or rear left occupant.The action signal can be selected based on the occupant state, for example, using a lookup table, which can be included in a data storage device included in the motor vehicle, and then output. The occupant state can be determined based on occupant parameters. Determined occupant parameters can be assigned to an occupant state in an assignment unit, for example, via a lookup table. The occupant state, in turn, is then assigned to an action signal to be output in another assignment unit.

[0026] It has proven particularly advantageous to assign an image data stream and thus the capture position assigned to it or the image stream assigned to a capture position to a perspective group. A perspective group can relate to a viewing angle or a perspective on an occupant, for example a perspective on the front passenger or on the driver, or to a specific perspective on one occupant at a time, such as from the front of the driver or from the side from the left or from the side from the right of the driver. A perspective group can therefore be assigned to a location in the motor vehicle interior; for example, a perspective group can relate to the steering wheel, in which case, for example, an occupant, in particular the driver, can be captured from the front. A location can have or comprise one or more training capture positions and / or capture positions.Another possible location or perspective group could, for example, refer to the rearview mirror.

[0027] Depending on their distance from training acquisition positions, acquisition positions are assigned to the processing paths associated with them, which are assigned to a perspective group, i.e. a acquisition location such as the steering wheel area or the rearview mirror.

[0028] According to the invention, the training acquisition positions and acquisition positions are each assigned to a perspective group classifying the camera perspective depending on their respective acquisition range, wherein the individual confidences assigned to a perspective group are weighted with a perspective group-specific weighting.

[0029] For example, if one or up to three training recording positions are in the area of ​​the steering wheel and one or up to three training recording positions are in the area of ​​the rearview mirror, a recording position that does not correspond to any training recording position can be assigned to the "steering wheel" perspective group if its direct distance to the respective training recording positions of the "steering wheel" perspective group is less than the direct distance or sum of the direct distances to the training recording positions of the rearview mirror perspective group. It can be provided that a recording position is only assigned to a single perspective group. For example, it can be provided that a recording position with the same direct spatial distance to training recording positions of two or more perspective groups is assigned to the perspective group that is defined as the dominant perspective group.This group can be called a master group. As already mentioned, a perspective group can comprise at least one or more training acquisition positions that are located, for example, on the same vehicle component, e.g., the instrument cluster or the rearview mirror. As already mentioned, each training acquisition position is assigned a processing path and vice versa. Each processing path that generates an individual confidence is assigned its relevance. However, a relevance can also be assigned to a processing path that is not fed with an image data stream and that does not generate an individual confidence. This means that the relevancies of the respective processing paths can be determined before it is known whether an image data stream is fed to the processing paths in a current configuration. The relevance can be a numerical value between 0 or 0% and 1 or 100%. An individual confidence is thus weighted with the processing path-specific relevance, e.g.multiplied. In addition to the relevance, a perspective group-specific weighting can be provided. This perspective group-specific weighting can represent a numerical value between 0 or 0% and 1 or 100%. This results in the advantage that individual confidences can be weighted with a further weighting in addition to a relevance. In particular, this has the advantage that several individual confidences generated by processing paths that are each assigned to the same perspective group can be weighted overall depending on the perspective group. Thus, a perspective group-specific characteristic that arises, for example, from the perspective or the construction location, can be taken into account by the weighting for all processing paths it encompasses, each of which is related to a training acquisition position.It can be provided that only one detection position or only one actually installed camera device is available or assigned to each perspective group.

[0030] The invention also includes further developments which result in additional advantages.

[0031] A further development of the invention provides that the respective occupant parameter is only included in the determination if the total confidence value assigned to the occupant parameter is above a predetermined threshold value.

[0032] The individual confidences, as well as the overall confidence value (where multiple overall confidence values ​​may be present), can be specified, for example, as numerical values ​​between 0 and 1 or as percentages between 0 and 100%. A value of 1 or 100% corresponds to a high degree of certainty with which an occupant parameter was detected. A numerical value of 0 or 0% can indicate that an occupant parameter is not detected. Before an occupant parameter with a respective overall confidence value can be used to determine an occupant state, the invention can provide that the overall confidence value must be greater than a predetermined threshold value. This threshold value can, for example, be between 0.6 or 60% and 1.0 or 100%. The expert can determine the exact threshold value in the context of test series based on the specifically specified configuration of detection positions and training detection positions.By means of a target / actual comparison, which can be carried out, for example, by a comparison unit that can be provided on a data storage device in the motor vehicle and / or on a backend server, it can be compared or checked whether an overall confidence value is greater than or at least equal to the threshold value set or specified for it. Only in this case, i.e. when the overall confidence value is equal to or greater than the respective threshold value, can an occupant parameter assigned to the overall confidence value be taken into account in the further process. This has the advantage that an incorrect occupant state is detected. This, in turn, has the advantage that no incorrect action signal is issued to the occupant or one of the occupants.

[0033] A further development of the invention provides that the perspective group-specific weighting is assigned dynamically during the method depending on an evaluation of an image quality of an image data stream which is supplied to this perspective group.

[0034] In order to take into account a temporary, i.e. non-permanent, peculiarity of this camera device or capture position, the perspective group-specific weighting can be adjusted or changed during the process, i.e. "during operation". Such a temporary or time-related peculiarity can, for example, be a deterioration or change in the image quality of the camera device. Such deterioration can be caused, for example, by an insect or dirt on a lens of the camera device. Additionally or alternatively, deterioration can be caused by dirt accumulating on the lens over the period of operation. For example, if a lens of the camera device is dirty and this impairs the image quality of the camera device, the perspective group-specific weighting to which the camera device is assigned can be reduced.The perspective group-specific weighting can be proportional to the occlusion of the lens or a field of view or detection of the camera device. For example, if the field of view of the camera device is reduced by 50% due to an obstacle (insect) or dirt, a correction factor or factor for the perspective group-specific weighting can be 0.5. This means that the perspective group-specific weighting is multiplied by a factor of 0.5. As long as contamination of the camera lens is detected, this reduced weighting can remain. If the contamination has been removed, the perspective group-specific weighting can be reset to the previously set value. The detection of contamination or a deterioration in image quality can be achieved using methods described in the prior art.The sum of all perspective-group-specific weightings can be 1 or 100%. If a perspective-group-specific weighting is dynamically reduced during the process due to a deterioration in image quality, the sum of all perspective-group-specific weightings can be less than 1 or less than 100%. The sum of all relevancy values ​​assigned to a perspective group can be 1 or 100%.

[0035] A further development of the invention provides that the relevance of a processing path that processes an image data stream is determined as a function of the number of processing paths that process this image stream and the distance of a capture position of the image data stream to the respective training capture positions of the processing paths, wherein the sum of all relevancies per perspective group is normalized to 1.

[0036] The dependence on the number of processing paths can mean that the sum of all relevancy per perspective group equals 1 or 100%. The relevance of a processing path can be determined linearly with the direct distance of a capture position from the training capture position to which a processing path is assigned. If, for example, a perspective group only includes one training capture position, the relevance can be determined linearly using a quotient whose denominator is formed from the difference between a specified maximum distance and the direct distance of the capture position from the training capture position, and whose denominator is the specified maximum distance.The specified maximum distance can preferably be selected such that, if the acquisition position were located at the maximum distance from the training acquisition position of the first perspective group, the acquisition position would be assigned to another perspective group, for example, a second perspective group. The expert determines the specific value of the maximum distance based on a specific configuration, for example, through test series. If the acquisition position is located, for example, between two training acquisition positions of a perspective group, the relevance can be determined by linear interpolation depending on the distances to the respective training acquisition positions compared to the total distance to both training acquisition positions.

[0037] In general, the term "processing path" refers to the software or program module in which the processing of the image data stream is implemented. A further development of the invention provides that each processing path includes an artificial intelligence, a machine learning algorithm, a neural network, a deterministic image processing algorithm, and / or a statistical image processing algorithm. Such algorithms can advantageously provide a single confidence level.

[0038] Depending on the existing configuration and complexity, the person skilled in the art selects the most advantageous method or process or tool from the prior art in order to efficiently implement the process according to the invention.

[0039] For use cases or application situations that may arise during the method and which are not explicitly described here, it may be provided that, in accordance with the method, an error message and / or a request to enter user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0040] As a further solution, the invention also encompasses a computer-readable storage medium comprising program code which, when executed by a computer or computer network, causes the computer to carry out an embodiment of the method according to the invention. The storage medium can be provided at least partially as a non-volatile data memory (e.g. as a flash memory and / or as an SSD - solid state drive) and / or at least partially as a volatile data memory (e.g. as a RAM - random access memory). The storage medium can be arranged in the computer or computer network. However, the storage medium can also be operated on the Internet, for example, as a so-called app store server and / or cloud server. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor.The program code may be provided as binary code and / or as assembly code and / or as source code of a programming language (e.g. C) and / or as a program script (e.g. Python).

[0041] The invention also encompasses combinations of the features of the described embodiments. The invention therefore also encompasses implementations that each comprise a combination of the features of several of the described embodiments, unless the embodiments are described as mutually exclusive.

[0042] Exemplary embodiments of the invention are described below. Shown are: Fig. 1 schematic structure of a first perspective group and processing of their individual confidences and contribution of their individual confidences to an overall confidence value of an occupant parameter; Fig. 2 process steps from an overall confidence value of an occupant parameter to the output of an action signal.

[0043] The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that can be considered independently of one another, each of which also develops the invention independently of one another. Therefore, the disclosure is intended to encompass combinations of the features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0044] In the figures, the same reference symbols designate elements with the same function.

[0045] Fig. Figure 1 shows a first perspective group and its influence on an overall confidence value GK of a first occupant parameter IP1. A symmetry line SL in Fig. 1 can represent that this first perspective group PG1 can be mirrored and then represent a second perspective group PG2 with the same structure as the first perspective group. The first perspective group PG1 can comprise two processing paths VP1,1 and VP2,1. The first number here and in further abbreviations indicates the number of the respective unit, i.e. in this case a processing path, and the second number indicates the number of the perspective group to which the unit is assigned. These can be trained with the training image data streams from the training acquisition positions TP1,1 and TP2,1. In addition, this first perspective group PG1 can have a camera device KE1. This camera device KE1 can be located at an acquisition position EP1. As shown, the acquisition position EP1 can lie between the training acquisition positions TP1,1 and TP2,1.The image data stream provided by KE1 can be provided to both processing paths VP1,1 and VP2,1 and processed by each of them. This allows an individual confidence to be generated for each processing path: EK1,1 and EK2,1. These individual confidences can be weighted with the processing path-specific relevancies R1,1 and R2,1. In this exemplary embodiment, the individual confidence EK1,1 can be 0.8, and the individual confidence EK2,1 of the second processing path VP2,1 can have a value of 0.6. The relevance of the first processing path R1,1 can be 0.7, and the relevance of the second processing path VP2,1, R2,1 can be 0.3. The two relevancies of perspective group 1 PG1 can add up to the value 1. The respective individual confidences EK1,1 and EK2,1 are weighted with their respective relevances R1,1 and R2,1, in this example multiplied.After weighting, a remaining confidence of 0.56 can result on the left side of perspective group 1 PG1, since the relevance R1,1 of the first processing path VP1,1 was multiplied by the individual confidence of the first processing path EK1,1 in this case. For the second processing path VP2,1, a confidence of 0.18 can result after weighting with the second relevance R2,1. The remaining confidences of the respective processing paths VP1,1 and VP2,1, weighted by the relevances, can then be summed, in . Fig. 1 represented by the circled plus sign. The remaining weighting can thus be 0.74. This remaining confidence can be weighted with a perspective group-specific weighting PGW1, which can mean that it is multiplied by this. If PGW1 is 0.9 in this example, a confidence of 0.67, i.e. 0.74 times 0.9, is included in the overall confidence value GK from the first perspective group PG1. This confidence value indicates the certainty with which the first perspective group PG1 can recognize an occupant parameter. The perspective group-specific weightings - in this case PGW1 with the value 0.9 - and a perspective group-specific weighting of a second perspective group PG2 (structured the same as the first perspective group PG1, but for example mirrored along the symmetry line SL to the perspective group PG2) can add up to 1 in this exemplary embodiment.This results in a value of 0.1 for the perspective group-specific weighting PGW2. Based on the perspective group-specific weightings, a perspective group with a higher or highest weighting, in this example PG1, can be designated as the master group, and the perspective group with a lower or lowest weighting, here PG2, can be designated as the slave group. In this exemplary embodiment, all weightings and confidences can be within a value range of 0.0 and 1.0. Thus, the overall confidence value GK can only result in a value between 0 and 1. The image data stream of the camera device KE1 can be tapped by an image quality unit BQ and checked for its image quality.If, for example, the image quality is impaired due to an obscuration of the lens, for example by an insect on the lens or dirt, the perspective group-specific weighting PGW1 can be temporarily changed using a correction factor KF1. The correction factor, in this case for perspective group 1, KF1, can be between a value of 0.0 and 1. The correction factor KF1 can, for example, be used to change the perspective group-specific weighting PGW1, i.e. to multiply it by the factor KF1, in this case by the perspective group-specific weighting PGW1. The relevancies R1,1 and R2,1 of the two processing paths can be determined as a function of the distance L1 between the two training acquisition positions TP1,1 and TP2,1 and the respective distance D1,1 of the acquisition position EP1 to the separation acquisition position 1,1 and the distance D2,1 to the training acquisition position TP2,1.For example, linear interpolation can be used. Thus, in this example, the relevance R1,1 of the first processing path VP1,1 can be calculated as the quotient of the difference between L1 and D2,1 divided by the total distance L1 between the two training acquisition positions TP1,1 and TP2,1. The relevance of the second processing path R2,1 can be determined analogously. R2,1 can be calculated as the quotient of the difference between L1 and D1,1 in the denominator divided by the total distance L1.

[0046] The Fig. The structure of perspective group 1 shown in Figure 1 and, mirrored, perspective group 2 can be multiplied with an image data stream from camera device KE1 for different occupant parameters. This means that, with the same structure, processing paths trained to evaluate additional occupant parameters can be used. The certainty with which these additional occupant parameters are correct can be indicated by additional individual confidence values ​​and overall confidence values.

[0047] Fig. Figure 2 shows the processing steps that follow the determination of an overall confidence value GK for an occupant parameter. An occupant parameter can be, for example, an eyelid blink frequency, the duration of an eyelid blink, or the frequency of a head nod, whereby these occupant parameters can indicate an occupant condition, such as fatigue. Another occupant parameter can be that the gaze of an occupant, in particular the driver of a motor vehicle, frequently deviates from the roadway and can thus indicate an occupant condition of distraction. The method presented here can be expanded by a person skilled in the art to include other occupant parameters and associated or attributable physical and / or psychological occupant conditions, such as emotional agitation and / or intoxication. The overall confidence value GK can be compared with a threshold value SW in a comparison unit.If, for example, the overall confidence value GK is smaller than the threshold value SW, the respective occupant parameters, for example IP1 and / or IP2, cannot be transferred to a first allocation unit ZE1 in a further process step. It should be emphasized here that with the method described in . Fig.1 method, only one overall confidence value GK is determined for one occupant parameter at a time. However, as already described, the method can be repeated in parallel to check the accuracy of several occupant parameters with only one image data stream fed to the respective processing paths. Several overall confidence values ​​GK can be fed to the first assignment unit ZE1 in order to determine the accuracy of an occupant state IZ. In the first assignment unit ZE1, an occupant state IZ can be assigned to one and / or more occupant parameters, in this example IP1 and IP2, for example via a first lookup table LUT1. The occupant state IZ determined in the first assignment unit ZE1 can be transferred to a second assignment unit ZE2. In this assignment unit ZE2, the occupant state IZ can be assigned to an action signal via a second lookup table LUT2.In this exemplary embodiment, for example, the occupant parameter 1 IP1 can describe frequent blinking of the eyelids and the second occupant parameter IP2 can describe frequent nodding of the head forward. In the assignment unit ZE1, these two occupant parameters are assigned, for example, to the occupant state IZ "tiredness" via the first lookup table LUT1. This means that a result value is determined via a lookup table if the prerequisites assigned to this result value, in this case "tiredness", or only one prerequisite, in this case the occupant parameters IP1 and IP2, are present or apply. In the second assignment unit ZE2, this occupant state can be assigned to an action signal via a second lookup table LUT2, which should, for example, prompt the driver to take a break.Thus, in the second lookup table, action instructions or an action instruction can be linked to one or more occupant states as prerequisites. This action signal HS can be transmitted to a display device AE included in the motor vehicle. The display device can be a screen known from the prior art that is used in motor vehicles. On this display device AE, for example, the driver can be shown a warning signal in conjunction with a text request to take a break soon or within a specified time interval. The determination or evaluation of the accuracy of the or an occupant parameter and / or one or more occupant states can be subject to a time cycle, i.e., can be carried out repeatedly at a specified time interval according to the method, for example every 10 to 60 seconds.

[0048] A system or method can use multiple optical sensors (ranging from intelligent cameras with computing power and memory to simple optical heads or camera devices for image acquisition that contain only lenses and image converters) to perceive the vehicle interior and centrally calculate the images. A so-called "computer vision perception stack" can be trained in the vehicle with various sensor positions. A "computer vision perception stack" can, for example, represent several cascaded layers or processing modules for determining an occupant parameter. The perception stack or the individual processing paths can be trained with data (images) that cover every installation position, training acquisition position, or acquisition position that is suitable and may be approved for vehicle production.Each position, or training acquisition position, or acquisition position i is assigned a weight W(i). The weight determines how relevant or important the trained position should be in the overall perception stack or procedure.

[0049] Through a software configuration, the system or method reacts to the availability or unavailability of an installation position, training acquisition position, or acquisition position for the optical head in the final vehicle model produced. In a very simplified model, the weighting of the available position, training acquisition position, or acquisition position may be 1, while the weighting of the unavailable position is 0. Based on the installation position or acquisition position of the optical sensors in a produced vehicle, a weighting engine in the optical interior perception system decides the relevance of the optical perception data set used to train the camera for the entire stack interpretation or the system or method.For a system with n optical sensors, j perspective groups and p different installation positions or training acquisition position or acquisition position for each of them, the relevance of each position or training acquisition position or acquisition position i, R (j,i), is a number between 0 and 1, depending on the distance of a training acquisition position to the actual installation position of the camera (R(j,i) increases with decreasing distance):. R(j,i)∈[0,1]∀j∈1:n,i∈1:p

[0050] An optical sensor with only one mounting position or training acquisition position or acquisition position has a relevance of R = 1 for that position. An optical sensor that is not installed in a vehicle configuration has R(i) = 0 for all installation positions. A weighting module calculates the weight of each camera position or training acquisition position or acquisition position for the considered algorithms, depending on the role of the optical sensor (master or slave) and the sensor position or training acquisition position or acquisition position. For an optical sensor in a perspective group j with p possible installation positions or training acquisition positions or acquisition positions, the sum of the weights W(i) of each position i adds up to 1, depending on the probability of the installation position in this vehicle model, if at least one sensor position or perspective group j is installed in the vehicle: Σ W(i) = 1 if at least P(j) >0 , ∀ i,j ε 1:n

[0051] Overall, the examples show how a method for outputting an action signal to an occupant in a motor vehicle can be provided.

Claims

[1] Method for issuing an action signal (HS) to an occupant in a motor vehicle, wherein the action signal (HS) is generated depending on an occupant state (IZ) which is determined by means of occupant parameters (IP), wherein a respective occupant parameter (IP) is assigned a total confidence value (GK), which represents the sum of weighted individual confidences (EK), wherein the individual confidences (EK) relating to the respective occupant parameter (IP) are each generated by processing a respective image data stream through a respective processing path (VP), wherein each processing path (VP) is provided trained for processing a training image data stream that has been acquired at only one of a plurality of predetermined training acquisition positions (TP), wherein a training acquisition position (TP) is characterized by a position-specific camera perspective, whereby each processing path (VP) and a respective training acquisition position (TP) are mutually associated, wherein the image data stream is associated with at least one processing path (VP) for processing depending on a spatial distance between its actual acquisition position (EP) and at least one training acquisition position (TP), wherein the individual confidences (EK) generated for each processing path (VP) are weighted with at least one processing path-specific relevance (R),which is determined depending on the spatial distance between the respective training acquisition position (TP) and the acquisition position (EP), whereby the training acquisition positions (TP) and acquisition positions (EP) are each assigned to a perspective group (PG) classifying the camera perspective depending on their respective acquisition range, where the individual confidences (EK) assigned to a perspective group (PG) are weighted with a perspective group-specific weighting (PGW). [2] Method according to claim 1, wherein the respective occupant parameter (IP) is only included in the determination if the total confidence value (GK) assigned to the occupant parameter (IP) is above a predetermined threshold value (SW). [3] Method according to claim 1 or 2, wherein the perspective group-specific weighting (PGW) is assigned dynamically during the method as a function of an evaluation of an image quality of an image data stream which is supplied to this perspective group (PG). [4] Method according to one of the preceding claims, wherein the relevance (R) of a processing path (VP) which processes an image data stream is determined as a function of the number of processing paths (VP) which process this image stream and the distance of a detection position (EP) of the image data stream to the respective training detection positions (TP) of the processing paths (VP), wherein the sum of all relevancies (R) per perspective group is normalized to 1. [5] Method according to one of the preceding claims, wherein each processing path (VK) is comprised of an artificial intelligence, a machine learning algorithm, a neural network, a deterministic image processing algorithm and / or a statistical image processing algorithm.

Images