Method of detecting presence of an object in a vehicle

Abstract

The present disclosure relates to a method of detecting a presence state of an object in a vehicle, in particular to a computer-implemented method of detecting a presence state of a given object in a vehicle, comprising the steps of: processing data received over time from various devices in the vehicle, said data comprising parameters related to the given object indicative of whether the given object is present in the vehicle, to determine a set of cumulative scores; determining, based on the set of cumulative scores, an object-related confidence value representative of a likelihood that the given object is present in the vehicle; comparing the object-related confidence value to a predetermined threshold value to detect the presence of the object in the vehicle if the current object-related confidence value exceeds the predetermined threshold value.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of detecting and managing the presence status of a given object in a vehicle. The object can be, for example, a child, an animal or a person that can require the attention of the driver and / or other passengers in the vehicle. BACKGROUND

[0002] In the automotive field, there is a need to detect children or animals in the vehicle cabin in order to avoid leaving children or animals in the vehicle after the driver has left the vehicle and to avoid heat stroke or health problems. It can take only about ten minutes for the temperature in the vehicle to increase by 20 degrees.

[0003] For example, to sense a child in the vehicle cabin, a camera sensor can be used. If the child is in the field of view of the camera sensor, the child can be directly detected from the image data captured by the camera sensor by a visual algorithm of object detection. However, to sense a child in the field of view of the camera sensor, it can be necessary to install several camera sensors at different locations of the vehicle to cover the entire cabin. Other sensors such as radar sensors can be used in the cabin by detecting materials. Such sensors are more flexible in terms of positioning. But many vehicles are not equipped with such radar sensors to monitor the interior of the cabin.

[0004] In many cases, it is not possible to directly sense a child with sensors in the vehicle, in particular when there is a limited number of camera sensors in the cabin. For example, in a vehicle equipped with only one camera sensor located at the front of the cabin and pointing towards the rear, it is not possible to directly detect a child in the rear-facing rear seats.

[0005] Therefore, there is a need to address the problem of overcoming the limitations of direct object visibility to detect the presence of a child or an animal in a vehicle, and more generally, the presence of a given object. In other words, there is a need to facilitate the detection of the presence status of a given object in a vehicle in various situations, including situations where the object cannot be directly seen from an image sensor device in the vehicle. SUMMARY

[0006] The present disclosure relates to a computer-implemented method of detecting the presence status of a given object in a vehicle, the computer-implemented method comprising the steps of:

[0007] - processing data received over time from various devices in the vehicle, the data comprising parameters related to the given object, indicative of whether the given object is present in the vehicle, to determine a set of cumulative scores;

[0008] - determining, based on the set of cumulative scores, an object-related confidence value representative of the likelihood that the given object is present in the vehicle;

[0009] - comparing the object-related confidence value to a predetermined threshold value to detect the presence of the object in the vehicle if the current object-related confidence value exceeds the predetermined threshold value.

[0010] The method allows to accumulate scores determined by processing data received over time from various devices in the vehicle, such as on-board sensors, the data comprising parameters related to a given object, these parameters being indicative of the presence or not of the given object in the vehicle. In other words, the method allows to accumulate the likelihood or score of the presence or not of an object in the vehicle. Typically while the vehicle is running, a confidence value can be determined based on the accumulation of scores determined from the received data over time. It is compared to a threshold value and when the confidence value reaches and / or exceeds the threshold value, it is considered high enough to determine that the given object is present in the vehicle.

[0011] In one embodiment, the step of determining an object-related confidence value can comprise updating the object-related confidence value when a new score is added to the set of accumulated scores.

[0012] In one embodiment, the step of processing data can be performed continuously over time while the vehicle is running to accumulate scores in the set of accumulated scores over time.

[0013] In one embodiment, the step of processing data can comprise obtaining the parameters related to the given object from the received data and mapping the parameters to respective scores.

[0014] Thus, data received from a plurality of on-board devices is processed to determine parameters related to a given object and the parameters are mapped to respective scores.

[0015] In one embodiment, the step of determining an object-related confidence value can comprise assigning a weight to respective accumulated scores.

[0016] In a particular embodiment, the step of determining an object-related confidence value can comprise computing a cumulative sum of the accumulated scores.

[0017] In one embodiment, scores can be weighted by respective weights in the cumulative sum.

[0018] In one embodiment, the step of processing data can use a machine learning model, such as a neural network.

[0019] The received data or parameters comprised in the received data can be provided as input to the machine learning model. The machine learning model can compute a confidence value representing the likelihood of the presence of the given object in the vehicle. The machine learning model, e.g. a neural network, can be trained to detect the presence of the given object from the collected data and / or from the parameters comprised in the received data. Such detection is very flexible and effective over time.

[0020] In one embodiment, the method can comprise the steps of:

[0021] detecting the presence of the given object in the vehicle by direct sensing;

[0022] collecting data received over time from various devices in the vehicle, the data being indicative of the presence of the given object in the vehicle;

[0023] using the collected data as training data to further train the machine learning model.

[0024] Thus, if the given object is directly detected, e.g. from captured image data, the data received from the on-board devices can be used as new training data to perform additional training of the machine learning model.

[0025] In one embodiment, the step of obtaining, from the received data, parameters related to the given object in the vehicle comprises at least one of the following steps:

[0026] - a step of estimating and tracking a body posture of a person in the vehicle from image data captured by image sensor devices to determine a behavior of the person towards the given object;

[0027] - a step of detecting, from data captured by one or more sensor devices, an action of placing a child in the vehicle through an open vehicle door, the given object being a child;

[0028] - a step of detecting, from data captured by sensor devices, a movement in a child seat area, the object being a child.

[0029] The task of estimating and tracking a body posture of a person (driver or passenger) in the vehicle can be performed to determine one or more parameters or information related to a body behavior of the person towards the given object, e.g. turning several times to the child seat during a vehicle ride.

[0030] The detection of a movement in a child seat area can also provide information or parameters indicative of the presence of a child in the vehicle.

[0031] The detection of an action of placing a child in the vehicle can also provide information or parameters indicative of the presence of a child in the vehicle.

[0032] In one embodiment, the vehicle can comprise a memory for storing a user profile of the driver indicating that the driver has children or driving history information indicating that children were present in the past several times in the case of the driver. In this case, the step of obtaining from the received data a parameter relating to the given object can comprise a step of reading in the memory information that the given object is present in the stored user profile or in the stored driving history information.

[0033] The vehicle can have in the memory a user profile of the driver (or other passenger) indicating that the driver has children or driving history information indicating that children were present in the past several times in the case of the driver. Such information can be a parameter or information indicating the presence of children in the vehicle.

[0034] In one embodiment, the step of obtaining from the received data a parameter relating to the given object can further comprise at least one of the following steps:

[0035] - a step of detecting from the data captured by the image sensor device the presence in the vehicle of one or more objects attached to the given object;

[0036] - a step of analyzing the captured audio data;

[0037] - the given object is a child, a step of analyzing the content played by the infotainment system of the vehicle.

[0038] For example, the presence of one or more objects attached to the object can also be detected from the data captured by the image sensor device. When a child is in the car, the image sensor device can capture images including toys, child equipment or any other object that can be related to the presence of a child in the vehicle. The detection of objects attached to the object in the vehicle provides information or parameters relating to the child indicating the presence of a child in the vehicle.

[0039] The object is a child, the method can further comprise a step of analyzing the content played by the infotainment system of the vehicle. If one or more contents played by the infotainment system of the vehicle can be classified as contents suitable for children (for example using a content filter as used in a parental control application), it can provide information or parameters indicating the presence of a child in the vehicle.

[0040] The detection of child speech in the audio data captured in the vehicle can also provide information or parameters indicating the presence of a child in the vehicle.

[0041] In one embodiment, the plurality of accumulated parameters indicative of the presence or absence of the given object in the vehicle can comprise parameters of different types, and different weights can be assigned to the scores related to the parameters of different types, respectively, to increase the confidence value.

[0042] In one embodiment, the method can further comprise an alerting step triggered when the given object is left in the vehicle after it is detected that one or more passengers have left the vehicle. Then, a warning action can be performed, including for example displaying a sign on a display in the vehicle and / or outputting an alert (light and / or sound alert) when the personnel in the vehicle leaves the vehicle without the object.

[0043] The present disclosure also relates to a data processing device for detecting the presence of a given object in a vehicle, said data processing device comprising

[0044] a receiving unit for receiving data from various devices in the vehicle; and

[0045] a processing unit adapted to perform the steps of the method as defined above.

[0046] The present disclosure also relates to:

[0047] a computer program comprising instructions which, when the program is executed by a computer, cause the computer to perform the steps of the method as defined above;

[0048] a computer-readable medium on which the computer program is stored;

[0049] a vehicle comprising the data processing device as defined above. BRIEF DESCRIPTION OF DRAWINGS

[0050] Other characteristics, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments, which is made with reference to the attached drawings.

[0051] Figure 1 A flowchart of a method for detecting the presence of a given object in a vehicle is shown, according to one embodiment.

[0052] Figure 2 A flowchart of the processing for accumulating scores and determining a confidence value related to the presence of a given object in a vehicle is shown, according to one embodiment.

[0053] Figure 3 A functional block diagram of a data processing device for detecting the presence of a given object in a vehicle is shown, according to one embodiment. DETAILED DESCRIPTION

[0054] The present disclosure relates to a computer-implemented method of detecting and managing a presence status of a given object in a vehicle 100, e.g. a car, and a data processing apparatus 200 comprising a receiving unit or interface 10 for receiving data from various apparatuses 30a, 30b,... in the vehicle 100 and a data processing unit 20 for executing the steps of the computer-implemented method. The processing unit 20 comprises one or more software modules of program instructions running on the processing unit 20 for implementing the different steps described below which are executed by the processing unit 20.

[0055] The object can comprise a child, an animal or a person, for example. Such an object can require the attention of a person (driver and / or other passenger) in the vehicle 100.

[0056] The software modules can comprise an object or person detection module, a tracking body module, an audio analyzer and / or any other appropriate functional module to implement the steps described below. In one embodiment, the software modules can comprise one or more machine learning modules trained to perform one or more steps or tasks.

[0057] Figure 3 The vehicle 100 schematically represented in Fig. 1 can be equipped with one or more sensor apparatuses. In one embodiment, the on-board sensor apparatuses can comprise at least one image sensor apparatus 30a, e.g. a camera sensor apparatus and / or a radar sensor. Optionally, the sensor apparatuses can comprise an audio sensor apparatus 30b, one or more door sensors 30c sensing the opening and closing of a vehicle door, a motion sensor 30d, an acceleration sensor or any other type of appropriate sensor apparatus.

[0058] The vehicle 100 can further comprise other apparatuses or systems, e.g. an infotainment apparatus or system 40a, one or more memory apparatuses 40b, and at least one on-board monitoring system, e.g. a driver monitoring system 40c and / or a cabin monitoring system 40d. The memory apparatuses 40b can store one or more user profiles and / or driving history information related to one or more users. The user profile and / or the driving history information related to a given user can comprise information that a given object, e.g. a child and / or an animal, has appeared in the vehicle 100 a number of times in the past when said user was in the vehicle 100.

[0059] The computer-implemented method allows for indirectly detecting a presence status of a given object, e.g. a child or baby, or an animal, in the vehicle 100. It can be used in situations where the given object is not directly visible by the on-board monitoring system 40c, 40d and / or where there is no direct visibility of the object from the image sensor apparatus 30a of the vehicle 100.

[0060] Reference will now be made to Figure 1Embodiments of a method of detecting the presence of a given object, such as a child, in a vehicle 100 will be described. It is assumed that the given object is not directly visible by the on-board monitoring system or image sensors.

[0061] In step S1, sensor devices 30a, 30b... of the vehicle 100 capture data and send the captured data to the data processing device 200. Step S1 can be performed continuously over time while the vehicle 100 is in operation. The vehicle 100 can be moving or stopped in operation. For example, the camera sensor device 30a can capture images of the interior of the cabin of the vehicle 100 within a given field of view. The captured images can cover the two front seats, the interior of the two front doors and part of the rear seats. For example, the sensor device 30c can also detect the opening and closing of the respective doors of the vehicle 100.

[0062] In step S2, the sensor data from the respective sensor devices 30a, 30b... is received by the data processing device 200, here by the receiving unit 10. The sensor data can be received continuously over time, typically in real time as it is captured.

[0063] In step S1 ', other on-board devices or systems such as the infotainment system 40a, the memory device 40b, the driver monitoring system 40c and / or the cabin monitoring system 40d can send data to the data processing device 200. For example, the infotainment system 40a can send information about the audio and / or video content that the infotainment system 40a is playing. Information about the presence of a child or an animal in the vehicle 100 when the current driver was driving the vehicle 100 in the past can be sent from the memory device 40a to the data processing device 200. Data or parameters detected by the on-board monitoring system 40c or 40d can also be sent to the data processing device 200. Thus, in step S2', the data processing device 200 can receive data from devices other than sensors over time while the vehicle 100 is in operation, in any order.

[0064] In steps S2 and S2', the data received by the data processing device 200 from the plurality of on-board devices 30a, 30b... 40a, 40b... over time, in parallel or in any order, can include image data, sound data, sensor data, infotainment data, stored data, etc. The received data can contain parameters related to a given object, such as a child in the illustrative and non-limiting example herein, which are indicative of the presence or absence of said object in the vehicle 100.

[0065] In step S3, the processing unit 20 can process the data received from the plurality of on-board devices 30a, 30b...40a, 40b... over time to obtain a plurality of parameters or information related to the presence of a given object in the vehicle 100 and determine a set of accumulated scores. The data can be received at successive points in time and then processed over time to accumulate scores over time.

[0066] The parameters or information contained in the data received by the data processing device 200 from the plurality of on-board devices 30a, 30b...40a, 40b... can indicate, with some uncertainty, the presence or absence of a given object in the vehicle 100. This means that the information or parameter can be related to the presence or absence of a given object in the vehicle 100. The parameter indicating the presence of an object in the vehicle is not a direct and certain detection of the object, for example as directly detecting the object from an image captured by an image sensor device. It can indicate the likelihood of the presence of an object in the vehicle 100.

[0067] The step S3 of processing the data received by the receiving unit 10 can comprise the execution, by the processing unit 20, of different types of detection using the received data to determine or obtain, from the received data, parameters or information related to a given object, for example a child. A non-exhaustive list of detections is given below:

[0068] - detection of an accessory object related to the presence of a given object, for example a toy in the case where the given object is a child;

[0069] - estimation and tracking of the body posture of the driver and / or passengers in the vehicle 100 to detect specific behaviors of the people towards the given object;

[0070] - classification of the audio and / or video content played by the infotainment system of the vehicle 100;

[0071] - detection of the rearview mirror check and / or adjustment;

[0072] - identification of information indicating the presence of a given object in the driver's user profile and / or driving history information;

[0073] - detection of movement in the child seat area;

[0074] - detection of the action of placing a child in the vehicle 100 through an open door;

[0075] - voice or audio detection.

[0076] Therefore, step S3 can comprise a step S30 of detecting and classifying one or more objects attached to the given object, which is indicative of the presence of the given object in the vehicle 100. For example, in the case where the given object is a child, the attached objects can comprise a child seat, a booster seat, toys, any other baby or child equipment, etc. In an embodiment, the attached objects can be detected by the processing unit 20 from the image data captured by the camera sensor device 30a. In this case, the processing unit 20 can implement the function of object detection. The information relating to the presence of one or more objects attached to the given object, for example a child, can be a parameter indicative of the presence of the given object in the vehicle 100. Alternatively, the processing unit 20 can use data from the sensors to detect the attached or unattached state of the child seat.

[0077] Step S3 can also comprise a step S31 of estimating and tracking the body posture of each person in the vehicle 100 from the image data captured by the camera sensor device 30a or the on-board monitoring system 40c, 40d, for example, to detect the behavior of said person towards said object. Step 31 can be performed by the processing unit 20 of the algorithm (software component) performing the body posture tracking to track the body behavior of the driver or other passengers in the vehicle 100. In step S31, the seating position of each person (driver or other passengers) can be detected and the body posture of this person can be monitored from the captured image data. In one embodiment, key body points (e.g. shoulder, hand, elbow, knee, hip points, facial feature points, etc.) can be detected and tracked. The information about the body key points can be two-dimensional or three-dimensional. Then, by monitoring the body key points over time, the processing unit 20 can estimate and track the body posture of the person and detect the body movements of turning around and / or looking at or touching the child seat. The following information or parameter elements can be further obtained from the captured image data: the time points at which the person starts and stops turning around, the duration of the person's turn around, the frequency of the person's turn around. If a child seat is detected at a given location in the vehicle 100, the processing unit 20 can obtain the interaction between the person seated in the vehicle 100 and the child seat based on tracking the person's hand in the vicinity of the child. Other information such as the frequency or duration of the interaction between the person and the child seat can be accumulated and used as information or parameters to indicate with a certain degree of uncertainty the presence of said object in the vehicle 100. In another variant, the processing unit 20 can determine the time series of the interaction of the hand with the child seat or with the child inside the child seat. The time series can also include the action of touching an object attached to the child (such as a toy, a pacifier, a book, a water bottle...) and moving it inside the child seat. Conversely, in the absence of a child in the vehicle 100, the data processing device 200 can detect in step S31 of estimating and tracking the body posture that the passengers in the vehicle 100 do not move, or move little, or do not have any movement related to a child. Thus, the estimation and tracking of the body posture related to the passengers of the vehicle 100 makes it possible to obtain a plurality of parameters or information indicating the presence or absence of a given object (e.g. a child) in the vehicle 100.

[0078] In case the given subject is a child, step S3 can further comprise a step S32 of analyzing and classifying the audio and / or video content (music, videos, etc.) played by the infotainment system 40a of the vehicle 100 to determine whether said content is a content suitable for children. It can also be determined whether the content is not suitable for children. The analysis and classification of the content can be performed using a content filter, as used in parental control applications. The analysis and classification can be performed on data captured by the audio sensor 30b or on audio data and / or metadata related to the content directly sent to the data processing device 200 by the infotainment system 40a. Thus, in step S32, the data processing device 200 can determine the classification of the content played by the infotainment system 40a, which can be a content suitable for children and a content not suitable for children. A content suitable for children is for example a children cartoon, while a content not suitable for children is for example hard rock music. The determined classification can be a parameter or information indicative of the presence of a child in the vehicle 100.

[0079] Step S3 can further comprise a step S33 of detecting an adjustment of the rearview mirror by the driver, which is indicative of the driver's attention to the use of the rearview mirror. Optionally, it can be further detected that the rearview mirror was adjusted in a way that the driver gets a better view and the given back seat can be monitored. The detection S33 can be performed using data from one or more sensors, such as motion sensor 30d and / or acceleration sensor, to obtain the absolute position and orientation of the rearview mirror, in particular in case of a manually adjusted rearview mirror. Alternatively, the rearview mirror can be controlled by electronic adjustment. In this case, a deviation between the ideal position to see the road through the rearview mirror and the ideal position to see the given back seat can be calculated and used to adjust the position of the rearview mirror. This information related to the adjustment of the rearview mirror by the driver can be used as a parameter or information indicative of the presence of a child in the vehicle 100. Conversely, in step S33, the data processing device 200 can detect that the driver did not make any specific adjustment of the rearview mirror and this information can be used as a parameter indicative of the absence of a child in the vehicle 100.

[0080] Step S3 can further comprise a step S34 of retrieving from the memory device 40b information on the presence of a given subject, such as a child, in the vehicle 100, which information is comprised in a stored user profile or in stored driving history information related to the current driver or passenger of the vehicle 100. The information on the presence of a subject in the stored driving history information and / or in the stored user profile can optionally be combined with timing information and / or location information, such as a GPS track, which corresponds to a pattern of picking up a child from school for example. This information on the presence of a given subject, such as a child, in the vehicle 100 extracted from the user profile or driving history information can be used as a parameter indicative of the presence of the given subject in the vehicle 100.

[0081] In case the object is a child, step S3 can further comprise a step S35: detecting movements in the child seat area. Any detected movement within the seat area with the child seat can increase the probability of a child being in the child seat. For example, trackable feature points in the child seat area and surrounding areas can be detected from the captured image data. Movements within the child seat area (feature displacements in the captured image data) can be compared to movements outside the seat area. The average movement outside the child seat area can be subtracted from the movements within the child seat area. The remaining delta corresponds to movements specific to the child area. This can be further fed into a classification module that distinguishes typical movements caused by a child from other movements caused by the vehicle 100 motion. Information related to the detected movements in the child seat area can be a parameter indicating the presence of a child in the vehicle 100. If no movements are detected in the child seat area, this information can be used as a parameter indicating the absence of a child in the vehicle 100.

[0082] In case the object is a child, step S3 can further comprise a step S36: detecting from the captured data by the camera sensor arrangement 30 an action of putting a child into the vehicle 100 through an open vehicle door. The detected action of putting a child into the vehicle 100 can be used as a parameter indicating the presence of a child in the vehicle 100. In one embodiment, the field of view of the camera sensor arrangement 30 can have at least one partial view on the vehicle 100 door and a person outside the cabin can be detected from the image data captured by the camera sensor arrangement 30. Optionally, by processing the captured image data, it can be determined that:

[0083] - one or more body parts of the outside person reach into the cabin (e.g. arms or hands);

[0084] - a child (e.g. a baby) can be put into the seat area.

[0085] In a variant, the processing unit 20 can detect the child when the person puts the child (e.g. a baby) into the vehicle 100 while the child is still in the field of view of the camera sensor arrangement 30 before being hidden in the child seat.

[0086] Such detection can be performed by an object detection function and / or a function estimating and tracking body keypoints, implemented on the processing unit 20 and using the image data captured by the camera sensor arrangement 30.

[0087] The processing unit 20 can further use data captured by the sensor arrangement 40 related to the vehicle door opening state and / or corresponding signals on the vehicle 100 message bus.

[0088] Step S3 can also comprise an audio detection step S37. A microphone arrangement can be used to capture audio signals in the cabin of the vehicle 100. The captured audio signals can subsequently be analyzed in step S37 by the processing unit 20 to detect, for example:

[0089] - special sounds related to a child or baby (e.g. a baby’s scream, cry or laughter), or special sounds related to an animal such as a dog; and / or

[0090] - keywords suggesting that a person in the vehicle 100 is talking to a child (child’s name, specific language, etc.) or an animal.

[0091] The detected audio information can be used as a parameter indicating the presence of a given object (e.g. a child or an animal) in the vehicle 100.

[0092] The processing unit can further classify the speaker age from the captured audio signals. The information about the speaker age can be used as a parameter indicating the presence of a child in the vehicle 100.

[0093] The detections S30 to S37 can be performed in parallel over time or in any order. They result in a plurality of parameters or information being collected over time. These parameters are related to a given object (e.g. a child) and indicate whether the given object is present in the vehicle 100. In other words, each parameter can be an indication of whether the given object is present in the vehicle 100. It has some degree of uncertainty related to whether the given object is present in the vehicle 100. The plurality of parameters resulting from any of the detection steps S30 to S37 is accumulated over time.

[0094] The method can also comprise a step S38 of mapping the parameters obtained from the data received and processed by the data processing arrangement 200 in steps S30 to S37 to respective scores. The scores can be predetermined values or determined by a scoring function. The score of a parameter can depend on the type of the parameter. The score to which a parameter is mapped can be fixed or can vary depending on various factors, such as the uncertainty related to said parameter.

[0095] The following gives an illustrative example of how parameters can be mapped to scores with respect to the detection steps S30 and S31.

[0096] Let us consider the score determined for the detection S30 of an accessory object related to a given object such as a child. This score can initially be set to zero. Each detected accessory object can proportionally increase this score depending on its likelihood of indicating the presence of a child. The likelihood can have predetermined values predefined for each object class or group of object classes.

[0097] In the present example, the likelihood can be further weighted with the confidence of the detection of the accessory object. For example, the confidence of the detection can be represented by a floating point value between 0 (no confidence) and 1 (highest confidence).

[0098] In another variant, the position of the detected accessory object can also be used as another factor to determine the score.

[0099] For example, the total score determined from the detection S30 of the accessory object in relation to a given object, such as a child, can be expressed as follows:

[0100]

[0101] where

[0102] - i is the index of the detection S30 of the accessory object;

[0103] - I is the total number of detections S30 of the accessory object;

[0104] - likelihood obj_class(i) is the likelihood value of the classification of the accessory object i indicating the presence of a child in the vehicle;

[0105] - confidence obj_class(i) is the confidence value of the detection of the accessory object i;

[0106] - likelihood obj_location(i) is the likelihood value of the position of the accessory object i indicating the presence of a child in the vehicle.

[0107] Let us consider the score determined for the detection S31 in relation to the movements of the person in the vehicle 100. This score can initially be set to zero. Each time the person makes a predetermined movement, such as reaching for the child seat, turning around, looking at the child seat, etc., a corresponding counter can be incremented by 1, all counters being initially set to zero. In one embodiment, the counters can also be decremented for certain actions, such as the person putting a large item on the child seat.

[0108] The respective values of the different counters can be combined to determine a final score. A function of the weighted sum of all counters can be used to compute the final score. Each counter can have a specific weight. These weights allow to selectively score or credit each feature or movement. Thus, the score of one feature such as touching the child seat can be higher than another feature such as looking at the child seat.

[0109] The step S38 can be performed by the data processing device 200 to accumulate the scores and produce a set of accumulated scores. The step S38 of determining and accumulating the scores can be performed continuously over time while the vehicle 100 is running. In this way, the scores can be continuously added in the set of accumulated scores over time.

[0110] The method further comprises a step S4 of determining an object-related confidence value of the presence state of the given object in the vehicle 100. Step S4 can be performed by the data processing device 200. For example, if the given object is a child, the determined confidence value is related to the child. The confidence value is a value representative of the likelihood of the given object being present in the vehicle 100.

[0111] The object-related confidence value is determined based on the scores accumulated in step S38. In a particular embodiment, the determination of the object-related confidence value can comprise a step of computing a cumulative sum of the accumulated scores. Optionally, respective weights can be assigned to the scores. The weights can depend on the parameters that have been mapped to said scores. Thus, in the step of computing a cumulative sum of the accumulated scores, the scores can be multiplied by the corresponding weights.

[0112] In one embodiment, scores indicative of the presence of the given object in the vehicle 100 can be positive, while scores indicative of the absence of the given object in the vehicle 100 can be negative. Thus, the object-related confidence value should increase by positive scores and decrease by negative scores. For example, when the infotainment system 40a of the vehicle 100 plays content that is not suitable for children (e.g. a piece of hard rock music), the confidence value can decrease by a certain amount, while when the infotainment system 40a of the vehicle 100 plays child content (e.g. a cartoon), the confidence value can increase by a certain amount.

[0113] The object-related confidence value can be updated upon adding a new score to the set of accumulated scores, preferably each time a new score is added to the set of accumulated scores.

[0114] In a particular embodiment, the object-related confidence value can be a likelihood value.

[0115] In one embodiment, the likelihood value can be expressed in percentage or be a value between 0 and 1. For example, a probability of 100% means that the object is certainly present in the vehicle 100, while a probability of 0% means that the object is certainly not present in the vehicle 100. In this case, the scores can be fractional values of the probability. In another embodiment, the confidence value and the scores can be scaled differently.

[0116] In one embodiment, the object-related confidence value can be determined or computed based on the set of accumulated scores determined while the vehicle 100 was running, when the vehicle 100 is stopped and the operation of the vehicle is ended.

[0117] In another embodiment, the object-related confidence value can be determined by a machine learning model.

[0118] The machine learning model can receive as input data the data received from the plurality of devices 30a, 30b... 40a, 40b... in the vehicle 100 and / or the parameters determined in step S3 indicating the presence or absence of the given object in the vehicle 100. The machine learning model can comprise a neural network.

[0119] The machine learning model can be trained with training data comprising:

[0120] - input training data comprising a plurality of sets of data (as previously described) from on-board devices such as sensors 30a, 30b... and other devices 40a, 40b... or parameters obtained from the received data (as described in steps S30-S37), each set of data or parameter corresponding to a scenario recorded in the past by the vehicle 100, and

[0121] - output training data comprising, for each set of data or parameter, a confidence value of the presence state of the given object in the vehicle 100, the confidence value being manually assigned to the corresponding scenario.

[0122] In another embodiment, the machine learning model can be trained to learn from the input data a binary state representing the state "child present" or the state "child absent". The confidence value can then be obtained from the output of the machine learning model.

[0123] In another variant, the input training data can be synthetic data obtained for example by simulation or a mix of real data and synthetic data.

[0124] After determining the object-related confidence value of the presence of the given object in the vehicle 100, the method further comprises a step S5 of comparing the determined object-related confidence value with a predetermined threshold value and detecting the presence of the object when the confidence value exceeds said threshold value.

[0125] The step S5 of comparing the confidence value with a predetermined threshold value and detecting the presence of the object in case the confidence value reaches or exceeds the predetermined threshold value is performed by a software module (or program instructions) running on the processing unit 20. The step S5 can be performed by the machine learning model determining the confidence value.

[0126] In one embodiment, the presence state of the given object (e.g. a child) in the vehicle 100 can be detected by direct sensing, typically from captured images, for example during operation of the vehicle 100. In this case, the method can further comprise the following steps:

[0127] - collecting or gathering the data received by the data processing device 300 from the plurality of devices in the vehicle 100 over time, these data comprising parameters indicating the presence of the given object in the vehicle 100,

[0128] - using the collected data as training data for additionally training the machine learning model.

[0129] The additional training can be performed by running an integrated software part on a processor or computer inside the vehicle 100, or the training data can be transmitted out of the vehicle 100, for example to a cloud backend or backend server, and processed externally.

[0130] Alternatively, the presence of the given object can be detected after the vehicle has just stopped after being in operation. In this case, data received by the data processing device 300 over time while the vehicle 100 was in operation can have been recorded and used as training data for additional training of the machine learning model.

[0131] In one embodiment, in case an object in the vehicle 100 is detected in step S5, the method can further comprise a step S6 of alerting that a given object is present in the vehicle 100. This alerting step S6 can be triggered when it is detected that the object was left in the vehicle 100 after all passengers have left the vehicle 100.

[0132] In one embodiment, the alerting can use the output means 50 of the vehicle 100, for example by displaying a sign indicating “child in car” on a display of the vehicle 100. Alternatively or additionally, the alerting can comprise outputting an audio and / or light signal by the vehicle 100 to alert the person leaving the vehicle 100 that a given object was left in the vehicle 100. In a variant, the alerting can be triggered on the driver’s mobile phone. More generally, the presence of the given object can be communicated through any appropriate communication channel, for example a message inside or outside the vehicle, a message on a smartphone, an email, a change of vehicle lighting inside and / or outside the vehicle, a horn or other external sound production.

[0133] In one embodiment, in case a given object, for example a child, is detected in the vehicle 100 in step S5, the method can further comprise a step of adapting features or parameters or functions of on-board systems of the vehicle 100 to the presence of the detected object in the vehicle 100, such as ADAS (Advanced Driver Assistance System) systems. For example, after detecting a child in step S5, safety parameters of the ADAS system can be programmed. As illustrative examples, the airbag system near the child seat can be disabled, the vehicle speed limit can be programmed, a soft brake mode can be activated, etc.

[0134] The steps S2 to S5 of indirectly detecting the presence of the given object are looped over newly received data over time to determine and check the confidence value. In one embodiment, once the confidence value reaches a threshold, the steps S2 to S5 are only partially executed or no longer executed. In another embodiment, the steps S2 to S5 continue to be executed even after the threshold is reached. This allows monitoring further interactions with the one or more objects.

[0135] The present disclosure also relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the aforementioned method.

[0136] The present disclosure can also include one or more of the following numbered clauses, which can describe and / or relate to additional aspects or features within the context of the present disclosure:

[0137] 1. A computer-implemented method of detecting the presence of an object of interest in a vehicle, the computer-implemented method comprising the steps of:

[0138] receiving data captured by at least one sensor device in the vehicle, processing the received data to gather a plurality of pieces of information indicative of the presence of the object of interest in the vehicle with a certain degree of uncertainty;

[0139] computing a confidence value of the presence of the object of interest in the vehicle, wherein the more pieces of information gathered, the more the confidence value increases;

[0140] comparing the confidence value to a threshold value, and detecting the presence of the object when the confidence value exceeds the threshold value.

[0141] 2. The method of clause 1, further comprising the step of estimating and tracking a body posture of a person in the vehicle from image data captured by an image sensor device to determine a behavior of the person towards the object of interest.

[0142] 3. The method of clause 1 or 2, further comprising the step of detecting the presence of one or more objects attached to the object of interest from data captured by an image sensor device.

[0143] 4. The method of any one of clauses 1 to 3, wherein the object of interest is a child, the method further comprising the step of analyzing content played by an infotainment system of the vehicle to determine whether the content is a specific content for children.

[0144] 5. The method of any one of clauses 1 to 4, further comprising the step of reading in a memory information that the object of interest is present in a user profile or a driving history of the vehicle.

[0145] 6. The method according to any one of clauses 1 to 5, wherein the object of interest is a child, the method further comprising the step of detecting movement in a child seat area from data captured by a sensor device.

[0146] 7. The method according to any one of clauses 1 to 6, wherein the object of interest is a child, the method further comprising the step of detecting an action of placing a child through an open vehicle door from data captured by one or more sensor devices.

[0147] 8. The method according to any one of clauses 1 to 7, wherein the object of interest is a child, the method further comprising the step of analyzing captured audio data and classifying the audio data as child speech.

[0148] 9. The method according to any one of clauses 1 to 8, wherein the plurality of pieces of information collected comprises different types of cue information, and different weights are assigned to the different types of cue information to increase the confidence value.

[0149] 10. The method according to any one of clauses 1 to 8, wherein the plurality of pieces of cue information is provided as input to a machine learning module that performs the task of calculating the confidence value.

[0150] 11. The method according to any one of clauses 1 to 10, the method further comprising the step of alerting a person of the vehicle that the object of interest is present in the vehicle.

[0151] 12. The method according to clause 11, the method further comprising the step of detecting from data captured by a sensor device that the object of interest is left behind in the vehicle after the one or more passengers have exited the vehicle, which step triggers the performance of the alerting step.

[0152] 13. A data processing device for detecting the presence of an object of interest in a vehicle, the data processing device comprising

[0153] a receiving unit for receiving data comprising data captured by at least one sensor device in the vehicle; and

[0154] a processing unit adapted to perform the steps of the method according to any one of clauses 1 to 12.

[0155] 14. A computer program comprising instructions for causing the device of clause 13 to perform the steps of the method of any one of clauses 1 to 12.

[0156] 15. A vehicle comprising the device according to clause 13.

Claims

1. A computer-implemented method of detecting a presence state of a given object in a vehicle (100), the computer-implemented method comprising the steps of: processing (S3) data received over time from various devices (30a, 30b...40a, 40b...) in the vehicle (100), the data comprising parameters related to the given object indicative of whether the given object is present in the vehicle (100), to determine a set of accumulated scores; determining (S4), based on the set of accumulated scores, an object-related confidence value representative of a likelihood that the given object is present in the vehicle (100), wherein the parameters comprise different types of parameters and wherein different weights are assigned to scores related to the different types of parameters, respectively; comparing (S5) the object-related confidence value to a predetermined threshold to detect the presence of the object in the vehicle (100) if the current object-related confidence value exceeds the predetermined threshold.

2. The computer-implemented method of claim 1, wherein, The step (S4) of determining the object-related confidence value comprises updating the object-related confidence value when a new score is added to the set of accumulated scores.

3. The computer-implemented method of claim 1 or 2, wherein, The step (S3) of processing data is performed continuously over time while the vehicle is in operation to continuously accumulate scores in the set of accumulated scores over time.

4. The computer-implemented method of claim 1, wherein, The step (S3) of processing data comprises obtaining the parameters related to the given object from the received data and mapping the parameters to respective scores.

5. The computer-implemented method of claim 1, wherein, The step (S4) of determining the object-related confidence value comprises computing a cumulative sum of the accumulated scores.

6. The computer-implemented method of claim 1, wherein, The step (S3) of processing data uses a machine learning model.

7. The computer-implemented method of claim 6, further comprising: detecting a presence state of the given object in the vehicle by direct sensing; collecting data received over time from various devices in the vehicle, the data comprising parameters indicative of the presence of the given object in the vehicle; using the collected data as training data to further train the machine learning model.

8. The computer-implemented method of any one of claims 4 to 7, wherein, The step of obtaining parameters related to the given object from the received data comprises at least one of the following steps: - a step (S31) of estimating and tracking a body posture of a person in the vehicle from image data captured by an image sensor device (30) to determine a behavior of the person with respect to the given object; - a step (S36) of detecting, from data captured by one or more sensor devices, an action of placing a child into the vehicle through an open vehicle door, the given object being a child; - a step (S35) of detecting, from data captured by a sensor device, a movement in a child seat area, the object being a child.

9. The computer-implemented method of claim 4, wherein, Said vehicle comprises a memory for storing a user profile of the driver indicating that the driver has children or driving history information indicating that in the case of the driver there were children in the past a plurality of times, the step of obtaining from the received data parameters relating to the given object comprises the step (S34) of reading in the memory information that the given object is present in the stored user profile or in the stored driving history information.

10. The computer-implemented method of claim 4, wherein, The step of obtaining from the received data parameters relating to the given object comprises at least one of the following steps: - a step (S30) of detecting from the data captured by the image sensor device the presence in the vehicle of one or more objects attached to the given object; - a step (S37) of analyzing captured audio data; - a step (S32) of the given object being a child, analyzing the content played by the infotainment system of the vehicle.

11. The computer-implemented method according to claim 1, further comprising an alert step (S6) triggered when the object is left in the vehicle after it has been detected that one or more passengers have left the vehicle.

12. A data processing device (200) for detecting the presence of a given object in a vehicle, said data processing device comprising: - a receiving unit (10) for receiving data from various devices in the vehicle; and - a processing unit (20) adapted to perform the steps of the computer- implemented method according to any one of claims 1 to 11.

13. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the computer- implemented method according to any one of claims 1 to 11.

14. A vehicle comprising a data processing device according to claim 12. ​

Images