Cabin object motion detection

Abstract

In-cabin object motion detection is provided. The invention relates to vehicle cabin safety, in particular to detection of objects and their movement in a vehicle cabin. At least one object in the vehicle cabin is determined based on one or more images showing an interior of the vehicle cabin. A number of features of the at least one object is determined and a degree of danger of the at least one object is determined based on the number of features. A warning signal is output in response to determining that the level of hazard meets the given criterion.

Description

Technical Field

[0001] The present disclosure generally relates to improvements in vehicle safety and control, and more particularly to methods and systems for detecting the movement of objects within a vehicle cabin. Background Art

[0002] Intelligent vehicles, such as smart cars, smart buses, etc., are significantly improving passenger safety. Such intelligent vehicles can be equipped with on-vehicle cameras and are capable of capturing images inside the vehicle. These images can then be used (sometimes in combination with other sensors) for different safety-related tasks, such as detecting objects and their movement within the vehicle, as well as tasks such as seatbelt assistance, detecting people in the vehicle, classifying people as adults or children, determining whether one of the vehicle doors is open, etc.

[0003] In a vehicle, storing a large amount of luggage and heavy objects or objects in the vehicle cabin, especially, may affect passenger safety in different ways. Without being fixed, for example, by a seatbelt, especially heavy objects such as suitcases pose a threat to the safety of vehicle drivers and passengers in the case of strong acceleration of the vehicle (such as emergency braking). In addition, heavy objects of relatively large mass moving during vehicle driving (such as suitcases or other luggage) will potentially and adversely affect the maneuverability of the vehicle and the ability of the driver to operate the vehicle. In addition, large objects may affect or completely block the driver's view of the traffic behind the vehicle (such as through the rear window). Summary of the Invention

[0004] The present invention relates to a vehicle cabin safety system. For the safety of vehicle drivers and passengers, objects and their movement within the vehicle cabin are detected before they may become potentially dangerous projectiles or otherwise negatively affect driving safety.

[0005] In this context, methods, systems, and computer program products as defined in the independent claims are proposed.

[0006] In this regard, according to a first aspect, there is provided a computerized method for vehicle cabin safety, the method comprising the steps of: determining at least one object in the vehicle cabin based on one or more images showing the interior of the vehicle cabin; determining several features of the at least one object; determining the degree of danger of the at least one object based on the several features; and outputting a warning signal in response to determining that the degree of danger meets a given criterion.

[0007] In another aspect, there is provided a vehicle cabin safety system for performing vehicle cabin safety functions, comprising a sensor system, a data processing system, and an interface for outputting a warning signal.

[0008] In another aspect, a vehicle is provided that includes a vehicle system as described herein.

[0009] Finally, a computer program including instructions is presented, which, when executed by a computer, cause the computer to perform the methods described herein.

[0010] The dependent claims present further improvements.

[0011] These and other objects, embodiments, and advantages will become apparent to those skilled in the art from the following detailed description of embodiments with reference to the accompanying drawings. The invention is not limited to any particular embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Aspects and examples of the present disclosure are described with reference to the following drawings, in which:

[0013] Figure 1 The interior of a vehicle cabin with an object and a person is shown.

[0014] Figure 2 A portion of the interior of a vehicle cabin is shown, where the object is located on the rear seat.

[0015] Figure 3 A computer-implemented sequence for detecting an object and its movement in a vehicle cabin as described herein is depicted.

[0016] Figure 4 Another computer-implemented sequence for detecting an object and its movement in a vehicle cabin as described herein is depicted.

[0017] Figure 5 A computer-implemented sequence for tracking the movement state of an object in a vehicle cabin as described herein is depicted.

[0018] Figure 6 Illustrates the Figure 1 bounding box of the object as shown.

[0019] Figure 7 A computer-implemented sequence for determining object characteristics as described herein is depicted.

[0020] Figure 8 Another computer-implemented sequence for detecting an object and its movement in a vehicle cabin is depicted.

[0021] Figure 9 A computer-implemented sequence for determining the level of danger of an object in a vehicle cabin as described herein is shown.

[0022] Figure 10Shows another computer-implemented sequence for determining the risk level of an object in a vehicle cabin.

[0023] Figure 11 Depicts a computer-implemented sequence for outputting a warning signal as described herein.

[0024] Figure 12 Illustrates the taking of an image as described herein.

[0025] Figure 13 Schematically shows a system for determining the movement of an object in a vehicle cabin as described herein.

[0026] Figure 14 Schematically illustrates another system for determining the movement of an object in a vehicle cabin.

[0027] Figure 15 Schematically shows another system for determining the movement of an object in a vehicle cabin as described herein.

[0028] Figure 16 Schematically shows another system for determining the movement of an object in a vehicle cabin.

[0029] Figure 17 Is a diagram of the internal components of a data processing system included in a system for determining the movement of an object in a vehicle cabin. Detailed Description

[0030] The present disclosure relates to improvements in the safety of vehicles and, in particular, to methods and systems for detecting an object in a vehicle cabin and for determining its movement.

[0031] Figure 1 Depicts the interior of vehicle cabin 1 as seen in the field of view of an imaging system / vehicle sensing system 101 that is part of a system 100 for determining an object in vehicle cabin 1 and its movement as shown in Figure 15 and is installed in cabin 1. In Figure 1 , the imaging system 101, which may include a camera, is located at the rearview mirror of the vehicle's front windshield ( Figure 1 not shown in

[0032] Three people 2, 3, and 4 are sitting in the vehicle. Person 4 is sitting in the driver's seat of the vehicle, person 3 is sitting in the back seat, and person 2 is sitting in the passenger seat adjacent to the driver's seat. Several objects are located within the cabin. Between persons 2 and 4, on the center console 8 of vehicle cabin 1, there is placed a bottle or can 6 such as cola. A box 7 is placed on the back seat of the vehicle. In addition, on top of the backrest of the back seat, there is placed a soft object 5, such as a jacket, which is, for example, being sat on by person 3 in the back seat.

[0033] Figure 2 Depicts a portion of the interior of the vehicle's cabin 1 as seen in the field of view of the imaging system / vehicle sensing system 101 installed in the cabin 1. Figure 2 Shows the Figure 1 box object 7 located on the rear seat. The box 7 is not fixed in its position on the rear seat by any means (such as, for example, Figure 2 the seat belt 9 shown in Figure 2 . When looking more closely at

[0034] Figure 3 is a basic flowchart of a method for detecting objects and their movements in the vehicle's cabin 1 as disclosed herein. The method determines at least one object 5, 6, 7 in the vehicle's cabin 1 based on one or more images showing the interior of the vehicle's cabin 1 in action 10. The method determines several features of the at least one object 5, 6, 7 in action 11, and determines the degree of danger of the at least one object 5, 6, 7 based on the several features in action 12. In response to determining that the degree of danger meets a given criterion, the method outputs a warning signal in action 13. This enhances the safety of the vehicle's driver 4 and passengers 2, 3, which is achieved by detecting these objects and their movements in the vehicle before all objects and their movements in the vehicle can become potentially dangerous projectiles or otherwise negatively affect the safety of driving.

[0035] Images can be obtained from one or more cameras that capture the interior of the vehicle's cabin 1. These cameras can produce visible light images and / or near-infrared images. The cameras can be installed at different positions within the vehicle. At least one camera can be installed in the cabin 1, for example, in the front and rear view mirror positions (as Figure 2 shown), and / or in the center of the dashboard, under the roof, above the second seat row, or in the trunk.

[0036] The image can include 2D images and / or 3D images. The image can also include a color image or a grayscale image. The image can also include a pixel-based image.

[0037] The determination 10 of at least one of the objects 5, 6, 7 can be performed based on the image by using an object detection algorithm. For each of the objects 5, 6, 7 in the cabin 1 of the vehicle, the algorithm will generate output data including image coordinates related to the position of the object 5, 6, 7, and data related to the class and / or type of the object 5, 6, 7. Although related everyday objects can be classified into separate object classes (e.g., phone, laptop, bottle, jacket, bag, etc.), less common objects such as a skateboard or a bicycle can be classified as "other objects".

[0038] The image data can be processed pixel by pixel by the object detection algorithm, which uses segmentation and classification algorithms to identify pixel regions of the objects 5, 6, 7 that may belong to parts of the cabin 1 of the vehicle that are not. In an embodiment, the object detection algorithm can use 3D clustering and segmentation algorithms, where the image data is processed in a preprocessing activity to generate a 3D point cloud or a depth map, e.g., based on time of flight (TOF), structured light, stereo, monocular depth estimation. The 3D data map is compared with a 3D reference model of the cabin 1 of the vehicle to identify potential objects 5, 6, 7. These objects 5, 6, 7 can be further classified in another activity to identify moving objects 5, 6, 7 from fixed cabin elements or persons (e.g., the driver 4 of the vehicle or the (passenger) persons 2 and 3).

[0039] The degree of danger can include a numerical scale, e.g., a scale including numerical values ranging from 0 to 9, where a value of 0 or 1 indicates no danger or negligible danger from the objects 5, 6, 7 within the vehicle cabin, e.g., a child seat fixed in the rear seat, where a value of 2 and 3 indicates, e.g., low danger, e.g., from a smartphone, a value of 5 indicates medium danger, e.g., from a cola bottle located on the center console, a value of 9 indicates high danger, e.g., from a suitcase stored on the rear seat but not, e.g., secured by a seatbelt. In some embodiments, the degree of danger can include a simple three-level indication, including the values 0, 1, and 2, corresponding to "no danger", "medium danger", and "extreme danger", respectively, or even more two-level indications, using binary values 0 and 1, corresponding to the levels "no danger" or "danger", corresponding to "static" or "moving" objects. The selection of an appropriate numerical scale can depend on the object type, the occupancy of the vehicle, and / or the driving state, but can also depend on, e.g., the vehicle type. This enables an accurate determination of the degree of danger based on the vehicle situation.

[0040] When the determined level of danger meets a given criterion, for example when the determined numerical danger level value exceeds the value corresponding to the "no danger" or "negligible danger" level, a warning signal can be output, for example, to warn the driver 4 and / or passengers 2 and 3. Otherwise, no warning signal may be output.

[0041] Outputting the warning signal 13 can include outputting a visual signal, such as a message on the infotainment display in the cabin 1 and / or the driver's dashboard and / or an audio signal. This makes the output suitable for the requirements of the driver 4 and / or passengers 2, 3.

[0042] Depending on the determined level of danger, different warning signals can be output. If no objects 3, 4, 5 are detected in the cabin 1, or as already described, there is no danger or only negligible danger from objects 5, 6, 7, no warning signal is output. If the determined level of danger is low (e.g., objects 5, 6, 7 are smartphones), the output signal can also be completely suppressed, or output without the need for confirmation, for example, by the driver 4 (or passengers 2, 3). If the estimated potential risk is high, a warning signal can be output, for example, via a pop-up message on the vehicle dashboard (smart vehicle) or the touchscreen of the driver 4's smartphone, a notification sound or a location-specific sound, which requires active confirmation by the driver 4 (or passengers 2, 3). The confirmation can include detection and confirmation of the driver 4's (or passengers 2, 3) gesture or head pose or voice command by the vehicle's computing / data processing system 102 ( Figure 15 ) or by the driver 4 (or passengers 2, 3) pressing a button or the touchscreen.

[0043] For potential objects 5, 6, 7, the level of danger can be determined, and a warning signal can be output independently of the movement detection of the objects 5, 6, 7, but based on the type and size of the detected objects 5, 6, 7, such as a large suitcase or a skateboard, so that a warning can be issued to the driver 4 (or passengers 2, 3) before driving off once the vehicle has started. In some embodiments, the vehicle's departure may even be blocked by this warning, and the driver is required to check the loaded objects and confirm that it is safe to start the vehicle.

[0044] In an embodiment, depending on the determined level of danger, different stages of warnings can be used. If a warning signal including a pop-up message that requires confirmation by the driver 4, for example, does not resolve the situation related to the objects 5, 6, 7 causing the danger and / or the determined level of danger further increases, additional more attention-grabbing display messages can be sent to the driver 4. In an embodiment, an audio warning beep with increased intensity can be sent to the driver 4 (or passengers 2, 3).

[0045] In an embodiment, when it is determined that the risk levels of objects 5, 6, and 7 can be determined while the vehicle is in the driving stage and the driver 4 is unable or unwilling to stop the vehicle, the advanced driver assistance system (ADAS) can be instructed to perform actions that cause the vehicle to increase its distance to the vehicle ahead on the street to a defined distance, and / or, if the traffic conditions around the vehicle permit such an action, perform a braking algorithm for less sudden deceleration of the vehicle, and / or reduce the total speed of the vehicle, and / or reduce the total acceleration of the vehicle, and / or drive appropriately in turns and curves.

[0046] In some embodiments, and as Figure 4 shown, the method further includes, in operation 14, tracking the motion states of objects 5, 6, and 7 based on a plurality of images showing the interior of the vehicle cabin 1, wherein determining 15 the risk level is further based on the motion states of the objects. In the case where the vehicle is driving or accelerating, for example increasing its speed, braking, or driving through a curve, the objects stored in the cabin 1 may start to move, such as sliding, rolling, or flying back and forth inside the vehicle cabin. Any such situation may hit a person in the vehicle cabin 1, or obstruct the driver 4 from seeing the surrounding traffic, or obstruct the unobstructed movement of the driver's leg that operates the vehicle pedal (e.g., the brake pedal). Tracking the objects moving in the vehicle while the vehicle is driving or decelerating / accelerating further enhances the safety of the driver 4 and / or the vehicle passengers, especially when the vehicle is in the driving state.

[0047] The motion states of objects 5, 6, and 7 may include the objects sliding or rolling on, for example, the vehicle seat or console or the trunk in the vehicle cabin 1 or from it. The motion may also include the flying of objects 5, 6, and 7 inside the vehicle cabin. The motion may include any other dynamic states of objects 5, 6, and 7, such as tilting, rotating, floating, etc.

[0048] In some embodiments, and as Figure 5 shown, tracking 14 the motion states of the objects includes determining 16 the bounding boxes of the objects in subsequent images. Tracking algorithms can be used to continuously track the positions of objects such as objects 5, 6, and 7 over time. The position can be defined by the bounding box position or reference point (e.g., the object center or other feature points) of an object over time. This enables the analysis of the potential motion of the objects (e.g., objects 5, 6, and 7). The tracking algorithm can utilize a Kalman filter, a particle filter, a machine learning-based method, or other tracking methods.

[0049] Bounding boxes can be used to calculate optical flow information on an input image or a specific region thereof to detect the movement of an object and derive displacement information related to the object (such as objects 5, 6, 7) in the image plane. As an example, trackable features within the bounding box can be detected and descriptors can be used to encode the features. Then the displacement of the descriptor-based features can be tracked. The sparse feature tracking method can be the Kanade-Lucas-Tomasi (KLT) tracker. Moreover, a dense optical flow method can be used. The optical flow calculation is not limited to the object bounding box. Instead, the calculation can be performed in a larger image region or over the entire image and can be used as an input for object detection. This can be performed on a sampled version of the image to reduce the computational complexity or only on a specific part of the image that shows the part of the vehicle cabin 1 where larger and potentially dangerous objects can typically be stored, such as the back seat or trunk of the vehicle.

[0050] Figure 6 Exemplarily shown are several bounding boxes 60, 61, and 62 respectively surrounding objects 5, 6, and 7. The bounding boxes can be defined by processing software applied to the data generated by the imaging system / vehicle sensing system 101( Figure 17 )). Objects 5, 6, and 7 can be clearly seen (as shown) by the imaging system / vehicle sensing system 101, or they can also be occluded to a certain extent. In Figure 6 , the following bounding boxes are determined, and the bounding boxes can be applied to each picture taken by, for example, the imaging system / vehicle sensing system 101:

[0051] · The bounding box 60 surrounding object 6 (such as a cola can);

[0052] · The bounding box 61 surrounding object 7 (such as a box containing a wine bottle);

[0053] · The bounding box 62 surrounding object 5 (such as a jacket).

[0054] The bounding box can be defined by at least the width, height, center x position, and center y position relative to an image taken by, for example, a camera of the imaging system / vehicle sensing system 101. However, depending on the occupancy of the vehicle cabin 1 and the structural boundary conditions of the cabin 1 itself, other shapes for the boundary region may be more suitable than a rectangular surface for successfully tracking the movement of objects (such as objects 5, 6, 7). In some embodiments, the boundary region surrounding an object includes a circular surface, which generally defines a closer region around the object compared to a rectangular surface. In some further embodiments, these regions include boundary regions of any spatial dimension, such as a three-dimensional boundary region (also refer to Figure 5) This is useful in cases where the vehicle sensing system 101 supports three-dimensional imaging of the vehicle cabin 1 and the person inside cabin 1. Additionally, if the vehicle sensing system 101 does not provide depth information, the 3D bounding box can be estimated, for example, through approximation from 2D sensors and machine learning methods. In a further embodiment, any shape of the bounding region suitable for indicating an object under a given sensing technology can be used, or alternatively, instead of defining the surrounding region, only the object key points are determined. In an embodiment, pixel-by-pixel object segmentation can be used for object detection, assigning object class labels and object IDs to individual pixels. For determining the bounding region, such as the bounding box around an object, the YOLO algorithm, such as YOLO 2, can be used. Other machine learning algorithms or traditional image processing and recognition algorithms can also be used.

[0055] In an embodiment, the bounding box can include a three-dimensional (3D) bounding box around an object (such as objects 5, 6, 7). If three-dimensional (3D) object detection is available, tracking the movement of the objects (such as objects 5, 6, 7) can also be done in the 3D coordinate system representing the interior of the vehicle cabin 1. By comparing the carved-out volumes of two consecutive time frames, the movement of the objects (such as objects 5, 6, 7) can be derived from the change in the center of gravity of objects 5, 6, 7, for example, using instance segmentation groups, enabling the tracking of the movement of a single object within a group or cluster of objects. In the absence of using instance segmentation groups, the change in the 3D bounding box indicates the overall movement of the group of objects inside the vehicle cabin 1. By additionally using radar-based information (also refer to Figure 5 ), the speed information from the radar sensor can be included in the object movement determination by fusing the speed determined based on vision-based tracking using a visible light camera and / or an infrared light camera with the speed determined based on the radar-based information. The radar sensor can provide a radial Doppler estimate, which can be fused with the laterally detected movement of the vision-based (visible light / IR) camera system to obtain a complete 3D velocity curve.

[0056] In an embodiment, the bounding box around an object (such as objects 5, 6, 7) can include data related to the type or class of the object. This enables reliable and dynamic determination of the degree of danger of the corresponding object.

[0057] In some embodiments and as Figure 5 shown, tracking the motion state of objects 5, 6, 7 includes performing a machine learning algorithm and / or based on a three-dimensional (3D) reference model of the vehicle cabin 1, enabling precise calculation and prediction of the movement of objects (such as objects 5, 6, 7).

[0058] In an embodiment, a system 100 for detecting an object and its movement in a vehicle cabin 1 using a 3D camera can be configured to estimate the height position of an object above a given seat or headrest (e.g., the driver's seat). An object positioned in the cabin 1 at a height above the headrest can be considered to have the highest risk, corresponding to a determined level of danger such as reaching "high". 3D image data related to a height above the seat (e.g., the driver's seat) can be obtained from 3D spatial position information and a bounding box related to the space above the headrest of the driver's seat. Regarding 2D image data, the 2D bounding box of the detected object can be compared with the 2D positions of the seat and the headrest. By using cabin-specific calibration and considering the viewing point and perspective characteristics of the camera, the relative height position of the object with respect to a person's head can be determined.

[0059] In some further embodiments, an adjustable seat, such as the driver's seat and a passenger seat near the driver, can be detected based on an image by using an object detection algorithm. This enables the detection of an object belonging to the cabin (e.g., a seat headrest) by combining the detection of an object (e.g., objects 5, 6, 7) in the cabin in one algorithm (e.g., a neural network model).

[0060] In some further embodiments, machine learning-based methods include neural networks and / or supervised learning and / or unsupervised learning and / or reinforcement learning and / or decision trees. In an embodiment, a danger level classifier as a neural network includes a first input channel configured to receive one (or more) current images from a camera system such as an imaging system / vehicle sensing system 101. The danger level classifier is trained to calculate a value indicating the danger level, such as a value from 0 to 9 from the above numerical scale, representing the danger level from "no danger" to "high danger". The danger level classifier can then also output the calculated value, and in a further embodiment, output the calculated value together with a confidence score. This reduces the possibility of incorrectly determining the danger level of an object (such as objects 5, 6, 7) based on the detected position and / or movement of an object in the cabin 1, for example, because the system 100 for detecting an object and its movement in the vehicle cabin 1 can also form the basis for the current determination of the danger level of an object such as objects 5, 6, 7 with the corresponding data for the early and correct determination of the object danger level, thereby enhancing the reliability of the system 100.

[0061] A three-dimensional (3D) reference model of the vehicle cabin 1 can include, for example, a 3D reference model of the position of the center console relative to the imaging system / vehicle sensing system 101, and the positions of the B-pillar, C-pillar, and rear window relative to the imaging system / vehicle sensing system 101. The various positions can be derived from the CAD model of the cabin 1.

[0062] In an embodiment, the semantic segmentation and depth estimation data of the vehicle cabin 1 can be used to create a 3D map of the vehicle cabin, which can then be used to optimize the determination of the level of danger. If loose items or luggage are detected on the rear seat or the front passenger seat of the vehicle, and the loose items or luggage may fit into a safe location in the cabin 1, such as the trunk, a message is sent to the driver 4 using the semantic segmentation and depth estimation data associated with the vehicle cabin 1.

[0063] In some embodiments and as Figure 7 shown, the determination 11 of the characteristics of the object is based on 18 a reference feature group. This enables the reliable determination of the type and characteristics of the objects 5, 6, 7 based on the predetermined characteristics for the object class.

[0064] By determining the current position of the objects (such as objects 5, 6, 7) in the cabin 1 along with the object type / class, the level of danger of the objects relative to one or more occupants within the cabin 1 can be determined independently of whether the objects are fixed in their positions.

[0065] In an embodiment, the detected objects can be classified into specific risk categories. For example, jackets, pillows, or blankets (usually soft objects) can be grouped into a lower risk category, where a lower level of danger can typically be determined, while boxes, suitcases (usually hard objects) can be grouped into a higher risk category with a default base level of danger, such as "medium", which can be exceeded depending on, for example, the occupancy of the cabin, the driving condition of the vehicle, whether the object is fixed in its position, etc. In a further embodiment, the level of danger will be directly estimated as a numerical value in the case of not being grouped into a risk category.

[0066] Since the bounding boxes around the detected objects (such as objects 5, 6, 7) may differ between consecutive frames although there is no movement of the objects, a feature-based tracking algorithm can additionally be applied to accurately track the movement of the detected objects 5, 6, 7, rather than relying solely on the detection accuracy of the previous step. This enables the reliable determination of the type of the objects (such as objects 5, 6, 7), which in turn enables a more reliable determination of the level of danger of the corresponding objects.

[0067] The reference feature group can depend on the type of the objects (such as objects 5, 6, 7). In the case of, for example, a mobile phone or a notebook, the reference features can include the shape of the object (usually including a flat rectangular shape) and the dimensions that the object typically has. In the case of a suitcase, as Figure 3 object 7 in shows, the generally rectangular form and several defined dimensions for a suitcase can also include the corresponding group. In the case of a skateboard and a bicycle, the form of the wheels and frame components can form part of the corresponding feature group. In a further embodiment, it can include for cans and bottles (such asFigure 1 and Figure 6 the cylindrical form of the feature group of the object 6).

[0068] In an embodiment, the objects 5, 6, 7 can also be classified as high-value based on the feature group (e.g., ancient books, porcelain vases, bowls or pots with food or fluids which, in the event of unexpected movement, may splash their contents onto the seat, e.g., the back seat of a vehicle), and although the risk of becoming dangerous during an accident or hard braking may be low for passengers and thus the determined level of danger is "low", however, for the driver 4, it is useful to receive an indication of the potential value of the objects 5, 6, 7 through the system 100, especially when the driver is not aware of the value of the said objects. Thus, a message with a warning signal can be output, the warning signal being that a potentially valuable object is moving during driving and may be damaged.

[0069] In a further embodiment, the objects 5, 6, 7 can be classified as pets by the system 100, e.g., a dog lying on the back seat is classified as being exposed to danger when not fastened to the back seat in some way. The system 100 can also classify pets such as dogs, cats, rabbits, etc. as being exposed to danger when not secured in, e.g., a pet carrier.

[0070] In some embodiments and as Figure 8 shown in, the method further includes: in operation 20, determining at least one person in the vehicle cabin 1 based on the one or more images; in operation 21, determining the position of at least one person in the vehicle cabin 1; wherein determining the level of danger 12 is also based on the position of the at least one person. This enables the level of danger to be determined for each individual occupying the vehicle cabin 1, especially for autonomous taxis, valet parking, etc.

[0071] Referring to Figure 1 and Figure 6 , the determination 20 can result in there being three people in the vehicle cabin 1, where person 4 is sitting in the driver's seat, person 2 is sitting in the front seat next to the driver and person 3 is sitting in the back seat. The objects 5, 6, 7 are located at different positions on the center console or the back seat and thus are at different positions relative to the vehicle occupants 2, 3, 4. Thus, each of the objects 5, 6, 7 can pose a different danger to the occupants, which can be reflected in the correspondingly determined level of danger.

[0072] In some embodiments and as Figure 9As shown in, the determination of the degree of danger 12 is also based on: an estimate 30 of the impact of the object on driving safety; an estimation 31 of the weight of the at least one object; the driving speed 32 of the vehicle; the position 33 of the at least one object in the vehicle; determining 34 whether the object is fixed in its position in the vehicle, and in the case where the determination is affirmative, reducing the degree of danger; whether the at least one person is the driver 4 or a passenger 2, 3 of the vehicle. This enables a flexible determination of the degree of danger based on object characteristics, how it is positioned and fixed within the cabin 1, the occupancy of the vehicle cabin 1, and the driving conditions of the vehicle.

[0073] Referring again to Figure 1 or Figure 6 , the impact of objects 5, 6, 7 on driving safety can be different. Objects 5 and 7, representing for example a jacket and a suitcase positioned on the back seat, may have a relatively small impact on overall driving safety as they are likely to have much less impact on the driver 4 of the vehicle than object 6, which is placed on the middle console and may thus have a greater impact on the driver's ability to drive the vehicle safely, for example when sliding or rolling in the footwell area of the cabin 1 (discussed in more detail below), although having a much lower estimated weight than for example object 5 and object 7.

[0074] However, a jacket represented by object 5, or other types of clothing or blankets, although they typically have a moderate weight and a soft nature, can pose a significant danger to the driver 4 for example, as these types of objects can easily fly around in the vehicle cabin 1, especially in the case of sharp acceleration or deceleration of the vehicle, such as during hard braking, and may eventually land on the driver 4 and may obscure the driver's view of the street and the surrounding traffic conditions. Therefore, even for an object 5 such as a jacket, a high degree of danger can be determined, especially with respect to the driver 4.

[0075] Furthermore, although not shown in the figure, an object located at a position comparable to the height of the driver 4 (or a passenger 2, 3) can pose a greater danger, especially to the driver 4 of the vehicle, as an impact of the object on the person's head is generally more difficult for the person (e.g., the driver 4) to endure and can also more easily block the unobstructed view of the driver 4. Therefore, the object position of an object (e.g., objects 5, 6, 7) at a height level comparable to the height position of the head of the vehicle occupant (e.g., the height position of the head of the driver 4) can lead to the determination of a high degree of danger for the object.

[0076] Further checking Figure 1 , Figure 2 or Figure 6, all objects 5, 6, 7 are not fixed in their positions. Taking object 7 as an example, it represents a suitcase with a weight of, for example, 20 to 30 kg. Figure 1 , Figure 2 or Figure 6 the inspection shows that the suitcase 7 is not fixed in its position, for example, by a safety belt 9 (see Figure 2 ). This may affect the determination of the risk level of object 7. When fixed in its position, the risk level of object 7 can be determined to be lower than when object 7 is not fixed in its position.

[0077] In an embodiment, the system 100 can detect a protection system between the trunk and the seat of the vehicle, such as for a passenger car (net or protection bar), and can suppress the output of a warning signal when the net and / or the protection bar is detected.

[0078] Further referring to Figure 1 or Figure 6 , for the same object, such as object 6 on the center console (as described above), the risk level determined for the driver 4 can generally be determined to be higher compared to passengers 2, 3, because the driver has the task of driving the vehicle and thus bears an increased responsibility for the safety of passengers 2, 3 and the traffic around the vehicle. In a first example, for example, the risk level for object 6 (representing, for example, a cola can) can be determined to be low because, for example, when an emergency brake is applied to the vehicle, the can 6 may move in the forward direction and thus miss the driver 4. Even when hitting the driver 4, for example, when moving during a right turn curve of the vehicle, due to the small size and light weight of the can 6, the risk that the driver 4 may be disturbed or even injured is low. However, the risk that the can 6 may slide or roll into the footwell area and block the driver's pedal (such as the brake pedal) can be determined to be high because the position of the can 6 on the center console is generally near the footwell area. Therefore, on, for example, the above three - level risk scale (including the values 0, 1, and 2), for the driver 4, the total risk level of the can 6 can be determined to be "high".

[0079] On the other hand, when determining the risk level for object 7 representing, for example, a larger suitcase, since the suitcase 7 has a larger size and heavier weight compared to the can 6, the risk level associated with the suitcase and determined for the driver 4 seems to be higher than that of the can 6 at first glance. However, in the case of movement, the chance that the suitcase 7 passes between or above the two front seats and hits the driver 4 or even further slides into the footwell area due to its size is considered to be low. Therefore, the determined risk level of the suitcase with respect to the driver 4 is "medium" on the three - level risk scale.

[0080] Further referring to Figure 1 or Figure 6, and now consider a passenger 2 sitting in the front seat near the driver 4. Due to the fact that the collision of the moving can 6 will not seriously injure the passenger 2 and the passenger 2 does not assume the driving responsibility, the determined risk level for the can 6 relative to the passenger 2 can be "low".

[0081] In some embodiments and as Figure 10 shown, when the vehicle starts, the risk level is initially determined 22, and is dynamically redetermined 23 based on the driving speed of the vehicle. This enables the determined risk level to flexibly adapt to the current driving conditions of the vehicle.

[0082] Referring again to Figure 1 or Figure 6 , the object 6 representing a cola can located on the center console may pose different types of risks when the vehicle starts and when the vehicle is driving at a normal driving speed. At the beginning, the acceleration is usually moderate, and the cola can 6 is most likely to remain in place (when not being consumed by the passenger 2, for example). Therefore, the determined risk level can be "low". However, when in the driving state, the vehicle and thus the can 6 may be subject to a higher acceleration, resulting in a higher likelihood of sudden displacement from its position on the center console and sliding or rolling into the footwell space of the cabin 1. Therefore, the risk level determined for the can relative to the driver 4 can be "high".

[0083] In some embodiments and as Figure 11 shown, the output 13 of the warning signal is also based on the occupancy state 36 of the vehicle, or prevents the vehicle from being started and / or operated 37 based on the risk level, or outputs one or more control signals 38 that change the operating state of the vehicle based on the risk level when the vehicle is driving. This enables the flexible output of warning signals based on the occupancy state, and also enables the activation of additional safety measures based on the current driving state of the vehicle.

[0084] Referring again to Figure 1 or Figure 6 , the warning signal can depend on whether only the driver 4 is sitting in the vehicle cabin 1, or whether there are also passengers 2, 3 occupying the cabin 1. In the first case, the warning signal can be directed only to the driver, for example, by displaying a message on the touch screen of the instrument panel indicating the risk level related to the driver 4 determined from the objects 5, 6, 7. In the case of other occupants of the vehicle cabin 1, additional warning signals can be output, for example, by also indicating to the driver 4 on the touch screen the corresponding risk levels for the passengers 2, 3. In addition, in the case of other occupants in the vehicle cabin 1, warning signals can also be output at the positions where the occupants are seated, for example, at the positions of the passengers 2, 3, for example, in the case of visual signals, such as a flashing red light or a corresponding message on the local touch screen.

[0085] In addition, in the case of a heavy and large volume object, such as Figure 2 the suitcase 7 in [reference document], the danger can be determined as "high". Based on this determination, the start of the vehicle can be blocked until the determined danger level has been reduced, for example to "medium", due to a remedial action performed by the driver 4 or the passengers 2, 3. Such a remedial action can include, for example, as Figure 2 shown, using the seat belt 9 to fix the suitcase 7 in its position. When the vehicle is being driven, the danger level determined for the suitcase may again become "high" during driving, for example in the case where the passenger 3 would loosen the seat belt 9 to open the suitcase. The determined danger level jump can then again change from "medium" to "high", also causing a change in the operating state of the vehicle, such as decelerating the vehicle, depending on the traffic situation, to a speed at which the determined danger level again changes back to "medium".

[0086] In some embodiments and as Figure 11 shown, the warning signal is also based on whether at least one person to whom the output signal is directed is the driver of the vehicle or the passenger 39.

[0087] The warning signal directed to the driver 4 can include information related to the danger level of the driver 4, information related to the individual danger levels of the passengers in the vehicle compartment 1 (e.g., information related to the danger levels of the passengers 2, 3 respectively), and information related to appropriate remedial actions. In addition, information related to a change in the operating state of the vehicle can be included in the warning signal directed to the driver 4, such as information indicating that the vehicle can decelerate in the case where the danger level has been determined to be "high". On the other hand, the warning signal directed to a passenger (e.g., the passengers 2, 3) can include only information related to the danger level of that passenger.

[0088] In an embodiment, the warning signal can also be based on additional information from external perception based on external sensors, such as the perception of the surrounding traffic situation, and / or a map, enabling prediction of short-term or long-term driving trajectories and providing additional information for determining the danger level. This enables the output of a warning signal even before any movement of the object (e.g., objects 5, 6, 7) within the compartment 1 can be tracked.

[0089] In some embodiments and as Figure 12As shown, one or more camera images 56 include images captured by one or more visible light cameras 50 and / or infrared light cameras 51 and / or in-cabin radars 52 located within the vehicle cabin 1. The images 56 may also include images captured by a time-of-flight camera 53 or a structured light camera 54 or a stereo camera 55. The method may also include irradiating the vehicle cabin 1 with infrared light and / or radio waves. This enables flexible determination of the motion state of objects within the vehicle cabin 1, adapting to changing lighting and occupancy conditions within the cabin 1, especially under low light and night conditions.

[0090] Visible light cameras 50 / infrared light cameras 51, including for example RGB-IR sensors, offer significant advantages as they allow capturing day and night images with the same sensor. RGB-IR image sensors operate in two ranges: the visible spectral range and the IR spectral range. By typically dedicating 25% of its pixel array pattern to infrared (IR) and 75% to RGB, the RGB-IR sensor can capture RGB and IR images simultaneously. The RGB-IR image sensor does not have any dedicated filters to improve signal quality: it measures everything and extracts the image IR and RGB. This results in some optical problems as the signals in both the IR and RGB domains are contaminated. All pixels in the pixel array of the RGB-IR sensor can receive the IR signal. This means that not only IR pixels receive the IR signal, but also RGB pixels receive the IR signal. Additionally, IR pixels also receive a small amount of visible light signal. In the automotive industry, infrared signals play a key role in automated image processing applications such as surveillance and driver monitoring. Typically, these applications require pure infrared images and they cannot operate on the raw RGB-IR input.

[0091] In an embodiment, the imaging system 101 within the vehicle cabin 1 includes cameras operating in the visible electromagnetic spectrum and / or the infrared electromagnetic spectrum. Separate cameras operating in the visible and infrared electromagnetic spectra can provide images based on pure visible spectra and pure infrared images.

[0092] By using, for example, pixel images generated by an indirect time-of-flight (TOF) camera 53, reference images can be added to the vehicle's image processing system. Images obtained from the TOF camera and the in-cabin radar can additionally be used as a authenticity check for safety-critical applications, such as determining the motion of objects (e.g., objects 5, 6, 7) within the cabin 1. Additionally, structured light and stereo imaging can be used to generate a 3D depth map of the vehicle cabin 1, which can be added to the vehicle's image processing system.

[0093] As Figure 13As shown, image data is generated by the imaging system 101 by irradiating at least the driver 41, one or more passengers 42, and at least one object 43. The imaging system includes a camera 40 mounted at the rearview mirror within the vehicle cabin 1. This enhances the accuracy of occupancy detection and object motion detection within the vehicle cabin because the infrared reflection characteristics of the points on the objects 43 and people 41, 42 occupying the vehicle are also used for the motion detection of the object 43 and the corresponding determination of their degree of danger.

[0094] In some embodiments and as Figure 14 shown, at least one of the one or more cameras 44 monitors the footwell area 45 of the vehicle 300. This further enhances the safety of the vehicle 300 and its occupants (such as people 2, 3, 4), especially during the driving of the vehicle 300, where it is important that the driver can operate the pedals, such as the brake pedal, unhindered to perform the safe operation of the vehicle 300. The camera 44 can be mounted at an appropriate position within the vehicle cabin 1, such as Figure 14 shown under the steering column or under the vehicle seat (such as the driver's seat). In an embodiment, the camera 44 can include radar.

[0095] According to one aspect and as Figure 15 shown, a system 100 for determining the motion of an object in a vehicle cabin of a vehicle is provided. The system 100 includes a sensor system 101, a data processing system 102, and an interface 103 for outputting a warning signal. The system is configured to perform any of the methods described in the above paragraphs.

[0096] Figure 16 An embodiment of the detection system 100 for detecting an object and its motion in the vehicle cabin 1 is shown, which implements the functions described herein. The imaging / vehicle sensing system 101 can include a visible light camera 50, an infrared light camera 51, and additional components such as an in-cabin radar 52 that capture an image 56 of the interior of the vehicle cabin 1. The data processing system 102 can perform an action 10 of determining the actions of the objects (such as objects 5, 6, 7) within the cabin based on the image 56. The data processing system 102 can also perform an action 11 of determining several characteristics of at least one of the objects 5, 6, 7 and an action 12 of determining the degree of danger of at least one of the objects 5, 6, 7 based on the several characteristics. In response to determining that the degree of danger meets a given criterion, the interface 103 outputs a warning signal in action 13.

[0097] According to one aspect, a vehicle 300 is provided that includes the system 100 for determining the movement of an object in the vehicle as described in the preceding paragraph, and the system performs any one of the methods described within the present disclosure. In the present disclosure, the term "vehicle" includes all types of vehicles, such as cars, autonomous vehicles, trams, railway vehicles, and the like.

[0098] Figure 17 is an illustration of internal components of a data processing system 200 that implements the functions described herein, such as Figure 15 the data processing system 102. The data processing system 200 may be located in the vehicle and includes at least one processor 201, a user interface 202, a network interface 203, and a main memory 206 that communicate with each other via a bus 205. Optionally, the data processing system 200 may further include a static memory 207 and a disk drive unit (not shown), which also communicate with each other via the bus 205. A video display, an alphanumeric input device, and a cursor control device may be provided as examples of the user interface 202.

[0099] In addition, the data processing system 200 may further include a designated sensing interface 204 that communicates with the imaging / sensor system 101 of the vehicle 300. Alternatively, the data processing system 200 may communicate with the imaging / sensor system 101 via the network interface 203. The imaging / sensor system 101 is used to generate interior compartment data for determining the movement of objects (such as objects 5, 6, 7 in the vehicle compartment 1). The data processing system 200 may also be connected to a database system (not shown) via the network interface, where the database system stores at least a portion of the images required to provide the functions described herein.

[0100] The main memory 206 may be a random access memory (RAM) and / or any other volatile memory. The main memory 206 may store program code for depth estimation system control 208 and determining the correct depth estimation 209. The memory 206 may also store additional program data required to provide the functions described herein. A portion of the program data 210, the determination of the correct depth estimation 209, and / or the depth estimation system control 208 may also be stored in a separate, such as cloud memory, and at least partially executed remotely. In such an exemplary embodiment, the memory 206 may store the depth estimation and the corrected depth estimation in a buffer 211 according to the methods described herein.

[0101] According to one aspect, a computer program including instructions is provided. When the computer executes the program, these instructions cause the computer to perform the methods described herein. The program code implemented in any of the systems described herein can be distributed individually or jointly as a program product in various different forms. Specifically, the program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform aspects of the embodiments described herein.

[0102] A computer-readable storage medium that is non-transitory in nature can include volatile and non-volatile, as well as removable and non-removable tangible media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. The computer-readable storage medium can also include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid state memory technologies, portable compact disc read-only memory (CD-ROM) or other optical storage, magnetic tape cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be read by a computer.

[0103] A computer-readable storage medium itself should not be construed as a transitory signal (e.g., radio waves or other propagating electromagnetic waves, electromagnetic waves propagating through a transmission medium such as a waveguide, or electrical signals transmitted through a wire). The computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device or another device, or downloaded to an external computer or external storage device via a network.

[0104] It should be understood that while specific embodiments and variations are described herein, further modifications and substitutions will be apparent to those skilled in the relevant art. In particular, examples are provided by way of illustration of the principles, and a variety of specific methods and arrangements are provided for making those principles effective.

[0105] In certain embodiments, the functions and / or actions specified in the flowcharts, sequence diagrams, and / or block diagrams can be reordered, serially processed, and / or concurrently processed without departing from the scope of the present invention. Additionally, any flowchart, sequence diagram, and / or block diagram can include more or fewer blocks than shown in accordance with the embodiments of the present invention.

[0106] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the embodiments of the present disclosure. It should also be understood that when used in this specification, the terms "comprises" and / or "comprising" specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. Further, insofar as the terms "comprise", "have", "include", "contain" or variants thereof are used in the detailed description or claims, these terms are intended to be inclusive in a manner similar to the term "comprising".

[0107] While the description of the various embodiments has illustrated all of the present invention and while these embodiments have been described in considerable detail, it is not the intention of the applicant to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and modifications will be obvious to those skilled in the art. Accordingly, the invention in its broader aspects is not limited to the specific details, representative devices and methods, and illustrative examples shown and described. The described embodiments are, therefore, to be considered as illustrative for the purpose of teaching the general features and principles and not as limiting the scope as defined by the appended claims.

Claims

1. A computerized method for vehicle cabin safety, the method comprising the following steps: determining at least one object in the vehicle cabin based on the one or more images showing an interior of the vehicle cabin; determining a plurality of characteristics of the at least one object; determining a dangerousness level of the at least one object based on the plurality of characteristics; In response to determining that the risk level meets a given criterion, a warning signal is output.

2. The method according to claim 1, further comprising: tracking a motion state of the object based on a plurality of images showing the interior of the vehicle cabin; Therein, determining the degree of risk is additionally based on the movement state of the object.

3. The method according to claim 2, wherein: Tracking the motion state of the object includes determining a bounding box for the object in subsequent images.

4. The method according to claim 2 or claim 3, wherein: Tracking the motion state of the object includes: executing a machine learning algorithm and / or based on a three-dimensional 3D reference model of the vehicle cabin.

5. A method according to any one of the preceding claims, wherein: Determining the features of the object is performed based on a reference feature set.

6. The method according to any one of the preceding claims, further comprising: determining at least one person in the vehicle cabin based on the one or more images; determining a position of the at least one person in the vehicle cabin; Wherein determining the risk level is also based on the location of the at least one person.

7. A method according to any one of the preceding claims, wherein: The determination of the stated degree of risk is also based on: estimating the impact of the object on driving safety; estimating a weight of the at least one object; the travel speed of the vehicle; the position of the at least one object in the vehicle; determining whether the object is secured in the position in the vehicle, and if the determination is positive, reducing the risk level; The at least one person is a driver or a passenger of the vehicle.

8. A method according to any one of the preceding claims, wherein: The level of hazard is initially determined when the vehicle is started and is dynamically re-determined based on the speed of travel of the vehicle.

9. A method according to any one of the preceding claims, wherein: Output the warning signal: also based on an occupancy state of said vehicle; including preventing the vehicle from being started and / or operated based on the risk level; This includes outputting one or more control signals that change the operating state of the vehicle based on the risk level when the vehicle is traveling.

10. The method according to any one of claims 6 to 9, wherein: Outputting the warning signal is also based on whether the at least one person to whom the output signal is directed is a driver or a passenger of the vehicle.

11. A method according to any one of the preceding claims, wherein The one or more images include images captured by one or more visible light cameras and / or infrared cameras and / or radars in the vehicle cabin, and / or the one or more images include images captured by a time-of-flight camera, a structured light camera, or a stereo camera; The method also includes illuminating the vehicle compartment with infrared light and / or radio waves.

12. The method according to claim 11, wherein: At least one of the one or more visible light cameras and / or infrared light cameras monitors a footwell area of ​​the vehicle.

13. A system for determining motion of an object in a vehicle compartment in a vehicle, the system comprising: - sensor system, - data processing systems, - an interface for outputting a warning signal, The system is configured to perform the method according to any one of claims 1 to 12.

14. A vehicle comprising a system according to claim 13.

15. A computer program product comprising instructions which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 12.

Images