Systems and methods for detecting and mitigating unsafe conditions in a vehicle

Abstract

Systems and methods for detecting and mitigating unsafe conditions in a vehicle. A system for detecting and mitigating unsafe conditions in a vehicle includes an image sensor configured to generate and output image data of one or more seats in a cabin of the vehicle, and a processing system operably connected to the image sensor and including at least one processor. The processing system is configured to receive the image data from the image sensor, process the image data to determine a location of at least one passenger in the cabin, detect that the at least one passenger is located outside of the one or more seats based on the determined location of the at least one passenger in the cabin, and in response to detecting that the at least one passenger is located outside of the one or more seats, operate at least one component of the vehicle in a predefined manner.

Description

[0001] Cross-references to related applications

[0002] This application is related to U.S. application Ser. No. 16 / 705,978, filed on the same date as the present application, entitled “System and Method for Detecting Abnormal Settling of Passengers in a Vehicle,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to vehicle cabin systems and, more particularly, to a system for detecting an unsafe condition caused by a passenger leaving his or her respective seat in a vehicle. Background Art

[0004] As technology advances toward autonomous driving, future cars will no longer have human drivers. However, this lack of a human driver presents a host of new challenges. In particular, the car itself may need to take on the task of understanding its internal state without a human driver.

[0005] More specifically, unsafe conditions can arise when a passenger is moving within the cabin and out of their seat. In conventional vehicles, the driver can identify unsafe situations and take necessary actions to mitigate them. However, in autonomous vehicles, there is no driver to identify and remedy unsafe conditions. As such, improvements to systems and methods for identifying and remediating unsafe conditions when a passenger is out of their seat would be beneficial. Summary of the Invention

[0006] In one embodiment, a system detects and mitigates an unsafe condition in a vehicle. The system includes an image sensor configured to generate and output image data of one or more seats in a cabin of the vehicle, and a processing system operably connected to the image sensor and including at least one processor. The processing system is configured to receive the image data from the image sensor, process the image data to determine a position of at least one passenger in the cabin, detect that the at least one passenger is outside the one or more seats based on the determined position of the at least one passenger in the cabin, and operate at least one component of the vehicle in a predefined manner in response to detecting that the at least one passenger is outside the one or more seats.

[0007] In an embodiment, the processing system is further configured to estimate a pose of the at least one passenger in the processing of the image data to determine a corresponding passenger position point for each of the at least one passenger.

[0008] In another embodiment, the processing system is further configured to estimate a hip position of each passenger in the estimation of the posture, and determine the corresponding passenger position point as a midpoint between the estimated hip positions.

[0009] In another embodiment, the processing system of the system may be further configured to estimate a shoulder position of each passenger in the estimation of the posture, and determine the corresponding passenger position point as a midpoint between the estimated shoulder positions.

[0010] The processing system may be further configured to detect that the at least one passenger is outside the one or more seats in response to at least one of the corresponding passenger location points being outside a first seat boundary of the at least one seat within a predetermined first time period.

[0011] In another embodiment, the at least one component comprises a drive system of a vehicle, and operating the at least one component of the vehicle in a predefined manner further comprises operating the drive system of the vehicle to slow or stop the vehicle.

[0012] In yet another embodiment, the at least one component includes a speaker configured to project audio into the vehicle cabin, and operating the at least one component of the vehicle in the predefined manner further includes operating the speaker to generate an audible alert.

[0013] In some embodiments, the at least one component includes at least one of a display and a light arranged in or adjacent to the vehicle cabin, and operating the at least one component of the vehicle in a predefined manner further includes operating at least one of the display and the light to generate a visual alert.

[0014] In another embodiment, a method for detecting and mitigating an unsafe condition in a vehicle includes receiving, with a processing system, image data of one or more seats in a cabin of the vehicle from an image sensor, and processing, with the processing system, the image data to determine a position of at least one passenger in the cabin. The method further includes detecting, with the processing system, that the at least one passenger is located outside the one or more seats based on the determined position of the at least one passenger in the cabin, and operating, with the processing system, at least one component of the vehicle in a predefined manner in response to detecting that the at least one passenger is located outside the one or more seats. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic top view of a vehicle having a cabin monitoring system with a passenger stability monitoring system and a seat exit detection system.

[0016] Figure 2 yes Figure 1 Schematic diagram of the components of a vehicle and passenger stability monitoring system.

[0017] Figure 3 Yes Operation Figure 1 A flow chart of a method for a passenger settlement monitoring system to generate an alert if a passenger is settling into a vehicle in an abnormal manner.

[0018] Figure 4 Yes Operation Figure 1 Flowchart of a method for mitigating a detected abnormal stability event by a vehicle.

[0019] Figure 5 yes Figure 1 Schematic diagram of a vehicle and components of a passenger exit seat detection system.

[0020] Figure 6 Yes Operation Figure 1 Flowchart of a method for a passenger departure seat detection system to generate an alert in the event that a passenger leaves his or her corresponding seat.

[0021] Figure 7 Yes Operation Figure 1 A flow chart of a method for a vehicle to mitigate a detected passenger leaving seat event.

[0022] Figure 8a yes Figure 1 Images of passengers on seats in a vehicle.

[0023] Figure 8b yes Figure 8a Images of passengers, Figure 8b Illustration of passenger location points and seat boundaries.

[0024] Figure 9a There are two passengers Figure 1 The image of the corresponding seat of the means of transport, Figure 9a Diagram showing the passenger's shoulder position, hip position, and passenger position points.

[0025] Figure 9b is the image of the two passengers in their respective seats, Figure 9b Illustration of passenger hip position and seat boundaries.

[0026] Figure 10a is an image of a passenger sitting between the seat boundaries.

[0027] Figure 10b are images of two passengers, one of them is inside the seat boundary and the other of them is outside the seat boundary. DETAILED DESCRIPTION

[0028] For the purpose of promoting an understanding of the principles of the embodiments described herein, reference is now made to the drawings and descriptions in the following written specification. The references are not intended to limit the scope of the present subject matter in any way. The present disclosure also encompasses any changes and modifications to the illustrated embodiments, as would normally occur to one skilled in the art, and includes further applications of the principles of the described embodiments.

[0029] The various operations may be described sequentially as multiple discrete actions or operations in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be interpreted as implying that the operations are necessarily order-dependent. In particular, the operations may not be performed in the order presented. The described operations may be performed in an order different from that of the described embodiments. In additional embodiments, various additional operations may be performed and / or the described operations may be omitted.

[0030] The terms "comprising," "including," "having," and the like as used with respect to embodiments of the present disclosure are synonymous. As used herein, the term "approximately" refers to a value within ±10% of a reference value.

[0031] Figure 1 A simplified block diagram of a vehicle 100 is shown having a cabin 104, a cabin monitoring system 108, a vehicle electronic control unit ("ECU") 112, and a drive system 116. As discussed in detail below, the cabin monitoring system 108 is configured to monitor the position of passengers in the vehicle 100 and determine whether the passenger is properly settled in the vehicle and whether the passenger has left his or her corresponding seat. The cabin monitoring system 108 improves upon conventional vehicle monitoring systems by advantageously using key points of a vehicle passenger to determine whether a passenger is in a seat in the vehicle. Additionally, the cabin monitoring system 108 enables generation of a signal corresponding to a passenger's failure to properly settle into the vehicle within a predetermined time period based on the passenger's position, thereby enabling the vehicle ECU 112 to advantageously take action to mitigate problems caused by the passenger's failure to settle into a vehicle seat. In particular, the vehicle ECU 112 can mitigate problems caused by a passenger's failure to secure themselves in a vehicle seat by, for example, preventing the vehicle from moving, operating speakers, lights, or displays to warn passengers that they are not properly secured in their seats, instructing passengers on how to properly sit in the vehicle, notifying passengers that the vehicle cannot move until the passengers are properly seated, and / or communicating with a remote server to allow human intervention.

[0032] Additionally, the cabin monitoring system 108 is an improvement over conventional vehicle monitoring systems by advantageously providing an accurate way to determine whether a passenger has left their respective seat without duplicate alarms or false positive alarms. Furthermore, the cabin monitoring system 108 advantageously provides for generating a signal corresponding to a passenger leaving his or her respective seat and communicating the signal to the vehicle ECU 112 so that the vehicle ECU 112 can take action to mitigate an unsafe condition caused by the passenger leaving his or her respective seat. As used herein, the phrase "unsafe condition" refers to a situation in which a passenger is located in a position in the vehicle (e.g., outside the vehicle seat) where the passenger is at increased risk of injury in the event of a sudden vehicle maneuver or accident. The vehicle ECU 112 can mitigate the unsafe condition by, for example, operating lights, speakers, and / or displays to instruct the passenger to return to his or her seat, and by operating the drive system 116 to slow the vehicle 100 and steer it to a safe position to stop the vehicle 100, thereby reducing the likelihood of sudden maneuvers or accidents that could cause injury to the passenger while outside of his or her seat. Thus, the cabin monitoring system 108 improves upon conventional vehicle monitoring systems by increasing vehicle safety and reducing the likelihood of injury to passengers in the vehicle 100.

[0033] Figure 1 The vehicle 100 depicted in FIG1 is a car, and the cabin 104 includes four seats 120, 122, 124, 126 in which passengers can sit. The reader should appreciate that the cabin 104 may include more or fewer seats depending on the configuration and type of vehicle 100. The drive system 116 of the vehicle 100 includes a drive motor, such as an internal combustion engine and / or one or more electric motors, that drives the wheels of the vehicle 100, as well as steering and braking components that enable the vehicle 100 to move in a controlled manner.

[0034] In other embodiments, the vehicle 100 can include any number of types of vessels, such as trains, buses, subways, aircraft, helicopters, drones, submarines, elevators, and passenger mobility pods, with one or more cabins 104 for moving people or cargo. A cabin 104 (which may also be referred to herein as a compartment) is a typically enclosed space for accommodating passengers or cargo. Although the vehicle 100 is illustrated as having a single cabin 104, the reader should appreciate that the vehicle 100 can include any number of separate and separate cabins 104 (e.g., multiple compartments or spaces within a train car).

[0035] In this exemplary embodiment, the vehicle 100 is a shared autonomous vehicle that is configured to autonomously drive to the passenger's location and then autonomously transport the passenger to the desired location using the public road network when the passenger enters the vehicle 100. For example, the passenger can use a smartphone or smart device application (i.e., an "app") to engage the services of the vehicle 100. The passenger is also referred to herein as an occupant, user, operator, or person. In other embodiments, the vehicle 100 is any type of passenger vehicle as described above, and in some embodiments, can be occupant-controlled or remotely controlled.

[0036] The vehicle 100 also includes a plurality of doors 132 that allow passengers to enter the cabin 104 and the seats 120 - 126. In addition, the vehicle 100 may include a rear compartment 136 that allows users to enter a cargo storage area of ​​the vehicle, such as a trunk or storage space behind the rear seats.

[0037] The cabin monitoring system 108 is configured to estimate the position of passengers in the cabin 104 and determine whether the passengers are in their corresponding seats 120-126. The cabin monitoring system 108 includes a passenger stability monitoring system 138 and a seat departure detection system 140, each of which is communicatively coupled to one or more image sensors 142, 144, 146, 148. The vehicle ECU 112, the passenger stability monitoring system 138, and the seat departure detection system 140 may be collectively referred to herein as the processing system 150 of the vehicle 100, and in some embodiments, may be integrated into a single control unit. In particular, the passenger stability monitoring system 138, the seat departure detection system 140, and / or the vehicle ECU 112 may be integrated into the vehicle's computer responsible for the autonomous navigation of the vehicle 100 and the operation of other components in the vehicle 100. In another embodiment, the processing system may be partially or entirely in the "cloud." For example, the vehicle ECU 112 can be configured to transmit data to a remote memory via the Internet (e.g., in the "cloud"), and the processes and functions of the vehicle ECU 112, passenger stability monitoring system 138, and / or seat exit detection system 140 described herein can be performed by one or more processors located remotely from the vehicle (e.g., in the "cloud").

[0038] The image sensors 142, 144 in the vehicle cabin 104 are each configured to generate an image of a portion of the cabin 104, while one or more image sensors 146, 148 can be arranged to generate an image of a portion of the exterior of the vehicle 100. The image sensors 142-148 can be video or still image cameras, each having, for example, a charge coupled device (CCD) or an active pixel sensor for generating digital image data. In other embodiments, the image sensors 142-148 can include thermal or infrared sensors, radar imaging systems, LIDAR imaging systems, or other suitable imaging systems.

[0039] In the illustrated embodiment, the cabin monitoring system 108 includes two image sensors 142, 144 in the cabin 104. The front image sensor 142 generates digital image data of the front of the cabin, including the front seats 120, 122, and the rear image sensor 144 generates digital image data of the rear of the cabin 104, including the rear seats 124, 126. In other embodiments, the cabin monitoring system 108 may include: a single image sensor that captures images of the entire cabin 104, including all seats 120-126; a separate image sensor directed at each of the seats 120-126; or any desired image sensor configuration for generating a digital image of each seat in the vehicle.

[0040] In one embodiment, the image sensors 142, 144 are disposed in or on the ceiling of the vehicle 100 and directed downwardly into the cabin 104 toward the corresponding one or more seats 120-126 for imaging. In other embodiments, the image sensors may be disposed in the seats or in the instrument panel of the vehicle 100. For example, in one particular embodiment, the image sensors for imaging the front seats 120, 122 are disposed in the instrument panel of the vehicle 100, while the image sensors for imaging the rear seats 124, 126 are disposed in the front seats 120, 122 that are directly in front of the corresponding rear seats 124, 126.

[0041] One or more external image sensors 146, 148 are configured to generate images of the area immediately surrounding the vehicle 100. For example, in the illustrated embodiment, the external image sensors 146, 148 may be mounted on or in place of the vehicle's rearview mirrors and directed toward the exterior of the door 132.

[0042] The vehicle ECU 112 is communicatively coupled to the seat exit detection system 140 and the drive system 116 via the communication bus 152. The vehicle ECU 112 may also be connected to various additional components in the vehicle 100. For example, Figure 2 and Figure 5 As illustrated in , the vehicle ECU 112 can be communicatively coupled to one or more lights 180, one or more speakers 184 and / or one or more display screens 188, which are positioned in the cabin 104 or configured to project light or sound into the cabin 104.

[0043] In addition, the vehicle ECU 112 can be communicatively coupled to a transceiver 196, which is also referred to as a wireless transmitter and receiver and is configured, for example, to wirelessly transmit data from the vehicle ECU 112 to another electronic device (not shown) via the Internet and to wirelessly receive data from another electronic device. Thus, the transceiver 196 is operable to connect the vehicle 100 to the Internet and other electronic devices. In other embodiments, the transceiver 196 sends and receives data using a cellular network, a wireless local area network ("Wi-Fi"), a personal area network, and / or any other wireless network. Thus, the transceiver 196 is compatible with any desired wireless communication standard or protocol, including, but not limited to, near field communication ("NFC"), IEEE 802.11, IEEE 802.15.1, and IEEE 802.15.2. Global System for Mobile Communications ("GSM") and Code Division Multiple Access ("CDMA").

[0044] For example, the vehicle ECU 112 may include one or more general or special purpose programmable processors and / or controllers. One of ordinary skill in the art will recognize that a "controller" or "processor" includes any hardware system, hardware mechanism, or hardware component that processes data, signals, or other information. At least one processor and / or controller of the vehicle ECU 112 is configured to execute program instructions stored on its associated memory to manipulate data or operate one or more components of the vehicle 100, such as the drive system 116, lights 180, speakers 184, displays 188, and transceivers 196, to perform the recited tasks or functions.

[0045] Passenger stability monitoring system

[0046] Now refer to Figure 2The passenger settling monitoring system 138 of the cabin monitoring system 108 is configured to determine, based on received sensor signals, whether a passenger has settled into his or her corresponding seat when loaded into the vehicle 100, and to generate a signal indicating an abnormal loading event if the passenger is not properly settled into the vehicle, and to communicate the signal to the vehicle ECU 112. The passenger settling monitoring system 138 is communicatively coupled to the image sensors 142, 144 via the communication bus 152 and is configured to receive a sensor signal from each of the image sensors 142, 144, which sensor signal may be an analog or digital signal.

[0047] The passenger stability monitoring system 138 includes at least one processor and / or controller operatively connected to an associated memory 204, hereinafter referred to as the processor 200. The processor 200 is configured to execute program instructions 208 stored on the memory 204 to manipulate data (e.g., data 220-240) or to operate one or more components of the passenger stability monitoring system 138 or the vehicle 100 to perform the recited tasks or functions.

[0048] The memory 204 is an electronic storage device configured to store data and program instructions, including at least the image data 220, the passenger location data 224, the configuration data 228, the settling timer data 232, the seated timer data 236, the abnormal settling event data 240, and the program instructions 208 for operating the exit-seat detection system 140. The memory 204 may include non-transitory computer-readable storage media and / or communication media, such as both volatile and non-volatile media, both write-capable and read-only media, and both removable and non-removable media implemented in any media or technology, including CD-ROMs, DVDs, optical disk storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices, or other known storage media technologies.

[0049] Image data 220 is electronic data representing one or more images captured by image sensors 142, 144 and, in some embodiments, external image sensors 146, 148. As discussed in detail below, passenger position data 224 represents one or more points corresponding to the passenger's position and / or associated body parts of the passenger. The points can be stored, for example, as xy coordinates corresponding to pixel locations in the associated image.

[0050] Configuration data 228 includes the boundaries of the seat, which in one embodiment may also be xy coordinates corresponding to pixel positions in the associated image. Configuration data 228 may include, for example, the boundaries of the hip area and / or shoulder area for each of seats 120-126. The boundaries may be rectangular, trapezoidal, circular, triangular, another polygon, a combination of the above shapes, or any other suitable shape. In some embodiments, the seat boundaries may be irregularly shaped so as to follow the desired contour of the seat area representing a safe position for a passenger. In addition, configuration data 228 includes predetermined default values ​​for a settling timer and a seated timer. Configuration data 228 may be pre-programmed into memory 204, or it may be user-configurable or remotely configurable, depending on the desired use and configuration of vehicle 100.

[0051] In some embodiments, one or both of the default values ​​for the settling timer and the seat timer may depend on the specific passenger or user. For example, in one embodiment, the default value for the settling timer may be a longer time or a larger frame count for a passenger who is, for example, elderly or has a disability that causes the passenger to load the vehicle at a slower rate than typical passengers. In another embodiment, the default value for the settling timer may be based on data collected from previous settling times for that specific user or passenger, such that an abnormal settling event is generated only when the settling time for a specific passenger is abnormally long. The passenger-specific default values ​​may be transmitted to the vehicle from a remote server, such as via transceiver 196, via known methods.

[0052] The settling timer data 232 includes a settling timer variable corresponding to a decrementing counter or timer representing the number of frames or lengths of time that a passenger must remain in their respective seats before an abnormal settling event is generated. The seating timer data 236 includes a seating timer variable corresponding to a decrementing timer or counter representing the number of consecutive frames or lengths of time a passenger must remain in their respective seats before the vehicle is determined to have been loaded. The abnormal settling event data 240 is alert data output by the passenger settling monitoring system 138 when a passenger has been detected settling in an abnormal manner, enabling the vehicle ECU 112 to implement actions to mitigate the abnormal loading event.

[0053] In some embodiments, the passenger stability monitoring system 138 further includes a communication interface assembly 212 having one or more interfaces configured to couple the passenger stability monitoring system 138 with the image sensors 142-148 and the vehicle ECU 112. The communication interface assembly 212 is configured to enable sensor data, control signals, software, or other information to be transmitted between the passenger stability monitoring system 138 and the image sensors 142-148 or the vehicle ECU 112 in the form of signals, such as electronic, electromagnetic, optical, or other signals capable of being received or transmitted by the communication interface assembly 212. In some embodiments, the communication interface assembly 212 may include physical terminals for connecting to a wired medium, such as a wired network or direct wired communication (e.g., the communication bus 152). In other embodiments, the communication interface assembly 212 may include one or more modems, bus controllers, etc., configured to enable communication with the image sensors 142-148 and the vehicle ECU 112. In some embodiments, the communication interface assembly 212 may include one or more wireless transceivers configured to enable wireless communications, such as acoustic, RF, infrared (IR), Wi-Fi, Bluetooth, and other wireless communication methods.

[0054] Abnormal passenger stability detection process

[0055] In operation, the passenger stability monitoring system 138 is configured to implement Figure 3 The method 300 illustrated in the flowchart of FIG is used to determine whether a passenger has settled into a seat 120-126 and completed vehicle loading, and to generate an alarm for an abnormal settling event if the passenger has not properly settled into the vehicle within a predetermined time period. In the description of the method 300, the statement that the method, process, module, processor, system, etc. is performing some task or function refers to a controller or processor (e.g., processor 200) executing programming instructions (e.g., program instructions 208) stored in a non-transitory computer-readable storage medium (e.g., memory 204) operably connected to the controller or processor to manipulate data or operate one or more components in the cabin monitoring system 108 and / or the vehicle 100 to perform the task or function. Additionally, the steps of the method can be performed in any feasible time order, regardless of the order shown in the figures or the order used to describe the steps therein. It will be appreciated that in some embodiments, the operations of the processor 200 described herein may be performed by the processing system 150 and / or other components of the cabin monitoring system 108, such as the vehicle ECU 112 or the integrated image processor of the sensors 142, 144, or the operations may be performed by one or more processors remote from the vehicle (e.g., in the "cloud").

[0056] Figure 3 Method 300 begins with processor 200 initiating a passenger entry event (block 302). In some embodiments, processor 200 may initiate a passenger entry event in response to a command received from vehicle ECU 112 indicating that vehicle 100 has reached a passenger boarding location. In another embodiment, vehicle ECU 112 or processor 200 communicates with external image sensors 146, 148 and / or door sensors (not shown) that sense the open or closed state of a door, and processor 200 initiates a passenger loading event in response to detecting that a passenger is being loaded into the vehicle.

[0057] Once the passenger loading event has been initiated in frame 302, method 300 continues as follows: interior image sensors 142, 144 capture images of the interior of cabin 104, particularly images of seats 120-126 (frame 304). Additionally, in some embodiments, exterior image sensors 146, 148 are also operated in frame 304 to capture images of the exterior of the vehicle.

[0058] The images may be one or more RGB images of the cabin 104, including the seats 120-126, or the images may be infrared images, three-dimensional radar or LIDAR images, or other desired images. The processor 200 of the seat detection system 140 receives the images from the image sensors 142, 144 via, for example, the communication interface assembly 212 and the communication bus 152. In some embodiments, the images may be resized by the image sensors 142, 144 or the processor 200 and stored in the image data 220 of the memory 204. Example images 600, 610 are shown in FIG. Figure 8a and Figure 9a Depicted in Figure 8a An example image 660 captured by the rear image sensor 144 is depicted illustrating a passenger 620 seated in the seat 124. Figure 9a An example image 610 captured by the rear image sensor 144 is illustrated, illustrating two passengers 624 , 626 seated in the rear seats 124 , 126 .

[0059] Refer back to Figure 3, process 300 continues by processing the image data to determine key points of the passengers in the cabin 104, and in some embodiments, key points of the persons located outside the vehicle and detected in the images captured by the external image sensors 146-148 (box 308). In particular, the processor 200 is configured to determine a plurality of key points in the image data to estimate the posture of the passenger. As used herein, the "posture" of a passenger refers to the relative and / or absolute position and / or orientation of one or more joints, limbs, or other body parts of the passenger. For example, in at least one embodiment, the posture of the passenger includes a plurality of points corresponding to the position of specific joints of the passenger, particularly the hip and / or shoulder joints. More particularly, the processor 200 processes the image data based on the image data 220 to approximate the position of certain joints of the passenger and returns the position as xy coordinates in the image. The processor 200 can be configured to perform any suitable posture estimation technique in a manner generally known in the art.

[0060] In one embodiment, the key points are the positions of the passenger's left and right hips (e.g., Figure 8b The hip positions 620a and 620b of the middle passenger 620, and Figure 9a In another embodiment, the key points may be the positions of the passenger's left and right shoulders (e.g., Figure 8b shoulder positions 621a-b of the middle passenger 620, and Figure 9a The key points are stored in the passenger position data 224 of the memory 204.

[0061] In one embodiment, if the processor 200 cannot determine one or both of the key point positions, the processor 200 reads the key point position(s) of the previous image from the passenger position data 224. However, if there is no previous key point data or the key point data is not available within a predetermined number of frames (e.g., approximately 3 frames, approximately 5 frames, or approximately 10 frames) or a predetermined time period (e.g., approximately 0.1 seconds, approximately 0.5 seconds, or approximately 1 second), the processor 200 can instead determine a different key point. For example, a system in which the processor is configured to use hip position as a key point can instead determine shoulder position as a key point if hip position is not available within a predetermined number of frames or time periods. The predetermined number of frames or time periods can, for example, be stored in the configuration data 228.

[0062] In some embodiments, the processor 200 uses generalized object tracking techniques rather than human pose estimation techniques to determine the position and boundaries of the passenger. The processor 200 can, for example, use the object tracking techniques to estimate a single center point of the passenger based on the image data 220 in a manner generally known in the art, and store the estimated key points in the passenger position data 224. The processor 200 can also be configured to use the object tracking techniques to track non-human objects in the cabin 104 and distinguish the non-human objects from the passengers in the cabin 104 before or after determining the key points.

[0063] Based on the key points, the method 300 continues by determining a single passenger location point for each passenger (block 312). Specifically, the processor 200 converts the multiple key points for each passenger into a single passenger location point for each passenger and stores the passenger location point in the passenger location data 224 of the memory 204. In one particular embodiment, the passenger location point for each passenger corresponds to the midpoint of the key points in the x- and y-directions.

[0064] For example, in Figure 8b In the embodiment of FIG, the passenger position point 620c is the midpoint between the two hip positions 620a, 620b. Alternatively, if the shoulder position is used as the key point instead, the passenger position point 621c is the midpoint between the two shoulder positions 621a, 621b. Similarly, if Figure 9a As depicted in FIG, passenger position points 624c and 626c correspond to the midpoints of the hip positions 624a-b and 626a-b of passengers 624, 626, respectively, or alternatively, passenger position points 625c, 627c correspond to the midpoints of the shoulder positions 625a-b and 627a-b of passengers 624, 626. Figure 9b As shown in FIG, the passenger location points 624c, 626c can be stored as xy coordinates within the image. Figure 9b , the xy coordinates of passenger location points 624c and 626c are [290, 80] and [140, 85] respectively.

[0065] The process 300 continues by determining whether a passenger is present in the cabin (block 320). In particular, the processor 200 is configured to detect the presence of a passenger if a passenger keypoint and / or at least one passenger location point is available, or alternatively, if a keypoint is not available, but a predetermined number of frames or time periods for which a previous keypoint can be used have not passed since the most recent availability of the keypoint.

[0066] If the processor 200 determines in block 320 that no passenger is present (i.e., no passenger location data is available), then the passenger has not properly entered the vehicle 100. Accordingly, the method 300 proceeds to block 324, where the processor 200 resets the seating timer in the seating timer data 236 to its default value, which is retrieved from the configuration data 228. Alternatively, because no passenger is currently being detected loading into the cabin 104 of the vehicle 100, the processor 200 resets the settling timer to its default value, also retrieved from the configuration data 228, to allow the passenger to perform necessary actions outside the vehicle, such as loading luggage into the vehicle 100, without starting the timer for triggering an abnormal settling event. The method 300 then proceeds to block 304, where the processor 200 continues the settling detection loop.

[0067] Returning to block 320 , if there is a detected passenger, the method 300 continues by determining whether the passenger is within the seat boundaries (block 332 ). Specifically, the processor 200 retrieves the passenger location point from the passenger location data 224 and compares the passenger location point with the seat boundaries retrieved from the configuration data 228 .

[0068] In some embodiments, the associated seat boundaries depend on whether the passenger position is based on the passenger's hip point or the passenger's shoulder point. For example, the seat boundary at the passenger's shoulders is higher on the seat than the seat boundary at the passenger's hips to reflect that a safe position for a passenger's shoulders is naturally higher than a safe position for a passenger's hips. Figure 8b In the example depicted in FIG, passenger location point 620c is located inside seat boundaries 640a, 640b (for purposes of the method 300, only one of seat boundaries 640a, 640b is used, although the reader will appreciate that both inner and outer seat boundaries may be used in a manner similar to that discussed below with reference to process 500). Figure 9b , the passengers' respective positions 624c, 626c are both within the associated seat boundaries 644, 646.

[0069] If one or more passengers are not within a seat boundary at block 332, the method 300 continues by resetting the seating timer in the seating timer data 236 to its default value (block 336). Specifically, the processor 200 retrieves the seating timer default value from the configuration data 228 and stores the default value as the seating timer value in the seating timer data 236.

[0070] The method 300 continues (block 340) by decrementing the settling timer in the settling timer data 232. Specifically, the processor 200 retrieves the settling timer value from the settling timer data 232, decrements it by, for example, one or the elapsed time, and stores the decremented value as the new settling timer value in the settling timer data 232.

[0071] The method 300 then checks whether the settling timer has reached zero (block 344). Specifically, the processor 200 determines whether the settling timer is zero. If the settling timer has not reached zero, the time allotted for the passenger to settle into the vehicle 100 has not elapsed, and therefore the method 300 continues to block 304 for the processor 200 to repeat the settling detection cycle without generating an alarm.

[0072] Alternatively, if the settling timer is zero in block 344, the passenger has not properly settled into the vehicle within the time allotted for settling in. Accordingly, the processor 200 generates an abnormal settling event alert, which is stored in the abnormal settling event data 240 and / or transmitted as an electronic signal to the vehicle ECU 112 via the communication interface assembly 212 (block 348).

[0073] The method 300 then continues by resetting the settling timer data 232 to its default value (block 352), and the passenger settling cycle continues at block 304. More specifically, the processor 200 retrieves the default value for the settling timer from the configuration data 228 and stores the default value as the settling timer value in the settling timer data 232. Since the settling timer is reset to the predetermined default value and the passenger settling cycle continues, if the passenger still does not settle before the settling timer reaches zero again, the processor 200 of the passenger settling monitoring system 138 can generate another abnormal settling alert. In this manner, the passenger settling monitoring system 138 is configured to generate repeated abnormal settling alerts until all passengers have properly settled into the vehicle.

[0074] Returning now to block 332, if the processor 200 determines that the passengers are all located within the permitted seating zones, the method 300 continues by decrementing the seat timer variable (block 356). Specifically, the processor 200 retrieves the seat timer variable from the seat timer data 236, decrements it by one or the elapsed time, and stores the decremented seat timer variable in the seat timer data 236 in the memory 204.

[0075] The method 300 then determines whether the seat timer variable has reached zero (block 360). More specifically, the processor 200 determines whether the seat timer variable has reached zero. If the seat timer variable has not reached zero, then the passenger has not been seated within the required length of time. As such, the method 300 proceeds to block 340 by decrementing the settling timer variable, as discussed above.

[0076] However, if the seat timer variable has reached zero in block 360, then all passengers have remained in their respective seats for the required length of time and are therefore properly settled. Thus, the process 300 terminates. In some embodiments, the process 300 terminates at block 364 by beginning the following reference to Figure 6 The discussion ends with the departure seat detection loop.

[0077] Mitigation of abnormal stability events

[0078] Figure 4 A method 380 of operating a vehicle 100 is illustrated. The method 380 mitigates an unsafe condition or assists passengers in properly settling by operating at least one component of the vehicle 100 in a predefined manner in response to the detection of an abnormal settling event. In the description of the method 380, the statement that a method, process, module, processor, system, etc. is performing some task or function refers to a controller or processor (e.g., the vehicle ECU 12) executing programmed instructions stored in a non-transitory computer-readable storage medium operatively connected to the controller or processor to manipulate data or operate one or more components of the cabin monitoring system 108 and / or the vehicle 100 to perform the task or function. Additionally, the steps of the method may be performed in any feasible chronological order, regardless of the order shown in the various figures or the order used to describe the steps. It will be appreciated that in some embodiments, the operations of the vehicle ECU 112 described herein may be performed by the processing system 150 and / or other components of the cabin monitoring system 108, and / or may be performed by a processor of the processing system 150 remote from the vehicle 100 (e.g., in the "cloud").

[0079] The method 380 begins by receiving an abnormal stability event (block 384 ). Specifically, the vehicle ECU 112 receives a signal generated by the passenger stability monitoring system 138 via, for example, the communication bus 154 , corresponding to an abnormal stability event alert.

[0080] The method 380 continues by preventing the vehicle from moving (block 388). More specifically, while the abnormal settling event is active, the vehicle ECU 112 prevents the drive system 116 from operating to initiate movement of the vehicle. For example, the vehicle ECU 112 prevents the vehicle 100 from moving by activating the brakes and disabling the motor of the drive system 116 until the passengers are properly seated in their respective seats and in a safe position for transport by the vehicle 100.

[0081] Additionally, method 380 also includes operating lights, displays, and / or speakers in the vehicle to generate visual and / or audible alerts (block 392). Specifically, in response to receiving the abnormal settling event alert, the vehicle ECU 112 operates the lights 180, speakers 184, and / or displays 188 to generate visual and / or audible alerts in the cabin 104 of the vehicle 100. For example, the vehicle ECU 112 may operate the speakers 184 to emit an audible alert tone or a pre-recorded message notifying the passengers that one or more passengers in the vehicle 100 are not secured in their seats. Additionally or alternatively, the vehicle ECU 112 may operate the lights 180 to illuminate and / or flash to alert the passengers that the ride cannot be initiated until all passengers are secured in their seats. In some embodiments, the vehicle ECU 112 operates the display 188 to flash, display a textual warning to the passengers, and / or display a video or animation instructing the passengers on how to safely sit in the vehicle 100.

[0082] In some embodiments, method 380 proceeds by transmitting an abnormal stability alert (block 396). Specifically, vehicle ECU 112 can be configured to transmit a signal corresponding to a detected abnormal stability event to a remote server via transceiver 196. For example, the signal corresponding to the abnormal stability event can be transmitted to a central control server to enable human intervention. For example, a remote operator can resolve the abnormal stability event by taking remote control of vehicle 100 or communicating with passengers via speaker 184 and display 188 to instruct passengers on how to properly load into vehicle 100.

[0083] In one embodiment, the vehicle ECU 112 can be configured to trigger corrective actions based on a predefined escalation method. For example, in block 388, the vehicle ECU 112 can first operate the display 188 to play a video on how to properly sit in the vehicle. If the abnormal settling alarm persists, the vehicle ECU 112 can then be configured to operate the speaker 184 in block 388 to generate an audible alarm that warns the passengers that the ride cannot be started until all passengers are properly seated. If the abnormal settling event is still active after the audible alarm is generated, the vehicle ECU 112 can communicate the alarm to a remote server via the transceiver 196 (block 392) so that a remote operator can intervene and determine why the passenger is not properly seated in the vehicle 100 and instruct the passenger on how to load the vehicle.

[0084] The passenger settlement monitoring system 138 is therefore configured to detect abnormal passenger settlement events in process 300, and process 380 mitigates unsafe conditions caused by abnormal settlement events and assists passengers in properly settling into the vehicle. For example, if a passenger enters the vehicle 100 with an unusual amount of luggage or belongings, the passenger may take an unusual amount of time (i.e., longer than the default settling timer value) to find space for the belongings and arrange them before taking their seats. The passenger settlement monitoring system 138 is advantageously configured to detect such abnormal settlement events, and a video or animation can be displayed on the display 188 to instruct the passenger on how to properly load their luggage and settle into the vehicle. If the passenger still cannot properly settle into their seat, the vehicle ECU 112 audibly alerts the passenger that the ride cannot begin and then, if the passenger still does not settle, communicates with a remote server via the transceiver 196 to enable human intervention.

[0085] As another example, the passenger settling monitoring system 138 detects a situation in which a passenger enters the vehicle more slowly or in an unusual mode due to a physical disability of the passenger. Detection of an unusual settling event by the processing system 150 advantageously enables human intervention to assist the disabled passenger in entering the vehicle, for example, by calling for help at the passenger's location or operating a feature in the vehicle to assist, such as by lowering a ramp or lowering a vehicle seat for easier access.

[0086] The passenger set-up detection system can also detect passengers entering from the side, passengers moving over another passenger, passengers frequently changing position, and passengers who otherwise exhibit unusual movements while settling into the vehicle. Detection of unusual settling events advantageously enables the processing system 150 to provide automated instructions for proper settling and enables the processing system 150 to communicate with a remote server to enable human intervention to assist the passenger in properly settling into the vehicle.

[0087] In another embodiment, in response to an abnormal settling alert, the vehicle ECU 112 can be configured to analyze the image data to determine if there is a problem with the seat that prevents the passenger from properly settling into the seat. For example, the vehicle ECU 112 can detect damage to the seat or an object or covering on the seat that prevents the passenger from properly using the seat, and operate the speaker 184, display 188, or transceiver 196 based on the detection of the seat problem.

[0088] In another embodiment, the vehicle ECU 112 may be configured to determine if an emergency situation exists that prevents a passenger from properly securing their seat. For example, in response to an abnormal securing event alert, the vehicle ECU 112 may analyze the image data to determine if the passenger is in distress due to, for example, a medical issue or criminal activity that prevents the passenger from securing their seat.

[0089] Leaving seat detection system

[0090] As briefly discussed above, the cabin monitoring system 108 also includes a passenger exit seat detection system 140 that operates after the passengers are settled to detect potentially unsafe conditions caused by passengers moving out of their respective seats while the vehicle 100 is in motion. Figure 5 The exit-seat detection system 140 of the cabin monitoring system 108 is configured to monitor the cabin 104 of the vehicle and determine whether a user has exited his or her corresponding seat based on received sensor signals, generate a signal representing the exit-seat determination, and communicate the signal to the vehicle ECU 112. The exit-seat detection system 140 is communicatively coupled to the image sensors 142, 144 via a communication bus 152 and is configured to receive a sensor signal from each of the image sensors 142, 144, which sensor signal may be an analog or digital signal.

[0091] The exit-seat detection system 140 includes at least one processor and / or controller 400, hereinafter referred to as the processor 400, operatively connected to an associated memory 404. The processor 400 of the exit-seat detection system 140 is configured to execute program instructions 408 stored on its associated memory 404 to manipulate data (e.g., data 420-444) or to operate one or more components of the exit-seat detection system 140 or the vehicle 100 to perform the recited tasks or functions.

[0092] The memory 404 is an electronic storage device configured to store data and program instructions, including at least image data 420, passenger location data 424, configuration data 428, alert wait time data 432, reset counter data 436, internal state data 440, exit-seat event data 444, and program instructions 408 for operating the exit-seat detection system 140. The memory 404 may include non-transitory computer-readable storage media and / or communication media, such as both volatile and non-volatile media, both write-capable and read-only media, and both removable and non-removable media implemented in any media or technology, including CD-ROMs, DVDs, optical disk storage devices, cassettes, magnetic tape, disk storage devices, or other known storage media technologies.

[0093] As noted above, in some embodiments, the seat exit detection system 140 may be integrated with one or both of the vehicle ECU 112 and the passenger inertia detection system 138. For example, the processor 400 and memory 404 may be the same processor 200 and memory 204 discussed above with respect to the passenger inertia monitoring system 138, and / or the processor 400 and memory 404 may be the same processor and memory as in the vehicle ECU 112.

[0094] Image data 420 is electronic data representing one or more images captured by image sensors 142, 144. As discussed in detail below, passenger position data 424 represents one or more points representing the location of a passenger and / or an associated body part of the passenger. The points may be stored, for example, as xy coordinates corresponding to pixel locations in the associated image.

[0095] Configuration data 428 includes the boundaries of the seats from the image sensor view, which in one embodiment can also be xy coordinates corresponding to pixel locations in the associated image. Configuration data 428 can include, for example, inner and outer boundaries for the passenger's hips and / or shoulders for each of seats 120-126. The boundaries can be rectangular, trapezoidal, circular, triangular, another polygon, a combination of the above shapes, or any other suitable shape. Alternatively, the seat boundaries can be irregularly shaped to follow the desired contour of the seating area. In addition, configuration data 428 includes predetermined default values ​​for internal state variables, reset counter variables, and alarm counter variables. Configuration data 428 can be pre-programmed into memory 404, or it can be user-configurable or remotely configurable, depending on the desired use or configuration of vehicle 100.

[0096] The alert wait time data 432 includes an alert wait time variable that decrements a timer corresponding to the number of frames or length of time a passenger must be out of the boundaries of their respective seat before an alert is generated. The reset counter data 436 includes a reset timer variable that decrements a timer corresponding to the number of frames or length of time a passenger must be back within the boundaries of their respective seat after an alert has been generated before the alert state is reset and another out-of-seat event alert can be generated again.

[0097] Internal state data 440 includes an internal state variable corresponding to an indication of whether a passenger exit seat event has been previously generated without the alarm state being reset. Exit seat event data 444 is alarm data output by the exit seat detection system 140 when a passenger exit seat event has been detected, so as to enable the vehicle ECU 112 to implement actions to mitigate a potentially unsafe condition caused by a passenger being outside of his or her respective seat.

[0098] In some embodiments, the exit-seat detection system 140 further includes a communication interface assembly 412 having one or more interfaces configured to couple the exit-seat detection system 140 with the image sensors 142, 144 and the vehicle ECU 112. The communication interface assembly 412 is configured to enable sensor data, control signals, software, or other information to be transferred between the exit-seat detection system 140 and the image sensors 142, 144 or the vehicle ECU 112 in the form of signals, such as electronic, electromagnetic, optical, or other signals capable of being received or transmitted by the communication interface assembly 412. In some embodiments, the communication interface assembly 412 may include physical terminals for connecting to a wired medium, such as a wired network or direct wired communication (e.g., the communication bus 152). In other embodiments, the communication interface assembly 412 may include one or more modems, bus controllers, etc., configured to enable communication with the image sensors 142, 144 and the vehicle ECU 112. In some embodiments, communication interface assembly 412 may include one or more wireless transceivers configured to enable wireless communications, such as acoustic, RF, infrared (IR), Wi-Fi, Bluetooth, and other wireless communication methods.

[0099] Leaving seat detection process

[0100] In operation, the seat exit detection system 140 is configured to implement Figure 6126. In the description of method 500, a statement that a method, process, module, processor, system, or the like is performing some task or function refers to a controller or processor (e.g., processor 400) executing programming instructions (e.g., program instructions 408) stored in a non-transitory computer-readable storage medium operably connected to the controller or processor to manipulate data or operate one or more components of the cabin monitoring system 108 and / or the vehicle 100 to perform the task or function. Additionally, the steps of the method may be performed in any feasible chronological order, regardless of the order shown in the various figures or the order used to describe the steps. It will be appreciated that in some embodiments, the operations of processor 400 described herein may be performed by processing system 150 and / or other components of cabin monitoring system 108, such as the vehicle ECU 112 or the integrated image processor of sensors 142, 144, or the operations may be performed by one or more processors remote from the vehicle (e.g., in the "cloud").

[0101] Figure 6 The method 500 begins with the image sensors 142, 144 capturing an image of the seats 120-126 (block 504). The image may be an RGB image of the seats 120-126, or the image may be an infrared image, a three-dimensional radar or LIDAR image, or other desired image. The image sensors 142, 144 transmit the image to the processor 400 off the seat detection system 140 via, for example, the communication interface assembly 412. In some embodiments, the image may be resized by the image sensors 142, 144 or the processor 400 and stored as image data 420 in the memory 404. Example images 600, 610 are shown in FIG. Figure 8a and Figure 9a Depicted in Figure 8a An example image 600 captured by the rear image sensor 144 is depicted, illustrating a passenger 620 seated in the seat 124. Figure 9a An example image 610 captured by the rear image sensor 144 is illustrated, illustrating two passengers 624 , 626 seated in the rear seats 124 , 126 .

[0102] Refer back to Figure 6, method 500 continues by processing the image data to determine key points of the passenger in cabin 104 (block 508). In particular, processor 400 is configured to determine a plurality of key points in the image data to estimate the passenger's posture. More particularly, processor 400 processes the image data to approximate the positions of certain joints of the passenger based on image data 420 and returns the positions as xy coordinates in the image. Processor 400 can be configured to perform any suitable posture estimation technique in a manner generally known in the art.

[0103] In one embodiment, the key points are the positions of the passenger's left and right hips (e.g., Figure 8b The hip positions 620a and 620b of the middle passenger 620, and Figure 9a In another embodiment, the key points may be the positions of the passenger's left and right shoulders (e.g., Figure 8b shoulder positions 621a-b of the middle passenger 620, and Figure 9a The key points are stored in the passenger position data 424 of the memory 404.

[0104] In one embodiment, if processor 400 can not determine one or both of the key point positions, then processor reads (one or more) key point positions of previous image from passenger position data 424, and reuses previous key point positions. However, if there is no previous key point data or key point data is unavailable in a predetermined number of frames (e.g., approximately 3 frames, approximately 5 frames, approximately 10 frames, or approximately 20 frames) or a predetermined time period (e.g., approximately 1 second, approximately 2 seconds, approximately 3 seconds, or approximately 5 seconds), then processor 400 can alternatively determine different key points. For example, a system in which processor 400 is configured to use hip position as a key point can alternatively determine shoulder position as a key point, if hip position is unavailable in a predetermined number of frames or time period. A predetermined number of frames or time period can, for example, be stored in configuration data 428.

[0105] In some embodiments, the processor 400 uses generalized object tracking techniques rather than human pose estimation techniques to determine the position and boundaries of the passenger. The processor 400 can, for example, use the object tracking techniques to estimate a single center point of the passenger based on the image data 420 in a manner generally known in the art, and store the estimated key points in the passenger position data 424. The processor 400 can also be configured to use the object tracking techniques to track non-human objects in the cabin 104 and distinguish the non-human objects from the passengers in the cabin 104 before or after determining the key points.

[0106] Based on the key points, the method 500 continues by determining a single passenger location point for each passenger (block 512). Specifically, the processor 400 converts the multiple key points for each passenger into a single passenger location point for each passenger and stores the passenger location point in the passenger location data 424 of the memory 404. In one particular embodiment, the passenger location point for each passenger corresponds to the midpoint of the key points in the x- and y-directions.

[0107] For example, in Figure 8b In the embodiment of FIG, the passenger position point 620c is the midpoint between the two hip positions 620a, 620b. Alternatively, if the shoulder position is used as the key point instead, the passenger position point 621c is the midpoint between the two shoulder positions 621a, 621b. Similarly, if Figure 9a As depicted in FIG, passenger position points 624c and 626c correspond to the midpoints of the hip positions 624a-b and 626a-b of passengers 624, 626, respectively, or alternatively, passenger position points 625c, 627c correspond to the midpoints of the shoulder positions 625a-b and 627a-b of passengers 624, 626. Figure 9b As shown in FIG, the passenger location points 624c, 626c can be stored as xy coordinates within the image. Figure 9b In, based on Figure 9b In the coordinate system of the image depicted in , the xy coordinates of the passenger position points 624c and 626c are [290, 80] and [140, 85], respectively.

[0108] The process 500 continues by determining whether a passenger is present in the cabin (block 520). In particular, the processor 400 is configured to detect the presence of a passenger if a passenger keypoint and / or at least one passenger location point is available, or alternatively, if a keypoint is not available, but a predetermined number of frames or time periods for which a previous keypoint is available has not passed since the most recent availability of the keypoint.

[0109] If the processor 400 determines in block 320 that no passenger is present (i.e., no passenger location data is available), the process 500 proceeds to check whether the internal state variable is zero (block 524). Specifically, the processor 400 retrieves the internal state variable from the internal state data 440 in the memory 404 and determines whether the variable is zero or non-zero (i.e., one). As noted above, the internal state variable indicates whether a previous out-of-seat event alert has been generated and not reset. For example, if the internal state variable is zero, then either an alert has not yet been generated or a previous alert has been reset. If the internal state variable is one, then an alert has previously been generated and not reset.

[0110] If the internal state variable at block 524 is zero (i.e., there is no previous alarm or the previous alarm has been reset) when no passenger is present, then no passenger is detected and no passenger exit-of-seat event is active. Consequently, no potentially unsafe condition is detected by the exit-of-seat detection system 400. Consequently, the method 500 proceeds to reset the alarm wait time variable to its default value (block 528). More specifically, the processor 400 retrieves the default value for the alarm wait time variable from the configuration data 428 and stores the default value as the alarm wait time variable in the alarm wait time data 432. The method 500 then continues at block 504.

[0111] Alternatively, if the internal state variable is not zero in block 524 (i.e., a LeaveSeat event was previously activated and not reset), then no passenger is present, but there is an active LeaveSeat event that has not yet been reset. Consequently, method 500 continues by decrementing the alarm reset timer (block 532). More specifically, processor 400 retrieves the default value for the alarm reset timer from configuration data 428 and stores the default value as an alarm reset variable in reset timer data 436 of memory 404.

[0112] The method 500 then continues by checking whether the alarm reset counter is zero (block 536). Specifically, the processor 400 calls the alarm reset variable from the reset timer data 436 and determines whether the reset timer has reached zero.

[0113] If, after decrementing the alarm reset timer, the alarm reset timer reaches zero (block 536), sufficient time has passed since the out-of-seat event was detected, and if another unsafe condition exists, another alarm can now be generated. Process 500 thus proceeds by setting the internal state variables in internal state data 440 to zero and resetting the alarm wait time and alarm reset timer variables in the corresponding alarm wait time data 432 and alarm reset data 436 to their respective default values ​​(block 540). In particular, processor 400 sets the internal state variables in internal state data 440 to zero, recalls the default values ​​for the alarm wait time and reset timer from configuration data 428, and stores the corresponding default values ​​as the alarm wait time and reset timer variables in alarm wait time data 432 and reset timer data 436, respectively. Method 500 then returns to block 504.

[0114] Alternatively, if the reset counter variable is not zero in block 536 after being decremented, then insufficient time has elapsed since the exit seat event was detected. Process 500 therefore returns to block 504 where the decremented reset timer variable is updated in alarm reset timer data 436.

[0115] Returning now to block 520, if the processor 400 determines that at least one passenger is present, the method 500 continues by determining whether the passenger is within the seat boundaries (block 544). The processor 400 retrieves the passenger location point from the passenger location data 424 and the associated seat boundaries from the configuration data 428, and compares the passenger location point with the associated seat boundaries to determine whether the passenger location point is within the seat boundaries.

[0116] In some embodiments, the associated seat boundaries depend on whether the passenger position is based on the passenger's hip position or the passenger's shoulder position, and whether the internal state is zero or one (i.e., whether the leave-seat event has been activated and not reset). For example, the seat boundary at the midpoint of the passenger's shoulders is higher on the seat than the seat boundary at the midpoint of the passenger's hips to reflect that the passenger's shoulders naturally sit higher on the seat than the passenger's hips.

[0117] Furthermore, when the internal state is zero, a larger seat boundary (e.g. Figure 8b 640a) is used by the processor 400 to determine whether to generate an alert for a passenger who has left their seat and caused a potentially unsafe condition. Alternatively, when the internal state is one, a smaller seat boundary (e.g., an internal seat boundary) is generated. Figure 8b 640b) is used by processor 400 to determine whether to reset the internal state because the passenger has returned to his or her seat and has therefore remedied the potentially unsafe condition. Because the area used to reset the internal state is smaller than the area used to activate the alarm, processor 400 will not generate repeated leave-the-seat event alarms in situations where the passenger's position is slightly moving across either seat boundary. In other words, because the passenger must be outside outer seat boundary 640a in order for processor 400 to trigger a leave-the-seat event alarm, and must return to within inner seat boundary 640b in order for processor 400 to reset the alarm, repeated movement across only outer seat boundary 640a will result in processor 400 triggering only a single alarm, while processor 400 will not generate an alarm due to repeated movement across inner seat boundary 640b.

[0118] If the processor 400 determines in block 544 that the passenger location is inside the permitted area, then the process continues to block 524 in a similar manner as discussed above in the absence of a passenger. For example, FIG8b depicts a passenger 620 whose passenger location point 620c is inside both the inner and outer boundaries 640b, 640a. Similarly, Figure 9b Passengers 624 , 626 are depicted with their respective positions 624 c , 626 c within associated seat boundaries 644 , 646 .

[0119] When the processor 400 determines that the passenger is within their respective seat boundaries, as described above, if the processor 400 determines in block 524 that the internal state is zero (i.e., no out-of-seat alert has been issued since the last reset), the processor 400 proceeds to reset the alert wait time data 432 to its default value (block 528) because the processor 400 is not currently detecting a potentially unsafe condition. Alternatively, as described above, if the processor 400 determines in block 524 that the internal state is one (i.e., the passenger has returned to his or her seat after an out-of-seat event that has not yet been reset), the processor 400 decrements the reset timer data 436 (block 532), and if the processor 400 determines that the alert reset variable is zero (block 536), the processor 400 resets all variables to their predetermined default values ​​(block 540) so that if an unsafe condition occurs, another out-of-seat event can be generated. The process 500 then returns to block 504.

[0120] Returning to block 544, if the processor 400 determines that one or more passenger positions are outside the associated seat boundaries (e.g., as in Figure 10a and Figure 10b ), the method 500 continues by checking whether the internal state variable is one (block 548). More specifically, the processor 400 calls the internal state variable of the internal state data 440 from the memory and determines whether the internal state variable is one.

[0121] If the internal state is one, then the passenger is still outside the seat boundaries after previously generating an out-of-seat alert. Accordingly, the method 500 continues by resetting the alert reset timer to its default value, such that an out-of-seat alert cannot be generated until the passenger returns to their seat for a predefined number of frames or time period (block 552). Specifically, the processor 400 retrieves the predetermined default value for the alert reset timer from the configuration data 428 and stores the default value as a reset timer variable in the reset timer data 436 of the memory 404. The process 500 then continues at block 504.

[0122] If the processor 400 determines in block 548 that the internal state is zero, the passenger has left his or her corresponding seat, but no leave-seat event is currently active. The method 500 continues by decrementing the alarm wait time timer (block 552). Specifically, the processor 400 retrieves the alarm wait time variable from the alarm wait time data 432, decrements it by one frame or the elapsed time, and stores the decremented alarm wait time variable in the alarm wait time data 432.

[0123] Method 500 continues by changing whether the alarm wait time timer is zero (frame 556). More specifically, processor 400 determines whether the alarm wait time timer is zero. If the alarm wait time that decreases in frame 556 is not yet zero, then the waiting period is not yet expired. Therefore processor 400 does not generate the leaving seat event alarm, thereby allows the passenger to have time to return to his or her seat before generating the alarm. In addition, the alarm wait time reduces the incidence (incidence) of false positives (that is, when the passenger does not leave his or her corresponding seat, generate an alarm), which may be caused by erroneous image analysis, obstructed sight lines, etc., because the passenger must leave his or her corresponding seat and reach multiple frames so that the alarm is generated. In one embodiment, the default alarm wait time timer value can be between two seconds and five seconds. In another embodiment, the default alarm wait time timer value can be approximately 3.5 seconds.

[0124] However, if the processor 400 determines in block 556 that the alarm wait time is zero, the internal state variables in the internal state data 440 are set to one, and the alarm wait time and the alarm reset timer are set to their default values ​​(block 560). Specifically, the processor 400 sets the internal state variables in the internal state data 440 to one, retrieves the default values ​​for the alarm wait time and the reset timer from the configuration data 428, and stores the corresponding default values ​​as the alarm wait time and reset timer variables in the alarm wait time data 432 and the reset timer data 436, respectively.

[0125] The process 500 also generates an out-of-seat event alert (block 564). The processor 400 stores data corresponding to the out-of-seat event alert in the out-of-seat event data 444 in the memory 404 and / or generates an electronic signal that is transmitted to the vehicle ECU 112 via the communication interface assembly 412 and the communication bus 154. The method 500 then continues with the out-of-seat detection loop at block 504.

[0126] Figure 10a An image 700 is shown of a passenger 704 who has moved between two seats 124, 126 of a vehicle 100. Because the midpoint 720 of the passenger's hip position is outside of the two seat areas 724, 726 ( Figure 10a The seating areas illustrated in the figure correspond to the outer seating areas described above), if the passenger 704 does not return to one of the seating areas 724, 726 before the alarm wait time timer expires, the processor 400 of the exit seat detection system 140 implementing the process 500 will generate an exit seat event alarm.

[0127] Figure 10bImage 800 illustrates two passengers 804 and 808, one of whom has moved between seats 124 and 126 of vehicle 100. Because the midpoint 820 of the passenger's hips has moved outside of seat area 826, if passenger 804 does not return to seat area 826 before the alarm wait timer expires, processor 400 of exit-of-seat detection system 140 will generate an exit-of-seat event alert for passenger 804. On the other hand, passenger 808 is seated such that the midpoint 822 of the passenger's hips is within seat area 828. As such, processor 400 of exit-of-seat detection system 140 does not generate an exit-of-seat event alert for passenger 808.

[0128] In the above description, method 500 is performed simultaneously by processor 400 for all seats 120-126 and passengers in cabin 104. In other embodiments, method 500 may be performed separately for each individual seat or for each individual passenger by processor 400. For example, if processor 400 detects a passenger in block 544 while having moved out of left front seat 120, and the processor 400 detects a passenger in a subsequent cycle in right front seat 122 in block 544 before the processor 400 decrements the alert wait time counter to zero (blocks 552-556), the processor 400 does not generate a leave-seat event alert (block 564) because the passenger is determined to be in the seat and an unsafe condition does not exist.

[0129] While the illustrated embodiment includes the out-of-seat detection system 140 implemented in a logic unit separate from the vehicle ECU 112, the reader should appreciate that in some embodiments, the out-of-seat detection system 140 may be implemented in the vehicle ECU 112. For example, the vehicle ECU 112 may include the processor 400 and the memory 404 as separate components within the vehicle ECU 112, or the functionality of the processor 400 may be performed by a processor and / or controller of the vehicle ECU 112, and the data 420-444 and instructions 408 may be stored in memory associated with the vehicle ECU 112.

[0130] Mitigation of unsafe conditions when leaving the seat

[0131] Figure 7A method 580 of operating a vehicle 100 is illustrated for mitigating an unsafe condition caused by a passenger leaving their seat by operating at least one component of the vehicle in a predefined manner in response to detecting that at least one passenger is outside of their seat. In the description of method 580, the statement that a method, process, module, processor, system, etc. is performing some task or function refers to a controller or processor (e.g., vehicle ECU 112) executing programmed instructions stored in a non-transitory computer-readable storage medium operatively connected to the controller or processor to manipulate data or operate one or more components of the cabin monitoring system 108 and / or the vehicle 100 to perform the task or function. Additionally, the steps of the method may be performed in any practicable chronological order, regardless of the order shown in the figures or the order in which the steps are described. It will be appreciated that in some embodiments, the operations of the vehicle ECU 112 described herein may be performed by the processing system 150 and / or other components of the cabin monitoring system 108, and / or may be performed by a processor of the processing system 150 that is remote from the vehicle 100 (e.g., in the “cloud”).

[0132] Process 580 begins by receiving an out-of-seat event alert (block 584 ). Specifically, the vehicle ECU 112 receives an electronic signal corresponding to an out-of-seat event alert generated by the out-of-seat detection system 140 via, for example, the communication bus 154 (block 584 ).

[0133] Method 580 proceeds with the vehicle ECU 112 operating the lights 180, speakers 184, and / or displays 188 to generate visual and / or audible alerts in the cabin 104 of the vehicle 100 (block 588). The vehicle ECU 112 may, for example, operate the speakers 184 to emit an alert tone or pre-recorded message that notifies the passengers that one or more passengers in the vehicle 100 have been detected leaving their respective seats, thereby alerting the passengers to a potential unsafe condition. Additionally or alternatively, the vehicle ECU 112 may operate the lights 180 to illuminate and / or flash to alert the passengers to a potential unsafe condition. In some embodiments, the vehicle ECU 112 operates the display 188 to flash, display a textual warning to the passengers, and / or display a video or animation to instruct the user of the potential unsafe condition and how to safely sit in the vehicle 100.

[0134] Method 580 also includes slowing and / or stopping the vehicle or preventing the vehicle from starting (block 592). Specifically, the vehicle ECU 112 operates the drive system 116 to slow or stop the vehicle or, depending on the current driving state of the vehicle 100, prevent the vehicle from starting to move. For example, if the vehicle 100 is traveling on a road, the vehicle ECU 112 may operate the motor and brakes of the drive system 116 to reduce the speed of the vehicle 100, identify a safe position to stop the vehicle 100, and operate the steering to steer the vehicle 100 to the identified position that is safe for stopping the vehicle 100. Once the vehicle 100 stops, or if the vehicle 100 is already stopped when the leave-seat event alert is received in block 584, the vehicle ECU 112 prevents the vehicle 100 from moving by activating the brakes and disabling the motor until the passengers return to their respective seats and the unsafe condition has been remedied.

[0135] In some embodiments, the vehicle ECU 112 may be configured to delay for a predetermined period of time after operating the lights, display, and / or speakers in block 588 before slowing or stopping the vehicle in block 592 to allow the passenger time to return to his or her seat and thereby remedy the unsafe condition. The predetermined period of time may be, for example, between approximately 5 seconds and approximately 15 seconds.

[0136] In some embodiments, the vehicle ECU 112 can be configured to transmit an electronic signal corresponding to a detected out-of-seat event to a remote server via the transceiver 196. For example, the vehicle ECU 112 can transmit the out-of-seat event signal to the central control server to enable human intervention, such as a remote operator mitigating the unsafe condition by taking remote control of the vehicle or alerting the passenger via the speaker 184 and / or display 188, to remedy the potentially unsafe condition. In another embodiment, the transmitted signal can be stored by the remote server in conjunction with the passenger's user account to identify the passenger as causing the potentially unsafe condition. The remote server can use the stored information to apply a penalty to the passenger's account and / or suspend or deactivate the passenger's account.

[0137] In another embodiment, in response to a detected seat exit event, the vehicle ECU 112 can be configured to analyze the image data to determine if there is a problem with the seat that caused the passenger to move from his or her seat. For example, the vehicle ECU 112 can detect that the seat is damaged or has an object or covering on it that prevents the passenger from properly using the seat, and operate the speaker 184, display 188, or transceiver 196 based on the detection of the seat problem.

[0138] In another embodiment, the vehicle ECU 112 may be configured to determine whether an emergency situation exists within the cabin 104 of the vehicle 100 that has caused a passenger to move out of their seat. For example, in response to a detected out-of-seat event, the vehicle ECU 112 may analyze the image data to determine whether the passenger is in distress due to, for example, a medical issue or criminal activity that has caused the passenger to move out of his or her seat.

[0139] It will be appreciated that variations of the foregoing and other features and functions, or alternatives thereto, may be desirably combined into many other different systems, applications, or methods. Those skilled in the art may subsequently make various currently unforeseen or unanticipated substitutions, modifications, variations, or improvements that are also intended to be encompassed by the foregoing disclosure.

Claims

1. A system for detecting and mitigating an unsafe condition in a vehicle, comprising: an image sensor configured to generate and output image data of one or more seats in a vehicle cabin; and A processing system operatively connected to the image sensor and comprising at least one processor, the processing system being configured to: receiving image data from an image sensor; processing the image data to determine a position of at least one passenger in the cabin, wherein the processing of the image data includes estimating a pose of the at least one passenger to determine a respective passenger position point for each of the at least one passenger; detecting that the at least one passenger is outside the one or more seats in response to at least one of the corresponding passenger location points being outside a first seat boundary of at least one seat within a predetermined first time period; and In response to detecting that the at least one passenger is located outside of the one or more seats, at least one component of the vehicle is operated in a predefined manner.

2. The system according to claim 1, wherein: The processing system is further configured to estimate a hip position of each passenger in the estimation of the posture, and determine the corresponding passenger position point as a midpoint between the estimated hip positions.

3. The system according to claim 1, wherein: The processing system is further configured to estimate a shoulder position of each passenger in the estimation of the posture, and determine the corresponding passenger position point as a midpoint between the estimated shoulder positions.

4. The system according to claim 1, wherein: The at least one component comprises a drive system of a vehicle, and operating the at least one component of the vehicle in a predefined manner further comprises: Operate the vehicle's drive system to slow or stop the vehicle.

5. The system according to claim 1, wherein: The at least one component includes a speaker configured to project audio into a vehicle cabin, and operating the at least one component of the vehicle in a predefined manner further includes: Operate the speaker to generate an audible alarm.

6. The system according to claim 1, wherein: The at least one component includes at least one of a display and a light disposed in or adjacent to a vehicle cabin, and operating the at least one component of the vehicle in a predefined manner further includes: At least one of the display and the light is operated to generate a visual alert.

7. A method for detecting and mitigating an unsafe condition in a vehicle, comprising: receiving, using a processing system, image data from an image sensor for one or more seats in a vehicle cabin; processing the image data with a processing system to determine a position of at least one passenger in the cabin, wherein the processing of the image data includes estimating a posture of the at least one passenger to determine a corresponding passenger position point for each of the at least one passenger; detecting, using a processing system, that the at least one passenger is outside the one or more seats in response to at least one of the corresponding passenger location points being outside a first seat boundary of at least one seat within a predetermined first time period; and At least one component of the vehicle is operated in a predefined manner using the processing system in response to detecting that the at least one passenger is located outside of the one or more seats.

8. The method according to claim 7, wherein: The processing of the image data further includes estimating, using the processing system, a hip position of each passenger, and determining the passenger position point as a midpoint between the estimated hip positions.

9. The method according to claim 7, wherein: The processing of the image data further includes estimating, using the processing system, a shoulder position of each passenger, and determining the passenger position point as a midpoint between the estimated shoulder positions.

10. The method according to claim 7, wherein: Operating at least one component of the vehicle in a predefined manner includes operating, with the processing system, a drive system of the vehicle to slow or stop the vehicle.

11. The method of claim 7, wherein operating at least one component of the vehicle in a predefined manner comprises operating, with the processing system, a speaker to generate an audible alarm in the cabin.

12. The method of claim 7, wherein operating at least one component of the vehicle in a predefined manner comprises operating, with the processing system, at least one of a display and a light to generate a visual alert in the cabin.

Images