AI-enabled alarm to detect locked-in passengers

Abstract

The disclosed embodiments are directed to detecting a person or animal trapped in a vehicle and providing automated assistance to such a person or animal. In one embodiment, a method is disclosed that includes detecting a vehicle stop; activating at least one camera and recording at least one image of the interior of the vehicle using the at least one camera; classifying the at least one image using a machine learning model; and operating at least one subsystem of the vehicle in response to detecting that the classification indicates the presence of a person or animal in the at least one image.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. patent application serial number 16 / 783,381, filed on February 6, 2020, entitled “ARTIFICIAL INTELLIGENCE-ENABLED ALARM FOR DETECTING PASSENGERS LOCKED IN VEHICLE,” the entire disclosure of which is hereby incorporated herein by reference. Technical Field

[0003] The disclosed embodiments are directed to automotive systems, and in particular, to systems for detecting the presence of an occupant (or other living being) locked within an automobile and taking remedial action based on the detection. Background Art

[0004] Many people and animals have been left behind in locked vehicles. Typically, injuries or deaths occur primarily in hot weather when the temperature inside the vehicle rises significantly when the vehicle is parked in direct sunlight or even in an environment with indirect sunlight. Various attempts have been made to address this problem. One such attempt is to warn the driver before the driver leaves the vehicle. Another such attempt is to use weight or motion sensors to detect that a person or animal is left in the vehicle after the vehicle is locked. There are various problems with such methods. In a first method, drivers often ignore such warnings, which are given whenever the driver stops the car. In a second method, the use of motion or weight sensors can provide false positives (attributed to luggage or other items on the seat) or can provide false negatives (attributed to the person or animal falling asleep or losing consciousness). Therefore, current systems for detecting people or animals (collectively referred to as "creatures") in vehicles are unable to adequately detect such creatures trapped in locked vehicles.

[0005] The disclosed embodiments address these and other technical problems by providing a machine learning (ML) system to automatically detect organisms trapped in a vehicle and proactively take steps to protect these organisms without intervention by the driver of the vehicle. Summary of the invention

[0006] In one aspect, the present application relates to a method comprising: receiving a signal from one or more sensors of a vehicle, the signal instructing the vehicle to stop; activating at least one camera in response to receiving the signal and recording at least one image of the interior of the vehicle using the at least one camera; classifying the at least one image using a machine learning model; and operating at least one subsystem of the vehicle in response to detecting that the classification indicates the presence of a person or animal in the at least one image.

[0007] In another aspect, the present application relates to a non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining the following steps: receiving a signal from one or more sensors of a vehicle, the signal instructing the vehicle to stop; activating at least one camera in response to receiving the signal and recording at least one image of the interior of the vehicle using the at least one camera; classifying the at least one image using a machine learning model; and operating at least one subsystem of the vehicle in response to detecting that the classification indicates the presence of a person or animal in the at least one image.

[0008] In another aspect, the present application relates to a device comprising: a processor; and a storage medium for tangibly storing thereon programming logic for execution by the processor, the stored programming logic causing the processor to perform the following operations: receiving a signal from one or more sensors of a vehicle, the signal instructing the vehicle to stop, activating at least one camera in response to receiving the signal and recording at least one image of the interior of the vehicle using the at least one camera, classifying the at least one image using a machine learning model, and operating at least one subsystem of the vehicle in response to detecting that the classification indicates the presence of a person or animal in the at least one image. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0010] Figure 1 is a flow chart illustrating a method of detecting the presence of a living being in a locked vehicle according to some embodiments of the present disclosure.

[0011] Figure 2 is a flow chart illustrating a method of automatically operating a vehicle in response to detecting a living being locked within the vehicle according to some embodiments of the present disclosure.

[0012] Figure 3 is a flow chart illustrating a method of operating a vehicle subsystem in response to detecting a temperature that is in an out-of-range condition according to some embodiments of the present disclosure.

[0013] Figure 4 is a flow chart illustrating a method of taking emergency measures immediately after detecting a trapped passenger according to some embodiments of the present disclosure.

[0014] Figure 5 is a block diagram illustrating a vehicle computing system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0015] The disclosed embodiments describe apparatus, methods, and computer-readable media for detecting the presence of a living being trapped in a vehicle using an ML model. By using an ML model and an onboard camera, the disclosed embodiments can more reliably predict the presence of a living being in a vehicle. In response to detecting a living being locked in a vehicle, the disclosed embodiments further describe techniques for assisting such a living being without assistance from the vehicle driver.

[0016] Figure 1 is a flow chart illustrating a method of detecting the presence of a living being in a locked vehicle according to some embodiments of the present disclosure.

[0017] In box 102, the method detects that the vehicle is stopped. In some embodiments, the method further detects (in box 102) whether the vehicle is locked after the creature stops.

[0018] In one embodiment, the method receives signals indicating when the vehicle is stopped, parked, and / or locked via a communication bus (e.g., a controller area network). In the illustrated embodiment, these signals are monitored on the bus and used to determine when to initiate the remaining blocks of the method.

[0019] In a first embodiment, the method may execute the remaining blocks upon determining that the vehicle is stopped. In this embodiment, the method may (as an example) monitor the speed of the car and determine that the vehicle has stopped when the speed is zero or close to zero. Alternatively, or in conjunction with the above, the method may analyze a signal indicating that the brakes have been applied.

[0020] In a second embodiment, the method may determine whether the vehicle is parked in a parking lot. In some embodiments, this embodiment may be performed after determining that the vehicle has stopped or may be performed independently. In one embodiment, the method determines that the vehicle is parked in a parking lot by detecting a signal generated by a transmitter control unit in response to the driver parking the vehicle in a parking lot. Other signals may be used alone or in combination with such techniques, including the use of automatic parking assistance or other similar techniques.

[0021] In a third embodiment, the method may determine whether one or more doors of the vehicle are locked. In some embodiments, this embodiment may be performed after determining that the vehicle has stopped or after the vehicle is parked in a parking lot. Alternatively, this third embodiment may be performed independently. In one embodiment, the method monitors the bus for a signal from a door lock control unit indicating that a user has locked the vehicle doors. This signal may be generated in response to detecting that a user has locked the vehicle doors by a remote control pouch, a mobile application, or another device.

[0022] In some embodiments, the vehicle may be configured to provide a dedicated signal to proceed to the remaining blocks. In this embodiment, the vehicle broadcasts a start signal via the bus causing the method to proceed. In one embodiment, this start signal may be issued after the vehicle internally executes any one or more of the three embodiments described above. Thus, the three embodiments described above are still executable but may be executed by the vehicle's operating system rather than by the method.

[0023] In block 104 , the method activates one or more onboard cameras.

[0024] In the illustrated embodiment, block 104 is performed only after a positive determination in block 102. That is, the onboard cameras are activated only after the vehicle is stopped and when it is enabled, locked, and parked. Alternatively, in some embodiments, the onboard cameras may be used by other applications during operation of the vehicle. In this scenario, the method reuses these onboard cameras when the vehicle is locked.

[0025] In one embodiment, activating a camera includes starting the camera for operation. The specific steps for activating a camera are unique to each type of camera and details of such specific procedures are not included herein. In one embodiment, activation of a camera may additionally include focusing or moving the camera field of view based on a set of rules or preferences for camera placement. For example, a camera embedded in the center windshield may be moved to focus on the front passenger seat, thereby removing at least a portion of the driver's seat from the field of view. By making this adjustment, the camera may record more of the seat of interest (containing potential passengers).

[0026] In some embodiments, a separate model may be used to orient the onboard camera. Specifically, an ML model that can recognize vehicle features may be used to orient the camera. That is, the camera may be moved and frames analyzed until the ML model predicts that the camera is focused on a desired area (e.g., a particular seat).

[0027] In other embodiments, activation of the camera may include setting the camera to record continuously until stopped. Alternatively, or in conjunction with the above, the activation may also cause the camera to pivot or move to capture as much of the possible field of view as technically possible, if possible.

[0028] In block 106 , the method captures an image of the interior of the vehicle.

[0029] In one embodiment, the method is performed by a dedicated processor that receives raw image frames from individual cameras. Each of these frames may be time stamped by the camera device or by the processor. Thus, the method receives a collection of one or more frames at various points in time when the vehicle is locked.

[0030] In some embodiments, as part of block 106, the method may pre-process the frame to improve the prediction accuracy of the ML model (as discussed in block 108). In one embodiment, the method may clean the image to remove artifacts or other noise elements. For example, the method may apply one or more transformations to remove structural elements of the vehicle interior (e.g., windows, doors) while retaining pixels that capture the vehicle seats. In some embodiments, this pre-processing is optional.

[0031] In block 108 , the method classifies the image using the ML model.

[0032] In the illustrated embodiment, the ML model comprises a prediction model trained using classified data. In one embodiment, the ML model comprises a neural network, such as an artificial neural network (ANN), a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, etc. In the illustrated embodiment, the number of layers of the model can be configured based on experimentation and training. Alternatively, the prediction model can include a non-neural network, such as a support vector machine (SVM), a decision tree, or a naive Bayes classifier.

[0033] In the illustrated embodiment, the ML model is trained using classified image data. Specifically, images of creatures in the vehicle are recorded and labeled as containing such creatures. These labeled images are used to train the ML model, ultimately resulting in a set of weights (e.g., in the case of a neural network) and other model parameters. These parameters are then stored by the vehicle and used to predict whether new images contain creatures. In some embodiments, the onboard network can be configured to retrain the network based on actual images captured and confirmed labels. For example, if the ML model predicts the presence of a creature and alerts the relevant authorities (see Figure 4 ), but later the predictions are identified as false positives, then these images can be properly classified and used to retrain the ML model.

[0034] In one embodiment, the ML model may include a model that detects living things regardless of orientation. In this embodiment, the ML model is typically trained using pictures of living things and can be quickly trained using any image data containing humans or animals. In some embodiments, existing ML models can be reused (and enhanced) in this embodiment.

[0035] In another embodiment, the ML model may include a model trained using images of creatures in a vehicle. In this embodiment, the training dataset contains only data recorded within the vehicle and does not contain any images outside of the vehicle. This type of training process provides performance improvements due to the model parameters being fine-tuned to the automotive context.

[0036] In another embodiment, the ML model may include a model trained using images of creatures in a specific make and / or model of vehicle. This embodiment is similar to the previous model, however, the training data is generated on a per-vehicle basis. In some embodiments, the model may be generated by the manufacturer during production and then installed in every vehicle manufactured. In this embodiment, the manufacturer may generate sets of images for each model of vehicle and use this data to train the ML model.

[0037] In some embodiments, the ML model may be further fine-tuned based on the orientation of the camera. For example, there may be a separate ML model for each camera position. In conjunction with the previous example, these per-camera models may also be generated on a make / model basis. Thus, a given make or model of vehicle may have multiple models for each camera position. In some embodiments, the ML model stored in the vehicle's memory system and used to predict the presence of a living being then varies depending on the make and model of the vehicle and the specific orientation of the camera in the vehicle. In other embodiments, the ML model may be stored in the memory of an individual camera.

[0038] Alternatively, or in conjunction with the above, the ML models described above may be stored remotely. In this embodiment, a central server manages all ML models. Thus, when images are captured in block 106, they are transmitted to the central server for classification. The central server then returns the classification results to the vehicle. In this way, the system minimizes the total number of ML models and can centralize model training and retraining, thereby quickly improving the prediction accuracy of the models.

[0039] Alternatively, or in conjunction with the above, the models generated by the central server may be distributed to the relevant vehicles. In this embodiment, the server may periodically update the onboard ML models as the models are optimized. In this way, the central server may take advantage of the large amount of data collected, while the individual onboard models may be used for rapid predictions (including predictions in areas without access to wide area (e.g., cellular) networks).

[0040] In block 110, the method analyzes the results of the ML classification. The classification analyzed in block 110 typically includes a positive classification (i.e., a passenger was detected in the vehicle) or a negative classification (i.e., no passenger was detected). In some embodiments, the classification is further associated with a confidence level. In this context, the method may only use positive classifications with a confidence level above a preconfigured threshold. This threshold may be stored in a vehicle or user profile or rule setting.

[0041] In block 112, after a negative classification is detected (i.e., no passenger is detected), the method gradually decreases the time interval. In one embodiment, this time interval controls when the method should stop (or pause) capturing images. Specifically, in some embodiments, continuously capturing images in a vehicle when no living being is locked inside the vehicle is a waste of computing resources and energy. For example, when a driver parks a car at work during the day. As another example, parking a car during the night after returning from work will result in a considerable length (e.g., fourteen hours) of recording and processing time.

[0042] To avoid this overfitting, the method uses a time interval that controls how often block 106 (and subsequent blocks) should be executed. In one embodiment, this time interval is set to a very short period of time (e.g., one second) when the vehicle is first stopped. In block 112, the method begins increasing the time interval length when a negative classification is received. In some embodiments, the method exponentially increases the time interval length so that the length increases slowly immediately after stopping and increases rapidly when a negative classification is received. Other increasing functions (e.g., a power function or a linear function) may be used.

[0043] In box 114, the method determines whether the time interval is still valid. In some embodiments, the method determines that the time interval is valid if the length of the time interval is less than a preconfigured value. This value may be stored in a user or vehicle profile or rule setting. As an example, the method may stop processing when the time interval is set to thirty minutes. If the time interval is within the preconfigured length, the method continues to process images at box 106 and below. Alternatively, when the time interval exceeds the preconfigured length, the method stops recording images and ends. If an exponentially increasing time interval is used, the time interval will increase so quickly that the method actually stops recording images and only maintains the timer after a number of repetitions.

[0044] In some embodiments, block 114 is optional. In this embodiment, the method increases the time interval according to an increasing function and continues to capture images based on the increasing time intervals.

[0045] In some embodiments, a time interval mechanism may not be used, and the method may instead use other sensors to determine when to perform block 106. For example, the method may use a motion or weight sensor to determine when to operate block 106. That is, block 106 may be performed when weight is detected on a seat or when motion is detected in a vehicle. In some embodiments, time interval processing may also be used with interrupt style events. In this way, the method may be combined with weight or motion sensing technology to improve biometric detection.

[0046] In box 116 , after the method detects a living being in the vehicle, the method transmits an alert.

[0047] In some embodiments, block 116 is optional. If implemented, the method may transmit an alert to the driver of the vehicle. In some embodiments, the vehicle maintains a preset profile for the driver, containing contact information, such as a mobile phone number, user account id, email address, or other data. The method uses this contact information to transmit an alert via a radio interface over a wide area network (e.g., a cellular network). In one embodiment, the alert may include a text (SMS) message or the like. In other embodiments, the driver's mobile device contains a mobile application and the alert may include a push notification to the user's mobile device.

[0048] In box 118, the method proactively assists the passenger.

[0049] In some embodiments, block 118 is optional. However, in the illustrated embodiment, either block 116 or block 118 is performed, while in some embodiments, both blocks are performed. In block 118, the method proactively operates one or more vehicle subsystems to assist the detected passenger. In the following Figures 2 to 4 Various details of these operations are described in more detail in , and the disclosure of which is not repeated here.

[0050] Figure 2 is a flow chart illustrating a method of automatically operating a vehicle in response to detecting a living being locked within the vehicle according to some embodiments of the present disclosure.

[0051] In block 202 , the method monitors a cabin temperature of a vehicle.

[0052] In the illustrated embodiment, the vehicle is equipped with one or more on-board temperature sensors that continuously monitor the instantaneous temperature inside the vehicle cabin. In some embodiments, the vehicle may have multiple temperature sensors installed in different locations. In the illustrated embodiment, the method monitors the temperatures reported by these sensors. In some embodiments, the method averages the readings of multiple sensors (if multiple sensors are installed). In other embodiments, the method selects the highest value recorded by a sensor (if multiple sensors are installed). In some embodiments, each camera may be equipped with a temperature sensor and therefore the number of temperature readings is equal to the number of cameras.

[0053] In some embodiments, the method may use an external temperature sensor in addition to or in place of the in-car sensor. As used herein, an external temperature sensor refers to a temperature sensor mounted on the outside of the car. In this embodiment, the method may use the external temperature to predict the internal temperature. In some embodiments, this prediction may be accomplished with a temperature prediction model. In this embodiment, the temperature prediction model may be trained using the external temperature and the corresponding internal temperature recorded by the vehicle during operation (or otherwise). Alternatively, in one embodiment, the method may proceed based solely on the external temperature. For example, a rule or setting may indicate that the method should proceed if the external temperature is above a preconfigured threshold (regardless of the car temperature).

[0054] In block 204, the method determines whether the cabin temperature exceeds a predefined threshold. In one embodiment, the method compares the measured cabin temperature identified in block 202 to a predefined threshold. Alternatively, the method may compare the outdoor (exterior) temperature to a predefined outdoor threshold. In some embodiments, these thresholds may be set by the manufacturer or may be set by the driver or owner of the vehicle.

[0055] In block 206, the method operates one or more vehicle subsystems when the temperature exceeds a predefined threshold. In the illustrated embodiment, these subsystems may include an air conditioning system, a window subsystem, a communication subsystem, or other subsystems. Figure 3 and 4 Details of operating such subsystems are provided in and are only briefly described below.

[0056] In one embodiment, the method opens one or more windows of the vehicle in block 206. In this embodiment, the method uses an electronic control unit (ECU) associated with a given window to open or lower the window. In some embodiments, the method may further identify the window closest to the living being locked in the vehicle based on the identity of the camera detecting the user.

[0057] In another embodiment, the method may operate the vehicle's heating, ventilation and air conditioning (HVAC or AC) system. In this embodiment, the method may operate the AC system to adjust the temperature to a desired level. In some embodiments, this may be set by the owner of the vehicle.

[0058] In another embodiment, the method may transmit a notification to an emergency system. For example, the method may dial 9-1-1 (in the United States) and play a pre-recorded message. In other embodiments, an E911 service may be used. In some embodiments, the method may report the vehicle's global positioning (GPS) coordinates and the number of occupants in the car and the length of time via a text-to-speech processing module.

[0059] In another embodiment, the method may operate the horn or lights of the vehicle to attempt to draw attention to the vehicle. Of course, other types of subsystems may be operated and the above examples are not intended to limit the breadth of the present disclosure. In addition, the above examples (and other examples) may be combined in various ways.

[0060] In block 208 , the method determines whether to enable an alternative trigger upon determining that the temperature does not exceed the preconfigured threshold.

[0061] In the illustrated embodiment, after determining that the temperature has not exceeded the threshold, the method may then determine whether the method should still proceed to operate the vehicle subsystem. In block 208, other sensor types may be used to determine whether a trigger has occurred.

[0062] For example, a humidity sensor may be used to detect the presence of excess moisture (attributable to exhalation). In this embodiment, the method determines whether the current humidity level is greater than a preconfigured threshold. Alternatively, the method may monitor the humidity level over time to determine if there is a constant increase in moisture.

[0063] As another example, the method may monitor the sound level in the vehicle. Typically, if the occupant is in the vehicle and the sound level is low, the method may determine not to operate the vehicle subsystem. However, if the sound level is high, this may indicate that the occupant is shouting for help. Therefore, the method will use the sound level (in decibels) as a trigger value and operate the vehicle subsystem when the sound level exceeds a threshold (block 206).

[0064] As another example, a timer may be used to trigger block 206. In this embodiment, the owner or manufacturer of the vehicle may set a fixed time in which a person should remain in a locked vehicle. For example, a timer may be set to trigger block 206 after a creature has been locked in the vehicle for fifteen minutes. Of course, other values ​​may be used.

[0065] As another example, the operation of a component of the vehicle while it is locked can trigger block 206. For example, after the driver locks the vehicle and leaves, the method can monitor the operation of the vehicle to determine if suspicious activity involving the vehicle's control system occurs. For example, the behavior of pressing the horn or attempting to open a door after the vehicle is parked and locked can trigger block 206.

[0066] These examples may be combined in any manner, and other examples may be implemented based on the nature of the vehicle.The above examples should not be construed as limiting the embodiments.

[0067] In box 210, the method determines whether the monitoring routine should be cleared. In the illustrated embodiment, box 210 is performed after the method determines that temperature and external triggers do not cause the method to operate a vehicle subsystem. In the illustrated embodiment, box 210 detects that the method should be cleared when the vehicle is unlocked and / or started. In other words, Figure 3 The method described in is configured to operate until an emergency situation is detected (triggering block 206 ) or the owner / operator of the vehicle returns and unlocks the vehicle.

[0068] Figure 3 is a flow chart illustrating a method of operating a vehicle subsystem in response to detecting a temperature that is in an out-of-range condition according to some embodiments of the present disclosure.

[0069] In block 302, the method determines whether the vehicle's cabin temperature is out of range. In the illustrated embodiment, out of range means the temperature is above a threshold (in the case of hot weather) or below a threshold (in the case of cold weather). Details of this detection can be found in Figure 2 , and specifically, can be seen in blocks 202 and 204.

[0070] In block 304, the method opens one or more windows of the vehicle. In the illustrated embodiment, the processor sends an instruction to the window ECU to cause the window to open. In some embodiments, the method sends this instruction to all windows of the vehicle. In some embodiments, the instruction causes the window to be partially opened or fully opened.

[0071] In the illustrated embodiment, block 304 is configured to be executed first. In this embodiment, block 304 may be configured to be executed when the monitored temperature is between a first threshold and a second threshold. These thresholds define a first range beyond the first threshold, and the degree of opening of the window is sufficient to reduce the temperature to the desired range.

[0072] In block 306, the method continues to monitor the cabin temperature after the windows are opened. Figure 2 Blocks 202 and 204 are completed in the same manner as described above.

[0073] In box 308, the method determines whether the monitored temperature is within range. As used, in box 308, the cabin temperature is within range when it is between a previously defined first threshold and a second threshold. That is, the opening of the windows has prevented the interior temperature from exceeding the second threshold. In most embodiments, when the temperature is not excessive, this second threshold is not crossed, and opening the windows provides adequate relief.

[0074] However, in some cases, opening the windows will not sufficiently cool the cabin. Figure 3In cold weather conditions, blocks 304 and 306 are not performed, but instead the method proceeds directly to attempting to adjust the temperature using the HVAC system.

[0075] If the temperature is within the range, the method continues to monitor the temperature. This monitoring may continue until the temperature is outside the range or the vehicle is unlocked ( Figure 2 In one embodiment (not illustrated), the method may also determine whether the temperature has returned to below the first threshold. If so, the method may operate the windows to close each window and return to block 302 when the temperature is out of range again.

[0076] In block 310 , the method operates the HVAC system of the vehicle.

[0077] In the illustrated embodiment, the HVAC system of the vehicle refers to any component configured to heat or cool the vehicle via, for example, forced air flow. In addition, the HVAC system may include any heating or cooling elements, such as seat heating or cooling elements.

[0078] In block 310, during warm weather conditions, the method determines that the opening of the windows has not caused a decrease or maintenance in the temperature of the cabin. Therefore, the method operates the vehicle's air conditioning system to further cool the vehicle. In some embodiments, the method may also raise the vehicle's windows while operating the air conditioning system. Conversely, during cold weather conditions, the method activates the vehicle's heating system to begin heating the vehicle.

[0079] In block 312, the method monitors the cabin temperature while operating the HVAC system. This may be done in block 302 and Figure 2 Blocks 202 and 204 are completed in the same manner as described above.

[0080] In block 314, the method determines whether the cabin temperature is within a second range. In one embodiment, the second range includes a second threshold (described in block 308) and a third threshold. In warm weather conditions, the third threshold is greater than the second threshold, and in cold weather conditions, the third threshold is lower than the second threshold. As with the first and second thresholds, the third threshold may be set by the manufacturer or may be stored in a preset file set by the owner of the vehicle.

[0081] If the monitored temperature is still within the range, the method continues to operate the HVAC system in block 310. In some embodiments (not illustrated), the method may also stop operating the HVAC system if the temperature changes to be between the first threshold and the second threshold. In this scenario, the method may return to block 306 to monitor the cabin temperature without operating the HVAC system.

[0082] In block 316, the method takes emergency measures when the temperature cannot be maintained via the above blocks. In this scenario, the opening of the windows and the operation of the HVAC system fail to maintain the cabin temperature at an acceptable level. In general, block 316 is primarily performed in extreme temperature scenarios (e.g., extremely hot or extremely cold weather) or for long durations (i.e., when operations have exhausted available energy in the form of gasoline or batteries). Figure 4 Details of taking emergency measures are provided in the description of , which is incorporated herein.

[0083] Figure 4 is a flow chart illustrating a method of taking emergency measures immediately after detecting a trapped passenger according to some embodiments of the present disclosure.

[0084] In block 402, the method alerts the relevant authorities. Figure 3 As described in , block 402 may be performed immediately after detecting that the vehicle is unable to adequately adjust the cabin temperature while a passenger is locked in the vehicle. In an alternative embodiment, Figure 4 The method in may be performed separately from the temperature control routine.

[0085] In the illustrated embodiment, the method alerts the appropriate authorities by transmitting an alert to an emergency system. In some embodiments, this includes transmitting a text (SMS) message or similar message to a designated emergency system. In other embodiments, the vehicle may automatically initiate a cellular telephone call (e.g., 9-1-1 or E-911) to an emergency dispatch system. If (E)911 is used, the vehicle may generate an automated message to play to the dispatcher. For example, the message may include the license plate number, GPS coordinates, current temperature, length of time the vehicle has been locked, and other details. This message may be played immediately after a connection to the dispatcher is detected.

[0086] In some embodiments, the method may implement an interactive voice response (IVR) system. This system may allow an emergency dispatcher to control the vehicle via an emergency call. For example, the IVR system may allow access to cameras in the vehicle as well as external cameras. The IVR system may also allow the dispatcher to open the doors of the vehicle after verifying the safe angles in which the doors of the vehicle may be opened. In some embodiments, the IVR system may allow the dispatcher to enter the GPS coordinates where the vehicle should be driven to (if an autonomous vehicle).

[0087] In block 404, the method waits for the emergency responders until it is detected (block 406) that the emergency responders have arrived.

[0088] In the illustrated embodiment, the method waits for responders after alerting the appropriate authorities. In one embodiment, the method may utilize various sensors to determine that emergency personnel have arrived. For example, the method may use one or more external cameras to determine whether emergency personnel have arrived. In one embodiment, a trained model may be used to distinguish other objects from emergency personnel. In other embodiments, a microphone sensor may be used to recognize the presence of a sirens or other signals and use this presence to detect emergency personnel.

[0089] In block 408, the method determines whether it is safe to open the door of the vehicle. In one embodiment, the method may utilize a camera or GPS coordinates to orient the vehicle relative to the road or other hazards. For example, if the vehicle is parked along a street, the method may determine that one side of the vehicle is a sidewalk and open the door on this street side.

[0090] In box 410, the method determines that it cannot confirm that any door can be safely opened and therefore waits for the person to appear nearby. In one embodiment, this may be due to road conditions or other conditions outside the vehicle. In this scenario, the method may use sensors, such as radar, sonar, or lidar sensors to confirm that the emergency personnel are nearby. In this scenario, the method may open the door closest to the emergency personnel (box 412) to allow passengers to exit or allow the emergency personnel to enter the vehicle. As explained, if in box 408, the method determines that it is safe to open the door immediately after the emergency personnel are detected, then the method may open the door immediately.

[0091] Figure 5 is a block diagram illustrating a vehicle computing system according to some embodiments of the present disclosure.

[0092] Figure 5 The system described in can be installed completely within the vehicle. In some embodiments, some components (eg, components and subsystems other than subsystem (504)) can include existing autonomous and non-autonomous vehicle subsystems.

[0093] The system optionally includes an optional autonomous vehicle subsystem (502). In the illustrated embodiment, the autonomous vehicle subsystem (502) includes a map database (502a), a radar device (502b), a lidar device (502c), a digital camera (502d), a sonar device (502e), a GPS receiver (502f), and an inertial measurement unit (502g). Each of the components of the autonomous vehicle subsystem (502) includes standard components provided in the latest autonomous vehicles. In one embodiment, the map database (502a) stores multiple high-definition three-dimensional maps for route selection and navigation. As known in the art, the radar device (502b), the lidar device (502c), the digital camera (502d), the sonar device (502e), the GPS receiver (502f), and the inertial measurement unit (502g) may include various corresponding devices installed at various locations throughout the autonomous vehicle. For example, these devices may be installed along the perimeter of the autonomous vehicle to provide position awareness, collision avoidance, and other standard autonomous vehicle functionality. In some embodiments, the autonomous subsystem (502) is built into the vehicle, while in other embodiments, the autonomous subsystem (502) comprises an aftermarket system.

[0094] A vehicle subsystem (506) is additionally included in the system. The vehicle subsystem (506) includes various anti-lock braking systems (506a), an engine control unit (502b), and a transmission control unit (502c). These components may be used to control the operation of the vehicle in response to data generated by the autonomous vehicle subsystem (502) and / or the ADAS subsystem (504). Standard autonomous vehicle interactions between the autonomous vehicle subsystem (502) and the vehicle subsystem (506) are well known in the art and are not described in detail herein.

[0095] The processing side of the system includes one or more processors (510), short-term memory (512), an RF system (518), a graphics processing unit (GPU) (516), long-term storage (518), and one or more interfaces (520).

[0096] The one or more processors (510) may include a central processing unit, an FPGA, or any range of processing devices required to support the operation of the autonomous vehicle. The memory (512) includes DRAM or other suitable volatile RAM for temporarily storing data required by the processor (510). The RF system (518) may include a cellular transceiver and / or a satellite transceiver. The long-term storage device (514) may include one or more high-capacity solid-state drives (SSDs). In general, the long-term storage device (514) may be used to store, for example, high-definition maps, routing data, and any other data that requires permanent or semi-permanent storage. The GPU (516) may include one or more high-throughput GPU devices for processing data received from the autonomous vehicle subsystem (502a). Finally, the interface (520) may include various display units (e.g., built-in screens) located within the autonomous vehicle.

[0097] The system further comprises a locked passenger subsystem (504) that performs the operations required by the method described in the previous figures. The black box subsystem (504) comprises a machine learning model (504a) that stores Figure 1 The black box subsystem (504) includes various subsystems of the vehicle (such as Figure 2-4 ) and a set of rules or preset files (504c) that control the operation of the subsystem control model. Figures 2 to 4 The details of the subsystem control module (504b) and the rules / preset files (504c) are more fully described in the description of , and their disclosure is not repeated here.

[0098] In some embodiments, it should be understood that the method steps described above can be implemented as a continuous process, for example, each step can be independently operated by monitoring input data, performing operations, and outputting data to subsequent steps. And, such steps for each method can be implemented as a discrete event process, for example, each step can be triggered by an event that it should trigger and produce a certain output. It should also be understood that each figure represents a minimum method within a possible larger method of a computer system that is more complex than the computer system presented in the description. Therefore, the steps depicted in each figure can be combined with other steps that supply access to other steps associated with the larger method of the more complex system.

[0099] It should be understood that unless otherwise specified, the vehicle described herein can be any type of vehicle. The vehicle may include automobiles, trucks, ships and aircraft, as well as vehicles or vehicle equipment for military, construction, agronomy or leisure purposes. The electronic devices used by the driver or passenger of a vehicle, a vehicle part or a vehicle may be considered as vehicle electronics. The vehicle electronics may include electronic devices for engine management, igniter, radio, onboard computer, vehicle information service, vehicle entertainment system and other parts of the vehicle. The vehicle electronics may be used with or for the igniter and engine and transmission control, which may be seen in vehicles with internal combustion driven machinery such as gasoline-driven automobiles, trucks, motorcycles, ships, aircraft, military vehicles, forklifts, tractors and excavators. In addition, the vehicle electronics may be used by or with related elements for controlling the electrical systems seen in hybrid vehicles and electric vehicles such as hybrid vehicles or electric vehicles. For example, electric vehicles may use power electronics for main propulsion motor control, and manage battery systems. Furthermore, autonomous vehicles rely almost entirely on vehicle electronics.

[0100] Some portions of the previous detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing can most effectively convey the substance of their work to others skilled in the art. In this article, and generally, an algorithm is conceived to be a self-consistent sequence of operations that produce a desired result. An operation is one that requires physical manipulation of physical quantities. Typically (but not necessarily), these quantities take the form of electrical or magnetic signals that can be stored, combined, compared, and otherwise manipulated. It has proven to be convenient at times, primarily for common reasons, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, and the like.

[0101] It should be borne in mind, however, that all of these and similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure may refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage systems.

[0102] The present disclosure also relates to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the desired purpose, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer-readable storage medium, such as any type of disk (including floppy disks, optical disks, CD-ROMs, and magnetic optical disks), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0103] The algorithms and displays presented herein are not inherently related to any computer or other device. Various general purpose systems can be used with programs according to the teachings herein, or the general purpose systems can prove to be convenient for construction to perform more specialized devices for the methods described. The structures of various these systems will be presented as set forth in the following description. In addition, the present disclosure is described without reference to any programming language. It should be understood that the teachings of the present disclosure as described herein can be implemented using various programming languages.

[0104] The present disclosure may be provided as a computer program product or software, which may include a machine-readable medium having stored thereon instructions that can be used to program a computer system (or other electronic device) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). In some embodiments, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) readable storage medium, such as a read-only memory ("ROM"), a random access memory ("RAM"), a magnetic disk storage medium, an optical storage medium, a flash memory component, etc.

[0105] In the foregoing description, embodiments of the present disclosure have been described with reference to specific example embodiments thereof. It will be apparent that various modifications may be made to the present disclosure without departing from the broader spirit and scope of the embodiments of the present disclosure as set forth in the appended claims. Therefore, the description and drawings should be viewed in an illustrative sense rather than a restrictive sense.

Claims

1. A method for operating a vehicle, comprising: receiving signaling from one or more sensors of the vehicle, the signaling instructing the vehicle to stop; activating at least one camera in response to receiving the signaling and recording at least one image of the vehicle interior using the at least one camera; classifying the at least one image using a machine learning model; operating at least one subsystem of the vehicle in response to detecting that the classification indicates the presence of a person or an animal in the at least one image; and In response to detecting that the classification does not indicate that the person or the animal is present in the at least one image, the classification frequency is gradually reduced, and the gradual reduction includes one or both of extending the time interval between capturing subsequent images of the vehicle interior and the time interval between classifying the subsequent images using the machine learning model. 2 . The method of claim 1 , wherein detecting that the vehicle is stopped comprises detecting that the vehicle is not in motion and one or more doors of the vehicle are locked.

3. The method of claim 1, wherein classifying the at least one image using the machine learning model comprises classifying the at least one image using a neural network. 4 . The method of claim 1 , further comprising recording a second image in response to determining that the at least one image does not contain a person or an animal. 5 . The method of claim 1 , said operating the at least one subsystem comprising monitoring a temperature of the vehicle and operating the at least one subsystem in response to detecting that the temperature exceeds a predefined threshold. 6 . The method of claim 5 , wherein operating the at least one subsystem comprises operating an air conditioning system of the vehicle.

7. The method of claim 5, wherein operating the at least one subsystem comprises opening at least one window of the vehicle.

8. The method of claim 5, said operating said at least one subsystem comprising transmitting an alert to an emergency system.

9. A non-transitory computer readable storage medium for tangibly storing computer program instructions executable by a computer processor, the computer program instructions defining the following steps: receiving signaling from one or more sensors of a vehicle, the signaling instructing the vehicle to stop; activating at least one camera in response to receiving the signaling and recording at least one image of the vehicle interior using the at least one camera; classifying the at least one image using a machine learning model; operating at least one subsystem of the vehicle in response to detecting that the classification indicates the presence of a person or an animal in the at least one image; and In response to detecting that the classification does not indicate that the person or the animal is present in the at least one image, the classification frequency is gradually reduced, and the gradual reduction includes one or both of extending the time interval between capturing subsequent images of the vehicle interior and the time interval between classifying the subsequent images using the machine learning model. 10 . The non-transitory computer-readable storage medium of claim 9 , wherein detecting that the vehicle is stopped comprises detecting that the vehicle is not in operation and one or more doors of the vehicle are locked.

11. The non-transitory computer-readable storage medium of claim 9, wherein classifying the at least one image using the machine learning model comprises classifying the at least one image using a neural network.

12. The non-transitory computer readable storage medium of claim 9, the instructions further defining the step of recording a second image in response to determining that the at least one image does not contain a person or an animal. 13 . The non-transitory computer-readable storage medium of claim 9 , the operating the at least one subsystem comprising monitoring a temperature of the vehicle and operating the at least one subsystem in response to detecting that the temperature exceeds a predefined threshold.

14. The non-transitory computer-readable storage medium of claim 13, the operating the at least one subsystem comprising operating an air conditioning system of the vehicle. 15 . The non-transitory computer-readable storage medium of claim 13 , the operating the at least one subsystem comprising opening at least one window of the vehicle.

16. The non-transitory computer-readable storage medium of claim 13, the operating the at least one subsystem comprising transmitting an alert to an emergency system.

17. An apparatus for operating a vehicle, comprising: processor; and A storage medium for tangibly storing thereon programming logic for execution by the processor, the programming logic causing the processor to perform the following operations: receiving signaling from one or more sensors of the vehicle, the signaling instructing the vehicle to stop, activating at least one camera in response to receiving the signaling and recording at least one image of the vehicle interior using the at least one camera, classifying the at least one image using a machine learning model, operating at least one subsystem of the vehicle in response to detecting that the classification indicates the presence of a person or an animal in the at least one image; and In response to detecting that the classification does not indicate that the person or the animal is present in the at least one image, the classification frequency is gradually reduced, and the gradual reduction includes one or both of extending the time interval between capturing subsequent images of the vehicle interior and the time interval between classifying the subsequent images using the machine learning model.

18. The apparatus of claim 17, said detecting that the vehicle is stopped comprises detecting that the vehicle is not in motion and one or more doors of the vehicle are locked.

19. The apparatus of claim 17, wherein classifying the at least one image using the machine learning model comprises classifying the at least one image using a neural network.

20. The apparatus of claim 17, wherein operating the at least one subsystem comprises monitoring a temperature of the vehicle and operating the at least one subsystem in response to detecting that the temperature exceeds a predefined threshold, wherein the at least one subsystem comprises a subsystem selected from the group consisting of an air conditioning system, a window subsystem, and a communication subsystem.

Images