System and method for identifying organism around standstill vehicle

Abstract

To provide a system and a method for informing an operator of a standstill vehicle about an organism existing near the standstill vehicle.SOLUTION: In one embodiment, the method includes identifying an organism existing near a standstill vehicle from sensor data collected from a sensor of the standstill vehicle. The method also includes tracking the movement of the organism existing near the standstill vehicle and determining the final period in which the organism exists within the visual field of the sensor based on the sensor data. The method also includes presenting notification for identifying the organism and the final period in which the organism exists within the visual field of the sensor.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The subject matter described herein relates generally to tracking the movement of objects around a stationary vehicle, and more specifically to notifying a user about living creatures in the vicinity of a stationary vehicle, even when the living creatures may currently be outside the field of view of the stationary vehicle's sensors. [Background technology]

[0002] A vehicle may be equipped with sensors that facilitate the perception of other vehicles, obstacles, pedestrians, and additional aspects of the surrounding environment. For example, a vehicle may have a light detection and ranging (LiDAR) sensor that uses light to scan the surrounding environment. At the same time, logic associated with the LiDAR analyzes the acquired data to detect the presence of objects and other features in the surrounding environment. In a further example, additional / alternative sensors, such as cameras, may be implemented to acquire information about the surrounding environment from which the system derives its understanding of aspects of the surrounding environment. This sensor data may be useful in various situations to improve perception of the surrounding environment.

[0003] However, in some cases, the sensor system may have a limited field of view. Thus, some objects may remain undetected or may be located around the vehicle outside the field of view of the sensor system. Generally, the more awareness developed by the vehicle about its surrounding environment, the more enhanced the vehicle and safety system operation. Summary of the Invention

[0004] In one embodiment, the exemplary system and method relates to a method for improving identification of living beings, such as children and animals, in the vicinity of a stationary vehicle and informing the owner of the stationary vehicle when a living being was last recognized in the vicinity of the stationary vehicle so that the owner can take any appropriate safety precautions when operating the vehicle.

[0005] In one embodiment, a biological notification system is disclosed that identifies a biological organism in the vicinity of a stationary vehicle and generates a notification about the biological organism and the last time the biological organism was detected in the vicinity of the stationary vehicle. The biological notification system includes one or more processors and a memory communicatively connected to the one or more processors. The memory stores instructions that, when executed by the one or more processors, cause the one or more processors to identify a biological organism in the vicinity of the stationary vehicle from sensor data collected from sensors in the stationary vehicle. The memory also stores instructions that, when executed by the one or more processors, cause the one or more processors to track the movement of the biological organism in the vicinity of the stationary vehicle and determine from the sensor data the last time the biological organism was within the field of view of the sensor. The memory also stores instructions that, when executed by the one or more processors, cause the one or more processors to present a notification identifying the biological organism and the last time the biological organism was within the field of view of the sensor.

[0006] In one embodiment, a non-transitory computer-readable medium is disclosed for identifying and generating a notification about a living thing in the vicinity of a stationary vehicle, the medium including instructions that, when executed by one or more processors, cause the one or more processors to perform one or more functions. The instructions include instructions for identifying a living thing in the vicinity of the stationary vehicle from sensor data collected from a sensor in the stationary vehicle. The instructions also include instructions for tracking the movement of the living thing in the vicinity of the stationary vehicle and determining from the sensor data the last time the living thing was within the field of view of the sensor. The instructions also include instructions for presenting a notification identifying the living thing and the last time the living thing was within the field of view of the sensor.

[0007] In one embodiment, a method for identifying and generating a notification about a living thing in the vicinity of a stationary vehicle is disclosed. In one embodiment, the method includes identifying a living thing in the vicinity of the stationary vehicle from sensor data collected from a sensor on the stationary vehicle. The method also includes tracking the movement of the living thing in the vicinity of the stationary vehicle and determining from the sensor data the last time the living thing was within the field of view of the sensor. The method also includes presenting a notification identifying the living thing and the last time the living thing was within the field of view of the sensor. [Brief explanation of the drawings]

[0008] The accompanying drawings, which are incorporated herein and constitute a part of this specification, illustrate various systems, methods, and other embodiments of the present disclosure. It will be understood that the boundaries of elements shown in the figures (e.g., boxes, groups of boxes, or other shapes) represent one embodiment of the boundaries. In some embodiments, one element may be designed as multiple elements, or multiple elements may be designed as one element. In some embodiments, an element shown as an internal component of another element may be implemented as an external component, and vice versa. Additionally, elements may not be drawn to scale.

[0009] [Figure 1A] FIG. 1 illustrates an environment in which certain organisms may not be detected by environmental sensors in a stationary vehicle. [Figure 1B] FIG. 1 illustrates an environment in which certain organisms may not be detected by environmental sensors in a stationary vehicle. [Figure 2] FIG. 1 illustrates an embodiment of a vehicle in which the systems and methods disclosed herein may be implemented. [Figure 3] FIG. 1 illustrates an embodiment of a bio-notification system associated with identifying bio-notification organisms in the vicinity of a stationary vehicle. [Figure 4] FIG. 1 illustrates a flowchart of an embodiment of a method associated with identifying living organisms in the vicinity of a stationary vehicle. [Figure 5]FIG. 1 illustrates an embodiment of a bio-notification system that distinguishes between different types of organisms. [Figure 6] FIG. 1 illustrates an embodiment of a biological notification system that estimates the location of a biological organism. [Figure 7A] FIG. 1 illustrates an example of a notification generated by a bio-notification system. [Figure 7B] FIG. 1 illustrates an example of a notification generated by a bio-notification system. [Figure 8] FIG. 10 illustrates one embodiment of operations for generating a notification regarding an identified organism in the vicinity of a stationary vehicle. DETAILED DESCRIPTION OF THE INVENTION

[0010] Disclosed herein are systems, methods, and other embodiments related to improving detection of certain classes of organisms, such as children and animals, that may not be routinely detected by a vehicle's sensor system, and providing notifications about these detected organisms to ensure the safety of the detected organisms when an operator is operating the vehicle. Vehicles have become commonplace in many parts of the world. However, their presence poses inherent risks due to the relative size and strength of the vehicle to nearby humans and animals. For example, vehicles may be parked in public, multi-family, and private garages where adults, children, and animals may roam. Even when not in a garage, it is common for humans and animals to be in the vicinity of the vehicle. For example, people may walk past parked vehicles, and children may play around vehicles parked in driveways or loiter around adjacent vehicles in a parking lot. While adults may understand the dangers posed by vehicles and take appropriate measures to ensure their own safety, children and animals may not fully understand the situation and may not take appropriate measures around the vehicle.

[0011] Although a vehicle may be equipped with sensors such as cameras that provide the vehicle operator with a view of the vehicle's surroundings, some living things, such as children and animals, may be in areas around the vehicle that are outside the field of view of the sensor system. Because adults are taller, they are more likely to be detected by the vehicle sensors than smaller children and animals, which may not be in the vehicle's field of view even when they are in the vicinity of the vehicle. For example, a vehicle's backup camera may show a view behind the vehicle from about 3 feet above. Thus, small children and / or animals may be behind the vehicle but outside the camera's field of view. As another example, animals may seek shelter from bad weather near a warm engine under the vehicle. In either example, the vehicle operator may not be aware of the children and / or animals and therefore may not take any specific safety precautions.

[0012] Even when a child or animal is within the field of view of the vehicle sensor system, the vehicle operator may spend only a few seconds checking the sensor feed to determine whether the vehicle path is clear. For example, a side-view camera may be tilted downward to provide a clear view into a specific zone by the trunk and rear doors of the vehicle. However, the vehicle operator may only glance briefly at the side-view mirror before backing up. Clearly, an undesirable situation can arise when a child / animal is not recognized by the driver and is not detected by the rear-view or side-view camera, which may not be aware that the vehicle is backing up and / or may not be familiar with how to interact with a backing vehicle.

[0013] Thus, the bio-notification system herein alerts a vehicle operator of detected children or animals near a stationary vehicle. That is, the bio-notification system identifies unattended, dynamic living creatures near a vehicle and notifies individuals, such as the vehicle operator. The system relies on sensors (e.g., side-view cameras, rearview cameras, etc.) that provide clear visibility into specific zones near the trunk and rear doors of the vehicle, which may constitute blind spots for the vehicle operator, to capture low-to-the-ground moving objects, such as children and pets. The system detects dynamic and static objects (e.g., children, animals, etc.) that are alive and close to the ground. These objects may be beyond the field of view of the sensor system and therefore may not be detected by the vehicle operator. In particular, the bio-notification system may use machine learning models to detect low-to-the-ground objects using data from a ground-pointing camera or other sensor (e.g., a side-view camera for blind spot detection). Given that a child or pet may not be within the field of view of the sensor system when the operator enters the vehicle, the system may identify the last time the child or pet was within the field of view of the sensor system. The system generates a notification as a safety alert to the vehicle operator, indicating the last time the vehicle's sensors detected a child / pet.

[0014] Thus, the system may use supervised or unsupervised machine learning models to detect motion activity from the camera. If the motion activity exceeds a certain duration and the object is larger than a threshold size (which distinguishes living organisms from non-living organisms, such as light glare, dust particles, fallen leaves, etc.), the biological organism notification system activates a classification model that classifies the object as an adult, a child, or an animal.

[0015] To classify moving creatures, the creature notification system may implement dynamic object localization and machine learning models to distinguish between various creatures (e.g., taller humans, shorter humans, animals, birds, etc.). In a specific example, the creature notification system may use machine learning models to perform pose detection to determine the creature's pose, as the creature's pose may indicate whether the creature is passing by a vehicle or may still be in the vicinity of the vehicle. As an example, the creature notification system may estimate the creature's height and location from a camera viewpoint and estimate the location of a corresponding object near the vehicle. From these parameters, the creature notification system may determine whether the creature is of adult, infant, child, or short animal height. All of this information, such as classification, location, and pose, may assist in classifying the creature as an adult, child, infant, or animal.

[0016] Generally, with regard to notifications, a bio-notification system may identify children and / or animals near a stationary vehicle and generate a safety alert to notify the operator that a child or animal is near the vehicle. Specifically, the bio-notification system may generate an alert indicating the last time a child or animal was recognized near the vehicle, so that the operator can take appropriate corrective action. For example, a notification may indicate that a child was recognized playing behind a vehicle parked in a private garage at 12:30 PM. In this example, an operator entering the vehicle at 12:35 PM may receive a notification on their mobile device and take appropriate corrective action, such as checking behind and around the vehicle to ensure that the child is away from the vehicle before operating. Thus, the bio-notification system detects these smaller objects and triggers a notification to the vehicle operator that includes information about the detected object, such as whether it is a child or an animal, the area in which the object is detected, and a timestamp indicating when the object was last recognized. Thus, the operator may check the vicinity of the vehicle before safely backing out of the parking spot.

[0017] In one example, the bio-notification system may estimate the location of a living creature that has left the field of view of a camera. For example, an animal may be detected as moving beyond the field of view of a side-view camera. Based on pose information about the animal, the direction of the animal's movement, and the lack of detection of the animal by another camera, the bio-notification system may infer that the animal is taking up residence underneath the vehicle. In this example, in addition to indicating the last time the animal was recognized, the notification may indicate that the animal may be underneath the vehicle, an area that the driver may not normally check before operating the vehicle.

[0018] Thus, the present creature generation system provides a view of the vehicle's surroundings that may be undetectable by the vehicle operator / vehicle sensor system or may be casually checked prior to vehicle operation. Additionally, the notification provides a timestamp as to when the creature is confirmed so that the operator can determine whether the creature is still likely to be in the area. Furthermore, by estimating the object's location, the creature notification system not only identifies creatures currently observed, but also provides notification of creatures that may still be in the vehicle's danger zone even though they are not currently detected by the vehicle sensor system.

[0019] As mentioned above, in one example, all of this may occur while the vehicle is in a standby or off state. That is, the backup and side-view cameras may only be actively capturing data when the vehicle is turned on. Thus, these cameras may not detect objects, such as children or animals, around the vehicle when it is off. The cameras on which the system relies are battery-powered and may capture and process images to detect living creatures even when the vehicle is off. Thus, the bio-notification system provides continuous offline monitoring of the vehicle's surroundings, while in other cases, the vehicle may not be monitored when offline. That is, the bio-notification system herein may provide notifications about potential obstacles in the vehicle's path that would not previously have been generated due to the vehicle not being monitored while it was off.

[0020] 1A and 1B illustrate an environment in which certain living things may not be detected by the environmental sensors of a stationary vehicle 100. As mentioned above, the stationary vehicle 100 may be equipped with various environmental sensors, such as cameras, LiDAR sensors, radar sensors, and others, that detect objects in the environment surrounding the stationary vehicle 100. As a specific example, the vehicle 100 may be equipped with a backup camera 102 and a side-view camera 110. While the backup camera 102 certainly provides value in being able to inform the vehicle operator of objects around portions of the vehicle 100 that may be difficult or impossible for the driver to see while operating the vehicle 100, due to its limited field of view, some objects may still not be detected by these sensors. For example, even if the backup camera 102 is generally pointed toward the ground, a small child 106 may be below the field of view 104 of the backup camera 102. Additionally, side-view camera 110 may detect an average-height adult 114, but again, even though side-view camera 110 is generally pointed toward the ground, an animal 108, such as a dog, may not be within side-view camera 110's field of view 112. As another example, an animal 108 below the vehicle, as depicted in FIG. 1A, may not be detectable by any of vehicle 100's environmental sensors. The undetected creature may still be in a dangerous position relative to vehicle 100.

[0021] When a vehicle operator is observing the sensor feeds (e.g., reviewing the feeds of the backup camera 102 and / or the side-view camera 110), these objects may not be detected by the cameras, but the sensors may have detected objects at some point in the past that are currently outside the fields of view 104 and 112 of the respective cameras 102 and 110. Thus, the present creature notification system provides notification that a creature has previously been detected that may not currently be within the driver's field of view, so that the vehicle operator may recognize that a particular creature may still be in the vicinity of the vehicle, even if it is not currently detected. That is, the creature notification system provides an alert that a creature has recently been detected in the vicinity of the vehicle 100, even when the sensor system does not currently detect the creature.

[0022] Although Figures 1A and 1B specifically depict a rearview camera 102 and a sideview camera 110, the vehicle 100 may be equipped with different types of cameras (e.g., forward-facing cameras) and different types of sensors (e.g., LiDAR sensors and radar sensors) that can detect living organisms and transmit information to a living organism notification system for identification and tracking of the living organisms.

[0023] Referring to FIG. 2 , an example of a vehicle 100 is shown. As used herein, a “vehicle” is any form of transportation that may be motorized or otherwise powered. In one or more implementations, the vehicle 100 is an automobile. While features related to automobiles are described herein, it will be understood that embodiments are not limited to automobiles. In some implementations, the vehicle 100 may be in the form of a robotic device or vehicle that includes, for example, sensors that perceive aspects of the surrounding environment and therefore benefit from the functionality described herein associated with notifying an operator of objects in the vicinity of the vehicle 100 that may not currently be detected by the environmental sensors 227.

[0024] Vehicle 100 also includes various elements. It will be understood that in various embodiments, vehicle 100 may not necessarily have all of the elements shown in FIG. 2 . Vehicle 100 may have different combinations of the various elements shown in FIG. 2 . Furthermore, vehicle 100 may have additional elements to those shown in FIG. 2 . In some arrangements, vehicle 100 may be implemented without one or more of the elements shown in FIG. 2 . While various elements are shown as being located within vehicle 100 in FIG. 2 , it will be understood that one or more of the elements may be located external to vehicle 100. Furthermore, the elements shown may be physically separated by large distances. For example, as discussed, one or more components of the system of the present disclosure may be implemented within the vehicle, while additional components of the system are implemented in a cloud computing environment or other system remote from vehicle 100.

[0025] Some of the possible elements of vehicle 100 are shown in FIG. 2 and described in conjunction with subsequent figures. However, a description of many of the elements of FIG. 2 is provided following the description of FIGS. 3-8 for brevity of this description. Furthermore, it will be understood that, for ease and clarity of illustration, reference numerals have been appropriately repeated among different figures to indicate corresponding or similar elements. Additionally, the description outlines numerous specific details to provide a thorough understanding of the embodiments described herein. However, those skilled in the art will appreciate that the embodiments described herein may be implemented using various combinations of such elements. In either case, vehicle 100 includes bio-notification system 270, which is implemented to perform the methods and other functions disclosed herein with respect to improving vehicle 100's detection of living organisms in its vicinity.

[0026] As described in more detail below, in various embodiments, bio notification system 270 is implemented partially within vehicle 100 as a cloud-based service. For example, in one approach, functionality associated with at least one module of bio notification system 270 is implemented within vehicle 100, while additional functionality is implemented within a cloud-based computing system. Thus, bio notification system 270 may include a local instance within vehicle 100 and a remote instance that operates within a cloud-based environment.

[0027] Additionally, bio-notification system 270 provided within vehicle 100 functions in conjunction with communication system 280. In one embodiment, communication system 280 communicates according to one or more communication standards. For example, communication system 280 may include multiple different antennas / transceivers and / or other hardware elements that communicate at different frequencies according to respective protocols. In one arrangement, communication system 280 communicates via a communication protocol, such as WiFi, dedicated short-range communications (DSRC), vehicle-to-infrastructure (V2I), vehicle-to-vehicle (V2V), or another protocol suitable for communicating between vehicle 100 and other entities in a cloud environment. Furthermore, in one arrangement, communication system 280 also communicates according to a protocol such as Global System for Mobile Communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), Long Term Evolution (LTE), 5G, or another communication technology, providing vehicle 100 with communication with various remote devices (e.g., cloud-based servers). In either case, the bio-notification system 270 may utilize various wireless communication technologies to provide communications to other entities, such as elements of a cloud computing environment.

[0028] Referring to Figure 3, one embodiment of the bio-notification system 270 of Figure 2 is further illustrated. The bio-notification system 270 is illustrated as including the processor 218 of the vehicle 100 of Figure 2. As such, the processor 218 may be part of the bio-notification system 270, the bio-notification system 270 may include a processor separate from the processor 218 of the vehicle 100, or the bio-notification system 270 may access the processor 218 through a separate data bus or other communication path from the vehicle 100. In one embodiment, the bio-notification system 270 includes a memory 344 that stores an identification module 346 and a notification module 348. The memory 344 may be a random access memory (RAM), a read-only memory (ROM), a hard disk drive, a flash memory, or another suitable memory that stores the modules 346 and 348. The modules 346 and 348 are, for example, computer-readable instructions that, when executed by the processor 218, cause the processor 218 to perform various functions disclosed herein. In an alternative arrangement, modules 346 and 348 are elements separate from memory 344, e.g., comprising hardware elements. Thus, modules 346 and 348 are alternatively application specific integrated circuits (ASICs), hardware-based controllers, configurations of logic gates, or another hardware-based solution.

[0029] Additionally, in one embodiment, bio-notification system 270 includes data store 220. In one embodiment, data store 220 is an electronic data structure stored in memory 344 or another data storage device and comprised of routines that may be executed by processor 218 to analyze stored data, provide stored data, organize stored data, etc. Thus, in one embodiment, data store 220 stores data used by modules 346 and 348 in performing various functions.

[0030] Generally, data store 220 stores sensor data 222 on which identification module 346 relies to 1) identify and track living organisms in the vicinity of vehicle 100, and 2) identify the last time a living organism was detected in the vicinity of vehicle 100. Specifically, identification module 346 processes sensor data 222, such as camera images, to detect objects in the vicinity of vehicle 100, identify any living organisms in the vicinity of vehicle 100, classify living organisms in the vicinity of vehicle 100 (e.g., as adults, children, or animals), detect the pose of the living organisms, track the movements of the living organisms, and in some cases estimate the location of the living organisms even when they are not within the field of view of environmental sensors 227 of vehicle 100. Thus, sensor data 222 may include camera images, LiDAR sensor output, radar sensor output, sonar sensor output, and / or any output collected from any of the multiple environmental sensors 227 provided in vehicle 100.

[0031] In one embodiment, data store 220 stores sensor data 222 along with metadata that characterizes various aspects of the sensor data 222, for example. For example, as described above, it may be that a previously detected organism is still in the vicinity of vehicle 100, but is no longer within the field of view of the sensor. In this example, the metadata may indicate a time / date stamp for when the separate sensor data 222 was generated, and the time / date stamp indicates when the organism was detected. Identification module 346 and notification module 348 may rely on this time / date stamp information when identifying the organism and generating notifications. For example, notification module 348 may include a timestamp in the generated notification and may use the timestamp as a trigger for generating the notification.

[0032] As another example, the identification module 346 may rely on timestamp metadata to estimate the location of the organism. In this example, the metadata may also include location coordinates (e.g., longitude and latitude) for the object. The identification module 346 may similarly rely on this location metadata to estimate the location of the organism outside the frame. Additional details regarding estimating the location of the organism are provided below in connection with FIG. 6. While specific reference is made to particular metadata, other types of metadata may be included with the sensor data 222 associated with the other types of metadata.

[0033] In one embodiment, data store 220 further includes classification model 250, upon which identification module 346 may rely to detect, identify, classify, and track organisms in the vicinity of vehicle 100. In one example, organism notification system 270 may be a machine learning system. Generally, machine learning systems identify patterns based on previously unrecognized data. In the context of the present application, machine learning organism notification system 270 relies on some form of machine learning, whether supervised, unsupervised, reinforced, or any other type, to identify organisms in environmental sensor outputs, classify the organisms (e.g., as adults, children, animals, transient organisms, and / or prowling organisms), and track / estimate the organisms' movements around vehicle 100.

[0034] In one example, the classification model 250 is a supervised model, where the system is trained using an input dataset and optimized to meet a specific set of outputs. In another example, the classification model 250 is an unsupervised model, where the model is trained using an input dataset but is not optimized to meet a specific set of outputs; instead, it is trained to classify based on common features. As another example, the classification model 250 may be a self-training, reinforcement model based on trial and error. In either case, the classification model 250 includes weights (both trainable and non-trainable) biases, variables, offsets, algorithms, parameters, and other elements that operate to output the likely identity, class, and movement of a detected organism based on the sensor data 222. Examples of machine learning models include, but are not limited to, logistic regression models, support vector machine (SVM) models, naive Bayes models, decision tree models, linear regression models, k-nearest neighbor models, random forest models, boosting algorithm models, and hierarchical clustering models. Although a particular model is described herein, the classification model 250 can be of various types that are intended to identify and classify organisms based on determined characteristics.

[0035] Organism notification system 270 further includes an identification module 346, which, in one embodiment, includes instructions that cause processor 218 to 1) identify a living organism in the vicinity of stationary vehicle 100 from sensor data 222 collected from sensors in stationary vehicle 100, 2) track the movement of the living organism in the vicinity of stationary vehicle 100, and 3) determine from sensor data 222 the last time the living organism was within the field of view of the sensors. As described above, sensor system 225 of vehicle 100 can alert a vehicle operator about objects (e.g., living organisms) in the vicinity of vehicle 100, but the ability of an associated warning system to prevent a potentially dangerous situation may be limited by 1) the field of view of sensor system 225 and 2) the operating state of vehicle 100. That is, a creature that is in the vicinity of vehicle 100 but is not currently within the field of view of sensor system 225 (e.g., because it is small and close to vehicle 100 as depicted in FIGS. 1A and 1B ), or that comes into the vicinity of vehicle 100 while the vehicle is turned off, may not be detected by vehicle sensor system 225 or observed by the vehicle operator. Creature notification system 270 and identification module 346 operate to inform the vehicle operator of such creatures (e.g., smaller creatures and those in the vicinity of vehicle 100 while it is turned off) that may otherwise go unnoticed.

[0036] Thus, the identification module 346 generally includes instructions that function to control the processor 218 to receive data input from one or more sensors of the vehicle 100. In one embodiment, the input is observations of one or more objects in an environment proximate the vehicle 100 and / or other aspects of the surroundings. In one embodiment, as provided herein, the identification module 346 acquires sensor data 222 including at least camera images. In a further mechanism, the identification module 346 acquires sensor data 222 from additional sensors, such as radar sensors 228, LiDAR sensors 229, and other sensors, as may be suitable for identifying the vehicle and its location. Thus, in one embodiment, the identification module 346 controls each sensor to provide data input in the form of sensor data 222. Furthermore, in one embodiment, the identification module 346 controls the sensors to acquire sensor data 222 regarding an area encompassing 360 degrees around the vehicle 100 to provide a comprehensive assessment of the surrounding environment.

[0037] As described above, the bio-notification system 270 may operate in a sentry mode, in which the environmental sensor 227 is battery-powered and senses the environment even when the vehicle 100 is off. Accordingly, the identification module 346 generally includes instructions that function to control the processor 218 to receive data input from the battery-powered environmental sensor 227. That is, the environmental sensor 227 may receive power from the primary or auxiliary vehicle battery and may operate even when the ignition is off.

[0038] In one approach, the identification module 346 implements and / or otherwise uses a machine learning algorithm. As described herein, machine learning algorithms include, but are not limited to, deep neural networks (DNNs), including transformer networks, convolutional neural networks, recurrent neural networks (RNNs), support vector machines (SVMs), clustering algorithms, hidden Markov models, and the like. It should be understood that different forms of machine learning algorithms may have separate applications, such as agent modeling, machine perception, and the like. In one configuration, a machine learning algorithm is incorporated within the identification module 346 to identify and classify living organisms based on the sensor data 222. In one specific example, the machine learning model may be a neural network that includes any number of: 1) input nodes that receive the sensor data 222; 2) hidden nodes that may be arranged in layers connected to the input nodes and / or other hidden nodes and include computational instructions for computing outputs; and 3) output nodes connected to the hidden nodes that generate outputs indicative of the presence, movement, and classification of living organisms in the vicinity of the vehicle 100.

[0039] Of course, in further embodiments, the identification module 346 may employ different machine learning algorithms or implement different approaches to identifying and classifying organisms. Regardless of the particular approach implemented by the identification module 346, the identification module 346 provides an output of the identification, classification, and / or estimated location of organisms in the vicinity of the vehicle 100. In either case, the output of the identification module 346 is transmitted to the notification module 348 for generating and presenting a notification of the detected organism. In this manner, the organism notification system 270 may alert the vehicle operator of an undetected organism in the vicinity of the vehicle 100. Additional details regarding the identification, classification, and location estimation of various organisms in the vicinity of the vehicle 100 are provided below in connection with FIG. 4.

[0040] Furthermore, it should be understood that machine learning algorithms are generally trained to perform a prescribed task. Thus, unless otherwise stated, training of a machine learning algorithm is understood to be separate from the normal use of the machine learning algorithm. That is, the bio-notification system 270 or another system generally trains the machine learning algorithm according to a particular training technique, which may include supervised training, self-supervised training, reinforcement learning, etc. In contrast to training / learning a machine learning algorithm, the bio-notification system 270 implements the machine learning algorithm to perform inference. Thus, the normal use of a machine learning algorithm is described as inference.

[0041] It should be understood that the identification module 346 may be combined with the classification model 250 to form a computational model, such as a neural network model. In either case, in one embodiment, when implemented with a neural network model or another model, the identification module 346 implements functional aspects of the classification model 250, while further aspects, such as learned weights, may be stored in the data store 220. Thus, the classification model 250 is generally incorporated into the identification module 346 as a cohesive functional structure.

[0042] The organism notification system 270 further includes a notification module 348, which, in one embodiment, includes instructions that cause the processor 218 to present a notification identifying 1) the organism and 2) the last time the organism was within the field of view of the sensor. As described above, the organism notification system 270 provides notifications that may not otherwise be generated. For example, existing systems may not provide notifications regarding organisms that have left the field of view of the sensor and may therefore only provide a real-time indication of detected organisms. However, such systems do not completely prevent dangerous situations, as an organism that was previously detected but is now outside the field of view of the sensor system 225 may still be in a danger zone around the vehicle 100. Therefore, the notification module 348 receives the output of the identification module 346 and generates a notification based thereon. The notification may indicate the last time the organism was detected, so that the vehicle operator may be informed of the organism's presence and may take any necessary measures to avoid the dangerous situation. The notification may take various forms, including a visual notification presented on the human-machine interface (HMI) of the vehicle 100. In another example, the notification may be sent to the user's personal device, such as a smartphone. In either case, the generated notification may be sent to the intended recipient via communication system 280, which, as described above, may be a Wi-Fi, LGE, or other wireless-based network device. Additional details regarding the generation and presentation of notifications are provided below in connection with FIG. 4.

[0043] Additional aspects of generating a notification about a previously detected organism in the vicinity of vehicle 100 are described in connection with Figure 4. Figure 4 illustrates a flowchart of a method 400 associated with presenting an indication of a detected organism and an indication of the last time the organism was detected by one of the vehicle's environmental sensors 227. Method 400 is described in terms of organism notification system 270 of Figures 2 and 3. While method 400 is described in conjunction with organism notification system 270, it should be understood that method 400 is not limited to implementation within organism notification system 270, but rather is an example of a system in which method 400 may be implemented.

[0044] As mentioned above, the method 400 described herein may be performed, at least in part, while the vehicle 100 is stationary. For example, a stationary vehicle 100 may be an unused vehicle, an abandoned vehicle, and / or a vehicle that is in an off state. In each case, the vehicle operator may or may not be present at the vehicle 100. For example, the vehicle 100 may be located in a private garage overnight, or the driver may leave the vehicle 100 to retrieve something left behind at their home. In any of these examples, a small child or animal may be in the vicinity of the vehicle 100 but may remain unnoticed by the driver and / or undetected by the environmental sensors 227 of the vehicle 100. In the case of a vehicle 100 that is in an off state, the environmental sensors 227 of the vehicle 100 may enter a sentry mode, in which the sensors continue to collect data about the surroundings of the vehicle 100, even though other vehicle systems are inactive and the vehicle 100 ignition is in the off position. Thus, in these examples, the bio notification system 270 continuously monitors for living creatures in the vicinity of the vehicle 100, even when the vehicle 100 is turned off. In one example, the environmental sensor 227 is connected to the vehicle battery or power source but operates to be active even when other systems are inactive. In another example, the environmental sensor 227 is powered by an independent power source. In one implementation, the environmental sensor 227 may be motion-activated; that is, detected movement may trigger the collection of sensor data 222. In either case, the bio notification system 270 provides comprehensive awareness of the vehicle 100's surroundings and notification of any living creatures detected therein, even when the vehicle is turned off.

[0045] At 410, the identification module 346 identifies organisms in the vicinity of the stationary vehicle 100 from the sensor data 222 collected from the sensors of the stationary vehicle 100. Generally, identifying organisms in the vicinity involves analyzing the output of the environmental sensors 227 to identify objects in the captured output (e.g., camera images). Accordingly, the identification module 346 controls the sensor system 225 to acquire the sensor data 222. In one embodiment, the identification module 346 controls the radar sensor 228 and the camera 231 of the vehicle 100 to observe the surrounding environment. Alternatively, or in addition, the identification module 346 controls the camera 231 and the LiDAR sensor 229 or another set of sensors to acquire the sensor data 222.

[0046] In one example, identifying a living organism in the vicinity of stationary vehicle 100 includes distinguishing the living organism from other objects captured in the output of environmental sensor 227. For example, other objects, such as fallen leaves, debris, and precipitation, may also be detected by the image processor of identification module 346. If such other elements are identified as potential living organisms, a false positive notification may be generated. Thus, identification module 346 may distinguish between non-living and living objects. In one example, such distinction may be supported by machine learning operations. That is, living organisms may have certain characteristics (such as size and / or movement patterns) that are distinct from the characteristics of non-living objects. In this example, relying on classification model 250, identification module 346 may analyze the characteristics (e.g., size and / or movement patterns, among others) of a detected object to identify it as a living organism or another non-living object. In one example, identification module 346 may rely on a machine learning algorithm trained on a dataset. That is, during training, identification module 346 may be presented with numerous images of objects (e.g., people, animals, dust, weather) and metadata characterizing the objects. The identification module 346 may be trained to identify patterns or features in a training set of object images that are characteristic of a particular object type. Thus, during object identification, the identification module 346 may be trained on this data set to identify features of objects in the images and, based on the identified features, characterize the objects in the images as animate or non-animate.

[0047] In one example, identifying creatures in the vicinity of a stationary vehicle 100 includes distinguishing between transient creatures versus creatures that are loitering / hanging out in the vicinity of the vehicle 100. For example, as depicted in Figure 5, the environmental sensors 227 may detect creatures that are loitering around the vehicle 100 and creatures that are simply passing by the vehicle 100, such as a user of an electric scooter. While it may be desirable to generate a notification of a loitering creature because a loitering creature is more likely to still be in the vicinity of the vehicle, it may be unnecessary to generate a safety alert for a transient user because a transient user is unlikely to be in the vicinity of the vehicle 100 for an extended period of time.

[0048] Transient creatures may exhibit different characteristics / features than wandering creatures. For example, transient creatures may move farther away from vehicle 100, may have a single movement pattern, may move at a faster average speed compared to wandering creatures, and may be captured in fewer camera images than wandering creatures. In comparison, wandering creatures may generally be closer to vehicle 100, may have zigzag, random, or back-and-forth movement patterns, may be stationary and / or move slower than transient creatures, and may be detected in more frames of sensor output than transient creatures. Thus, identification module 346 may include an image processor that analyzes the sensor output (e.g., camera images) and / or series of sensor outputs to identify patterns / features characteristic of transient creatures and patterns / features characteristic of hanging or wandering creatures. Like distinguishing between animate and non-animate objects, distinguishing between transient and roaming organisms may be based on machine learning algorithms, whether supervised or unsupervised, that analyze output frames and infer the type of organism (e.g., transient or roaming).

[0049] In this example, notification generation may be triggered by the identification of a living (and therefore non-transient) entity (as opposed to a non-living entity such as dust, debris, or other non-living object) loitering in the vicinity of vehicle 100. In comparison, non-living and transient living entities may not trigger subsequent operations of biological notification system 270 (e.g., classification, tracking, location estimation, and notification generation).

[0050] At 420, the identification module 346 characterizes the living thing. That is, the identification module 346 includes instructions that, when executed by the processor 218, cause the processor 218 to classify the living thing based on the detected characteristics of the living thing. As described below, the classification of the living thing may affect the generation and presentation of a notification. For example, as described above, it may be assumed that adults are more aware of potential hazards around them and around an unoccupied vehicle 100. Furthermore, due to their larger size, adults are more likely to be perceived by the driver and environmental sensors 227. In comparison, an animal or a child 1) may not be aware that the vehicle 100 is on and the driver is reversing, and / or 2) may not be detected by the driver and environmental sensors 227 when the vehicle 100 shifts into reverse gear. Thus, in these examples, the identification module 346 specifically classifies the living thing as at least one of a child or an animal and generates a notification accordingly. For example, a notification may be generated when a child or an animal is detected in the vicinity of the vehicle 100 within a threshold period of time. In contrast, when an adult is detected in the vicinity of the vehicle 100, no notification may be generated.

[0051] Like the identification of a living thing, the classification of the living thing may be based on an analysis of the output of the environmental sensor 227. For example, an adult may have different physical characteristics compared to a child, such as different heights, limb lengths, etc. Similarly, an animal may have different physical characteristics compared to a human. Thus, the identification module 346 may include an image processor that classifies the living thing as an adult, a child, or an animal.

[0052] In one example, the identification module 346 may be presented with a large number of images of living organisms (e.g., adults, children, various types of animals) and metadata characterizing the living organisms. The identification module 346 may be trained to identify patterns or features in the training set images of the organisms that are characteristic of a particular class of organisms. The identification module 346 may then be trained on this dataset to identify features of the organisms in the sensor output and classify the organisms based on the identified features. Similarly, during training, the identification module 346 may rely on a machine learning algorithm trained on the dataset to classify the organisms. That is, during training, the identification module 346 may be presented with a large number of images of humans (e.g., adults of various heights, ages, and builds, children of various ages, heights, and builds) and metadata characterizing the humans. The identification module 346 may be trained to identify human patterns or features that are characteristic of a particular class of humans. The identification module 346 may then be trained on this dataset to identify features of the humans in the sensor output and classify the humans based on the identified features.

[0053] In addition to classifying creatures based on type, the identification module 346 may also determine the pose of the classified creature. That is, the pose of the creature may be used to infer whether the creature is likely to remain in the vicinity of a vehicle. For example, a child sitting on the ground and curled up may indicate that the child is playing with a toy and is likely to remain in place for a period of time. In comparison, a child in a running pose may indicate that the child is near the vehicle but is unlikely to remain there for very long. Thus, the identification module 346 may include instructions to cause the processor 218 to analyze the images and detect pose and pose changes over time for the classified creature. Pose detection may be performed for both human and non-human creatures.

[0054] At 430, the identification module 346 tracks the movement of the living thing. That is, the identification module 346 may include instructions that cause the processor 218 to track the movement of the living thing in the vicinity of the stationary vehicle 100. In some examples, the movement of the living thing may indicate 1) whether the living thing is still in the vicinity of the vehicle 100 even if it is outside the field of view of the environmental sensor 227, and 2) the location of the living thing outside the field of view of the environmental sensor 227.

[0055] For example, the head of the child 106 may be periodically detected at the lower end of the field of view of the backup camera 102 and then disappear from the field of view. Thus, the tracking movement of the child's head in and out of the lower portion of the field of view of the backup camera 102 may indicate that the child 106 remains at the rear of the vehicle 100, even if not detected by the backup camera 102.

[0056] 6, the identification module 346 may track the movement of an animal 108 beyond the field of view of the passenger side-view camera 110 and toward the vehicle 100. However, it may be that the animal 108 continues along a path outside the field of view of the side-view camera 110 (i.e., underneath the vehicle 100). In this example, the identification module 346 may track the movement of the organism through various frames of collected output to estimate a likely trajectory for the organism after it leaves the field of view of the sensor. For example, the identification module 346 may compare the relative positions of the organism in successive frames of output and extrapolate from that relative position the organism may be found after it leaves the field of view of the sensor.

[0057] In one example, the movement of an organism may be tracked across various sensors of vehicle 100. For example, following the detection of an animal 108 moving toward vehicle 100, driver's side view camera 110 may detect the animal moving laterally away from vehicle 100, as depicted in FIG. 6 . Relying solely on passenger side view camera 110, identification module 346 may infer that animal 108 is underneath vehicle 100. However, by aggregating the outputs of multiple vehicle sensors, identification module 346 may infer that animal 108 has left the vicinity of vehicle 100.

[0058] At 440, the identification module 346 determines the last time the living thing was within the field of view of the sensor system 225 of the vehicle 100. That is, the identification module 346 may include instructions that cause the processor 218 to determine from the sensor data 222 the last time the living thing was within the field of view of the sensor. As described above, it may be that a living thing that is outside the field of view of the sensor system 225 of the vehicle 100 when the operator enters the vehicle 100 was within the field of view of the sensor at some point. Accordingly, the identification module 346 analyzes metadata associated with the sensor data 222 to determine the last time the living thing was recognized in the vicinity of the vehicle 100. This information may be included in the notification or the basis for generating the notification. For example, a notification may be generated when the operator activates the vehicle 100 (e.g., by turning on the vehicle 100, unlocking a vehicle door, opening a vehicle door, etc.). During such operation, a notification may be presented to the driver via the HMI of vehicle 100 or the driver's personal device, the notification indicating the last time a target class creature (e.g., a child or animal) was recognized in the vicinity of vehicle 100. The operator may then take any appropriate corrective action. As another example, a notification may be presented if the last time a creature was recognized is within a certain time range (e.g., 10 minutes, 20 minutes, 30 minutes, or 60 minutes, as examples) from when the driver operates vehicle 100. In either case, the driver may carefully scan the vicinity of vehicle 100 to identify whether the creature is still in the vicinity and therefore should be considered before operating vehicle 100.

[0059] At 450, the notification module 348 presents a notification identifying the living thing and the last time the living thing was recognized in the vicinity of the vehicle 100. That is, the notification module 348 includes instructions that cause the processor 218 to present a notification identifying the living thing and the last time the living thing was within the field of view of the sensor. The notification module 348 may generate a variety of different types of notifications. For example, the notification may be an audio notification, a visual notification, or a combination thereof.

[0060] Generally, the notification indicates the last time the classified creature was recognized. For example, as described above, the sensor data 222 may include metadata indicating a timestamp of the recorded sensor data 222. This metadata may be presented in a notification informing a user, such as a vehicle operator, of the creature's last appearance. The notification may include other information as well. For example, the notification may indicate the location where the creature was detected. The location may be related to the sensor / camera that detected the creature. For example, the notification may indicate, "A child was last recognized behind the vehicle at 12:30 PM." The notification may include other information, such as the detected classification (e.g., child, animal). In another example, the notification may include an image of the creature or other sensor output. Whatever is included in the notification, the notification may be presented on a variety of devices. In one example, the notification may be presented on a human-machine interface (HMI), such as a vehicle's infotainment center. In another example, the notification may be presented on a user device, such as the operator's smartphone.

[0061] In one example, generation of a notification may be triggered by any number of circumstances. For example, a notification may be in response to an operator operating a stationary vehicle 100. That is, as described above, monitoring of the vehicle 100's surroundings may be continuous even when the vehicle is left unattended and / or turned off. However, a notification may be presented when the operator operates the vehicle 100. Examples of operator actions include entering the vehicle 100 (i.e., unlocking and / or opening a vehicle door), turning the vehicle 100 on, and shifting a gear in the vehicle 100. Accordingly, the notification module 348 may communicate the action with other systems in the vehicle 100 that detect the user's operation of the vehicle 100. In another example, a notification may be triggered based on detected movement of the operator toward the vehicle 100. In yet another example, a notification may be triggered based on detection of an approaching operator based on short-range communication between the vehicle 100 and a user device.

[0062] In one example, the generation and presentation of a notification may be based on the classification of the creature. For example, it may be assumed that an adult is more easily detectable by the sensor system 225, is aware of the potential danger associated with loitering around the stationary vehicle 100, and may take appropriate measures. Thus, a notification may not be generated when the detected creature is classified as an adult. In comparison, the notification module 348 may generate a notification when the detected creature is an animal / child, which may not understand the potential risk and / or may not know how to respond to impending vehicle movement.

[0063] In another example, notification generation and presentation may be based on a timestamp for the last detection of a living organism and the time the operator operates the vehicle 100. For example, the notification module 348 may generate a notification for a living organism detected within a certain threshold range (e.g., 20 minutes, 10 minutes, 5 minutes) from when the operator operates the vehicle 100. It is not necessarily the case that a last recognized detected living organism outside this time range will not trigger notification generation. Examples of generated notifications are presented below in connection with Figures 7A and 7B.

[0064] In addition to generating a notification, in some examples, the bio-notification system 270 may take additional corrective measures, such as disabling portions of the vehicle systems, such as the throttle system 238 and the transmission system 239, thus preventing movement of the vehicle 100 until the situation is resolved (e.g., the operator inspects the vehicle's surroundings and confirms this through the HMI).

[0065] As described above, method 400, or at least portions of method 400, may be performed while the stationary vehicle is in an off state. That is, identification module 346 identifies a living creature in the vicinity of stationary vehicle 100, tracks the creature's movement, and determines the last time the living creature was within the field of view of the sensors while stationary vehicle 100 was in an off state. Thus, identification module 346 may operate in sentry mode, where environmental sensors 227 are active and continuously monitoring for potential out-of-frame prowling creatures, regardless of the state of other systems on vehicle 100. Thus, method 400 improves user safety by generating notifications about living creatures that may be in the vicinity of the vehicle, even when they are not within the field of view of the vehicle's sensors and while the vehicle is in an off state.

[0066] FIG. 5 illustrates one embodiment of a biological notification system 270 that distinguishes between different biological organisms. As described above, different biological organisms have different physical characteristics / features. For example, as depicted in FIG. 5, an adult 114 may be taller than a child 106 and have different physical characteristics than the child 106. Similarly, an animal 108 may have different physical characteristics that can be detected by the identification module 346 and used to classify the biological organism. As an example, the identification module 346 may analyze the physical characteristics of a biological organism, e.g., as a wireframe, to identify those characteristics that define the organism as a child 106-1 or 106-2, an animal 108, or an adult 114. In a similar manner, the identification module 346 may be able to distinguish between non-living and living organisms, as described above, based on, for example, the size, location, and movement patterns of the detected object. Accordingly, the identification module 346 includes instructions that cause the processor to distinguish between living organisms and other elements detected by the sensor.

[0067] As described above, a wandering creature may trigger the generation of a notification, while a transient creature may not. Accordingly, the identification module 346 includes instructions that cause the processor to infer from the sensor data 222 that a creature is a wandering creature when the creature is outside the field of view of the sensors. As described above, a wandering creature may have different physical characteristics / behaviors and different movement characteristics / behaviors than a transient creature. For example, a child 106-2 riding a scooter may have a straighter trajectory, move at a faster speed, and be in fewer frames of sensor output than a wandering child 106-1. The identification module 346, which may rely on machine learning, may identify the characteristics indicative of either a wandering creature or a transient creature and classify the creature as either a wandering creature or a transient creature. In one example, a creature's state may be considered wandering or transient even when the creature is no longer in the frame of the vehicle 100's sensors. For example, wandering child 106-1 may be detected in multiple frames of side-view camera 110, disappear off the bottom of the field of view, and eventually reappear several frames later. Based on this information, identification module 346 may infer that wandering child 106-1 is sitting near the vehicle despite being outside the field of view of side-view camera 110. In either case, notification module 348 may cause the processor to provide a notification based on the creature wandering near stationary vehicle 100.

[0068] FIG. 6 illustrates one embodiment of a creature notification system 270 that estimates the location of a creature. As described above, in some examples, a creature, such as an animal 108, may leave the field of view of a camera but may still be near the vehicle 100 and, therefore, may be in a potentially dangerous position. Accordingly, the creature notification system 270 may infer or estimate the location of a creature that is outside the field of view of a vehicle sensor but still near the vehicle 100. That is, the identification module 346 may cause the processor 218 to estimate the location of the creature when it is not within the field of view of the sensor based on the tracking movement of the creature. For example, the identification module 346 may identify a detected path 650 of the animal 108 while it is in the field of view 112-1 of the passenger-side sideview camera 110-1. As depicted in FIG. 6, the detected path 650 of the animal 108 may be heading toward the vehicle 100. However, the animal 108 may be moving outside the field of view 112-1 of the passenger-side sideview camera 110-1, and in the example depicted in FIG. 6 , underneath the vehicle 100. In this example, the identification module 346 may extrapolate an estimated path 652 of the animal 108 from the sensor data 222. In one example, the estimated location may be based on a combination of sensor data 222 from multiple vehicle sensors, or lack thereof. For example, assuming the animal 108 is not detected within the field of view 112-2 of the driver-side sideview camera 110-2, the animal 108 may be inferred to be located underneath the vehicle 100, and an appropriate notification may be generated.

[0069] Thus, the identification module 346 identifies creature behaviors that indicate the creature is still in the vicinity of the vehicle 100, even when outside the field of view of the vehicle's 100 environmental sensors 227. This may be done by relying on machine learning analysis of the number of frames captured of the creature, the creature's movement patterns, pose, location, and the like. In one example, the notification module 348 may present a notification that identifies the creature's estimated location. In one example, the generation of the notification may be based on the creature's estimated location. For example, if the creature leaves the vehicle 100 from underneath the vehicle 100, continuing on its path as depicted by the driver's side view camera 110-2, no notification may be generated.

[0070] 7A and 7B illustrate example notifications generated by the creature notification system 270. As described above, notifications can be presented on a variety of devices, including a user device 754, such as a smartphone as depicted in FIG. 7A, or an HMI 756, such as a vehicle infotainment system, as depicted in FIG. 7B. The content of the notification can also be varied. For example, as depicted in FIG. 7A, the notification can include a text display of the creature's last location and a timestamp of when the creature was last seen. In one example, a log of notifications can be presented. In the example depicted in FIG. 7A, a log of multiple notifications spanning multiple days is presented. This log can be cleared manually or automatically. In another example, a notification can be generated and presented when a notification occurs within a predetermined window, such as a predetermined period of time before the operator activates the vehicle 100.

[0071] As depicted in the example of FIG. 7B, the notification may include sensor data output indicating the last location of the detected organism. For example, the notification may include a graphic of the vehicle 100 and an indication of the general location where the organism was detected (e.g., the zone around the vehicle). In another example, the notification may include output (e.g., an image) of the detected organism. Although particular reference is made to particular forms and types of notifications, other types of notifications or other content may be presented in the notifications.

[0072] 8 illustrates one embodiment of an operation for generating a notification regarding an identified organism in the vicinity of a stationary vehicle 100. As described above, monitoring of organisms in the vicinity of the vehicle 100 may occur continuously, for example, via the organism notification system 270 and environmental sensors 227, which are battery-powered and active even when other systems of the vehicle 100 may not be active. In comparison, generation of a notification may be triggered when the vehicle 100 is active, for example, when the ignition is turned on. In this example, when the ignition is turned on, the vehicle 100 may communicate the object's location and associated metadata to a remote device 858 that sends, and in some cases generates, the notification.

[0073] For example, when the vehicle is off, the identification module 346 may receive camera images and perform motion detection to identify living beings in the vicinity of the vehicle 100. As described above, this may include 1) distinguishing between living and non-living objects in the vicinity of the vehicle 100 and 2) distinguishing between transient and prowling living beings. The identification module 346 may also classify the living being (e.g., as either a human or an animal) and perform pose detection, which may yield an inferred / estimated position / movement of the living being. The identification module 346 also determines the living being's distance from the vehicle 100 and, in some cases, estimates the living being's location. When the vehicle 100 is turned on and connected to a remote device 858 via the communication system 280, the living being's location, and in some cases, a generated notification, is passed to the remote device 858. The remote device 858 then wirelessly transmits the generated notification to a device, such as the user device 754. In another example, the remote device 858 generates a notification and wirelessly transmits it to the user device 754 or another device.

[0074] 2 is now described in greater detail as an exemplary environment in which the systems and methods disclosed herein may operate. In some examples, vehicle 100 is configured to selectively switch between an autonomous mode, one or more semi-autonomous modes, and / or a manual mode. "Manual mode" means that all or most of the control and / or steering of the vehicle is performed according to inputs received via a manual human-machine interface (HMI) (e.g., steering wheel, accelerator pedal, brake pedal, etc.) of vehicle 100 when operated by a user (e.g., a human driver). In one or more arrangements, vehicle 100 may be a manually controlled vehicle configured to operate exclusively in manual mode.

[0075] In one or more mechanisms, vehicle 100 implements a level of automation to operate autonomously or semi-autonomously. As used herein, automated control of vehicle 100 is defined along a spectrum according to the SAE J3016 standard. The SAE J3016 standard defines six levels of automation, Levels 0-5. Generally, as described herein, semi-autonomous mode refers to Levels 0-2, while autonomous mode refers to Levels 3-5. Thus, autonomous mode generally involves controlling and / or steering vehicle 100 along a travel route via a computing system with minimal or no input from a human driver. In contrast, semi-autonomous mode, which may also be referred to as an advanced driver assistance system (ADAS), provides some control and / or steering of the vehicle along a travel route via a computing system to a vehicle operator (i.e., a driver) who provides at least some of the control and / or steering of vehicle 100.

[0076] 2 , vehicle 100 includes one or more processors 218. In one or more arrangements, processor 218 may be the primary / centralized processor of vehicle 100 or may represent multiple distributed processing units. For example, processor 218 may be an electronic control unit (ECU). Alternatively, or in addition, processor may include a central processing unit (CPU), a graphics processing unit (GPU), an ASIC, a microcontroller, a system-on-chip (SoC), and / or other electronic processing unit that supports operation of vehicle 100.

[0077] Vehicle 100 may include one or more data stores 220 that store one or more types of data. Data store 220 may comprise volatile and / or non-volatile memory. Examples of memory that may form data store 220 include RAM (random access memory), flash memory, ROM (read-only memory), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), registers, magnetic disks, optical disks, hard drives, solid-state drives (SSDs), and / or other non-transitory electronic storage media. In one configuration, data store 220 is a component of processor 218. Generally, data store 220 is operably connected to processor 218 for use by processor 218. As used throughout this specification, the term "operably connected" includes direct or indirect connections and may include connections without direct physical contact.

[0078] In one or more arrangements, one or more data stores 220 include various data elements that support vehicle 100 functionality, such as semi-autonomous and / or autonomous functions. Accordingly, data store 220 may store map data 221 and / or sensor data 222. In at least one approach, map data 221 includes maps of one or more geographic areas. In some examples, map data 221 may include information about roads (e.g., lane and / or road maps), traffic control devices, road markings, structures, features, and / or landmarks in one or more geographic areas. In at least one approach, map data 221 may be characterized as a high-definition (HD) map that provides information to autonomous and / or semi-autonomous functions.

[0079] In one or more mechanisms, map data 221 may include one or more terrain maps 223. Terrain maps 223 may include information about the ground, topography, roads, terrain, and / or other features of one or more geographic areas. Terrain maps 223 may include elevation data for one or more geographic areas. In one or more mechanisms, map data 221 includes one or more stationary obstacle maps 224. Stationary obstacle maps 224 may include information about one or more stationary obstacles located within one or more geographic areas. A "stationary obstacle" is a physical object whose position and general attributes do not substantially change over time. Examples of stationary obstacles include trees, buildings, curbs, fences, etc.

[0080] Sensor data 222 is data provided by one or more sensors of sensor system 225. Thus, sensor data 222 may include observations about the environment surrounding vehicle 100 and / or information about vehicle 100 itself. In some examples, one or more data stores 220 located onboard vehicle 100 store at least a portion of map data 221 and / or sensor data 222. Alternatively or additionally, at least a portion of map data 221 and / or sensor data 222 may be located in one or more data stores 220 located remotely from vehicle 100.

[0081] As described above, vehicle 100 may include sensor system 225. Sensor system 225 may include one or more sensors. As used herein, a "sensor" refers to an electronic and / or mechanical device that generates an output (e.g., an electrical signal) in response to a physical phenomenon, such as electromagnetic radiation (EMR), sound, etc. Sensor system 225 and / or one or more sensors may be operatively connected to processor 218, data store 220, and / or another element of vehicle 100.

[0082] Various examples of different types of sensors are described herein. However, it will be understood that embodiments are not limited to the particular sensors described. In various configurations, the sensor system 225 includes one or more vehicle sensors 226 and / or one or more environmental sensors. The vehicle sensors 226 function to sense information about the vehicle 100 itself. In one or more arrangements, the vehicle sensors 226 include one or more accelerometers, one or more gyroscopes, an inertial measurement unit (IMU), a dead reckoning system, a global navigation satellite system (GNSS), a global positioning system (GPS), and / or other sensors that monitor aspects related to the vehicle 100.

[0083] As mentioned, sensor system 225 may include one or more environmental sensors 227 that sense the vehicle 100's surrounding environment (e.g., exterior) and / or, in at least one arrangement, the environment of the vehicle's passenger compartment. For example, one or more environmental sensors 227 detect objects in the vehicle 100's surrounding environment. The obstacles may be stationary and / or dynamic objects. Various examples of sensors for sensor system 225 are described herein. Exemplary sensors may be part of one or more environmental sensors 227 and / or one or more vehicle sensors 226. However, it will be understood that embodiments are not limited to the particular sensors described. By way of example, in one or more arrangements, sensor system 225 may include one or more radar sensors 228, one or more LIDAR sensors 229, one or more sonar sensors 230 (e.g., ultrasonic sensors), and / or one or more cameras 231 (e.g., monocular, stereo, RGB, infrared, etc.).

[0084] Continuing with the description of elements of FIG. 2 , vehicle 100 may include input system 232. Input system 232 generally encompasses one or more devices that enable a machine to obtain information from an external source, such as an operator. Input system 232 may receive input from vehicle passengers (e.g., the driver / operator and / or passengers). Additionally, in at least one configuration, vehicle 100 includes output system 233. Output system 233 includes, for example, one or more devices that enable information / data to be provided to an external target (e.g., a person, a vehicle passenger, another vehicle, another electronic device, etc.).

[0085] Further, in various configurations, vehicle 100 includes one or more vehicle systems 234. Various examples of one or more vehicle systems 234 are shown in FIG. 2 . However, vehicle 100 may include different configurations of vehicle systems. While certain vehicle systems are defined separately, it should be understood that each or any of the systems or portions thereof may be otherwise combined or separated via hardware and / or software within vehicle 100. As shown, vehicle 100 includes a propulsion system 235, a braking system 236, a steering system 237, a throttle system 238, a transmission system 239, a signaling system 240, and a navigation system 241.

[0086] Navigation system 241 may include one or more devices, applications, and / or combinations thereof for determining the geographic location of vehicle 100 and / or determining travel routes for vehicle 100. Navigation system 241 may include, for example, one or more mapping applications that determine travel routes for vehicle 100 according to map data 221. Navigation system 241 may include, or at least provide a connection to, a global positioning system, a local positioning system, or a geolocation system.

[0087] In one or more configurations, vehicle systems 234 function in cooperation with other components of vehicle 100. For example, processor 218, bio-notification system 270, and / or autonomous driving module 243 may be operatively connected to communicate with various vehicle systems 234 and / or their individual components. For example, processor 218 and / or autonomous driving module 243 may be in communication to send and / or receive information from various vehicle systems 234 to control navigation and / or steering of vehicle 100. Processor 218, bio-notification system 270, and / or autonomous driving module 243 may control some or all of these vehicle systems 234.

[0088] For example, when operating in autonomous mode, processor 218 and / or self-driving module 243 controls the direction and speed of vehicle 100. Processor 218 and / or self-driving module 243 accelerates vehicle 100 (e.g., by increasing the supply of energy / fuel provided to the motor), slows vehicle 100 (e.g., by applying the brakes), and / or changes direction of vehicle 100 (e.g., by steering the two front wheels). As used herein, "cause" or "causing" means to make, force, compel, instruct, command, order, and / or enable an event or action to occur, either directly or indirectly.

[0089] As shown, in at least one configuration, vehicle 100 includes one or more actuators 242. Actuators 242 are elements operable to move and / or control mechanisms, such as vehicle systems 234 or one or more of its components, in response to, for example, electronic signals or other inputs from processor 218 and / or autonomous driving module 243. One or more actuators 242 may include motors, pneumatic actuators, hydraulic pistons, relays, solenoids, piezoelectric actuators, and / or another form of actuator that generates the desired control.

[0090] As mentioned above, vehicle 100 may include one or more modules, at least some of which are described herein. In at least one arrangement, the modules are implemented as non-transitory computer-readable instructions that, when executed by processor 218, implement one or more of the various functions described herein. In various arrangements, one or more of the modules are components of processor 218, or one or more of the modules execute on and / or are distributed among other processing systems to which processor 218 is operatively connected. Alternatively, or in addition, one or more of the modules are implemented at least partially in hardware. For example, one or more of the modules may comprise a combination of logic gates (e.g., metal-oxide-semiconductor field-effect transistors (MOSFETs)) arranged to achieve the described functions, an ASIC, a programmable logic array (PLA), a field-programmable gate array (FPGA), and / or another electronic hardware-based implementation that implements the described functions. Furthermore, in one or more arrangements, one or more of the modules may be distributed among multiple modules described herein. In one or more arrangements, two or more of the modules described herein may be combined into a single module.

[0091] Additionally, vehicle 100 may include one or more autonomous driving modules 243. In at least one approach, autonomous driving module 243 receives data from sensor system 225 and / or other systems associated with vehicle 100. In one or more mechanisms, autonomous driving module 243 uses the data to perceive the vehicle's surroundings. Autonomous driving module 243 determines the position of vehicle 100 in the surroundings and maps aspects of the surroundings. For example, autonomous driving module 243 determines the location of obstacles or other environmental features, including traffic signs, trees, shrubs, nearby vehicles, pedestrians, etc.

[0092] Autonomous driving module 243 may be configured, independently or in combination with bio-notification system 270, to determine a travel path, a current autonomous driving maneuver for vehicle 100, a future autonomous driving maneuver, and / or modifications to the current autonomous driving maneuver based on data obtained by sensor system 225 and / or another source. Generally, autonomous driving module 243 functions to implement various levels of automation, including, for example, advanced driver assistance (ADAS) functions, semi-autonomous functions, and fully autonomous functions, as described above.

[0093] Detailed embodiments are disclosed herein. However, it should be understood that the disclosed embodiments are intended merely as examples. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to variously employ the aspects of the present specification in substantially any suitable detailed configuration. Furthermore, the terms and phrases used herein are not intended to be limiting, but rather to provide an understandable description of possible implementations. While various embodiments are shown in FIGS. 1-8, the embodiments are not limited to the structures or applications shown.

[0094] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, comprising one or more executable instructions that implement the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved.

[0095] The above-described systems, components, and / or processes may be implemented in hardware or a combination of hardware and software, and may be implemented in a centralized fashion within one processing system, or in a distributed fashion with various elements spread across several interconnected processing systems. The systems, components, and / or processes may also be embodied in a computer-readable storage, such as a machine-readable computer program product or other data program storage device, tangibly embodying a program of instructions executable by a machine to perform the methods and processes described herein. These elements may also be embodied in an application product with features that enable implementation of the methods described herein and that, when loaded on a processing system, are capable of executing these methods.

[0096] Furthermore, the mechanisms described herein may take the form of a computer program product in which computer-readable program code is embodied, e.g., stored, in one or more computer-readable media. Any combination of one or more computer-readable media may be utilized. The phrase "computer-readable storage medium" refers to a non-transitory storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. A non-exhaustive list of computer-readable storage media may include the following: a portable computer diskette, a hard disk drive (HDD), a solid-state drive (SSD), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), an optical storage device, a magnetic storage device, or a combination thereof. In the context of this specification, a computer-readable storage medium is a tangible medium that stores a program for use by or in connection with an instruction execution system, apparatus, or device.

[0097] Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wired, fiber optic, cable, RF, or the like, or any suitable combination thereof. Computer program code for performing operations for aspects of the present mechanism may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, or the like, and traditional procedural programming languages ​​such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection to the external computer may be made (e.g., through the Internet using an Internet Service Provider).

[0098] The terms "a" and "an," as used herein, are defined as one or more than one. The term "plurality," as used herein, is defined as two or more than two. The term "another," as used herein, is defined as at least a second or more. The terms "including" and / or "having," as used herein, are defined as comprising (i.e., open language). The phrase "and at least one of," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items. As an example, the phrase "at least one of A, B, and C" includes A only, B only, C only, or any combination thereof (e.g., AB, AC, BC, or ABC).

[0099] Aspects of the present specification may be embodied in other forms without departing from the spirit or essential attributes thereof, and reference should accordingly be made to the following claims, rather than the foregoing specification, as indicating the scope of the present specification.

Claims

1. 1. A system comprising: a processor; a memory storing machine-readable instructions; the machine-readable instructions, when executed by the processor, cause the processor to: Identifying living things in the vicinity of a stationary vehicle from sensor data collected from sensors of the stationary vehicle; tracking the movement of the living thing in the vicinity of the stationary vehicle; determining from the sensor data the last time the organism was within the field of view of the sensor; The system causes a notification to be presented that identifies the organism and the last time the organism was within the field of view of the sensor.

2. 2. The system of claim 1, wherein the machine-readable instructions that, when executed by the processor, cause the processor to present the notification comprise machine-readable instructions that, when executed by the processor, cause the processor to present the notification in response to an operator operating the stationary vehicle.

3. 2. The system of claim 1, wherein the machine-readable instructions, when executed by the processor, cause the processor to identify the living thing in the vicinity of the stationary vehicle, track the movement of the living thing, and determine the last time the living thing was within the field of view of the sensor, are executed while the stationary vehicle is off.

4. The machine-readable instructions, which when executed by the processor, cause the processor to identify the living organism in the vicinity of the stationary vehicle, when executed by the processor, cause the processor to: Distinguishing between living organisms and other elements detected by the sensor; The system of claim 1 , comprising machine-readable instructions for causing the organism to be characterized as at least one of a transient organism or a wandering organism.

5. the machine-readable instructions that, when executed by the processor, cause the processor to characterize the organism comprise machine-readable instructions that, when executed by the processor, cause the processor to infer from the sensor data that the organism is a prowling organism when outside the field of view of the sensor; 5. The system of claim 4, wherein the machine-readable instructions that, when executed by the processor, cause the processor to present the notification comprise machine-readable instructions that, when executed by the processor, cause the processor to present the notification based on the living thing loitering in the vicinity of the stationary vehicle.

6. the machine-readable instructions further comprising machine-readable instructions that, when executed by the processor, cause the processor to classify the organism based on detected characteristics of the organism; 2. The system of claim 1, wherein the machine-readable instructions that, when executed by the processor, cause the processor to present the notification comprise machine-readable instructions that, when executed by the processor, cause the processor to indicate a class of the organism.

7. 7. The system of claim 6, wherein the machine-readable instructions that, when executed by the processor, cause the processor to classify the creature comprise machine-readable instructions that, when executed by the processor, cause the processor to classify the creature as at least one of a child or an animal.

8. the machine-readable instructions further comprising machine-readable instructions that, when executed by the processor, cause the processor to estimate a location of the living thing when not within the field of view of the sensor based on tracking movement of the living thing; 2. The system of claim 1, wherein the machine-readable instructions that, when executed by the processor, cause the processor to present the notification comprise machine-readable instructions that, when executed by the processor, cause the processor to identify a probable location of the living organism.

9. 2. The system of claim 1, wherein the machine-readable instructions that, when executed by the processor, cause the processor to present the notification comprise machine-readable instructions that, when executed by the processor, cause the processor to present at least one of a text display or sensor data output indicating the organism's last known location.

10. A non-transitory machine-readable medium comprising instructions that, when executed by a processor, cause the processor to: Identifying living things in the vicinity of a stationary vehicle from sensor data collected from sensors of the stationary vehicle; tracking the movement of the living thing in the vicinity of the stationary vehicle; determining from the sensor data the last time the organism was within the field of view of the sensor; a non-transitory machine-readable medium for causing presentation of a notification identifying the organism and the last time the organism was within the field of view of the sensor;

11. 11. The non-transitory machine-readable medium of claim 10, wherein the instructions that, when executed by the processor, cause the processor to present the notification comprise instructions that, when executed by the processor, cause the processor to present the notification in response to an operator operating the stationary vehicle.

12. 11. The non-transitory machine-readable medium of claim 10, wherein the instructions that, when executed by the processor, cause the processor to identify the living thing in the vicinity of the stationary vehicle, track the movement of the living thing, and determine the last time the living thing was within the field of view of the sensor are executed while the stationary vehicle is off.

13. The instructions, which when executed by the processor, cause the processor to identify the living organism in the vicinity of the stationary vehicle, when executed by the processor, cause the processor to: Distinguishing between living organisms and other elements detected by the sensor; The non-transitory machine-readable medium of claim 10 comprising instructions for causing the organism to be characterized as at least one of a transient organism or a prowling organism.

14. The machine-readable medium further comprises instructions that, when executed by the processor, cause the processor to classify the organism based on detected characteristics of the organism; 11. The non-transitory machine-readable medium of claim 10, wherein the instructions that, when executed by the processor, cause the processor to present the notification comprise instructions that, when executed by the processor, cause the processor to indicate a class of the organism.

15. The machine-readable medium further comprises instructions that, when executed by the processor, cause the processor to estimate a location of the living thing when not within the field of view of the sensor based on tracking movement of the living thing; 11. The non-transitory machine-readable medium of claim 10, wherein the instructions that, when executed by the processor, cause the processor to present the notification comprise instructions that, when executed by the processor, cause the processor to identify a probable location of the living organism.

16. identifying living organisms in a vicinity of a stationary vehicle from sensor data collected from sensors of the stationary vehicle; tracking the movement of the living thing in the vicinity of the stationary vehicle; determining from the sensor data the last time the organism was within the field of view of the sensor; presenting a notification identifying the organism and the last time the organism was within the field of view of the sensor; A method comprising:

17. The method of claim 16 , wherein presenting the notification comprises presenting the notification in response to an operator actuating the stationary vehicle.

18. Identifying the organism in the vicinity of the stationary vehicle includes: distinguishing between living organisms and other elements detected by the sensor; characterizing the organism as at least one of a transient organism or a prowling organism; 17. The method of claim 16, comprising:

19. further comprising classifying the organism based on the detected characteristics of the organism; The method of claim 16 , wherein presenting the notification includes indicating a class of the organism.

20. estimating a location of the living thing when it is not within the field of view of the sensor based on the tracking movement of the living thing; The method of claim 16 , wherein presenting the notification includes identifying a probable location of the organism.

21. For the processor, identifying living organisms in a vicinity of a stationary vehicle from sensor data collected from sensors of the stationary vehicle; tracking the movement of the living thing in the vicinity of the stationary vehicle; determining from the sensor data the last time the organism was within the field of view of the sensor; presenting a notification identifying the organism and the last time the organism was within the field of view of the sensor; A computer program that performs the following:

Images