A camera system for monitoring occupants inside a vehicle and detecting their activities.

Abstract

To provide a monitoring system and method for exactly determining whether or not an activity of a crew member in a vehicle is in a safe state.SOLUTION: In a monitoring system, a vehicle 102 comprises: a memory 110 which stores image data; and a first camera (internal camera 116a) which fetches first image data including an activity of a crew member in the vehicle. An electronic control unit 108 acquires the first image data including the crew member from the first camera, predicts the activity of the crew member on the basis of the first image data, and determines that the activity of the crew member becomes different from a reference activity of the crew member. The electronic control unit records and fetches the first image data in periods before and after the determination that the activity of the crew member becomes different from the reference activity by using the first camera in the memory.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This disclosure relates to monitoring the activities of vehicle occupants, detecting, predicting, and documenting any changes in the activities.

Background Art

[0002] While driving a vehicle, the driver usually cannot see the rear compartment of the vehicle. In certain situations, even through the rearview mirror, the vehicle occupants may completely disappear from the driver's field of vision. For example, infants or toddlers under the age of 2 may be sitting in a rear-facing child seat in the rear row of the vehicle. To monitor the child, the driver can install a mirror in front of the child seat and use the vehicle's rearview mirror to see the child. However, this arrangement is limited by the positioning and size of the mirror and can only see a very small part of the vehicle's passenger compartment. In some cases, the driver may also want to monitor other young occupants in the vehicle, pets or other animals in the vehicle, and / or fragile components in the vehicle that may not be visible to the driver using the mirror system. Additionally, the driver may want to anticipate the activities of the occupants before certain activities occur or observe events that occur outside the vehicle that cause some action by the occupants. Similarly, with the development of fully autonomous vehicles, an individual may want to monitor the occupants when there is no passenger in the vehicle. Further, a driver with hearing impairment may not receive a warning when an infant or animal occupant is in distress or when a fragile component is no longer in a safe state or has fallen over.

[0003] Therefore, there is a need for systems and methods for safely anticipating, predicting, and / or reporting the activities of vehicle occupants inside and around the vehicle.

Summary of the Invention

[0004] In general, one aspect of the subject matter described herein may be embodied in a vehicle monitoring system. The monitoring system includes a first camera configured to capture first image data including one or more occupants in a vehicle. The monitoring system includes a memory configured to store the image data. The monitoring system includes an electronic control unit coupled to the first camera and the memory. The electronic control unit is configured to acquire first image data including one or more occupants from the first camera. The electronic control unit is configured to determine the activity of each of the one or more occupants based on the first image data. The activity is one or more of a movement and / or a facial expression. The electronic control unit is configured to determine that the activity of each of the one or more occupants will be different from the respective baseline activity of each of the one or more occupants. The electronic control unit is configured to record and acquire, in memory, first image data using the first camera for the period before and after the determination that the activity of each of the one or more occupants will be different from the baseline activity.

[0005] These embodiments, and other embodiments, may optionally include one or more of the following features: The first camera may include one or more interior cameras. The one or more interior cameras may be configured to capture different views of the interior of the vehicle.

[0006] The electronic control unit may include an output device. The output device may be configured to notify the user of the output device that an activity has occurred in response to a determination that the activity of at least one of the occupants differs from a baseline activity. The output device may include one or more displays. The electronic control unit may be configured to notify the user of the output device that an activity has occurred by displaying an icon corresponding to the activity on the output device. The electronic control unit may be configured to notify the user of the output device that an activity has occurred by displaying live video feed from one or more internal cameras on the output device. One or more displays may be vehicle displays, including one or more of a head-up display, a multi-information display, and an in-vehicle infotainment display, located in the front compartment of the vehicle. One or more displays may include a portable display located on a portable device. The portable display may be located away from the vehicle.

[0007] The surveillance system may further include a user interface configured to receive commands from a user to display live video feeds from one or more internal cameras on one or more displays.

[0008] The monitoring system may further include a second camera. The second camera may consist of one or more external cameras configured to capture a view of the surrounding environment outside the vehicle. The user interface may be configured to receive commands from the user to display live video feeds from one or more external cameras on one or more displays.

[0009] The monitoring system may further include a navigation unit. The navigation unit may be configured to acquire navigation map information, including the assignment of real-world objects and the vehicle's current location. The electronic control unit may be configured to notify the user of the route to the real-world object when the monitoring system determines that activity has occurred in at least one of the one or more occupants.

[0010] In another embodiment, the subject may be embodied in a vehicle monitoring system. The monitoring system comprises a first camera, configured to capture first image data including occupants inside the vehicle. The monitoring system comprises a memory configured to store the image data. The monitoring system comprises an electronic control unit coupled to the first camera and the memory. The electronic control unit is configured to acquire first image data including occupants from the first camera. The electronic control unit is configured to determine the occupants' baseline activity. The electronic control unit is configured to determine the occupants' activity based on the first image data. The electronic control unit is configured to determine that the occupants' activity will differ from the occupants' baseline activity. The electronic control unit is configured to record and capture in memory first image data using the first camera for the period before and after the determination that the occupants' activity will differ from the baseline activity.

[0011] The electronic control unit may be configured to observe occupant activity. The electronic control unit may be configured to determine movement and facial expression patterns associated with occupant activity. The electronic control unit may be configured to establish the most frequently occurring occupant activity as the occupant's baseline activity. The monitoring system may include a user interface configured to receive data input from the user. The electronic control unit may be configured to receive data input from the user to assign occupant movement and facial expression patterns to the occupant's baseline activity.

[0012] In another embodiment, the subject may be embodied in a method for notifying a user of an output device of the activities of an occupant inside a vehicle. This method includes the processor acquiring first image data of a vehicle compartment, including the occupant, from a first camera. This method includes the processor determining the activities of the occupant inside the vehicle compartment. This method includes the processor determining that the occupant's activities will differ from the occupant's baseline activities. This method includes the processor recording and storing the first image data from the first camera in memory. This method includes the processor transmitting the first image data to an output device, thereby notifying a user of the occupant's activities. Recording the first image data may include recording the first image data for the period before and after the step of determining that the occupant's activities will differ from the baseline activities. This method may further include the processor acquiring second image data of the surrounding environment outside the vehicle from a second camera. This method may further include the processor recording and storing the second image data from the second camera in memory. The method may further include the processor determining that the crew's activity will differ from the crew's baseline activity, and then transmitting second image data to the output device. Recording the second image data may include recording the second image data for the period before and after the determination that the crew's activity will differ from the baseline activity. [Brief explanation of the drawing]

[0013] Other systems, methods, features, and advantages of the present invention will be apparent to those skilled in the art by examining the following figures and detailed description. The components shown in the drawings are not necessarily to a specific scale and may be exaggerated to better illustrate the important features of the present invention. [Figure 1] A block diagram of an exemplary monitoring system according to one aspect of the present invention. [Figure 2]A flowchart illustrating an exemplary process for acquiring image data to predict and / or detect activity using the monitoring system of Figure 1, according to one aspect of the present invention. [Figure 3] A flowchart illustrating an exemplary process for detecting or predicting activity and taking action using the monitoring system of Figure 1, according to one aspect of the present invention. [Figure 4] A flowchart illustrating an exemplary process for generating, establishing, or determining crew baseline activities using the monitoring system of Figure 1, according to one aspect of the present invention. [Figure 5] An illustrative diagram of the positioning of one or more internal cameras in the monitoring system of Figure 1 inside a vehicle according to one aspect of the present invention. [Figure 6] An illustrative diagram of the positioning of one or more external cameras of the monitoring system shown in Figure 1 on a vehicle according to one aspect of the present invention. [Figure 7] An illustrative diagram of an output device for the monitoring system in a vehicle shown in Figure 1, according to one aspect of the present invention. [Modes for carrying out the invention]

[0014] This specification discloses systems, vehicles, and methods for monitoring occupants in a vehicle and detecting certain activities of occupants in a vehicle. Specific embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages: The monitoring system includes an in-vehicle camera to capture live video and transmit it to an electronic control unit (e.g., an in-vehicle infotainment (IVI) system). The electronic control unit runs an edge computing artificial intelligence (AI) algorithm trained with respect to the live video feed to identify occupant types, e.g., infants, toddlers, pets, or fragile components, and transmits live video and / or data such as notification icons to a multi-information display (MID), head-up display (HUD), IVI display, and / or other user device when certain occupant activity is detected.

[0015] The monitoring system uses AI, including machine algorithm learning, in conjunction with a model that timely anticipates, predicts, or determines when specific occupant activities ("activities") are occurring or about to occur. The monitoring system may function to proactively anticipate activities by anticipating, predicting, or determining when activities are occurring or about to occur, and to report, or otherwise record or document, activities. For example, the monitoring system may alert the user if an occupant is in distress, e.g., if an infant is crying or choking, if a pet or other animal is agitated, or if a component has moved to a position where it could damage a component or the interior of the vehicle. The monitoring system may learn from each instance of an activity or situation.

[0016] The monitoring system captures and records image data before, during, and after an activity occurs, documenting the overall situation surrounding the activity. Furthermore, the monitoring system may be manually activated by the user to display live, real-time video of the vehicle's interior at any time, such as when a red light is encountered or while the vehicle is temporarily parked.

[0017] Figure 1 is a block diagram of the monitoring system 100. The monitoring system 100 may be retrofitted to, coupled to, encompass, or contained within the vehicle 102. The monitoring system 100 may also couple to, connect to, or contain an external database 104. The monitoring system 100 may connect the external database 104 to a network 106 associated with the vehicle 102. The network 106 may be a local area network (LAN), wide area network (WAN), cellular network, internet, or a combination thereof, connecting, coupling, and / or communicating between the vehicle 102 and the external database 104.

[0018] The monitoring system 100 monitors the occupants of the vehicle and detects, identifies, predicts, and / or forecasts activities occurring within the vehicle 102, and / or potentially occurring in the near future. The monitoring system 100 may be activated immediately before an activity occurs and may record and capture image data before, during, and after the activity occurs. The monitoring system 100 may use artificial intelligence, including machine learning algorithms, to predict when an activity is about to occur or has occurred. The monitoring system may combine, connect to, or incorporate an internal edge computing device 118 for rapid and efficient processing at the location of the vehicle 102. The internal edge computing device may comprise one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural processing unit (NPU), etc. The edge computing device 118 may include a relational database or behavioral model database that provides models of the normal motion and / or characteristics of different objects or individuals. The edge computing device 118 may be updated and / or provide updates in real time. The edge computing device 118 may save the model and / or provide it to the ECU 108.

[0019] The monitoring system 100 may encompass the vehicle 102, be retrofitted to the vehicle, or be coupled to it. The vehicle 102 is a means of transport capable of transporting people, objects, or permanently or temporarily attached devices. The vehicle 102 may be a self-propelled wheeled vehicle such as an automobile, a sports utility vehicle, a truck, a bus, a van, or other motor-driven, battery-driven, or fuel cell-driven vehicle. For example, the vehicle 102 may be an electric vehicle, a hybrid vehicle, a hydrogen fuel cell vehicle, a plug-in hybrid vehicle, or any other type of vehicle having a fuel cell stack, motor and / or generator. Other examples of vehicles include bicycles, trains, airplanes or boats and any other form of transport capable of being moved. The vehicle 102 may be semi-autonomous or autonomous; that is, the vehicle 102 may be self-piloting and navigating without human input. The autonomous vehicle may be equipped with and use one or more sensors and / or navigation units for autonomous driving.

[0020] The monitoring system 100 comprises one or more processors, such as an electronic control unit (ECU) 108, and memory 110. The monitoring system 100 may also comprise other components such as a navigation unit 112, one or more sensors 114 including one or more internal cameras 116a, one or more external cameras 116b, a network access device 120, a user interface 122, and an output device 124. The monitoring system 100 may also comprise other sensors 136, such as a vehicle speed sensor and a proximity sensor. The monitoring system 100 may also combine, connect, and / or incorporate one or more vehicle components, such as a motor and / or generator 126, an engine 128, a battery 130, a transmission 132, and / or a battery management control unit (BMCU) 134.

[0021] ECU 108 may be implemented as a single ECU or a plurality of ECUs. ECU 108 may be electrically coupled to some or all of other components within vehicle 102, such as motor and / or generator 126, transmission 132, engine 128, battery 130, battery management control unit (BMCU) 134, memory 110, network access device 120, and / or one or more sensors 114. ECU 108 may include one or more processors or controllers specially designed to predict activities within vehicle 102. ECU 108 may generate a prediction model and use a machine learning algorithm to anticipate activities before they occur.

[0022] ECU 108 may analyze the internal environment of vehicle 102, compare data against a standard, and / or input data into a model to anticipate, predict, or determine any activity within the environment. If an activity is predicted or otherwise detected, ECU 108 may act to record, document, provide, or otherwise mitigate the results of the activity. ECU 108 may be coupled to memory 110 and execute instructions stored in memory 110.

[0023] Memory 110 may be coupled to ECU 108 and store instructions for ECU 108 to execute. Memory 110 may comprise one or more of random access memory (RAM) or other volatile memory or non-volatile memory. Memory 110 may be a non-transitory memory or data storage device, such as a hard disk drive, semiconductor disk drive, hybrid disk drive, or other suitable data storage device, and may further store machine-readable instructions that can be read and executed by ECU 108. Additionally, memory 110 may record, store, and use image data before, after, and / or during the occurrence of an activity to document the activity.

[0024] The monitoring system 100 may include a user interface 122. The monitoring system 100 may display one or more notifications on the user interface 122. The one or more notifications on the user interface 122 may be notified to the vehicle occupants when the monitoring system 100 is initialized or operated. The user interface 122 may include an input / output device that receives user input from user interface elements, buttons, dials, microphones, keyboards, or touchscreens. For example, the user interface 122 may receive user input including a configuration regarding the amount of image data or the length of video to record when an activity is detected. Additionally, the user interface 122 may receive user input including a configuration regarding when to activate cameras 116a - b, when to play back recordings, and when to transmit live video to the output device 124. An example of the output device 124 located in the front compartment of the vehicle is shown in FIG. 7. The output device 124 may be, for example, a display such as a head-up display (HUD) 502 in the windshield, a multi-information display (MID) 504 in the dashboard, and / or an IVI display 506. For example, each of the HUD 502, MID 504, and IVI display 506 may display a notification icon 508 indicating a specific activity of an occupant, such as a crying child, and / or a live video 510 of the occupant compartment. In addition to or alternatively to this, the output device 124 may be a display on a portable device. The portable device may also include a user interface 122. In other examples, the output device 124 may be a speaker, an audio and / or visual indicator, or a braille display.

[0025] The monitoring system 100 may include a network access device 120. The network access device 120 may include one or more communication ports or channels, such as a Wi-Fi unit, a Bluetooth® unit, a radio frequency identification (RFID) tag or reader, or a cellular network unit for accessing a cellular network (such as 3G, 4G, or 5G). The network access device 120 may send and receive data to and from an external database 104. For example, the ECU 108 may communicate with the external database 104 via the network 106 to obtain information about real objects near the location of the vehicle 102.

[0026] The monitoring system may include a navigation unit 112 and / or one or more sensors 114. The navigation unit 112 may be integrated with the vehicle 102, or it may be a separate unit coupled to the vehicle 102, such as a personal device with navigation capabilities. If the navigation unit 112 is separated from the vehicle 102, it may communicate with the vehicle 102 via a network access device 120. The vehicle 102 may instead include a Global Positioning System (GPS) unit (not shown) for detecting location data, including the vehicle 102's current location and date / time information. In this regard, the ECU 108 may perform the functions of the navigation unit 112 based on data received from the GPS unit. At least one of the navigation unit 112 or the ECU 108 may predict or propose a route set, including a start and destination location. The navigation unit 112 or the ECU 108 may perform navigation functions. The navigation function may include, for example, predicting routes and route sets, providing navigation instructions, and receiving user input such as verifying predicted routes and route sets or destinations.

[0027] The navigation unit 112 may provide and acquire navigation map information including location data that may include the current location, starting location, destination location, and / or the route between the starting location or current location and the destination location of the vehicle 102. The navigation unit 112 may have a memory (not shown) for storing route data. The navigation unit 112 may receive data from other sensors that can detect data corresponding to location information. For example, other sensors may include a gyroscope or an accelerometer.

[0028] Navigation map information may include real-world object information. This real-world object information may include the locations of places of interest, such as hospitals or veterinary clinics. The locations of such various real-world objects may be useful in situations where certain actions by the occupants occur, such as when an infant or animal is injured or becomes ill.

[0029] One or more sensors 114 may include one or more internal cameras 116a, one or more external cameras 116b and / or other sensors 136. One or more internal cameras 116a may include multiple cameras positioned within the vehicle 102 to capture different views of the interior of the vehicle 102, for example, as shown in Figure 5. One or more internal cameras 116a may be positioned within the vehicle 102, such as behind the front seats to capture the view of the rear seats, or behind the rear seats to capture the view of the cargo area behind the seats of the vehicle. One or more internal cameras 116a may be positioned in front of a second row of seats to capture the view of the occupants in the seats of the car, for example. One or more internal cameras 116a may be positioned near the rear window and facing outwards from the pickup truck, or mounted on the railing of the truck bed to capture a view of the truck bed, for example. One or more internal cameras may be positioned on the ceiling. One or more internal cameras 116a may acquire image data including a single frame or image or a continuous video of the interior of the vehicle 102 or the environment inside the vehicle. Different views within the vehicle may be used to form a panoramic or 360-degree image of the entire vehicle interior, so that the monitoring system 100 can capture activity inside the vehicle 102, such as a crying infant, a barking dog, or an object that has moved to an undesirable position. One or more internal cameras 116a may be portable cameras that can be easily mounted or installed inside the vehicle 102 by the user and may be powered via the vehicle 102's power supply, such as via a USB connector.

[0030] One or more external cameras 116b may include multiple cameras positioned outside the vehicle 102, for example as shown in Figure 6, to acquire different views of the surrounding environment outside the vehicle 102. One or more external cameras 116b may be positioned along the frame 602 of the vehicle 102, such as on the roof 604, trunk 606, or front 608 of the vehicle 102. Different views of the surrounding environment may be used to form a panoramic or 360-degree image of the surrounding environment outside the vehicle 102. One or more external cameras 116b may acquire image data including a single frame or image or a continuous video of the surrounding environment outside the vehicle 102, so that the surveillance system 100 can capture activities outside the vehicle 102 that may affect the activities of occupants inside the vehicle 102, such as an individual approaching the vehicle or attempting to open the vehicle door.

[0031] The monitoring system 100 may combine, connect, and / or include one or more vehicle components. One or more vehicle components may include a motor and / or generator 126. The motor and / or generator 126 may convert electrical energy into mechanical power such as torque, and may convert mechanical power into electrical energy. The motor and / or generator 126 may be coupled to a battery 130. The motor and / or generator 126 may convert energy from the battery 130 into mechanical power, for example, by returning energy to the battery 130 via regenerative braking. The vehicle 102 may have one or more additional power generation devices, such as an engine 128 or a fuel cell stack (not shown). The engine 128 supplies power by burning fuel instead of, and / or in addition to, the power supplied by the motor and / or generator 126.

[0032] The battery 130 may be coupled to the motor and / or generator 126, and may supply electrical energy to the motor and / or generator 126 and receive electrical energy from the motor and / or generator 126. The battery 130 may include one or more rechargeable batteries.

[0033] The BMCU134 may be coupled to the battery 130 and may control and manage the charging and discharging of the battery 130. For example, the BMCU134 may use a battery sensor to measure parameters used to determine the state of charge (SOC) of the battery 130. The BMCU134 may also control the battery 130.

[0034] One or more vehicle components may include a transmission 132. The transmission 132 may have different gears and / or modes such as parking, drive and / or neutral, and may shift between different gears. The transmission 132 controls the amount of power supplied to the wheels of the vehicle 102 when a certain amount of speed is given. One or more vehicle components may include a steering system 138. The steering system 138 controls the direction of movement of the vehicle to follow a desired path.

[0035] The monitoring system 100 may include or be linked to an external database 104. A database is any collection of information organized for search and retrieval by a computer or the like, and the database may be organized in tables, diagrams, queries, reports or any other data structures. The database may use any number of database management systems. The external database 104 may include a third-party server or website that stores or provides information. The information may include real-time information, regularly updated information or user-entered information. The server may be a computer on the network used to provide services to other computers on the network, such as access to files or sharing of peripherals.

[0036] The external database 104 may be a relational database or a behavioral model database that provides models of the normal motion and / or characteristics of different objects or individuals. The external database 104 may be updated in real time and / or provide updates. The external database 104 may store and / or provide models to the ECU 108.

[0037] Figure 2 is a flowchart of an exemplary process 200 for acquiring internal image data for use in predicting and / or detecting activity. One or more computers or one or more data processing devices, for example, the ECU 108 of the appropriately programmed monitoring system 100 in Figure 1, may perform process 200.

[0038] The monitoring system 100 determines whether to activate the internal camera 116a and / or the external camera 116b (208). The monitoring system 100 may also determine whether to activate the internal camera 116a and / or the external camera 116b based on the activity of one or more occupants inside the vehicle 102.

[0039] In some implementations, the monitoring system 100 is connected to an external database 104 via a network 106. The external database 104 may be, for example, a traffic information database. The monitoring system 100 may provide the traffic information database with the current location of the vehicle 102 and, accordingly, receive situational factors including the status of one or more traffic lights or signals located near the current location of the vehicle 102.

[0040] When the monitoring system 100 is activated, the monitoring system 100 determines whether to activate the internal camera 116a (202). The monitoring system 100 may be activated automatically when the vehicle starts. Alternatively, the monitoring system 100 may be activated manually by the driver, passenger in the front seat, or remote user of the autonomous vehicle. The monitoring system 100 may determine whether to activate the internal camera 116a based on the presence of a rear passenger in the vehicle 102. When the monitoring system 100 activates one or more internal cameras 116a, the monitoring system 100 uses one or more internal cameras 116a to capture image data (204).

[0041] In particular, one or more internal cameras 116a may capture image data of the environment inside the vehicle 102. One or more internal cameras 116a may be positioned within the range of the interior of the vehicle 102 and directed towards the interior of the vehicle 102 to monitor or capture image data of the occupants inside the vehicle 102. Each of the one or more internal cameras 116a may be directed towards different parts of the interior of the vehicle 102, for example, the rear of the front passenger compartment, the front of one or more rows of seats in the rear passenger compartment, the cargo area behind the seats, or outward toward the truck bed. One or more cameras 116a may also be positioned inside the trunk of the vehicle 102 to capture image data of the trunk. The image data captured by one or more internal cameras 116a may be a single image and / or a multi-frame video. The single image and / or multi-frame video may be stored in memory 110 or buffered. Different viewpoints and / or views may be captured and later stitched together, merged, or otherwise combined to form panoramic images and / or panoramic videos.

[0042] When the internal camera 116a is activated, the monitoring system 100 determines whether to activate the external camera 116b (206). The external camera may be activated based on the specific activity of one or more occupants, for example, a crying infant or an excited animal. When the monitoring system 100 activates one or more external cameras 116b, the monitoring system 100 uses one or more external cameras 116b to capture image data (208). The image data may be a single image and / or a video of multiple frames. The single image and / or video of multiple frames may be stored in memory 110 or buffered.

[0043] In particular, one or more external cameras 116b may capture image data of the surrounding environment outside the vehicle 102. One or more external cameras 116b may be positioned outside the vehicle 102. Each of the one or more external cameras 116b may capture different images from different viewpoints of the surrounding environment outside the vehicle 102, and as a result, all of the one or more external cameras 116b together may capture a 360-degree perspective view of the surrounding environment. The different images may later be stitched together, merged, or otherwise combined to form a panoramic image and / or panoramic video.

[0044] Image data from all cameras 116a to 116b may be combined with images or videos capturing the entire environment inside and outside the vehicle 102 for a certain period of time, or otherwise combined, and the image data may include different occupants inside the vehicle 102 and the environment outside the vehicle.

[0045] Figure 3 is a flowchart of an exemplary process 300 for detecting or predicting occupant activity when a monitoring system is activated. One or more computers or one or more data processing devices, for example, the ECU 108 of the monitoring system 100 in Figure 1, which is appropriately programmed, may perform process 300.

[0046] When the monitoring system 100 begins acquiring image data and monitoring activity, the monitoring system 100 acquires or generates a baseline or baseline model of occupants and their corresponding activities (302). The baseline or baseline model is used to predict, determine, or otherwise detect certain occupant activities. For example, the monitoring system 100 may predict when an infant occupant is sleeping or agitated. In another example, the monitoring system 100 may predict when an animal occupant will become agitated by an event occurring outside the vehicle. In yet another example, the monitoring system 100 may predict when an upright member will become unsafe or unbalanced. Figure 4 further illustrates the acquisition or generation of baseline or baseline models. The generation and use of baseline or baseline models provides the monitoring system 100 with a control model for comparing the current situation, thereby enabling the monitoring system 100 to predict when activity is about to occur. By modeling typical occupant activities, the monitoring system 100 adaptively uses machine learning to predict occupant activity.

[0047] The monitoring system 100 recognizes occupants from image data (304). To recognize different occupants in the image data, the monitoring system 100 may divide, outline, or otherwise map the figures in the image data using multiple joints and segments. Segments may represent linear representations or contours of occupants, and joints may represent vertices, contours, or other angles between different segments.

[0048] Once the occupant's contour is mapped, the monitoring system 100 may identify the occupant by comparing the representation of multiple joints and divisions with objects in the already mapped occupant database. For example, the monitoring system 100 may compare the contour of an occupant, such as a child or pet, with a stored occupant contour and determine that the contour shapes match, thereby recognizing the occupant as a child or pet. Once the occupant is identified as a child or pet, the monitoring system 100 monitors the occupant by dividing the image into specific regions of interest, thereby reducing the amount of image processing required by the monitoring system 100. For example, a child may be monitored in the areas of their face, head, and arms. As another example, a pet may be monitored in the areas of its head, mouth, legs, and tail.

[0049] After a crew member is recognized and matched with a corresponding crew member in the crew member database, the monitoring system 100 determines the crew member's corresponding baseline activity (306). The baseline and / or baseline model may include multiple activities that correspond to and are associated with the crew member in the crew member database that match the recognized crew member, such as an infant being awake and in a good mood, or an animal being sitting or lying down. Such multiple movements may be considered the crew member's baseline activity and may be used to predict the activity by comparing it with different crew member activities. The monitoring system 100 may utilize computer vision processing via the edge computing device 118 to efficiently process images and classify multiple desired recognitions to avoid unnecessary delays in alerting the user. This may be done by setting the frequency and sequence of prioritized recognition steps, simplifying the image comparison process, and reducing the detection zone in the image through dynamic segmentation. For example, the baseline and / or baseline model may include a set of frequencies for detecting a particular activity, and the particular activity may be identified in prioritized steps to ensure the accuracy of the prediction. For example, when an infant wakes up, a reference model might determine that the infant is awake by detecting head movements first, then mouth movements, then arm / hand / leg movements, then eye movements, and then movements of other parts of the body in that specific order.

[0050] Next, the monitoring system 100 determines the activity of the occupants (308). Occupant activity may include, for example, a crying infant, an animal occupant standing in the vehicle, or an object occupant that has fallen from an upright position. The monitoring system 100 tracks the occupant's segment or contour across multiple frames of image data acquired over a period of time. The time between the acquisition of different frames of image data may be pre-configured or pre-set. The monitoring system 100 may predict or determine the activity using a machine learning algorithm that uses a standard or reference model to make predictions or decisions using patterns. In addition, the monitoring system 100 may process self-learning to create / modify new standards or reference models to adapt to specific occupants. New standards or reference models may be shared with other vehicles via an external database 104.

[0051] In response to determining that crew activity is occurring, the monitoring system 100 may buffer, record, or acquire internal image data (310). The monitoring system 100 may have already buffered internal image data in memory 110 before detecting that activity is occurring. The monitoring system 100 may continuously record and / or acquire internal image data in a loop recording in memory 110. The monitoring system 100 may stop recording and / or acquiring internal image data when crew activity returns to a baseline. Separately, when the monitoring system 100 detects that activity is occurring, the monitoring system 100 may set a timer to stop buffering internal image data after a certain period of time. After the period of time has elapsed, the monitoring system 100 may stop recording and / or acquiring internal image data and save the internal image data in memory 110. The period of time may be pre-configured or configured via user input through the user interface 122.

[0052] Any image data captured outside the time frame between the start of buffered image data and the end of a certain period of time may be deleted from memory 110 or otherwise removed in order to conserve computing resources. Since the buffered internal image data starts recording before activity is detected, the monitoring system 100 records and stores image data before, during, and after the activity for a certain period of time thereafter.

[0053] In some implementations, the monitoring system 100 starts buffering and recording image data after determining that activity has occurred. By activating image data recording and / or buffering after detecting activity, the monitoring system 100 reduces the amount of energy required to operate the internal camera 116a and / or external camera 116b, thereby improving the energy efficiency of the vehicle 102.

[0054] The monitoring system 100 may provide image data to an output device (312). The output device is a computing device for the person using the monitoring system 100. The image data may also be notification icons indicating the activity of the occupants, such as an infant sleeping or a dog barking. In addition to or separately from this, the image data may also be live video footage of data captured from the internal camera 116a.

[0055] The monitoring system 100 may operate or control one or more vehicle components in response to the detection of specific occupant activities (314). The monitoring system may activate the external camera 116b when it detects that an occupant is disturbed by an event occurring outside the vehicle 102. The monitoring system 100 may slow down the vehicle 102 and pull over to the side of the road if it detects that a particular occupant activity may be dangerous to the occupant or distract the driver, for example, if an animal in the back of a truck is not in a safe position and / or has fallen off the vehicle. In another example, the navigation unit 112 may, in the case of an autonomous or semi-autonomous vehicle, notify the driver of a nearby hospital or reroute the vehicle to a nearby hospital when the monitoring system detects that an occupant is ill or in a physical emergency.

[0056] Figure 4 is a flowchart of an exemplary process 400 for generating, establishing, or determining crew standard activities. One or more computers or one or more data processing devices, such as the ECU 108 of the appropriately programmed monitoring system 100 in Figure 1, may perform process 400.

[0057] The monitoring system 100 observes the activities of the occupants in the vehicle, and if an occupant's activity is repeated many times, the monitoring system 100 determines that there is a pattern of movement and facial expression associated with a certain activity of occupant 402. For example, the monitoring system 100 may observe that an infant occupant frequently displays a happy expression with its eyes open. In addition, the monitoring system 100 may observe that an infant occupant closes its eyes while the infant is sleeping. In addition, the monitoring system 100 may observe that an infant occupant displays an excited expression with its mouth wide open when crying. As another example, the monitoring system 100 may observe that a dog occupant frequently sits and lies down. In addition, the monitoring system 100 may observe that a dog occupant stands on all four legs, or that the dog rapidly moves its head when barking. Using this data, the monitoring system 100 establishes that the most frequently occurring occupant activity is the baseline activity of occupant 406. If the frequency of occupant standard activities increases, the monitoring system 100 may increase the likelihood that the occupant activity is part of the standard. Conversely, if the frequency of occupant activity decreases, the monitoring system 100 may decrease the likelihood that the occupant activity is part of the standard. The percentage or frequency required for inclusion in the standard may be determined in advance and / or configured by the user. Separately, the monitoring system 100 may receive data entered by the user to indicate certain occupant activities and manually assign such activities as standards 404. For example, the monitoring system may observe occupants in the vehicle 102 and ask the user to identify and assign occupant activities as standards, for example, a fragile lamp installed upright.

[0058] Exemplary embodiments of the present invention have been disclosed in an exemplary manner. Therefore, terms used throughout this specification should be read non-limitingly. While minor modifications to the teachings herein may occur to those familiar with the art, it should be understood that any embodiments intended to be limited within the scope of the patents guaranteed herein are those reasonably included within the scope of the technological advance to which this specification contributes, and that scope is not limited except as the appended claims and their equivalents are taken into consideration.

Claims

1. A vehicle monitoring system, A first camera configured to capture first image data including one or more occupants inside the vehicle, A memory configured to store reference data relating to the sequence of multiple activities, movements, or facial expressions of at least one of the occupants of the vehicle, wherein the multiple activities, movements, or facial expressions of at least one of the occupants are prioritized. An electronic control unit coupled to the first camera and the memory, The first image data, including the one or more occupants, is acquired from the first camera. Based on the first image data, the order in which multiple activities, movements, or facial expressions of the at least one crew member are observed is determined. Based on the aforementioned reference data, a reference sequence of multiple activities, movements, or facial expressions of at least one occupant is identified. Based on a comparison of the aforementioned reference order and the aforementioned observation order, the future activities, movements, or facial expressions of at least one crew member are predicted. The first camera captures and records in the memory second image data of the at least one occupant during the periods before, during, and after the time associated with the predicted future activity, movement, or facial expression. An electronic control unit configured as follows, A monitoring system equipped with the following features.

2. The surveillance system according to claim 1, wherein the first camera includes one or more interior cameras configured to capture one or more different views inside the vehicle.

3. A vehicle monitoring system, A first camera configured to capture first image data including the occupants inside the vehicle, A memory configured to store reference data relating to the sequence of multiple activities, movements, or facial expressions of an occupant in the vehicle, wherein the multiple activities, movements, or facial expressions of the occupant are prioritized. An electronic control unit coupled to the first camera and the memory, First image data is acquired from the first camera. Based on the first image data, the order in which to observe multiple activities, movements, or facial expressions of the crew members is determined. Based on the aforementioned reference data, a reference sequence of multiple activities, movements, or facial expressions of the crew is identified. Based on a comparison of the aforementioned reference order and the aforementioned observation order, the future activities, movements, or facial expressions of the crew members are predicted. Second image data of the occupant during the periods before, during, and after the predicted future activity, movement, or facial expression is captured using the first camera and recorded in the memory. An electronic control unit configured as follows, A monitoring system equipped with the following features.

4. A method for notifying the user of an output device of the activities, movements, or facial expressions of occupants inside a vehicle, The processor acquires first image data of the vehicle's compartment, including images of the occupants, from the first camera. The processor determines, based on the first image data, the order in which to observe multiple activities, movements, or facial expressions of the occupants in the vehicle. The processor determines a reference order of multiple occupant activities, movements, or facial expressions based on reference data stored in a memory coupled to the processor, wherein the multiple occupant activities, movements, or facial expressions are prioritized. The processor predicts the occupant's future activities, movements, or facial expressions based on a comparison of the reference order and the observation order. The processor records second image data related to the predicted future activity, movement, or facial expression using the first camera. The processor stores the second image data in the memory, A method comprising transmitting the second image data to the output device by the processor, thereby notifying the user of the occurrence of the predicted future activity, movement, or facial expression.

5. The method according to claim 4, wherein recording the second image data includes recording the second image data for a period before and after a time related to the predicted future activity, movement or facial expression.

6. The processor acquires image data of the surrounding environment outside the vehicle from the second camera, The processor records and stores the image data in the memory using the second camera. The method according to claim 4, further comprising the processor transmitting the image data to the output device after predicting the future activity, movement or facial expression.

7. The method according to claim 6, wherein recording the image data includes recording the image data during a period before and after the time related to the predicted future activity, movement or facial expression.

Images