System and method for detecting abnormal passenger behavior in an autonomous vehicle

Abstract

Systems and methods for detecting abnormal passenger behavior in an autonomous vehicle are provided. Methods and systems for monitoring passengers within a cabin of a vehicle and determining whether the passengers are engaging in abnormal behavior are disclosed. The methods and systems use novel vectors to robustly and numerically represent the activities of passengers in corresponding frames, which vectors are referred to herein as "activity vectors." Additionally, the methods and systems utilize Gaussian mixture models to distinguish between normal and abnormal passenger behavior. Cluster components of the Gaussian mixture model are advantageously learned using an unsupervised approach in which the training data is not labeled or annotated to indicate normal and abnormal passenger behavior. In this way, the Gaussian mixture model can be trained at a very low cost.

Description

Technical Field

[0001] The apparatus and methods disclosed in this document relate to in-vehicle sensing systems, and more particularly, to detecting abnormal passenger behavior in an autonomous vehicle. Background Art

[0002] Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

[0003] In the near future, driverless cars, such as autonomous taxis operating for on-demand mobility services, will play an important role in transportation. It will soon become common practice for passengers who are strangers to each other to share autonomous taxis. Unlike traditional taxis where the driver can supervise the passengers, autonomous taxis will require monitoring systems to monitor the safety of the passengers. Any abnormal behavior of passengers, such as violent activities, should be detected and monitored for their prevention. Therefore, it would be beneficial to provide a monitoring system for monitoring passengers in the cabin of an autonomous vehicle and intelligently detecting abnormal passenger behavior. Summary of the invention

[0004] A method for detecting abnormal passenger behavior in a vehicle is disclosed. The method includes: receiving, using a processing system, a first image frame of at least one passenger in a cabin of the vehicle from an image sensor. The method further includes: determining, using the processing system, based on the first image frame, a first numerical vector representing a posture and motion of the at least one passenger in the first image frame. The method further includes: detecting, using the processing system, based on the first numerical vector, abnormal passenger behavior in the first image frame using a mixture model having a plurality of cluster components representing normal passenger behavior.

[0005] A system for detecting abnormal passenger behavior in a vehicle is disclosed. The system includes an image sensor configured to generate and output an image frame of at least one passenger in a cabin of the vehicle. The system further includes a processing system operably connected to the image sensor and including at least one processor. The processing system is configured to receive a first image frame from the image sensor. The processing system is further configured to determine a first numerical vector representing a posture and motion of the at least one passenger in the first image frame based on the first image frame. The processing system is further configured to detect abnormal passenger behavior in the first image frame using a mixture model having a plurality of cluster components representing normal passenger behavior based on the first numerical vector. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The foregoing aspects and other features of the method and system are explained in the following description considered in conjunction with the accompanying drawings.

[0007] Figure 1 is a schematic top view of a vehicle with a cockpit monitoring system.

[0008] Figure 2 yes Figure 1 Schematic view of the components of the vehicle and cockpit monitoring system.

[0009] Figure 3 A logic flow diagram of a method for detecting abnormal passenger behavior in a cabin of a vehicle is shown.

[0010] Figure 4 A logic flow diagram of a method for deriving an activity vector for an image frame is shown.

[0011] Figure 5 An exemplary image frame is shown in which two passengers are riding in the rear seats of a vehicle.

[0012] Figure 6 An exemplary sequence of five image frames is shown in which two passengers are riding in the rear seats of a vehicle.

[0013] Figure 7 A further exemplary image frame is shown in which a passenger is pushing another passenger.

[0014] Figure 8 The diagram shows a Figure 7 Graph of activity vectors calculated for an exemplary image frame. DETAILED DESCRIPTION

[0015] For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the accompanying drawings and described in the following written specification. It should be understood that no limitation of the scope of the present disclosure is thereby intended. It should be further understood that the present disclosure includes any changes and modifications to the illustrated embodiments, and includes further applications of the principles of the present disclosure, as would normally occur to a person skilled in the art to which the present disclosure belongs.

[0016] System Overview

[0017] refer to Figure 1-2, discloses an exemplary embodiment of a vehicle 100 having a cabin monitoring system 104. The cabin monitoring system 104 is advantageously configured to monitor passengers within a cabin 108 of the vehicle 100 and determine whether the passengers are engaging in abnormal behavior. In addition to the cabin monitoring system 104, the vehicle 100 also includes a vehicle electronic control unit ("ECU") 112 configured to operate a drive system 116, and various electronic devices of the vehicle other than the cabin monitoring system 104, such as lights, locks, speakers, displays, etc. The drive system 116 of the vehicle 100 includes: a drive motor (e.g., an internal combustion engine and / or one or more electric motors) that drives the wheels of the vehicle 100, and steering and braking components that enable the vehicle 100 to move in a controlled manner.

[0018] exist Figure 1 In the illustrated embodiment of the vehicle 100, the vehicle 100 takes the form of a car. However, in other embodiments, the vehicle 100 may include any number of types of containers having one or more cabins 108 for moving personnel, such as trains, buses, subways, airplanes, helicopters, passenger drones, submarines, elevators, and passenger moving pods. The cabin 108 (which may also be referred to herein as a compartment) is a typical enclosed room for accommodating passengers. Although the vehicle 100 is illustrated as having a single cabin 108, it will be appreciated that the vehicle 100 may include any number of separate and separate cabins 108 (e.g., multiple compartments or rooms inside a train car). In the illustrated embodiment, the cabin 108 includes four seats 120, 122, 124, 126 in which passengers can sit. However, depending on the configuration and type of the vehicle 100, the cabin 108 may include more or fewer seats. The vehicle 100 also includes one or more doors (not shown) that allow passengers to enter the cabin 108 and the seats 120-126. Additionally, the vehicle 100 may include a rear hatch (not shown) that enables a user to access a cargo storage area of ​​the vehicle 100 , such as a trunk or storage space behind the rear seats 124 , 126 .

[0019] In at least one embodiment, the vehicle 100 is a shared autonomous car that is configured to provide an autonomous transportation service in which the vehicle 100 autonomously drives to the location of a passenger and then autonomously transports the passenger to a desired location using a public road network when the passenger enters the vehicle 100. The passenger can use, for example, a smart phone or smart device application (i.e., an "app") to engage in the autonomous transportation service of the vehicle 100. Passengers are also referred to herein as occupants, users, operators, or personnel. In other embodiments, as described above, the vehicle 100 is any type of passenger vehicle, and in some embodiments, the vehicle 100 can be occupant-controlled or remotely controlled.

[0020] The cockpit monitoring system 104 includes a vehicle computer 130 that is operably connected to one or more image sensors 134, 138 disposed throughout the vehicle. The image sensors 134, 138 may be video or still image RGB cameras, each having, for example, a charge coupled device (CCD) or an active pixel sensor for generating digital image data in the form of image frames. In other embodiments, the image sensors 134, 138 may include thermal or infrared sensors, radar imaging systems, LIDAR imaging systems, or another suitable imaging system.

[0021] In the illustrated embodiment, the cabin monitoring system 104 includes two interior image sensors 134, 138 that are disposed within the cabin 108 and configured to generate an image of a portion of the cabin 108. In one embodiment, the interior image sensors 134, 138 are disposed in or on the ceiling of the vehicle 100 and are pointed downwardly into the cabin 108 toward the respective one or more seats 120-126 for imaging. In other embodiments, the interior image sensors 134, 138 may be disposed in the seats or in the instrument panel of the vehicle 100. For example, in one particular embodiment, the image sensors for imaging the front seats 120, 122 are disposed in the instrument panel of the vehicle 100, while the image sensors for imaging the rear seats 124, 126 are disposed in the front seats 120, 122 that are directly in front of the respective rear seats 124, 126. In some embodiments, additional exterior image sensors (not shown) may be disposed on the exterior of the vehicle 100 to generate an image of a portion of the exterior of the vehicle 100 .

[0022] In the illustrated embodiment, the front image sensor 134 generates digital image data of the front of the cabin including the front seats 120, 122, and the rear image sensor 138 generates digital image data of the rear of the cabin 108 including the rear seats 124, 126. In other embodiments, the cabin monitoring system 104 may include a single image sensor that captures an image of the entire cabin 108 including all of the seats 120-126, separate image sensors directed toward each seat 120-126, or any desired configuration of image sensors to generate a digital image of each seat in the vehicle.

[0023] The vehicle computer 130 is configured to process image data received from one or more of the image sensors 134, 138 to monitor passengers within the cabin 108 of the vehicle 100 and determine whether the passengers are engaging in abnormal behavior. The vehicle computer 130 may additionally be configured to perform other complex tasks, such as autonomous navigation of the vehicle 100, and interfacing with passengers or smartphones owned by passengers to provide autonomous transportation of passengers.

[0024] Reference now Figure 2 , describes exemplary components of the vehicle computer 130 of the cabin monitoring system 104. In the illustrated embodiment, the vehicle computer 130 includes at least a processor 200 and an associated memory 204. The memory 204 is configured to store program instructions that, when executed by the processor 200, enable the vehicle computer 130 to perform various operations described elsewhere herein, including at least monitoring passengers within the cabin 108 of the vehicle 100 and determining whether the passengers are engaging in abnormal behavior. The memory 204 can be any type of device capable of storing information accessible to the processor 200, such as any of a memory card, ROM, RAM, hard drive, disk, flash memory, or various other computer-readable media that act as data storage devices, as will be recognized by those of ordinary skill in the art. Additionally, it will be recognized by those of ordinary skill in the art that a "processor" includes any hardware system, hardware mechanism, or hardware component that processes data, signals, or other information. The processor 200 may include a system having a central processing unit, a graphics processing unit, a plurality of processing units, a dedicated circuit for implementing functionality, programmable logic, or other processing systems.

[0025] In the illustrated embodiment, the vehicle computer 130 further includes a communication interface 208 configured to enable the vehicle computer 130 to communicate with the image sensors 134, 138 and with the vehicle ECU 112 via one or more communication buses 142, which may take the form of one or more controller area network (CAN) buses. The communication interface 212 may include physical terminals for connecting to a wired medium (e.g., the communication bus 142). Additionally, the communication interface assembly 212 may include one or more modems, bus controllers (e.g., a suitable CAN bus controller), or other such hardware configured to enable communication with the image sensors 134, 138 and the vehicle ECU 112.

[0026] In the illustrated embodiment, the vehicle computer 130 further includes one or more radio transceivers 212 configured to communicate with a remote server (e.g., a cloud service) and with a smartphone or other smart device owned by a passenger for the purpose of providing autonomous transportation services. The radio transceiver(s) 212 may include a transceiver configured to communicate with the Internet via a wireless telephone network, such as a Global System for Mobile ("GSM") or Code Division Multiple Access ("CDMA") transceiver. Additionally, the radio transceiver(s) 212 may include a Bluetooth® or Wi-Fi transceiver configured to communicate locally with a smartphone or other smart device owned by a passenger.

[0027] As will be described in more detail below, the memory 204 of the vehicle computer 130 stores program instructions corresponding to an abnormal behavior detection program 216. The abnormal behavior detection program 216 includes program instructions and learning parameters corresponding to a gesture detection model 220 and an activity classification model 224. Additionally, the memory 204 stores image data 228 including image frames received from the image sensors 134, 138, and activity data 232 representing the activity of the passenger in each image frame.

[0028] Method for detecting abnormal passenger behavior

[0029] The cabin monitoring system 104 is advantageously configured to monitor passengers within the cabin 108 of the vehicle 100 and determine whether the passengers are engaging in abnormal behavior. For the purpose of explanation only, it is noted that abnormal passenger behavior can include violent behavior (such as arguing, fighting, grabbing, kicking, punching, pushing or slapping) as well as non-violent behavior (such as stripping). In contrast, normal passenger behavior can include behavior such as talking, touching, hugging, sitting quietly, drinking coffee or crossing legs.

[0030] As will be discussed in more detail below, the cabin monitoring system 104 uses a novel vector to robustly and numerically represent the activity of the passenger in the corresponding frame, which is referred to herein as the "activity vector" of the corresponding image frame. Additionally, the activity classification model 224 includes a mixture model, particularly a Gaussian mixture model (GMM), which the cabin monitoring system 104 utilizes to distinguish between normal and abnormal passenger behavior. In particular, based on training data in the form of videos of passengers riding in the cabin 108 of the vehicle 100, Gaussian mixture modeling is used to learn cluster components representing activity vectors corresponding to normal passenger behavior. Therefore, the cabin monitoring system 104 can determine whether a passenger is engaging in abnormal behavior by comparing the activity vector representing actual passenger behavior with the learned cluster components representing normal passenger behavior. Therefore, it will be appreciated that, as used herein, as it relates to the activity classification model 224 and / or its mixture model, the term "abnormal behavior" or "abnormal passenger behavior" refers only to passenger behavior that is uncommon or rare in the training data, and no specific qualitative or value-based meaning is given to the term.

[0031] Advantageously, an unsupervised approach can be utilized in which the training data is not labeled or annotated to indicate normal and abnormal passenger behavior. In particular, because abnormal behaviors such as violence are generally rare, unannotated videos of passengers riding in the cabin 108 of the vehicle 100 can be used to learn cluster components representing normal passenger behavior. The unsupervised approach is advantageous because a large corpus of training data can be collected at very low cost and used for training. Additionally, because the definition of abnormal behaviors such as violence varies across individuals, the quality of annotations will be questionable in supervised approaches, which in turn will lead to poor performance. In addition, since abnormal behaviors such as violence rarely occur in practice, it will be difficult to gather all possible abnormal behaviors in the training data in supervised approaches. In addition, supervised approaches tend to rely on a large number of artificially crafted features, which may work well in the case of existing training data, but when future abnormal behaviors are different from those in the training data, these features may not be universal for detecting the future abnormal behaviors.

[0032] Figure 3 A logical flow diagram of a method 300 for detecting abnormal passenger behavior in a cabin of a vehicle is shown. In the description of the method 300, the statement that a method, process, module, processor, system, etc. is performing a certain task or function refers to a controller or processor (e.g., processor 200) executing programming instructions (e.g., program instructions 208) stored in a non-transitory computer-readable storage medium (e.g., memory 204) operatively connected to the controller or processor to manipulate data or operate one or more components in the cabin monitoring system 108 and / or the vehicle 100 to perform the task or function. Additionally, the steps of the method can be performed in any feasible time order, regardless of the order shown in the various figures or the order in which the steps are described. It will be appreciated that in some embodiments, the operations of the processor 200 described herein can be performed by other components of the vehicle 100 and / or the cabin monitoring system 108, such as the vehicle ECU 112 or the integrated image processor of the sensor 134, 138. Additionally, in some embodiments, the operations of the processor 200 described herein may be performed by a remote server, such as in a cloud computing system.

[0033] The method 300 begins with a step of receiving an image frame and incrementing a frame count (block 310). In particular, the processor 200 of the vehicle computer 130 operates at least one of the image sensors 134, 138 to receive a video feed consisting of a sequence of image frames at a defined frame rate (e.g., 25 frames per second). In at least one embodiment, the processor 200 stores the received image frames as image data 228 in the memory 204. It will be appreciated that each image frame includes a two-dimensional array of pixels. Each pixel has at least corresponding photometric information (e.g., intensity, color, and / or brightness). In some embodiments, the image sensors 134, 138 may also be configured to capture geometric information corresponding to each pixel (e.g., depth and / or distance). In such embodiments, the image sensors 134, 138 may, for example, take the form of two RGB cameras configured to capture stereo images from which depth and / or distance information may be derived; and / or an RGB camera with an associated IR camera configured to provide depth and / or distance information.

[0034] As will be discussed below, in at least some embodiments, certain processes of method 300 are performed with respect to each image frame, while other processes are performed only every so many frames (e.g., every 75 frames or every 3 seconds). As described below, this can be accomplished with a hyperparameter having a value (e.g., 75) detect_every_frame ( Every_frame_detection ). Therefore, in at least some embodiments, when each image frame is received and processed, the processor 200 of the vehicle computer 130 is configured to use frame count ( Frame Count ) increases, frame count For example, it is stored in the memory 204 .

[0035] The method 300 continues with the step of deriving an activity vector based on the image frames (block 320). In particular, the processor 200 of the vehicle computer 130 calculates the activity vector for each image frame received from the image sensors 134, 138. X i ,in i Indicates the index of the image frame. As used herein, an "activity vector" refers to a numerical vector representing at least the following: (i) the posture of at least one passenger in the image frame, and (ii) the movement of at least one passenger in the image frame. As used herein, the "posture" of a passenger refers to the position, posture, orientation, etc. of the passenger. In particular, in the detailed embodiments described herein, the activity vector represents the position of multiple key points corresponding to specific joints and body parts of each passenger in the image frame, as well as the direction and speed of movement of those key points.

[0036] Figure 4 A logic flow diagram of a method 400 for deriving an activity vector for an image frame is shown. In the description of the method 400, the statement that a method, process, module, processor, system, etc. is performing a task or function refers to a controller or processor (e.g., processor 200) executing programming instructions (e.g., program instructions 208) stored in a non-transitory computer-readable storage medium (e.g., memory 204) operatively connected to the controller or processor to manipulate data or operate one or more components in the cabin monitoring system 108 and / or the vehicle 100 to perform the task or function. Additionally, the steps of the method can be performed in any feasible time order, regardless of the order shown in the various figures or the order in which the steps are described. It will be appreciated that in some embodiments, the operations of the processor 200 described herein can be performed by other components of the vehicle 100 and / or the cabin monitoring system 108, such as the vehicle ECU 112 or the integrated image processor of the sensor 134, 138, etc. Additionally, in some embodiments, the operations of the processor 200 described herein can be performed by a remote server such as in a cloud computing system.

[0037] Given an image frame, method 400 begins by detecting ( e In the step of detecting key points of each of the passengers in the image frame (block 410), the processor 200 of the vehicle computer 130 uses the pose detection model 220 to detect a plurality of key points corresponding to specific joints or body parts of each passenger in the image frame. In at least one embodiment, the processor 200 also uses the pose detection model 220 to detect the number of passengers in the image frame (block 410). e ). In at least one embodiment, the gesture detection model 220 includes a deep neural network (DNN) that has been trained based on a corpus of training data (which is different from the training data used to train the GMM of the activity classification model 224 discussed above). The processor 200 executes program instructions of the gesture detection model 220 with reference to a set of learned parameters, weights, and / or kernel values ​​learned during the training of the gesture detection model 220 to detect a plurality of key points for each passenger. In at least one embodiment, each key point is represented by a two-dimensional coordinate pair ( x t , y t ) in the form of x t represents the horizontal position in the image frame, y t represents the vertical position in the image frame, and trepresents the time or frame number of the image frame. However, it will be appreciated that where the image sensors 134, 138 provide depth and / or distance information, three-dimensional coordinate triplets may also be used.

[0038] Figure 5 An exemplary image frame 500 is shown in which two passengers are riding in the rear seats of the vehicle 100. A plurality of key points 510 are identified for each of the two passengers. In the illustrated example, the posture detection model 220 is configured to detect 25 key points, including: (1) right eye, (2) left eye, (3) nose, (4) right ear, (5) left ear, (6) neck, (7) right shoulder, (8) left shoulder, (9) right elbow, (10) left elbow, (11) right wrist, (12) left wrist, (13) right hip, (14) mid hip, (15) left hip, (16) right knee, (17) left knee, (18) right ankle, (19) left ankle, (20) right heel, (21) left heel, (22) right big toe, (23) left big toe, (24) right little toe, and (25) left little toe. However, it will be appreciated that for a particular image frame, some keypoints 510 may be outside the frame or occluded.

[0039] In at least one embodiment, the processor 200 is configured to smooth the values ​​of the coordinate values ​​predicted by the gesture detection model 220 for each passenger's key points. In particular, due to limitations in model performance, the predicted coordinate values ​​provided by the gesture detection model 220 may have some undesirable jitter between image frames. To overcome this artifact, the processor 200 is configured to calculate the coordinate value of each key point as the average of the sequence of predicted coordinate values ​​from the gesture detection model 220. In particular, the processor 200 calculates the time or frame number according to the following equation t The coordinate values ​​of each key point at :

[0040]

[0041] in is determined by the posture detection model 220 at a time or frame number t The predicted coordinate values ​​provided at Pose smooth is a smoothing hyperparameter taking an integer value (e.g., 10). In other words, the processor 200 calculates the coordinate value of each key point as the predicted coordinate value of the current image frame and the predetermined number Pose smooth The average of the predicted coordinate values ​​of the previous image frames.

[0042] Return to Figure 4, the method 400 continues with the step of determining an optical flow vector for each key point (block 420). In particular, the processor 200 of the vehicle computer 130 calculates an optical flow vector for each key point, the optical flow vector representing the direction and speed of movement of the corresponding key point. In some embodiments, the processor 200 calculates each optical flow vector of the key point as the difference between the coordinate value of the key point in the current image frame and the coordinate value of the key point in the previous image frame. In particular, in one embodiment, the processor 200 calculates the time or frame number according to the following equation t The optical flow vector of the key point at:

[0043]

[0044] in It's time t The key points at x t , y t ), and Flow smooth is a smoothing hyperparameter that takes an integer value (e.g. 3).

[0045] Figure 6 An exemplary sequence 600 of five image frames is shown, in which two passengers are riding in the rear seats of the vehicle 100. The optical flow vector of the keypoint corresponding to the left ear of the passenger on the right side of the image frame is calculated as follows: t =5 frame with the coordinates of the left ear key point t =2 in the frame where the coordinate values ​​of the left ear key point are compared.

[0046] Return to Figure 4 The method 400 continues with the following steps: for each passenger, the key points are classified into a × b The key points are classified into the cells of the grid based on the optical flow angle of the key points. d - Bin Block ( d -bin) histogram (block 430). In particular, the received image frame is divided into a × b The cells of the grid, where a is a grid height hyperparameter that takes an integer value (e.g., 7), and b is a grid width hyperparameter that takes an integer value (e.g. 13). a × b Each cell of the grid represents a range of horizontal coordinate values ​​and a range of vertical coordinate values ​​within the image frame. In at least one embodiment, a × b Each cell of the grid has equal size. For example, Figure 5 , the exemplary image frame 500 is divided into a 7×13 unit grid 520. Figure 6 The sequence of five image frames 600 is similarly divided into a grid of cells.

[0047] Additionally, for a × b Each cell of the grid is defined for each passenger d -Bin histogram (e.g., 3-bin histogram). Each d The bins represent the range of optical flow angles. For example, a 3-bin histogram may include a first bin representing the optical flow angle range of 0°-120°, a second bin representing the optical flow angle range of 120°-240°, and a third bin representing the optical flow angle range of 240°-360°. The optical flow angles may be relative to any arbitrary zero angle, such as relative to an image frame and / or a × b The horizontal x-axis of the grid. It will be appreciated that the To calculate the optical flow angle of the optical flow vector relative to the horizontal x-axis, It's time t The key points at x t , y t ) The optical flow angle.

[0048] The processor 200 classifies each specific passenger's key points into a × b In the cells of the grid: the coordinate values ​​of the key points are compared with the corresponding a × b The value of each specific cell of the grid is compared to the range. In other words, if the coordinate value of the key point ( x t , y t ) in the definition a × b If the value of a specific cell of the grid is within the range, the processor 200 classifies the key point into a × b Next, the processor 200 converts a × b The key points of each passenger in each cell of the grid are classified into a × b The corresponding cells of the grid correspond to the corresponding passengers d- in one of the bins in the bin histogram: the optical flow angle of the keypoint is compared to the range of the optical flow angle range of the corresponding bin of the histogram. In other words, if the keypoint has an optical flow angle within the range of the optical flow angle range defined by a particular bin, the processor classifies the keypoint into that particular bin. It will be appreciated that since there is a e Each of the passengers has d - Bin histogram a × b unit, so each key point depends on its coordinate value ( x t , y t ) and its optical flow angle and is classified in a × b × d × e In a corresponding one of the different bin blocks.

[0049] The method 400 continues with the following steps: for each histogram bin of each cell of each passenger, a value is calculated to obtain an activity vector for a given image frame (block 440). In particular, the processor 200 calculates a value for each bin of each histogram in each cell of each passenger that is equal to the sum of the magnitudes of the optical flow vectors of the keypoints that have been classified into the corresponding bin. More particularly, the processor 200 calculates the magnitude of the optical flow vector for each keypoint. It will be appreciated that the magnitude of the optical flow vector can be calculated according to the equation To calculate the magnitude of the optical flow vector, It's time t The key points at x t , y t ) Finally, the processor 200 calculates the value of each bin as the sum of the magnitudes of the optical flow vectors of the key points classified into the corresponding bin. These calculated values ​​form a matrix with dimension a × b × d × e The activity vector X i ,in i Indicates the index of the image frame. It will be appreciated that the magnitude of the calculated value is proportional to the amount of activity in the image frame in the corresponding region and direction defined by the corresponding cell and histogram bin. In this way, the activity vector X i The movements and / or activities of the two passengers within the image frame are encoded in a numerical form that can be more easily evaluated.

[0050] Figure 7 A further exemplary image frame 700 is shown in which a passenger is pushing another passenger. Figure 8 A graph 800 illustrating activity vectors calculated based on an exemplary image frame 700 is shown. In graph 800, cells 810 correspond to cells 710 of the exemplary image frame 700. In each cell 810 of graph 800, a 3-bin histogram is shown for each passenger. In particular, the optical flow vectors and key points for passengers on the right hand side of the image frame 700 are represented by solid black histogram bins 820 in graph 800. Conversely, the optical flow vectors and key points for passengers on the left hand side of the image frame 700 are represented by diagonally shaded histogram bins 830 in graph 800. The height of each histogram bin corresponds to the activity vector X i As can be seen, there is only minimal overlap of the key points of these two passengers (i.e., only one cell shows a histogram for both passengers). Additionally, as can be seen, the cell corresponding to the left arm of the passenger on the left hand side of image frame 700 shows a diagonally shaded histogram bin of considerable height, which indicates motion of considerable magnitude (i.e., rapid motion).

[0051] Return to Figure 3 , the method 300 continues with the following step: classifying the image frame into the cluster with the highest posterior probability based on the activity vector (block 330). In particular, for each image frame, the processor 200 determines the activity vector X i Most likely corresponds to multiple learned cluster components C i More specifically, the processor 200 refers to a plurality of learned cluster components. C i to execute the program instructions of the activity classification model 224 to convert the activity vector X i Classify as most likely to belong to a specific learned cluster component C i In other words, the cluster components C i is considered as a latent variable describing the activity class represented in the image frame and is based on the measured activity vector X i And predicted.

[0052] As noted above, the activity classification model 224 includes a Gaussian mixture model (GMM) that defines a plurality of cluster components corresponding to normal passenger behavior. C i Cluster componentsC i Each is included in the dimension a × b × d × e (i.e., with the activity vector X i same dimensions) ,in Is a dimension a × b × d × e the cluster centers and / or medians of , and Is a dimension a × b × d × e The covariance matrix of the activity classification model 224 is given by k Different cluster components C i In other words, given the cluster components, the activity vector for each frame comes from p -dimensional multivariate normal:

[0053]

[0054] The variables C Is has K The category distribution of different categories, p 1 , p 2 ,… p k Is a dimension a × b × d × e The density function of the indicator variable C Get a specific value c possibility, and Is a specific value c The normal distribution of .

[0055] Activity vector based on a specific image frame X i , the processor 200 classifies the image frame into the cluster component with the highest posterior probability according to the following equation C i middle:

[0056] .

[0057] In other words, for each value i = 1,…, k, processor 200 calculates the posterior probability , which indicates the activity vector X i Belongs to a specific cluster component C i The processor will be active vector X i The class with the highest posterior probability Cluster components C i . Activity vector X i The cluster component to which it belongs C i In this article, it is marked as c i In at least one embodiment, processor 200 converts the activity vector X i The component of the identified cluster that it most likely belongs to c i Stored in memory 204.

[0058] As alluded to above, prior to deploying the cabin monitoring system 104, the plurality of cluster components are learned based on unlabeled training data in the form of videos of passengers riding in the cabin 108 of the vehicle 100. C i In particular, based on the above Figure 4 The described method derives a large set of training activity vectors from image frames from a training video. X i For exporting k Different cluster components C i The GMM k Different cluster components C i Optimally apply this large set of training activity vectors X i Use the expectation maximization algorithm to estimate the components of each cluster C i The unknown parameters and .

[0059] Additionally, it will be appreciated that GMM requires the number of cluster components to be k is pre-specified. In at least one embodiment, the number of cluster components k The selection is done by the Akaike Information Criterion (AIC). AIC is defined as:

[0060]

[0061] in P are the unknown parameters to be estimated (i.e., , and ,in l = 1,…, K ) and L is the likelihood function, or in other words the observed training activity vector X i The density at i = 1,…, n ,in n is the training activity vector X i The total number of .

[0062] A smaller AIC indicates a better fit for the model while penalizing the use of a complex model due to the number of unknown parameters. P In one embodiment, for k A predetermined range of values ​​(e.g. k = 1,…,20) to calculate AIC, and get the one with the lowest AIC k The values ​​are used to derive the GMM of the activity classification model 224 .

[0063] In at least one embodiment, the training process is uniquely performed for different numbers of passengers using unlabeled training data in the form of videos of corresponding numbers of passengers riding in the cabin 108 of the vehicle 100. In particular, corresponding multiple cluster components may be learned for a single passenger riding alone, for two passengers riding together, for three passengers riding together, and so on up to some reasonable upper limit on the number of passengers expected to ride in a particular area of ​​the cabin 108 within the field of view of the image sensor. C i .

[0064] Method 300 continues with the step of determining the posterior density of the image frame (block 340). In particular, once the activity vector X i The cluster component that most likely belongs to c i Once determined, the processor 200 calculates the posterior density according to the following equation:

[0065]

[0066] in f () is given by the activity vector X i and the identified cluster components ci In at least one embodiment, the processor calculates the probability density function of the GMM to be evaluated in the case of Posterior density i ( Posterior density i ) are stored in memory 204.

[0067] As described below, if the determined posterior density of an image frame is below a predetermined threshold, the image frame may be considered abnormal or include abnormal passenger behavior. In this way, the processor 200 may detect abnormal passenger behavior on a frame-by-frame basis by comparing the posterior density of each image frame with a predetermined threshold. However, it is generally not necessary to detect whether an abnormality occurs with every frame (e.g., every 1 / 25=0.04 seconds) because the abnormal behavior situation will not change at such a high frequency. Therefore, in at least one embodiment, the processor 200 instead detects abnormal passenger behavior only every many frames based on the average posterior density over several frames.

[0068] Method 300 repeats steps 310-340 to determine the posterior density of the sequence of image frames until the frame count equals the threshold number of frames (block 350). In particular, as noted above, as each frame is received, processor 200 causes frame_count As each image frame is received, the processor 200 repeats the following process: deriving the activity vector X i , determine the activity vector X i The cluster component that most likely belongs to c i , and calculate the image frame Posterior density i ,until frame_count Equal to the hyperparameter detect_every_frame (For example, 75, so that abnormal behavior is detected every 3 seconds at 25 frames per second).

[0069] The method 300 continues with the following step: checking whether the average posterior density of the image frame sequence is less than a threshold (block 360). In particular, the processor 200 calculates the average posterior density of the image frame sequence since the last reset. frame_count and all image frames received since the last abnormal behavior detection Posterior density i The average value of and compares the average value with a predetermined abnormal threshold. In other words, the processor 200 evaluates the following equation:

[0070] .

[0071] If the average posterior density is less than the threshold, the method 300 continues to detect abnormal passenger behavior (block 370). In particular, in response to the average posterior density being less than a predetermined abnormal threshold, the processor 200 detects that abnormal passenger behavior has occurred. In at least one embodiment, in response to detecting abnormal passenger behavior, the processor 200 operates the transceiver 212 to transmit an abnormal notification message to a remote server, such as a cloud backend or a remote database. The abnormal notification message may include information about the image frame and / or activity vector for which the abnormal passenger behavior was detected. X i .

[0072] The remote server may be accessible, for example, by an operator of an autonomous taxi service or other similar autonomous vehicle service or shared vehicle service, and may interface with an external cloud service associated with the service. In one embodiment, the remote server is configured to notify the operator (e.g., via email, etc.) in response to detecting abnormal behavior. In other embodiments, the operator may access the relevant image data and / or abnormal behavior event data stored on the remote server via a web portal.

[0073] In a further embodiment, in response to detecting abnormal passenger behavior, the processor 200 may operate a speaker or display screen (not shown) disposed within the cabin 108 of the vehicle 100 to display, play, or otherwise output an alarm or warning to the passenger, such as urging the passenger to stop the abnormal behavior.

[0074] Regardless of whether the average posterior density is less than the threshold, method 300 continues with the step of resetting the frame count (block 380) before completely repeating method 300. In particular, after abnormal behavior detection, before performing abnormal behavior detection again, processor 200 will frame_count Reset to zero and repeat the following process: receive image frame, derive activity vector X i , determine the activity vector X i The cluster component that most likely belongs to c i , and calculate the posterior density of each image frame i ,until frame_ count Equal to the hyperparameter detect_every_frame until.

[0075] Although the present disclosure has been illustrated and described in detail in the drawings and the foregoing description, it should be considered illustrative rather than restrictive in nature. It should be understood that only preferred embodiments have been presented and all changes, modifications and further applications falling within the spirit of the present disclosure are desired to be protected.

Claims

1. A method for detecting abnormal passenger behavior in a vehicle, the method include: receiving, with a processing system, a first image frame of at least one passenger in a cabin of a vehicle from an image sensor; determining, with the processing system, based on the first image frame, a first numerical vector representing a posture and a motion of the at least one passenger in the first image frame; as well as detecting abnormal passenger behavior in the first image frame using, with the processing system, a mixture model having a plurality of cluster components representing normal passenger behavior based on the first numerical vector; Wherein determining the first numerical vector further comprises: determining, with the processing system, a respective plurality of key points for each of the at least one passenger, each key point comprising a coordinate pair corresponding to a location of a respective joint or body part of the at least one passenger within the first image frame; determining, with the processing system, an optical flow vector for each respective key point of the respective plurality of key points for each of the at least one passenger, the optical flow vector indicating motion of the respective key point in the first image frame relative to the at least one previous image frame; categorizing, using the processing system, each respective key point of the respective plurality of key points for each of the at least one passenger into a respective cell of a two-dimensional grid of cells based on the coordinate pairs of the respective key points, wherein each respective cell of the grid corresponds to a range of coordinates within the first image frame; classifying, using a processing system, each corresponding keypoint classified into each corresponding cell of the grid into a corresponding bin of a corresponding histogram for each of the at least one passenger based on an optical flow angle of an optical flow vector of the corresponding keypoint, wherein each bin of the corresponding histogram for each of the at least one passenger corresponds to an optical flow angle range; determining, with the processing system, a value for each bin of the corresponding histogram for each of the at least one passenger as a sum of optical flow values ​​of the optical flow vectors for each keypoint classified into the corresponding bin; and A first numerical vector is formed, using a processing system, having a numerical value for each bin of a corresponding histogram for each of the at least one passenger.

2. The method according to claim 1, further comprising determining a plurality of key points corresponding to each of the at least one passenger. include: The processing system is used to determine the coordinate pair of each key point in the corresponding multiple key points of each of the at least one passenger as: the average position of the corresponding joint or body part of the at least one passenger on multiple image frames, and the multiple image frames include a first image frame and at least one previous image frame.

3. The method according to claim 1, further comprising: determining the optical flow vector include: A difference between a coordinate pair of corresponding key points in a first image frame and a previous coordinate pair of corresponding key points in a previous image frame is determined using a processing system.

4. The method of claim 1 , wherein the first numerical vector has dimensions a×b×d×e, wherein a×b is the dimension of the grid, d is the number of bins in the corresponding histogram of each of the at least one passenger, and e is the number of passengers in the at least one passenger.

5. The method according to claim 1, wherein detecting abnormal passenger behavior further comprises: using a processing system to determine, for each respective cluster component of the plurality of cluster components of the mixture model, the posterior probability that a first numerical vector belongs to the respective cluster component; using a processing system to classify a first image frame as belonging to a first cluster component having the highest posterior probability among the plurality of cluster components of the mixture model.

6. The method according to claim 5, wherein detecting abnormal passenger behavior further comprises: using a processing system to determine a first posterior density based on the first numerical vector and the first cluster component among the plurality of cluster components of the mixture model.

7. The method according to claim 6, wherein detecting abnormal passenger behavior further comprises: using a processing system to compare the first posterior density with a predetermined threshold; and detecting abnormal passenger behavior in the first image frame in response to the first posterior density being less than the predetermined threshold.

8. The method according to claim 6, wherein detecting abnormal passenger behavior further comprises: using a processing system to determine an average posterior density over a plurality of image frames, the plurality of image frames including the first image frame and at least one previous image frame; using a processing system to compare the average posterior density with a predetermined threshold; and detecting abnormal passenger behavior in the first image frame in response to the average posterior density being less than the predetermined threshold.

9. The method according to claim 1, wherein unlabeled training data is used to learn the plurality of cluster components, the unlabeled training data including a video corpus of at least one passenger riding in a vehicle.

10. The method according to claim 1, further comprises: using a transceiver to transmit a message to a remote server in response to detecting abnormal passenger behavior.

11. The method according to claim 1, further comprises: outputting an alert to the at least one passenger using a speaker or a display screen in response to detecting abnormal passenger behavior.

12. A system for detecting abnormal passenger behavior in a vehicle, the system comprises: an image sensor configured to generate and output an image frame of at least one passenger in a cockpit of the vehicle; a processing system operatively connected to the image sensor and including at least one processor, the processing system being configured to: receive a first image frame of at least one passenger in a cockpit of the vehicle from the image sensor; determine a first numerical vector representing the pose and motion of the at least one passenger in the first image frame; and detect abnormal passenger behavior in the first image frame using a mixture model having a plurality of cluster components representing normal passenger behavior based on the first numerical vector; the processing system is further configured to, when determining the first numerical vector: determine respective multiple key points for each of the at least one passenger, each key point including a coordinate pair corresponding to the position of a respective joint or body part of the at least one passenger within the first image frame; determining, for each respective key point of a respective plurality of key points for each of the at least one passenger, an optical flow vector indicating motion of the respective key point in a first image frame relative to at least one previous image frame; categorizing each respective key point of the respective plurality of key points for each of the at least one passenger into a respective cell of a two-dimensional grid of cells based on the coordinate pairs of the respective key points, wherein each respective cell of the grid corresponds to a range of coordinates within the first image frame; classifying each corresponding keypoint classified into each corresponding cell of the grid into a corresponding bin of a corresponding histogram of each of the at least one passenger based on an optical flow angle of an optical flow vector of the corresponding keypoint, wherein each bin of the corresponding histogram of each of the at least one passenger corresponds to an optical flow angle range; determining a value for each bin of the corresponding histogram for each of the at least one passenger as a sum of optical flow values ​​of the optical flow vectors for each keypoint classified into the corresponding bin; as well as A first numerical vector is formed having a numerical value for each bin of the corresponding histogram for each of the at least one passenger.

13. The system of claim 12, wherein the processing system is further configured to, when detecting abnormal passenger behavior: determine, for each respective cluster component of the plurality of cluster components of the mixture model, a posterior probability that the first numerical vector belongs to the respective cluster component; classifying a first image frame as a first cluster component having a highest posterior probability among the plurality of cluster components belonging to the mixture model; as well as A first posterior density is determined based on a first numerical vector and a first cluster component of the plurality of cluster components of the mixture model.

14. The system of claim 13, wherein the processing system is further configured to, when detecting abnormal passenger behavior: determine an average posterior density over a plurality of image frames, the plurality of image frames comprising a first image frame and at least one previous image frame; comparing the average posterior density to a predetermined threshold; and Abnormal passenger behavior in the first image frame is detected in response to the average posterior density being less than a predetermined threshold.

15. The system according to claim 12, further comprising: include: a transceiver operably connected to the processing system, Wherein the processing system is configured to operate the transceiver to transmit a message to a remote server in response to detecting abnormal passenger behavior.

Images