Behavior prediction device, behavior prediction method, behavior prediction program, sudden appearance prediction device, and driving support system

Abstract

To improve the accuracy of predictions in predicting human behavior. [Solution] An action prediction device that predicts the future actions of a person under surveillance based on the person's past actions, and comprises a controller. The controller acquires time-series images of the person under surveillance, acquires emotional information relating to the person under surveillance's emotions, and predicts the person under surveillance's actions based on the time-series images and the emotional information.

Description

Technical Field

[0006] , , ,

[0005] , ,

[0001] The present invention relates to an action prediction device, an action prediction method, an action prediction program, a jump-out prediction device, and a driving support system.

Background Art

[0002] Conventionally, an action prediction technique has been proposed for predicting a pedestrian's jump-out onto a road or the like based on the posture and movement of the pedestrian as a monitoring target. For example, a technique is known in which the pedestrian shape detected from image data is compared with pedestrian shape information related to the precursor behavior of a pedestrian having the possibility of jumping out, and the jump-out of the pedestrian is predicted (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, although a person's movement is generally estimated based on the immediate previous movement (motion) of the person such as the pedestrian shape as described above, a person's movement is also affected by the surrounding environment, for example, the weather, etc., in addition to the immediate previous movement (motion) of the person. Therefore, there is a concern that the prediction accuracy may decrease if such points are not considered.

[0005] In view of the above problems, an object of the present invention is to provide a technique capable of improving the prediction accuracy in predicting a person's action.

Means for Solving the Problems

[0006] An exemplary behavior prediction device of the present invention is a behavior prediction device that predicts the future behavior of a person under surveillance based on the past behavior of the person under surveillance, and comprises a controller. The controller acquires time-series images of the person under surveillance, acquires emotional information relating to the emotions of the person under surveillance, and predicts the behavior of the person under surveillance based on the time-series images and the emotional information. [Effects of the Invention]

[0007] According to the present invention, the prediction result of the behavior prediction of the monitored person is corrected according to the type of emotion of the monitored person, thereby obtaining a prediction result corrected based on emotion. In other words, the behavior prediction of the monitored person can be made while taking into account the influence of the monitored person's emotions. Therefore, it becomes possible to improve the prediction accuracy in predicting human behavior, and to provide more appropriate driving assistance such as collision avoidance. [Brief explanation of the drawing]

[0008] [Figure 1] Overall configuration diagram of the driver assistance system of this embodiment [Figure 2] Block diagram showing the configuration of the driver assistance system in Figure 1. [Figure 3] This figure shows a two-dimensional model (psychological plane), which is an example of an emotion estimation model. [Figure 4] Figure showing an example of a correction parameter table. [Figure 5] Conceptual diagram showing predictions and corrections, etc. [Figure 6] Table structure diagram showing the confidence data table for calculating confidence level and position variation range. [Figure 7] An explanatory diagram showing an example of the prediction results of human behavior in a behavior prediction device that does not take emotions into account. [Figure 8] Schematic diagram showing the situation around the vehicle in Figure 7, viewed from above. [Figure 9] This diagram illustrates an example of a prediction result for human behavior in the behavior prediction device that takes emotion (excitement) into consideration according to this embodiment. [Figure 10]Figure 1 shows an example of a processing flow for the driver assistance system. [Figure 11] Conceptual diagram illustrating the human behavior prediction process using a behavior prediction device for modified organisms. [Figure 12] Conceptual diagram illustrating the learning process of a behavior prediction device (behavior prediction AI model) using a learning device. [Modes for carrying out the invention]

[0009] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the embodiments described below.

[0010] <1. Driver assistance systems> Figure 1 is an overall configuration diagram of the driver assistance system 1 of this embodiment. In this embodiment, the driver assistance system 1 is a system that provides driving assistance to the driver D1 of vehicle Ca1, and is a system that provides driving assistance to avoid collisions between vehicle Ca1 and the monitored person P1. In detail, vehicle Ca1 is an automobile, but it may be other types of vehicles such as motorcycles. In detail, the monitored person P1 is a pedestrian, but it may be a person driving a vehicle such as a bicycle.

[0011] In this embodiment, the driver assistance system 1 is mounted on the vehicle Ca1 (the vehicle itself). The driver assistance system 1 may also be included in, for example, a drive recorder or navigation system installed in the vehicle Ca1. As shown in Figure 1, the driver assistance system 1 comprises a camera 2, a behavior prediction device 10, and a driver assistance device 20.

[0012] Camera 2 is, for example, a front-facing camera installed at the front end of vehicle Ca1, which photographs the area in front of vehicle Ca1. If vehicle Ca1 is equipped with a drive recorder, the drive recorder's front-facing camera may be used as camera 2.

[0013] In Fig. 1, the camera 2 is depicted as being provided in the vehicle Ca1. However, the camera 2 may be, for example, a camera included in infrastructure facilities installed on a road or the like. In that case, the captured image of the camera 2 is transmitted to the driving support system 1 (behavior prediction device 10) mounted on the vehicle Ca1 by wireless communication. The camera 2 may be provided in at least one of the vehicle Ca1 and the infrastructure facilities installed on a road or the like, and may be plural. The camera 2 is a device for acquiring (capturing) appearance information (captured image) related to the movement of the monitored person P1 when predicting the behavior of the monitored person P1.

[0014] The behavior prediction device 10 is a device that predicts the behavior of the monitored person P1 based on the appearance information related to the movement of the monitored person P1. In the present embodiment, the monitored person P1 whose behavior is to be predicted is a pedestrian existing in front of the vehicle Ca1 (own vehicle) on which the behavior prediction device 10 is mounted. The dangerous behavior of the monitored person P1 is jumping out into the passing area of the vehicle Ca1. That is, in the present embodiment, the behavior prediction device 10 is a jumping-out prediction device that predicts the jumping out of a pedestrian. Note that the object of behavior prediction may include, for example, a person driving a vehicle such as a bicycle instead of or in addition to a pedestrian.

[0015] Also, in predicting the behavior of the monitored person P1, the behavior prediction device 10 adds the influence of the emotion of the monitored person P1 to calculate the prediction result. For this reason, the behavior prediction device 10 acquires emotion information related to the emotion of the monitored person P1.

[0016] The emotion information of the monitored person P1 is the appearance information (captured image) of the monitored person P1 captured by the camera 2. The behavior prediction device 10 estimates the emotion based on this appearance information. Details of the behavior prediction device 10 will be described later.

[0017] Furthermore, the emotions of the monitored person P1 can also be estimated based on the monitored person P1's biometric information (biometric signals). In this case, the behavior prediction device 10 acquires the monitored person P1's biometric information detected by the sensing device of the terminal device 60 worn by the monitored person P1 via a communication network N such as a mobile communication network, and estimates the monitored person P1's emotions based on the acquired biometric information.

[0018] The driver assistance device 20 is a device that provides driver assistance to avoid a collision between the vehicle Ca1 and the monitored person P1. More specifically, the driver assistance device 20 of this embodiment provides driver assistance to avoid a collision between the vehicle Ca1 and the monitored person P1 based on the prediction result data of the monitored person P1's behavior prediction obtained from the behavior prediction device 10. In other words, the driver assistance device 20 provides driver assistance to avoid a collision between the vehicle Ca1 and the monitored person P1 based on the prediction result data of the behavior prediction that takes into account the influence of the monitored person P1's emotions. Details of the driver assistance device 20 will be described later.

[0019] Figure 2 is a block diagram showing the configuration of the driver assistance system 1 in Figure 1. Figure 2 shows the components necessary to explain the features of this embodiment, and general components are omitted. Figure 2 also shows the devices mounted on the vehicle Ca1 that operate in conjunction with the driver assistance system 1.

[0020] Vehicle Ca1 is equipped with a vehicle control device 31, an actuator device 32, and a notification device 33 (see Figure 2).

[0021] The vehicle control device 31 is, for example, an ECU (Electronic Control Unit) for vehicle control and is mounted on the vehicle Ca1. The vehicle control device 31 performs various vehicle controls, such as controlling the direction and speed of the vehicle Ca1, based on driving operations performed by the driver D1 of the vehicle Ca1 and information from various connected sensors (not shown), such as a vehicle speed sensor, air-fuel ratio sensor, steering angle sensor, etc.

[0022] Furthermore, the vehicle control device 31 stores data used in vehicle control processing, such as control processing programs, processing coefficient data, and temporary storage data during processing. The vehicle control device 31 also stores a collision avoidance processing information data table that associates collision prediction information received from the driver assistance device 20 with control signals from the vehicle Ca1 and notification information to the driver D1.

[0023] The actuator device 32 consists of various drive components, such as motors, that realize various operations of the vehicle Ca1, and its operation is driven and controlled by the vehicle control device 31. Specifically, the actuator device 32 includes, for example, an engine and motor that generate driving force in the vehicle Ca1, a steering actuator that drives the steering of the vehicle Ca1, a brake actuator that drives the brakes of the vehicle Ca1, and so on.

[0024] The notification device 33 notifies the driver D1 inside the vehicle of notification information from the vehicle control device 31 or the driver assistance device 20 by visual, auditory, or other means. Specifically, the notification device 33 consists of, for example, a display device such as a liquid crystal display or HUD (Head-Up Display) that conveys notification information to the driver D1 in the form of text, images, or videos; a buzzer and speaker that conveys notification information in the form of warning sounds or voices; a vibration device that is built into the seats inside the vehicle and conveys notification information by vibration; and a light-emitting device that conveys notification information by lighting up, flashing, or other means.

[0025] <2. Behavior Prediction Device> The behavior prediction device 10 comprises a communication unit 11, a storage unit 12, and a controller 13.

[0026] The communication unit 11 is an interface for communicating data with other devices (such as the driver assistance device 20 and the terminal device 60) via a communication network. The communication unit 11 includes communication devices for wired and wireless communication with other devices. The wireless communication device consists of, for example, a transceiver for a 5G (fifth-generation mobile communication system) mobile telephone network.

[0027] The storage unit 12 is configured to include volatile memory and non-volatile memory, and stores various information necessary for content playback processing. The volatile memory is composed of, for example, RAM (Random Access Memory). The non-volatile memory is composed of, for example, ROM (Read Only Memory), flash memory, or a hard disk drive. Programs and data that can be read by the controller 13 are stored in the non-volatile memory. At least a portion of the programs and data stored in the non-volatile memory may be obtained from other computer devices connected by wired or wireless connections, or from portable recording media.

[0028] The memory unit 12 stores the behavior prediction program 121, the emotion estimation model 122, and the correction parameter table 123. The contents of these programs, data tables, etc., stored in the memory unit 12 are described separately. In addition, the memory unit 12 stores multiple data tables, etc., for various processing.

[0029] The controller 13 consists of a processor that performs calculations and other processing, and controls various operations in the behavior prediction device 10. The processor includes, for example, a CPU (Central Processing Unit). The controller 13 executes the behavior prediction program 121 stored in the memory unit 12 and performs prediction processing of human behavior. The behavior prediction program 121 includes various programs that realize various functions of the behavior prediction device 10.

[0030] The controller 13 includes, as its functions, an acquisition unit 131, an action prediction unit 132, an emotion estimation unit 133, a prediction result correction unit 134, and a provision unit 135. In this embodiment, the functions of the controller 13 are realized by the processor executing calculation processing according to the action prediction program 121 stored in the memory unit 12.

[0031] The acquisition unit 131 acquires images captured by the camera 2 via the communication unit 11. In a configuration where images are acquired from multiple cameras 2, it becomes possible to predict human behavior with high accuracy. The acquisition unit 131 acquires captured images from the camera 2 at regular intervals. In other words, the acquisition unit 131 acquires video footage (time-series still images) captured by the camera 2. That is, the acquisition unit 131 acquires (captures) visual information (captured images) related to the movement of a person (surveillance subject P1). The captured images (image data) from the camera 2 are stored in the storage unit 12.

[0032] Furthermore, the acquisition unit 131 performs analysis processing on the captured image (video) from camera 2 to detect people (surveillance target P1) and roadway areas (vehicle Ca1 traffic areas) in the captured image. The method for detecting surveillance target P1 and roadway areas is not particularly limited, and surveillance target P1 and roadway areas may be detected using known appropriate image analysis techniques such as pattern matching technology. For example, surveillance target P1 and roadway areas may be detected from the captured image (video) from camera 2 using object detection AI (Artificial Intelligence).

[0033] When the driver assistance system 1 estimates the emotions of the monitored person P1 based on the monitored person P1's biometric information, the acquisition unit 131 acquires the monitored person P1's biometric information detected by the sensing device of the terminal device 60 worn by the monitored person P1 via the communication unit 11. The monitored person P1's biometric information includes, for example, electroencephalogram (EEG) information (EEG signals) and heart rate information (heart rate signals). The EEG information and heart rate information are stored in the storage unit 12. The acquisition unit 131 acquires the captured image, EEG information, and heart rate information at substantially the same time and stores them as a single dataset in a single data record in the data table of the storage unit 12. This dataset is used to estimate the emotions of the monitored person P1.

[0034] The behavior prediction unit 132 tracks specific points (e.g., joints) of the monitored person P1 detected on the captured image (video) acquired by the acquisition unit 131 using known skeletal estimation techniques. Then, based on the monitored person P1's past movement state (movement trajectory), the behavior prediction unit 132 predicts the future movement trajectory of the monitored person P1 (a predetermined prediction period of a set length: a period during which sudden movement would be dangerous (for example, set by the design and development team based on experiments, etc.)) using known movement prediction methods such as differential approximation. It is also possible to predict the movement trajectory of the monitored person P1 based on the captured image using a movement prediction AI.

[0035] Subsequently, the behavior prediction unit 132 predicts a sudden departure from the road based on the relationship between the predicted movement trajectory of the monitored person P1 and the detected road area. Specifically, the behavior prediction unit 132 predicts a sudden departure if the predicted movement trajectory is located within the road area within the prediction period. In this embodiment, the point where the person deviates most far into the road (the point with the longest distance from the boundary between the sidewalk and the road in the predicted movement trajectory) is used as the predicted sudden departure point, but the predicted position of the monitored person P1 at the end of the prediction period may also be used as the predicted sudden departure point. In other words, the behavior prediction unit 132 predicts the monitored person P1's sudden departure into the road from the video (time-series image information) acquired over a predetermined time-length prediction data collection period.

[0036] Furthermore, the prediction of people suddenly running into the road may be performed, for example, using an AI model for predicting such occurrences. This AI model predicts a person's sudden running behavior based on visual information related to their movement. This AI model for predicting sudden running is generated by training the model using a large amount of video (time-series image information) accumulated from various scenes in which sudden running has actually occurred as supervised training data.

[0037] The behavior prediction unit 132 outputs (calculates) the prediction result of the monitored person P1 running out as a probability of running out. For example, the behavior prediction unit 132 calculates the probability of running out according to the longest distance from the road area boundary to the predicted movement position of the monitored person P1 during the prediction period (distance from the road area boundary to the predicted running out point), that is, the predicted running distance (longest distance) of the monitored person P1 into the road. The probability of running out increases as the running distance increases. Such probabilities can be calculated based on a predetermined calculation formula or data table (determined by the designer / developer based on experiments, etc.) that uses the running distance as a parameter value. Alternatively, it is also possible to calculate the probability of running out using a statistical model such as a Gaussian distribution model, or to use the prediction probability output by an AI model as the probability of running out.

[0038] Furthermore, the behavior prediction unit 132 estimates the reliability of the predicted position and the range of positional variation of the monitored person P1 based on the monitored person P1's movement speed, acceleration, and directional variation status. This data can be calculated using a data table that stores data on reliability and range of positional variation, with movement speed, acceleration, and directional variation status as parameters (axis values ​​in the data table), or using a calculation formula appropriately set based on these parameters.

[0039] The emotion estimation unit 133 estimates the emotions of the monitored person P1 based on the captured images of the person acquired by the acquisition unit 131. The estimation of a person's emotions is not particularly limited, and known emotion estimation methods may be used. Regarding the estimation of a person's emotions, for example, a known emotion estimation model 122 is stored in the memory unit 12.

[0040] Emotions can be estimated based on electroencephalogram (EEG) and heart rate information. Specifically, emotions are estimated based on two emotional indicators that represent the mental and physical state related to emotions. One of the emotional indicators is the level of central nervous system arousal (hereinafter referred to as arousal), and its value can be calculated from the "beta waves / alpha waves of the electroencephalogram." The other emotional indicator is the level of autonomic nervous system activity (hereinafter referred to as activity), and its value can be calculated from the "standard deviation of the heart rate LF (Low Frequency) component (low-frequency component of the heart rate waveform signal)."

[0041] The emotion estimation model 122, which estimates emotions based on arousal and activity levels, consists of a model for emotion estimation based on arousal and activity levels (calculation formulas and conversion data tables). The emotion estimation model consists of, for example, a two-dimensional model that estimates emotions using arousal and activity levels as parameters. This two-dimensional model is created based on medical evidence (papers, etc.) that shows the relationship between arousal and activity levels and emotions.

[0042] Figure 3 shows a two-dimensional model (psychological plane) which is an example of an emotion estimation model. In the psychological plane shown in Figure 3, the vertical axis represents "arousal level (aroused-unaroused)" and the horizontal axis represents "autonomic nervous system activity level (sympathetic nervous system activity (strong emotion)-parasympathetic nervous system activity (weak emotion))." In this psychological plane, each of the four quadrants separated by the vertical and horizontal axes is assigned a corresponding emotion type. The distance from each axis indicates the intensity of the corresponding emotion.

[0043] Furthermore, emotions can be estimated from the coordinates obtained by plotting two types of emotional indicator values ​​(arousal and activity levels) obtained based on biosignals onto a psychological plane. Specifically, emotions and their intensity can be estimated based on which quadrant the plotted coordinates lie in on the psychological plane, their position within that quadrant, and their distance from the origin.

[0044] It should be noted that emotion estimation models are not limited to the psychological plane model of arousal and activity levels shown in Figure 3; other emotion estimation models, such as Russell's circular emotion model, can also be used.

[0045] In this embodiment, the emotion estimation model 122 includes an AI model that estimates emotions based on a captured image (face image) of the person being monitored P1. This AI model can be generated as follows.

[0046] The learning data generation environment involves attaching biosensors (EEG sensor, heart rate sensor) to subjects for learning data generation, and setting up a camera to capture images of the subjects' faces. Then, biosensor data and facial images of the subjects are acquired at a sampling time length suitable for emotion estimation (set based on experiments, etc.). Subsequently, emotions are estimated using the acquired biosensor data (EEG, heart rate) with the method described above (emotion estimation model). This estimated emotion for the subject is used as the ground truth data (output data), and the corresponding facial image (facial image from the same sampling) is used as input data to generate learning data. To ensure that the AI ​​model can learn sufficiently, a large amount of such learning data is generated in various surrounding environments and with many subjects, and a learning dataset is created from this learning data. Then, the pre-training emotion estimation AI model is trained using the created learning dataset, and a trained emotion estimation AI model is generated.

[0047] The emotion estimation unit 133 estimates the emotion (emotion type) of the monitored person P1 by applying (inputting) the captured image of the monitored person P1 acquired by the acquisition unit 131 (preferably with a sampling time length suitable for emotion estimation) to the emotion estimation model 122 created in this manner.

[0048] The prediction result correction unit 134 corrects the prediction result data (probability of jumping out) calculated by the behavior prediction unit 132 based on the emotion type estimated by the emotion estimation unit 133. More specifically, the prediction result correction unit 134 uses the correction parameter table 123 stored in the memory unit 12 to correct the prediction result data of the behavior prediction of the monitored person P1 using correction parameters corresponding to the emotion type estimated by the emotion estimation unit 133.

[0049] Figure 4 shows an example of the correction parameter table 123. As shown in Figure 4, the items in the correction parameter table 123 include "Correction Parameter ID", "Emotion Type", "Confidence Correction Parameter", "Jump Location Correction Parameter", and "Jump Probability Correction Parameter".

[0050] The "Correction Parameter ID" is identification information used to identify a dataset of correction parameters related to human emotions. The Correction Parameter ID data is also the primary key of the data record in the Correction Parameter Table 123. In other words, in the Correction Parameter Table 123, a data record is created for each Correction Parameter ID data, and the data for each item associated with the data ID is stored in that data record.

[0051] "Emotion type" is a data item related to the type of emotion a person is feeling. For example, emotion types include data such as "calm," "excited," "apathetic," and "nervous."

[0052] The "reliability correction parameter," "jumping position correction parameter," and "jumping probability correction parameter" are data items of correction data that correct the reliability of the prediction results calculated by the behavior prediction unit 132.

[0053] In the example of the correction parameter table 123 shown in Figure 4, when the emotion (type) is "calm," each parameter value is set to the baseline (1.0). When the emotion type is "excited," the confidence correction parameter is 0.8, which is a correction value that reduces the confidence of the prediction result (predicted position), the jump position correction parameter is 1.3, which is a correction value that widens the predicted range of the predicted jump position, and the jump probability correction parameter is 1.2, which is a correction value that increases the probability of jumping.

[0054] The prediction result correction unit 134 performs correction processing on the prediction result data (confidence data, sudden appearance location data, and sudden appearance probability data) calculated by the behavior prediction unit 132, using correction parameters corresponding to the emotion type estimated by the emotion estimation unit 133. With this configuration, the risk of a person suddenly appearing can be communicated quickly and appropriately to the driver D1 of the vehicle Ca1, enabling appropriate vehicle control. This makes it possible to more effectively avoid collisions between the vehicle Ca1 and people.

[0055] Next, specific examples of the prediction content of the behavior prediction unit 132 and the correction content of the prediction result correction unit 134 will be explained using Figures 5 and 6. Figure 5 is a conceptual diagram showing the prediction content and correction content, etc. Figure 6 is a table structure diagram showing the confidence data table for calculating the confidence level and position variation range.

[0056] For the sake of clarity, the explanation will use an example where the parameters determining reliability are the movement speed level of the monitored person P1 (stratified by appropriate speed ranges) and weather. However, if other parameters are used, this can be accommodated by changing the items and dimensions of the data table shown in Figure 6. Furthermore, the movement speed level of the monitored person P1 (average movement speed over the prediction period) is assumed to be V1, the surrounding weather is sunny, and the emotional state of the monitored person P1 is assumed to be excited.

[0057] The behavior prediction unit 132 estimates the predicted point of departure EP (referred to as the predicted point of departure, including situations where the person does not step into the roadway) where the monitored person P1 will be located at the end of the prediction period (prediction time), based on camera images of the monitored person P1 during a prediction data collection period of a predetermined duration suitable for prediction. The behavior prediction unit 132 also calculates the average movement speed of the monitored person P1 during the prediction period based on the position and time of each point in the movement trajectory, and calculates the movement speed level V(V1) by stratifying the calculated movement speed v(v1). The values ​​in parentheses are examples of the actual values.

[0058] Furthermore, the behavior prediction unit 132 calculates the probability OP (OP1) of the monitored person P1 running into the roadway based on the positional relationship between the predicted running point EP and the roadway area, as described above. The running probability OP can be calculated using a calculation formula or data table appropriately created by the design and development team based on experiments, etc. The behavior prediction unit 132 compares the calculated movement speed level V (V1) with the weather (sunny: estimated from illuminance sensors, temperature, wiper operation status, or internet information, etc.) using the confidence data table shown in Figure 6, and extracts the confidence TFV (TFV1) and position variation range PFV (PFV1).

[0059] The prediction result correction unit 134 compares the emotion (excitement) estimated by the emotion estimation unit 133 with the correction parameter table 123 shown in Figure 4 and extracts the confidence correction parameter value (0.8), the jumping position correction parameter value (1.3), and the jumping probability correction parameter value (1.2). Then, the prediction result correction unit 134 corrects the confidence TFV (TFV1), position variation range PFV (PFV1), and jumping probability OP (OP1) predicted by the behavior prediction unit 132 using these correction values.

[0060] Specifically, the corrected confidence CTFV of the predicted jump point EP is the value obtained by multiplying the confidence TFV (TFV1) by the confidence correction parameter value (0.8) (TFV1 × 0.8). The corrected variation range ΔCP of the predicted jump point EP is the value obtained by multiplying the position variation range PFV (PFV1) by the jump position correction parameter value (1.3) (PFV1 × 1.3). The corrected jump probability COP is the value obtained by multiplying the jump probability OP (OP1) by the jump probability correction parameter value (1.2) (OP1 × 1.2).

[0061] Returning to Figure 2, the supply unit 135 provides the driver assistance device 20 with the corrected prediction result data (corrected confidence data, corrected sudden appearance position data, and corrected sudden appearance probability data) corrected by the prediction result correction unit 134. This enables the driver assistance device 20 to perform driver assistance that avoids collisions between the vehicle Ca1 and the monitored person P1, based on the corrected prediction result data of behavioral prediction that takes into account the influence of the monitored person P1's emotions.

[0062] According to the above configuration, the prediction result of the behavior prediction of the monitored person P1 is corrected according to the emotional type of the monitored person P1, thereby obtaining a prediction result corrected based on emotion. In other words, the behavior prediction of the monitored person P1 can be made while taking into account the influence of the monitored person P1's emotions. Therefore, it becomes possible to improve the prediction accuracy in predicting human behavior, and to provide more appropriate driving assistance such as collision avoidance.

[0063] <3. Driving assistance systems> The driver assistance device 20 comprises a communication unit 21, a storage unit 22, and a controller 23.

[0064] The communication unit 21 is an interface for data communication between the behavior prediction device 10, the vehicle control device 31, the actuator device 32, and the notification device 33, and is, for example, a CAN (Controller Area Network) interface.

[0065] The memory unit 22, like the memory unit 12, is composed of various types of memory and stores data used by the controller 23 for processing, such as the driving support program 221, processing coefficient data, and temporary storage data during processing. The memory unit 22 also stores the corrected prediction result data received from the behavior prediction device 10, and multiple data tables for various processing.

[0066] The controller 23 consists of a processor that performs calculations and other processing, and controls various operations in the driver assistance device 20. The controller 23 executes the driver assistance program 221 stored in the memory unit 22 and performs support processing related to the driving of the vehicle Ca1. The driver assistance program 221 includes various programs that realize various functions of the driver assistance device 20.

[0067] The controller 23 provides driving assistance to avoid a collision between the vehicle Ca1 and the monitored person P1, based on the corrected prediction result data of the monitored person P1's behavior prediction obtained from the behavior prediction device 10. In other words, the controller 23 provides driving assistance to avoid a collision between the vehicle Ca1 and the monitored person P1, based on the prediction result of the monitored person P1's behavior prediction, which takes into account the influence of the monitored person P1's emotions. Driving assistance includes, for example, notification processing to inform the driver D1 of the vehicle Ca1 that there is a risk of the monitored person P1 suddenly running out, and avoidance processing to have the vehicle Ca1 automatically perform evasive maneuvers to avoid a collision with the monitored person P1.

[0068] Specifically, the notification process involves causing the notification device 33 installed in the vehicle Ca1 to perform a notification action to alert the driver of the danger of someone suddenly appearing. The notification action may include, for example, a display on a display device, sound or voice output from a buzzer or speaker, vibration from a vibration device, or light emission from a light-emitting device.

[0069] Specifically, for example, the notification process performs the following: if the corrected sudden appearance probability COP1 exceeds the emergency threshold BV1, which has the highest value, an emergency audio warning at maximum volume and an emergency display warning with large brightness fluctuations are issued; if the corrected sudden appearance probability COP1 exceeds the alarm threshold BV2, which has the second highest value (below or equal to the emergency threshold BV1), an audio warning and a display warning are issued; if the corrected sudden appearance probability COP1 exceeds the caution threshold BV3, which has the third highest value (below or equal to the alarm threshold BV2), only a display warning (caution display) is issued; and if the corrected sudden appearance probability COP1 is below the caution threshold BV3, no notification is issued.

[0070] Furthermore, the emotional state (excitement) of the monitored person P1, and the corrected confidence level CTFV of this information regarding the monitored person P1's potential sudden appearance, may be communicated using numerical values, characters or other graphics, or the color of the display screen. The predicted sudden appearance point EP and the range (spread) of its error (corrected variation range ΔCP) may also be displayed on the display device. In addition, if the corrected sudden appearance probability OP1 increases rapidly (above a threshold), the notification device 33 may inform the driver D1 of the sudden increase in the sudden appearance probability.

[0071] Figure 7 is an explanatory diagram showing an example of the prediction results of human behavior in a behavior prediction device that does not take emotions into consideration. Figure 8 is a schematic diagram viewed from above showing the situation around vehicle Ca1 in Figure 7. Figure 7 schematically depicts the state in which the predicted position data of the monitored person P1 suddenly appearing is displayed on the windshield Fw of vehicle Ca1 by a HUD as a driving assistance.

[0072] Based on the prediction result of the monitored person P1 suddenly appearing onto the road (the vehicle Ca1's travel area Rp), the controller 23 displays a monitored person icon g1 near the position where the monitored person P1 is visible on the windshield Fw, an arrow-shaped direction icon g2 indicating the predicted direction of movement of the monitored person P1, and a predicted position icon g3 indicating the predicted position of the monitored person P1 at the end of the prediction period. The predicted position icon g3 is displayed as a range including the error in the predicted position (closed area curve: in this case an ellipse (a circle when viewed from above)).

[0073] The monitored target icon g1 is an image that suggests the presence of the monitored person P1 and is displayed near the position where the monitored person P1 is visible on the windshield Fw to alert the driver D1 of vehicle Ca1. The direction icon g2 is an image that shows the predicted direction of movement (predicted trajectory) of the monitored person P1, and a shape that indicates direction, such as an arrow, is used. The predicted position icon g3 is an image that suggests the predicted point of departure EP and is displayed as a range that includes the error of the predicted position (closed region curve: in this case an ellipse (a circle when viewed from above, and the predicted point of departure EP is the center of the circle)). The confidence level TFV and the probability of departure OP may also be displayed on the windshield Fw of vehicle Ca1 by the HUD, either as their values ​​(TFV1 and OP1) or as shapes that suggest their values.

[0074] Figure 9 is an explanatory diagram showing an example of the prediction results of human behavior in the behavior prediction device 10 that takes emotion (excitement) into consideration in this embodiment. Similar to Figure 7, Figure 9 schematically depicts the state in which the sudden appearance position data, which is one of the prediction result data, is displayed on the windshield Fw of the vehicle Ca1 by the HUD.

[0075] When the controller 23 predicts that the monitored person P1 will suddenly step out onto the roadway (the vehicle Ca1's travel area Rp) (when the probability of stepping out OP exceeds a predetermined threshold), it displays the monitored person icon g1, the direction icon g2, and the corrected predicted position icon g4 on the windshield Fw, aligned with the viewing position of the monitored person P1 and other external scenery on the windshield Fw, that is, it displays them using AR (Augmented Reality) technology.

[0076] The display position (center of the circle) of the corrected predicted position icon g4 is the predicted departure point EP, as explained using Figure 5, and its size (radius) is the position variation range PFV1 × 1.3. When emotions are not considered, the display position of the corrected predicted position icon g3 is the predicted departure point EP, and its size is the position variation range PFV1. Note that if the predicted departure point EP is also corrected considering emotions (for example, in the case of an unstable field emotion (e.g., excitement), it is corrected closer to the center of the road for safety reasons), the display position of the corrected predicted position icon g3 will be slightly shifted from the predicted departure point EP (e.g., closer to the center of the road). In that case, the direction icon g2 will also move slightly (or change its shape and size) in accordance with the shift of the predicted departure point EP.

[0077] Furthermore, the range of predicted jumping positions is not limited to circles or ellipses, but is determined according to the emotion type, and its shape and size may differ for each emotion type. In addition, the corrected confidence score CTFV and the corrected jumping probability COP may be displayed on the windshield Fw of the vehicle Ca1 via a HUD, either as their values ​​(corrected confidence score CTFV1 and corrected jumping probability COP1) or as shapes suggesting those values.

[0078] Specifically, the avoidance process involves instructing the vehicle control device 31, which controls the movement of the vehicle Ca1, to perform an avoidance action to avoid a collision with the monitored person P1. The avoidance action may include, for example, deceleration or stopping using automatic braking, or changing the direction of travel using automatic steering.

[0079] The driver assistance device 20 then instructs the vehicle control device 31 to perform vehicle control based on the corrected probability of sudden appearance COP during the avoidance process, such as throttle valve control, brake control, and steering control. Specifically, for example, the device calculates the collision risk based on the corrected probability of sudden appearance COP, the distance between the vehicle Ca1 and the monitored person P1 (current position or predicted sudden appearance point EP), and the vehicle speed of the vehicle Ca1 (velocity component in the direction relative to the monitored person P1) (calculated based on an appropriate calculation formula or data table), and performs control such as braking according to the collision risk. In other words, when the monitored person P1 is excited, it is assumed that there is a higher possibility (higher risk) of taking sudden action (larger difference from the prediction) compared to when they are calm, so the vehicle control is geared towards safety by making the brake sensitivity easier to increase (making it easier to stop) and making it easier for the automatic brakes to engage.

[0080] According to the above configuration, the driver assistance device 20 provides driving assistance to avoid a collision between the vehicle Ca1 and the monitored person P1, based on the prediction of the monitored person P1's behavior, taking into account the influence of the monitored person P1's emotions. As a result, the driver assistance device 20 can provide appropriate driving assistance to the vehicle Ca1 based on a more accurate prediction of the monitored person P1's behavior. Therefore, it becomes possible to further improve the safety of the vehicle Ca1.

[0081] <4. Driving support processing flow (example of operation)> Figure 10 is a processing flow diagram showing an example of the driver assistance processing flow performed by the driver assistance system 1 in Figure 1. In this processing flow diagram, actions related to predicting the actions of a person (person being monitored P1) are realized by a computer program (action prediction program 121) executed by the controller 13 (computer constituting the controller 13) of the action prediction device 10, and actions related to driver assistance for the driver D1 of the vehicle Ca1 are realized by a computer program (driver assistance program 221) executed by the controller 23 (computer constituting the controller 23) of the driver assistance device 20.

[0082] The computer program that implements the behavior prediction method and driving support method according to this embodiment in a computer device is installed in a computer device such as the behavior prediction device 10 and the driving support device 20 to realize the various functions described above. Furthermore, such a computer program is provided to the computer device via a computer-readable non-volatile recording medium. For example, optical discs on which the computer program is recorded are distributed and sold, or computer programs stored on the hard disk of a server device are distributed and sold via a network environment. In addition, the computer program that implements the behavior prediction method and driving support method according to this embodiment in a computer device may consist of only one program or may consist of multiple programs.

[0083] The controller 13 of the behavior prediction device 10 executes the behavior prediction program 121 when the monitored person P1 is detected in the camera image while the driver assistance system 1 is operating. The controller 23 of the driver assistance device 20 executes the driver assistance program 221 when it receives the corrected prediction result data provided by the behavior prediction device 10.

[0084] In step S101, the controller 13 (acquisition unit 131) of the behavior prediction device 10 acquires the biological information (biometric signals) of the monitored person P1 from the terminal device 60 worn by the detected monitored person P1, and proceeds to step S102. More specifically, the controller 13 (acquisition unit 131) acquires, for example, electroencephalogram information (electroencephalogram signals) and heart rate information (heart rate signals) of the monitored person P1 via the communication unit 11.

[0085] Note that the process in step S101 is performed when applying a method that estimates the emotions of the monitored person P1 based on the biological signals of the monitored person P1, but is not performed (deleted from the program) when applying a method that estimates the emotions of the monitored person P1 based on the facial image of the monitored person P1, etc.

[0086] In step S102, the controller 13 (acquisition unit 131) acquires time-series image information (video: a series of still images taken over time) of the monitored person P1 from the camera images (video), and then proceeds to step S103. The time-series image information is acquired by collecting images (video) taken over a predetermined time period of predicted data collection.

[0087] In step S103, the controller 13 (behavior prediction unit 132) uses the time-series image information of the monitored person P1 acquired in step S102 to perform behavior prediction processing for the monitored person P1, and then proceeds to step S104. More specifically, the controller 13 (behavior prediction unit 132) calculates the following prediction result data for the behavior prediction of the monitored person P1: prediction confidence data (confidence TFV), sudden departure location data (predicted sudden departure point EP and fluctuation range ΔP), and sudden departure probability data (sudden departure probability OP).

[0088] In step S104, the controller 13 (emotion estimation unit 133) performs emotion estimation processing on the monitored person P1 using the biometric information (biosignals) of the monitored person P1 acquired in step S101, or the time-series image information of the monitored person P1 acquired in step S102, obtains the emotion type, and proceeds to step S105. More specifically, the controller 13 (emotion estimation unit 133) estimates (obtains) the emotion type of the monitored person P1 using the emotion estimation model 122 based on the captured image of the monitored person P1, or the electroencephalogram information and heart rate information.

[0089] In step S105, the controller 13 (prediction result correction unit 134) performs a correction process on the prediction result data (confidence data (confidence TFV), jump location data (predicted jump location EP and fluctuation range ΔP), and jump probability data (jump probability OP)) calculated in step S103, based on the emotion type obtained in step S104, and proceeds to step S106. More specifically, the controller 13 (prediction result correction unit 134) calculates corrected confidence data (corrected confidence CTFV), corrected jump location data (predicted jump location EP and corrected fluctuation range ΔCP), and corrected jump probability data (corrected jump probability COP) as corrected prediction results for the behavior prediction of the monitored person P1.

[0090] In step S106, the controller 13 (providing unit 135) provides (transmits) the corrected prediction result data (corrected reliability data (corrected reliability CTFV), corrected sudden appearance position data (predicted sudden appearance point EP and corrected variation range ΔCP), and corrected sudden appearance probability data (corrected sudden appearance probability COP)) that has been corrected in step S106 to the driving support device 20, and terminates processing.

[0091] When the controller 23 of the driver assistance device 20 receives the corrected prediction result data from the behavior prediction device 10, it executes the processes from steps S201 to S203.

[0092] In step S201, the controller 23 of the driver assistance device 20 determines the degree of danger (the possibility of vehicle Ca1 colliding with monitored person P1) based on corrected sudden appearance probability data (corrected sudden appearance probability COP), determines a driver assistance method according to the determination result, and proceeds to step S202. More specifically, for example, the controller 23 uses a data table that associates the corrected sudden appearance probability COP with the driver assistance method to determine a driver assistance method according to the corrected sudden appearance probability COP.

[0093] In step S202, the controller 23 performs notification processing via voice, display, etc., based on the driving assistance method determined in step S201, and then proceeds to step S203. More specifically, as shown in Figure 9, the controller 23 displays the corrected prediction result data of the monitored person P1 (corrected reliability data (corrected reliability CTFV), corrected sudden appearance position data (predicted sudden appearance point EP and corrected variation range ΔCP), and corrected sudden appearance probability data (corrected sudden appearance probability COP)) and data such as emotions on the windshield Fw of the vehicle Ca1 using shapes, characters, colors, etc., and also outputs this information as voice through a speaker in appropriate wording.

[0094] In step S203, the controller 23 performs avoidance processing by vehicle control based on the driving assistance method determined in step S201, and then terminates the process. More specifically, the controller 23 performs vehicle control such as activating the vehicle's alarm and emitting an alarm sound outside the vehicle (to the monitored person P1), automatically closing the throttle valve (stopping engine drive), and activating automatic brakes.

[0095] <5. Variation> The behavior prediction device 10 may be configured with a behavior prediction AI model that outputs a prediction result of a person's behavior, taking into account the influence of the person's emotions, based on the captured images (video (time-series images)) of the person (surveillance subject P1) as described with reference to Figure 2. In this modified example, the behavior prediction AI model is used to predict the behavior of surveillance subject P1 based on captured images (time-series images) of the surveillance subject P1, and further taking into account the emotions of the surveillance subject P1.

[0096] Figure 11 is a conceptual diagram illustrating the human behavior prediction process by the modified behavior prediction device 10. Figure 12 is a conceptual diagram illustrating the learning process of the behavior prediction device 10 (behavior prediction AI model) by the learning device 40. In this modified example, components similar to those shown in Figure 2 are denoted by the same reference numerals, and detailed explanations are omitted. The behavior prediction device 10 includes a trained behavior prediction AI model 124M that includes the functions of the behavior prediction unit 132, emotion estimation unit 133, and prediction result correction unit 134 described using Figure 2.

[0097] The behavior prediction device 10 inputs the captured images (time-series images) of a person (monitored person P1) acquired by the acquisition unit 131 as input data to the trained behavior prediction AI model 124M. The behavior prediction AI model 124M is a trained AI model that outputs prediction results of a person's behavior, taking into account the influence of the person's emotions, based on the input values ​​of the captured images of the person. The behavior prediction AI model 124M outputs corrected prediction result data (corrected jump position data), which is the prediction result that takes into account the influence of emotions, as an output value.

[0098] Furthermore, since the AI ​​model has the function to calculate probabilities for each class (in this example, each jump position) (usually the class (jump position) with the highest probability is output as the estimated result), this probability can also be used to calculate corrected confidence data or corrected jump probability data.

[0099] The learning device 40 is equipped with a pre-training (or training) behavior prediction AI model 124m. The learning device 40 performs training on the pre-training (or training) behavior prediction AI model 124m using a training dataset consisting of the number of training data required for training.

[0100] The input data for the training dataset consists of time-series images (videos) (first input data) (predetermined time-series data) (first input data) of people (trainees) located near roads over a predetermined time period, and the emotions of the people (trainees) near the roads (second input data). The emotions of the people (trainees) are estimated based on their images (facial expressions). This emotion estimation can be achieved by using an emotion estimation AI model trained on training data in which the emotions estimated based on the subject's biosignals (brainwaves, heart rate) at the time the image was acquired are used as input data.

[0101] Furthermore, the ground truth data in the training data is the result of the jump (subsequent destination) in the scene in question (the jump position (the destination at the predicted time)). In other words, the ground truth data can be described as the behavior under the required prediction conditions (what conditions the prediction data under is required (in this example, the condition is the state of jumping out onto the road to the maximum extent during the prediction period, and the data is the position)). Using the training dataset, which consists of the number of these training data required for training, the learning device 40 trains the behavior prediction AI model 124m.

[0102] Specifically, the learning device 40 uses a learning algorithm such as backpropagation to adjust parameters such as weights in the behavior prediction AI model 124m based on the input supervised learning dataset, thereby generating a trained behavior prediction AI model 124M for use in the behavior prediction device 10.

[0103] According to the above configuration, the behavior prediction AI model 124M estimates and outputs corrected prediction result data in response to an input image of a person who has suddenly stepped out into the street. This allows the functions of the behavior prediction unit 132, emotion estimation unit 133, and prediction result correction unit 134, as explained using Figure 2, to be implemented within the behavior prediction AI model 124M. Therefore, a model for estimating emotions from emotion index values ​​(such as a two-dimensional model (psychological plane)) becomes unnecessary, which has the advantage of simplifying the design of the behavior prediction device.

[0104] <6. Points to note> The various technical features disclosed as embodiments herein can be modified in various ways without departing from the spirit of the technical creation. That is, the above embodiments are illustrative in all respects and not restrictive. The technical scope of the present invention is indicated by the claims rather than by the above descriptions of embodiments, and includes all modifications that fall within the meaning and scope equivalent to the claims. Furthermore, the multiple embodiments shown herein may be combined as appropriate to the extent possible.

[0105] Furthermore, although the above embodiment explains that various functions are implemented in software through CPU arithmetic processing according to a program, at least some of these functions may be implemented by electrical hardware resources. These hardware resources may be implemented entirely or partially by, for example, ASICs (Application Specific Integrated Circuits) or FPGAs (Field Programmable Gate Arrays). Conversely, at least some of the functions implemented by hardware resources may be implemented in software.

[0106] Furthermore, the system may include a computer program that enables a processor (computer) to implement at least some of the functions of each of the driver assistance systems 1 (behavior prediction device 10, driver assistance device 20). Such a computer program can be stored and provided (sold, etc.) on a computer-readable non-volatile recording medium (for example, in addition to the non-volatile memory mentioned above, optical recording medium (e.g., optical disc), magneto-optical recording medium (e.g., magneto-optical disc), USB memory, or SD card, etc.), and can also be provided from a server device via a communication line such as the Internet, a method known as download. [Explanation of symbols]

[0107] 1. Driver assistance system 2 cameras 10. Behavior prediction device (sudden escape prediction device) 11 Communications Department 12 Storage section 13 Controllers 20 Driving support systems 21 Communications Department 22 Memory section 23 Controllers 31 Vehicle control system 32 Actuator device 33. Notification device 40 Learning device 60 Terminal devices 121 Behavior Prediction Program 122 Emotion Estimation Models 123 Correction Parameter Table 124M, 124m Behavior Prediction AI Model 131 Acquisition Department 132 Behavioral Prediction Unit 133 Emotion estimation part 134 Prediction Result Correction Unit 135 Provision Department 221 Driving Assistance Program Ca1 Vehicle D1 Driver P1 Persons under surveillance

Claims

1. A behavior prediction device that predicts the future behavior of a monitored person based on the monitored person's past behavior, comprising a controller, The aforementioned controller, A time-series image of the person being monitored is acquired, We obtain emotional information relating to the emotions of the person being monitored. Based on the aforementioned time-series images and the aforementioned emotional information, predict the behavior of the person being monitored. Behavior prediction device.

2. The aforementioned controller, The location of the specific point of the monitored person in each of the acquired time-series images is detected. Based on the movement of the detected specific point, predict the actions of the person being monitored. The behavior prediction device according to claim 1.

3. A behavior prediction method that predicts the future behavior of a person under surveillance based on the person's past behavior, A time-series image of the person being monitored is acquired, We obtain emotional information relating to the emotions of the person being monitored. The controller performs the following actions to predict the behavior of the monitored person based on the aforementioned time-series images and the aforementioned emotional information. Methods for predicting behavior.

4. A behavior prediction program that predicts the future behavior of a monitored person based on the monitored person's past behavior, A time-series image of the person being monitored is acquired, We obtain emotional information relating to the emotions of the person being monitored. The computer is instructed to perform a process that predicts the behavior of the person being monitored based on the aforementioned time-series images and the aforementioned emotional information. Behavior prediction program.

5. A behavior prediction device that predicts the future behavior of a monitored person based on the monitored person's past behavior, Controller and A behavioral prediction AI model, which is executed by the controller and predicts the behavior of the monitored person based on time-series images of the monitored person and input of the monitored person's emotions, Equipped with, The aforementioned behavior prediction AI model, During a predetermined time-series period, time-series images of the subject captured by a camera are used as the first input data. The emotion estimated based on the subject's biosignals during the prediction period is used as the second input data. The behavior of the subject under the required prediction conditions after the aforementioned prediction period is used as the ground truth data, and is trained using training data. Behavior prediction device.

6. A device for predicting when a person under surveillance will run into the roadway based on the person's past behavior, comprising a controller, The aforementioned controller, A time-series image of the person being monitored is acquired, We obtain emotional information relating to the emotions of the person being monitored. The location of the specific point of the monitored person in each of the acquired time-series images is detected. Based on the movement of the detected specific point, the data predicting the sudden appearance of the monitored person is used to predict the sudden appearance of the person. The predicted jump-out result data is corrected based on the aforementioned emotional information. Pop-out prediction device.

7. The aforementioned data predicting the sudden departure is, The predicted departure point and the error variation range of the said predicted departure point, The device for predicting sudden movement according to claim 6.

8. A driver assistance system that provides driving support to the driver of a vehicle, It comprises a behavior prediction device that predicts the actions of the person being monitored, and a driving assistance device that assists the driver of the vehicle, The behavior prediction device is A time-series image of the person being monitored is acquired, We obtain emotional information relating to the emotions of the person being monitored. The location of the specific point of the monitored person in each of the acquired time-series images is detected. Based on the movement of the detected specific point, the data predicting the sudden appearance of the monitored person is used to predict the sudden appearance of the person. Based on the aforementioned emotional information, the predicted jump-out result data is corrected. The aforementioned driving support device, Based on the corrected prediction result data, the system provides driving assistance to avoid collisions between the vehicle and the monitored person. Driver assistance system.

9. The corrected prediction result data includes the predicted departure point and the error variation range of the predicted departure point. The aforementioned driving assistance device displays a driving assistance image that suggests the predicted point of sudden departure and the error variation range. The driving assistance system according to claim 8.

10. The aforementioned driver assistance system includes a display device that displays an image on the windshield, The display device displays the driver assistance image in accordance with the viewing position of the exterior scenery on the windshield. The driving assistance system according to claim 9.

Images