Information processing device, information processing method, information processing program, emotion estimation device, and driving assistance system

Abstract

To improve estimation accuracy of a condition of a user without imposing troublesome work, etc., on the user.SOLUTION: An exemplary information processing device estimates a condition of a user by performing weighting processing on outputs of a plurality of AI models estimating the condition of the user. The information processing device detects a target event of calibration processing with respect to the weighting processing based on a peripheral environment of the user and adjusts a weight in the weighting processing in accordance with an output situation of each of the plurality of AI models when the target event is detected.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, an information processing program, an emotion estimation device, and a driving assistance system. [Background technology]

[0002] For example, AI (Artificial Intelligence) models have been developed that estimate human states such as emotions. It is desirable for such AI models to be able to accommodate individual differences among users, in other words, to be versatile.

[0003] One method for realizing a versatile AI model is to provide multiple types of AI models, use the estimation results from each AI model, and weight the resulting combination according to the user to produce the final estimation result. With this method, it is necessary to appropriately adjust the weights according to the user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2019-56970 Summary of the Invention [Problem to be solved by the invention]

[0005] One method for determining the weights is to request a calibration task from the user and calibrate the weights according to the estimation results when the user is performing the task. Another method for determining the weights is to request the user to evaluate the estimation results and calibrate the weights according to the evaluation (see, for example, Patent Document 1). Both methods have the problem of being cumbersome for the user. Another problem with the method of requesting a task is that it cannot be performed when the user is unable to perform the task (for example, when the user is driving).

[0006] In view of the above circumstances, an object of the present invention is to improve the accuracy of estimating a user's state without imposing cumbersome tasks on the user. [Means for solving the problem]

[0007] An exemplary information processing device of the present invention is an information processing device that estimates a user's state by performing weighting processing on the outputs of multiple AI models that estimate the user's state, detects a target event for calibration processing for the weighting processing based on the user's surrounding environment, and adjusts the weights in the weighting processing depending on the output status of each of the multiple AI models at the time the target event is detected. [Effects of the Invention]

[0008] According to the exemplary embodiment of the present invention, a target event is automatically detected based on the user's surrounding environment, and weights are adjusted according to the output status of each of the multiple AI models at the time the target event is detected. Therefore, the state estimation function is appropriately calibrated without requiring the user to perform tedious tasks, thereby improving the accuracy of estimating the user's state.

[0009] Furthermore, according to the exemplary present invention, target events are automatically detected based on the user's surrounding environment, and calibration does not require the user to perform a task, so weights can be adjusted even when the user is unable to perform a task (for example, when the user is a driver and is driving). [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a conceptual diagram illustrating processing by a driving assistance device; [Figure 2] Diagram showing an example of the configuration of a driving assistance system [Figure 3] An example of a target event table [Figure 4] Flowchart showing information processing of a driving assistance device [Figure 5]FIG. 10 is a diagram showing an example of a first change detection history database; [Figure 6] FIG. 10 is a diagram showing an example of a second change detection history database. [Figure 7] FIG. 10 is a diagram showing an example of a target event occurrence history database. [Figure 8] A diagram showing a first example of a weight adjustment database. [Figure 9] A diagram showing a second example of a weight adjustment database. [Figure 10] A diagram showing a third example of a weight adjustment database. [Figure 11] Conceptual diagram showing the AI ​​model learning process performed by a learning device DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[0012] <1. Conceptual configuration of driving assistance device> First, the driving support device will be described. Fig. 1 is a conceptual explanatory diagram showing the processing performed by the driving support device 10.

[0013] In the usage example of the driving assistance device 10 shown in FIG. 1, a user U1 (observed person) is the driver of a vehicle V1. The first AI model 121M1 and the second AI model 121M2 used in the processing by the driving assistance device 10 estimate the emotions of the driver (user U1). The first AI model 121M1 and the second AI model 121M2 are each separately trained using data of subjects with similar characteristics (age, gender, hobbies, tastes, personality, etc.) as training data, provided as trained models, and installed in the driving assistance device 10. The similar characteristics of the subjects in the training data of the first AI model 121M1 and the similar characteristics of the subjects in the training data of the second AI model 121M2 are different from each other. Details of the driving assistance device 10 will be described later.

[0014] The driving assistance device 10 can be realized by a computer device installed in the vehicle V1, or by a server connected to the vehicle V1 via a network. The server may be a physical server or a virtual server.

[0015] Vehicle V1 is equipped with on-board sensors, including a camera C, a microphone M, a brake sensor SN1 (not shown in FIG. 1), and a steering angle sensor SN2 (not shown in FIG. 1). Gaze data and facial orientation data of user U1 (driver) based on an image captured by camera C, and voice data of user U1 (driver) acquired by microphone M, are input to first AI model 121M1 and second AI model 121M2. Because the input data for first AI model 121M1 and second AI model 121M2 are generated using non-contact sensors that do not come into contact with user U1, the user U1 is less likely to feel annoyed than if the input data for first AI model 121M1 and second AI model 121M2 were generated using contact sensors that come into contact with user U1. The brake sensor SN1 detects the amount of operation of a brake pedal (the amount of operation by the driver) provided on vehicle V1 and outputs brake information representing the detected amount of operation. The steering angle sensor SN2 detects the steering angle (steering angle) of the vehicle V1 and outputs steering angle information indicating the detected steering angle.

[0016] In the illustrated example, the behavior determination unit 132 processes the image captured by the camera C using image recognition processing or the like, and processes it into gaze data (images of the eyeballs or data processed into text and numerical data indicating the gaze (direction)) and facial direction data (images of the face or data processed into text and numerical data indicating the facial direction), and inputs the data to the first AI model 121M1 and the second AI model 121M2, but it is also possible to input the image captured by the camera C. The first AI model 121M1 and the second AI model 121M2 (two AI models designed according to the input data type and trained differently) corresponding to the input data type will be installed in the driving assistance device 10.

[0017] The first AI model 121M1 and the second AI model 121M2 are AI models that input gaze data, facial direction data, and voice data of user U1 and output estimated emotions of user U1, and are designed with a structure suitable for such input and output. The first AI model 121M1 and the second AI model 121M2 are trained using a large number of training data sets that use gaze data, facial direction data, and voice data as input data and emotions corresponding to the input data as correct answer data. In other words, the first AI model 121M1 and the second AI model 121M2 are trained AI models. Details of the training method will be described later.

[0018] Then, the first AI model 121M1 and the second AI model 121M2 output the emotions estimated (output) based on the gaze data, facial direction data, and voice data of the user U1 to the weighting processing unit 133. The weighting processing unit 133 performs weighting processing on the output of the first AI model 121M1 and the output of the second AI model 121M2, and finally outputs the estimated emotions to the driving assistance processing unit 134. The driving assistance processing unit 134 provides advice regarding driving, for example, based on the input emotions.

[0019] <2. Driving assistance systems> FIG. 2 is a diagram illustrating a configuration example of a driving assistance system SYS. The driving assistance system SYS includes a driving assistance device 10, a camera C, a microphone M, a brake sensor SN1, a steering angle sensor SN2, a content playback device D1, a traffic information output device D2, and a user identification information output device D3. FIG. 2 illustrates components necessary for explaining the features of this embodiment, and omits general components. The driving assistance device 10 includes an information processing device (emotion estimation device) that estimates the state (emotion) of a user U1 (driver). However, the driving assistance device 10 may be separate from the information processing device (emotion estimation device) that estimates the state (emotion) of the user U1 (driver). When the driving assistance device 10 and the information processing device (emotion estimation device) that estimates the state (emotion) of the user U1 (driver) are separate entities, the driving assistance device 10 and the information processing device (emotion estimation device) that estimates the state (emotion) of the user U1 (driver) are connected via wired or wireless communication. The driving assistance device 10 includes a communication unit 11, a storage unit 12, and a controller 13.

[0020] The storage unit 12 includes an AI model storage unit 121 that stores the trained first AI model 121M1 and second AI model 121M2 shown in FIG.

[0021] The first AI model 121M1 and the second AI model 121M2 are stored in the AI ​​model storage unit 121, for example, in the following manner.

[0022] The manufacturer or the like of the driving assistance device 10, when assembling the device, installs a memory into which the trained first AI model 121M1 and second AI model 121M2 are written as the AI ​​model storage unit 121. Alternatively, the manufacturer or the like of the driving assistance device 10 connects an external storage device in which the trained first AI model 121M1 and second AI model 121M2 are stored to the driving assistance device 10, reads the first AI model 121M1 and second AI model 121M2, and writes them into the AI ​​model storage unit 121. Alternatively, the manufacturer or the like of the driving assistance device 10 receives the trained first AI model 121M1 and second AI model 121M2 from a learning device (described later) via a network and writes them into the AI ​​model storage unit 121.

[0023] The storage unit 12 also stores a target event table 122 and databases 123 to 126 (first change detection history database 123, second change detection history database 124, target event occurrence history database 125, and weight adjustment database 126). The target event table 122 is a table that defines the types of target events to be registered in the target event occurrence history database 125 and the conditions for determining the types, as shown in Fig. 3, for example.

[0024] The target events of types TY1 and TY2 in FIG. 3 are changes in the driving conditions of the vehicle V1 in which the user U1 is riding. The target event of type TY3 in FIG. 3 is a change in the traffic conditions around the vehicle V1 in which the user U1 is riding. The target events of types TY4 to TY6 in FIG. 3 are scene changes in the content that the user U1 is watching. The target event of type TY7 in FIG. 3 is a change in the state of conversation of the user U1. The event type discrimination conditions (TX1 to TX7) are conditions for detecting the corresponding event, and for example, in the case of type TY1 (sudden steering), sudden steering discrimination conditions such as a judgment threshold for the steering angle and a judgment threshold for the steering angular velocity are stored. The controller 13 then determines the occurrence of each event by processing such as comparing a signal (e.g., a steering angle signal) from a sensor of the vehicle with the event type discrimination conditions.

[0025] The target events of types TY1 to TY7 are selected to be events that occur with a moderately high frequency, and the update frequency of the weight adjustment in the weight adjustment unit 136 can be increased moderately compared to events that occur very infrequently, such as being involved in a traffic accident or being honked at by a vehicle behind.

[0026] Note that a specific scene in the content can be detected by performing a known scene detection process (scene determination based on scene information included in the content information, or scene estimation based on image analysis or voice recognition) on the content information (image information of the content, voice information of the content) output from the content playback device D1. Also, a sudden change in tone of voice in the conversation of the user U1 can be detected by analyzing voice information collected by the microphone M (vocabulary recognition, analysis of pitch, etc.). The type of the target event registered in the target event table 122 may be added, for example, by communication between the cloud server and the driving assistance device 10. The databases 123 to 126 will be described later.

[0027] The controller 13 controls various operations of the driving assistance device 10, includes a processor that performs arithmetic processing, and is configured by, for example, a CPU (Central Processing Unit). The controller 13 has, as its functions, an acquisition unit 131, a behavior determination unit 132, a weighting processing unit 133, a driving assistance processing unit 134, a target event detection unit 135, and a weight adjustment unit 136. The functions of the controller 23 are realized by the processor executing arithmetic processing in accordance with a program stored in the storage unit 22.

[0028] The acquisition unit 131 acquires various types of appearance information (image information, audio information) of the user U1, who is the person being observed, captured by the camera C and the microphone M via the communication unit 11. Note that the appearance information here includes not only image data but also information that can be detected by non-contact sensors (remote detection sensors: camera, microphone, etc.), such as audio. The acquisition unit 131 stores the acquired various types of appearance information in a data table formed in the storage unit 12 as needed for subsequent processing. Note that the acquisition unit 131 acquires these pieces of appearance information at substantially the same time and stores them in the data table as one data set. These pieces of data are then used to detect the behavioral state at that time.

[0029] Furthermore, the acquisition unit 131 acquires brake information output from the brake sensor SN1 via the communication unit 11, and outputs steering angle information output from the steering angle sensor SN2.

[0030] The acquisition unit 131 acquires content information (image information of the content, audio information of the content) output from a content playback device D1 mounted on the vehicle V1 via the communication unit 11. The content playback device D1 is, for example, an optical disc playback device, a television broadcast receiving device, or the like. The acquisition unit 131 acquires traffic information output from a traffic information output device D2 mounted on the vehicle V1 via the communication unit 11. The traffic information output device D2 is, for example, a navigation device capable of receiving VICS (registered trademark) (Vehicle Information and Communication System) information, or the like. The acquisition unit 131 acquires user identification information output from a user identification information output device D3 mounted on the vehicle V1 via the communication unit 11. The user identification information output device D3 is, for example, a fingerprint sensor, a face authentication device that performs face authentication using an image captured by a camera C, or the like.

[0031] The behavior determination unit 132 determines predetermined types of behavior of the user U1, such as the line of sight, facial direction, facial expression, and voice, by performing an analysis process on the image information and voice information of the user U1 acquired by the acquisition unit 131. The predetermined types of behavior are behaviors corresponding to the types of data input to the first AI model 121M1 and the second AI model 121M2, and in the case of this embodiment, specifically, the line of sight, facial direction, facial expression, and voice (content).

[0032] Therefore, the behavior determination unit 132 performs various processes according to the input specifications (determined by design conditions, learning conditions, etc.) of the first AI model 121M1 and the second AI model 121M2. The various processes include, for example, cutting out images (a group of still images or video) and audio data of the processing target time length, cutting out and detecting the orientation of the face and eyeballs of the user U1 from the image captured by the camera C, and frequency analysis of the output signal from the microphone M.

[0033] The first AI model 121M1 and the second AI model 121M2 output appropriate estimated (correct) labels for input data of the same type (content and format) as the input data used during learning. Therefore, the output data type (content type and format) of the behavior determination unit 132 must be the same as the input data used during learning for the first AI model 121M1 and the second AI model 121M2. The behavior determination unit 132 performs such data processing. Programs and data for realizing these operations of the behavior determination unit 132 are stored in the memory unit 12.

[0034] The weighting processing unit 133 performs weighting processing on the output of the first AI model 121M1 and the output of the second AI model 121M2, and outputs the final estimated emotion to the driving support processing unit 134.

[0035] The driving support processing unit 134 provides, for example, advice regarding driving based on the input emotion.

[0036] The target event detection unit 135 detects a target event for calibration processing for weighting processing based on the surrounding environment of the user U1, using the information acquired by the acquisition unit 131. The target event is an event that is thought to be likely to cause a change in the emotion of the user U1.

[0037] The weight adjuster 136 adjusts the weights used in the weighting process by the weighting processor 133 according to the output states of the first AI model 121M1 and the second AI model 121M2 when the target event is detected.

[0038] In the driving assistance device 10, a target event is automatically detected by the target event detection unit 135 based on the surrounding environment of the user U1, and the weights are adjusted by the weight adjustment unit 136 according to the output states of the first AI model 121M1 and the second AI model 121M2 at the time the target event is detected. Therefore, the emotion estimation function is appropriately calibrated without imposing cumbersome work on the user U1, thereby improving the accuracy of estimating the emotion of the user U1.

[0039] Furthermore, in the driving assistance device 10, the target event detection unit 135 automatically detects target events that allow calibration (adjustment of weights) of the emotion estimation function without performing a special task based on the surrounding environment of the user U1, and performs calibration. Therefore, the calibration does not require the user U1 to perform a task, and the weights can be adjusted even when the user U1 cannot perform a task (for example, when the user U1 is driving the vehicle V1).

[0040] <3. Information processing for driving assistance devices> 4 is a flowchart showing information processing executed by the controller 13 of the driving assistance device 10 of FIG. 2. This flowchart shows the technical content of a computer program that causes a computer device to perform information processing. The computer program is stored in various readable non-volatile recording media, or is provided (sold, distributed, etc.) by distribution over a communication network from a server (where the computer program is stored in a storage device within the server). The computer program may consist of only one program, or may consist of multiple programs that work together.

[0041] The information processing shown in FIG. 4 is initiated when the vehicle V1 starts and becomes ready to travel, and is then repeatedly executed throughout the time period in which estimation of the emotion of the user U1 is required.

[0042] In step S10, the controller 13 identifies the user U1 based on the user identification information acquired by the acquisition unit 131. The information processing device included in the driving assistance device 10 is an information processing device capable of estimating the emotions of multiple users, and stores databases 123 to 126 for each of the multiple users in the storage unit 12. The controller 13 identifies the user U1 in step S10 and determines the databases 123 to 126 to be used in the subsequent processing. When the processing of step S10 ends, the process proceeds to step S20.

[0043] In step S20, the controller 13 (behavior determination unit 132) determines the behavior of the user U1, specifically, the gaze, facial direction, facial expression, and voice (content), based on the various data (camera captured images, microphone collected audio) acquired by the acquisition unit 131, and supplies each piece of data based on the behavior determination results to the first AI model 121M1 and the second AI model 121M2. The first AI model 121M1 and the second AI model 121M2 each output (estimate) the emotions of the user U1. When the processing of step S20 ends, the processing proceeds to step S30.

[0044] In step S30, the controller 13 determines whether the current output of the first AI model 121M1 has changed from the previous output of the first AI model 121M1, and whether the current output of the second AI model 121M2 has changed from the previous output of the second AI model 121M2. For example, if the type of emotion output from the model has changed, it is determined that there has been a change in the model output. Also, if the model outputs an emotion with a score (a score related to likelihood), it may also be determined that there has been a change in the model output when the type of emotion output from the model has not changed but the score has changed by exceeding a threshold.

[0045] If there is a change in at least one of the output of the first AI model 121M1 and the output of the second AI model 121M2, the process proceeds to step S40. On the other hand, if there is no change in either the output of the first AI model 121M1 or the output of the second AI model 121M2, the process proceeds to step S50.

[0046] In step S40, if there is a change in the output of the first AI model 121M1, the controller 13 updates the first change detection history database 123, which indicates the history of changes in the output of the first AI model 121M1. The first change detection history database 123 records the date and time when there was a change in the output of the first AI model 121M1, as shown in FIG.

[0047] In step S40, if there is a change in the output of the second AI model 121M2, the controller 13 updates the second change detection history database 124, which indicates the history of changes in the output of the second AI model 121M2. The second change detection history database 124 records the date and time when there was a change in the output of the second AI model 121M2, as shown in Fig. 6, for example. After the processing of step S40 ends, the processing proceeds to step S50.

[0048] In step S50, the controller 13 (target event detection unit 135) determines whether a target event has occurred. If a target event has occurred, the process proceeds to step S60. On the other hand, if a target event has not occurred, the process proceeds to step S70.

[0049] In step S60, the controller 13 updates the target event occurrence history database 125. The target event occurrence history database 125 is a database in which the date and time when a target event occurred is linked to the type of the target event, as shown in Fig. 7, for example. If the type of the target event is not taken into consideration when adjusting the weight, the target event occurrence history database 125 may be a database in which only the date and time are recorded, similar to the first change detection history database 123 and the second change detection history database 124. After the processing of step S60 is completed, the process proceeds to step S70.

[0050] In step S70, the controller 13 updates the weight adjustment database 126. Specifically, the controller 13 updates the weight adjustment database 126 as follows.

[0051] The controller 13 checks the consistency between the date and time in the target event occurrence history database 125 and the date and time in the first change detection history database 123. If the first change detection history database 123 contains a date and time within a certain error range of a certain date and time Tc4 in the target event occurrence history database 125, the controller 13 determines that the first AI model 121M1 detected a change in emotion when the target event occurred at the certain date and time Tc4, and updates the weight adjustment database 126 based on the determination result. Since only the consistency of the date and time is checked and the target event is not associated with the specific emotion of the user U1, the burden of the confirmation process is reduced. Note that if the target event occurs only once, the adverse impact of a false positive in the target event occurrence would be significant. Therefore, it is desirable that the target event occur multiple times and the number of times a change in emotion is detected be tallied.

[0052] Similarly, the controller 13 checks the consistency between the date and time in the target event occurrence history database 125 and the date and time in the second change detection history database 124, and if the second change detection history database 124 contains a date and time that is within a certain error range of a certain date and time Tc4 in the target event occurrence history database 125, the controller 13 determines that the second AI model 121M2 detected a change in emotion at the timing of the target event occurrence at the certain date and time Tc4, and updates the weight adjustment database 126 based on the determination result.

[0053] Fig. 8 is a diagram showing a first example of the weight adjustment database 126. The weight adjustment database 126 shown in Fig. 8 is a database in which the type of AI model, whether or not an emotion change has been detected for each target event, and the number of emotion change detections are linked together. The number of events that can be stored is essentially unlimited (subject to limitations imposed by memory capacity).

[0054] FIG. 9 is a diagram illustrating a second example of the weight adjustment database 126. The weight adjustment database 126 illustrated in FIG. 9 is a database in which the type of AI model, whether or not an emotion change has been detected for each target event, and the number of emotion changes detected are linked. In the weight adjustment database 126 illustrated in FIG. 9, the number of stored target events is limited to a fixed number of the most recent events. Therefore, when the number of target events reaches a fixed number, the oldest target event is deleted so that the number of target events does not exceed the fixed number. The second example of the weight adjustment database 126 can suppress an increase in the size of the database.

[0055] FIG. 10 is a diagram illustrating a third example of the weight adjustment database 126. The weight adjustment database 126 illustrated in FIG. 10 is a database in which the type of AI model, whether or not a change in emotion has been detected for each type of target event, and the number of times a change in emotion has been detected are linked. In the weight adjustment database 126 illustrated in FIG. 10, whether or not a change in emotion has been detected for each type of target event is narrowed down to the most recent time (the emotion change detection data is updated when the most recent type of target event is detected). It is also possible to narrow down the number of times to a certain number of times (for example, two times) rather than the most recent time. The third example of the weight adjustment database 126 can prevent weights from being adjusted when the types of target events that have occurred are biased.

[0056] Note that the weight adjustment database 126 does not need to treat all detections of target events equally. For example, when emotion estimation is performed in March, the number of detections may be calculated only for target events that occurred in March of the previous year. Also, for example, the number of detections may be calculated by placing more importance on data related to target events that occurred most recently (for example, data (number of detections) within the last week may be doubled).

[0057] When the process of step S70 is completed, the process proceeds to step S80.

[0058] In step S80, the weight adjustment unit 136 adjusts the weight data used in the weighting process in the weighting processing unit 133 based on the weight adjustment database 126. After the process of step S80 ends, the process proceeds to step S90. In step S90, the weighting processing unit 133 performs weighting processing of the output of the first AI model 121M1 and the output of the second AI model 121M2. After the process of step S90 ends, the process proceeds to step S100. In step S100, the weighting processing unit 133 determines the emotion with the highest score as the estimation result, and outputs the emotion estimation result to the driving assistance processing unit 134. After the process of step S100 ends, the process proceeds to step S110, which will be described later.

[0059] The processes of steps S80, S90 and S 100 will be described in detail below. Specific examples of weighting processes include weighted average processing, most frequently detected model selection processing, Bayesian network processing, and the like.

[0060] In weighted averaging when the first AI model 121M1 and the second AI model 121M2 output only emotion types, for example, if the output of the first AI model 121M1 is "anger," the output of the second AI model 121M2 is "sadness," the first AI model 121M1 has detected the target event eight times, and the second AI model 121M2 has detected the target event two times, the outputs of each model (one emotion type each) are weighted based on the ratio of the number of detections of each model, and the output of the weighting processor 133 is set to "anger score 80%, sadness score 20%." Then, in subsequent processing, it is determined how to handle this output "anger score 80%, sadness score 20%." For example, processing methods such as simply adopting the emotion type with the higher score, or adopting the emotion type with the highest score above a predetermined adoption threshold (if no such emotion is detected, it is treated as no emotion).

[0061] In weighted averaging processing when emotions are output from the first AI model 121M1 and the second AI model 121M2 with scores (scores related to likelihood), for example, if the output of the first AI model 121M1 is "anger 70%" and "sadness 30%", the output of the second AI model 121M2 is "anger 10%" and "sadness 90%", the number of times the target event has been detected by the first AI model 121M1 is "8 times", and the number of times the target event has been detected by the second AI model 121M2 is "2 times", the output of each model is weighted by the ratio of the number of times each model has been detected, and the output of the weighting processing unit 133 is "anger score 58% (= 70% × 0.8 + 10% × 0.2) and sadness score 42% (= 30% × 0.8 + 90% × 0.2)".

[0062] In the most frequently detected model selection process when the first AI model 121M1 and the second AI model 121M2 output only the type of emotion, for example, if the output of the first AI model 121M1 is "anger," the output of the second AI model 121M2 is "sadness," the first AI model 121M1 has detected the target event "8 times," and the second AI model 121M2 has detected the target event "2 times," the output of the first AI model 121M1, which is the most frequently detected model, is selected, and the output of the weighting processing unit 133 is set to "anger."

[0063] In the case where emotions are output from the first AI model 121M1 and the second AI model 121M2 with scores (scores related to likelihood), the most frequently detected model selection process, for example, if the output of the first AI model 121M1 is "anger 70%" and "sadness 30%", the output of the second AI model 121M2 is "anger 10%" and "sadness 90%", the number of times the target event has been detected by the first AI model 121M1 is "8 times", and the number of times the target event has been detected by the second AI model 121M2 is "2 times", then the output of the first AI model 121M1, which is the most frequently detected model, is selected, and the output of the weighting processing unit 133 is set to "anger score 70%, sadness score 30%".

[0064] In step S110, the driving assistance processing unit 134 determines whether or not it is necessary to provide advice regarding vehicle driving based on the emotion output from the weighting processing unit 133 (for example, it determines that advice is necessary in the case of the emotion "anger"). If it is necessary to provide advice, the driving assistance processing unit 134 provides advice for the estimated emotion in step S120, and then returns to step S20, where the first AI model 121M1 and the second AI model 121M2 perform the next emotion estimation. On the other hand, if it is not necessary to provide advice, step S120 is skipped, and the process returns to step S20, where the first AI model 121M1 and the second AI model 121M2 perform the next emotion estimation.

[0065] <4. AI model learning method> Next, the learning method of the first AI model 121m1 will be described. To make it easier to distinguish between the pre-learning and post-learning models, the pre-learning model is referred to as the first AI model 121m, and the post-learning model is referred to as the first AI model 121M1, with uppercase and lowercase letters used in the symbols to distinguish between the pre-learning and post-learning models. Figure 11 is a conceptual explanatory diagram showing the AI ​​model learning process performed by the learning device 20.

[0066] In this embodiment, user U2 is a plurality of subjects (subjects for generating learning data) with similar characteristics for generating learning data. In this embodiment, since the driving assistance device 10 is assumed to be an in-vehicle device, learning is performed using an in-vehicle device with a similar environment, and the subject user U2 is the driver of vehicle V2.

[0067] It is possible to mount the learning device 20 on the vehicle V2 and have it perform learning, or to install the learning device 20 in a location other than the vehicle V2, such as a research and development laboratory, and have it perform learning. In the latter case, the emotion of the subject user U2 riding in the vehicle V2 is input to the learning device 20 via a recording medium or a communication line. In this embodiment, an example will be described in which the learning device 20 is mounted on the vehicle V2 and has it perform learning.

[0068] An AI model storage unit is provided in a storage device (memory, etc.) of the learning device 20 mounted on the vehicle V2, and the pre-learning first AI model 121m1 is stored in the storage device. Then, the learning device 20 executes the learning process described below, whereby the pre-learning first AI model is learned and a trained first AI model, i.e., the first AI model 121M1 to be implemented in the AI ​​application device, is generated.

[0069] The trained first AI model 121M1 is used in the driving assistance device 10 shown in FIGS. 1 and 2. Therefore, the input data of the training data for the first AI model 121m1 is behavioral information such as the user U2's gaze, facial direction, facial expression, and voice (appearance information, data of the same type and format as the input data of the first AI model 121M1 of the driving assistance device 10), and the correct data is emotion. The correct data, which is emotion, may be generated, for example, from the results of a questionnaire given to the user U2 or analysis results based on the user U2's biosignals. Specifically, an electroencephalogram (EEG) sensor and a heart rate sensor are attached to the user U2 to measure their EEG and heart rate, and their emotion is estimated based on the measured EEG and heart rate. The estimated emotion is used as the correct data.

[0070] The learning method of the second AI model 121m2 is the same as the learning method of the first AI model 121m1 except for the characteristics of the user U2.

[0071] <Notes, etc.> Various technical features disclosed in the description of the present invention may be modified in various ways without departing from the spirit of the technical creation. Furthermore, multiple embodiments and modifications disclosed in the description of the present invention may be combined to the extent possible.

[0072] In the above-described embodiment, the user's emotion is estimated, but a state of the user other than emotion may also be estimated. Examples of the state of the user other than emotion include the user's concentration level, the user's behavior, etc.

[0073] In the above-described embodiment, two AI models (first AI model 121m1 and second AI model 121m2) are used in the information processing device, but three AI models may be used in the information processing device. [Explanation of symbols]

[0074] 10 Driving assistance devices 20 Learning Device 11 Communications Department 12 Storage section 13 Controller 121 AI model memory unit 121m1 1st AI model 121m2 2nd AI model 131 Acquisition Department 132 Behavior determination unit 133 Weighting processing unit 134 Driving assistance processing unit 135 Target event detection unit 136 Weight adjustment unit C Camera D1 Content playback device D2 Traffic information output device D3 User specific information output device M microphone SN1 Brake Sensor SN2 steering angle sensor SYS Driving Assistance System U1 User V1 vehicle

Claims

1. An information processing device that estimates a user's state by performing weighting processing on outputs of a plurality of AI models that estimate a user's state, Detecting a target event for a calibration process for the weighting process based on the surrounding environment of the user; An information processing device that adjusts weights in the weighting process according to the output status of each of the plurality of AI models when the target event is detected.

2. The information processing device according to claim 1 , wherein each of the plurality of AI models estimates an emotion of the user.

3. 2. The information processing device according to claim 1, wherein the target event includes at least one of a change in the driving state of the vehicle in which the user is riding, a change in a scene of the content the user is watching, a change in the state of the user's conversation, and a change in the traffic state of the vehicle in which the user is riding.

4. Detecting whether or not there is a change in the output of each of the plurality of AI models when the target event occurs; The information processing device according to claim 1 , wherein the weights in the weighting process are adjusted based on the number of detected changes in each AI model output.

5. Recognizes the user, The information processing apparatus according to claim 1 , wherein the weights in the weighting process are adjusted using weight adjustment data corresponding to the recognized user.

6. An information processing method for estimating a user's state by performing weighting processing on outputs of a plurality of AI models that estimate a user's state, Detecting a target event for a calibration process for the weighting process based on the surrounding environment of the user; An information processing method, comprising adjusting weights in the weighting process according to the output status of each of the plurality of AI models when the target event is detected.

7. An information processing program that estimates a user's state by performing weighting processing on outputs of a plurality of AI models that estimate a user's state, Detecting a target event for a calibration process for the weighting process based on the surrounding environment of the user; adjusting weights in the weighting process according to output conditions of each of the plurality of AI models when the target event is detected; An information processing program that causes a computer to execute the following.

8. 1. A feeling estimation device that estimates a user's feeling by performing a weighting process on outputs of a plurality of AI models that estimate a user's feeling based on an image captured by the user, Detecting a target event for a calibration process for the weighting process based on the surrounding environment of the user; and adjusting weights in the weighting process according to output states of the plurality of AI models when the target event is detected.

9. A driving assistance system including: an emotion estimation device that estimates an emotion of a user; and a driving assistance device that assists in driving a vehicle based on the emotion estimated by the emotion estimation device, the emotion estimation device, 1. A feeling estimation device that estimates a user's feeling by performing a weighting process on outputs of a plurality of AI models that estimate a user's feeling based on an image captured by the user, Detecting a target event for a calibration process for the weighting process based on the surrounding environment of the user; adjusting weights in the weighting process according to output states of the plurality of AI models at the time of detecting the target event; Driver assistance system.

Images