Estimation device, on-vehicle system, and estimation program

Abstract

To provide a technology to make a balance between a load of emotion estimation processing and accuracy of emotion estimation an appropriate one.SOLUTION: An exemplary estimation device detects a scene surrounding an emotion estimation target person, and performs resource adjustment for adjusting a resource amount spent for emotion estimation processing for estimating an emotion of the emotion estimation target person on the basis of each output of a plurality of emotion estimation sensors according to the detected scene.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for estimating emotions. [Background technology]

[0002] Conventionally, there is known a technology for estimating a person's emotions based on the outputs of multiple emotion estimation sensors, such as an electroencephalogram sensor, a heart rate sensor, a camera (for detecting facial expressions), a microphone (for detecting vocal sounds), etc., and using the estimated emotions for various types of control (for example, vehicle driving control, control of in-vehicle equipment, etc.) (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-103462 A Summary of the Invention [Problem to be solved by the invention]

[0004] In order to estimate emotions with high accuracy, it is necessary to provide many different emotion estimation sensors. However, the more the number of emotion estimation sensors increases, the greater the load on the emotion estimation process becomes. When the load on the emotion estimation process increases, there is a risk of processing delays occurring. Furthermore, in order to prevent processing delays, it becomes necessary to use expensive devices to improve processing capabilities.

[0005] In view of the above, an object of the present invention is to provide a technique that can appropriately balance the load of emotion estimation processing and the accuracy of emotion estimation. [Means for solving the problem]

[0006] An exemplary estimation device of the present invention detects a scene surrounding a subject of emotion estimation, and performs resource adjustment to adjust the amount of resources consumed for emotion estimation processing that estimates the emotions of the subject of emotion estimation based on the outputs of multiple emotion estimation sensors, in accordance with the detected scene. Effect of the Invention

[0007] According to the exemplary present invention, the amount of resources consumed for emotion estimation processing can be adjusted according to the scene surrounding the emotion estimation target. Although reducing the load of emotion estimation processing and improving the accuracy of emotion estimation are in a trade-off relationship, since the amount of resources consumed for emotion estimation processing can be adjusted according to the scene surrounding the emotion estimation target, it is possible to appropriately balance the load of emotion estimation processing and the accuracy of emotion estimation. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an estimation system according to a first embodiment; [Diagram 2] FIG. 1 is a diagram showing an example of the configuration of an estimation device according to a first embodiment; [Diagram 3] FIG. 13 is a diagram showing an example of a weighting value table. [Figure 4] 1 is a flowchart showing an example of processing executed by the estimation device according to the first embodiment. [Diagram 5] FIG. 13 is a diagram showing an example of the configuration of an estimation system according to a second embodiment; [Figure 6] FIG. 13 is a diagram showing an example of the configuration of an estimation device according to a second embodiment. [Figure 7] FIG. 13 is a diagram showing an example of a weighting value table for various users. [Figure 8] 11 is a flowchart showing an example of processing executed by an estimation device according to a second embodiment. [Figure 9] FIG. 1 is a diagram showing an example of the configuration of a learning model generation device. [Figure 10] A diagram showing an example of an emotion estimation model (psychological plane) [Figure 11] FIG. 13 is a diagram showing an example of a sensor table. [Figure 12]A diagram showing an example of a psychological plane table. [Figure 13] FIG. 13 is a diagram showing another example of the configuration of a learning model generating device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[0010] <1. First embodiment> [1-1. Estimation system] 1 is a diagram showing an example configuration of an estimation system 1 according to a first exemplary embodiment of the present invention. In this embodiment, a user U1 is an occupant of a vehicle 20. The estimation system 1 is configured as a system that estimates the emotion of the user U1.

[0011] 1 shows only one user U1, there may be multiple users U1. For example, user U1 may be an e-sports player, a patient at a medical institution, a student at an educational institution, a viewer of content such as video or music, etc. In the case of these users, a part of the configuration of the estimation system 1 (such as a biosensor) will be installed at the location where user U1 is present, for example, the location where the e-sports operation terminal is installed, a hospital, a school, a content viewing room, etc.

[0012] 1, the estimation system 1 includes a server 10 (computer), an in-vehicle camera 21, an in-vehicle microphone 22, an exterior camera 23, an exterior microphone 24, and an in-vehicle device 25. In this embodiment, the server 10 is disposed outside the vehicle 20, for example, in an emotion estimation service center building or the like, and the in-vehicle camera 21, the in-vehicle microphone 22, the exterior camera 23, the exterior microphone 24, and the in-vehicle device 25 are mounted on the vehicle 20.

[0013] The server 10 may be a physical server or a virtual server. In this embodiment, the server 10 constitutes an estimation device that estimates emotions. That is, the estimation system 1 includes the estimation device 10. Hereinafter, the server 10 will be referred to as the estimation device 10. Details of the estimation device 10 will be described later. Note that the estimation device 10 may be constituted by one server or multiple servers.

[0014] The in-vehicle camera 21, the in-vehicle microphone 22, the exterior camera 23, and the exterior microphone 24 are each an emotion estimation sensor. The in-vehicle camera 21 captures images of the interior of the vehicle 20. The in-vehicle microphone 22 collects sounds generated within the interior of the vehicle 20. The exterior camera 23 captures images of the surroundings of the vehicle 20. The exterior microphone 24 collects sounds generated around the vehicle 20. The in-vehicle device 25 is equipped with a weather sensor included in the emotion estimation sensor. The weather sensor detects the weather by performing image analysis on images captured by the exterior camera 23.

[0015] The in-vehicle device 25 includes a wireless unit that performs wireless communication with the server 10. The in-vehicle device 25 transmits the outputs of the emotion estimation sensors to the server 10, and uses the emotion estimation results received from the server 10 for, for example, emotion information notification, vehicle control, and the like.

[0016] Here, the flow of processing in the estimation system 1 will be described with reference to FIG.

[0017] The in-vehicle camera 21, the in-vehicle microphone 22, the outside camera 23, and the outside microphone 24 each output a sensing result to the in-vehicle device 25. The in-vehicle camera 21 outputs image information in the cabin of the vehicle 20. The in-vehicle microphone 22 outputs audio information in the cabin of the vehicle 20. The outside camera 23 outputs image information of the surroundings of the vehicle 20. The outside microphone 24 outputs audio information of the surroundings of the vehicle 20. The outputs (sensor signals) of the in-vehicle camera 21, the in-vehicle microphone 22, the outside camera 23, and the outside microphone 24 and the output of the weather sensor are transmitted to the estimation device (server) 10 via the network N1 by the wireless function (wireless unit) of the in-vehicle device 25 (step S1). For example, the sensor signals of the in-vehicle camera 21, the in-vehicle microphone 22, the outside camera 23, and the outside microphone 24 and the output of the weather sensor may be transmitted to the estimation device 10 using a communication device equipped in the vehicle 20.

[0018] The estimation device 10 detects a scene surrounding the user U1 (step S2). More specifically, the estimation device 10 detects a scene surrounding the user U1 using the outputs (in-vehicle images and sounds, and outside the vehicle (vehicle surrounding images and sounds)) of a plurality of emotion estimation sensors.

[0019] The estimation device 10 adjusts the amount of resources consumed for the emotion estimation process according to the scene detected in step S2 (step S3), and executes the emotion estimation process to estimate the emotion of the user U1 (step S4). The estimation device 10 provides (transmits) the estimation result to the vehicle 20 (step S5).

[0020] A control device (on-board device 25) mounted on the vehicle 20 controls the vehicle 20 using, for example, the estimation result provided by the estimation device 10, and outputs (displays and outputs audio) driving guidance (advice) information according to the estimated emotion. For example, an angry driver tends to operate the accelerator, brake, and steering wheel roughly. Therefore, when the user U1 is the driver and the estimation result is "anger," the control device mounted on the vehicle 20 performs control to reduce sensitivity to the accelerator, brake, and steering wheel operations (to convert them into operation signals that change gently).

[0021] In another application example, when the user U1 is a sports player, the estimation result can be used for the mental training of the user U1. For example, when the user U1 feels unfavorable emotions (anxiety, anger, etc.) during playing a video game, it is determined that intensive training corresponding to the emotional state is necessary. Therefore, the estimation result of the estimation device 10 is used in the training by providing the player, etc. with information on the estimated emotions in each sports scene or providing appropriate advice information on the estimated emotions.

[0022] In addition, various types of e-sports include a type in which multiple users U1 play together and a type in which multiple users U1 play against each other. In such types of e-sports, if the emotional state of each player is displayed, it becomes possible to perform advanced game play, such as changing game tactics according to the emotional state. In addition, it is also possible to configure the e-sports spectators to understand the emotional state of each player while watching the game, which can increase the fun factor of watching the game.

[0023] Furthermore, if the user U1 is a patient at a medical institution, the estimation result can be used for examinations, treatment, etc. For example, the staff at the medical institution can understand that the patient is feeling anxious based on the estimated emotion and provide countermeasures such as counseling.

[0024] In addition, if the user U1 is a student in an educational institution, the estimation result can be used to improve the content of lessons. For example, a teacher can understand that a student finds a lesson boring based on the estimated emotion and improve the content of the lesson to make it more interesting for the student.

[0025] Furthermore, if the user U1 is a viewer of content such as video or music, the estimation result can be used to create further content. For example, a distributor of video content can create a highlight video by collecting scenes that viewers found enjoyable based on the estimated emotions.

[0026] In the above, an example has been given in which the server 10 disposed outside the vehicle 20 is the estimation device, but the estimation device may be mounted on the vehicle 20. In addition, for example, the estimation device may be composed of a plurality of devices. For example, a device disposed outside the vehicle 20 and a device mounted on the vehicle 20 may perform a process for estimating emotions in a distributed manner.

[0027] [1-2. Estimation device] FIG. 2 is a diagram showing an example of the configuration of an estimation device 10 according to a first exemplary embodiment of the present invention. Note that FIG. 2 shows components necessary for explaining the features of this embodiment, and a description of general components is omitted. As shown in FIG. 2, the estimation device 10 includes a communication unit 11 and a storage unit 12. The estimation device 10 also includes a controller 13. The estimation device 10 may be a so-called computer device. Note that the estimation device 10 may be configured to include an input device such as a keyboard and an output device such as a display.

[0028] The communication unit 11 is an interface for communicating data with other devices via the network N1. The communication unit 11 is, for example, a network interface card (NIC).

[0029] The storage unit 12 includes a volatile memory and a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM). The non-volatile memory may include, for example, a read only memory (ROM), a flash memory, or a hard disk drive. The non-volatile memory stores computer-readable programs and data. At least a part of the programs and data stored in the non-volatile memory may be obtained from another computer device connected by wire or wirelessly, or from a portable recording medium.

[0030] As shown in Fig. 2, in this embodiment, the storage unit 12 includes a table (data table) 121 and a (machine) learning model 122. In detail, the table 121 includes a plurality of tables for various processes. The table 121 includes a weighting value table 121a. Fig. 3 is a diagram showing an example of the weighting value table 121a.

[0031] 3, the items of the weighting value table 121a include "scene," "weighting value of the output of the in-vehicle camera," "weighting value of the output of the in-vehicle microphone," "weighting value of the output of the outside-vehicle camera," "weighting value of the output of the outside-vehicle microphone," "weighting value of weather information," "priority," and "learning model." The items of the table correspond to data storage cells (storage slots).

[0032] The item "scene" of the weighting value table 121a stores the type of scene surrounding the user U1. The type of scene surrounding the user U1 is also the primary key of the data record in the weighting value table 121a. That is, in the weighting value table 121a, a data record is configured for each type of scene surrounding the user U1, and data of each item associated with the type of scene surrounding the user U1 is stored in the data record.

[0033] The item "Weighting value of in-vehicle camera output" in the weighting value table 121a stores a weighting value for the output of the in-vehicle camera 21. The item "Weighting value of in-vehicle microphone output" in the weighting value table 121a stores a weighting value for the output of the in-vehicle microphone 22. The item "Weighting value of outside-vehicle camera output" in the weighting value table 121a stores a weighting value for the output of the outside-vehicle camera 23. The item "Weighting value of outside-vehicle microphone output" in the weighting value table 121a stores a weighting value for the output of the outside-vehicle microphone 24. The item "Weighting value of weather information" in the weighting value table 121a stores a weighting value for weather information. In this example, four levels of "high", "medium", "low", and "OFF" are used as the weighting values, but the weighting values ​​may be specific numerical values ​​other than the four levels (above or below).

[0034] The learning model 122 includes a plurality of learning models for each scene. Each learning model is generated so that output data of each sensor according to the weighting value of the corresponding scene is used as an input. Note that adjustment of the output data of each sensor according to the weighting value (detail or output) can be realized by using an adjustment function of each sensor (if the sensor has the function) or by installing an adjustment device (a device for changing the resolution of an image, etc.) at the rear stage of each sensor.

[0035] For example, a learning model corresponding to a traffic jam scene may be a learning model trained by machine learning using learning data that uses as input data the output of in-vehicle camera 21 with the level of detail adjusted to a high level, the output of in-vehicle microphone 22 with the level of detail adjusted to a low level, and the output of exterior camera 23 with the level of detail adjusted to a medium level, and uses as correct answer data the emotions of the person to be subjected to data creation for creating the learning data at the time when the input data was acquired.

[0036] As another example, a learning model corresponding to a scene of driving on snowy roads may be a learning model machine-learned using learning data in which the input data is the output of the exterior camera 23 with the level of detail adjusted to a high level, the output of the exterior microphone 24 with the level of detail adjusted to a medium level, and weather information with the level of detail adjusted to a medium level, and the correct answer data is the emotions of the person to be used to create the learning data at the time the input data was acquired.

[0037] Returning to FIG. 2, an example of the configuration of the estimation device 10 will be described. The controller 13 includes a processor that performs arithmetic processing and the like. The processor may include, for example, a CPU (Central Processing Unit). The controller 13 may be configured with one processor or multiple processors. When configured with multiple processors, the processors only need to be connected to each other so that they can communicate with each other. When the estimation device 10 is configured as a cloud server, the CPU that configures the processor may be a virtual CPU.

[0038] 2, the controller 13 includes, as its functions, an acquisition unit 131, a scene detection unit 132, a weighting value extraction unit 133, a resource adjustment unit 134, a feeling estimation unit 135, and a providing unit 136. In this embodiment, the functions of the controller 13 are realized by a processor executing arithmetic processing in accordance with a program stored in the storage unit 12.

[0039] The scope of the present embodiment may include a computer program that causes a processor (computer) to realize at least a part of the functions of the estimation device 10. The scope of the present embodiment may also include a computer-readable non-volatile recording medium that records such a computer program. The non-volatile recording medium may be, for example, an optical recording medium (e.g., an optical disk), a magneto-optical recording medium (e.g., a magneto-optical disk), a USB memory, or an SD card, in addition to the above-mentioned non-volatile memory.

[0040] Further, each of the functional units 131 to 136 may be realized by one program, or may be realized by a separate program for each functional unit. Further, each of the functional units 131 to 136 may be realized as a separate server. Further, each of the functional units 131 to 136 may be realized by having a processor execute a program, that is, by software, as described above, or may be realized by other methods. At least a part of each of the functional units 131 to 136 may be realized by using, for example, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. That is, each of the functional units 131 to 136 may be realized by hardware using a dedicated IC, or the like. Further, each of the functional units 131 to 136 may be realized by using both software and hardware. Further, each of the functional units 131 to 136 is a conceptual component. A function executed by one component may be distributed to multiple components. Further, functions possessed by multiple components may be integrated into one component.

[0041] The acquisition unit 131 acquires the outputs (sensor signals) of the in-vehicle camera 21, the in-vehicle microphone 22, the outside camera 23, and the outside microphone 24 via the communication unit 11. The acquisition unit 131 also acquires the output of a weather sensor via the communication unit 11. The acquisition unit 131 may acquire weather information for the location where the vehicle 20 is located from a specific website or the like. The acquisition unit 131 stores the acquired data in the memory unit 12 as necessary for subsequent processing.

[0042] The scene detection unit 132 detects the scene surrounding the user U1 using the outputs (sensor signals) of the in-vehicle camera 21, the in-vehicle microphone 22, the outside camera 23, and the outside microphone 24. For example, when the scene detection unit 132 detects a scene using a learning model for scene detection, each sensor output data in various scenes is collected in advance, each scene is manually labeled with a scene type, each sensor output data is used as input data, and scene detection learning data in which the labels are correct data is created. Then, the scene detection unit 132 uses the scene detection learning model trained with the scene detection learning data. Note that the scene detection unit 132 may detect a scene without using the outputs (sensor signals) of the in-vehicle camera 21, the in-vehicle microphone 22, the outside camera 23, and the outside microphone 24. For example, the scene detection unit 132 may detect a scene based on data from a navigation device (a road determination using a current position and map data), vehicle speed information, an operating state of an audio device, and the like.

[0043] The weighting value extraction unit 133 extracts weighting values ​​corresponding to the scenes detected by the scene detection unit 132 from the weighting value table 121a (a weighting value for the output of the in-vehicle camera 21, a weighting value for the output of the in-vehicle microphone 22, a weighting value for the output of the outside-vehicle camera 23, a weighting value for the output of the outside-vehicle microphone 24, and a weighting value for weather information).

[0044] The resource adjustment unit 134 adjusts the amount of resources consumed for the emotion estimation process executed by the emotion estimation unit 135 according to the scene detected by the scene detection unit 132. In detail, the resource adjustment unit 134 uses the weighting values ​​extracted by the weighting value extraction unit 133 to control the usage of the outputs (sensor signals) and weather information of the in-vehicle camera 21, the in-vehicle microphone 22, the outside vehicle camera 23, and the outside vehicle microphone 24.

[0045] In detail, the resource adjustment unit 134 selects a sensor for emotion estimation used in emotion estimation processing in the emotion estimation unit 135. The resource adjustment unit 134 does not supply a sensor signal with a weighting value of "OFF" or weather information with a weighting value of "OFF" to the emotion estimation unit 135. For example, when the scene detection unit 132 detects a scene of traffic congestion (information about the surroundings of the vehicle (outside the vehicle) surrounding the user U1), the resource adjustment unit 134 supplies only the output of the in-vehicle camera 21, the output of the in-vehicle microphone 22, and the output of the outside vehicle camera 23 to the emotion estimation unit 135. In other words, when the scene detection unit 132 detects a scene of traffic congestion, the number of sensor signals used in emotion estimation processing in the emotion estimation unit 135 is controlled to three. In addition, when the scene detection unit 132 detects a scene of music appreciation (information about the vehicle interior situation surrounding the user U1), the resource adjustment unit 134 supplies only the output of the in-vehicle camera 21 to the emotion estimation unit 135. That is, when the scene detection unit 132 detects a music appreciation scene, the number of sensor signals used in the emotion estimation process in the emotion estimation unit 135 is controlled to 1. When multiple scenes are detected at the same time, the scene detection unit 132 is set to detect a scene with a high priority (determined based on the priority data in the weighting value table 121a).

[0046] As described above, the resource adjustment unit 134 selects emotion estimation sensors to be used in the emotion estimation process in the emotion estimation unit 135, so that it is possible to reduce the number of sensor signals to be used in the emotion estimation process in the emotion estimation unit 135, and thus it is possible to significantly reduce the load of the emotion estimation process.

[0047] Alternatively, the resource adjustment unit 134 adjusts the detail of at least a part of the sensor signal used in the emotion estimation process in the emotion estimation unit 135. When the sensor signal is an image signal, the adjustment of the detail can be realized, for example, by converting (changing) at least one of the resolution of the sensor signal and the frame rate of the sensor signal. When the sensor signal is an audio signal, the adjustment of the detail can be realized, for example, by converting (changing) at least one of the resolution of the sensor signal and the sampling rate of the sensor signal.

[0048] For example, when the weighting value of a sensor signal (e.g., an image captured by a camera) is "high", the resource adjustment unit 134 leaves the detail of the sensor signal as it is (e.g., the camera capture resolution), when the weighting value of the sensor signal is "medium", it performs a conversion (change) process to reduce the detail of the sensor signal (e.g., converting the image resolution to medium resolution (e.g., a resolution that is 50% of the camera capture resolution)), and when the weighting value of the sensor signal is "low", it performs a conversion (change) process to reduce the detail of the sensor signal more than when the weighting value is "medium" (e.g., converting the image resolution to low resolution (e.g., a resolution that is 20% of the camera capture resolution)). In addition, for example, when the weighting value of the weather information is "high", the resource adjustment unit 134 leaves the weather information as it is (for example, information on type (sunny, cloudy) and precipitation in 1mm units (upper limit 50mm)), when the weighting value of the weather information is "medium", it performs a conversion (change) process to reduce the detail of the weather information (for example, convert to information on type (sunny, cloudy) and 10mm units (upper limit 50mm)), and when the weighting value of the weather information is "low", it performs a conversion (change) process to reduce the detail of the weather information to a greater extent than when the weighting value is "medium" (for example, convert to information on type (sunny, cloudy, rainy)).

[0049] As described above, the resource adjustment unit 134 adjusts the detail of at least a part of the sensor signals used in the emotion estimation process in the emotion estimation unit 135, so that it is possible to suppress a reduction in the number of sensor signals used in the emotion estimation process in the emotion estimation unit 135, and thus it is possible to suppress a decrease in the accuracy of the emotion estimation process.

[0050] The resource adjustment unit 134 selects a learning model to be used by the feeling estimation unit 135, based on the learning model data in the weighting value table 121a. That is, the resource adjustment unit 134 switches the learning model to be used by the feeling estimation unit 135 depending on the scene detected by the scene detection unit 132. This makes it possible to use a learning model suitable for the scene detected by the scene detection unit 132 (suitable for the weighting value of each sensor (for example, learning has been performed according to the weighting value)), and it is possible to suppress a decrease in the accuracy of feeling estimation even if the amount of resources consumed for the feeling estimation process is reduced by resource adjustment.

[0051] The emotion estimation unit 135 estimates the emotion of the user U1 based on the outputs (sensor signals) of the in-vehicle camera 21, the in-vehicle microphone 22, the outside camera 23, the outside microphone 24, etc., controlled by the resource adjustment unit 134. In detail, the emotion estimation unit 135 applies the outputs of the in-vehicle camera 21, the in-vehicle microphone 22, the outside camera 23, the outside microphone 24, etc., controlled by the resource adjustment unit 134 to the learning model switched by the resource adjustment unit 134, to estimate the emotion of the user U1.

[0052] The providing unit 136 provides the estimation result estimated by the emotion estimation unit 135 to the in-vehicle device 25 in the vehicle 20 via the network N1. Based on the estimated emotion provided to the in-vehicle device 25, the vehicle is controlled, driving guidance (advice) is provided to the driver, and the like.

[0053] [1-3. Estimation method] Next, a method for estimating an emotion executed by the estimation device 10 will be described.

[0054] The estimation method of the present embodiment can be realized by a computer program that causes a computer device to operate in accordance with the estimation method. Such a computer program is provided to an administrator of the estimation device 10 by a computer-readable non-volatile recording medium or by using a communication network (so-called download), and is installed in the estimation device 10. Such a computer program is included in the scope of the present embodiment. Furthermore, the computer program that causes a computer device to realize the estimation method of the present embodiment may be composed of only one program, or may be composed of multiple programs.

[0055] Fig. 4 is a flowchart showing an example of a process executed by the estimation device 10. The process shown in Fig. 4 is started when emotion estimation is started, for example, when the emotion of the user U1 is estimated from the time when the user U1 gets into the vehicle 20 and used for various controls of the vehicle, by the acquisition unit 131 of the estimation device 10 acquiring a signal notifying door unlocking from the vehicle 20.

[0056] In step S10, the scene detection unit 132 (controller 13) detects a scene surrounding the user U1 using the outputs (sensor signals) of the in-vehicle camera 21, the in-vehicle microphone 22, the outside camera 23, and the outside microphone 24. When the scene surrounding the user U1 is detected by the scene detection unit 132, the process proceeds to the next step S20.

[0057] In step S20, the weighting value extraction unit 133 (controller 13) extracts, from the weighting value table 121a, a weighting value corresponding to the scene detected by the scene detection unit 132. Once the weighting value extraction unit 133 has extracted the weighting value, the process proceeds to the next step S30.

[0058] In step S30, the resource adjustment unit 134 (controller 13) uses the weighting values ​​extracted by the weighting value extraction unit 133 to control (adjust) each sensor signal, thereby adjusting the amount of resources consumed for the emotion estimation process executed by the emotion estimation unit 135 in accordance with the scene detected by the scene detection unit 132. When the resource amount is adjusted by the resource adjustment unit 134, the process proceeds to the next step S40.

[0059] In step S40, the feeling estimation unit 135 (controller 13) selects a learning model corresponding to the scene detected by the scene detection unit 132, based on the learning model data in the weighting value table 121a. Once the learning model is selected by the resource adjustment unit 134, the process proceeds to the next step S50.

[0060] In step S50, the feeling estimation unit 135 (controller 13) estimates the feeling of the user U1 based on the sensor signal controlled by the resource adjustment unit 134 and the learning model selected in step S40. When the feeling estimation unit 135 estimates the feeling of the user U1, the process proceeds to the next step S60.

[0061] In step S60, the providing unit 136 provides (transmits) the estimation result estimated by the emotion estimation unit 135 to the in-vehicle device 25 in the vehicle 20 via the network N1 using the communication unit 11. Then, the in-vehicle device 25 performs various controls according to the provided estimated emotion.

[0062] When the estimation result is provided to the in-vehicle device in the vehicle 20 by the providing unit 136, the process returns to step S10, and the above-mentioned process is repeated.

[0063] In addition, when the operation of the device that uses the emotion estimation result is terminated, for example, when the acquisition unit 131 of the estimation device 10 acquires a signal from the vehicle 20 notifying that the doors are locked (end of various controls while the vehicle is running), an interrupt process for terminating the process shown in FIG. 4 is started, and the process shown in FIG. 4 is terminated.

[0064] By the above-described processing, it becomes possible to control the in-vehicle device based on the emotion of the user U1.

[0065] In this embodiment, the resource adjustment unit 134 adjusts resources according to the scene detected by the scene detection unit 132. Although reducing the load of the emotion estimation process and improving the accuracy of emotion estimation are in a trade-off relationship, the amount of resources consumed for the emotion estimation process can be adjusted according to the scene surrounding the user U1, so that an appropriate balance can be achieved between the load of the emotion estimation process and the accuracy of emotion estimation.

[0066] <2. Second embodiment> FIG. 5 is a diagram showing an example of the configuration of an estimation system 1 according to the second exemplary embodiment of the present invention.

[0067] In this embodiment, the estimation device 10 detects not only the scene surrounding the user U1 but also the type of the user U1 (step S2). Then, the estimation device 10 adjusts the amount of resources consumed for the emotion estimation process according to the scene and the type of the user U1 detected in step S2 (step S3). In other respects, this embodiment is similar to the first embodiment.

[0068] FIG. 6 is a diagram showing an example of the configuration of an estimation device 10 according to the second exemplary embodiment of the present invention.

[0069] In this embodiment, the controller 13 of the estimation device 10 is equipped with an occupant determination unit 137, the memory unit 12 stores multiple weighting value tables, and the weighting value extraction unit 133 extracts weighting values ​​corresponding to the scene and type of user U1 detected by the scene detection unit 132, but is similar to the first embodiment in other respects.

[0070] The occupant determination unit 137 determines the type of the user U1 based on the output of the in-vehicle camera 21 and the output of the in-vehicle microphone 22. The occupant determination unit 137 determines the type of the user U1 using at least one of an image analysis method such as machine learning and pattern recognition and a voice analysis method such as machine learning and pattern recognition. In the machine learning image analysis method, a learning model is used that uses a face image of the user U1 as input data and outputs the type of the user U1. In the image analysis method such as pattern recognition, a matching determination is made between the face image of the user U1 and a pattern image for each type of person. In the machine learning voice analysis method, a learning model is used that uses the voice of the user U1 as input data and outputs the type of the user U1. In the voice analysis method such as pattern recognition, a matching determination is made between the voice of the user U1 and a pattern voice for each type of person. When both the image analysis method and the voice analysis method are used, for example, a comprehensive determination may be made by weighting the image analysis result and the voice analysis result. In the case of machine learning, instead of performing image analysis and voice analysis separately, the image and voice may be analyzed comprehensively using a learning model that uses the face image and voice of the user U1 as input data and outputs the type of the user U1. The occupant determination unit 137 may store (register) the relationship between the type of the user U1 and the identification information of the user U1, and determine the type of the user U1 based on the identification information of the user U1. Examples of the identification information of the user U1 include a signal transmitted from a mobile device or wearable device possessed by the user U1 (identification information of the device owner (owner information registered in the device) or device identification information (substituted as user identification information)), and biometric information of the user U1 (user-specific information that can identify the user, such as facial features, iris features, and fingerprint features).

[0071] The storage unit 12 stores a plurality of weighting value tables. For example, as shown in Fig. 7, the storage unit 12 stores weighting value tables for various users, such as weighting value table 121a, weighting value table 121b, weighting value table 121c, weighting value table 121d, etc., shown in Fig. 3. For ease of explanation, in weighting value table 121b shown in Fig. 7, data storage cells (storage frames) in which values ​​different from those in weighting value table 121a shown in Fig. 3 (also shown in Fig. 7) are stored are colored gray.

[0072] The weighting value table 121a is a table from which weighting values ​​are extracted by the weighting value extracting unit 133 when the user U1 is a young couple. The weighting value table 121b is a table from which weighting values ​​are extracted by the weighting value extracting unit 133 when the user U1 is a single senior citizen.

[0073] For example, since it is considered that seniors are more likely to speak up than young people when they are stuck in a traffic jam or when they are subjected to aggressive driving, in weighting value table 121b shown in FIG. 7, the weighting value of the output of in-vehicle microphone 22 in a traffic jam scene and the weighting value of the output of in-vehicle microphone 22 in a scene where they are subjected to aggressive driving are made higher than in weighting value table 121a shown in FIG. 3.

[0074] For example, when driving at high speeds or on snowy roads, it is considered that seniors are more likely to speak out and have a stern expression than younger people, so in weighting value table 121b shown in FIG. 7, the weighting value of the output of in-vehicle camera 21 in high-speed driving scenes, the weighting value of the output of in-vehicle microphone 22 in high-speed driving scenes, the weighting value of the output of in-vehicle camera 21 in snowy road driving scenes, and the weighting value of the output of in-vehicle microphone 22 in snowy road driving scenes are made higher than those in weighting value table 121a shown in FIG. 3.

[0075] Here, two weighting value tables 121a and 121b are illustrated, but when the types of user U1 determined by the occupant determination unit 137 are three or more, a weighting value table for each type of user U1, that is, three or more weighting value tables, may be stored in the storage unit 12 (FIG. 7 shows an example in which four types of tables are clearly shown). Each weighting value table 121 is created by, for example, determining appropriate weighting values ​​by performing a sampling experiment on the target users and storing them in each table.

[0076] FIG. 8 is a flowchart showing an example of processing executed by the estimation device 10 according to the second embodiment.

[0077] The flowchart shown in Fig. 8 is a flowchart in which steps S11 and S12 are added to the flowchart shown in Fig. 4. In the flowchart shown in Fig. 8, steps S11 and S12 are provided between steps S10 and S20. Note that the order of execution of steps S10 and S11 may be reversed. Alternatively, steps S10 and S11 may be executed simultaneously.

[0078] When the scene surrounding the user U1 is detected by the scene detection unit 132 in step S10, the process proceeds to the next step S11.

[0079] In step S11, the occupant determination unit 137 (controller 13) determines the type of the user U1 by using the outputs (sensor signals) of the in-vehicle camera 21 and the in-vehicle microphone 22. When the type of the user U1 is determined by the occupant determination unit 137, the process proceeds to the next step S12.

[0080] In step S12, the weighting value extraction unit 133 (controller 13) selects a weighting value table corresponding to the type of the user U1 determined by the occupant determination unit 137. When the weighting value table is selected by the weighting value extraction unit 133, the process proceeds to the next step S20.

[0081] In step S20, the weighting value extraction unit 133 (controller 13) extracts a weighting value corresponding to the scene detected by the scene detection unit 132 from the weighting value table selected in step S20. When the weighting value extraction unit 133 extracts the weighting value, the process proceeds to the next step S30.

[0082] The process from step S30 onwards is similar to that in the first embodiment, and therefore the description will be omitted.

[0083] In this embodiment, the resource adjustment unit 134 performs resource adjustment according to the scene detected by the scene detection unit 132 and the type of the user U1 determined by the occupant determination unit 137. This is expected to achieve a balance between the load of the emotion estimation process and the accuracy of emotion estimation that is more appropriate for the user than in the first embodiment.

[0084] <3. Learning model> 9 is a diagram showing an example of the configuration of a learning model generation device 30 that generates the above-mentioned learning model 122. The learning model generation device 30 may be built into the estimation device 10, or may be a device separate from the estimation device 10.

[0085] The learning model generation device 30 includes an emotion estimation unit 31, an emotion estimation model 32, a resource adjustment unit 33, a weighting value table 34, and a learning model generation unit 35.

[0086] The learning model generation device 30 includes, for example, a controller and a storage unit. The controller realizes, as its functions, an emotion estimation unit 31, a resource adjustment unit 33, and a learning model generation unit 35 by processing operations based on a program, and the storage unit stores the emotion estimation model 32, the weighting value table 34, the program, etc.

[0087] The multiple types of biosignals are multiple types of biosignals of a subject for creating learning data, measured by a biosensor.

[0088] The biosensor includes a first sensor and a second sensor. The first sensor is a brainwave sensor. The second sensor is a heart rate (pulse) sensor. However, the first sensor and the second sensor may be changed to other biosensors depending on the bioinformation to be acquired. The other biosensor may be, for example, a blood pressure monitor, a NIRS (Near Infrared Spectroscopy) device, etc.

[0089] The biosensor may be composed of only a sensor worn by the data creation subject, for example. The biosensor may also be composed of a sensor worn by the data creation subject and a sensor equipped in the vehicle, for example. The biosensor may also be composed of only a sensor equipped in the vehicle, for example. The sensor worn by the data creation subject may be, for example, a headgear-type brainwave sensor, a wristband-type optical heart rate (pulse) sensor, etc. The sensor equipped in the vehicle may be, for example, an optical heart rate (pulse) sensor installed on the steering wheel, etc.

[0090] The biosensor may be configured as a single sensor, not a plurality of sensors, so long as it is capable of measuring a plurality of types of biosignals of the subject of data creation.

[0091] The emotion estimation unit 31 estimates the emotion of a data creation subject based on a plurality of types of biosignals.

[0092] The emotion deduction unit 31 generates an index value, which is a value of an index (physiological response) indicating a mental and physical state, based on a plurality of types of input biological signals. The index value, which is a value of an index (physiological response) indicating a mental and physical state, is also an emotion index value related to emotion. In this embodiment, the emotion deduction unit 31 generates index values ​​of indexes indicating two mental and physical states related to electroencephalograms and heartbeats. For example, the index indicating the mental and physical state related to electroencephalograms is a central nervous system arousal level (hereinafter, simply referred to as arousal level), and the index value can be given by "β waves / α waves of electroencephalograms". In addition, for example, the index indicating the mental and physical state related to heartbeats is an activity level of the autonomic nervous system, and the index value can be given by "standard deviation of heartbeat LF (low frequency range) component". The index value is calculated using a calculation model (calculation formula or conversion data table) stored in advance in a memory or the like.

[0093] The biosensor may be configured to have a calculation function (a configuration having a built-in computer or the like) and to calculate the index value.

[0094] The emotion estimation unit 31 estimates the emotion of the data creation subject based on multiple types of emotion index values ​​of the data creation subject. More specifically, the emotion estimation unit 31 estimates the emotion of the data creation subject using the calculated index values ​​and the emotion estimation model 32. The emotion estimation model 32 is created based on medical evidence (papers, etc.).

[0095] FIG. 10 is a diagram showing an example of an emotion estimation model (mental plane). According to various medical evidence related to psychology, it is said that mentality can be estimated based on two types of indices that indicate physical conditions. The psychological plane with the two types of indices of mental and physical conditions shown in FIG. 10 as axes is an example of an emotion estimation model according to the technical idea. In FIG. 10, the vertical axis is "arousal level (arousal-unaroused)" and the horizontal axis is "activity level of the autonomic nervous system (sympathetic nervous activity (strong emotion)-parasympathetic nervous activity (weak emotion)" and is an example of an emotion estimation model devised by the applicant of the present application that is expected to have high estimation accuracy.

[0096] In the psychological plane shown in FIG. 10, psychological states are assigned to each of the four quadrants separated by the vertical and horizontal axes. The distance from each axis indicates the intensity of the corresponding psychological state. In the example of FIG. 10, the psychological states of "fun, joy, anger, sadness" are assigned to the first quadrant. The psychological state of "melancholy" is assigned to the second quadrant. The psychological states of "relaxation, calmness" are assigned to the third quadrant. The psychological states of "anxiety, fear, unpleasantness" are assigned to the fourth quadrant.

[0097] The psychological state can be estimated from the coordinates obtained by plotting two types of mental and physical state index values ​​(emotion index values) obtained based on the measurement results of biosignals on a psychological plane. Specifically, the psychological state and its intensity can be estimated based on which quadrant of the psychological plane the plotted coordinates are in and how far they are from the origin.

[0098] In this example, in a model in which the vertical axis represents "alertness (awake-unaroused): calculated, for example, based on the beta / alpha waves of the brainwaves" and the horizontal axis represents "activity of the autonomic nervous system (sympathetic activity (strong emotion)-parasympathetic activity (weak emotion): calculated, for example, based on the standard deviation of the heart rate LF component," the subject's emotions are estimated to be "fun, joy, anger, sadness," "melancholy," "relaxed, calm" or "anxiety, fear, unpleasantness" depending on which quadrant the subject's state is in. This model has been verified as significant through experiments etc. by the applicant.

[0099] In the example shown in FIG. 10, the emotion estimation model is a two-dimensional plane, but it may be a three- or higher-dimensional space.

[0100] Furthermore, since error in emotion estimation is larger than in physical measurements, etc., it is possible to take into account the error in the emotion estimation result and each parameter / index related to emotion estimation, set a neutral region NR as shown in Figure 10, and make a judgment that emotion estimation is impossible (no emotion).

[0101] The learning model generation device 30 stores a sensor table 36 and a psychological plane table 37 corresponding to the emotion estimation model 32 in a storage unit (not shown in FIG. 9) in the learning model generation device 30. FIG. 11 is a diagram showing an example of the sensor table 36. FIG. 12 is a diagram showing an example of the psychological plane table 37.

[0102] 11, the items of the sensor table 36 include “sensor ID”, “sensor type”, “biological signal type”, “corresponding index ID”, “corresponding index type”, and “index conversion information”. The items of the table correspond to data storage cells (storage slots).

[0103] The item "sensor ID" of the sensor table 36 stores sensor ID data, which is identification information for identifying a data record in the sensor table 36. The sensor ID data is also the primary key of the data record in the sensor table 36. That is, in the sensor table 36, a data record is configured for each sensor ID data, and data of each item linked to the sensor ID data is stored in the data record.

[0104] The item "sensor type" of the sensor table 36 stores information for identifying the type of the biosensor. In this example, the sensor name (or data such as model number) is stored.

[0105] The item "biosignal type" of the sensor table 36 stores the type of measurement value based on the biosignal detected by the biosensor. This biosignal type data is data correlated with the corresponding index type data. Academically, it is recognized that the corresponding index type data can be estimated (calculated) by acquiring the corresponding biosignal type data.

[0106] The item "Corresponding index ID" of the sensor table 36 stores identification information for identifying a mental and physical state index generated (calculated) based on a signal from a biosensor that detects a biosignal. The item "Corresponding index type" of the sensor table 36 stores the type of the index (such as a name).

[0107] The item "index conversion information" of the sensor table 36 stores conversion information (such as an arithmetic formula or a conversion data table) for calculating an index value based on a signal obtained from a biosensor that detects a biosignal. In other words, the index value of the mental and physical condition index identified by the corresponding index ID is estimated (calculated) by converting the biosignal detected by the biosensor corresponding to the sensor ID data according to the index conversion information.

[0108] For example, the data record of the sensor ID "SN01" in the sensor table 36 shown in Fig. 11 has the following information. "Electroencephalogram beta / alpha waves" are measured by the output signal of "electroencephalogram sensor BA". Then, this "electroencephalogram beta / alpha waves" is converted using the index conversion information of "FX01" to obtain an index value of "wakefulness".

[0109] As shown in Fig. 12, the psychological plane table 37 is a two-dimensional matrix table with index types (specifically, index ID data is used) as parameters of the vertical and horizontal axes. In the psychological plane table 37, data of psychological plane types that can be used with the two index type data are stored in memory cells (memory frames) determined by two index types. For example, if the index types used as indexes are index type VSm and index type VSn, the psychological plane used for emotion estimation is psychological plane mn. Information for performing processing using psychological plane mn is read out and used for emotion estimation processing.

[0110] In addition, the sensor table 36 and the psychological plane table 37 use a common index ID. That is, based on the sensor table 36 and the psychological plane table 37, the psychological plane mn corresponding to the two types of biosensors attached to the user U1 can be determined. For example, the types of sensors attached to the data creation subject are "brain wave sensor BA (index ID: VS01)" and "heartbeat sensor HA (index ID: VS02)". In this case, the index ID data corresponding to "brain wave sensor BA (index ID: VS01)" and "heartbeat sensor HA (index ID: VS02)" are determined based on the sensor table 36. Then, based on the psychological plane table 37, the psychological plane 01-02 with "arousal level" (VS01) and "autonomic nervous system activity level" (VS02) as indexes is determined as the psychological plane to be used for emotion estimation.

[0111] The emotion estimation unit 31 selects a model to be used for estimating emotion candidates based on the biosignal of the biosensor signal. Specifically, the emotion estimation unit 31 compares the biosensor signal with the sensor table 36 to extract data of the “corresponding index type” (index type data) corresponding to the sensor type and biosignal type included in the biosensor signal. Then, the emotion estimation unit 31 compares the “index ID” ​​of the extracted index type data with the psychological plane table 37 to select a model (psychological plane) to be used for estimating emotion candidates. For example, when the sensor IDs are “SN01” (sensor type: brainwave sensor BA) and “SN02” (sensor type: heartbeat sensor HA), the corresponding index IDs are “VS01” (index type: arousal level) and “VS02” (index type: autonomic nervous system activity level) (see FIG. 11). Based on these, the emotion estimation unit 31 selects the psychological plane 01-02 as the emotion estimation model to be used.

[0112] In addition, when the type of sensor to be used by the learning model generation device 30 is predetermined (when the sensor type is fixedly connected (installed) to the learning model generation device 30, etc.), the emotion estimation model to be used is also a predetermined emotion estimation model (emotion estimation model selection process is not required).

[0113] The emotion deduction unit 31 executes an emotion candidate deduction process using the selected psychological plane. The emotion deduction unit 31 converts information based on the biosensor signal into an index value using index conversion information in the sensor table 36. In this embodiment, an index value of the arousal level and an index value of the autonomic nervous system activity level are obtained using data obtained from the first sensor (sensor type: electroencephalogram sensor BA) and the second sensor (sensor type: heart rate sensor HA).

[0114] Then, the emotion estimation unit 31 estimates emotion candidates according to the coordinate positions obtained by plotting each index value on the selected psychological plane. In this embodiment, emotion estimation is performed according to the coordinate positions obtained by plotting the index value of the arousal level and the index value of the autonomic nervous system activity level on a psychological plane (see FIG. 10) with the arousal level on the vertical axis and the autonomic nervous system activity level on the horizontal axis. Note that, as a specific method for determining the area in which the coordinates of each index value are located (area in which an emotion is set), a method of calculating the positional relationship between each area boundary and each index value based on each function formula corresponding to each area boundary, or a method of extracting an area corresponding to each combination of index values ​​using a data table simulating a psychological plane (a data table in which area values ​​(corresponding to emotions) are stored for each combination of index values) can be applied. As a method for determining an emotion from the area determined in this manner, a method of determining an emotion corresponding to an area using a data table in which areas and emotions are associated can be applied.

[0115] Moreover, the feeling estimation unit 31 obtains a plurality of types of index values. In this embodiment, the plurality of types is two. However, a configuration may be adopted in which three or more types of index values ​​are obtained. In this case, the feeling estimation model may be a three or more dimensional space, rather than a two-dimensional plane.

[0116] The resource adjustment unit 33 acquires scene information and type information of the data creation subject when a plurality of types of biosensor signals of the data creation subject for creating learning data are obtained. The weighting value table 34 is a weighting value table for the type of the data creation subject, similar to the weighting value tables 121a to 121d in FIG. 7. However, unlike the weighting value tables 121a to 121d in FIG. 7, the weighting value table 34 does not have learning model data. The resource adjustment unit 33 selects a weighting value table corresponding to the type of the data creation subject based on the acquired scene information and type information of the data creation subject. Then, the resource adjustment unit 33 uses the selected weighting value table to perform resource adjustment on at least a part of the output of the in-vehicle camera 21, the output of the in-vehicle microphone 22, the output of the outside camera 23, the output of the outside microphone 24, and the weather information, and supplies the resource-adjusted data to the learning model generation unit 35.

[0117] The learning model generation unit 35 uses as input data at least a part of the output of the in-vehicle camera 21, the output of the in-vehicle microphone 22, the output of the outside camera 23, the output of the outside microphone 24, and the weather information after the resource adjustment unit 33 has been performed, and performs machine learning with the learning data in which the emotion estimated by the emotion estimation unit 31 based on the biological signal at the same timing as the input data is used as correct answer data to generate a learning model. Furthermore, each input data becomes data whose resources have been adjusted according to the scene. Therefore, the generated learning model is a learning model that estimates emotions based on data from each sensor whose resources have been adjusted according to the scene.

[0118] Then, from the learning models for each scene thus generated, a model is selected based on the scene in which emotion estimation is to be performed, and used for emotion estimation. Therefore, an appropriate learning model according to the scene is used, and the emotion estimation accuracy can be improved.

[0119] For example, when generating a learning model corresponding to a traffic jam scene, the output of the in-car camera 21 with the detail level adjusted to a high level, the output of the in-car microphone 22 with the detail level adjusted to a low level, and the output of the outside camera 23 with the detail level adjusted to a medium level are used as input data for resource adjustment corresponding to the traffic jam scene, and the learning model is trained with learning data in which emotions estimated based on biological signals at the same timing as the input data are used as correct answer data to generate a trained learning model. This results in a learning model that estimates emotions based on data from each sensor with resources adjusted according to the traffic jam scene. This learning model is then used for emotion estimation in the traffic jam scene.

[0120] Instead of the learning model generation device 30 shown in FIG. 9, a learning model generation device 40 shown in FIG. 13 may be used.

[0121] The learning model generation device 40 shown in FIG. 13 includes a resource adjustment unit 41, a weighting value table 42, a first learning model generation unit 43, and a second learning model generation unit 44.

[0122] The learning model generation device 40 includes, for example, a controller and a memory unit, and the controller includes, as its functions, a resource adjustment unit 41, a first learning model generation unit 43, and a second learning model generation unit 44, and the memory unit stores a weighting value table 42, a program, etc.

[0123] The resource adjustment unit 41 acquires scene information and type information of the data creation subject when the arousal state index value and the emotion intensity index value of the data creation subject for creating learning data are obtained. The arousal state index value and the emotion intensity index value are calculated based on the biological signal of the data creation subject as described above. The weighting value table 42 is a weighting value table for the type of the data creation subject similar to the weighting value tables 121a to 121d in FIG. 7. However, unlike the weighting value tables 121a to 121d in FIG. 7, the weighting value table 42 does not have learning model data. The resource adjustment unit 41 selects a weighting value table corresponding to the type of the data creation subject based on the acquired scene information and type information of the data creation subject. Then, the resource adjustment unit 41 performs resource adjustment on at least a part of the output of the in-vehicle camera 21, the output of the in-vehicle microphone 22, the output of the outside camera 23, the output of the outside microphone 24, and the weather information using the selected weighting value table, and supplies the resource-adjusted data to the first learning model generation unit 43 and the second learning model generation unit 44.

[0124] The first learning model generation unit 43 uses as input data at least a part of the output of the in-vehicle camera 21, the output of the in-vehicle microphone 22, the output of the outside camera 23, the output of the outside microphone 24, and the weather information after the resource adjustment unit 33 has been performed, and performs machine learning with the first learning data in which the alertness index value indicating the alertness state of the data creation subject at the same timing as the input data is the first correct answer data to generate a first learning model. In addition, each input data becomes data whose resources have been adjusted according to the scene. Therefore, the generated first learning model becomes a learning model that estimates an alertness index value based on data from each sensor whose resources have been adjusted according to the scene.

[0125] The second learning model generation unit 44 uses as input data at least a part of the output of the in-vehicle camera 21, the output of the in-vehicle microphone 22, the output of the outside camera 23, the output of the outside microphone 24, and the weather information after the resource adjustment unit 33 has been performed, and performs machine learning with the second learning data in which the emotion intensity index value indicating the intensity of the emotion of the data creation subject at the same timing as the input data is the second correct answer data to generate a second learning model. In addition, each input data becomes data whose resources have been adjusted according to the scene. Therefore, the generated second learning model becomes a learning model that estimates an emotion intensity index value based on data from each sensor whose resources have been adjusted according to the scene.

[0126] Then, from the first learning model and the second learning model for each scene thus generated for each scene, a model is selected based on the scene when emotion estimation is performed, and used for emotion estimation (used to estimate the arousal state index value and the emotion intensity index value used for emotion estimation). Therefore, an appropriate learning model according to the scene is used, and the emotion estimation accuracy can be improved.

[0127] For example, when generating a first learning model corresponding to a traffic jam scene, the output of the in-vehicle camera 21 with a high level of detail, the output of the in-vehicle microphone 22 with a low level of detail, and the output of the outside camera 23 with a medium level of detail are used as input data for resource adjustment corresponding to the traffic jam scene, and the first learning model is trained with the first learning data in which the alertness index value at the same timing as the input data is used as correct answer data to generate a trained learning model. This results in a learning model that estimates an alertness index value based on data from each sensor with resources adjusted according to the traffic jam scene. This learning model is then used for emotion estimation in a traffic jam scene.

[0128] Similarly, for example, when generating a second learning model corresponding to a traffic jam scene, the output of the in-car camera 21 with the detail level adjusted to a high level, the output of the in-car microphone 22 with the detail level adjusted to a low level, and the output of the outside camera 23 with the detail level adjusted to a medium level are used as input data for resource adjustment corresponding to the traffic jam scene, and the second learning model is trained with the second learning data in which the emotion intensity index value at the same timing as the input data is used as the correct answer data to generate a trained learning model. This results in a learning model that estimates the emotion intensity index value based on the data from each sensor with the resource adjusted according to the traffic jam scene. This learning model is then used for emotion estimation in the traffic jam scene.

[0129] When the configuration of the estimation device 10 is changed to a configuration having the above-mentioned first learning model and second learning model, the storage unit 12 of the estimation device 10 may store the above-mentioned emotion estimation model. The controller 13 (resource adjustment unit 134) performs resource adjustment processing (using a weighting value table corresponding to the scene (and user)) on at least a part of the output of the in-vehicle camera 21, the output of the in-vehicle microphone 22, the output of the outside camera 23, the output of the outside microphone 24, and the weather information according to the scene. The controller 13 (emotion estimation unit 135) obtains an arousal state index value output from the first learning model by applying the output data of the in-vehicle camera 21 subjected to the resource adjustment processing, etc., to the first learning model. Similarly, the controller 13 (resource adjustment unit 134) performs resource adjustment processing (using a weighting value table corresponding to the scene (and user)) on at least a part of the output of the in-vehicle camera 21, the output of the in-vehicle microphone 22, the output of the outside camera 23, the output of the outside microphone 24, and the weather information according to the scene. The controller 13 (emotion estimation unit 135) obtains an emotion intensity index value output from the second learning model by applying the output data of the in-vehicle camera 21, for which the resource adjustment process has been performed, to the second learning model. Then, the controller 13 (emotion estimation unit 135) applies the arousal state index value and the emotion intensity index value to the emotion estimation model (model shown in FIG. 10) to estimate the emotion of the user U1.

[0130] <4. Things to keep in mind> Various technical features disclosed in the description of the invention may be modified in various ways without departing from the spirit of the technical creation. Furthermore, the embodiments and modifications disclosed in the description of the invention may be combined to the extent possible.

[0131] For example, the estimation process of the estimation device 10 may be started when the vehicle 20 starts control using the estimation result of the estimation device 10. In this way, the processing load on the estimation device 10 can be reduced.

[0132] In the above-described embodiment, an emotion estimation model for each scene is used in the emotion estimation process, but an emotion estimation model common to each scene may be used in the emotion estimation process. In this case, if the emotion estimation model is a model in which the processing load changes depending on the presence or absence of input data and the image resolution (a model including processing according to the image resolution (including processing to skip processing of non-existent dots) including processing to skip related processing for data types that are not input), the processing load and memory usage of the controller can be reduced.

[0133] For example, a learning model common to each scene may be a learning model trained by machine learning using learning data that uses as input data the output of in-vehicle camera 21 with the level of detail adjusted to a high level, the output of in-vehicle microphone 22 with the level of detail adjusted to a high level, the output of outside-vehicle camera 23 with the level of detail adjusted to a high level, the output of outside-vehicle microphone 24 with the level of detail adjusted to a high level, and weather information, and uses as correct answer data the emotions of the person to be used to create the learning data at the time the input data was acquired.

[0134] For example, the learning model common to each scene may be a learning model machine-learned using learning data in which input data includes the output of in-vehicle camera 21 with the level of detail adjusted to a high level, the output of in-vehicle microphone 22 with the level of detail adjusted to a high level, the output of exterior camera 23 with the level of detail adjusted to a high level, the output of exterior microphone 24 with the level of detail adjusted to a high level, weather information, the weighting value of the output of in-vehicle camera 21, the weighting value of the output of in-vehicle microphone 22, the weighting value of the output of exterior camera 23, the weighting value of the output of exterior microphone 24, and the weighting value of the weather information, and the emotions of the person to be used to create the learning data at the time the input data was acquired are used as correct answer data. In this case, the output of the in-vehicle camera 21, the output of the in-vehicle microphone 22, the output of the exterior camera 23, the output of the exterior microphone 24, weather information, a weighting value of the output of the in-vehicle camera 21, a weighting value of the output of the in-vehicle microphone 22, a weighting value of the output of the exterior camera 23, a weighting value of the output of the exterior microphone 24, and a weighting value of the weather information are supplied to the emotion estimation unit 135, and the emotion estimation unit 135 also serves as the resource adjustment unit 134. [Explanation of symbols]

[0135] 1. Estimation system 10... Estimation device 11. Communications Department 12...Storage section 13. Controller 20. Vehicle 21. In-car camera 22. In-car microphone 23. Exterior camera 24. Outside microphone 25...In-vehicle equipment U1...User

Claims

1. Detect the scene surrounding the emotion estimation subject, An estimation device that performs resource adjustment to adjust an amount of resources consumed for emotion estimation processing that estimates emotions of the emotion estimation subject based on outputs of a plurality of emotion estimation sensors, in accordance with the detected scene.

2. The estimation device according to claim 1 , wherein the resource adjustment is selection of the emotion estimation sensor to be used in the emotion estimation process.

3. The estimation device according to claim 1 , wherein the resource adjustment is a detail adjustment for at least a part of an output of the emotion estimation sensor used in the emotion estimation process.

4. The emotion estimation process is a process performed by a learning model that receives outputs from a plurality of emotion estimation sensors as input data and outputs emotions, 4. The estimation device according to claim 1, wherein the learning model used in the emotion estimation process is selected from a plurality of learning models corresponding to scenes in accordance with the detected scene.

5. The estimation device according to claim 1 , wherein the resource adjustment is performed in accordance with the detected scene and a type of the person whose emotion is to be estimated.

6. An estimation device having a learning model that estimates emotions of an emotion estimation subject based on outputs of a plurality of emotion estimation sensors, The learning model is Calculating an alertness index value representing an alertness state based on the brain waves of a data creation subject for creating learning data; calculating an emotion intensity index value representing an emotion intensity based on the heart rate of the data generation subject; estimating an emotion by case-by-case classification of a combination of the binarized value of the arousal state index value and the binarized value of the emotion intensity index value; generating learning data using outputs from a plurality of emotion estimation sensors at timings corresponding to timings of measuring the electroencephalogram and the heart rate as input data and using the estimated emotion as correct answer data; A pre-training model is generated by training the training data, Detect the scene surrounding the emotion estimation subject, performing resource adjustment to adjust an amount of resources consumed for an emotion estimation process for estimating an emotion of the emotion estimation target person based on outputs of a plurality of emotion estimation sensors in accordance with the detected scene; the emotion estimation device estimates the emotion of the emotion estimation subject by applying outputs of the plurality of emotion estimation sensors, for which the resource adjustment has been performed, to the learning model, in the emotion estimation process.

7. An estimation device comprising: a first learning model that estimates an arousal state index value representing an arousal state of a subject of emotion estimation based on outputs from a plurality of emotion estimation sensors; and a second learning model that estimates an emotion intensity index value representing an intensity of an emotion, and estimates an emotion by case-by-case classification of a combination of a binary value of the arousal state index value and a binary value of the emotion intensity index value, The first learning model is Calculating an alertness index value representing an alertness state based on the brain waves of a data creation subject for creating learning data; generating first learning data using outputs from a plurality of emotion estimation sensors at timings corresponding to the electroencephalogram measurement timings as input data and using the calculated arousal state index value as correct answer data; A pre-training model is generated by training the first training data, The second learning model is calculating an emotion intensity index value representing an emotion intensity based on the heart rate of the data generation subject; generating second learning data using outputs from a plurality of emotion estimation sensors at timings corresponding to the heartbeat measurement timings as input data and using the calculated emotion intensity index value as correct answer data; A pre-learning model is generated by training the pre-learning model with the generated second training data, Detect the scene surrounding the emotion estimation subject, performing resource adjustment to adjust an amount of resources consumed for an emotion estimation process for estimating an emotion of the emotion estimation target person based on outputs of a plurality of emotion estimation sensors in accordance with the detected scene; the emotion estimation device estimates the emotion of the emotion estimation subject by applying outputs of the plurality of emotion estimation sensors, for which the resource adjustment has been performed, to the first learning model and the second learning model in the emotion estimation process.

8. A plurality of emotion estimation sensors installed in a vehicle; a feeling estimation device that estimates a feeling of a subject of feeling estimation according to each output of the feeling estimation sensor; an in-vehicle control device that controls an in-vehicle device in accordance with the emotion estimated by the emotion estimation device, The emotion estimation device includes: Detect the scene surrounding the emotion estimation subject, performing resource adjustment to adjust an amount of resources consumed for an emotion estimation process for estimating an emotion of the emotion estimation target person based on outputs of the plurality of emotion estimation sensors in accordance with the detected scene; An in-vehicle system that controls an in-vehicle device based on the emotion of the emotion estimation subject estimated in the emotion estimation process.

9. Detecting a scene surrounding a subject for emotion estimation; performing resource adjustment to adjust an amount of resources consumed for an emotion estimation process for estimating an emotion of the emotion estimation target person based on outputs of a plurality of emotion estimation sensors in accordance with the detected scene; performing the emotion estimation process; The estimation program causes a computer to execute the above.

Images