Biological information estimation device, biological information estimation method, and biological information estimation program

Abstract

To improve the accuracy of biometric information estimation. [Solution] The biometric information estimation device installed in the vehicle includes a computer that performs the following actions: acquiring image data of the driver (user); inputting the acquired user image data into a trained model that takes the image data as input and outputs biometric information, in order to estimate the user's biometric information. The trained model includes a trained model for right-hand drive vehicles and a trained model for left-hand drive vehicles, and the computer switches between the trained model for right-hand drive vehicles and the trained model for left-hand drive vehicles based on information representing the position of the vehicle's steering wheel.

Description

Technical Field

[0001] The present disclosure relates to a biological information estimation device such as emotion mounted on a vehicle, a biological information estimation method, and a biological information estimation program.

Background Art

[0002] Conventionally, an emotion estimation system that estimates emotions based on biological signals (for example, heartbeat and brain waves) is known (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In emotion estimation using biological signals, data on heartbeat and brain waves obtained by contact-type sensors is used. However, when a driver uses the system while driving a vehicle, the driver often finds it troublesome to wear contact-type sensors. Therefore, in the case of an in-vehicle device, it is convenient to use data obtained non-contact, such as a face image, as input data to the system. However, a camera mounted in the vehicle interior is often attached near the center in the vehicle width direction, such as near the rearview mirror. In such a case, due to the positional relationship between the camera and the driver, there is a bias in the portion of the driver's face captured in the image. Such a problem also occurs when estimating a biological state other than emotion using an image.

[0005] An object of the present technology is to provide a technology for improving the accuracy of output values when using an image of a user captured from obliquely in front as input data to a machine learning model for estimating a biological state mounted on an in-vehicle device.

Means for Solving the Problems

[0006] The biometric information estimation device installed in the vehicle includes a computer that performs the following actions: acquiring image data of the driver (user); inputting the acquired user image data into a trained model that takes the image data as input and outputs biometric information, in order to estimate the user's biometric information. The trained model includes a trained model for right-hand drive vehicles and a trained model for left-hand drive vehicles, and the computer switches between the trained model for right-hand drive vehicles and the trained model for left-hand drive vehicles based on information representing the position of the vehicle's steering wheel.

[0007] Furthermore, the biometric information estimation device installed in the vehicle includes a computer that performs the following actions: acquiring image data of the driver (user); inputting the acquired user image data into a trained model that takes the image data as input and outputs biometric information, in order to estimate the user's biometric information. The trained model is suitable for either right-hand drive or left-hand drive vehicles. The computer processor, based on information representing the position of the vehicle's steering wheel, determines that the vehicle is one of the two types and inputs the user's image data to the trained model after horizontally inverting it. If the vehicle is determined to be one of the two types, the computer processor inputs the user's image data to the trained model without horizontally inverting it. [Effects of the Invention]

[0008] This technology reduces the impact of differences in user-captured images between left-hand and right-hand drive vehicles, thereby improving the accuracy of biometric information estimation. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a diagram illustrating the conceptual configuration of an emotion estimation device. [Figure 2] Figure 2 is a diagram illustrating an example of image data created by a camera. [Figure 3] Figure 3 is a block diagram showing an example of the configuration of a vehicle control system installed in vehicle 1. [Figure 4] Figure 4 is a flowchart showing the emotion estimation process performed by the emotion estimation device's processor. [Figure 5] Figure 5 is a diagram illustrating the machine learning method. [Figure 6] Figure 6 is a flowchart showing an example of the learning process performed by the learning device. [Figure 7] Figure 7 is a flowchart showing the emotion estimation process related to the modified form. [Figure 8] Figure 8 is a flowchart showing an example of the learning process related to the modified example. [Modes for carrying out the invention]

[0010] Embodiments of this disclosure will be described below with reference to the drawings. However, the configurations of the following embodiments are illustrative, and the present invention is not limited to the configurations of these embodiments.

[0011] <Overview> First, we will describe an emotion estimation device, which is an example of a biometric information estimation device. Figure 1 is a configuration diagram showing the conceptual configuration of the emotion estimation device. A vehicle 1, such as an automobile, is equipped with an emotion estimation device 11, a camera 12, and a microphone (hereinafter referred to as "microphone") 13.

[0012] Camera 12 is an imaging device that converts light into electrical signals using an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), and creates and outputs image data. Camera 12 outputs video data of the interior of the vehicle, including user 2, who is the driver of vehicle 1, to the emotion estimation device 11.

[0013] Furthermore, camera 12 is installed near the center of the vehicle 1 in the width direction, such as near the rearview mirror, and photographs the interior of the vehicle. At this time, depending on the position of the steering wheel of vehicle 1, the position in which user 2 is captured in the image data will be biased to either the left or the right. Figure 2 is a diagram to explain an example of image data created by the camera. Figure 2 (A) in the upper part shows an example of a left-hand drive vehicle 1A. Camera 12A, installed near the rearview mirror, is positioned diagonally in front of user 2 to the right, and outputs image data 121A in which the right half of user 2's body (left side from user 2's perspective) is captured relatively broadly. In other words, image data 121A can be said to have higher resolution data on the right side of user 2's face than on the left side (in other words, there is a bias in the data obtained on the left and right sides of user 2). Figure 2 (B) in the upper part shows an example of a right-hand drive vehicle 1B. Camera 12B, installed near the rearview mirror, is positioned diagonally in front of user 2 to the left, and generates image data 121B in which the left half of user 2's body (right side from user 2's perspective) is captured relatively broadly. In other words, image data 121B shows that the left side of user 2's face has higher resolution than the right side.

[0014] The microphone 13 in Figure 1 is an audio input device that converts sound into electrical signals using methods such as dynamic, condenser, or piezoelectric types. The microphone 13 is also installed inside the vehicle and collects sound from inside the vehicle, including the voice of user 2, and outputs the audio data to the emotion estimation device 11.

[0015] The emotion estimation device 11 is a computer mounted on the vehicle 1. The emotion estimation device 11 uses video data input from the camera 12 and audio data input from the microphone 13 to estimate the emotions of user 2 and outputs information representing the estimated emotions. AI (Artificial Intelligence) models (1st AI model 1122 and 2nd AI model Emotion estimation is performed using (1123) and the emotion estimation model (1124).

[0016] The first AI model 1122 is a machine learning model that estimates the central nervous system arousal level (hereinafter referred to as "arousal level") based on the image data input via the camera 12 and the voice data input via the microphone 13. The arousal level is an index value that can be calculated by "β wave / α wave of electroencephalogram". The first AI model 1122 is constructed by performing machine learning using the image data and voice data of the subject as input data and the arousal level corresponding to the input data as correct answer data. In the example of FIG. 3, the first AI model 1122 includes the first AI model (left) 11221 for a left-handle vehicle and the first AI model (right) 11222 for a right-handle vehicle.

[0017] The second AI model 1123 is a machine learning model that estimates the activity level of the autonomic nervous system (hereinafter referred to as "activity level") based on the image data input via the camera 12 and the voice data input via the microphone 13. The activity level is an index value that can be calculated as "standard deviation of the LF (Low Frequency) component of the heartbeat (low-frequency component of the heartbeat waveform signal)". The second AI model 1123 is also constructed by performing machine learning using the image data and voice data of the subject as input data and the activity level corresponding to the input data as correct answer data. In the example of FIG. 3, the second AI model 1123 includes the second AI model (left) 11231 for a left-handle vehicle and the second AI model (right) 11232 for a right-handle vehicle.

[0018] ​The AI models are each constructed as pre-trained models through prior machine learning and stored in the storage device of the emotion estimation device 11. Also, in the present embodiment, the AI models include AI models for left-hand drive vehicles (first AI model (left) 11221, second AI model (left) 11231) and AI models for right-hand drive vehicles (first AI model (right) 11222, second AI model (right) 11232), and the model to be used is determined according to the position of the steering wheel of the vehicle 1. That is, when the user 2 is photographed by the camera 12 installed near the center in the width direction of the vehicle 1, there is a bias in the data obtained on the left and right of the user 2 in the image data, and when a single general-purpose AI model is used, the accuracy of the output result may not be sufficient. In the present embodiment, by making the AI models for left-hand drive vehicles and right-hand drive vehicles switchable, the accuracy of the output result can be improved.

[0019] The emotion estimation model 1124 of the emotion estimation device 11 estimates emotions based on a plurality of emotion index values that are indicators of the mental and physical states related to emotions. The emotion index values used in the present embodiment are the arousal level and activity level described above. The emotion estimation model 1124 is composed of a model for emotion estimation (calculation formula and conversion data table) based on the arousal level and activity level. The emotion estimation model 1124 is composed of a multi-dimensional model (here, a two-dimensional model with the arousal level and activity level as two axes) that estimates emotions using the arousal level and activity level as parameters. The two-dimensional model is created based on medical evidence (papers, etc.) showing the relationship between each of the plurality of indicators and emotions (the relationship between the arousal level / activity level and emotions). Alternatively, the two-dimensional model is created based on the questionnaire results of many subjects (data consisting of the emotion declarations by the subjects and the arousal level / activity level (based on electroencephalogram / heart rate measurement values) at that time). Note that the emotion estimation model 1124 can also be a multi-dimensional model of three dimensions or more, not just a two-dimensional model.

[0020] Figure 1 shows an example of a multi-dimensional (2D) model of emotion estimation (also called the "psychological plane") within the emotion estimation model 1124. According to various medical evidence related to psychology, psychology can be estimated based on two types of indicators that show the physical state. In the psychological plane shown in Figure 1, the vertical axis is "arousal level (aroused-unaroused)" and the horizontal axis is "autonomic nervous system activity level (sympathetic nervous system activity (strong emotion)-parasympathetic nervous system activity (weak emotion))."

[0021] In this psychological plane, each of the four quadrants separated by the vertical and horizontal axes is assigned a corresponding psychological state. The distance from each axis indicates the intensity of the corresponding psychological state. The first quadrant is assigned to the psychological states of "happiness, joy, anger, and sadness." The second quadrant is assigned to the psychological state of "melancholy." The third quadrant is assigned to the psychological states of "relaxation and calmness." The fourth quadrant is assigned to the psychological states of "anxiety, fear, and unpleasantness."

[0022] The axis position will be set appropriately based on experiments (measuring the arousal and activity levels of subjects and performing statistical processing), etc. Possible methods include using the center of the neutral region determined by a predetermined method as the axis, or measuring the range of fluctuations of normalized emotion index values ​​using a predetermined normalization method for many subjects and using the average of the median values ​​of these measured fluctuation ranges as the axis.

[0023] Furthermore, by plotting two types of indicator values ​​for mental and physical states (arousal level and activity level), obtained based on biosignals, on a psychological plane, the psychological state can be estimated from the resulting coordinates. Specifically, the psychological state and its intensity can be estimated based on which quadrant the plotted coordinates lie in on the psychological plane, their position within that quadrant, and their distance from the origin. Note that the emotion estimation model 1124 shown in Figure 1 is a two-dimensional plane, but it can become a multi-dimensional space of three or more dimensions depending on the number of indicators used.

[0024] Since estimating emotional intensity involves relatively large errors, it may be better to determine only the type of emotion based on the quadrant in which the emotional index value is located on the psychological plane, and then use the determined emotional type in subsequent processing.

[0025] Furthermore, when emotional intensity is strong, that is, when the emotional index value swings significantly towards the maximum or minimum value, the accuracy of emotion estimation increases. However, when emotional intensity is weak, that is, when the emotional index value is near the median, the accuracy of emotion estimation decreases. For this reason, the region near the median of the emotional index value may be designated as the neutral region, and a judgment of "no estimated emotion" or "emotion estimation impossible" may be made. In other words, in the psychological plane shown in the emotion estimation model 1124 in Figure 1, for example, a predetermined range of "activity level" centered on the vertical axis and a predetermined range of "arousal level" centered on the horizontal axis are designated as the neutral region. The upper and lower limits of the neutral region for the emotional index of "arousal level," and the upper and lower limits of the neutral region for the emotional index of "activity level," can be appropriately set based on experiments, for example.

[0026] The emotion estimation device 11 estimates emotions by applying the input emotion indices "arousal level" and "activity level" to the emotion estimation model 1124 shown on the psychological plane described above (plotting them as coordinates).

[0027] A typical emotion estimation model takes electroencephalogram (EEG) data and heart rate data as input and estimates emotions using the arousal and activity levels of emotion indices calculated based on the EEG and heart rate data. On the other hand, the emotion estimation device 11 according to this embodiment estimates emotions using the arousal and activity levels of emotion indices estimated by an AI model based on images and sound, instead of EEG and heart rate data. For example, since it is not practical to have a vehicle driver wear a contact-type sensor, this embodiment uses a camera 12 and a microphone 13, which are non-contact sensors, making it particularly suitable for applications such as estimating the emotions of a driver.

[0028] <Vehicle control system> Next, a vehicle control system that performs vehicle control using estimated emotion data will be explained with reference to Figure 3. Figure 3 is a block diagram showing an example of the configuration of a vehicle control system 10 installed in vehicle 1. Figure 3 shows components characteristic of this embodiment, The general components are not described.

[0029] The vehicle control system 10 comprises an emotion estimation device 11, a vehicle control device 14, an actuator 15, and a notification unit 16. The vehicle control system 10 may further include a user interface (UI) such as an input device like a keyboard or touch panel, or an output device like a display.

[0030] <Emotion estimation device> The emotion estimation device 11 comprises a processor 111, a memory unit 112, and a communication unit 113. Since the emotion estimation device 11 generally requires immediate control response, it is mounted on the vehicle 1 as in this example, but it can also be configured as a server connected to the vehicle 1 via a network.

[0031] The processor 111 is a processing unit that includes at least one of the following: CPU (Central Processing Unit), MCU (Micro Controller Unit), MPU (Micro Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), ASIC (Application Specific IC), ASSP (Application Specific Standard Product), etc. The processor 111 controls the operation of the emotion estimation device 11 and functions as an acquisition unit 1111, an emotion estimation unit 1112, and a provision unit 1113. The functions of the processor 111 are realized by the processor executing arithmetic processing according to the program stored in the memory unit 112. Details of the functions will be described later.

[0032] The memory unit 112 includes main memory such as RAM (Random Access Memory) and ROM (Read Only Memory), and auxiliary memory (secondary memory) such as HDD (Hard Disk Drive), SSD (Solid State Drive), and flash memory. The main memory temporarily stores programs read by the processor 111 and reserves workspace for the processor 111. The auxiliary memory stores programs executed by the processor 111, steering wheel position 1121 representing the position of the steering wheel of the vehicle 1, trained AI models (first AI model 1122 and second AI model 1123), emotion estimation models including the psychological plane shown in Figure 1, and other data. In other words, a computer is configured to perform various controls with the processor 111 and the memory unit 112 as its main components.

[0033] The communication unit 113 is an interface, such as CAN (Controller Area Network), for communicating with the camera 12, microphone 13, vehicle control device 14, etc.

[0034] Furthermore, the acquisition unit 1111 acquires image data captured by the camera 12 and audio data collected by the microphone 13 via the communication unit 113. The acquisition unit 1111 also stores the acquired data in the storage unit 112. The acquisition unit 1111 stores the image data and audio data from corresponding time periods (essentially the same time period) as a single dataset in a data table. This data is then used to estimate emotions at the end of that time period.

[0035] The emotion estimation unit 1112 reads the steering wheel position 1121 from the memory unit 112 and, depending on whether the steering wheel position 1121 indicates left or right, reads either an AI model for a left-hand drive vehicle (first AI model (left) 11221, second AI model (left) 11231) or an AI model for a right-hand drive vehicle (first AI model (right) 11222, second AI model (right) 11232) from the memory unit 112. Note that the steering wheel position 1121 information is stored in the memory unit 112 as vehicle information during vehicle manufacturing, etc., but the steering wheel position may also be obtained by image analysis processing (image recognition of the steering wheel) of image data of the vehicle interior captured by the camera 12. Then, the emotion estimation unit 1112 estimates the emotion of user 2 using the read AI model. Specifically, the acquisition unit 1111 inputs the acquired image data and audio data to the first AI model 1122 and the second AI model 1123, respectively. As a result, the first AI model 1122 and the second AI model 1123 output the emotion index values ​​(arousal and activity levels) of user 2 estimated based on the image data and audio data. The emotion estimation unit 1112 reads the emotion estimation model 1124 from the memory unit 112 and inputs the estimated emotion index values ​​of user 2 output by the first AI model 1122 and the second AI model 1123 into the emotion estimation model 1124. Then, the emotion estimation unit 1112 can determine user 2's emotion corresponding to the emotion index values ​​of user 2 as an estimation result from the emotion estimation model 1124.

[0036] Furthermore, the data used for the emotion estimation described above (image data, audio data, estimated emotion index values, etc.) may be data obtained by statistically processing (averaging, low-pass filtering, etc.) data over an appropriate period of time, and emotions may be estimated using this data. In this case, the statistical processing is performed appropriately by the acquisition unit 1111, the emotion estimation unit 1112, etc., depending on the type of data to be processed.

[0037] The provisioning unit 1113 provides the vehicle control device 14 with information representing the emotions of user 2, which has been estimated by the emotion estimation unit 1112. This enables the vehicle control device 14 to control vehicle 1 based on the emotions of user 2.

[0038] <Vehicle control devices, etc.> The vehicle control device 14 is, for example, an ECU (Electronic Control Unit) for vehicle control. The vehicle control device 14 is installed in vehicle 1. The vehicle control device 14 comprises a processor 141, a storage unit 142, and a communication unit 143.

[0039] Processor 141 is a processing unit similar to, for example, processor 111. Processor 141 may perform various vehicle controls based on driving operations by user 2, who is the driver of vehicle 1, and information from various connected sensors. Examples of connected sensors include a vehicle speed sensor, an air-fuel ratio sensor, and a steering angle sensor. Examples of various vehicle controls include controlling the direction and speed of vehicle 1. Processor 141 also receives information representing user 2's emotions (estimated emotions) from emotion estimation device 11 via communication unit 143. Processor 141 may then perform vehicle controls, such as controlling speed, accelerator sensitivity, and brake sensitivity, using the received estimated emotion information. Specifically, when the driver is excited, processor 141 performs driving control that is on the safe side, that is, reduces the influence of the driver's level of excitement, by lowering the accelerator sensitivity (making it harder to accelerate), increasing the brake sensitivity (making it easier to stop), and lowering the upper limit of the maximum speed control.

[0040] Furthermore, the processor 141 provides notification of the estimated emotion information itself, as well as notification corresponding to that estimated emotion information. Specifically, if the driver is in an excited state, the processor 141 displays or provides voice guidance such as "Let's calm down," and controls the quality of the voice in various voice guidance to be in a calm tone and expression.

[0041] The memory unit 142, like the memory unit 112, is composed of various types of memory and stores data used by the processor 141. For example, programs, processing coefficient data, and temporary storage data during processing are stored therein. The memory unit 142 is also provided with multiple data tables for various processing, which associate information related to emotions received from the emotion estimation device 11 with control signals of the vehicle 1 and notification information to the user 2 inside the vehicle.

[0042] The communication unit 143 is an interface for communicating data between the emotion estimation device 11, the actuator 15, and the notification unit 16, and is, for example, a CAN (Controller Area Network) interface.

[0043] The actuator 15 consists of various drive components, such as motors, that realize various operations of the vehicle, and its operation is driven and controlled by the vehicle control device 14. Specifically, for example, the actuator 15 includes the engine and motor that generate driving force in the vehicle 1, the steering actuator that drives the steering of the vehicle 1, and the brake actuator that drives the brakes of the vehicle 1.

[0044] The notification unit 16 notifies the user 2 inside the vehicle of notification information from the vehicle control device 14 by visual, auditory, or other means. Specifically, for example, the notification unit 16 consists of a liquid crystal display that conveys notification information to the user 2 using text, images, or videos, and a speaker that conveys notification information using voice, warning sounds, or the like.

[0045] As described above, the vehicle control system 10 estimates the emotions of user 2, who is the driver of vehicle 1, and performs the operation of vehicle 1, such as driving control, taking the estimated emotions into consideration. With this configuration, vehicle 1 can be controlled in accordance with user 2's emotions. In this embodiment, emotion indicators are estimated using AI models that correspond to the steering wheel position. Depending on the positional relationship between camera 12 and user 2, the estimation accuracy can be improved by using AI models appropriate for cases where the left side of user 2's face is captured more prominently in the image data, and cases where the right side is captured more prominently.

[0046] <Emotion estimation processing> Figure 4 is a flowchart illustrating the emotion estimation process performed by the processor 111 (computer) of the emotion estimation device. The emotion estimation process is performed by the processor 111 of the emotion estimation device 11 executing a computer program. This computer program is stored and provided on a computer-readable non-volatile recording medium. This computer program may consist of only one program, or it may consist of multiple programs working together.

[0047] The process shown in Figure 4 is initiated and repeatedly executed at the timing when various controls based on estimated emotions are started after vehicle 1 has started.

[0048] In step S1, the processor 111 (emotion estimation unit 1112) acquires handle position information by reading the handle position 1121 from the memory unit 112.

[0049] After step S1, in step S2, the processor 111 (emotion estimation unit 1112) determines, for example, whether the vehicle is right-hand drive. If it is determined to be right-hand drive (S2: YES), in step S3, the processor 111 reads out AI models for right-hand drive vehicles (first AI model (right) 11222 and second AI model (right) 11232) from the memory unit 112. If it is determined in S2 that the vehicle is not right-hand drive (i.e., left-hand drive) (S2: NO), in step S4, the processor 111 reads out AI models for left-hand drive vehicles (first AI model (left) 11221 and second AI model (left) 11231) from the memory unit 112.

[0050] After step S3 or S4, in step S5, the processor 111 (acquisition unit 1111) acquires data indicating the user 2's behavior, specifically image data obtained from the camera 12 and audio data obtained from the microphone 13, and stores them in the storage unit 112.

[0051] After step S5, in step S6, the processor 111 (emotion estimation unit 1112) processes the image data and audio data acquired in step S5 into the first AI model 1122 (i.e., the first AI model (right) 11222 read out in step S3, or step S The first AI model (left) 11221) read out in step 4 is input to obtain an emotional index of arousal level. The processor 111 (emotion estimation unit 1112) also inputs the image data and audio data acquired in step S5 to the second AI model 1123 (i.e., the second AI model (right) 11232 read out in step S3, or the second AI model (left) 11231 read out in step S4) to obtain an emotional index of activity level.

[0052] After step S6, in step S7, the processor 111 (emotion estimation unit 1112) applies (inputs) the two types of emotion index values ​​(arousal level, activity level) obtained in step S6 to the emotion estimation model 1124 to estimate the emotion of user 2.

[0053] After step S7, in step S8, the processor 111 (providing unit 1113) provides (outputs) the estimated emotion information of user 2 estimated by the emotion estimation unit 1112 to the vehicle control device 14.

[0054] Furthermore, after step S8, in step S9, the processor 111 (emotion estimation unit 1112) determines whether to terminate the process. For example, if the vehicle control device 14 receives notification that it has terminated various controls based on estimated emotions (typically, when the termination of vehicle operation (based on engine stoppage, etc.) is detected), the processor 111 determines to terminate the emotion estimation process. If it is determined to terminate the process (S9: YES), the process shown in Figure 4 is terminated. On the other hand, if it is determined to repeat the process (not terminate) (S9: NO), the process returns to step S5 and is repeated.

[0055] Furthermore, since the emotions estimated for each individual instance may be negatively affected (misjudgment) by noise in the sensor output, it is preferable to use estimated emotions that have undergone statistical processing, such as adopting the average or mode over a predetermined period. For this reason, it is preferable for the processor 111 (providing unit 1113) to add a timestamp to the estimated emotions, store them, and provide the statistically processed estimated emotions to the vehicle control device 14 (step S8). It is also possible for the vehicle control device 14 to store the estimated emotion information, perform statistical processing on it, and use the results for control.

[0056] <Machine Learning Methods> Next, the learning methods for the first AI model 1122 and the second AI model 1123 will be described. Figure 5 is a diagram illustrating the machine learning method. User 6, who is the driver of vehicle 5, is the subject for generating training data. The subject is not necessarily the same as the user of the emotion estimation device 11. Also, vehicle 5 is not necessarily the same model as vehicle 1, which is equipped with the emotion estimation device 11, and the position of the steering wheel may be different. Vehicle 5 may also be a driving simulator.

[0057] The vehicle 5 in Figure 5 is equipped with an on-board device 51, a camera 52, and a microphone 53. The camera 52 is the same device as the camera 12 shown in Figure 1. The microphone 53 is the same device as the microphone 13 shown in Figure 1. The user 6 drives the vehicle 5 wearing an electroencephalogram (EEG) sensor 54 to detect brain waves and a heart rate sensor 55 to detect heart rate. The data (learning data) output by the camera 52, microphone 53, EEG sensor 54, and heart rate sensor 55 during driving is stored in the on-board device 51. For example, the EEG sensor 54 may be a headgear-type EEG sensor. For example, the heart rate sensor 55 may be a chest belt-type electrocardiogram heart rate sensor. Other biosensors may be added or changed depending on the biometric information to be acquired, wearability, etc. Other biosensors may include, for example, an optical heart rate (pulse) sensor, a blood pressure monitor, or a NIRS (Near Infrared Spectroscopy) device. The on-board device 51 is a computer equipped with a processor and a memory unit. The processor and memory unit are the same as the processor 111 and memory unit 112 of the emotion estimation device 11 shown in Figure 3, but the processing performed and the data stored are different. The data is different.

[0058] The learning data is stored in association with information such as the date and time of creation and the position of the steering wheel of vehicle 5. The learning data is also copied from the in-vehicle device 51 to the learning device 7. The learning device 7 is a computer equipped with a processor and a memory unit, and creates the first AI model 1122 and the second AI model 1123 by performing machine learning using the learning data. The processor and memory unit of the learning device 7 are similar to the processor 111 and memory unit 112 of the emotion estimation device 11 shown in Figure 3, but the processing performed and the data stored are different. In the example in Figure 5, the in-vehicle device 51 that stores the learning data and the learning device 7 that performs machine learning are separate devices, but the in-vehicle device 51 and the learning device 7 may be a single integrated device.

[0059] The processor of the learning device 7 functions as an index value calculation unit 71 that calculates index values. The index value calculation unit 71 calculates the level of alertness using the electroencephalogram (EEG) data output by the EEG sensor 54. The index value calculation unit 71 also calculates the activity level using the heart rate data output by the heart rate sensor 55. The calculation method is as described above, with the level of alertness calculated using the "beta / alpha waves of the EEG" and the activity level calculated using the "standard deviation of the heart rate LF (Low Frequency) component (low-frequency component of the heart rate waveform signal)". These sentiment indicators may also be normalized values.

[0060] The first AI model 1122 is created by supervised learning, using image data and audio data as input values ​​and arousal level as the ground truth value. The second AI model 1123 is also created by supervised learning, using image data and audio data as input values ​​and activity level as the ground truth value. The learning device 7 uses learning data (especially image data) obtained from the left-hand drive vehicle 5 to create AI models for left-hand drive vehicles (first AI model (left) 11221 and second AI model (left) 11231), and uses learning data (especially image data) obtained from the right-hand drive vehicle 5 to create AI models for right-hand drive vehicles (first AI model (right) 11222 and second AI model (right) 11232). The AI ​​models thus created (trained models) are mounted on the emotion estimation device 11 shown in Figure 1. By creating AI models for left-hand drive vehicles and AI models for right-hand drive vehicles and making them switchable in the emotion estimation device 11 shown in Figure 1, the accuracy of the output results of the emotion estimation device 11 can be improved.

[0061] <Learning Process> Figure 6 is a flowchart illustrating an example of the learning process performed by the learning device 7. The process shown in Figure 6 is executed based on an operation to start the learning process by the user of the learning device 7. It is also assumed that the learning device 7's memory unit has learning data (image data, audio data, electroencephalogram data, heart rate data) pre-stored in it.

[0062] In step S11, the processor of the learning device 7 reads the learning data from the memory unit. The learning data includes information on the handle position associated with the learning data (e.g., image data). The learning data is read out as a single unit consisting of a dataset for a predetermined period.

[0063] After step S11, in step S12, the processor determines whether the vehicle 5 used to create the training data is, for example, right-hand drive. If it is determined to be right-hand drive (S12: YES), in step S13, the processor classifies the read training data as training data for right-hand drive vehicles. If it is determined in S12 that it is not right-hand drive (i.e., left-hand drive) (S12: NO), in step S14, the processor classifies the read training data as training data for left-hand drive vehicles.

[0064] After step S13 or step S14, in step S15, the processor performs machine learning processing. In this step, the processor calculates the arousal level and activity level (or normalized arousal level and activity level) of the emotion index values ​​based on the electroencephalogram and heart rate data of the biological signals. Alternatively, the arousal level and activity level of the emotion index values ​​may be calculated and stored based on the acquired electroencephalogram and heart rate data of the biological signals before this process (for example, when collecting training data), and these emotion index values ​​may be used. The processor also performs supervised learning using image data and audio data as input values, with the arousal level as the correct value, and updates the parameters of the first AI model 1122, for example. At this time, the training data classified for right-hand drive vehicles in S13 is used to construct the first AI model (right) 11222 for right-hand drive vehicles, and the training data classified for left-hand drive vehicles in S14 is used to construct the first AI model (left) 11221 for left-hand drive vehicles. Similarly, the processor takes image data and audio data as input values ​​and performs supervised learning with activity levels as the correct values, updating the parameters of, for example, the second AI model 1123. At this time, the training data classified for right-hand drive vehicles in S13 is used to construct the second AI model (right) 11232 for right-hand drive vehicles, and the training data classified for left-hand drive vehicles in S14 is used to construct the second AI model (left) 1123 for left-hand drive vehicles.

[0065] After step S15, in step S16, the processor decides whether to terminate the learning process. For example, if there is no unused learning data in the memory, the processor terminates the learning process (step S15: YES). On the other hand, if there is unused learning data in the memory, the processor decides not to terminate the learning process (step S15: NO) and returns to step S11 to continue processing.

[0066] Furthermore, after the learning process is completed, the trained first AI model 1122 and second AI model 1123 are stored in the emotion estimation device 11 of the vehicle 1 shown in Figure 1.

[0067] Alternatively, by reversing the image data horizontally, the training data collected for one type of vehicle (right-hand drive or left-hand drive) can be classified as training data for the other type. For example, in step S13, the image data classified as training data for right-hand drive vehicles is reversed horizontally in step S12 and used as training data for left-hand drive vehicles. Similarly, in step S14, the image data classified as training data for left-hand drive vehicles is reversed horizontally in step S12 and used as training data for right-hand drive vehicles. In this way, even if there is a bias in the available training data, the same amount of training data can be prepared for both right-hand drive and left-hand drive vehicles. Note that this method of creating training data may be applied to only one of the two types of vehicles (right-hand drive or left-hand drive). That is, if there is a bias in the available training data, the training data for the vehicle lacking training data can be supplemented by reversing the image data of the other vehicle horizontally.

[0068] <Example 1> The AI ​​model installed in the emotion estimation device 11 may be either for left-hand drive vehicles or right-hand drive vehicles. If the steering wheel position of vehicle 1 does not match the intended use of the AI ​​model, the image data from camera 12 is horizontally flipped and input to the AI ​​model.

[0069] Figure 7 is a flowchart showing the emotion estimation process related to the modified example. Note that the same reference numerals are used for processes corresponding to those in Figure 4, and their explanations are omitted. Furthermore, it is assumed that the AI ​​model is predetermined to be for either left-hand drive or right-hand drive vehicles.

[0070] In step S21, the processor 111 (emotion estimation unit 1112) Read the steering wheel position 1121 and the AI ​​model from there.

[0071] Following step S21, in step S23, the processor 111 (acquisition unit 1111) acquires data indicating the user 2's behavior, specifically image data obtained from the camera 12 and audio data obtained from the microphone 13, and stores them in the storage unit 112. This step is the same as step S5 in Figure 4.

[0072] After step S22, in step S23, the processor 111 (emotion estimation unit 1112) determines whether the vehicle's steering wheel position matches the intended use of the AI ​​model (whether it is for left-hand drive or right-hand drive). If it is determined that they do not match (S23: NO), in step S24, the processor 111 reverses the image data horizontally.

[0073] If a match is determined in step S23 (S23: YES), or if the process from step S6 onwards is performed after step S24, the process from step S6 onwards is the same as in Figure 4.

[0074] Figure 8 is a flowchart showing an example of the learning process related to the modified example. Note that the same reference numerals are used for processes corresponding to those in Figure 6, and their explanations are omitted. Furthermore, it is assumed that the AI ​​model is predetermined to be for either left-hand drive or right-hand drive vehicles.

[0075] In step S31, the processor of the learning device 7 reads the learning data (image data, audio data, electroencephalogram data, heart rate data) from the memory unit. The learning data includes information on the handle position associated with the learning data (e.g., image data). This step is the same as step S11 in Figure 6.

[0076] After step S31, in step S32, the processor determines whether the vehicle's steering wheel position matches the intended use of the AI ​​model (whether it is for left-hand drive or right-hand drive). If it is determined that they do not match (S32: NO), in step S33, the processor reverses the image data included in the training data horizontally.

[0077] If it is determined in step S32 that there is a match (S32: YES), or if the process from step S15 onwards is performed after step S33, the process from step S15 onwards is the same as in Figure 6.

[0078] According to this modified version, emotion estimation can be performed using a single model.

[0079] <Modification 2> The AI ​​models installed in the emotion estimation device 11 are not limited to those for left-hand drive and right-hand drive vehicles. For example, more types of AI models may be prepared based on the positional relationship between the camera 12 and the user 2. For example, the position of the user 2's face (in other words, the headrest of the driver's seat) relative to the position of the camera 12 may be mapped to coordinates in three-dimensional space. In this case, the X, Y, and Z coordinates may be classified into three stages, and AI models may be created for 27 possible patterns, with the appropriate AI model being used depending on the installation position of the camera 12 in the vehicle 1.

[0080] This modified version can also improve the accuracy of emotion estimation.

[0081] <Other> While embodiments have been described above, this disclosure is not limited thereto, and various modifications based on the knowledge of those skilled in the art are possible without departing from the spirit of the claims. The configurations shown in the embodiments and modified examples can be combined as appropriate.

[0082] In the embodiments and modifications described above, an emotion estimation model 1124 was used to estimate emotions using emotion index values ​​(arousal level and activity level) as parameters, but the method is not limited to this. That is, an AI model that directly estimates emotions based on image data may be created and installed in the emotion estimation device 11. An AI model that directly estimates emotions can be created, for example, by learning the relationship between emotions estimated based on brain waves and heart rate during training and image data, etc. The method for estimating emotions based on brain waves and heart rate is not limited to the embodiments described above, and various methods can be adopted. Furthermore, the accuracy of the estimation can be improved by creating an AI model for right-hand drive vehicles and an AI model for left-hand drive vehicles.

[0083] Furthermore, the technologies shown in the embodiments and modified examples can be used not only for estimating emotions, but also for estimating alertness, drowsiness, heart rate, and various other biometric information. In such cases, the accuracy of the estimation can be improved by creating AI models for right-hand drive vehicles and AI models for left-hand drive vehicles.

[0084] Furthermore, the present invention includes a computer program that performs the processing method described above, and a computer-readable recording medium on which the program is recorded. By having a computer read the program stored on the recording medium and execute the program, a device equipped with the computer can realize the above-described operation. A computer-readable recording medium refers to a recording medium that stores information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer. Examples of such recording media that are removable from a computer include flexible disks, magneto-optical disks, optical disks, magnetic tapes, and memory cards. Examples of recording media fixed to a computer include hard disk drives and ROMs. In addition, a method is also applicable in which a computer program is stored on a server's storage medium (hard disk drive, etc.), and the server distributes the computer program to an in-vehicle device, etc., via a communication network environment. [Explanation of Symbols]

[0085] 1: Vehicle, 10: Vehicle control system 11: Emotion estimation device, 111: Processor, 1111: Acquisition unit, 1112: Emotion estimation unit, 1113: Provision unit, 112: Memory unit, 1121: Handle position, 1122: First AI model, 1123: Second AI model, 1124: Emotion estimation model, 113: Communication unit 12: Camera, 13: Microphone 14: Vehicle control device, 141: Processor, 142: Memory unit, 143: Communication unit 15: Actuator, 16: Notification Department 5: Vehicle, 51: In-vehicle device, 52: Camera, 53: Microphone, 54: EEG sensor, 55: Heart rate sensor 7: Learning device, 71: Index value calculation unit

Claims

1. A biometric information estimation device mounted on a vehicle, Obtaining image data of the user who is the driver, The process involves inputting the acquired user's image data into a trained model that takes image data as input and outputs biometric information, in order to estimate the user's biometric information. Equipped with a computer that runs, The aforementioned trained model includes a trained model for right-hand drive vehicles and a trained model for left-hand drive vehicles. The computer switches between the learned model for a right-hand drive vehicle and the learned model for a left-hand drive vehicle based on information representing the position of the vehicle's steering wheel. A device for estimating biological information.

2. The aforementioned trained model for right-hand drive vehicles is a trained model constructed by machine learning using supervised training data generated based on biometric information derived from biosignals acquired from a subject driving a right-hand drive vehicle, and image data of the subject acquired at a timing corresponding to the acquisition of said biosignals. The aforementioned trained model for left-hand drive vehicles is a trained model constructed by machine learning using supervised training data generated based on biometric information derived from biosignals acquired from a subject driving a left-hand drive vehicle, and image data of the subject acquired at a timing corresponding to the acquisition of said biosignals. The biological information estimation device according to claim 1.

3. A biometric information estimation device mounted on a vehicle, Obtaining image data of the user who is the driver, The process involves inputting the acquired user's image data into a trained model that takes image data as input and outputs biometric information, in order to estimate the user's biometric information. Equipped with a computer that runs, The aforementioned trained model is suitable for either right-hand drive or left-hand drive vehicles. If the computer determines, based on information representing the position of the vehicle's steering wheel, that the vehicle is one of the two types, it inputs the user's image data into the trained model after horizontally flipping it; if the computer determines that the vehicle is one of the two types, it inputs the user's image data into the trained model without horizontally flipping it. A device for estimating biological information.

4. The aforementioned trained model is a trained model constructed by machine learning using supervised training data generated based on biometric information obtained from a subject driving a vehicle that is either a right-hand drive or left-hand drive vehicle, and image data of the subject obtained at a timing corresponding to the acquisition of the biometric signals. The biological information estimation device according to claim 3.

5. The aforementioned biological information is emotion. A biological information estimation device according to any one of claims 2 to 4.

6. A method for estimating biometric information performed by a computer installed in a vehicle, Obtaining image data of the user who is the driver, The process involves inputting the acquired user's image data into a trained model that takes image data as input and outputs biometric information, in order to estimate the user's biometric information. Execute, The aforementioned trained model includes a trained model for right-hand drive vehicles and a trained model for left-hand drive vehicles. Furthermore, based on information representing the position of the vehicle's steering wheel, the system switches between the learned model for right-hand drive vehicles and the learned model for left-hand drive vehicles. Methods for estimating biological information.

7. A method for estimating biometric information performed by a computer installed in a vehicle, Obtaining image data of the user who is the driver, The process involves inputting the acquired user's image data into a trained model that takes image data as input and outputs biometric information, in order to estimate the user's biometric information. Execute, The aforementioned trained model is suitable for either right-hand drive or left-hand drive vehicles. Furthermore, based on information representing the position of the vehicle's steering wheel, if it is determined that the vehicle is different from the other, the user's image data is horizontally flipped and input into the trained model. If it is determined that the vehicle is one of the two, the user's image data is input into the trained model without horizontal flipping. Methods for estimating biological information.

8. A biometric information estimation program executed on a computer installed in a vehicle, Obtaining image data of the user who is the driver, The process involves inputting the acquired user's image data into a trained model that takes image data as input and outputs biometric information, in order to estimate the user's biometric information. The computer is made to execute the above, The aforementioned trained model includes a trained model for right-hand drive vehicles and a trained model for left-hand drive vehicles. Furthermore, based on information representing the position of the vehicle's steering wheel, the system will switch between the learned model for the right-hand drive vehicle and the learned model for the left-hand drive vehicle. A program for estimating biological information.

9. A biometric information estimation program executed on a computer installed in a vehicle, Obtaining image data of the user who is the driver, The process involves inputting the acquired user's image data into a trained model that takes image data as input and outputs biometric information, in order to estimate the user's biometric information. The computer is made to execute the above, The aforementioned trained model is suitable for either right-hand drive or left-hand drive vehicles. Furthermore, based on information representing the position of the vehicle's steering wheel, if it is determined that the vehicle is different from the other vehicle, the user's image data is horizontally flipped and input into the trained model. If it is determined that the vehicle is one of the two vehicles, the user's image data is input into the trained model without horizontal flipping. A program for estimating biological information.

Images