Information processing device, vehicle system, information processing method, and program

Abstract

To improve convenience for users of vehicles.SOLUTION: An information processing device provided herein estimates emotion of a driver of a vehicle on the basis of an image acquired by a camera mounted on the vehicle, and identifies a first point where a predetermined emotion is estimated as the emotion of the driver.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to video sharing services. [Background technology]

[0002] A technology has been proposed for sensing and utilizing the emotions of a vehicle driver. In this regard, Patent Document 1 discloses a system that estimates the driver's emotions and generates a route that does not cause discomfort to the driver. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-106530 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure aims to improve convenience for vehicle users. [Means for solving the problem]

[0005] A first aspect of the present disclosure is an information processing device having a control unit that performs the following operations: estimating an emotion of a driver of a vehicle based on an image acquired by a camera mounted on the vehicle; and identifying a first location where a predetermined emotion is estimated as the emotion of the driver.

[0006] A second aspect of the present disclosure is a vehicle system including an on-board device mounted in a vehicle and a server device that manages a plurality of the vehicles, wherein the on-board device has a first control unit that executes the following: estimating an emotion of a driver of the vehicle based on an image acquired by a camera mounted on the vehicle; and transmitting emotion data to the server device, the emotion data being data that associates the estimated emotion with location information; and the server device has a second control unit that generates data that compiles the emotions felt by each of the plurality of drivers for each location or road section, based on the emotion data received from the plurality of on-board devices.

[0007] Furthermore, a third aspect of the present disclosure is an information processing method including the steps of estimating an emotion of a driver of a vehicle based on an image acquired by a camera mounted on the vehicle, and identifying a first location where a predetermined emotion is estimated as the emotion of the driver.

[0008] Another aspect of the present disclosure is a program for causing a computer to execute the above-described information processing method, and a computer-readable storage medium that non-temporarily stores the program. Effect of the Invention

[0009] According to the present disclosure, convenience for vehicle users can be improved. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an overview of a vehicle system according to a first embodiment. [Diagram 2] 1 is a system configuration diagram of an in-vehicle device 100 according to a first embodiment. [Diagram 3] 4 shows an example of emotion data generated by the in-vehicle device 100. [Figure 4] 2 is a diagram for explaining data transmitted and received between modules of the in-vehicle device 100. [Diagram 5] 2 is a diagram for explaining data transmitted and received between modules of the in-vehicle device 100. [Figure 6] 3 shows an example of an emotion map generated by the in-vehicle device 100. [Figure 7] 4 is a flowchart of a first process executed by the in-vehicle device 100. [Figure 8] 10 is a flowchart of a second process executed by the in-vehicle device 100. [Figure 9] FIG. 11 is a diagram illustrating an overview of a vehicle system according to a second embodiment. [Figure 10] FIG. 11 is a system configuration diagram of an in-vehicle device 100 according to a second embodiment. [Figure 11] FIG. 11 is a system configuration diagram of a server device 200 according to a second embodiment. [Figure 12] FIG. 11 is a sequence diagram of data transmitted and received in the second embodiment. [Figure 13] 13 is an example of an emotion map generated in the second embodiment. [Figure 14] 13 shows an example of emotion data generated in the third embodiment. [Figure 15] 13 shows an example of emotion data generated in the fourth embodiment. [Figure 16] 13 is an example of an emotion map generated in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] There is a known system that supports driving based on an image obtained by capturing the face of a vehicle driver. For example, it is possible to detect whether the driver is feeling drowsy based on the face image of the driver and encourage the driver to take a break. It is also possible to detect the driver's emotions and provide appropriate information.

[0012] The information processing device according to the present disclosure provides a technique for providing appropriate route guidance based on the driver's emotions.

[0013] An information processing device according to one aspect of the present disclosure is characterized in having a control unit that performs the following operations: estimating an emotion of a driver of a vehicle based on an image acquired by a camera mounted on the vehicle; and identifying a first location where a predetermined emotion is estimated as the emotion of the driver.

[0014] The camera is, for example, an in-vehicle camera installed facing the inside of the vehicle, but is not limited to this as long as it can capture the face of the driver. For example, a camera equipped in an omnidirectional drive recorder that can capture images in 360 degrees can also be used. The control unit estimates the driver's emotion based on the image captured by the vehicle-mounted camera, and identifies a first location where the predetermined emotion is detected. The predetermined emotion may be any one or more of a plurality of predetermined emotions. For example, multiple emotions such as "anger," "irritation," "confusion," and "joy" may be targeted.

[0015] The estimation result may be stored in association with the first location. By accumulating such data (referred to as emotion data), it becomes possible to determine that a particular emotion tends to occur at a particular location (or road section). Based on the emotion data, the locations (or road sections) where drivers tend to have a particular emotion may be mapped onto a road map.

[0016] The control unit may transmit the emotion data to an external device that compiles emotion data. With this configuration, emotion data transmitted from a plurality of vehicles can be compiled by the external device. This makes it possible, for example, to identify road sections that are difficult for many drivers to pass through comfortably and generate a map that indicates this fact.

[0017] A vehicle system according to one aspect of the present disclosure includes an on-board device mounted on a vehicle and a server device that manages a plurality of the vehicles, the on-board device estimating an emotion of a driver of the vehicle based on an image acquired by a camera mounted on the vehicle, and transmitting emotion data, which is data in which the estimated emotion is associated with location information, to the server. and transmitting the emotion data to a server device, the server device having a second control unit that generates data in which the emotions felt by each of a plurality of drivers are compiled for each location or road section based on the emotion data received from the plurality of in-vehicle devices. In this way, the server device may compile information regarding emotions.

[0018] Specific embodiments of the present disclosure will be described below with reference to the drawings. Unless otherwise specified, the hardware configuration, module configuration, functional configuration, and the like described in each embodiment are not intended to limit the technical scope of the disclosure to only those.

[0019] First Embodiment An overview of a vehicle system according to a first embodiment will be described with reference to FIG. The vehicle system according to this embodiment includes an in-vehicle device 100 mounted on a vehicle. The in-vehicle device 100 has a camera capable of capturing images of the interior of the vehicle, and is configured to be able to estimate the driver's emotions based on images captured by the camera. The in-vehicle device 100 periodically estimates the driver's emotion while the vehicle is traveling, and when a specific emotion is detected, stores the result in association with the location information. The in-vehicle device 100 also maps the estimated emotion on a road map based on the stored data.

[0020] The in-vehicle device 100 will now be described in detail. The in-vehicle device 100 is a computer mounted on a vehicle. The in-vehicle device 100 may be a device that provides information to the vehicle occupants (for example, a car navigation device). The in-vehicle device 100 is also called a car navigation device, an infotainment device, or a head unit. The in-vehicle device 100 can provide navigation and entertainment to the vehicle occupants.

[0021] Furthermore, the in-vehicle device 100 accumulates data while the vehicle 10 is traveling, and provides information to a user (typically the driver) of the vehicle based on the accumulated data. In this embodiment, the in-vehicle device 100 detects the emotion of the driver of the vehicle 10, generates a road map in which the detected emotion is mapped for each location or road section, and outputs the road map. The in-vehicle device 100 includes a control unit 101, a storage unit 102, a communication unit 103, an input / output unit 104, a camera 105, and a position information acquisition unit 106.

[0022] The in-vehicle device 100 can be configured by a general-purpose computer. That is, the in-vehicle device 100 can be configured as a computer having a processor such as a CPU or a GPU, a main storage device such as a RAM or a ROM, and an auxiliary storage device such as an EPROM, a hard disk drive, or a removable medium. The auxiliary storage device stores an operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, various functions that match a predetermined purpose, as described below, can be realized. However, some or all of the functions may be realized by hardware circuits such as an ASIC or an FPGA.

[0023] The control unit 101 is a calculation device that controls the in-vehicle device 100. The control unit 101 can be realized by a calculation processing device such as a CPU (Central Processing Unit). . The control unit 101 is configured to have three functional modules: an emotion estimation unit 1011, a data generation unit 1012, and a map generation unit 1013. Each functional module may be realized by executing a stored program by a CPU.

[0024] The emotion estimation unit 1011 estimates the driving state of the vehicle 10 using the camera 105 described later while the vehicle 10 is moving. A facial image of the driver is acquired, and the driver's emotion is estimated based on the facial image. The emotion estimation can be performed using a known technique. For example, the emotion estimation unit 1011 converts the facial image into a feature amount, and inputs the obtained feature amount to a machine learning model for emotion estimation. For example, the machine learning model classifies the input feature amount into one of a plurality of classes, and outputs the result together with the likelihood. This makes it possible to obtain classified emotions and corresponding likelihoods. When there is a class for which a likelihood equal to or greater than a predetermined value is obtained, the emotion estimation unit 1011 can determine that the driver has an emotion corresponding to that class. The determination result is transmitted to the data generation unit 1012.

[0025] The data generation unit 1012 generates data (emotion data) that associates the estimated emotion with a location based on the result of the estimation performed by the emotion estimation unit 1011. 3 is an example of emotion data generated by data generating unit 1012. In this embodiment, the emotion data includes date and time information, a trip identifier (trip ID), location information, and an emotion identifier (emotion ID). The date and time information is the date and time when the emotion estimation is performed. A trip is a unit of travel from when the system power of the vehicle is turned on to when the system power is turned off. The data generating unit 1012 assigns an identifier (trip ID) corresponding to a new trip every time the system power of the vehicle is turned on. The position information is the position information (latitude, longitude) of the vehicle 10 acquired by a position information acquisition unit 106 described later. The emotion identifier is a predefined identifier. For example, if the emotion estimation unit 1011 can identify six types of emotions, one of the six identifiers is stored in the trip ID. Data generation unit 1012 generates emotion data every time emotion estimation is performed, and stores the generated emotion data in storage unit 102, which will be described later.

[0026] The map generation unit 1013 maps points or road sections where specific emotions occur onto a road map based on the stored emotion data, and outputs the result. Hereinafter, the road map onto which emotions are mapped is referred to as an emotion map. The emotion data used for mapping may be extracted based on any criteria. For example, emotion data generated during a certain period of time in the past may be extracted and used for mapping. The certain period of time may be specified by the user or may be determined by the system. By referring to the emotion map, the user of the vehicle 10 can ascertain locations or road sections where driving is comfortable, or locations or road sections where driving is stressful.

[0027] The storage unit 102 is a means for storing information, and is configured with a storage medium such as a RAM, a magnetic disk, or a flash memory. The storage unit 102 includes a main storage device and an auxiliary storage device. The main storage device is a memory in which the programs executed by the control unit 101 and data used by the control programs are expanded. The auxiliary storage device is a device in which the programs executed by the control unit 101 and data used by the control programs are stored. The auxiliary storage device may store the programs executed by the control unit 101 packaged as applications. It may also store an operating system for executing these applications. The programs stored in the auxiliary storage device are loaded into the main storage device and executed by the control unit 101, thereby performing the processing described below.

[0028] The storage unit 102 stores an estimation model 102A, emotion data 102B, and road data 102C.

[0029] The estimation model 102A is a machine learning model for estimating emotions. The estimation model 102A performs class classification using feature amounts obtained from an image including a human face as input, and outputs the result. For example, the estimation model 102A classifies the feature amounts into one of a plurality of predetermined emotions. Examples of the plurality of emotions include surprise, excitement, happiness, vigilance, satisfaction, relaxation, calmness, sleepiness, boredom, depression, pessimism, tension, excitement, and dissatisfaction. The estimation model 102A is constructed in advance based on image data for learning. Note that the estimation model 102A may be capable of outputting the likelihood together with the emotion that is the classification result.

[0030] Emotion data 102B is a collection of multiple pieces of emotion data generated by emotion estimation unit 1011. The road data 102C is road map data that is the basis for generating an emotion map. The road data 102C is data that defines, for example, the geographical positions and connection relationships of road links.

[0031] The communication unit 103 includes an antenna and a communication module for wireless communication. The antenna is an antenna element for inputting and outputting wireless signals. In this embodiment, the antenna is suitable for mobile communication (e.g., mobile communication such as 3G, LTE, and 5G). The antenna may be configured to include multiple physical antennas. For example, when performing mobile communication using radio waves in a high frequency band such as microwaves and millimeter waves, multiple antennas may be distributed and arranged to stabilize communication. The communication module is a module for performing mobile communication.

[0032] The input / output unit 104 is a means for accepting an input operation performed by a user and presenting information to the user. Specifically, it is composed of a touch panel and its control means, and a liquid crystal display and its control means. In this embodiment, the touch panel and the liquid crystal display are composed of one touch panel display. The input / output unit 104 may include a unit for outputting sound (an amplifier and a speaker), a unit for inputting sound (a microphone), etc.

[0033] The camera 105 is an optical unit including an image sensor for acquiring an image. In this embodiment, the camera 105 is installed at a position where an image including the face of the driver of the vehicle 10 (a facial image) can be acquired. The location information acquisition unit 106 includes a GPS antenna and a positioning module for measuring location information. The GPS antenna is an antenna that receives a positioning signal transmitted from a positioning satellite (also called a GNSS satellite). The positioning module is a module that calculates location information based on the signal received by the GPS antenna.

[0034] 2 is an example, and all or part of the functions shown in the figure may be executed using a dedicated circuit. Also, the programs may be stored or executed using a combination of a main memory device and an auxiliary memory device other than those shown in the figure.

[0035] Next, the process executed by the in-vehicle device 100 will be described in detail. 4 and 5 are diagrams for explaining the flow of data between components (modules) of the in-vehicle device 100. FIG. FIG. 4 corresponds to a process for generating emotion data based on a face image, and FIG. 5 corresponds to a process for generating an emotion map based on the emotion data.

[0036] First, the process of generating emotion data based on a face image (FIG. 4) will be described. The emotion estimation unit 1011 acquires a face image from the camera 105 while the vehicle 10 is traveling. The face image includes the face of the driver of the vehicle 10. The emotion estimation unit 1011 estimates the acquired face The image is converted into features and input to the estimation model 102B. As described above, the estimation model 102B is a machine learning model that performs class classification based on feature quantities. As a result, emotions to be classified (e.g., “satisfied,” “calm,” “gloomy,” “tense,” “dissatisfied,” etc.) and their likelihoods can be obtained. The emotion estimation unit 1011 estimates, for example, the emotion with the highest likelihood as the emotion felt by the driver. The classification result is transmitted to the data generation unit 1012.

[0037] The data generation unit 1012 generates emotion data as shown in FIG. 3 based on the classification result and the position information of the vehicle 10. 4 is executed periodically while the vehicle 10 is traveling. As a result, a plurality of records of emotion data are accumulated in the storage unit 102.

[0038] Next, a process for generating an emotion map based on emotion data will be described with reference to FIG. The map generation unit 1013 extracts emotion data (multiple records) to be used for generating an emotion map from the storage unit 102. The emotion data to be extracted may be specified by the user or may be determined by the system. For example, emotion data that occurred during a predetermined period in the past may be the emotion data to be extracted. The map generating unit 1013 generates a map (emotion map) in which emotions are mapped to points (or road sections) on a road based on the acquired emotion data and the road data 102C stored in the storage unit 102. FIG. 6 is an example of an emotion map generated by the map generating unit 1013. As shown in the figure, the map generating unit 1013 maps points where any emotion is detected on a road map. The emotion to be mapped may be negative, such as "dissatisfied" or "melancholy," or may be positive, such as "happiness" or "satisfied." This makes it possible to show the user points (or road sections) where it is recommended to pass through and points (or road sections) where it is not recommended to pass through. In the example of FIG. 6, icons indicating emotions are mapped on a road map. The map generating unit 1013 may execute mapping when, for example, a user instructs it to do so, or when a predetermined condition is satisfied.

[0039] Next, a flowchart of the process executed by the in-vehicle device 100 will be described with reference to FIGS. 7 is a flowchart of a process for generating emotion data by the in-vehicle device 100. This process is executed periodically while the vehicle 10 is traveling. First, in step S11, the feeling estimation unit 1011 acquires an image (face image) of the driver via the camera 105. When the camera 105 also serves as a camera of a drive recorder, the feeling estimation unit 1011 may request the drive recorder to acquire an image.

[0040] Next, in step S12, the emotion estimation unit 1011 estimates the emotion of the driver based on the acquired face image. A known method can be adopted for estimating the emotion. For example, the emotion estimation unit 1011 converts the acquired face image into a feature amount and inputs it to the estimation model 102B. In addition, the emotion estimation unit 1011 acquires the classification result and likelihood output from the estimation model 102B, and determines the emotion with the highest likelihood as the emotion of the driver.

[0041] Next, in step S13, it is determined whether the emotion determined by the emotion estimation unit 1011 corresponds to any one of a plurality of preset emotions (e.g., “dissatisfied,” “gloomy,” “happy,” “satisfied,” etc.). The determination may be made based on the likelihood output by the estimation model 102B. If the determination in this step is positive, the process proceeds to step S14. If the determination in this step is negative, the process ends. For example, If there is no class having a likelihood equal to or greater than the predetermined value, the determination in this step is negative.

[0042] In step S14, the data generation unit 1012 generates emotion data based on the result of the estimation performed by the emotion estimation unit 1011. The emotion data is data in which an identifier of the estimated emotion is associated with location information, as shown in Fig. 3. The generated emotion data is stored in the storage unit 102. Emotion data for each trip is accumulated by repeatedly executing the process shown in Fig. 7. Note that data generation section 1012 may delete emotion data that satisfies a predetermined condition (e.g. emotion data that has been generated for a predetermined number of days).

[0043] Next, a process of generating an emotion map based on the accumulated emotion data will be described. Fig. 8 is a flowchart of the process of the map generating unit 1013 generating an emotion map. The illustrated process can be executed when a predetermined trigger occurs. The predetermined trigger may occur, for example, when the vehicle 10 finishes a trip, or may occur based on a user's instruction.

[0044] First, in step S21, emotion data to be used for generating an emotion map is extracted from the storage unit 102. The emotion data to be extracted may be specified by the user, or may be extracted according to a predetermined rule. For example, if the process starts when a trip ends, emotion data generated in the most recent trip may be the target. Also, if there is a rule that "emotion data corresponding to trips in the past month is used," emotion data generated in the past month may be acquired.

[0045] In step S22, the prerequisite conditions for generating an emotion map (hereinafter, generation conditions) are acquired, and the emotion data is filtered according to the generation conditions. For example, road conditions may vary depending on the day of the week or the time period, such as "weekday morning," "weekday evening," and "holiday." Therefore, the emotion data used to generate the emotion map may be filtered according to the day of the week or the time period. The generation conditions may be specified by the user, or may be automatically determined by the system. For example, if the current date and time is a weekday evening, the emotion map is generated using only the emotion data generated on that weekday evening. This makes it possible to visualize, for example, "points that should not be passed on weekday evenings."

[0046] In step S23, the acquired emotion data is mapped onto a road map based on the road data 102C to generate an emotion map. The generated emotion map is output via the input / output unit 104.

[0047] As described above, the in-vehicle device 100 according to the first embodiment can estimate the driver's emotion and map the result on a road map. This makes it possible to visualize points or road sections where negative emotions have occurred, or points or road sections where positive emotions have occurred. Furthermore, by referring to the emotion map, the user of the vehicle can recognize points (or road sections) that are recommended to pass through and points (road sections) that are not recommended to pass through.

[0048] In this embodiment, an example has been given in which the emotion map is generated after the vehicle 10 has finished traveling, but the emotion map may be generated in real time (i.e., while the vehicle 10 is traveling). In this case, the driver may be notified in real time of the points where the predetermined emotion is detected, and the points may be mapped in real time on a road map (for example, output by a navigation device).

[0049] Second Embodiment In the first embodiment, the emotion map is generated by the in-vehicle device 100. In contrast, in the second embodiment, the in-vehicle devices 100 mounted on a plurality of vehicles 10 transmit emotion data to the server device 200, and the server device 200 generates the emotion map based on the emotion data transmitted from the plurality of vehicles. FIG. 9 is a schematic diagram illustrating a system configuration according to the second embodiment.

[0050] 10 is a system configuration diagram of an in-vehicle device 100 in the second embodiment. As shown in the figure, in the second embodiment, the in-vehicle device 100 does not have a map generating unit 1013, and does not store road data 102C. The in-vehicle device 100 according to the second embodiment has a function of transmitting generated emotion data to a server device 200. The other components are similar to those in the first embodiment, and therefore detailed description thereof will be omitted.

[0051] 11 is a system configuration diagram of the server device 200. The server device 200 is a device that receives and accumulates emotion data from a plurality of in-vehicle devices 100, generates an emotion map based on the accumulated data, and distributes the emotion map to the plurality of in-vehicle devices 100.

[0052] The server device 200 can be configured by a general-purpose computer. That is, the server device 200 can be configured as a computer having a processor such as a CPU or a GPU, a main storage device such as a RAM or a ROM, and an auxiliary storage device such as an EPROM, a hard disk drive, or a removable medium. The auxiliary storage device stores an operating system (OS), various programs, various tables, and the like, and the programs stored therein are loaded into a working area of ​​the main storage device and executed. Each component is controlled through the execution of the programs, thereby realizing each function that matches a predetermined purpose, as described below. However, some or all of the functions may be realized by a hardware circuit such as an ASIC or an FPGA.

[0053] The server device 200 includes a control unit 201 , a storage unit 202 , and a communication unit 203 . The control unit 201 is a calculation device that manages the control performed by the server device 200. The control unit 201 can be realized by a calculation processing device such as a CPU. The control unit 201 is configured to have two functional modules: a data collection unit 2011 and a map generation unit 2012. Each functional module may be realized by executing a stored program by a CPU.

[0054] The data collection unit 2011 receives emotion data from a plurality of vehicle-mounted devices 100, and stores the emotion data in the storage unit 202 in association with the vehicle identifiers. The map generating unit 2012 generates an emotion map based on a plurality of emotion data stored in the storage unit 202. The map generating unit 2012 may generate an emotion map based on a request transmitted from the in-vehicle device 100. For example, the map generating unit 2012 generates an emotion map according to generation conditions included in the request, and transmits the generated emotion map to the in-vehicle device 100 that transmitted the request.

[0055] The storage unit 202 includes a main storage device and an auxiliary storage device. The main storage device is a memory in which the programs executed by the control unit 201 and data used by the control programs are expanded. The auxiliary storage device is a device in which the programs executed by the control unit 201 and data used by the control programs are stored.

[0056] The storage unit 202 stores emotion data 202A and road data 202B. Emotion data 202A is a collection of emotion data received from a plurality of in-vehicle devices 100. Each piece of emotion data is associated with an identifier of the vehicle that generated the emotion data. The road data 202B is road map data that is the basis for generating an emotion map. The road data 202B is the same data as the road data 102C.

[0057] The communication unit 203 is a communication interface for connecting the server device 200 to a network. The communication unit 203 includes, for example, a network interface board and a wireless communication interface for wireless communication.

[0058] Next, a description will be given of the flow of data exchanged between the in-vehicle device 100 and the server device 200 in the second embodiment.

[0059] The in-vehicle device 100 periodically transmits emotion data generated while the vehicle 10 is traveling to the server device 200. The process by which the in-vehicle device 100 generates emotion data is similar to the process described with reference to FIG. The server device 200 (data collection unit 2011) stores the received emotion data in the storage unit 202 in association with the vehicle identifier (step S31).

[0060] In step S32, the in-vehicle device 100 requests the server device 200 to generate an emotion map. Specifically, similar to step S22, the in-vehicle device 100 obtains conditions (generation conditions) that are prerequisites for generating an emotion map, and transmits a request (generation request) including the generation conditions to the server device 200. The generation conditions may be input by the user via the input / output unit 104.

[0061] In step S33, the server device 200 (map generator 2012) generates an emotion map based on the received request. Specifically, emotion data 202A (i.e., emotion data transmitted from a plurality of vehicles) that matches the generation conditions is extracted, and mapping is performed on the road map recorded in the road data 202B to generate an image.

[0062] In addition, when multiple emotion data are generated at the same point or road section, the data may be tallied and the results may be mapped. For example, a breakdown of multiple emotions associated with the same point may be generated and the emotion with the highest ratio may be mapped. This makes it possible to clearly indicate points or road sections where multiple drivers tend to have specific emotions. In addition, multiple emotions may be broadly classified into "positive," "neutral," and "negative," and the results of the classification may be mapped. This makes it possible to clearly indicate points or road sections where driving is recommended (or where driving is not recommended).

[0063] Also, a list showing the breakdown of emotions may be generated and attached to the emotion map. The list is displayed, for example, by a user operation (operation to select a location or section). Fig. 13 shows an example of an emotion map generated by the server device 200. The generated emotion map is transmitted to the in-vehicle device 100 and output via the input / output unit 104.

[0064] As described above, in the second embodiment, the server device 200 collects emotion data from a plurality of in-vehicle devices 100, and generates an emotion map based on the collected emotion data. According to this embodiment, points (or road sections) that are recommended to pass through and points (road sections) that are not recommended to pass through can be more appropriately visualized based on the probe data.

[0065] In the present embodiment, the vehicle-mounted device 100 uploads emotion data at any time. However, the emotion data may be uploaded at a predetermined timing (for example, when the trip of the vehicle 10 ends). You may upload the file via messaging.

[0066] Also, when a predetermined emotion is detected in the vehicle 10, the driver may be notified of the emotion and the user may be asked whether or not to transmit emotion data. For example, when it is detected that the driver has a dissatisfied look on his / her face, a question such as "Do you want to transmit data showing negative emotion to share problems occurring on the road?" may be asked.

[0067] In addition, in this embodiment, the server device 200 generates the emotion map, but the emotion map may be generated by the in-vehicle device 100. In this case, the server device 200 may tally up emotions for each point or section, and transmit the result (tally up data) to the in-vehicle device 100. The in-vehicle device 100 may generate the emotion map based on the tally up data transmitted from the server device 200.

[0068] In addition, in the present embodiment, an example has been given in which the server device 200 generates an emotion map based on a request from the in-vehicle device 100. However, the server device 200 may periodically generate and store an emotion map, and transmit the emotion map to the in-vehicle device 100 upon receiving a request from the in-vehicle device 100.

[0069] Third embodiment In the first embodiment, the emotion data used when generating the emotion map is filtered according to the day of the week and the date and time, but filtering may be performed using other elements.

[0070] In the third embodiment, the data generating unit 1012 adds data on the driving environment of the vehicle 10 to the emotion data, and the map generating unit 1013 performs filtering using the data. FIG. 14 is an example of emotion data generated in the third embodiment. As shown in the figure, in this example, environmental data is added to the emotion data. The environmental data is data that represents the driving environment of the vehicle 10. The driving environment can be, for example, elements that affect the stress on the driver when the vehicle 10 is driving, such as weather and traffic volume. The driving environment may be acquired by a sensor provided in the vehicle 10, or may be acquired from an external device. For example, if there is a road that is strongly affected by strong winds, information such as "the driver felt uneasy in an environment where strong winds are blowing" is recorded in the emotion data.

[0071] In the third embodiment, in step S22, emotion data is filtered using the driving environment. The driving environment may be specified by the user or may be determined by the system. For example, if the current driving environment is "strong wind", filtering can be performed by the weather of "strong wind". In this way, in the third embodiment, an emotion map corresponding to a specific driving environment can be generated. For example, if the current driving environment is "strong wind", emotions occurring in the same environment are mapped. This makes it possible to generate an appropriate emotion map according to the driving environment. The third embodiment may be applied to the second embodiment.

[0072] (Fourth embodiment) The fourth embodiment is an embodiment in which an image captured by an on-board camera is presented together with an emotion map.

[0073] In the fourth embodiment, when the data generating unit 1012 generates emotion data, an image outside the vehicle 10 (typically, an image in front of the vehicle) is acquired via an on-board camera. The on-board camera may be used in combination with the camera 105. For example, the camera 105 may have a 360-degree angle of view. If the camera has a camera, an image of the outside of the vehicle and an image of the driver's face can be acquired at the same time. In this case, the data generating unit 1012 may crop the area corresponding to the outside of the vehicle. Furthermore, the data generating unit 1012 associates the acquired image with the emotion data. Fig. 15 is an example of emotion data generated in the fourth embodiment. As shown in the figure, in this example, image data is added to the emotion data. By referring to the image data, it is possible to know later what caused the change in the driver's emotion.

[0074] In the fourth embodiment, the map generating unit 1013 generates an emotion map in which each point is associated with an image. Fig. 16 is an example of an emotion map generated in the third embodiment. As shown in the figure, in this embodiment, an image is associated with each point to which an emotion is mapped, and the image can be confirmed based on a user's operation. With this configuration, it becomes possible to confirm after the fact what caused the change in the driver's emotions.

[0075] The fourth embodiment may be applied to the second embodiment. Furthermore, although an image outside the vehicle is exemplified in this example, an image of the driver's face may be included in the image data. Furthermore, although an example of using a still image is given in this example, the image data may be video data. For example, a still image or a video including both an image in front of the vehicle and an image of the driver's face may be output. Such an image (or video) may be made retrievable from the in-vehicle device 100 separately from the emotion data. With this configuration, it becomes possible to provide the user with an image (or video) of the moment a particular emotion occurs.

[0076] (Modification) The above-described embodiment is merely an example, and the present disclosure can be modified and implemented as appropriate without departing from the spirit and scope of the present disclosure. For example, the processes and means described in this disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0077] In addition, in the description of the embodiment, an example was given of outputting only an emotion map, but if there are points or road sections for which it is recommended to pass, or points for which it is not recommended to pass, the reasons for this may be specifically notified. Furthermore, in the description of the embodiment, an example has been given in which emotion is estimated from a face image, but the emotion of the driver may be estimated based on other biological information (for example, voice, etc.).

[0078] Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by one device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0079] The present disclosure can also be realized by supplying a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors of the computer read and execute the program. Such a computer program may be provided to the computer by a non-transitory computer-readable storage medium connectable to the system bus of the computer, or may be provided to the computer via a network. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (floppy disk, hard disk drive (HDD), etc.), an optical disk (CD-ROM, DVD disk, Blu-ray disk, etc.), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, and any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0080] 100...In-vehicle equipment 101, 201...Control unit 102,202...Storage section 103,203···Communications Department 104...Input / output section 105···Camera 106...Location information acquisition unit 200 Server device

Claims

1. Estimating an emotion of a driver of the vehicle based on an image captured by a camera mounted on the vehicle; Identifying a first location where a predetermined emotion is estimated as the emotion of the driver; generating emotion data that associates the estimated emotion with the first location, and transmitting the emotion data to a server device; receiving from the server device a road map in which the emotions felt by each of a plurality of drivers are tallied for each location or road section, and the emotion that prevails most among a plurality of emotions associated with the same location or road section is mapped, the road map being accompanied by a list showing the breakdown of the emotions; An information processing device having a control unit that executes the above.

2. The control unit further acquires an image of an exterior of the vehicle at the first location, and further associates the image with the emotion data. The information processing device according to claim 1 .

3. The camera is a camera capable of simultaneously capturing a view in front of the vehicle and the face of the driver. The information processing device according to claim 2 .

4. A vehicle system including an on-board device mounted in a vehicle and a server device that manages a plurality of the vehicles, The in-vehicle device includes: Estimating an emotion of a driver of the vehicle based on an image acquired by a camera mounted on the vehicle; transmitting emotion data, which is data associating the estimated emotion with location information, to the server device; A first control unit that executes The server device includes: Based on the emotion data received from the plurality of in-vehicle devices, the emotions felt by each of the plurality of drivers are collected for each location or road section, and associated with the same location or the same road section. a second control unit that generates road map data in which a list showing the breakdown of emotions is attached to the road map data, the road map data being generated by mapping the emotion that is most prevalent among a breakdown of a plurality of emotions displayed on the road map, Vehicle systems.

5. the second control unit periodically generates the road map data based on the emotion data.

5. The vehicle system of claim 4.

6. The second control unit transmits the road map data to the vehicle-mounted device, The first control unit outputs the road map data.

6. A vehicle system according to claim 4 or 5.

7. estimating an emotion of a driver of the vehicle based on images acquired by a camera mounted on the vehicle; Identifying a first location where a predetermined emotion is estimated as the emotion of the driver; generating emotion data that associates the estimated emotion with the first location, and transmitting the emotion data to a server device; receiving from the server device a road map in which the emotions felt by each of a plurality of drivers are tallied for each location or road section, and the emotion that prevails most among a plurality of emotions associated with the same location or road section is mapped, the road map being accompanied by a list showing the breakdown of the emotions; An information processing method comprising:

8. further acquiring an image of an exterior of the vehicle at the first location, and further associating the image with the emotion data; The information processing method according to claim 7.

9. The camera is a camera capable of simultaneously capturing a view in front of the vehicle and the face of the driver. The information processing method according to claim 8.

10. A program for causing a computer to execute the information processing method according to any one of claims 7 to 9.

Images