Control device, control method and program

Abstract

To provide a control device which can appropriately provide an opportunity of content reproduction for a passenger.SOLUTION: A control device comprises: recognition means for recognizing a running environment of a vehicle; estimation means for estimating emotion of a passenger on the vehicle; determination means for determining a content for reproduction on the basis of the fact that the running environment recognized by the recognition means and the emotion estimated by the estimation means have changed; and control means for performing control to reproduce the content determined by the determination means.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a control device, a control method, and a program for controlling the playback of content. [Background technology]

[0002] Patent document 1 describes an apparatus that determines the vehicle's driving conditions, selects content having attributes corresponding to the vehicle's driving conditions from content that can be played by a sound source, and suggests to passengers that they play the selected content. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-63133 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when content playback is suggested based on the results of judging the vehicle's driving conditions, the suggested content may be played back frequently, which may be annoying to passengers.

[0005] An object of the present invention is to provide a control device, a control method, and a program that provide passengers with an opportunity to appropriately play back content. [Means for solving the problem]

[0006] A control device according to the present invention includes a recognition means for recognizing a driving environment of a vehicle, a first determination means for determining whether or not there is a change in the driving environment recognized by the recognition means, an estimation means for estimating an emotion of an occupant of the vehicle, a determination means for determining content to be reproduced based on the determination by the first determination means that there is a change in the driving environment and the change in the emotion estimated by the estimation means, and a control means for performing control to reproduce the content determined by the determination means. The first determination means determines that there is a change in the driving environment when the driving scene of the vehicle becomes a driving scene in which a driving load of a passenger of the vehicle is predicted to be reduced, and when it is determined that at least one of the driving environment and the emotion estimated by the estimation means has not changed, the determination means is controlled so that the content to be reproduced is not determined by the determination means and the control means is not controlled so that the content is not reproduced by the reproduction means. It is characterized by:

[0007] A control method according to the present invention is a control method executed in a control device, and includes a recognition step of recognizing a driving environment of a vehicle, a determination step of determining whether or not there is a change in the driving environment recognized in the recognition step, an estimation step of estimating an emotion of an occupant of the vehicle, a determination step of determining a content to be reproduced based on the determination step that there is a change in the driving environment and the change in the emotion estimated in the estimation step, and a control step of performing control to reproduce the content determined in the determination step. In the determination step, when a driving scene of the vehicle becomes a driving scene in which a driving load of a passenger of the vehicle is predicted to be reduced, it is determined that there is a change in the driving environment, and when it is determined that at least one of the driving environment and the emotion estimated in the estimation step has not changed, the determination of the content to be reproduced in the determination step and the reproduction of the content in the control step are not performed. It is characterized by:

[0008] The program according to the present invention causes a computer to function as recognition means for recognizing a driving environment of a vehicle, determination means for determining whether or not there is a change in the driving environment recognized by the recognition means, estimation means for estimating emotions of an occupant of the vehicle, determination means for determining content to be played back based on the determination by the determination means that there is a change in the driving environment and the change in the emotion estimated by the estimation means, and control means for performing control to play back the content determined by the determination means. The determination means determines that there is a change in the driving environment when the driving scene of the vehicle becomes a driving scene in which a driving load of a passenger of the vehicle is predicted to be reduced, and when it is determined that at least one of the driving environment and the emotion estimated by the estimation means has not changed, the determination means is controlled so that the content to be reproduced is not determined by the determination means and the control means is not performed to reproduce the content. It is characterized by: Effect of the Invention

[0009] According to the present invention, it is possible to appropriately provide passengers with an opportunity to play back content. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram illustrating a configuration of a vehicle control device. [Diagram 2] FIG. 2 is a diagram showing functional blocks of a control unit. [Diagram 3] FIG. 2 is a diagram illustrating a configuration of a server. [Figure 4] 13 is a flowchart showing a content playback control process. [Diagram 5] 13 is a flowchart showing a content playback control process. [Figure 6] FIG. 13 is a diagram showing a message screen recommending content playback. [Figure 7] 13 is a flowchart showing a content playback control process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] Fig. 1 is a block diagram of a vehicle control device (cruising control device) according to one embodiment of the present invention, which controls a vehicle 1. In Fig. 1, the vehicle 1 is shown in a schematic plan view and a side view. As an example, the vehicle 1 is a sedan-type four-wheeled passenger car.

[0013] [First embodiment] The driving control device in Fig. 1 includes a control unit 2. The control unit 2 includes a plurality of ECUs 20 to 29 communicably connected via an in-vehicle network. Each ECU includes a processor such as a CPU, a storage device such as a semiconductor memory, an interface with an external device, and the like. The storage device stores programs executed by the processor, data used by the processor for processing, and the like. Each ECU may include a plurality of processors, storage devices, interfaces, and the like.

[0014] The following describes the functions and the like handled by each of the ECUs 20 to 29. Note that the number of ECUs and the functions handled by them can be designed appropriately, and they can be divided more finely than in this embodiment, or can be integrated.

[0015] The ECU 20 executes control related to the automatic driving of the vehicle 1. In the automatic driving, at least one of the steering and the acceleration / deceleration of the vehicle 1 is automatically controlled.

[0016] The ECU 21 controls the electric power steering device 3. The electric power steering device 3 includes a mechanism for steering the front wheels in response to a driver's driving operation (steering operation) on a steering wheel 31. The electric power steering device 3 also includes a motor for assisting the steering operation or for generating a driving force for automatically steering the front wheels, a sensor for detecting a steering angle, and the like. When the driving state of the vehicle 1 is autonomous driving, the ECU 21 automatically controls the electric power steering device 3 in response to an instruction from the ECU 20, and controls the traveling direction of the vehicle 1.

[0017] ECUs 22 and 23 control detection units 41-43 that detect the surrounding conditions of the vehicle and process information on the detection results. Detection unit 41 is a camera that captures images of the area ahead of vehicle 1 (hereinafter, may be referred to as camera 41), and in this embodiment, is attached to the inside of the passenger compartment of the windshield at the front of the roof of vehicle 1. By analyzing images captured by camera 41, it is possible to extract, for example, the contours of targets and lane markings (white lines, etc.) on a road.

[0018] The detection unit 42 is a Light Detection and Ranging (LIDAR) that detects targets around the vehicle 1 and measures the distance to the targets. In the present embodiment, five detection units 42 are provided, one at each corner of the front of the vehicle 1, one at the rear center, and one on each side of the rear. The detection unit 43 is a millimeter wave radar (hereinafter, may be referred to as radar 43), which detects targets around the vehicle 1 and measures the distance to the targets. In the present embodiment, five radars 43 are provided, one at the front center of the vehicle 1, one at each front corner, and one at each rear corner.

[0019] The ECU 22 controls the one camera 41 and each detection unit 42 and processes information on the detection results. The ECU 23 controls the other camera 41 and each radar 43 and processes information on the detection results. By providing two sets of devices that detect the surrounding conditions of the vehicle, the reliability of the detection results can be improved, and by providing different types of detection units such as cameras and radars, the surrounding environment of the vehicle can be analyzed from multiple perspectives.

[0020] The ECU 24 controls the gyro sensor 5, the GPS sensor 24b, and the communication device 24c, and processes information on the detection results or communication results. The gyro sensor 5 detects the rotational motion of the vehicle 1. The path of the vehicle 1 can be determined based on the detection results of the gyro sensor 5, the wheel speed, etc. The GPS sensor 24b detects the current position of the vehicle 1. The communication device 24c wirelessly communicates with a server that provides map information, traffic information, and weather information, and acquires this information. The ECU 24 can access a database 24a of map information built in a storage device, and the ECU 24 performs route search from the current location to the destination, etc. Note that a database of the above traffic information, weather information, etc. may be built in the database 24a.

[0021] The ECU 25 includes a communication device 25a for vehicle-to-vehicle communication. The communication device 25a performs wireless communication with other surrounding vehicles and exchanges information between the vehicles. The communication device 25a has various communication functions, such as a dedicated short range communication (DSRC) function and a cellular communication function. The communication device 25a may be configured as a TCU (Telematics Communication Unit) including a transmitting / receiving antenna.

[0022] The ECU 26 controls the power plant 6. The power plant 6 is a mechanism that outputs a driving force to rotate the drive wheels of the vehicle 1, and includes, for example, an engine and a transmission. The ECU 26 controls the output of the engine in response to a driving operation (accelerator operation or acceleration operation) by the driver detected by an operation detection sensor 7a provided on the accelerator pedal 7A, for example, and switches the gear stage of the transmission based on information such as the vehicle speed detected by a vehicle speed sensor 7c. When the driving state of the vehicle 1 is autonomous, the ECU 26 automatically controls the power plant 6 in response to an instruction from the ECU 20, and controls the acceleration and deceleration of the vehicle 1.

[0023] The ECU 27 controls lighting devices (headlights, taillights, etc.) including turn signals 8 (blinkers). In the example of Fig. 1, the turn signals 8 are provided at the front, door mirrors, and rear of the vehicle 1.

[0024] The ECU 28 controls the input / output device 9. The input / output device 9 outputs information to the driver and accepts information input from the driver. The audio output device 91 notifies the driver of information by audio. The display device 92 notifies the driver of information by displaying an image. The display device 92 is disposed, for example, in front of the driver's seat and constitutes an instrument panel or the like. Note that, although audio and display are exemplified here, information may be notified by vibration or light. Information may be notified by a combination of two or more of audio, display, vibration, and light. Furthermore, the combination may be changed or the notification mode may be changed depending on the level of the information to be notified (for example, urgency). The display device 92 may also include a navigation device.

[0025] The input device 93 is a group of switches that are arranged in a position operable by the driver to give instructions to the vehicle 1, but may also include a voice input device such as a microphone.

[0026] The ECU 29 controls the brake device 10 and the parking brake (not shown). The brake device 10 is, for example, a disk brake device provided on each wheel of the vehicle 1, and applies resistance to the rotation of the wheel to decelerate or stop the vehicle 1. The ECU 29 controls the operation of the brake device 10 in response to the driving operation (brake operation) of the driver detected by, for example, an operation detection sensor 7b provided on the brake pedal 7B. When the driving state of the vehicle 1 is automatic driving, the ECU 29 automatically controls the brake device 10 in response to an instruction from the ECU 20, and controls the deceleration and stop of the vehicle 1. The brake device 10 and the parking brake can be operated to maintain the stopped state of the vehicle 1. In addition, when the transmission of the power plant 6 is equipped with a parking lock mechanism, the parking lock mechanism can also be operated to maintain the stopped state of the vehicle 1.

[0027] The following describes the control related to the automatic driving of the vehicle 1 executed by the ECU 20. When the driver instructs the ECU 20 to enter a destination and to perform automatic driving, the ECU 20 automatically controls the driving of the vehicle 1 toward the destination according to the guide route searched for by the ECU 24. During automatic control, the ECU 20 acquires and recognizes information (external world information) related to the surrounding conditions of the vehicle 1 from the ECUs 22 and 23, and issues instructions to the ECUs 21, 26, and 29 based on the acquired information and the recognition results to control the steering, acceleration, and deceleration of the vehicle 1.

[0028] FIG. 2 is a diagram showing functional blocks of the control unit 2 and the processing unit 230 used in this embodiment. The control unit 200 corresponds to the control unit 2 in FIG. 1, and includes an external environment recognition unit 201, a self-location recognition unit 202, an interior recognition unit 203, an action planning unit 204, a drive control unit 205, and a device control unit 206. Each block is realized by one ECU or multiple ECUs shown in FIG. 1. The function of the control unit 200 is mainly executed in the automatic driving of the vehicle 1, but the function of the control unit 200 can be realized even when the vehicle 1 is driven by a passenger. For example, even when the vehicle 1 is driven by a passenger, the processing unit 230 that realizes the operation of this embodiment and the control unit 200 cooperate with each other to execute the functions of the external environment recognition unit 201 and the interior recognition unit 203. The configuration of the control unit 200 and the processing unit 230 in FIG. 2 can be a computer that executes a program.

[0029] The external environment recognition unit 201 recognizes external information of the vehicle 1 based on signals from the external environment recognition camera 211 and the external environment recognition sensor 212. Here, the external environment recognition camera 211 is, for example, the camera 41 in FIG. 1, and the external environment recognition sensor 212 is, for example, the detection units 42 and 43 in FIG. 1. The external environment recognition unit 201 recognizes, for example, scenes such as intersections, railroad crossings, and tunnels, free spaces such as road shoulders, and behaviors (speed, traveling direction, etc.) of other vehicles based on signals from the external environment recognition camera 211 and the external environment recognition sensor 212. The self-position recognition unit 202 recognizes the current position of the vehicle 1 based on a signal from the GPS sensor 215. Here, the GPS sensor 215 corresponds to, for example, the GPS sensor 24b in FIG. 1.

[0030] The in-vehicle recognition unit 203 identifies the passenger of the vehicle 1 based on signals from the in-vehicle recognition camera 213 and the in-vehicle recognition sensor 214, and recognizes the state of the passenger. The in-vehicle recognition camera 213 is, for example, a near-infrared camera installed on the display device 92 inside the vehicle 1, and detects, for example, the direction of the passenger's gaze from the captured image data. The in-vehicle recognition sensor 214 is, for example, a sensor for detecting the passenger's biological signal and acquiring biological information. The biological information is, for example, information related to the living body, such as pulse, heart rate, weight, body temperature, blood pressure, sweating, and sleep information. The in-vehicle recognition sensor 214 may acquire such biological information from, for example, a wearable device of the passenger. The in-vehicle recognition unit 203 recognizes, based on those signals, that the passenger is in a drowsy state, is in a state of working other than driving, and the like.

[0031] The behavior planning unit 204 plans the behavior of the vehicle 1, such as an optimal route and a risk avoidance route, based on the results of recognition by the external environment recognition unit 201 and the self-position recognition unit 202. The behavior planning unit 204 performs, for example, entry judgment based on the start point and end point of an intersection or a railroad crossing, and behavior planning based on the predicted results of the behavior of other vehicles. The drive control unit 205 controls the driving force output device 216, the steering device 217, and the brake device 218 based on the behavior plan by the behavior planning unit 204. Here, the driving force output device 216 corresponds to, for example, the power plant 6 in FIG. 1, the steering device 217 corresponds to the electric power steering device 3 in FIG. 1, and the brake device 218 corresponds to the brake device 10.

[0032] The device control unit 206 controls devices connected to the control unit 200. For example, the device control unit 206 controls the speaker 219 and the microphone 220 to output a predetermined voice message such as a warning or a message for navigation, and detects a voice signal uttered by a passenger in the vehicle to acquire voice data. For example, the device control unit 206 controls the display device 221 to display a predetermined interface screen. The display device 221 corresponds to, for example, the display device 92. For example, the device control unit 206 controls the navigation device 222 to acquire setting information in the navigation device 222. The device control unit 206 controls a content playback device 223 that plays content such as music, and plays music through the speaker 219. The communication interface 224 corresponds to, for example, the communication device 25a in FIG. 1, and controls communication between an external device such as a server and another vehicle. The communication interface 224 includes a configuration corresponding to each of a plurality of communication methods, and has a configuration corresponding to short-range wireless communication such as Wifi and Bluetooth (registered trademark). Wired communication may be performed in part or all of the communication. In this case, a connector for wired connection is provided separately. The communication interface 224 can be connected to a wireless communication network such as a mobile phone network such as 3G, cellular, or LTE communication, or an Internet line.

[0033] The control unit 200 may include other functional blocks than those shown in FIG. 2 as appropriate, and may include, for example, an optimal route calculation unit that calculates an optimal route to a destination based on map information acquired via the communication device 24c. The control unit 200 may also acquire information from other than the cameras and sensors shown in FIG. 1, and may acquire information on other vehicles via the communication device 25a, for example. The control unit 200 receives detection signals not only from the GPS sensor 215 but also from various sensors provided in the vehicle 1. For example, the control unit 200 receives detection signals from door opening / closing sensors and door lock mechanism sensors provided in the door units of the vehicle 1 via an ECU configured in the door units. Thereby, the control unit 200 can detect unlocking of the doors and opening / closing operations of the doors.

[0034] In this embodiment, the processing unit 230 is configured to be able to communicate with the control unit 200. The processing unit 230 recognizes the voices and facial images of passengers in the vehicle 1, and estimates the emotions of the passengers from the recognition results. The processing unit 230 also acquires content equivalent to music, determines content to be played from the acquired content, and requests the control unit 200 to play the content on the content playback device 223.

[0035] The voice recognition unit 231 analyzes the voice signal input by the microphone 220 and performs language analysis. At that time, the voice recognition unit 231 acquires features such as intonation and tempo of the voice signal. The image recognition unit 232 analyzes image data captured by the in-vehicle recognition camera 213 and performs face recognition. The emotion estimation unit 233 estimates the emotion of the passenger of the vehicle 1 based on the analysis results of the voice recognition unit 231 and the image recognition unit 232. The emotion estimation estimates a change in emotion, for example, from a smile estimated based on at least one of the pitch, intonation, and tempo of the voice recognized by the voice recognition unit 231 and the change in the position and shape of the passenger's facial features (nose, eyebrows, etc.) recognized by the image recognition unit 232. The estimation of the change in emotion means, for example, estimating an uplifting emotion by recognizing a smile or laughter. The emotion estimation unit 233 may use signals and data other than those of the voice recognition unit 231 and the image recognition unit 232. For example, the emotion estimation unit 233 may estimate a change in emotion using biological information detected by the in-vehicle recognition sensor 214.

[0036] The content storage unit 234 stores content corresponding to music. The content stored in the content storage unit 234 is, for example, content information such as content and metadata acquired from a mobile terminal such as a smartphone held by a passenger of the vehicle 1 or from the outside via the Internet. The above content information includes lyrics, melody, tempo, wavelength analysis data, release date, artist name, instrument type, tie-up information, song name or album name, country of production, label type, and shooting location information. The content information may further include information generated based on the above information. The content analysis unit 235 analyzes the content and content information stored in the content storage unit 234. For example, the content analysis unit 235 performs text analysis on the lyric information stored in the content storage unit 234. The content determination unit 236 determines the content to be reproduced from the content stored in the content storage unit 234 based on the analysis result by the content analysis unit 235. When the content to be reproduced is determined by the content determination unit 236, the processing unit 230 transmits information on the determined content to be reproduced and a reproduction request to the control unit 200. Then, the control unit 200 causes the content playback device 223 to play back the determined content to be played back.

[0037] The driving information analysis unit 237 acquires and analyzes driving information from the control unit 200 at a predetermined time interval. Here, the driving information includes position information of the vehicle 1 and the recognition result obtained by the external environment recognition unit 201. The driving information analysis unit 237 determines, for example, from the acquired recognition result, whether or not the scene in which the vehicle 1 is traveling has changed.

[0038] 3 is a diagram showing the configuration of a server 300 with which the vehicle 1 can communicate via the communication interface 224. The server 300 has a database 330 that stores data associating vehicle driving information with information on played content for each pre-registered user (passenger).

[0039] The server 300 includes a control unit 310 and a storage unit 320. The control unit 310 includes a processor 311 and a memory 312, and controls the server 300 in an integrated manner. The operation of the server 300 in this embodiment is realized by the processor 311 executing a program stored in the memory 312. The communication interface 313 has a communication interface configuration according to an external network medium. The server 300 and the vehicle 1 can communicate with each other via the communication interface 313. For example, the server 300 receives driving information and information on the reproduced content from the vehicle 1, and registers the information in a database 330 configured in the storage unit 320 in association with passenger information. For example, as shown in FIG. 3, driving information 341 and content information 342 transmitted from the vehicle A are registered as history information 340 associated with passenger A of the vehicle A. In addition, driving information 351 and content information 352 transmitted from the vehicle B are registered as history information 350 associated with passenger B of the vehicle B. The driving information may be, for example, the position information of the vehicle 1. Furthermore, the content information may be, for example, a title or an artist name, as long as it is information that can identify the content. Furthermore, the server 300 has databases corresponding to the map information, traffic information, and weather information described in FIG.

[0040] Fig. 4 is a flowchart showing the process of content playback control in this embodiment. The process in Fig. 4 is realized, for example, by a processor such as an ECU constituting the processing unit 230 reading and executing a program stored in a memory (not shown). The process in Fig. 4 is started when the control unit 200 notifies the processing unit 230 that the vehicle 1 has started traveling.

[0041] In S101, the processing unit 230 uses the content analysis unit 235 to analyze the characteristics of the text information of the content stored in the content storage unit 234. For example, the processing unit 230 clusters similar words included in the lyric information stored in the content storage unit 234, and tags the stored content with characteristics such as place names, scenes, seasons, etc. based on the analysis results. For example, for content A, if the lyrics contain many specific place names, the content is tagged with the place names. Also, if the lyrics contain many words related to the "sea", the content is tagged with the scene name "sea".

[0042] In S102, the processing unit 230 acquires the position information of the vehicle 1 from the control unit 200 and the recognition result of the external environment recognition unit 201 as information of the driving environment through the driving information analysis unit 237, and analyzes them. Meanwhile, in parallel with the processing of S102, in S103, the processing unit 230 performs voice recognition based on the voice signal of the passenger of the vehicle 1 through the voice recognition unit 231, and performs face recognition based on the image information of the passenger of the vehicle 1 through the image recognition unit 232. Then, the processing unit 230 estimates the emotion of the passenger through the emotion estimation unit 233. Note that the processing of S102 and the processing of S103 are repeatedly performed at predetermined time intervals while the vehicle 1 is traveling.

[0043] In S104, the processing unit 230 determines whether or not there is a change in the driving environment based on the recognition result of the external environment recognition unit 201 by the driving information analysis unit 237. In this embodiment, the change in the driving environment refers to a change to a predetermined driving scene. The change to the predetermined driving scene refers to a change to a scene in which the driving load of the participants is estimated to be reduced. The reduction in the driving load here includes not only the reduction in the driving load estimated from the torque related to the driving operation such as moving from an uphill to a downhill, but also the reduction in the mental load of the driver. Examples of changes to the driving scene that reduce the mental load include a change in traffic volume such as a traffic jam being resolved, a change in the weather improving, joining the expressway from a service area, approaching within a predetermined distance to the destination, and crossing the border of a region on the driving route (for example, crossing a prefectural border or entering a city where the destination is located). At that time, the change in traffic volume may be determined based on, for example, traffic information acquired from the traffic information database 362 of the server 300 or a change in the speed of the vehicle 1. In addition, whether or not the vehicle 1 has come within a predetermined distance to the destination may be determined based on, for example, the position information and map information of the vehicle 1. In addition, a change in weather may be determined based on, for example, weather information acquired from the weather information database 363 of the server 300, or information on a camera, a sensor, a thermometer, and a hygrometer provided in the vehicle 1. In addition, a merger with a highway or a change in distance to the destination may be determined based on, for example, map information acquired from the map information database 361 of the server 300. In addition, a change in a scene that reduces the mental load of the driver includes a change to a driving scene that satisfies a condition in which there are fewer buildings and the field of view is expanded, such as the sea being visible, exiting a tunnel, or the scenery being expanded. Such a change to a driving scene may be determined based on, for example, whether or not a change in the recognition result by the external environment recognition unit 201 satisfies the above conditions. In addition, as described above, a change in a scene that reduces the mental load of the driver includes approaching within a predetermined distance to the destination, and the reduction in the mental load of the driver may be quantified based on the degree of approach to the destination.For example, each time a predetermined distance (such as 50 km) is approached to the destination, it may be determined that the mental stress is reduced by the ratio of the approaching distance to the distance to the destination.

[0044] In addition, although the above describes the determination of whether or not there is a change to a predetermined driving scene, other configurations may be used. For example, the location information acquired in S102 may be transmitted to the server 300, and an inquiry may be made as to whether or not the location information or nearby location information is stored in the history information corresponding to the vehicle 1 or the passenger. In this case, if the result of the inquiry shows that the location information is stored, it may be determined that the vehicle 1 is traveling again through a place where it has traveled in the past, and that there is a change in the driving environment. With such a configuration, for example, when the vehicle 1 is traveling again through a place that is memorable to the passenger, this can be used as a trigger for content playback.

[0045] Also, for example, the processing unit 230 may detect signals such as the torque of the driver's driving operation or biological information of the passenger from the control unit 230, and based on the changes, it may be determined in S104 that there has been a change in the driving environment.

[0046] If it is determined in S104 that there has been no change in the driving environment, the process proceeds to S108. On the other hand, if it is determined in S104 that there has been a change in the driving environment, the process proceeds to S105.

[0047] In S105, the processing unit 230 determines whether or not there is a change in the estimation result of the emotion of the passenger by the emotion estimation unit 233. In this embodiment, a change in the estimation result of the emotion means that an emotion of elation such as "joy" is recognized. For example, when laughter is recognized by the voice recognition unit 231 or a smile is recognized by the image recognition unit 232, the emotion estimation unit 233 recognizes the emotion of "joy". If it is determined in S105 that there is no change in the estimation result of the emotion, the process proceeds to S108. On the other hand, if it is determined in S105 that there is a change in the estimation result of the emotion, the process proceeds to S106.

[0048] In S106, the processing unit 230 determines, by the content determination unit 236, content to be reproduced from the content stored in the content storage unit 234. For example, the content determination unit 236 determines, as the content to be reproduced, content tagged with a place name corresponding to the current location information. Also, for example, the content determination unit 236 determines, as the content to be reproduced, content tagged with a scene in which the vehicle 1 is currently traveling. The criteria for determining such content may be determined according to the analysis result of the content stored in the content storage unit 234 by the content analysis unit 235. For example, when multiple types of tags such as "place name", "scene", and "season" are acquired as a result of the analysis, content determined to match the current traveling environment of the vehicle 1 (position, scene, season, etc.) may be determined as the content to be reproduced. In this case, the determined content may be one or multiple.

[0049] In S107, the processing unit 230 requests the control unit 200 to reproduce the content determined by the content determination unit 236 on the content reproduction device 223. In response to the request from the processing unit 230, the control unit 200 causes the content determined by the content determination unit 236 to be reproduced on the content reproduction device 223.

[0050] In S108, the processing unit 230 judges whether or not the current position of the vehicle 1 is near the end point. The judgment in S108 may be made by the processing unit 230 inquiring of the control unit 200. If it is judged in S108 that the current position is not near the end point, the processing from S102 to S103 is repeated. On the other hand, if it is judged in S108 that the current position is near the end point, the processing proceeds to S109.

[0051] In S109, the processing unit 230 transmits the driving information acquired by the driving information analysis unit 237 and the information of the played content together with the identification information of the vehicle 1 or the passenger to the server 300. The driving information transmitted in S109 includes at least the position information of the vehicle 1 when the content was played. That is, the recognition result of the external environment recognition unit 201 does not need to be transmitted. The server 300 stores the driving information and the content information received from the vehicle 1 in the database 330 in association with the passenger information. After S109, the processing of FIG. 4 ends.

[0052] As described above, according to the present embodiment, when it is determined that there is a change in the vehicle's driving environment and that there is a change in the passenger's emotions, music that corresponds to the driving environment is played. With such a configuration, it is possible to provide playback of music that is suitable for the current driving environment at an appropriate frequency in accordance with the passenger's changes in emotions.

[0053] [Second embodiment] The second embodiment will be described below with respect to the differences from the first embodiment. In the first embodiment, when the content to be played is determined in S106, music is played in S107 without any user operation. In this embodiment, when the content to be played is determined, information about the content is notified to the passenger. Then, when a playback instruction is received from the passenger, the content is played.

[0054] Fig. 5 is a flowchart showing the process of content playback control in this embodiment. The process of Fig. 5 is realized, for example, by a processor such as an ECU constituting the processing unit 230 reading and executing a program stored in a memory (not shown). The process of Fig. 5 is started when the control unit 200 notifies the processing unit 230 that the vehicle 1 has started traveling.

[0055] S201 to S206 are the same as those in S101 to S106 in FIG. 4, and therefore their description will be omitted.

[0056] When the content to be played is determined in S206, in S207, the processing unit 230 requests the control unit 200 to display information about the content to be played on the display device 221. When the control unit 200 receives the information about the content to be played from the processing unit 230, it displays on the display device 221 a message screen recommending playback of the content.

[0057] FIG. 6 is a diagram showing an example of a message screen recommending content playback. As shown in FIG. 6, a message 601 recommending playback of a song is displayed on the message screen 600. Information 602 of the content to be played is displayed below the message 601. In FIG. 6, the title and artist name of the song are displayed as the information 602. Note that the information 602 displays information about only one piece of content, but multiple pieces of content may be displayed. Also, the information 602 is not limited to the title and artist name, and other information may be displayed. For example, playback time or the like may be displayed. Buttons 603 and 604 are displayed on the message screen 600. Button 603 is a button that can accept an instruction from the passenger to play a song. Button 604 is a button that can accept an instruction from the passenger not to play a song. When multiple pieces of content are displayed as the information 602, a scroll key or the like may be displayed. In this case, the passenger selects a desired piece of content from the multiple pieces of content, and the passenger presses button 603 to play the desired piece of content.

[0058] In S208, the processing unit 230 determines whether or not a content playback instruction has been received. For example, when the processing unit 230 detects pressing of the button 603, it determines that a content playback instruction has been received. When it is determined in S208 that a content playback instruction has not been received, the processing from S202 and S203 is repeated. On the other hand, when it is determined in S208 that a content playback instruction has been received, the processing proceeds to S209.

[0059] In S209, the processing unit 230 requests the control unit 200 to reproduce the content determined by the content determination unit 236 and selected on the message screen 600 on the content reproduction device 223. In response to the request from the processing unit 230, the control unit 200 causes the content determined by the content determination unit 236 to be reproduced on the content reproduction device 223.

[0060] Since S210 and S211 are the same as those in S108 and S109, their explanation will be omitted.

[0061] As described above, according to this embodiment, when it is determined that the vehicle's driving environment has changed and the passenger's emotions have changed, a message is displayed recommending the playback of music that matches the driving environment. In addition, when there are multiple recommended contents, they are displayed, so that the passenger can play his / her favorite music among the music that matches the current vehicle's driving environment.

[0062] [Third embodiment] The third embodiment will be described below with respect to the differences from the first and second embodiments. In the first and second embodiments, it has been described that the content to be played is determined from the content stored in the content storage unit 234. However, the content to be played may be acquired from an external server 300.

[0063] As described in FIG. 3, the server 300 accumulates the traveling information and the content information in association with the passenger of the vehicle 1. That is, the content information and the position information at the timing when the content is reproduced are accumulated in association with each other. In this embodiment, as in the first and second embodiments, when it is determined that the traveling environment of the vehicle 1 has changed and the passenger's emotion has changed, the vehicle 1 transmits the current position information of the vehicle 1 to the server 300 and requests the content. Then, the server 300 searches the database for the transmitted position information and the corresponding content information, and transmits the information to the vehicle 1. Then, the vehicle 1 reproduces the content based on the content information transmitted from the server 300. With such a configuration, if the reproduced content is a song that was reproduced in a place where the passenger has traveled in the past, it can be reproduced as a song that is memorable to the passenger. Also, if the reproduced content is a song that another passenger has reproduced in that place, a fresh feeling can be provided to the passenger.

[0064] Fig. 7 is a flowchart showing the process of content playback control in this embodiment. The process of the vehicle 1 in Fig. 7 is realized, for example, by a processor such as an ECU constituting the processing unit 230 reading and executing a program stored in a memory (not shown). The process of the server 300 in Fig. 7 is realized, for example, by a processor 311 of the control unit 310 reading and executing a program stored in a memory 312. The process in Fig. 7 is started when it is determined in S105 in Fig. 4 or S205 in Fig. 5 that there is a change in the emotion of the passenger.

[0065] If it is determined that the passenger's emotion has changed, in S301, the processing unit 230 of the vehicle 1 transmits the current location information of the vehicle 1 and requests content information. In S302, the control unit 310 of the server 300 refers to the database 330 based on the location information transmitted in S301, and searches for content corresponding to the location information. In S302, the control unit 310 searches for location information stored in the database 330 that matches or is close to the location information transmitted in S301. In this case, not only history information corresponding to the passenger of the vehicle 1, but also history information corresponding to other passengers is subject to the search. The condition may be that the location information is included within a predetermined range, rather than matching or being close to the location information. For example, it may be an area such as a "prefecture" that includes the current location information of the vehicle 1.

[0066] In S303, the control unit 310 determines whether there is any location information corresponding to the location information transmitted in S301, for example, matching the transmitted location information or nearby. If it is determined that there is no corresponding location information, in S308, the control unit 310 transmits information indicating that no search was performed to the vehicle 1, and ends the processing in Fig. 7. On the other hand, if it is determined that there is corresponding location information, the process proceeds to S304.

[0067] In S304, the control unit 310 acquires content information corresponding to the searched location information. In S304, the content information may be acquired from history information corresponding to an occupant of the vehicle 1, or may be acquired from history information corresponding to an occupant different from the occupant of the vehicle 1. In S305, the control unit 310 transmits the content information acquired in S304 to the vehicle 1 as a response to the content request in S301.

[0068] In S306, the processing unit 230 of the vehicle 1 acquires the content information transmitted in S305. At that time, the processing unit 230 may acquire the content information via the communication interface 224 and the control unit 200, or may acquire the content information from the communication interface 224 without via the control unit 200.

[0069] In S307, the processing unit 230 requests the control unit 200 to display information about the content to be played on the display device 221 based on the content information transmitted from the server 300. When the control unit 200 receives the information about the content to be played from the processing unit 230, it displays a message screen recommending content playback on the display device 221. Then, the processing in FIG. 7 ends.

[0070] The message screen displayed in S307 displays the content information recommended for playback. The number of pieces of content information displayed may be one or more. All of the content information transmitted from the server 300 in S305 may be displayed in S307, or a part of the content information may be displayed. For example, among the content information transmitted from the server 300, content information corresponding to the content stored in the content storage unit 234 may be identified, and the identified content information may be displayed. In that case, the message screen displayed in S307 may include a button for accepting a playback instruction, as in FIG. 6. If the content information transmitted from the server 300 corresponds to content not stored in the content storage unit 234, the message screen displayed in S307 may not include a button for accepting a playback instruction and may only provide information about the music. Even if the content information corresponds to content not stored in the content storage unit 234, if there is no restriction on music playback, a button for accepting a playback instruction may be displayed. In this case, when a playback instruction is received, the processing unit 230 acquires the content from the outside via the communication interface 224 and requests the control unit 200 to play back the content on the content playback device 223 .

[0071] As described above, according to this embodiment, the content to be played is acquired from the external server 300. With such a configuration, if the content to be played is a piece of music that was played in a place where the passenger has traveled in the past, it can be played as a piece of music that is memorable to the passenger. Also, if the content to be played is a piece of music that another passenger has played in that place, information on new music can be provided to the passenger.

[0072] In FIG. 4 and FIG. 5, the content stored in the content storage unit 234 is the target of playback. However, the present invention is not limited to such a configuration. For example, the content stored in a mobile terminal such as a smartphone held by a passenger of the vehicle 1 may be the target of playback. In this case, the processing unit 234 communicates with the mobile terminal via the communication interface 224 in S101 and S201, and analyzes information on the content stored in each mobile terminal. Then, when the content to be played is determined, the processing unit 234 may instruct the mobile terminal holding the content to play it. Also, in FIG. 4, FIG. 5, and FIG. 7, the content information is described as being displayed on the display device 221 of the vehicle 1. However, the present invention is not limited to such a configuration. For example, the information may be displayed on a mobile terminal such as a smartphone held by a passenger of the vehicle 1. Also, the above-mentioned configuration may be realized by downloading an app that can communicate with the processing unit 230 to the mobile terminal in advance, and the passenger executes the app.

[0073] 2 has been described as being configured in the vehicle 1, the present invention is not limited to such a configuration. For example, the processing unit 230 may be configured in an external device such as a server. Also, the processing unit 230 may be configured to be able to execute some of the functions of the control unit 200, or the processing unit 230 may be configured to be able to execute some of the functions of the processing unit 230.

[0074] <Summary of the embodiment> The control device of the above embodiment includes a recognition means (201) for recognizing the driving environment of the vehicle, an estimation means (233) for estimating the emotion of the occupant of the vehicle, a determination means (236) for determining the content to be reproduced based on the change in the driving environment recognized by the recognition means and the emotion estimated by the estimation means, and a control means (230) for controlling the reproduction of the content determined by the determination means. The control device further includes a first determination means (S204) for determining the presence or absence of a change in the driving environment recognized by the recognition means, and the determination means determines the content to be reproduced when the first determination means determines that there is a change in the driving environment. The first determination means determines that there is a change in the driving environment when the driving scene becomes a predetermined driving scene. The predetermined driving scene is a driving scene in which the driving load of the occupant of the vehicle is reduced. The first determination means determines the presence or absence of a change in the driving environment based on at least one change in a signal indicating the driving operation of the occupant of the vehicle and bioinformation. The control device further includes a second judgment means (S205) for judging whether or not there is a change in the emotion estimated by the estimation means, and the determination means determines the content to be played back when the second judgment means judges that there is a change in the emotion.

[0075] With such a configuration, it is possible to appropriately provide the passenger with an opportunity to play back the content based on changes in the driving environment and the passenger's emotions.

[0076] Moreover, the control means controls a content reproduction device (223) to reproduce the content determined by the determination means.

[0077] With such a configuration, the content can be played back without any user operation.

[0078] The control means also causes a display device (221, FIG. 6) to display a screen including information on the content determined by the determination means. The screen is also capable of receiving an instruction to play the content (603), and when the control means receives the instruction, the control means controls the content playback device to play the content.

[0079] With such a configuration, it is possible to play back content in response to a user instruction.

[0080] The control device further includes an analysis means (235) for analyzing information about the content, and the determination means determines, based on the analysis result by the analysis means, that the content determined to correspond to the traveling environment recognized by the recognition means as the content to be reproduced. The analysis means also analyzes characteristics of text information about the content. The characteristics include at least one of a place name, a scene, and a season.

[0081] With such a configuration, for example, when the vehicle 1 is traveling through a place name that is frequently included in the lyric information of a content, the content can be reproduced.

[0082] The control device further includes a request means (S301) that requests information on content corresponding to the location information of the vehicle from an external device based on the location information of the vehicle, and the determination means determines the content obtained from the external device in response to the request by the request means as the content to be played.

[0083] Such a configuration may allow, for example, playback of content that was played in a location that the passenger or other passengers have previously traveled over.

[0084] The recognition means recognizes the traveling environment based on at least one of the vehicle information, traffic information, map information, and weather information.

[0085] With such a configuration, it is possible to recognize changes in the driving scene based on, for example, vehicle position information, speed information, traffic congestion information, weather, and destination.

[0086] The estimation unit estimates the emotion of the vehicle occupant based on at least one of a voice and a face image. With this configuration, it is possible to provide an opportunity to play back content when the occupant smiles, for example.

[0087] In addition, the content is music. With this configuration, it is possible to appropriately provide the passenger with an opportunity to play music based on the driving environment and the passenger's emotions.

[0088] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0089] 1 Vehicle: 200 Control unit: 230 Processing unit: 231 Voice recognition unit: 232 Image recognition unit: 233 Emotion estimation unit: 234 Content storage unit: 235 Content analysis unit: 236 Content decision unit: 237 Travel information analysis unit: 300 Server

Claims

1. A recognition means for recognizing a driving environment of a vehicle; a first determination means for determining whether or not there is a change in the traveling environment recognized by the recognition means; An estimation means for estimating an emotion of an occupant of the vehicle; a determination means for determining a content to be reproduced based on the determination by the first determination means that there is a change in the traveling environment and the change in the emotion estimated by the estimation means; a control means for performing control for playing back the content determined by the determination means; Equipped with The first determination means determines that there is a change in the driving environment when a driving scene of the vehicle becomes a driving scene in which a driving load of a passenger of the vehicle is expected to be reduced, When it is determined that at least one of the driving environment and the emotion estimated by the estimation means has not changed, the determination means is not allowed to determine the content to be reproduced and the control means is not allowed to reproduce the content. A control device comprising:

2. 2. The control device according to claim 1, wherein the first determination means determines whether or not there is a change in the driving environment based on a change in at least one of a signal indicating a driving operation of a vehicle occupant and biological information.

3. A second determination means for determining whether or not there is a change in the emotion estimated by the estimation means, The determination means determines the content to be reproduced when the second determination means determines that there is a change in the emotion.

3. The control device according to claim 1 or 2.

4. 4. The control device according to claim 1, wherein the control means controls a content reproducing device to reproduce the content determined by the determination means.

5. 5. The control device according to claim 4, wherein the control means causes a display device to display a screen including information on the content determined by the determination means.

6. the screen is capable of receiving an instruction to play back the content; The control device according to claim 5 , wherein the control means controls the content playback device to play back the content when the control means receives the instruction.

7. An analysis means for analyzing information of the content, The determination means determines, based on a result of the analysis by the analysis means, the content determined to correspond to the traveling environment recognized by the recognition means as the content to be reproduced.

7. The control device according to claim 1, wherein the control device is a control unit.

8. The control device according to claim 7 , wherein the analysis means analyzes characteristics of text information of the content.

9. The control device according to claim 8 , wherein the features include at least one of a place name, a scene, and a season.

10. a request unit for requesting information on a content corresponding to the position information from an external device based on the position information of the vehicle; the determining means determines the content acquired from the external device in response to the request by the requesting means as the content to be reproduced; 7. The control device according to claim 1, wherein the control device is a control unit.

11. 11. The control device according to claim 1, wherein the recognition means recognizes the driving environment based on at least one of information about the vehicle, traffic information, map information, and weather information.

12. 12. The control device according to claim 1, wherein the estimation means estimates the emotion of the passenger of the vehicle based on at least one of a voice and a face image.

13. The control device according to claim 1 , wherein the content is a piece of music.

14. A control method executed in a control device, comprising: A recognition step of recognizing a driving environment of a vehicle; a determination step of determining whether or not there is a change in the traveling environment recognized in the recognition step; an estimation step of estimating an emotion of an occupant of the vehicle; a determination step of determining a content to be reproduced based on the determination step that there is a change in the traveling environment and the change in the emotion estimated in the estimation step; a control step of performing control for playing back the content determined in the determination step; having In the determination step, when a driving scene of the vehicle becomes a driving scene in which a driving load of a passenger of the vehicle is expected to be reduced, it is determined that there is a change in the driving environment, When it is determined that at least one of the traveling environment and the emotion estimated in the estimation step has not changed, the determination of the content to be reproduced in the determination step and the reproduction of the content in the control step are not performed. A control method comprising:

15. Computer, A recognition means for recognizing a driving environment of a vehicle; a determination means for determining whether or not there is a change in the traveling environment recognized by the recognition means; An estimation means for estimating an emotion of an occupant of the vehicle; a determination means for determining content to be reproduced based on the determination by the determination means that there is a change in the traveling environment and the change in the emotion estimated by the estimation means; a control means for performing control for playing back the content determined by the determination means; Functioning as a The determination means determines that there is a change in the driving environment when a driving scene of the vehicle becomes a driving scene in which a driving load of a passenger of the vehicle is expected to be reduced, When it is determined that at least one of the driving environment and the emotion estimated by the estimation means has not changed, the determination means is not allowed to determine the content to be reproduced and the control means is not allowed to reproduce the content. A program characterized by:

Images