Occupant support system and occupant support method

Abstract

To allow a user to cope with a requirement while continuing to travel when having any requirement for a shop visually recognized during travelling.SOLUTION: An occupant support system 100 that supports an occupant in a vehicle, detects a target existing around the vehicle on the basis of external environment data relating to an environment around the vehicle, detects a visual line direction of the occupant on the basis of occupant data indicating a state of the occupant in the vehicle, identifies a visual observation target visually observed by the occupant on the basis of the target and the visual line direction of the occupant, acquires additive information relating to the visual observation target when a kind of the visual observation target is a building, and identifies a requirement for the visual observation target and executes action for coping with the requirement, on the basis of an utterance content of the occupant at the time when visually observing the visual observation target and additive information relating to the visual observation target.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an occupant assistance system and an occupant assistance method. [Background technology]

[0002] Patent document 1 discloses a conventional information presentation system that presents to the driver information associated with a specific location in the outside world that the driver sees through the windshield (e.g., an advertisement for a store located at the specific location). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2002-267470 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional information presentation system described above presents predetermined information stored in a database to the driver as information associated with a specific location, and therefore, in this conventional system, when there is a requirement for a store that is viewed while driving, the requirement cannot be processed while continuing driving.

[0005] The present invention has been made in response to such problems, and has as its object to make it possible, when a requirement is found for a store that is viewed while driving, to process the requirement while continuing driving. [Means for solving the problem]

[0006] In order to solve the above problems, an occupant assistance system for assisting an occupant of a vehicle according to one aspect of the present invention is configured to detect targets present around the vehicle based on external environmental data related to the environment surrounding the vehicle, detect the occupant's line of sight based on occupant data representing the state of the vehicle occupant, identify the visual target that the occupant is looking at based on the target and the occupant's line of sight, obtain additional information related to the visual target if the type of visual target is a building, and identify requirements for the visual target and perform actions to address the requirements based on the content of the occupant's speech when looking at the visual target and the additional information related to the visual target.

[0007] In addition, according to one aspect of the present invention, an occupant assistance method for assisting a vehicle occupant detects targets present around the vehicle based on external environmental data related to the environment around the vehicle, detects the occupant's line of sight based on occupant data representing the state of the vehicle occupant, identifies the visual target that the occupant is looking at based on the target and the occupant's line of sight, obtains additional information related to the visual target if the type of visual target is a building, and identifies requirements for the visual target and performs actions to address the requirements based on the content of the occupant's speech when looking at the visual target and the additional information related to the visual target. Effect of the Invention

[0008] According to these aspects of the present invention, when there is a requirement for a store related to a visual landmark seen while driving, the vehicle occupant can process the requirement for the store seen while driving by combining the direction of gaze and the content of speech while continuing driving. [Brief description of the drawings]

[0009] [Figure 1] 1 is a schematic configuration diagram of an occupant assistance system according to an embodiment of the present invention; [Diagram 2] 5 is a diagram showing an example of an operation sequence of the occupant assistance system according to the embodiment of the present invention. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the following description, like components are designated by like reference numerals.

[0011] FIG. 1 is a schematic diagram of an occupant assistance system 100 according to an embodiment of the present invention.

[0012] The passenger assistance system 100 is a system for providing assistance to a passenger of a vehicle while the vehicle is traveling. In this embodiment, the passenger assistance system 100 will be described as being applied to a driver of the vehicle.

[0013] The occupant assistance system 100 includes an external environment sensor 1, a driver sensor 2, a positioning sensor 3, an HMI (Human Machine Interface) 4, a communication device 5, and a control device 6. The external environment sensor 1, the driver sensor 2, the positioning sensor 3, the HMI 4, the communication device 5, and the control device 6 are communicatively connected to each other via an in-vehicle network 110 that complies with standards such as a controller area network or in-vehicle Ethernet.

[0014] The external environment sensor 1 is a sensor for generating data related to the environment around the vehicle (hereinafter referred to as "external environment data"). In this embodiment, the external environment sensor 1 includes one or more external cameras 11 for capturing images of the surroundings of the vehicle. The external camera 11 captures images of the surroundings of the vehicle at a predetermined frame rate (for example, 10 [Hz] to 40 [Hz]) and generates an surrounding image showing the surroundings of the vehicle. Every time the external camera 11 generates an surrounding image, it transmits the generated surrounding image to the control device 6 as external environment data via the in-vehicle network 110.

[0015] The driver sensor 2 is a sensor for generating data representing the state of the driver (hereinafter referred to as "driver data"). In this embodiment, the driver sensor 2 includes a driver monitor camera 21 for capturing an image of the driver's appearance including the driver's face. The driver monitor camera 21 captures an image of the driver's appearance at a predetermined frame rate (for example, 10 [Hz] to 40 [Hz]) and generates an appearance image showing the driver's appearance. Every time the driver monitor camera 21 generates an appearance image of the driver, it transmits the generated appearance image to the control device 6 as driver data via the in-vehicle network 110.

[0016] The positioning sensor 3 acquires data indicating the current position of the vehicle, such as latitude and longitude (hereinafter referred to as “current position data”). Every time the positioning sensor 3 acquires current position data, the positioning sensor 3 transmits the acquired current position data to the control device 6 via the in-vehicle network 110.

[0017] The HMI 4 is a user interface for exchanging information between the vehicle and its occupants. The HMI 4 includes an output device 41 for notifying the vehicle occupant through the vehicle occupant's bodily senses (e.g., vision, hearing, touch, etc.), and an input device 42 for the vehicle occupant to perform input operations and response operations. In this embodiment, the output device 41 includes displays (e.g., a meter display, a center display, a head-up display, etc.) 411 and a speaker 412, and the input device 42 includes a touch panel 421 and a microphone 422.

[0018] The HMI 4 displays information (e.g., text information or image information) corresponding to a display signal received from the control device 6 on a display 411, and outputs a sound corresponding to a sound signal from a speaker 412. The HMI 4 also transmits data input from a panel or by voice by a vehicle occupant via an input device 42 (hereinafter referred to as "occupant input data") to the control device 6 via the in-vehicle network 110.

[0019] The HMI 4 may be pre-installed in the vehicle as in this embodiment, or may be a terminal such as a smartphone owned by the driver. In this case, information may be exchanged between the driver's terminal and the vehicle by short-range wireless communication, or information may be exchanged indirectly with the vehicle via communication between the driver's terminal and an external server (not shown).

[0020] The communication device 5 has a communication interface circuit for connecting to a network consisting of optical communication lines or the like, and is configured to connect the vehicle to, for example, an external server (not shown) via the network so as to enable mutual communication therebetween.

[0021] The control device 6 is an ECU (Electronic Control Unit) including a communication unit 61, a storage unit 62, and a processing unit 63.

[0022] The communication unit 61 includes an interface circuit for connecting the control device 6 to the in-vehicle network 110. The communication unit 61 supplies various data received from the sensors 1 to 3, the HMI 4, the communication device 5, etc. to the processing unit 63. The communication unit 61 also transmits various signals output from the processing unit 63 to the HMI 4, the communication device 5, etc.

[0023] The storage unit 62 has a storage medium such as a hard disk drive (HDD), a solid disk drive (SSD), or a semiconductor memory, and stores various computer programs and data used for processing in the processing unit 63.

[0024] The processing unit 63 has one or more CPUs (Central Processing Units) and their peripheral circuits, and executes various computer programs stored in the storage unit 62, and is, for example, a processor. The processing unit 63 executes processing according to the computer programs, thereby functioning as a target detection unit 71, a gaze direction detection unit 72, a visual target identification unit 73, a voice detection unit 74, and a requirement processing unit 75, and operates as functional units (modules) that realize predetermined functions. In the following description, when the processing is described with each of the functional units 71 to 75 as the subject, it indicates that the processing unit 63 is executing a program that realizes each of the functional units 71 to 75.

[0025] The following describes the specific processing carried out by the control device 6. That is, the following describes the contents of each of the functional units 71 to 75 realized by the processing unit 63 executing processing according to a program.

[0026] The target detection unit 71 sequentially inputs the surrounding images received from the external camera 11 to a classifier, thereby detecting an area in the surrounding images where targets such as other vehicles, pedestrians, and buildings are depicted, and the type of the target depicted in the area. The classifier can be, for example, a convolutional neural network (CNN) having a plurality of convolution layers connected in series from the input side to the output side. The target detection unit 71 estimates the distance from the external camera 11 to the target using the standard size of the target stored in the storage unit 62 for each type of target and the size of the target detected in the surrounding images, and calculates the position and speed of the target by tracking the target detected in the surrounding images in a time series. Note that the method of detecting the target is not limited to such a method, and various known methods may be used for detection.

[0027] The gaze direction detection unit 72 detects the position of the pupil center, which is the moving point (i.e., the driver's viewpoint position), and the position of the Purkinje image (corneal reflection image), which is the reference point, by sequentially performing image processing on the driver's appearance image (face image) received from the driver monitor camera 21, and detects the driver's gaze direction based on the positional relationship between the Purkinje image and the pupil center (the position of the moving point relative to the reference point). Note that the method for detecting the driver's gaze direction is not limited to this method, and various known methods may be used for detection.

[0028] The visual target identification unit 73 identifies a target (hereinafter referred to as a "visual target") that the driver is looking at based on targets around the vehicle and the driver's line of sight. The visual target identification unit 73 acquires additional information related to the visual target as necessary. For example, in this embodiment, if the type of visual target is a building, the visual target identification unit 73 acquires additional information related to the visual target, such as the name of the building, the name of the store in the building, and the service content, based on the image of the visual target in the surrounding image, the current position data, the distance to the visual target, and the map data stored in the storage unit 62. Note that the visual target identification unit 73 can also use the Internet via the communication device 5 to acquire the additional information.

[0029] The voice detection unit 74 detects the speech of the driver (vehicle occupant) via the microphone 422, and converts it into desired data that can be processed by the requirement processing unit 75 (for example, text data of the speech).

[0030] The requirement processing unit 75 identifies the driver's requirements using a trained machine learning model (AI) such as a neural network based on the data of the spoken voice, the visual target, and additional information of the visual target, and executes requirement processing to deal with the driver's requirements. When identifying the driver's requirements, for example, the driver's gestures (pointing action, predetermined action, etc.) that can be detected from the driver's appearance image can also be added to the judgment information.

[0031] The requirement processing includes a search process for the website of the store related to the visual target using the Internet, a process for browsing a specific page in the website of the store, a process for starting a dedicated application that enables reservations for the store, etc., various confirmation processes for the store on the website or dedicated application (e.g., checking availability, etc.), reservation processing and order processing for the store from the website or dedicated application, and telephone contact processing for the store, etc. The requirement processing also includes a notification process for notifying the driver (vehicle occupant) of the results of each process such as the above-mentioned search process via the HMI 4.

[0032] In this way, the passenger assistance system 100 according to the present embodiment identifies the driver's requirements using the visual target and its additional information in addition to the utterance contents. Therefore, even if the driver does not have information on the visual target such as the name of the store, the driver can easily handle various requirements while driving (for example, checking the availability of the viewed store, making a reservation for the viewed store, viewing and ordering products of the viewed store, calling the viewed store, starting an application related to the viewed store, checking information on the viewed store, etc.) by combining the line of sight and the utterance contents and uttering, for example, "Tell me the menu of that store" or "Make a reservation for that store."

[0033] FIG. 2 is a diagram showing an example of an operation sequence of the occupant assistance system 100. As shown in FIG.

[0034] In step S1, the passenger assistance system 100 identifies a visual target of the driver based on the targets around the vehicle and the line of sight of the driver, and obtains additional information related to the visual target as necessary.

[0035] In step S2, when the occupant assistance system 100 detects the driver's speech via the microphone 422, it identifies the driver's requirements based on the speech data, visual targets, and additional information of the visual targets, and performs requirement processing to address the driver's requirements.

[0036] In step S3, the occupant assistance system 100 notifies the driver (or a vehicle occupant) of the result of the requirement processing via the HMI 4.

[0037] In step S4, if the occupant assistance system 100 detects the driver's speech via the microphone 422, for example, within a predetermined time from the notification of the processing result, it identifies additional requirements of the driver that have arisen in accordance with the processing result of the requirement processing, and further performs requirement processing to address the additional requirements.

[0038] In step S5, the passenger assistance system 100 notifies the driver (or a vehicle occupant) of the processing result that addresses the additional requirement via the HMI 4. By repeating the processing of steps S4 and S5, the driver can repeatedly execute the interaction with the passenger assistance system 100 while driving until the requirement for the store related to the visual target is achieved.

[0039] The occupant support system 100 according to the present embodiment described above is configured to detect targets present around the vehicle based on external environment data related to the environment around the vehicle, detect the line of sight of the occupant based on occupant data indicating the state of the occupant of the vehicle, identify the visual target that the occupant is looking at based on the target and the line of sight of the occupant, obtain additional information related to the visual target if the type of visual target is a building, and identify requirements for the visual target and execute actions to address the requirements based on the speech content of the occupant when looking at the visual target and the additional information related to the visual target. The requirements for the visual target are, for example, requirements related to services provided at a store related to the visual target.

[0040] As a result, even if the driver does not have information about a visual landmark such as the name of a store, by combining the direction of gaze with the content of speech, the driver can take various actions, such as requesting the provision of services, toward the store associated with the visual landmark while continuing driving, without having to stop to check or search for the visual landmark.

[0041] In addition, the occupant assistance system 100 of this embodiment is configured to notify the driver of the results of dealing with the requirements via the HMI 4, which is capable of transmitting information between the vehicle and its driver (vehicle occupant), and if the driver speaks in response to the notification of the results, to identify additional requirements based on the content of the driver's speech and to perform actions to deal with the additional requirements.

[0042] This makes it possible to provide the driver with information that the driver desires with high accuracy, and to accurately address the driver's requirements.

[0043] Although the embodiments of the present invention have been described above, the above-mentioned embodiments merely show some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above-mentioned embodiments.

[0044] For example, in the above embodiment, the requirement processing is executed by the processing unit 63 of the control device 6, but a part or all of the requirement processing may be executed by an external server by communicating with the external server via the communication device 5. In this case, text data of the uttered voice, visual targets, and additional information of the visual targets may be transmitted to an operator on the server side, and a part of the requirement processing such as reservation processing may be executed by the operator. In addition, at that time, it may be possible to have a voice call with the operator, or a surrounding image received from the external camera 11 may be transmitted to the operator to share the surrounding image.

[0045] Furthermore, in the above-described embodiment, the computer program executed in the control device 6 may be provided in a form recorded on a computer-readable portable recording medium, such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. [Explanation of symbols]

[0046] 4. HMI 100 Occupant Assistance Systems

Claims

1. An occupant assistance system for assisting an occupant of a vehicle, comprising: Detecting targets present around the vehicle based on external environment data relating to an environment around the vehicle; Detecting a line of sight direction of an occupant based on occupant data representing a state of the occupant of the vehicle; Identifying a visual target that the occupant is viewing based on the target and the line of sight direction of the occupant; When the type of the visual target is a building, additional information regarding the visual target is obtained; Identifying requirements for the visual target and performing an action to address the requirements based on the speech content of the occupant while visually viewing the visual target and additional information regarding the visual target. Occupant assistance systems.

2. notifying the occupant of the result of the response to the requirement via an HMI capable of transmitting and receiving information between the vehicle and the occupant; 10. An occupant assistance system as claimed in claim 1.

3. if the occupant makes an utterance in response to the notification of the result, identifying an additional requirement based on the content of the occupant's utterance, and performing an action to address the additional requirement; 3. An occupant assistance system as claimed in claim 2.

4. The requirements are requirements regarding services provided at the store related to the visual target, An occupant assistance system according to any one of claims 1 to 3.

5. An occupant assistance method for assisting an occupant of a vehicle, comprising: Detecting targets present around the vehicle based on external environment data relating to an environment around the vehicle; Detecting a line of sight direction of an occupant based on occupant data representing a state of the occupant of the vehicle; Identifying a visual target that the occupant is viewing based on the target and the line of sight direction of the occupant; When the type of the visual target is a building, additional information regarding the visual target is obtained; Identifying requirements for the visual target and performing an action to address the requirements based on the speech content of the occupant while visually viewing the visual target and additional information regarding the visual target. Crew assistance methods.

Images