Occupant assistance system and occupant assistance method

Abstract

The invention relates to an occupant assistance system and an occupant assistance method. An occupant assistance system (100) that assists an occupant of a vehicle detects an object present around the vehicle on the basis of external environment data relating to the environment around the vehicle, and detects the line-of-sight direction of the occupant on the basis of occupant data indicating the state of the occupant of the vehicle. An object to be viewed by the occupant is identified on the basis of the object and the line-of-sight direction of the occupant, additional information relating to the object to be viewed is acquired when the type of the object to be viewed is a building, and the object to be viewed is identified on the basis of the content of speech when the occupant is viewing the object to be viewed and the additional information relating to the object to be viewed. A demand for a visual target is determined, and an action for coping with the demand is performed. As a result, when there is a certain demand for a visually recognized shop during driving, it is possible to cope with the demand while continuing driving.

Description

Technical Field

[0001] The present invention relates to an occupant assistance system (including a driver and other passengers in a vehicle) and an occupant assistance method. Background Art

[0002] Patent Document 1 discloses a conventional information presentation system that presents information (eg, advertisements of stores located at specific locations) associated with a specific location in the outside world that the driver of a vehicle sees through a windshield to the driver.

[0003] Prior art literature

[0004] Patent Document 1: Japanese Patent Application Publication No. 2002-267470 Summary of the invention

[0005] Problems to be solved by the invention

[0006] However, the above-mentioned conventional information prompting system is a system that prompts the driver with predetermined information stored in a database as information associated with a specific location. Therefore, in the conventional system, there is a problem that when there is a certain demand (request, important matter) for a store visually recognized during driving, the demand cannot be processed while continuing driving.

[0007] The present invention has been made with focus on such a problem, and an object of the present invention is to realize that, when there is a demand for a store visually recognized during driving, the demand can be processed while the vehicle continues driving.

[0008] Technical solutions to solve problems

[0009] In order to solve the above-mentioned problem, a technical solution of the present invention is an occupant assistance system for assisting a vehicle occupant, which is constructed by detecting a target object existing around the vehicle based on external environment data related to the environment around the vehicle, detecting the line of sight of the occupant based on occupant data indicating the status of the vehicle occupant, determining a visual target viewed by the occupant based on the target object and the line of sight of the occupant, obtaining additional information related to the visual target when the type of the visual target is a building, determining a need for the visual target based on the speech content of the occupant when viewing the visual target and the additional information related to the visual target, and executing an action to cope with the need.

[0010] In addition, a technical solution of the present invention is an occupant assistance method for assisting a vehicle occupant, including: detecting a target object existing around the vehicle based on external environment data related to the environment around the vehicle, detecting the line of sight of the occupant based on occupant data indicating the status of the vehicle occupant, determining a visual target viewed by the occupant based on the target object and the line of sight of the occupant, obtaining additional information related to the visual target when the type of the visual target is a building, determining a need for the visual target based on the speech content of the occupant when viewing the visual target and the additional information related to the visual target, and executing an action to respond to the need.

[0011] Effects of the Invention

[0012] According to the technical solutions of the present invention, when there is a demand for a store related to a visual target object visually recognized during driving, the vehicle occupants can handle the demand for the store visually recognized during driving while continuing to drive by combining the line of sight direction and the content of speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic configuration diagram of an occupant assistance system according to one embodiment of the present invention.

[0014] Figure 2 This is a diagram showing an example of an operation sequence of the occupant assistance system according to one embodiment of the present invention.

[0015] Description of symbols

[0016] 4. HMI (Human Machine Interface)

[0017] 100: Occupant assistance systems DETAILED DESCRIPTION

[0018] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, the same reference numerals are given to the same components.

[0019] Figure 1 It is a schematic configuration diagram of an occupant assistance system 100 according to an embodiment of the present invention.

[0020] The occupant assistance system 100 is a system for assisting an occupant of the vehicle while the vehicle is traveling. In the present embodiment, an example in which the occupant assistance system 100 is applied to a driver of the vehicle is described.

[0021] The passenger assistance system 100 includes an external environment sensor 1, a driver sensor 2, a positioning sensor 3, an HMI 4, a communication device 5, and a control device 6. The external environment sensor 1, the driver sensor 2, the positioning sensor 3, the HMI 4, the communication device 5, and the control device 6 are connected to each other so as to be communicable via an in-vehicle network 110 conforming to a standard such as a controller area network or an in-vehicle Ethernet.

[0022] The external environment sensor 1 is a sensor for generating data related to the environment around the vehicle (hereinafter referred to as "external environment data"). In the present embodiment, as the external environment sensor 1, one or more external cameras (camera) 11 for photographing the surroundings of the vehicle are provided. The external camera 11 photographs the surroundings of the vehicle at a predetermined frame rate (for example, 10 [Hz] to 40 [Hz]) and generates a surrounding image reflecting the surroundings of the vehicle. Whenever the external camera 11 generates a surrounding image, the generated surrounding image is sent to the control device 6 via the in-vehicle network 110 as the external environment data.

[0023] The driver sensor 2 is a sensor for generating data indicating the state of the driver (hereinafter referred to as "driver data"). In the present embodiment, as the driver sensor 2, a driver monitoring camera 21 for capturing the appearance of the driver including the driver's face is provided. The driver monitoring camera 21 captures the appearance of the driver at a predetermined frame rate (e.g., 10 [Hz] to 40 [Hz]) and generates an appearance image reflecting the appearance of the driver. Whenever the driver monitoring camera 21 generates an appearance image of the driver, it sends the generated appearance image as driver data to the control device 6 via the in-vehicle network 110.

[0024] The positioning sensor 3 acquires data indicating the current position of the vehicle, such as latitude and longitude (hereinafter referred to as “current position data”). Whenever the positioning sensor 3 acquires the current position data, it transmits the acquired current position data to the control device 6 via the in-vehicle network 110 .

[0025] HMI4 is a user interface for exchanging information between the vehicle and its passengers. HMI4 includes an output device 41 for notifying the vehicle passengers through the vehicle passengers' body senses (e.g., vision, hearing, touch, etc.) and an input device 42 for the vehicle passengers to perform input operations and response operations. In this embodiment, as the output device 41, a display type (e.g., instrument display, center display, head-up display, etc.) 411 and a speaker 412 are provided, and as the input device 42, a touch panel 421 and a microphone 422 are provided.

[0026] The HMI 4 displays information (e.g., text information, image information) corresponding to the display signal received from the control device 6 on the display 411, and outputs voice corresponding to the voice signal from the speaker 412. In addition, the HMI 4 transmits data inputted by the vehicle occupant through the panel or voice input via the input device 42 (hereinafter referred to as "occupant input data") to the control device 6 via the in-vehicle network 110.

[0027] In addition, HMI4 can be pre-installed in the vehicle as in the present embodiment, or can be a terminal such as a smart phone owned by the driver. In this case, information can be exchanged between the driver's terminal and the vehicle by using short-range wireless communication, or the driver's terminal can communicate with an external server (not shown) to exchange information with the vehicle indirectly via the server.

[0028] The communication device 5 has a communication interface circuit for connecting to a network constituted by an optical communication line or the like, and is configured to connect the vehicle to, for example, an external server (not shown) via the network and to enable mutual communication with the vehicle.

[0029] The control device 6 is an ECU (Electronic Control Unit) including a communication unit 61 , a storage unit 62 , and a processing unit 63 .

[0030] The communication unit 61 includes an interface circuit for connecting the control device 6 to the in-vehicle network 110. The communication unit 61 supplies various data received from the sensors 1 to 3, the HMI 4, the communication device 5, etc. to the processing unit 63. In addition, the communication unit 61 transmits various signals output from the processing unit 63 to the HMI 4, the communication device 5, etc.

[0031] The storage unit 62 includes storage media such as a HDD (Hard Disk Drive), an SSD (Solid Disk Drive), and a semiconductor memory, and stores various computer programs and data used for processing in the processing unit 63 .

[0032] The processing unit 63 has one or more CPUs (Central Processing Units) and their peripheral circuits, and executes various computer programs stored in the storage unit 62, such as a processor. The processing unit 63 performs the functions of the target object detection unit 71, the line of sight detection unit 72, the visual target determination unit 73, the voice detection unit 74, and the demand processing unit 75 by executing the processing according to the computer program, and acts as a functional unit (module) that realizes the predetermined function. In the following description, when the processing is explained with each functional unit 71 to 75 as the subject, it means that the processing unit 63 is executing the program that realizes each functional unit 71 to 75.

[0033] Hereinafter, the content of specific processing performed in the control device 6 will be described. That is, the content of each functional unit 71 to 75 realized by the processing unit 63 executing processing according to the program will be described.

[0034] The target object detection unit 71 detects areas representing targets such as other vehicles, pedestrians, and buildings in the surrounding images and the types of targets shown in the areas by sequentially inputting the surrounding images received from the external camera 11 into the recognizer. The recognizer can be, for example, a convolutional neural network (CNN) having a plurality of convolutional layers connected in series from the input side to the output side. The target object detection unit 71 estimates the distance from the external camera 11 to the target object using the standard size of the target object stored in the storage unit 62 according to the various types of the target object and the size of the target object detected in the surrounding images, and calculates the position and speed of the target object by tracking (tracing) the target object detected in the surrounding images in a time series. In addition, the method for detecting the target object is not limited to such a method, and detection can be performed using various known methods.

[0035] The sight direction detection unit 72 detects the position of the pupil center (i.e., the driver's viewpoint position) as the moving point and the position of the Purkinje image (corneal reflection image) as the reference point by sequentially performing image processing on the driver's appearance image (facial image) received from the driver monitoring camera 21, and detects the driver's sight direction based on the positional relationship between the Purkinje image and the pupil center (the position of the moving point relative to the reference point). In addition, the detection method of the driver's sight direction is not limited to such a method, and detection can be performed using various known methods.

[0036] The visual target determination unit 73 determines the target that the driver is visually viewing (hereinafter referred to as the "visual target") based on the targets around the vehicle and the driver's line of sight. The visual target determination unit 73 obtains additional information related to the visual target as needed. For example, in the present embodiment, if the type of the visual target is a building, the visual target determination unit 73 obtains additional information related to the visual target such as the name of the building, the name of the store in the building, and the content of the service based on the image of the visual target in the surrounding image, the current position data, the distance to the visual target, and the map data stored in the storage unit 62. In addition, when obtaining the additional information, the visual target determination unit 73 can also use the Internet via the communication device 5.

[0037] The voice detection unit 74 detects the speech sound of the driver (vehicle occupant) via the microphone 422 , and converts it into desired data (for example, text data of the speech sound, etc.) that can be processed by the request processing unit 75 .

[0038] The demand processing unit 75 determines the driver's demand based on the speech data, the visual target, and the additional information of the visual target, for example, using a machine learning model (AI) that has been learned such as a neural network, and performs demand processing for responding to the driver's demand. When determining the driver's demand, for example, the driver's gesture (finger pointing action, predetermined action, etc.) that can be detected from the driver's appearance image can also be added to the judgment material.

[0039] The demand processing includes: searching the website of the store involved in the visual target using the Internet, browsing a specific page in the website of the store, starting a dedicated application for making reservations at the store, etc., various confirmation processes (such as confirmation of availability, etc.) of the store on the website and / or the dedicated application, making reservations and / or ordering at the store from the website and / or the dedicated application, and contacting the store by phone, etc. In addition, the demand processing also includes: notifying the driver (vehicle occupant) of the processing results of each process such as the aforementioned search processing via the HMI 4.

[0040] In this way, the passenger assistance system 100 of the present embodiment uses the visual target and its additional information in addition to the speech content to determine the driver's needs. Therefore, even if the driver happens not to have information related to the visual target such as the name of the store, by combining the line of sight and the speech content, such as saying "Tell me the menu of that store" or "Please make a reservation for me at that store", it is possible to easily handle various needs (for example, confirmation of the availability of a visually recognized store, reservation of a visually recognized store, browsing and ordering of products at a visually recognized store, calling a visually recognized store, activation of an application associated with a visually recognized store, and confirmation of information related to a visually recognized store, etc.) while driving.

[0041] Figure 2 1 is a diagram showing an example of an operation sequence of the occupant assistance system 100 .

[0042] In step S1 , the occupant assistance system 100 determines the driver's visual target based on the targets around the vehicle and the driver's line of sight, and acquires additional information related to the visual target as needed.

[0043] In step S2 , when the driver's speech is detected via the microphone 422 , the occupant assistance system 100 determines the driver's demand based on the speech data, the visual target, and the additional information of the visual target, and executes demand processing for responding to the driver's demand.

[0044] In step S3 , the occupant assistance system 100 notifies the driver (or the vehicle occupant) of the processing result of the request processing via the HMI 4 .

[0045] In step S4 , if the occupant assistance system 100 detects the driver's voice via the microphone 422 within a predetermined time from notification of the processing result, for example, it determines the driver's additional demand generated by the processing result of the demand processing and further executes demand processing for responding to the additional demand.

[0046] In step S5, the passenger assistance system 100 notifies the driver (or vehicle occupant) of the processing result of the additional demand via the HMI 4. By repeatedly performing the processing of steps S4 and S5, the driver can repeatedly interact with the passenger assistance system 100 even while driving until the demand for the store involved in the visual target object is met.

[0047] The occupant assistance system 100 of the present embodiment described above is configured to detect a target object existing around the vehicle based on external environment data related to the environment around the vehicle, detect the line of sight of the occupant based on occupant data indicating the state of the occupant of the vehicle, determine the visual target viewed by the occupant based on the target object and the line of sight of the occupant, obtain additional information related to the visual target when the type of the visual target is a building, determine the need for the visual target based on the speech content of the occupant when viewing the visual target and the additional information related to the visual target, and perform an action for responding to the need. The need for the visual target is, for example, a need related to a service provided by a store related to the visual target.

[0048] Therefore, even if the driver happens not to have information related to the visual target, such as the name of a store, by combining the line of sight and the content of the speech, the driver can confirm and search for the visual target without stopping the car, and take various actions such as requesting service provision for the store involved in the visual target while continuing to drive.

[0049] In addition, the occupant assistance system 100 of the present embodiment is configured to notify the driver of the result of responding to the demand via the HMI4 that is capable of transmitting and receiving information (giving and receiving) between the vehicle and its driver (vehicle occupant), and when the driver speaks in response to the result notification, additional demand is determined based on the driver's speech content, and actions for responding to the additional demand are executed.

[0050] As a result, it is possible to present the driver with information required by the driver with high accuracy, and to meet the driver's needs with high accuracy.

[0051] Although the embodiments of the present invention have been described above, the above embodiments are merely a part of application examples of the present invention and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments.

[0052] For example, in the above-mentioned embodiment, the processing unit 63 of the control device 6 performs the demand processing, but the external server may perform part or all of the demand processing by communicating with the external server via the communication device 5. In this case, the text data of the speech sound, the visual target object, and the additional information of the visual target object may be transmitted to the operator on the server side, and the operator may perform part of the demand processing such as the reservation processing. In addition, at this time, it may be possible to make a voice call with the operator, or it may be possible to send the surrounding image received from the external camera 11 to the operator and share the surrounding image.

[0053] In the above-described embodiment, the computer program executed in the control device 6 may be provided in the form of being recorded on a computer-readable removable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

Claims

1. An occupant assistance system is a system that assists the occupants of a vehicle. detecting a target object existing around the vehicle based on external environment data related to the environment around the vehicle, detecting a line of sight direction of the occupant based on occupant data indicating a state of the occupant of the vehicle, Based on the target object and the line of sight of the occupant, determining the visual target object that the occupant is visually viewing, When the type of the visual target object is a building, obtaining additional information related to the visual target object, Based on the speech content of the occupant when visually viewing the visual target and the additional information related to the visual target, a need for the visual target is determined, and an action for responding to the need is performed.

2. The occupant assistance system according to claim 1, The result of responding to the demand is notified to the occupant via a human-machine interface capable of transmitting and receiving information between the vehicle and the occupant.

3. The occupant assistance system according to claim 2, When the occupant speaks in response to the notification of the result, an additional demand is determined based on the content of the occupant's speech, and an action for responding to the additional demand is executed.

4. The occupant assistance system according to any one of claims 1 to 3, The demand is a demand related to a service provided by a store related to the visual target.

5. An occupant assistance method is a method for assisting an occupant of a vehicle, comprising: detecting a target object existing around the vehicle based on external environment data related to the environment around the vehicle, detecting a line of sight direction of the occupant based on occupant data indicating a state of the occupant of the vehicle, Based on the target object and the line of sight of the occupant, determining the visual target object that the occupant is visually viewing, When the type of the visual target object is a building, obtaining additional information related to the visual target object, Based on the speech content of the occupant when visually viewing the visual target and the additional information related to the visual target, a need for the visual target is determined, and an action for responding to the need is performed.

Images