In-vehicle systems, vehicles, information processing devices, information processing methods, and programs

Abstract

To improve the operability of in-vehicle devices when vehicle users operate them. [Solution] An in-vehicle system comprising a first display device and a second display device, comprising: a storage unit that stores first data which is data for comparing a gesture made by a user of vehicle 1 detected by a sensor with a first gesture; and a control unit that, based on the first data and the output of the sensor, detects that a user has made a first gesture, and, when a first gesture is detected, outputs a predetermined user interface included in the content output to the first display device to the second display device.

Description

Technical Field

[0001] This disclosure relates to vehicle technology.

Background Art

[0002] There is a technology for providing information to reduce the burden on a driver who drives an automobile. In this regard, for example, Patent Document 1 discloses an in-vehicle system in which a display for information provision is arranged near a steering wheel.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] This disclosure aims to improve the operability when a user of a vehicle operates in-vehicle equipment.

Means for Solving the Problems

[0005] One aspect of this disclosure is an in-vehicle system including a first display device and a second display device, the in-vehicle system having a storage unit that stores first data which is data for collating a gesture made by a user of the vehicle, detected by a sensor, with a first gesture, and a control unit that detects that the user has made the first gesture based on the first data and an output of the sensor, and outputs a predetermined user interface included in content output to the first display device to the second display device when detecting the first gesture. It is an in-vehicle system.

[0006] One aspect of this disclosure is, An in-vehicle system comprising a first display device and a second display device, the system having a control unit that uses a sensor to detect a first gesture made by a vehicle user, and, when the first gesture is detected, outputs a predetermined user interface included in the content output to the first display device to the second display device. It is an in-vehicle system.

[0007] One aspect of this disclosure is, An information processing device for an in-vehicle system comprising a first display device and a second display device, wherein the device uses a sensor to detect a first gesture made by the vehicle user, and, when the first gesture is detected, outputs a predetermined user interface included in the content output to the first display device to the second display device. A control unit that performs the following: It is an information processing device.

[0008] One aspect of this disclosure is, An information processing method performed by an information processing device in an in-vehicle system comprising a first display device and a second display device, the method comprising: detecting a first gesture made by a vehicle user using a sensor; and, when the first gesture is detected, outputting a predetermined user interface included in the content output to the first display device to the second display device. It is an information processing method.

[0009] Other embodiments include a vehicle equipped with the above-mentioned in-vehicle system, a program for causing a computer to execute the above-mentioned information processing method, or a computer-readable storage medium that non-temporarily stores the program. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to improve the operability when a user of a vehicle operates in-vehicle equipment.

Brief Description of the Drawings

[0011] [Figure 1] The figure showing the interior of the vehicle 1 according to the first embodiment. [Figure 2] The enlarged view of the steering wheel 4 and its surroundings. [Figure 3] The figure showing an example of the components of the vehicle 1. [Figure 4] The figure showing an example of the components of the vehicle ECU 10. [Figure 5] The figure showing a list of screens output to the outer display. [Figure 6] The figure showing an example of a screen output to the outer display. [Figure 7] The figure showing an example of a screen output to the outer display. [Figure 8] The figure showing a list of screens output to the inner display. [Figure 9] The figure showing an example of a screen output to the inner display. [Figure 10] The figure showing an example of a screen output to the inner display. [Figure 11] The figure showing an example of a screen output to the center display. [Figure 12] The figure showing the state where the operation interface moves from the center display. [Figure 13] The flowchart of the process executed by the HMI control unit according to the first embodiment. [Figure 14] The flowchart of the process executed by the HMI control unit according to the second embodiment. [Figure 15] The figure showing the state where the operation interface moves from the outer display. [Figure 16] The figure showing an example of the display device according to the modification.

BEST MODE FOR CARRYING OUT THE INVENTION

[0012] In recent years, with the advancement of automobiles, displays for providing information are becoming larger, or the number of displays is increasing.

[0013] For example, a method has been proposed in which a large touch panel display (hereinafter referred to as a center display) is arranged between the driver's seat and the passenger seat in the front of the vehicle interior to perform operations and information provision related to the vehicle collectively.

[0014] However, in such a method, when operating components of the vehicle or checking the output information, the driver's line of sight and hands move significantly. For the safety of vehicle driving, it is preferable that the amount of movement of the driver's line of sight and hands be minimized.

[0015] To solve this problem, a vehicle system has been devised in which a display device equipped with a touch panel is arranged near the steering wheel. This makes it possible to operate in-vehicle devices and the like without moving the hands significantly. However, since the display device arranged near the steering wheel is small in size, it cannot replace the center display. That is, when attempting to use the display device arranged near the steering wheel, the amount of information cannot be ensured, and when attempting to use the center display, the amount of hand movement becomes large, which causes a problem.

[0016] Therefore, in the in-vehicle system of the present disclosure, when a driver gives an instruction by a predetermined gesture, a part of the content output to the display device existing in the front of the vehicle interior is moved to the display device near the steering wheel.

[0017] An in-vehicle system according to one aspect of the present disclosure is An in-vehicle system comprising a first display device and a second display device, the system having a control unit that uses a sensor to detect a first gesture made by a vehicle user, and, when the first gesture is detected, outputs a predetermined user interface included in the content output to the first display device to the second display device.

[0018] The sensor is, for example, an image sensor that detects images from an in-vehicle camera. Alternatively, the sensor may be an infrared image sensor or the like.

[0019] The first gesture is a predetermined gesture. For example, the first gesture may be a hand movement from the direction of the first display device to the direction of the second display device, as viewed from the driver. In other words, the first gesture may be a hand movement from the first display device to the second display device.

[0020] When the control unit detects a first gesture made by the user, it causes a predetermined user interface included in the content (screen, etc.) output to the first display device to be output to the second display device. The predetermined user interface is typically a user interface component for touch operation, such as a panel, button, or slider.

[0021] Thus, an in-vehicle system according to one aspect of this disclosure can output a user interface for operating in-vehicle equipment, which is displayed on one display, to another display device (for example, a display device in the driver's hand). This makes it possible to move only the user interface portion of the content displayed on the center display to the driver's hand, thereby improving the operability when operating in-vehicle equipment.

[0022] The first display device is a display device located in the center of the vehicle width direction at the front of the vehicle, and the second display device may be a display device located on both sides of the steering wheel and including a pair of display units having touch panels.

[0023] The first display device may be a center display or the like, and the second display device may be a display device installed near the driver's hands. The pair of display units may have a configuration that allows for touch operation via a touch panel.

[0024] With this configuration, it becomes possible to move the user interface from a center display located at the front and center of the vehicle in the width direction to a display device closer to the driver's hands, making it easier for the driver to operate.

[0025] Furthermore, the control unit may erase the user interface output to the second display device in response to the first gesture from the screen of the first display device. In other words, the user interface may be moved between display devices.

[0026] Furthermore, the predetermined user interface may include GUI components for operating the functions provided by the vehicle.

[0027] Furthermore, if the control unit detects a second gesture of the vehicle user using the sensor, it may erase the user interface output to the second display device from the screen of the second display device and output the user interface to the first display device again.

[0028] The second gesture can be a gesture to return the user interface that has been moved to the second display device back to the first display device.

[0029] Embodiments of this disclosure will be described below with reference to the drawings. The configurations of the following embodiments are illustrative, and this disclosure is not limited to the configurations of these embodiments.

[0030] (First embodiment) An overview of the vehicle system according to this embodiment will be described. The vehicle system according to this embodiment is configured to include vehicle 1.

[0031] Vehicle 1 will be described with reference to Figure 1.

[0032] Figure 1 shows the interior of vehicle 1. As shown in the figure, the interior of vehicle 1 is equipped with a driver's seat 2 and a passenger seat 3. In this embodiment, vehicle 1 is a right-hand drive vehicle, and the driver's seat 2 is located on the right side of the vehicle.

[0033] A steering wheel 4 is positioned in front of the driver's seat 2. The steering wheel may be circular, or it may be U-shaped (a so-called irregularly shaped steering wheel) as shown in the illustration.

[0034] The steering wheel 4 is connected to the steering system of the vehicle 1 via the steering column. The steering column is covered by a column cover.

[0035] The passenger seat 3 is located to the left of the driver's seat 2, and a center display 24 is positioned midway between the driver's seat 2 and the passenger seat 3, towards the front of the vehicle. The center display 24 displays, for example, supplementary information such as car navigation, or entertainment content such as audio and television.

[0036] A meter display 21 is provided on the front side of the steering wheel 4. The meter display 21 is a full-screen display located directly in front of the driver's seat 2, providing various information to the driver of the vehicle 1. The meter display 21 displays information necessary for driving the vehicle, such as vehicle speed, engine speed (if the vehicle 1 has an engine), charge / discharge status (if the vehicle 1 has a drive battery), and mileage.

[0037] A pair of display devices 27 are provided in front of the driver's seat 2 inside the vehicle. The pair of display devices 27 are positioned on both sides of the steering wheel 4. This includes a touch panel display. In this embodiment, a display located closer to the inside of the vehicle is referred to as an inner display, and a display located closer to the outside of the vehicle is referred to as an outer display. The inner display 22 and the outer display 23 are connected to the instrument panel of the vehicle 1. Either the inner display 22 or the outer display 23 corresponds to the "pair of display units" in this disclosure.

[0038] The detailed structures of the inner display 22 and the outer display 23 will be described with reference to Figure 2.

[0039] The inner display 22 and outer display 23 are vertically oriented displays and are connected to a support portion 5, which is a member that supports each display. The support portion 5 has a shape that protrudes in the left-right direction from the steering wheel. The outer display 23 is connected to the right end of the support portion 5 when facing forward of the vehicle, and the inner display 22 is connected to the left end when facing forward of the vehicle. This allows the display surface to be positioned near the grip portion of the steering wheel 4.

[0040] The above explanation is an example where vehicle 1 is a right-hand drive vehicle; if vehicle 1 is a left-hand drive vehicle, the left and right sides will be reversed. That is, the outer display 23 will be connected to the left end of the support part 5 facing the front of the vehicle, and the inner display 22 will be connected to the right end facing the front of the vehicle.

[0041] The support portion 5 may also serve as a column cover that covers the steering column. The support portion 5 extends from the portion through which the steering column passes in both the left-right direction (vehicle width direction), and each of these extends toward the rear of the vehicle, and is shaped to bend along the way so that the angle of the steering wheel with respect to the left-right direction becomes smaller.

[0042] Furthermore, the inner display 22 and the outer display 23 are angled with respect to the left-right direction of the steering wheel, and are positioned so as to tilt toward the steering wheel. That is, the inner display 22 protrudes further to the rear of the vehicle the closer it is to the inside of the vehicle, and the outer display 23 protrudes further to the rear of the vehicle the closer it is to the outside of the vehicle. In this way, the inner display 22 and the outer display 23 are positioned so as to face the driver.

[0043] The area 4R shown by the dashed line in Figure 2 is the part (grip area) where the steering wheel 4 is gripped by the right hand, and the area 4L shown by the dashed line in Figure 2 is the part (grip area) where the steering wheel 4 is gripped by the left hand. By making the support part 5 the shape described above, the positions of the inner display 22 and the outer display 23 in the longitudinal direction of the vehicle can be brought closer to the grip area of ​​the steering wheel 4. Therefore, the amount of hand movement when the driver performs a touch operation on the inner display 22 or the outer display 23 while driving can be reduced.

[0044] [System Configuration] Next, we will describe the configuration of each device that makes up the vehicle system of vehicle 1.

[0045] First, let's describe the components of Vehicle 1. Figure 3 is a schematic diagram showing an example of the configuration of Vehicle 1. Vehicle 1 consists of a vehicle system, an electrical system, and an input / output system.

[0046] A vehicle system is a system that controls the motion of a vehicle. A vehicle system includes multiple components related to the vehicle's movement.

[0047] An input / output system is a system that provides information to the occupants of a vehicle and receives input from them. An input / output system includes multiple components that perform information input and output.

[0048] An electrical system is a system that controls the electrical components of a vehicle's body. It includes multiple components related to the electrical components of the vehicle's body.

[0049] The components included in each system may include multiple electrical components and an ECU that controls those electrical components.

[0050] The vehicle ECU 10 and the body ECU 30 can be configured as computers having a processor (CPU, GPU, etc.), main memory (RAM, ROM, etc.), and auxiliary memory (EPROM, hard disk drive, removable media, etc.). The auxiliary memory stores an operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, various functions (software modules) that match a predetermined purpose, as described later, can be realized. However, some or all of the functions may be realized as hardware modules by hardware circuits such as ASICs and FPGAs.

[0051] The vehicle ECU10 is connected to several components related to the vehicle's operation.

[0052] The GPS unit 11 is a unit for acquiring the location information of vehicle 1. The GPS unit 11 includes a GPS antenna and a positioning module for determining the location information. The GPS antenna is an antenna that receives positioning signals transmitted from positioning satellites (also called GNSS satellites). The positioning module is a module that calculates location information based on the signals received by the GPS antenna.

[0053] The ETC (Electronic Toll Collection System) unit 12 is a unit for collecting tolls on toll roads via wireless communication. The ETC unit 12 consists of an antenna and a communication module.

[0054] The wireless communication unit 13 includes an antenna and a communication module for wireless communication. The antenna is an antenna element that inputs and outputs wireless signals. In this embodiment, the antenna is suitable for mobile communication (e.g., 3G, 4G, 5G, etc.). The antenna may be configured to include multiple physical antennas. The communication module is a module for performing mobile communication.

[0055] Furthermore, the wireless communication unit 13 may include communication modules other than those for mobile communications. For example, the wireless communication unit 13 may include a Bluetooth® communication module. This makes it possible to establish a wireless connection with a user terminal located inside the vehicle, enabling the provision of services in cooperation with the user terminal (e.g., hands-free calling).

[0056] Sensor group 14 is a collection of multiple sensors for acquiring sensor data used during vehicle operation. Sensors may acquire physical quantities or image data, etc. In this embodiment, examples of sensors include a sensor for detecting vehicle speed (vehicle speed sensor 141) and an image sensor for acquiring images of the area in front of the vehicle (onboard camera 142).

[0057] The vehicle ECU10 also contains components belonging to the drivetrain system (for example, the engine). Further components such as a motor, battery, and transmission may be connected.

[0058] The body ECU30 is connected to multiple electrical components of the vehicle.

[0059] In this embodiment, examples of electrical components in a vehicle include a car air conditioner, wipers, headlights, side mirrors, and power seats.

[0060] The air conditioning unit 31 is the car air conditioning system for vehicle 1. The air conditioning unit 31 includes, for example, a cooling unit, a heating unit, a compressor, a condenser, etc. The air conditioning unit 31 may also include a control unit for controlling these components.

[0061] The wiper unit 32 includes a front wiper, a rear wiper, and a motor for driving them.

[0062] The headlight unit 33 includes the headlights of the vehicle 1 and a unit for driving the headlights. The headlights are configured to switch between high beams and low beams, for example. If the headlights of the vehicle 1 are adaptive headlights, the illumination of the light can also be controlled based on commands from the body ECU. The headlight unit 33 may further include taillights, brake lights, turn signals, and / or units for driving these.

[0063] The mirror actuator 34 includes side mirrors positioned on the left and right sides of the vehicle 1, and a drive unit for driving the mirrors. The mirror actuator 34 allows the angle of the side mirrors to be adjusted up, down, left, and right.

[0064] The seat actuator 35 includes a drive unit for adjusting the position of a seat provided in the vehicle 1. The seat actuator 35 allows adjustment of the position, height, angle, etc., of the seat surface and backrest.

[0065] Next, we will describe the devices used for inputting and outputting information.

[0066] In this embodiment, the vehicle 1 has four displays for outputting information: a meter display 21, an inner display 22, an outer display 23, and a center display 24.

[0067] The meter display 21 is a display positioned in front of the driver and outputs information about the vehicle 1. The meter display 21 mainly outputs information about the vehicle's status, such as vehicle speed, engine speed, fuel level, battery level, charge / discharge status, and mileage, but other information may also be output.

[0068] The inner display 22 and outer display 23 are touch-panel displays located on either side of the steering wheel. As mentioned above, the inner display 22 is located towards the center of the vehicle, and the outer display 23 is located on the outside of the vehicle.

[0069] If vehicle 1 is a right-hand drive vehicle, the inner display 22 is positioned on the left side in the direction of travel, and the outer display 23 is positioned on the right side in the direction of travel. If vehicle 1 is a left-hand drive vehicle, the inner display 22 is positioned on the right side in the direction of travel, and the outer display 23 is positioned on the left side in the direction of travel.

[0070] The center display 24 is a display located in the center of the vehicle. The center display 24 is located in the same place where the car navigation system's display was previously located. The center display 24 can display, for example, supplementary information such as car navigation, or entertainment content such as audio or television.

[0071] The touch panel 25 is a unit for detecting touch operations performed on the aforementioned displays. In this embodiment, the inner display 22, outer display 23, and center display 24 each have a touch panel 25, allowing for touch operation. The touch panel 25 detects touch operations performed on these displays and transmits the detected information to the vehicle ECU 10.

[0072] The audio unit 26 is a unit that performs input and output of audio information. The audio unit 26 may include, for example, a speaker and a microphone.

[0073] Next, we will explain the detailed configuration of the vehicle ECU10.

[0074] The vehicle ECU 10 is connected to the vehicle system and the drivetrain system, thereby enabling control of vehicle 1. Furthermore, the vehicle ECU 10 has the function of inputting and outputting information through components included in the input / output system, and the function of controlling the electrical system via the body ECU 30, which is responsible for the electrical system.

[0075] Figure 4 is a diagram illustrating the detailed configuration of the vehicle ECU 10.

[0076] The vehicle ECU 10 is comprised of a control device 101, a storage device 102, and a communication module 103.

[0077] The control device 101 is a calculation unit that realizes various functions of the vehicle ECU 10 by executing a predetermined program. The control device 101 can be implemented by a hardware processor such as a CPU. The control device 101 may also be configured to include RAM, ROM (Read Only Memory), cache memory, etc.

[0078] In this embodiment, the control device 101 of the vehicle ECU 10 is configured with six software modules: an HMI control unit 1011, a vehicle control unit 1012, an ADAS control unit 1013, an entertainment control unit 1014, a navigation unit 1015, and an assistant unit 1016. Each software module may be implemented by the control device 101 (CPU, etc.) executing a program stored in a storage device 102, which will be described later. The information processing performed by the software modules is synonymous with the information processing performed by the control device 101 (CPU, etc.).

[0079] The HMI control unit 1011 generates a user interface for controlling various components of the vehicle and provides it to the vehicle occupants via the input / output system. The HMI control unit 1011 also passes information input via the input / output system (for example, commands to implement user-specified controls) to the corresponding software module.

[0080] This makes it possible to control any component of the vehicle through a user interface.

[0081] In the following explanation, the components of Vehicle 1 generated by the user will be used. Commands used to control the vehicle are called "user commands." For example, commands to change the air conditioner temperature, change the drive mode, make a phone call, or play music are examples of "user commands."

[0082] Furthermore, the components of a vehicle may include electrical components such as headlights and wipers. Also, the components of a vehicle may include an on-board computer, or software modules executed by the on-board computer (e.g., autonomous driving software, navigation software, etc.).

[0083] The HMI control unit 1011 selects the output destination of the user interface from among several displays (meter display 21, inner display 22, outer display 23, center display 24) depending on the type of component being controlled. The specific configuration will be described later.

[0084] The vehicle control unit 1012 controls components of the vehicle system or components of the drive system.

[0085] The vehicle control unit 1012 may control a target component (e.g., a drive motor) based on a command (e.g., an acceleration command) received from another component (e.g., an ADAS control unit 1013, described later), or it may control a target component (e.g., a headlight unit) based on a user command (e.g., a headlight activation command) obtained via the HMI control unit 1011.

[0086] The ADAS control unit 1013 controls the Advanced Driver-Assistance Systems. For example, the ADAS control unit 1013 generates vehicle control commands based on sensor information acquired from the on-board sensors included in the sensor group 14, thereby realizing driver assistance such as cruise control, pre-collision safety, and lane keeping assist. For example, the ADAS control unit 1013 controls the vehicle system and the drive system by transmitting control commands to the vehicle control unit 1012.

[0087] The entertainment control unit 1014 controls the entertainment provided to the vehicle's occupants. Examples of entertainment functions include audio functions and television / radio functions.

[0088] The navigation unit 1015 provides a car navigation function. Based on location information obtained from the GPS unit 11 and traffic information obtained from the wireless communication unit 13, the navigation unit 1015 can generate and provide vehicle route guidance. The navigation unit 1015 may, for example, accept the vehicle's destination and waypoints via the HMI control unit 1011, and output information regarding the generated route.

[0089] The assistant unit 1016 provides information to the vehicle occupants by executing a language model capable of natural language interaction. The language model may be, for example, a Large Language Model (LLM). For example, by executing an LLM that has information about the vehicle occupants, information can be provided in natural language. If there are multiple vehicle occupants, the assistant unit 1016 may obtain a language model corresponding to each person via the network and execute any of the language models.

[0090] The storage device 102 is a means for storing information and is composed of storage media such as RAM, magnetic disks, and flash memory. The storage device 102 stores programs executed by the control device 101, data used by those programs, and so on.

[0091] The communication module 103 is a communication interface for connecting the vehicle ECU 10 to the vehicle network. The communication module 103 is, for example, a CAN (Controller The system is configured to communicate with components of vehicle 1 via a network such as an Area Network.

[0092] The specific configuration of the vehicle ECU 10 can be appropriately modified, with components omitted, replaced, and added depending on the embodiment. For example, the control device 101 may include multiple hardware processors. The hardware processors may consist of microprocessors, FPGAs, GPUs, etc. Input / output devices other than those exemplified (e.g., optical drives, etc.) may also be added. Furthermore, the vehicle ECU 10 may be composed of multiple computers. In this case, the hardware configurations of each computer may or may not be identical.

[0093] [Interface screen provided by the HMI control unit] Next, the user interface provided by the HMI control unit 1011 will be described.

[0094] The HMI control unit 1011 can generate multiple types of graphical user interface (GUI) screens and output them via the multiple displays mentioned above.

[0095] In this embodiment, the HMI control unit 1011 aggregates information related to vehicle control on the outer display 23 and aggregates and outputs information related to occupant convenience and comfort on the inner display 22. This allows occupants of the vehicle 1 to intuitively recognize whether the desired information is output to the left or right side. Furthermore, the touch panel operating surfaces of the inner display 22 and outer display 23 are positioned near the steering wheel. This minimizes the amount of eye and hand movement required by the vehicle driver.

[0096] The HMI control unit 1011 can switch between multiple types of screens that are output to the outer display 23.

[0097] Figure 5 illustrates the transitions between multiple types of screens output to the outer display 23. As shown, the HMI control unit 1011 can output multiple screens, centered around the main screen 501, to the outer display 23.

[0098] (1) Main screen 501 The main screen 501 is the screen that is normally displayed on the outer display while driving. Figure 6(A) shows an example of the main screen 501. The content displayed on the main screen can be customized by the user.

[0099] (2) Headlight control screen 502 This screen is used to set the illumination status of the headlights of vehicle 1. Figure 7(A) shows an example of the headlight control screen 502.

[0100] The headlight control screen 502 allows the user to select the headlight lighting status (automatic, on, off, small lights, daytime running lights, etc.). The vehicle control unit 1012 generates a user command based on the touch operation acquired via the HMI control unit 1011 and controls the headlight unit 33 by transmitting the user command to the body ECU 30.

[0101] Furthermore, the vehicle control unit 1012 may acquire data indicating the headlight illumination status from the body ECU 30 and reflect the illumination status on the screen via the HMI control unit 1011. For example, the vehicle control unit 1012 may reflect the current headlight illumination status in the vehicle graphic drawn on the screen.

[0102] (3) Wiper control screen 503 This screen is used to set the operating status of the wipers (front and rear) of vehicle 1. Figure 7(B) shows an example of the wiper control screen 503.

[0103] On the wiper control screen 503, the operating state of the wipers (stop, automatic operation, intermittent operation, continuous operation, etc.) can be selected. The vehicle control unit 1012 generates a user command based on the touch operation acquired via the HMI control unit 1011 and controls the wiper unit 32 by transmitting the user command to the body ECU 30.

[0104] Furthermore, the vehicle control unit 1012 may acquire data indicating the wiper's operating status from the body ECU 30 and reflect the operating status on the screen via the HMI control unit 1011. For example, the vehicle control unit 1012 may reflect the current wiper's operating status in the vehicle graphic drawn on the screen.

[0105] (4) Mirror control screen 504 This screen is used to adjust the deployment state and angle of the side mirrors of vehicle 1. Figure 7(C) shows an example of the mirror control screen 504.

[0106] On the mirror control screen 504, the status of the side mirror (folded / unfolded state and angle) can be selected. The vehicle control unit 1012 generates a user command based on the touch operation acquired via the HMI control unit 1011 and controls the mirror actuator 34 by transmitting the user command to the body ECU 30.

[0107] Furthermore, the vehicle control unit 1012 may acquire data indicating the status of the side mirrors from the body ECU 30 and reflect the operating status on the screen via the HMI control unit 1011. For example, the vehicle control unit 1012 may reflect the current extended / retracted state of the side mirrors in the vehicle graphic drawn on the screen.

[0108] (5) Driving position adjustment screen 505 This screen allows you to adjust the position and angle of the seats (seat cushion and backrest) and the steering wheel of vehicle 1.

[0109] The driving position adjustment screen 505 allows users to set the position and angle of the seat and steering wheel. The vehicle control unit 1012 receives touch operations via the HMI control unit 1011, generates user commands based on these operations, and transmits these user commands to the body ECU 30 to control the seat actuator 35 and other components.

[0110] (6) ADAS settings screen 506 This screen displays the operating status of the advanced driver assistance system in vehicle 1 and allows for setting adjustments.

[0111] For example, if the advanced driver assistance system of vehicle 1 includes an auto cruise function, the user of vehicle 1 can set the cruising speed, distance between vehicles, whether or not to follow the vehicle in front, whether or not to use lane tracing, etc., via the ADAS setting screen 506.

[0112] The information included in the ADAS settings screen 506 can be generated by the ADAS control unit 1013.

[0113] (7) Drive mode setting screen 507 This is the screen for setting the drive mode of vehicle 1. Figure 7(D) shows an example of the drive mode setting screen 507.

[0114] For example, the user of vehicle 1 can select a mode from several modes, such as normal mode, sport mode, eco mode, and snow mode, via the drive mode setting screen 507. The vehicle control unit 1012 changes the vehicle's drive mode based on this selection.

[0115] (8) Vehicle function setting screen 508 This screen is for using other functions of Vehicle 1. Examples of other functions of Vehicle 1 include opening and closing the electric tailgate and adjusting the suspension.

[0116] (9) Display mode setting screen 509 This screen is for setting the types of information displayed on the meter display 21. For example, if the meter display 21 can display multiple pieces of information other than speed, the user of the vehicle 1 can customize the information displayed on the meter display 21 via the display mode setting screen 509.

[0117] The multiple screens exemplified above can be switched by performing a predetermined operation, such as swiping left or right, on the touch panel provided on the outer display 23.

[0118] Furthermore, the HMI control unit 1011 can display a shortcut screen 510 on the outer display 23 for jumping to each screen. Figure 6(B) shows an example of the shortcut screen 510. The shortcut screen can be accessed, for example, by performing a predetermined operation (for example, swiping from the bottom of the screen upwards) on each screen, including the main screen 501. The shortcut screen 510 has multiple interfaces (for example, buttons) for transitioning to each screen, and by operating these interfaces, the user can move to the desired screen.

[0119] In this way, the HMI control unit 1011 aggregates and outputs information related to vehicle control to the outer display 23.

[0120] Information related to vehicle control typically refers to information related to the vehicle's driving control. Examples of information related to vehicle driving control include information for controlling components belonging to the vehicle system and information for controlling components belonging to the drive system. Furthermore, information for controlling components belonging to the electrical system, such as lights and wipers, which are necessary for the vehicle to drive properly, can also be considered information related to vehicle driving control.

[0121] Furthermore, information related to vehicle control, even if not directly related to vehicle driving control, can be output to the outer display 23. For example, information for controlling the vehicle's electric doors may be output to the outer display 23.

[0122] Similarly, the HMI control unit 1011 can switch the screen output to the inner display 22 to multiple types of screens.

[0123] Figure 8 illustrates the transitions between multiple types of screens output to the inner display 22. As shown, the HMI control unit 1011 can output multiple screens, centered around the main screen 801, to the inner display 22.

[0124] (1) Main screen 801 The main screen 801 is the screen that is normally displayed on the inner display while driving. Figure 9(A) shows an example of the main screen 801. The content displayed on the main screen can be customized by the user.

[0125] (2) Navigation screen 802 This is a screen related to car navigation. Figure 10(A) shows an example of the navigation screen 802. The information contained in the navigation screen 802 is generated by the navigation unit 1015.

[0126] Furthermore, when map information from the car navigation system is displayed on the center display 24, the navigation screen 802 may output supplementary information regarding the vehicle 1's route. Examples of supplementary route information include guidance on the next turn, estimated time of arrival, and remaining distance.

[0127] Furthermore, the navigation screen 802 may include some of the information and user interface displayed on the center display 24.

[0128] For example, frequently used user interfaces such as a button to start directions to home, a button to access bookmarks, and a button to start directions to a charging station can be placed on the navigation screen 802.

[0129] (3) Audio screen 803 This is a screen related to the audio functions of vehicle 1. Figure 10(B) shows an example of the audio screen 803. The information contained in the audio screen 803 is generated by the entertainment control unit 1014. The entertainment control unit 1014 can acquire and play music via a network or media. The audio screen 803 can include information about the music currently being played and an interface for selecting music.

[0130] (4) Air conditioner settings screen 804 This is a screen for controlling the car air conditioner of vehicle 1. Figure 10(C) shows an example of the air conditioner setting screen 804.

[0131] The air conditioning settings screen 804 allows users to set the temperature and airflow of the car's air conditioning system, as well as the air conditioning devices to be operated (steering wheel heater, seat heater, defroster, etc.). The vehicle control unit 1012 generates user commands based on the operations performed on the screen and transmits these user commands to the body ECU 30 to control the air conditioning unit 31 and other components.

[0132] Although the car air conditioner is a vehicle 1 piece of equipment controlled by external commands, it relates to the comfort of the occupants, so the air conditioner setting screen 804 is displayed on the inner display 22, not the outer display 23.

[0133] (5) Phone screen 805 This screen is for wirelessly connecting with the user terminal and making hands-free calls. If the wireless communication unit 13 has a module for wireless communication with the user terminal, it can make (or receive) phone calls from the user terminal via this module.

[0134] (6) AI Partner Screen 806 This is a screen for interacting with a virtual agent (AI partner) capable of natural language dialogue. Figure 10(D) shows an example of the AI ​​partner screen 806. This screen includes a virtual agent character, and the vehicle occupant can obtain information by interacting with this character. The natural language dialogue service is provided by the assistant unit 1016.

[0135] The AI ​​partner screen 806 may include an interface for giving instructions to the agent. If there are multiple occupants in the vehicle, the assistant unit 1016 may be configured to run any agent corresponding to each person. The AI ​​partner screen 806 may also include interfaces for starting and ending interactions with the agent, and interfaces for switching agents.

[0136] The multiple screens exemplified above can be switched by performing a predetermined operation, such as swiping left or right, on the touch panel provided on the inner display 22.

[0137] Furthermore, the HMI control unit 1011 can display a shortcut screen 810 on the inner display 22 for jumping to each screen. Figure 9(B) shows an example of the shortcut screen 810. The shortcut screen can be accessed, for example, by performing a predetermined operation (for example, swiping from the bottom of the screen upwards) on each screen, including the main screen 801. The shortcut screen 810 has multiple interfaces (for example, buttons) for transitioning to each screen, and by operating these interfaces, the user can move to the desired screen.

[0138] In this way, the HMI control unit 1011 aggregates and outputs information regarding the convenience and comfort of the occupants to the inner display 22.

[0139] Furthermore, information about components that are part of the vehicle, such as car air conditioners, but relate to the convenience or comfort of the occupants, may be output to the inner display 22 without being treated as information related to vehicle control.

[0140] Thus, if the information in question applies to both vehicle control information and information regarding occupant convenience (or comfort), the HMI control unit 1011 may selectively output such information to either the inner display 22 or the outer display 23 according to predetermined criteria.

[0141] [Move the control interface] The HMI control unit 1011 can output the aforementioned multiple screens, which can be output to the inner display 22 or outer display 23, to the center display 24 as well. Figure 11 shows an example of a screen output to the center display. For example, Figure 10(B) is an example where the audio screen 803 is output to the inner display 22, but since the center display 24 has a larger area than the inner display 22, when the same screen is output to the center display 24, the screen design and amount of information may change, as shown in Figure 11. The screen 40 shown in Figure 11 includes a title field 41 and the operation interface, which is the rewind function. This includes a button 42, a play button 43, a fast-forward button 44, and a volume control bar 45. Here, the operation interface can be any buttons or the like for operating the functions of the in-vehicle equipment. The operation interface is a specific example of the "predetermined user interface" in this disclosure. The input interface is a specific example of the content output to the inner display 22 or the outer display 23.

[0142] When a driver attempts to operate the above-mentioned operation interface displayed on the center display 24 from a seated position, they may find it difficult to reach. Therefore, in this embodiment, when the HMI control unit 1011 detects that the vehicle user has performed a predetermined gesture, it moves the operation interface included in the screen currently displayed on the center display 24 from the center display 24 to the inner display 22 or outer display 23. Figure 12 shows the operation interface described above moved to the outer display 23. As shown in Figure 12, when the HMI control unit 1011 detects the first gesture, it outputs the operation interface included in the screen 40 (rewind button 42, play button 43, fast forward button 44, and volume control bar 45) to the outer display 23. At this time, the HMI control unit 1011 may erase these operation interfaces from the screen of the center display 24.

[0143] As a result, an in-vehicle system according to one aspect of this disclosure can improve the operability when a vehicle user operates the in-vehicle equipment.

[0144] The process described above will now be explained using a flowchart. Figure 13 is a flowchart of the process executed by the HMI control unit 1011 according to the first embodiment. The process shown in Figure 13 is repeatedly executed while one of the aforementioned multiple screens is output to the center display 24. If the process in Figure 13 is to be executed again within a predetermined period after it has been executed once, the process in Figure 13 may start from step S12. The target screen can be, for example, any of the navigation screen 802, audio screen 803, air conditioning setting screen 804, telephone screen 805, or AI partner screen 806 as described above.

[0145] First, in step S11, the HMI control unit 1011 acquires first data, which is data for matching a gesture made by the vehicle user with a first gesture. Specifically, the HMI control unit 1011 communicates with an external device to acquire the first data. The first data may be, for example, feature data indicating the form of the first gesture, which is used for pattern recognition to match a gesture made by the user with the first gesture.

[0146] Here, the first gesture is a gesture that represents some predetermined movement. For example, the first gesture could be a gesture of moving the hand from the vicinity of the center display towards the vicinity of the outer display.

[0147] In step S12, the HMI control unit 1011 switches to a mode for detecting gestures made by the user. The HMI control unit 1011 switches to a mode in which it waits to detect gestures made by the user using sensors such as an in-vehicle camera.

[0148] In step S13, the HMI control unit 1011 determines whether or not it has detected any gesture made by the user. If the HMI control unit 1011 determines that it has detected any gesture made by the user, this step results in a positive determination.

[0149] If the result in this step is positive, the process proceeds to step S14.

[0150] If the result in this step is negative, the process proceeds to step S12.

[0151] When the process transitions to step S14, the HMI control unit 1011 compares the detected gesture with the first gesture included in the first data. Specifically, the HMI control unit 1011 uses pattern recognition to determine the form of the gesture performed by the user detected by the sensor and the form of the first gesture, and then compares the two to match the detected gesture with the first gesture.

[0152] In step S15, the HMI control unit 1011 determines whether the gesture performed by the user detected by the sensor is the first gesture. This step results in a positive determination if the HMI control unit 1011 determines that the gesture performed by the user detected by the sensor is the first gesture.

[0153] If the result in this step is positive, the process proceeds to step S16.

[0154] If the result in this step is negative, the process will proceed back to step S12.

[0155] If the process proceeds to step S16, the HMI control unit 1011 outputs the operation interface included in the screen currently being output to the center display 24 to the inner display 22 or the outer display 23. The output destination of the operation interface may be determined as appropriate depending on the type of screen. In this example, the output destination is the outer display 23. For example, if the HMI control unit 1011 is outputting the audio screen 803 to the center display 24, the HMI control unit 1011 may also output the rewind button 42, play button 43, fast forward button 44, and volume control bar 45 to the outer display 23.

[0156] In step S17, the HMI control unit 1011 erases the operation interface on the screen of the center display 24 that corresponds to the operation interface output to the outer display 23 in step S16. For example, if the HMI control unit 1011 outputs the rewind button 42, the play button 43, the fast forward button 44, and the volume control bar 45 to the outer display 23, the HMI control unit 1011 erases the corresponding components from the screen of the center display 24.

[0157] As described above, in this embodiment, when the vehicle ECU 10 detects a predetermined gesture made by the user, it moves the operation interface included in the screen being displayed from the display device located in the front center of the vehicle to the display device held by the user.

[0158] According to such a configuration, an in-vehicle system according to one aspect of this disclosure can improve the operability when a vehicle user operates the in-vehicle equipment.

[0159] (Second Embodiment) In the first embodiment, when a predetermined gesture is detected, the operation interface contained on the screen is moved from the center display to the user's handheld display device. However, it is conceivable that there may be cases where it is more convenient for the vehicle user to return the operation interface from the driver's handheld display device to the center display, such as when a person seated in the passenger seat is operating the vehicle on behalf of the driver. Therefore, in the second embodiment, the operation interface that has been moved from the center display to the driver's handheld display device is returned to the center display again.

[0160] Figure 14 is a flowchart of the process executed by the HMI control unit 1011 according to the second embodiment. The process shown in Figure 14 is executed following step S17 in Figure 13. If the process in Figure 14 is to be executed again within a predetermined period after it has been executed once, the process in Figure 14 may start from step S21.

[0161] In step S20, the HMI control unit 1011 acquires second data, which is data for comparing a gesture made by the vehicle user with a second gesture. Specifically, the HMI control unit 1011 communicates with an external device to acquire the second data. The second data may be, for example, feature data indicating the form of the second gesture, which is used for pattern recognition to compare a gesture made by the user with the second gesture. For example, the second gesture may be a gesture in which the hand moves in the opposite direction to the first gesture.

[0162] In step S21, the HMI control unit 1011 determines whether or not it has detected a second gesture made by the user. At this time, the HMI control unit 1011 compares the detected gesture with the second gesture included in the second data. Specifically, the HMI control unit 1011 uses pattern recognition to understand the form of the gesture made by the user detected by the sensor and the form of the second gesture, and compares the two. Then, the HMI control unit 1011 compares the detected gesture with the second gesture, and if the two match, it determines that it has detected a second gesture. In this step, if the HMI control unit 1011 determines that it has detected a second gesture made by the user, the determination is affirmative.

[0163] If the result in step S21 is positive, the process proceeds to step S22.

[0164] If the result in step S21 is negative, the process terminates.

[0165] In step S22, the HMI control unit 1011 erases the operation interface (for example, the rewind button 42, the play button 43, the fast forward button 44, and the volume control bar 45) that has been output to the inner display 22 or the outer display 23 from the screen.

[0166] In step S23, the HMI control unit 1011 outputs the operation interface that was erased in step S22 to the center display 24. For example, the HMI control unit 1011 performs the process of returning the rewind button 42, play button 43, fast forward button 44, and volume control bar 45 to the center display 24.

[0167] Figure 15 shows how the operating interface moves from the outer display 23. As shown in Figure 15, the screen 40 is output to the center display 24. The screen 40 is assumed to be an audio screen 803 similar to that of the first embodiment. As shown in Figure 15, the screen 40 includes a title field 41 and the operating interface, which includes a rewind button 42, a play button 43, a fast forward button 44, and a volume control bar 45.

[0168] As shown in Figure 15, when the HMI control unit 1011 detects the second gesture, it outputs the operation interface that was erased in step S17 to the center display 24 again. At this time, the HMI control unit 1011 may erase the play button 43 from the screen of the outer display 23.

[0169] As described above, in the second embodiment, the vehicle ECU 10 outputs the operation interface displayed on the driver's handheld display device to the driver's handheld display device. Switch back to the center display.

[0170] With such a configuration, the in-vehicle system of this disclosure can present the user interface in a more preferred position.

[0171] (Other variations) The embodiments described above are merely examples, and this disclosure may be modified as appropriate without departing from its essence.

[0172] For example, the processes and means described in this disclosure can be freely combined and implemented, as long as no technical inconsistencies arise.

[0173] Furthermore, although the embodiments illustrate configurations in which the meter display 21, inner display 22, and outer display 23 are independent of each other, each display does not necessarily have to be physically independent. The display device according to this disclosure may be realized by a single display device 1201, for example, as shown in Figure 16. In the example of Figure 16, the display device 1501 has three display areas (display sections) separated by dashed lines. The display device 1501 may have a structure in which the display surface is bent or curved at the portion indicated by the dashed line. In the illustrated example, display area 1502 functions as the inner display 22, display area 1503 functions as the outer display 23, and display area 1504 functions as the meter display 21.

[0174] Furthermore, a process described as being performed by a single device may be divided and executed by multiple devices. Conversely, a process described as being performed by different devices may be executed by a single device. In a computer system, the hardware configuration (server configuration) by which each function is implemented can be flexibly changed.

[0175] The present disclosure can also be realized by supplying a computer program implementing the functions described in the embodiments above to a computer, and having one or more processors in the computer read and execute the program. Such a computer program may be provided to the computer by a non-temporary computer-readable storage medium that can be connected to the computer's system bus, or it may be provided to the computer via a network. Non-temporary computer-readable storage mediums include, for example, any type of disk such as magnetic disks (floppy disks, hard disk drives (HDDs), etc.), optical disks (CD-ROMs, DVDs, Blu-ray discs, etc.), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards, flash memory, optical cards, and any type of medium suitable for storing electronic instructions. [Explanation of Symbols]

[0176] 1. Vehicle 10. Vehicle ECU 101... Control device 102...Storage device 103...Communication module 21... Meter display 22. Inner display 23. Outer display 24... Center display 25...Touch panel 26..Audio Unit 30...Body ECU

Claims

1. An in-vehicle system comprising a first display device and a second display device, A storage unit that stores first data, which is data for comparing a gesture made by the vehicle user detected by a sensor with a first gesture, Based on the first data and the output of the sensor, it is detected that the user has performed the first gesture. When the first gesture is detected, a predetermined user interface included in the content output to the first display device is output to the second display device. A control unit that performs the following: In-vehicle systems.

2. An in-vehicle system comprising a first display device and a second display device, Using sensors, the system detects a first gesture made by the vehicle user, When the first gesture is detected, a predetermined user interface included in the content output to the first display device is output to the second display device. A control unit that performs the following: In-vehicle systems.

3. The first display device is This is a display device located in the center of the vehicle's width direction at the front of the vehicle. The second display device is A display device that includes a pair of display units, each having a touch panel, positioned on either side of the steering wheel. The in-vehicle system according to claim 2.

4. The control unit, The user interface output to the second display device in response to the first gesture is erased from the screen of the first display device. The in-vehicle system according to claim 2.

5. The aforementioned predetermined user interface is This includes GUI components for operating the functions provided by the vehicle, The in-vehicle system according to claim 2.

6. The control unit, When the sensor detects a second gesture of the vehicle user, the user interface output to the second display device is erased from the screen of the second display device, and the user interface is output again to the first display device. The in-vehicle system according to claim 2.

7. A vehicle equipped with the in-vehicle system according to any one of claims 2 to 6.

8. An information processing device included in an in-vehicle system comprising a first display device and a second display device, Using sensors, the system detects a first gesture made by the vehicle user, When the first gesture is detected, a predetermined user interface included in the content output to the first display device is output to the second display device. A control unit that performs the following: Information processing device.

9. The first display device is This is a display device located in the center of the vehicle's width direction at the front of the vehicle. The second display device is A display device that includes a pair of display units, each having a touch panel, positioned on either side of the steering wheel. The information processing apparatus according to claim 8.

10. The control unit, The user interface output to the second display device in response to the first gesture is erased from the screen of the first display device. The information processing apparatus according to claim 8.

11. The aforementioned predetermined user interface is This includes GUI components for operating the functions provided by the vehicle, The information processing apparatus according to claim 8.

12. The control unit, When the sensor detects a second gesture of the vehicle user, the user interface output to the second display device is erased from the screen of the second display device, and the user interface is output again to the first display device. The information processing apparatus according to claim 8.

13. An information processing method performed by an information processing device in an in-vehicle system comprising a first display device and a second display device, A step of using a sensor to detect a first gesture made by the vehicle user, When the first gesture is detected, the first step is to output a predetermined user interface included in the content output to the first display device to the second display device. including, Information processing methods.

14. The first display device is This is a display device located in the center of the vehicle's width direction at the front of the vehicle. The second display device is A display device that includes a pair of display units, each having a touch panel, positioned on either side of the steering wheel. The information processing method according to claim 13.

15. The further step includes erasing the user interface output to the second display device in response to the first gesture from the screen of the first display device. The information processing method according to claim 13.

16. The aforementioned predetermined user interface is This includes GUI components for operating the functions provided by the vehicle, The information processing method according to claim 13.

17. The further step includes, when the sensor detects a second gesture of the vehicle user, erasing the user interface output to the second display device from the screen of the second display device and outputting the user interface again to the first display device. The information processing method according to claim 13.

18. A program for causing a computer to execute the information processing method described in any one of claims 13 to 17.

Images