In-vehicle display control system and in-vehicle display control method

Abstract

To accurately select an object displayed on an onboard display.SOLUTION: Provided is an onboard display control system for controlling an onboard display which is installed in a vehicle interior. The onboard display control system comprises: one or a plurality of sensors for acquiring the state of an occupant in the vehicle interior; a line of sight direction identification unit for identifying a line of sight direction of the occupant on the basis of the output of the sensors; a finger identification unit for identifying a finger of the occupant on the basis of the output of the sensors; and a selection target specification unit for specifying, on the basis of a visible range of the occupant which is assumed from an identification result of the line of sight direction, a designated position on the display which is set from an identification result of the finger, and a positional relation with an object which is displayed on the display, an object as the target to be selected that exists within the visible range and is located at a position that corresponds to the designated position.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an in-vehicle display control system and an in-vehicle display control method. [Background technology]

[0002] Conventionally, pointing devices such as a mouse or touch panels have been used to select objects displayed on a display. However, technologies that allow for contactless selection of desired objects have also been considered.

[0003] For example, Patent Document 1 states that "We provide a method for teaching the spatial position of an object by pointing, which can be done naturally and easily." and "In a teaching method for determining the position and orientation of an object placed in the workspace of a computer on a surface basis, a finger on a human hand is extracted, indices are set at at least two points on the finger, the spatial coordinates of two points of the finger are calculated, the calculation of the spatial coordinates is performed by a stereo method, corresponding points in the stereo vision are searched for based on the indices set on the finger, a three-dimensional straight line is fitted to the spatial coordinates of the two points, and the intersection of the straight line and the object plane whose three-dimensional coordinates are known is taught as the position of the object." Furthermore, Patent Document 2 states that "the object is to provide a technology for accurately identifying a device that a driver wishes to operate from among in-vehicle devices and for assisting in the operation of that device. The operation assistance device of the present invention comprises: a gaze direction acquisition unit that acquires the gaze direction of the driver of the vehicle; a characteristic behavior acquisition unit that acquires characteristic behavior other than gaze by the driver; a device identification unit that identifies at least one in-vehicle device from among multiple in-vehicle devices mounted on the vehicle as an operation target device that the driver wishes to operate based on the gaze direction and the characteristic behavior; and a display control unit that displays an operation screen of the operation target device on a display device mounted on the vehicle." Patent document 3 also states, "We present a user interface device, computer program, computer-readable medium, and method for selecting a selectable object on a display screen. The display screen displays one or more selectable objects. Information regarding the tracked position of the user's hand and head or eye position is obtained. Based on the tracked position of the hand and head or eye position, it is determined whether the selectable object is at a first screen position. Here, the first screen position is a position on the display screen where the first hand at least partially obstructs the user's view of the selectable object. If it is determined to be so located, it is determined that the selectable object has been selected." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-069734 [Patent Document 2] Patent No. 6851482 [Patent Document 3] Japanese Patent Application Laid-Open No. 2017-539035 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional technology, it has been difficult to achieve accurate selection operations without malfunctions when an occupant selects an object displayed on an in-vehicle display. For example, since vehicle occupants do not always check or operate the display, selecting an object at an unnecessary time can result in malfunctions. Also, due to limited freedom of seating position and obstructions such as the steering wheel, it is sometimes impossible to create a situation where the object is in line with the occupant's line of sight and hand.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an in-vehicle display control system and an in-vehicle display control method that enable accurate selection of an object displayed on an in-vehicle display. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, one representative in-vehicle display control system of the present invention is an in-vehicle display control system that controls an in-vehicle display installed in a vehicle cabin, and is characterized by comprising: one or more sensors that acquire the status of occupants in the vehicle cabin; a gaze direction identification unit that identifies the gaze direction of the occupant based on the output of the sensor; a finger identification unit that identifies the occupant's fingers based on the output of the sensor; and a selection target identification unit that identifies, as a selection target, an object located within the visibility range of the occupant at a position corresponding to the indicated position, based on the positional relationship between the occupant's visibility range estimated from the gaze direction identification result, a pointed position on the display set from the finger identification result, and an object displayed on the display. [Effects of the Invention]

[0008] According to the present invention, it is possible to accurately select an object displayed on an in-vehicle display. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of an in-vehicle display control system according to a first embodiment. [Figure 2] Configuration diagram of an in-vehicle display control system according to the first embodiment [Figure 3] Flowchart showing the processing procedure of the in-vehicle display control system [Figure 4] Illustration of setting the indicated position [Figure 5] Illustration of specific selection [Figure 6] Example of object selection (part 1) [Figure 7] Example of object selection (part 2) [Figure 8] Example of object selection (part 3) [Figure 9] Examples of object manipulation [Figure 10] First Modification [Figure 11] Second Variant DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment will be described with reference to the drawings. [Example]

[0011] FIG. 1 is an explanatory diagram illustrating an overview of an in-vehicle display control system according to a first embodiment. A display 30, which is a display device, is attached to the dashboard of a vehicle 10. As an example, the display area of ​​the display 30 is from the driver's seat side to the passenger seat side of the dashboard. The in-vehicle display control system installed in the vehicle 10 acquires the state of an occupant U1, who is the driver, using a sensor, identifies the position of the head, the direction of the line of sight, and the fingers of the occupant U1, and controls the display on the display 30.

[0012] Specifically, the in-vehicle display control system processes an image captured by an exterior camera based on the position of the occupant U1's head, and displays on the display 30 an image of the exterior of the vehicle that is estimated to be visible from the occupant U1's viewpoint, thereby providing a field of view V2 that is virtually transmitted through the display 30. As a result, the field of view V1 that the occupant U1 can directly see and the virtual field of view V2 become continuous, thereby expanding the field of view of the occupant U1.

[0013] 1 shows a case where an obstacle D1 is present ahead of the vehicle 10. The obstacle D1 is hidden by the hood of the vehicle 10 and is not included in the directly visible field of view V1. However, by using the display on the display 30 as a virtual field of view V2, the occupant U1 can visually recognize the obstacle D1.

[0014] The image of the obstacle D1 displayed on the display 30 can be selected as an object and used as a target for operation. However, due to its position relative to the seating position and the presence of handles and other obstacles, it can be difficult to operate the display 30 by directly touching it.

[0015] Therefore, the disclosed in-vehicle display control system selects an object by non-contact remote control based on the expected visual range of occupant U1 from the direction of occupant U1's line of sight, the indicated position on the display set from the finger identification results, and the positional relationship between the object displayed on the display.

[0016] Here, the term "hand" includes at least one of a hand and a finger. The pointing position on the display set based on the identification result of the hand and finger is, for example, an object that exists in the direction where the occupant is pointing. Furthermore, when selecting an object, specifically, an object that is located within the visible range and at a position corresponding to the pointed position may be specified as the object to be selected.

[0017] 2 is a configuration diagram of the in-vehicle display control system of Example 1. As shown in FIG. 2, the in-vehicle display control system includes an outside camera 11, an inside camera 12, a processing unit 20, and a display 30.

[0018] The exterior camera 11 is an imaging device that captures images of the surroundings of the vehicle. The in-vehicle camera 12 is an imaging device that captures images of the interior of the vehicle, and functions as a sensor that acquires the status of the occupants in the vehicle.

[0019] The processing unit 20 has an occupant head identification unit 21, an occupant finger identification unit 22, a virtual transparent display processing unit 23, a selection target identification unit 24, and a target operation execution unit 25. When the processing unit 20 is realized by a computer, the functions corresponding to the occupant head identification unit 21, the occupant finger identification unit 22, the virtual transparent display processing unit 23, the selection target identification unit 24, and the target operation execution unit 25 can be realized by a CPU (Central Processing Unit) executing a program.

[0020] The occupant head identifying unit 21 identifies the head position and line of sight direction of the occupant U1 using the image captured by the in-vehicle camera 12. That is, the occupant head identifying unit 21 includes a function as a line of sight direction identifying unit.

[0021] The occupant finger identification unit 22 is a finger identification unit that identifies the fingers of the occupant U1 using the images captured by the in-vehicle camera 12. In identifying the fingers, the fingertips and joints of each finger can be identified.

[0022] The virtual transparent display processing unit 23 processes the image capture results of the outside vehicle camera 11 based on the head position output by the occupant head identification unit 21, and displays on the display 30 an image of the outside of the vehicle that is estimated to be visible from the viewpoint of the occupant U1, thereby providing the occupant U1 with a view that is virtually transparent to the display 30.

[0023] The selection target identification unit 24 estimates the occupant's visible range from the line-of-sight direction identification result by the occupant head identification unit 21, and sets the pointed position on the display from the hand and finger identification result by the occupant hand and finger identification unit 22. The selection target identification unit 24 also acquires the display content on the display 30 from the virtual transparent display processing unit 23. Then, the selection target identification unit 24 identifies, as the selection target, an object that is located within the visible range and at a position corresponding to the pointed position.

[0024] When setting the pointing position, the selection target specifying unit 24 sets the intersection of a straight line connecting two predetermined points included in the fingers and the display surface of the display 30 as the pointing position. When identifying a selection target, the selection target identification unit 24 sets an object located outside the visible range to a state where it is not ready to be selected, sets an object located inside the visible range to a state where it is ready to be selected, and when the pointed position overlaps an object that is ready to be selected, sets the object to a selected state.

[0025] The target operation execution unit 25 is an operation execution unit that identifies a gesture from the identification result of the occupant's fingers by the occupant's finger identification unit 22, and executes an operation associated with the identified gesture on the selected target.

[0026] Gestures can be any movement of the fingers, such as bringing two specific fingertips closer together, moving two specific fingertips apart, extending and moving a specific number of fingertips, opening the hand, closing the hand, or the like.

[0027] As an operation on a selection object, any operation can be used, such as enlarging or reducing the frame surrounding the selection object, enlarging or reducing the image of the selection object, or displaying the image recognition results related to the selection object.

[0028] 3 is a flowchart showing the processing procedure of the in-vehicle display control system. The processing unit 20 of the in-vehicle display control system repeatedly executes the processing procedure shown in FIG.

[0029] The processing unit 20 first acquires images from the exterior camera 11 and the interior camera 12 (step S101). After that, the processing unit 20 identifies the head position and gaze direction of the occupant using the occupant head identification unit 21 and identifies the occupant's fingers using the occupant finger identification unit 22 for the image acquired from the interior camera 12 (step S202).

[0030] After step S202, the virtual transparent display processing unit 23 generates a display screen for virtual transparent display from the occupant's head position and the image from the outside camera 11 (step S103), and performs virtual transparent display processing by displaying it on the display 30 (step S104).

[0031] After step S104, the selection target specifying unit 24 identifies a selectable object from the virtual see-through display screen, and extracts a rectangular area surrounding the object as a target area (step S105).

[0032] After step S105, the selection target specification unit 24 sets the visible range based on the identification result of the occupant's line of sight direction (step S106). Also, the selection target specification unit 24 sets the pointing position based on the identification result of the occupant's finger (step S107).

[0033] After step S107, the selection target specifying unit 24 specifies an object that is located within the visible range and at a position corresponding to the indicated position as the selection target (step S108). The target operation executing unit 25 identifies a gesture from the identification result of the occupant's fingers by the occupant's finger identifying unit 22, and executes an operation associated with the identified gesture on the selection target (step S109).

[0034] 4 is an explanatory diagram of setting the pointing position. The selection target specifying unit 24 sets the intersection of a line connecting two predetermined points included in the fingers and the display surface of the display 30 as the pointing position P. In FIG. 4, the tip and third joint of the index finger are identified by image processing and used as the two points. Any point, such as any joint in the finger, the palm, or the center of mass of the hand, can also be identified and used to set the pointing position.

[0035] 5 is an explanatory diagram of how a selection target is specified. The selection target specification unit 24 extracts a rectangle that surrounds an object as a target area. If the target area is located outside the visible range, the target area is set to an unprepared state for selection.

[0036] When a target area in a selection-unprepared state is inside the visible range, the selection target specification unit 24 sets the target area in a selection-prepared state. When a target area in a selection-prepared state is outside the visible range, the selection target specification unit 24 sets the target area in a selection-unprepared state.

[0037] When the pointed position enters a target area in a selection preparation state, the selection target identification unit 24 places the target area in a selected state. If a gesture to deselect is detected when there is a target area in a selected state, the selection target identification unit 24 places the target area in a selection preparation state. Examples of gestures to deselect include clenching the hand (curling all fingers) or opening the hand (stretching all fingers).

[0038] If a gesture associated with an operation is detected when there is a target area in a selected state, the target operation execution unit 25 executes the corresponding operation, and then returns to the selected state after the operation is completed.

[0039] 6 to 8 show specific examples of object selection. In FIG. 6, an image of the outside world corresponding to the viewpoint of an occupant U1 is displayed on a display 30 provided on the dashboard. For example, an image of an obstacle D1 present on the other side of the hood is displayed on the display 30 as an object T1. This allows the occupant U1 to recognize the obstacle D1 as if it were visible through the hood.

[0040] 7 shows the visible range A1 of the occupant U1 with a dashed-line rectangle. If the object T1 is outside the visible range A1, the object T1 is not ready for selection. If the object T1 is inside the visible range A1, the object T1 is ready for selection.

[0041] Figure 8 shows the relationship between objects in a selection-ready state and the pointed-to position. If the pointed-to position is outside the rectangular area (target area) that contains the objects in a selection-ready state, object T1 is in a selection-ready state. If the pointed-to position is inside the rectangular area (target area) that contains the objects in a selection-ready state, object T1 is in a selected state.

[0042] FIG. 9 shows a specific example of an object operation. FIG. 9 shows a case where an action of moving the tips of the index finger and thumb apart (pinch out action) is identified as a gesture. If a vertical pinch out action is detected, the target operation execution unit 25 increases the size of the rectangular area including the object in the vertical direction. If a horizontal pinch out action is detected, the target operation execution unit 25 increases the size of the rectangular area including the object in the horizontal direction. If a diagonal pinch out action is detected, the target operation execution unit 25 increases the size of the rectangular area including the object while maintaining the aspect ratio.

[0043] FIG. 9 shows a case where the size of a rectangular area including an object is changed, but any operation can be set, such as enlarging the image of the object itself by pinching out.

[0044] In the above explanation, an example has been given in which the present invention is applied to a configuration in which a virtual field of view is provided by a display 30 placed on a dashboard. Also, an example has been given in which a part of the display area of ​​the display 30 is set as the visible range. However, the present invention is not limited to this.

[0045] FIG. 10 is an explanatory diagram of a first modified example. In the configuration of FIG. 10, display 40 provides information related to the vehicle's travel route, such as map information. The display area of ​​display 40 is smaller than that of display 30 and is smaller than the visible range. In this modified example, the selection target identification unit prepares an object displayed on display 40 for selection, provided that the entire display area of ​​display 40 is included in visible range A2. When designated position P2 overlaps the object, the object is selected and an operation using a gesture is accepted. As an example, a map can be enlarged by pinching out.

[0046] FIG. 11 is an explanatory diagram of a second modified example. In the configuration of FIG. 11, the display 50 performs display output by projecting onto the windshield. In this modified example, an object that the occupant is directly looking at can be recognized as an object and used as a target for operation. In FIG. 11, a sign is included in the visible range A3 as object T2, and a designated position P3 is superimposed on object T2. Therefore, the selection target identification unit accepts operations with object T2 in a selected state. As an example, it is possible to have signs read aloud.

[0047] As described above, the disclosed system is an in-vehicle display control system that controls an in-vehicle display installed in a vehicle cabin, and includes an in-vehicle camera 12 as one or more sensors that acquire the status of occupants in the vehicle cabin, an occupant head identification unit 21 as a gaze direction identification unit that identifies the gaze direction of the occupant based on the output of the sensor, an occupant hand and finger identification unit 22 as a finger identification unit that identifies the fingers of the occupant based on the output of the sensor, and a selection target identification unit 24 that identifies an object located within the visibility range and at a position corresponding to the indicated position as a selection target, based on the positional relationship between the occupant's visibility range estimated from the gaze direction identification result, a pointed position on the display set from the finger identification result, and an object displayed on the display. This configuration and operation allows accurate selection of an object displayed on the in-vehicle display, and this selection can be performed remotely and without contact.

[0048] The disclosed system further includes an object operation execution unit 25 as an operation execution unit that identifies a gesture from the identification result of the fingers and executes an operation associated with the identified gesture on the selected object. This allows the occupant to perform an operation on the selected object.

[0049] The sensor is an in-vehicle camera that captures an image of the interior of the vehicle, and the gaze direction identification unit and the hand and finger identification unit identify the gaze direction and the hand and finger from the image of the occupant. Therefore, the state of the occupants can be acquired with a simple configuration, and the in-vehicle camera can be shared with other systems installed in the vehicle.

[0050] The selection target specifying unit also sets, as the pointed position, an intersection between a straight line connecting two predetermined points included in the fingers and a display surface of the display. Therefore, the designated position can be easily set.

[0051] In addition, the selection target identification unit sets the object located outside the visible range to a state where it is not ready to be selected, sets the object located inside the visible range to a state where it is ready to be selected, and when the indication position overlaps an object in the selection-ready state, sets the object to a selected state. In this way, by making a selection step by step with reference to a plurality of elements, it is possible to prevent malfunctions.

[0052] In addition, the disclosed system further comprises an exterior camera 11 that captures images of the area around the vehicle, and a virtual transparent display processing unit 23 that processes the image captured by the exterior camera based on the position of the occupant's head identified from the output of the sensor, and displays on the display an image of the outside of the vehicle that is estimated to be visible from the occupant's viewpoint, thereby providing a field of view that is virtually transparent to the display, and the visible range is a part of the display area of ​​the display. According to this configuration, a part of the vehicle body can be virtually displayed through the display, and the displayed object can be remotely selected.

[0053] Furthermore, according to the disclosed system, the display provides information related to the vehicle's driving route, and the selection target identification unit enables selection of an object displayed on the display, provided that the entire display area of ​​the display is included in the visible range. According to this configuration, a display that is smaller than the visible range, such as a navigation system screen, can be selected.

[0054] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, not only can the configurations be deleted, but also replacements and additions of configurations are possible.

[0055] For example, when there are other occupants in addition to the driver, the present invention may be applied to the other occupants.Furthermore, a specific occupant may be designated as the target of application of the present invention among the multiple occupants.

[0056] The state of the occupants may also be acquired from any sensor other than the in-vehicle camera, such as a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) or a stationary smartphone, etc. Furthermore, the outputs of multiple sensors or multiple types of sensors may be combined and used. [Explanation of symbols]

[0057] 11: exterior camera, 12: interior camera, 20: processing unit, 21: occupant head recognition unit, 22: occupant finger recognition unit, 23: virtual transparent display processing unit, 24: selection target identification unit, 25: target operation execution unit, 30: display

Claims

1. An in-vehicle display control system that controls an in-vehicle display installed in a vehicle interior, one or more sensors for acquiring the state of an occupant in the vehicle interior; a gaze direction identification unit that identifies the gaze direction of the occupant based on an output of the sensor; a finger identification unit that identifies the fingers of the occupant based on an output of the sensor; a selection target specifying unit that specifies, based on a positional relationship between the occupant's visible range estimated from the line-of-sight direction identification result, a designated position on the display set from the finger identification result, and an object displayed on the display, an object that is located within the visible range and at a position corresponding to the designated position, as a selection target; an operation execution unit that identifies a gesture from the identification result of the hand and fingers and executes an operation associated with the identified gesture on the selection object; Equipped with the selection target specifying unit sets the object located outside the visible range to a state not yet prepared for selection, sets the object located inside the visible range to a state prepared for selection, and when the pointing position is superimposed on the object in the selection-prepared state, sets the object to a selected state; an exterior camera that captures images of the vehicle's surroundings; a virtual transparency display processing unit that processes an image captured by the exterior camera based on the position of the occupant's head identified from the output of the sensor, and displays on the display an image of the exterior of the vehicle that is estimated to be visible from the occupant's viewpoint, thereby providing a field of view that is virtually transparent to the display; and Furthermore, the selection target specifying unit identifies a selectable object from the virtual transparent display screen provided by the virtual transparent display processing unit, extracts an area surrounding the identified object as a target area, sets the target area located inside the visible range to the selection preparation state, and when the pointing position enters the target area in the selection preparation state, sets the target area to the selected state; The in-vehicle display control system is characterized in that, when a target area in the selected state is present and a gesture associated with an image operation is detected, the operation execution unit executes the image operation on the target area in the selected state.

2. An in-vehicle display control system that controls an in-vehicle display installed in a vehicle interior, one or more sensors for acquiring the state of an occupant in the vehicle interior; a gaze direction identification unit that identifies the gaze direction of the occupant based on an output of the sensor; a finger identification unit that identifies the fingers of the occupant based on an output of the sensor; a selection target specifying unit that specifies, based on a positional relationship between the occupant's visible range estimated from the line-of-sight direction identification result, a designated position on the display set from the finger identification result, and an object displayed on the display, an object that is located within the visible range and at a position corresponding to the designated position, as a selection target; an operation execution unit that identifies a gesture from the identification result of the hand and fingers and executes an operation associated with the identified gesture on the selection object; Equipped with the selection target specifying unit sets the object located outside the visible range to a state not yet prepared for selection, sets the object located inside the visible range to a state prepared for selection, and when the pointing position is superimposed on the object in the selection-prepared state, sets the object to a selected state; the display provides information regarding the vehicle's route; the selection target specification unit sets an object displayed on the display to the selection preparation state on condition that the entire display area of ​​the display is included in the visible range, and sets the object to the selection preparation state when the pointing position is superimposed on the object; The in-vehicle display control system is characterized in that, when there is an object in the selected state, the operation execution unit identifies a gesture from the finger identification result and executes an operation associated with the identified gesture.

3. the sensor is an in-vehicle camera that captures an image of the interior of the vehicle; 3. The in-vehicle display control system according to claim 1, wherein the gaze direction identifying unit and the hand / finger identifying unit identify the gaze direction and the hand / finger from the image of the occupant.

4. 3. The in-vehicle display control system according to claim 1, wherein the selection target specifying unit sets, as the pointing position, an intersection of a straight line connecting two predetermined points included in the fingers and a display surface of the display.

5. An in-vehicle display control method for controlling an in-vehicle display installed in a vehicle interior, comprising: acquiring a status of an occupant in the vehicle cabin from one or more sensors; a gaze direction identifying step of identifying a gaze direction of the occupant based on an output of the sensor; a finger identification step of identifying the occupant's finger based on an output of the sensor; a selection target specifying step of specifying, as a selection target, an object located within the visible range and at a position corresponding to the pointed position on the display, based on a positional relationship between the visible range of the occupant estimated from the identification result of the line of sight direction, the pointed position on the display set from the identification result of the finger, and an object displayed on the display; an operation execution step of identifying a gesture from the identification result of the hand and fingers and executing an operation associated with the identified gesture on the selection target; Including, the selection target specifying step sets the object located outside the visible range to a state not yet prepared for selection, sets the object located inside the visible range to a state prepared for selection, and when the pointing position is superimposed on the object in the selection-prepared state, sets the object to a selected state; The method further includes a virtual transparency display processing step of processing an image of the surroundings of the vehicle captured by an exterior camera based on the position of the occupant's head identified from the output of the sensor, and displaying on the display an image of the outside of the vehicle that is estimated to be visible from the occupant's viewpoint, thereby providing a field of view that is virtually transparent to the display; The selection target specifying step identifies a selectable object from the virtual transparent display screen provided by the virtual transparent display processing step, extracts an area surrounding the identified object as a target area, sets the target area located inside the visible range to the selection preparation state, and when the pointing position enters the target area in the selection preparation state, sets the target area to the selected state; The operation execution step executes the image operation on the selected target area when a gesture associated with the image operation is detected while the selected target area is present.

10. A method for controlling an in-vehicle display.

6. An in-vehicle display control method for controlling an in-vehicle display installed in a vehicle interior, comprising: acquiring a status of an occupant in the vehicle cabin from one or more sensors; a gaze direction identifying step of identifying a gaze direction of the occupant based on an output of the sensor; a finger identification step of identifying the occupant's finger based on an output of the sensor; a selection target specifying step of specifying, as a selection target, an object located within the visible range and at a position corresponding to the pointed position on the display, based on a positional relationship between the visible range of the occupant estimated from the identification result of the line of sight direction, the pointed position on the display set from the identification result of the finger, and an object displayed on the display; an operation execution step of identifying a gesture from the identification result of the hand and fingers and executing an operation associated with the identified gesture on the selection target; Including, the selection target specifying step sets the object located outside the visible range to a state not yet prepared for selection, sets the object located inside the visible range to a state prepared for selection, and when the pointing position is superimposed on the object in the selection-prepared state, sets the object to a selected state; the display provides information regarding the vehicle's route; the selection target specifying step includes: setting an object displayed on the display to the selection preparation state on condition that the entire display area of ​​the display is included in the visible range; and setting the object to a selected state when the pointing position is superimposed on the object in the selection preparation state; The operation execution step includes identifying a gesture from the identification result of the hand and fingers when the selected object is present, and executing an operation associated with the identified gesture.

10. A method for controlling an in-vehicle display.

Images