Non-contact operation device for a vehicle and vehicle

Abstract

The present invention relates to a non-contact operation device for a vehicle and a vehicle, aiming to improve the operation device of the vehicle. The non-contact operation device (40) of the vehicle includes: a projection device (43) that projects an image into the passenger compartment (3) of the vehicle (1); a generation unit (51) that generates and updates the image projected by the projection device; an operation detection device (44) that detects the operation part of an occupant who performs a non-contact operation on the image projected into the space of the passenger compartment; an operation determination unit (52) that determines the non-contact operation of the occupant on the image projected into the passenger compartment space based on the position or movement of the occupant's operation part relative to the image; and a motion detection device (48) that detects the motion of the vehicle or the motion of the occupant caused by the motion of the vehicle. The operation determination unit makes adjustments based on the detection of the motion detection device to suppress the motion of the occupant caused by the motion of the vehicle, and determines the operation performed by the operation part of the occupant on the image projected into the empty space in the passenger compartment.

Description

Technical Field

[0001] The present invention relates to a non-contact operation device for a vehicle. Background Art

[0002] Various operation parts are provided in a vehicle. For example, a vehicle has a start switch, a steering wheel, a shift lever, pedals, etc. for controlling the running of the vehicle. In addition, operation parts for an air conditioner device, a navigation device, an audio device, an image receiving device, a hands-free call device, and other equipment and machines are provided in the vehicle. Thus, a plurality of operation parts are provided in the vehicle. The plurality of operation parts are parts arranged with respect to the inner surface of the vehicle compartment, but in recent years, the layout surface of the vehicle compartment has shown a shortage. In the future, for example, when considering adding a game device, a network communication device, an entertainment device, etc., it may not be possible to add operation parts for these devices any more.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2017-027401 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] Therefore, regarding a vehicle, as disclosed in Patent Document 1, it is considered to develop a new operation device for operating an image projected onto the vehicle compartment.

[0008] In Patent Document 1, an image is presented in front of an occupant sitting on a seat. In addition, according to an operation corresponding to the presented image, a tactile stimulus caused by an ultrasonic wave field is output to an operation part of the occupant. Therefore, regarding the operation of the image, the occupant can obtain a sense of reality of the operation.

[0009] However, a vehicle moves while running. A running vehicle accelerates and decelerates for running, and left and right turns, etc. cause loads to act. When the vehicle accelerates or decelerates or a load acts on the vehicle, the body of the occupant moves like swaying. In this case, for example, the hand or finger of the occupant operating the image sways and moves due to the movement of the occupant caused by the movement of the vehicle.

[0010] And, in such a running state, as in Patent Document 1, if an image is projected fixedly, an operation on the image is determined fixedly, and a tactile stimulus for the operation is output fixedly, it is impossible to correctly determine the operation that the occupant wants to perform on the image, or it is impossible to correctly output a tactile stimulus to the operation part that operates the image.

[0011] Thus, the vehicle needs to improve the operation device.

[0012] Means for Solving the Problems

[0013] The non-contact operation device of the vehicle of the present invention includes: a projection device that projects an image into the passenger compartment of the vehicle; a generation unit that generates and updates the image projected by the projection device; an operation detection device that detects the operation part of an occupant who performs a non-contact operation on the image projected in the space of the passenger compartment; an operation determination unit that determines the non-contact operation of the occupant on the image projected into the space of the passenger compartment according to the position or movement of the occupant's operation part relative to the image; and a motion detection device that detects the motion of the vehicle or the motion of the occupant caused by the motion of the vehicle. Among them, the operation determination unit makes adjustments according to the detection of the motion detection device to suppress the motion of the occupant caused by the motion of the vehicle, and determines the operation of the occupant's operation part on the image projected into the empty space in the passenger compartment.

[0014] Preferably, it may include: a stimulation output device that is arranged for the occupant sitting in the passenger compartment and can output a tactile stimulation caused by an ultrasonic wave field to the occupant who operates on the image; and a stimulation response output unit that outputs a tactile stimulation caused by an ultrasonic wave field to the operation part of the occupant who operates on the projected image as a response to the operation determined by the operation determination unit. Among them, the stimulation response output unit outputs the response of the tactile stimulation to the occupant operation determined by the operation determination unit through the detection of the motion detection device to the position of the operation part of the occupant detected by the operation detection device instead of through the detection of the motion detection device.

[0015] Preferably, the operation determination unit can use an inputtable area set for the image projected by the projection device to determine the operation of the occupant, and offset and adjust the position of the inputtable area according to the detection of the motion detection device in a manner that suppresses the motion of the occupant caused by the motion of the vehicle, so as to determine the operation of the occupant.

[0016] Preferably, the operation determination unit can use an inputtable area set for the image projected by the projection device to determine the operation of the occupant, and adjust it in a manner that expands the inputtable area according to the magnitude of the motion of the occupant caused by the motion of the vehicle according to the detection of the motion detection device, so as to determine the operation of the occupant.

[0017] Preferably, it may include: a stimulation output device configured for an occupant sitting in the passenger compartment, capable of outputting a tactile stimulation caused by an ultrasonic field to the occupant operating on the image; and a stimulation response output unit that outputs a tactile stimulation caused by an ultrasonic field to the operating part of the occupant operating on the projected image as a response to the operation determined by the operation determination unit. When the operation determination unit adjusts and determines the occupant's operation through the detection of the motion detection device, the stimulation response output unit outputs a response to the tactile stimulation of the occupant's operation to a wider range than the case where the determination is made without adjustment through the detection of the motion detection device.

[0018] Preferably, the operation determination unit may determine the occupant's operation based on the motion of the occupant with respect to the image projected by the projection device, and adjust the motion of the occupant detected by the operation detection device in a manner that suppresses the motion caused by the vehicle's motion according to the detection of the motion detection device, thereby determining the occupant's operation.

[0019] Preferably, the motion detection device may detect the driving state of the vehicle.

[0020] Preferably, the motion detection device may detect the motion of the occupant caused by the vehicle's motion based on an image of the occupant.

[0021] The vehicle of the present invention includes: the non-contact operation device according to any one of the above; and a plurality of controlled devices connected to the non-contact operation device through a network to control the vehicle. Each of the plurality of controlled devices obtains, through the network, input information generated by the non-contact operation device according to the non-contact operation of the occupant on the image projected into the passenger compartment.

[0022] Advantages of the Invention

[0023] The present invention detects the motion of the vehicle or the motion of the occupant caused by the vehicle's motion. And it will be adjusted according to the detection of the motion detection device to suppress the motion of the occupant caused by the vehicle's motion, and determine the operation of the occupant's operating part on the empty image projected into the passenger compartment.

[0024] Thus, in the present invention, it is possible to appropriately determine the non-contact operation of the occupant on the image projected into the empty space, which is independent of the vehicle's motion, without being affected by the motion of the vehicle during driving. In the present invention, even when the occupant's body is in a swaying state due to the vehicle's movement, it is possible to correctly determine the occupant's operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG. 1(A)-(B) is a schematic explanatory view of an automobile according to an embodiment of the present invention;

[0026] Figure 2 It is a schematic explanatory diagram of the control system of the vehicle in FIGS. 1(A)-(B);

[0027] Figure 3 It is a configuration diagram of the non-contact operation device provided in the vehicle in FIGS. 1(A)-(B);

[0028] Figure 4 It is an explanation of Figure 3 a diagram showing an example of the arrangement in the passenger compartment of the main components of the non-contact operation device of

[0029] Figure 5 It is a diagram showing Figure 3 the flowchart of the non-contact projection operation process performed by the non-contact operation device of

[0030] Figure 6 It is a flowchart showing the process of the operation determination process and the response output process as the determination result in the first embodiment;

[0031] Figure 7 It is related to Figure 6 a diagram explaining an example of the operation determination process corresponding to

[0032] Figure 8 It is a flowchart showing the process of the operation determination process and the response output process as the determination result in the second embodiment;

[0033] Figure 9 It is related to Figure 8 a diagram explaining an example of the operation determination process corresponding to

[0034] Figure 10 It is a flowchart showing the process of the operation determination process and the response output process as the determination result in the third embodiment;

[0035] Figure 11 It is related to Figure 10 a diagram explaining an example of the operation determination process corresponding to

[0036] Symbol Explanation

[0037] 1…Automobile (vehicle); 2…Vehicle body; 3…Passenger compartment; 4…Seat; 10…Control system; 20…Operation ECU; 40…Non-contact operation device; 43…3D image projection device (projection device); 44…Operation detection device; 45…Stimulation output device; 46…Sound output device; 48…Travel state detection device (motion detection device); 51…Image generation unit (generation unit); 52…Operation determination unit; 53…Stimulation response output unit; 54…Sound response output unit; 61…Display screen; 62…Semi-reflective mirror; 63…Stereo camera; 64…Quadrilateral frame; 65…Ultrasonic element; 70…Cube; 71…Button (part of the image); 72, 73, 74…Inputtable area; 75…Sphere 7 (image) Detailed implementation mode

[0038] Based on the accompanying drawings, the implementation modes of the present invention will be described below.

[0039] [First implementation mode]

[0040] Figs. 1(A)-(B) are schematic explanatory views of the automobile 1 according to the implementation mode of the present invention.

[0041] Fig. 1(A) is a top view, and Fig. 1(B) is a side view.

[0042] The automobile 1 in Figs. 1(A)-(B) is an example of a vehicle. The automobile 1 has a vehicle body 2 provided with a passenger compartment 3 in which a passenger sits. A seat 4 for the passenger to sit on is provided in the passenger compartment 3. In the front part of the passenger compartment 3, a steering wheel 5, a shift lever 6, a brake pedal 7, and an accelerator pedal 8 are provided for the passenger to operate the travel of the automobile 1. As shown in Fig. 1(B), the passenger sits on the seat 4 in a driving posture, and thus can reach out to these operation parts such as the steering wheel 5 to perform operations.

[0043] Figure 2 is a schematic explanatory view of the control system 10 of the automobile 1 in Figs. 1(A)-(B). Figure 2 Among them, a plurality of control devices constituting the control system 10 are represented and indicated by respectively incorporated control ECUs (Electronic Control Units).

[0044] Specifically,[[]] Figure 2Shown are a drive ECU 11, a steering ECU 12, a brake ECU 13, an autonomous driving / driving support ECU 14, a driving operation ECU 15, a detection ECU 16, an air-conditioning ECU 17, an occupant monitoring ECU 18, an external communication ECU 19, an operation ECU 20 of a non-contact operation device 40, and a system ECU 21. These multiple control ECUs are connected to a central gateway (CGW) 27 as a relay device via a vehicle network 26 such as CAN (Controller Area Network) or LIN (Local Interconnect Network) adopted in the vehicle 1.

[0045] Moreover, in each control module, the control ECU is connected to electronic devices used in the vehicle 1. The activated control ECU performs various processes and controls the operations of the respectively connected electronic devices according to the information (data) obtained from the vehicle network 26. In addition, the control ECU also outputs information (data) such as the operation states of the respectively connected electronic devices to the vehicle network 26.

[0046] Connected to the driving operation ECU 15 are operation detection sensors such as a steering wheel 5, a brake pedal 7, an accelerator pedal 8, and a shift lever 6 that an occupant operates to control the running of the vehicle 1. The driving operation ECU 15 outputs control information corresponding to the operation amount to the vehicle network 26. The drive ECU 11, the steering ECU 12, and the brake ECU 13 obtain information from the vehicle network 26 and control the running of the vehicle 1.

[0047] The operation ECU 20 obtains information from the driving operation ECU 15 via the vehicle network 26 and displays it on a display device (not shown). Connected to the operation ECU 20 are a display device and an operation device (not shown) operated by the occupant. The operation ECU 20, for example, constitutes an operation device together with the display device and the operation device. The operation device can be assembled in the non-contact operation device 40 described later. When an operation is performed on the operation device for the information obtained from the driving operation ECU 15 via the vehicle network 26, the operation ECU 20 outputs the input information of the operation device to the driving operation ECU 15 via the vehicle network 26. The driving operation ECU 15 obtains the input information via the vehicle network 26 and executes control based on the input information.

[0048] Connected to the detection ECU 16 are a speed sensor 31 of the vehicle 1, an acceleration sensor 32 capable of detecting the acceleration of the vehicle 1 caused by a collision or the like, an external camera 33, etc. The detection ECU 16 outputs the values of the speed sensor 31 and the acceleration sensor 32 of the vehicle 1, the image of the external camera 33, etc. to the vehicle network 26. The detection ECU 16 can also predict a collision based on the image of the external camera 33 and output the prediction result to the vehicle network 26. The central gateway 27 forwards the information.

[0049] The autonomous driving / driving support ECU 14 controls, for example, the vehicle 1 to travel to the destination of the vehicle 1. The autonomous driving / driving support ECU 14 obtains information from the detection ECU 16 via the vehicle network 26 and outputs control information for safe driving to the vehicle network 26. The drive ECU 11, the steering ECU 12, and the brake ECU 13 obtain information from the vehicle network 26 and control the travel of the vehicle 1.

[0050] The operation ECU 20 obtains information from the detection ECU 16 and the autonomous driving / driving support ECU 14 via the vehicle network 26 and displays it on the display device. When an operation device is operated for the information obtained from the detection ECU 16 and the autonomous driving / driving support ECU 14 via the vehicle network 26, the operation ECU 20 outputs the input information of the operation device to the vehicle network 26. The detection ECU 16 and the autonomous driving / driving support ECU 14 obtain the input information via the vehicle network 26 and perform control based on the input information.

[0051] An in-vehicle camera 34 is connected to the occupant monitoring ECU 18. The occupant monitoring ECU 18 identifies the occupants in the vehicle 1 based on the images from the in-vehicle camera 34 and monitors the riding states of the occupants. The occupant monitoring ECU 18 outputs this information to the vehicle network 26. The operation ECU 20 obtains the information of the occupant monitoring ECU 18 via the vehicle network 26 and displays it on the display device. When an operation device is operated for the information obtained via the vehicle network 26, the operation ECU 20 outputs the input information of the operation device to the vehicle network 26. The occupant monitoring ECU 18 obtains the input information via the vehicle network 26 and performs control based on the input information.

[0052] The air conditioner ECU 17 controls the environment such as the temperature and humidity in the passenger compartment 3. The air conditioner ECU 17 outputs information such as the detected temperature, solar radiation amount, and humidity to the vehicle network 26. The operation ECU 20 obtains the information of the air conditioner ECU 17 via the vehicle network 26 and displays it on the display device. When an operation device is operated for the information obtained via the vehicle network 26, the operation ECU 20 outputs the input information of the operation device to the vehicle network 26. The air conditioner ECU 17 obtains the input information via the vehicle network 26 and performs control based on the input information.

[0053] The external communication ECU 19 wirelessly communicates with, for example, a communication base station 81 existing outside the vehicle 1 and a communication device of another vehicle 82. The communication base station 81 and the communication device of another vehicle 82, together with the server device 83, constitute a traffic system 80. The external communication ECU 19 receives information of the traffic system 80 from the communication base station 81 or the communication device of another vehicle 82 and outputs it to the vehicle network 26. The operation ECU 20 acquires the information of the external communication ECU 19 through the vehicle network 26 and displays it on the display device. When an operation is performed on the operation device for the information acquired through the vehicle network 26, the operation ECU 20 outputs the input information of the operation device to the vehicle network 26. The external communication ECU 19 acquires the input information through the vehicle network 26 and wirelessly transmits the input information to the communication base station 81 and the communication device of another vehicle 82. The transmitted information can be used, for example, in the server device 83 or another vehicle 82 in the traffic system 80. Thus, the operation ECU 20 of the vehicle 1 can transmit and receive information (data) between the server device 83 or another vehicle 82 of the traffic system 80 through the external communication ECU 19.

[0054] In addition, power is supplied from the battery 91 provided in the vehicle 1 to each part, enabling the Figure 2 control system 10 shown to operate. The power supply lines leading from the battery 91 to each part are laid in the vehicle 1 together with, for example, the communication cables of the vehicle network 26. In addition to the battery 91, power can also be supplied to the control system 10 from a generator or a power receiver.

[0055] In this way, the operation ECU 20 is connected to the control ECU of other control devices, which are controlled devices, through the vehicle network 26 of the vehicle 1, and can transmit and receive information (data) with the control ECU of other control devices. In addition, in Figure 2 , the control ECU of each of the multiple controlled devices other than the operation ECU 20 acquires input information from the operation ECU 20 of the non-contact operation device 40 through the vehicle network 26 and controls the vehicle 1.

[0056] In addition, various operation parts are provided in the vehicle 1. For example, the vehicle 1 has a start switch, a steering wheel 5, a shift lever 6, pedals, etc. for controlling the driving of the vehicle 1. In addition, operation parts for an air conditioning device, a navigation device, an audio device, an image receiving device, a hands-free call device, and other equipment and machines are provided in the vehicle 1. Thus, a variety of operation parts are provided in the vehicle 1. These various operation parts are parts arranged relative to the inner surface of the passenger compartment 3, but in recent years, the layout surface of the passenger compartment 3 has shown a shortage. In the future, for example, when considering adding game devices, network communication devices, entertainment devices, etc., it may not be possible to add operation parts for these devices any more.

[0057] Therefore, regarding the vehicle 1, it is required to develop a new type of operating device to operate the image projected into the passenger compartment 3 of the vehicle 1. As such an operating device, for example, a non-contact operating device 40 can be considered. The non-contact operating device 40 projects an image in front of the occupant sitting on the seat 4. The non-contact operating device 40 generates input information according to the operation corresponding to the projected image and can output it to the vehicle network 26. In addition, in the non-contact operating device 40, a tactile stimulus caused by an ultrasonic field can also be output to the operating part of the occupant according to the operation corresponding to the projected image. By obtaining the response generated by this tactile stimulus, although the image projected into the space of the passenger compartment 3 is operated in a non-contact manner, the occupant can also obtain a sense of reality in operation as in the case of operating an actual object.

[0058] However, the vehicle 1 is in motion. The moving vehicle 1 accelerates and decelerates for driving, and loads such as turning left and right act on it. When the vehicle 1 accelerates or decelerates or a load acts on the vehicle 1, the body of the occupant will move like swaying. In this case, for example, the hand or finger of the occupant operating the image will sway and move due to the movement of the occupant caused by the movement of the vehicle 1.

[0059] Moreover, in such a driving state, as in Patent Document 1, if the image is projected fixedly, the operation on the image is judged fixedly, and the tactile stimulus for the operation is output fixedly, it is impossible to correctly judge the operation that the occupant wants to perform on the image, or it is impossible to correctly output the tactile stimulus to the operating part that operates the image.

[0060] Thus, the vehicle 1 needs to improve the operating device.

[0061] Figure 3 It is a configuration diagram of the non-contact operating device 40 provided in the vehicle 1 of FIGS. 1(A)-(B).

[0062] Figure 3 The non-contact operating device 40 has an in-vehicle communication unit 41, a timer 42, a 3D image projection device 43, an operation detection device 44, a stimulus output device 45, a sound output device 46, a driving state detection device 48, a memory 47, and an operation ECU 20 connected to them.

[0063] The memory 47 is, for example, a non-volatile memory 47. The memory 47 records the non-contact operation program and data. For the program, it can also be a program executed by AI. The program can include a learning program for AI processing. The data includes three-dimensional model data of the image projected during non-contact operation, etc. The three-dimensional model data is image data for non-contact operation and has, for example, a plurality of polygon data (polygon data) constituting the surface of the model.

[0064] The operation ECU 20 can be, for example, a microcomputer such as a central processing unit, an ASIC, or a DSP. The operation ECU 20 reads the program for non-contact operation from the memory 47 and executes it. Thus, in the operation ECU 20, the control unit of the non-contact operation device 40 is implemented. The control unit of the non-contact operation device 40 controls the operation of the entire non-contact operation device 40 to implement various functions for non-contact operation in the non-contact operation device 40. For example, in the operation ECU 20, as various functions for non-contact operation, an image generation unit 51, an operation determination unit 52, a stimulation response output unit 53, and a sound response output unit 54 are implemented.

[0065] The in-vehicle communication unit 41 is connected to the vehicle network 26. The in-vehicle communication unit 41 transmits and receives information (data) via the vehicle network 26, for example, between other control ECUs such as the air conditioner ECU 17, the autonomous driving / driving support ECU 14, and the external communication ECU 19 as illustrated in the figure. For example, the in-vehicle communication unit 41 obtains an instruction to display the air conditioner operation image from the air conditioner ECU 17 and outputs it to the operation ECU 20. The in-vehicle communication unit 41 obtains an instruction to display the setting operation image for autonomous driving / driving support from the autonomous driving / driving support ECU 14 and outputs it to the operation ECU 20. The in-vehicle communication unit 41 obtains content data such as 3D model data from the external communication ECU 19 and outputs it to the operation ECU 20.

[0066] The timer 42 measures the elapsed time or the time of day.

[0067] The image generation unit 51 generates and updates the image projected by the 3D image projection device 43. The image generation unit 51 obtains 3D model data from the memory 47 or the in-vehicle communication unit 41 according to the display instruction in order to generate the data of the projected image. The image generation unit 51 generates a 3D model based on the obtained 3D model data. The image generation unit 51 determines the projection position and direction of the 3D model in the passenger compartment 3 seen by the occupant, and generates the image data for projection according to the 3D model (stereoscopic model). The image generation unit 51 outputs the image data for projection to the 3D image projection device 43. The image generation unit 51 can also generate a 2D model (plane model) based on the 2D model data and generate the image data for projection. In addition, even without a display instruction, the image generation unit 51 can obtain content data such as a moving image or a still image from the memory 47 or the in-vehicle communication unit 41, generate the image data of the 3D image (stereoscopic image) or 2D image (plane image) of the content, and output it to the 3D image projection device 43.

[0068] The 3D image projection device 43 projects an image that can be operated by the occupant into the passenger compartment 3 of the vehicle 1. The 3D image projection device 43 can be, for example, a display device or a projector. The 3D image projection device 43 can project the image into the hollow space of the passenger compartment 3, for example, by means of holography or mirroring. The 3D image projection device 43 can also be arranged, for example, in front of the seat 4 on which the occupant is seated, such as on the instrument panel, the rearview mirror, or the roof of the passenger compartment 3. A 3D (three-dimensional) image or a 2D (two-dimensional) image is projected into the space of the passenger compartment 3 of the vehicle 1. Thereby, a stereoscopic image is projected at the projection position in the passenger compartment 3 so that the occupant can visually confirm the three-dimensional model.

[0069] The operation detection device 44 detects a specified operation part of the occupant. The operation detection device 44 can detect the movement of the body, upper body, shoulders, head, etc. of the occupant that vibrates due to the movement of the vehicle 1. The operation detection device 44 can detect the operation part of the occupant who performs a non-contact operation on the image projected into the space of the passenger compartment 3. Non-contact operations include, for example, hand operations on the image and movement operations for moving the image. The operation detection device 44 can also be arranged, for example, in front of the seat 4 on which the occupant is seated, such as on the instrument panel, the rearview mirror, or the roof of the passenger compartment 3. When the internal camera 34 of the occupant monitoring device is arranged on the instrument panel or the rearview mirror, the operation detection device 44 can be used in combination with the internal camera 34. The operation detection device 44 can be, for example, a stereoscopic camera 63 in which two imaging elements are arranged side by side. In this case, the operation detection device 44 detects the operation of the occupant on the image projected into the hollow space of the passenger compartment 3 based on the images of the two imaging elements.

[0070] The operation determination unit 52 obtains detection information such as an image of the stereo camera 63 from the operation detection device 44, and determines a non-contact operation of the occupant on the image projected into the space of the passenger compartment 3 based on the detection information. The operation determination unit 52 determines the position and movement of the operation part as the operation of the occupant on the image projected into the space of the passenger compartment 3. The movement includes movement information of the operation part such as the direction, speed, and acceleration of movement. The operation determination unit 52 obtains, for example, the pixel positions including the fingertip features of the occupant from the image through AI processing, and generates the position information of the fingertip by the correlation triangulation method of the images of the stereo camera 63. The operation determination unit 52 generates the movement of the fingertip, such as information on the movement direction, movement speed, and acceleration of movement, based on the shooting results at different times. The operation determination unit 52 determines the position and movement of the occupant's operation part with reference to the projection position of the image, and determines the operation of the occupant on the image projected into the space of the passenger compartment 3. The operation determination unit 52 determines whether the occupant's operation part contacts the projected image, the remaining distance to contact, the depth of contact, etc. The operation determination unit 52 outputs the determined operation information to each part of the operation ECU 20. The operation determination unit 52 outputs the operation information to, for example, the image generation unit 51, the stimulus response output unit 53, and the sound response output unit 54. The image generation unit 51 updates the image data for projection according to the operation of the occupant and outputs it to the 3D image projection device 43. Thus, the image projected into the passenger compartment 3 by the 3D image projection device 43 is updated in response to the operation.

[0071] In addition, when the operation determination unit 52 determines that the occupant has operated on the image projected into the space of the passenger compartment 3 according to the operation information, it outputs the input information generated by the operation of the occupant to each part of the vehicle 1 through the in-vehicle communication unit 41. For example, in a state where an operation button for changing the setting of the air conditioner temperature is projected, when the operation button is operated, the operation determination unit 52 generates input information corresponding to the operation button and outputs it to the in-vehicle communication unit 41. The in-vehicle communication unit 41 outputs the input information to the air conditioner ECU 17 as the controlled device through the vehicle network 26. The air conditioner ECU 17 performs air conditioner control to change the target temperature of the air conditioner according to the input information, so that the temperature of the passenger compartment 3 becomes the target temperature.

[0072] The driving state detection device 48 detects the driving state of the vehicle 1. The driving state detection device 48 detects, for example, the movement of the vehicle 1 caused by driving as the driving state of the vehicle 1. The information indicating the movement of the vehicle 1 caused by driving includes: the load acting on the vehicle 1, the speed of the vehicle 1, the acceleration of the vehicle 1, the yaw rate, etc.

[0073] The stimulation output device 45 is arranged for the operating part of the occupant operating on the operation image. The stimulation output device 45 can be, for example, any device that can output a tactile stimulation through an electrical signal. As a device for applying a tactile sensation non - contact, for example, it includes a device that generates an ultrasonic wave field and outputs a tactile stimulation caused by the ultrasonic wave field to the skin of the operating part by applying the ultrasonic wave field or a change in the field to the operating part of the occupant. The stimulation output device 45 can be, for example, an element array in which a plurality of ultrasonic elements 65 are arranged in a planar shape. The stimulation output device 45 can be arranged facing the operating part of the occupant sitting in the passenger compartment 3 and at a position away from, for example, the lower side of the hand. By selectively outputting ultrasonic waves from the plurality of ultrasonic elements 65, a tactile sensation as if the skin is in contact with an object can be generated at a local part of the hand, such as the fingertips, of the occupant operating on the image.

[0074] The stimulation response output unit 53 outputs a tactile stimulation caused by an ultrasonic wave field from the stimulation output device 45 to the operating part of the occupant operating on the projected image. The stimulation response output unit 53 outputs the tactile stimulation as a response to the occupant's operation. The stimulation response output unit 53 outputs an electrical signal to the stimulation output device 45 and selectively outputs ultrasonic waves corresponding to the operation from the plurality of ultrasonic elements 65. Thereby, the stimulation response output unit 53 can generate an ultrasonic wave field in the passenger compartment 3. The stimulation response output unit 53 locally applies an ultrasonic wave field or a change in the field to the operating part of the occupant determined by the operation determination unit 52 to be in contact with the image. When a person, for example, extends their hand into the ultrasonic wave field, the ultrasonic wave field can be felt through the skin on the hand surface. Thus, the stimulation response output unit 53 can output a tactile stimulation caused by an ultrasonic wave field as a response to the operation for the operating part of the occupant who performs a non - contact operation on the image projected into the space of the passenger compartment 3. The stimulation response output unit 53 locally applies an ultrasonic wave field or a change in the field to, for example, the surface of the occupant's operating part that is in virtual contact with the image.

[0075] In addition, the stimulation response output unit 53 can also have, for example, an element array in which a plurality of pressure output elements are arranged. In this case, the stimulation response output unit 53 can apply pressure to a person's skin by individually controlling the operation of the plurality of pressure output elements. Thereby, the occupant can obtain a tactile sensation corresponding to the operation.

[0076] The sound output device 46 can be, for example, a speaker driven by a sound signal. The sound output device 46 can be arranged in the passenger compartment 3.

[0077] The sound response output unit 54 outputs a sound signal to the sound output device 46 and outputs a sound corresponding to the operation from the sound output device 46. The sound response output unit 54 selects and acquires the sound data stored in the memory 47 and outputs a sound signal generated based on the sound data to the speaker serving as the sound output device 46. Thereby, the occupant can hear a sound corresponding to the operation.

[0078] Figure 4 This is an explanatory Figure 3 diagram showing an example of the arrangement of the main components of the non-contact operation device 40 in the vehicle cabin 3.

[0079] Figure 4 Schematically shows Figure 3 a specific combination example of the 3D image projection device 43, the operation detection device 44, and the stimulation output device 45.

[0080] Specifically, as the 3D image projection device 43, a display screen 61 and a half mirror 62 are illustrated. A stereo camera 63 is illustrated as the operation detection device 44. An element array is illustrated as the stimulation output device 45. These components of the non-contact operation device 40 are provided in the vehicle cabin 3 of the automobile 1.

[0081] The stimulation output device 45 has a hollow quadrilateral frame 64. Element arrays each having a plurality of ultrasonic elements 65 are provided on the four surfaces of the quadrilateral frame 64. By operating the element arrays on the upper, lower, left, and right surfaces of the quadrilateral frame 64, as shown in the figure, an ultrasonic field can be locally applied to the fingertips entering the quadrilateral frame 64. Thereby, a person can obtain the tactile sensation of the fingertips coming into contact.

[0082] The half mirror 62 is provided in front of the hollow quadrilateral frame 64 of the stimulation output device 45. The half mirror 62 is arranged facing the occupant. Below the half mirror 62, a display screen 61 for displaying a three-dimensional image (stereoscopic image) or a two-dimensional image (planar image) is arranged. Therefore, a three-dimensional image (stereoscopic image) or a two-dimensional image (planar image) can be visually confirmed and projected inside the hollow quadrilateral frame 64 of the stimulation output device 45. The occupant can visually confirm the image projected in the hollow of the vehicle cabin 3 inside the hollow quadrilateral frame 64 of the stimulation output device 45. The occupant can visually confirm that the sphere shown as a three-dimensional image in the figure seems to float inside the hollow quadrilateral frame 64.

[0083] The stereo camera 63 can be provided, for example, on the side of the half mirror 62 opposite to the stimulation output device 45. The two imaging elements of the stereo camera 63 can image the fingers etc. of the occupant located inside the quadrilateral frame 64. Thereby, the stereo camera 63 can image the fingers etc. entering the hollow quadrilateral frame 64 of the stimulation output device 45. Hereinafter, as needed, with Figure 4 as a reference, the direction along the axis of the quadrilateral frame 64 is called the Z direction, and the directions perpendicular to the Z direction are called the Y direction and the X direction. Here, the Z direction is the direction along the front-rear direction of the automobile 1.

[0084] In addition, Figure 3The components of the non-contact operation device 40 do not have to be densely arranged as Figure 4 shown.

[0085] For example, when the hollow quadrilateral frame 64 of the stimulation output device 45 is arranged in front of the occupant, it may interfere with the operation when the occupant operates the steering wheel 5 or the like. As shown in FIGS. 1(A)-(B), the hollow quadrilateral frame 64 can be arranged in a quadrilateral frame shape along the outer peripheral surface of the vehicle body 2. In this case, structures such as the quadrilateral frame 64 are not provided in front of the seat 4 where the occupant is seated. Even in this case, for example, as shown by the spherical image indicated by the dotted circle in the figure, the 3D image projection device 43 can project an image into the inner space of the hollow quadrilateral frame 64 provided in the passenger compartment 3. The occupant can perform a non-contact operation on the image by operating in the operation area inside the quadrilateral frame 64. In addition, the hollow quadrilateral frame 64 does not necessarily have to be in the shape of a square frame. The element array can also be arranged only along the outer periphery of the vehicle body 2. The element array can also be arranged integrally on the inner surface of the passenger compartment 3.

[0086] The semi-reflective mirror 62 can basically place the hollow quadrilateral frame 64 of the stimulation output device 45 in the middle and be arranged opposite to the occupant. In addition, if it is only for projecting an image into the inner hollow of the quadrilateral frame 64, a total reflection mirror can also be used instead of the semi-reflective mirror 62. In addition, the hollow quadrilateral frame 64 of the stimulation output device 45 can be placed in the middle, and the display screen 61 itself can be arranged at a position opposite to the occupant. In this case, the semi-reflective mirror 62 or the total reflection mirror is not required. Such a semi-reflective mirror 62 or total reflection mirror and the display screen 61 can be arranged, for example, on the instrument panel, the interior mirror, or the roof. In addition, it can also be integrally provided with the occupant monitoring device.

[0087] Detection devices such as the stereo camera 63 can be arranged, for example, on the instrument panel, the interior mirror, or the rearview mirror. In addition, the detection device can also be used in combination with the imaging element of the occupant monitoring device such as the DMS. Detection devices such as the stereo camera 63 only need to be able to photograph the operation parts of the occupant's fingertips and the like. In addition, the detection device can also use laser detection of the occupant's fingertips and the like by, for example, laser scanning the passenger compartment 3 instead of using images.

[0088] Figure 5 represents Figure 3 the flowchart of the non-contact projection operation process performed by the non-contact operation device 40.

[0089] In step ST1, the operation ECU 20 determines whether to start the non-contact operation of projecting an image. For example, when there is a display instruction or when there is content to be displayed, the operation ECU 20 determines to start the non-contact operation and advances the process to step ST2. In other cases, the operation ECU 20 determines not to start the non-contact operation and ends Figure 5 the process.

[0090] In step ST2, the operation ECU 20, acting as the image generation unit 51, generates and displays the initial 3D image. The operation ECU 20 generates a 3D model based on the three-dimensional model data obtained from the memory 47 or the in-vehicle communication unit 41, and then generates the image data for projection. The operation ECU 20 generates the image data for projection from the 3D model according to the settings of the initial projection position and display direction of the image preset for the 3D model. The operation ECU 20 may also temporarily record the generated 3D model in the memory 47. In this case, the operation ECU 20 can read the 3D model from the memory 47 in the next generation process for updating the image and generate the image data for projection. The operation ECU 20 outputs the generated image data for projection to the 3D image projection device 43. The 3D image projection device 43 projects the image generated based on the image data for projection into the space of the passenger compartment 3. Thus, for example, when the initial projection position is the standard projection position for projecting in front of the occupant in the driving posture, as Figure 4 shown, at the standard projection position in front of the occupant, the 3D model is displayed in a specified display direction. The specified display direction can be, for example, the front of the 3D model when it has a direction. The occupant in the driving posture can reach the image by directly reaching out.

[0091] In step ST3, the operation ECU 20, acting as the operation determination unit 52, determines whether an operation of the occupant on the image is detected. The operation ECU 20 obtains detection information such as the image of the stereo camera 63 from the operation detection device 44, extracts a specified part of the occupant such as a fingertip, and detects the operation of the occupant on the image according to the specified operation part, such as the position change and the presence or absence of movement in the passenger compartment 3. In the operation detection at this time, even if the operation part of the occupant does not operate the image, the operation ECU 20 can determine that an operation of the occupant on the image is detected. When no operation of the occupant on the image is detected, the operation ECU 20 repeats the determination process of this step. When an operation of the occupant on the image is detected, the operation ECU 20 advances the process to step ST4.

[0092] In step ST4, the operation ECU 20 acts as the operation determination unit 52 to determine the operation of the occupant on the image. The operation ECU 20 determines whether the operation part of the occupant is in a state of contacting the image based on the projection position on the surface of the image. When the operation part is in a state of operating the image, it further determines the contact shape, position, and movement (direction, speed) of the operation part. The contact shape can be, for example, the number of contacting fingers, the position of the hand, etc. Furthermore, the operation ECU 20 can also determine the remaining distance from the operation part to the image contact, the depth of the operation part in contact with the image, etc. based on the projection position on the surface of the image.

[0093] In addition, when the operation ECU 20 determines based on the operation information that the occupant has operated a specified image part such as a button projected into the space of the passenger compartment 3, it generates input information generated by the operation of the occupant.

[0094] In step ST5, the operation ECU 20 acts as the operation determination unit 52 and outputs the operation information related to the determined occupant operation and the input information generated by the operation to the inside and outside of the device. The operation ECU 20 outputs the operation information to the image generation unit 51, the stimulus response output unit 53, and the sound response output unit 54. The operation ECU 20 outputs the input information to the control ECUs of a plurality of controlled devices provided in each part of the vehicle 1 through the in-vehicle communication unit 41.

[0095] In step ST6, the operation ECU 20 outputs a response to the operation through touch and sound according to the operation information.

[0096] The operation ECU 20 acts as the stimulus response output unit 53, determines the position of the occupant operation part in a state of contacting the image according to the operation information, selects a plurality of ultrasonic elements 65 for outputting ultrasonic waves in such a way as to output ultrasonic waves to this position, and outputs an electrical signal to the stimulus output device 45. The stimulus output device 45 outputs ultrasonic waves from the selected plurality of ultrasonic elements 65. The occupant can obtain the tactile sensation of operating the image by using the response of the ultrasonic waves.

[0097] The operation ECU 20 acts as the sound response output unit 54, selects sound data from the memory 47 according to the movement of the occupant operation part in a state of contacting the image and the contact part of the image determined based on the operation information, and outputs a sound signal to the sound output device 46. The sound output device 46 outputs a sound based on the sound signal to the passenger compartment 3. Thus, the occupant can hear different sounds as response sounds of the operation according to the movement of the occupant operation part and the contact part of the image.

[0098] In step ST7, the operation ECU 20 acts as the image generation unit 51, updates the projected 3D image according to the operation information, and responds to the operation. The operation ECU 20 reads the three-dimensional model recorded in the memory 47, updates the image data for projection, and outputs it to the 3D image projection device 43. The 3D image projection device 43 projects the updated image into the passenger compartment 3. Thus, the occupant can visually recognize that the image is being operated on through the change in the image. The operation ECU 20 updates according to the operation information of the operation part of the same occupant detected by the operation detection device 44 to synchronize the output of the image and the stimulus.

[0099] In step ST8, the operation ECU 20 determines whether to end the non-contact operation. The operation ECU 20 determines that the non-contact operation is ended, for example, when the input information based on the operation is output, when there is no new display instruction, or when the content being displayed ends, and ends Figure 5 the process. In other cases, the operation ECU 20 returns the process to step ST3. The operation ECU 20 repeats the processes of steps ST3 to ST8 until it is determined in step ST8 that the non-contact operation is ended. During this repeated process, the operation ECU 20 repeats the processes of responding to the stimulus of the occupant's operation part, responding to the sound, and responding based on the image update according to the occupant's operation on the image. That is, when the operation ECU 20 updates the image according to the operation, it changes the output of the stimulus caused by the ultrasonic wave field corresponding to the updated image. The stimulus caused by the ultrasonic wave field can be synchronously controlled to follow the update of the image.

[0100] Figure 6 is a flowchart showing the operation determination process in the first embodiment and the process of outputting a response as a determination result.

[0101] Figure 6 The process corresponding to Figure 5 is the process of steps ST4 to ST6 enclosed by the dashed box in Figure 6 The process of outputting the input information to the outside in step ST5 and the process of outputting the sound response in step ST6 are omitted in

[0102] In step ST11, the operation ECU 20 obtains the operation of the occupant from the operation detection device 44 for operation determination.

[0103] In step ST12, the operation ECU 20 also obtains the driving state from the driving state detection device 48 for operation determination.

[0104] In step ST13, the operation ECU 20 determines whether there is movement such as the swaying of the occupant caused by the driving of the vehicle 1.

[0105] For example, when the acceleration or deceleration of the driving state obtained is less than or equal to a specified value, the operation ECU20 determines that there is no occupant movement caused by the driving of the vehicle 1. In this case, the operation ECU20 causes the process to proceed to step ST16.

[0106] For example, when the acceleration or deceleration of the driving state obtained is not less than or equal to the specified value, the operation ECU20 determines that there is occupant movement caused by the driving of the vehicle 1. In this case, the operation ECU20 causes the process to proceed to step ST14.

[0107] In step ST16 in the case where there is no movement caused by the movement of the vehicle 1, the operation ECU20 determines the operation of the occupant on the image without adjusting the detected occupant operation state.

[0108] The operation ECU20 uses the inputtable area set for the image to determine whether the operation part of the occupant detected by the operation detection device 44 is in a state of contacting the inputtable area for the operation on the image. When the operation part is in a state of operating the inputtable area, the operation ECU20 further determines the contact shape, position, and movement (direction, speed) of the operation part. In addition, when the operation ECU20 determines that the occupant has operated a specified image part such as a button on the image projected into the space of the passenger compartment 3 according to the operation information, the operation ECU20 generates input information generated by the operation of the occupant.

[0109] In step ST17, the operation ECU20 performs a response output for the operation determined in step ST16.

[0110] The operation ECU20 outputs a tactile stimulus for the operation determined in step ST16 from the stimulus output device 45 to the position of the operation part used for determination in step ST16.

[0111] When there is no movement caused by the movement of the vehicle 1, the operation ECU20 instructs the stimulus output device 45 to output a tactile stimulus to the position of the occupant operation part that has not been adjusted by the detection of the driving state detection device 48.

[0112] In step ST14 in the case where there is movement caused by the movement of the vehicle 1, the operation ECU20 adjusts the detected occupant operation and determines the operation of the occupant on the image.

[0113] The operation ECU 20 corrects the position of the inputtable area of the image and adjusts the movement of the occupant based on the movement of the occupant caused by the movement of the vehicle 1 detected by the driving state detection device 48. Also, the operation ECU 20 determines whether an operation on the image detected by the operation detection device 44 is being performed on the inputtable area whose position has been adjusted. Thereby, it is possible to determine the operation that the occupant intends to perform on the image. The operation ECU 20 determines the operation on the inputtable area by the adjusted operation, thereby determining whether the operation part of the occupant is in a state of contacting the image, and further determining the contact shape, position, and movement (direction, speed) of the operation part when the operation part is in a state of operating the image. In addition, when the operation ECU 20 determines based on the operation information that the occupant has operated a specified image part such as a button projected onto the space of the passenger compartment 3, the operation ECU 20 generates input information generated by the operation of the occupant.

[0114] In step ST15, the operation ECU 20 performs a response output of a tactile stimulus for the operation determined in step ST14.

[0115] The operation ECU 20 outputs a tactile stimulus for the operation determined in step ST14 from the stimulus output device 45.

[0116] When there is movement caused by the movement of the vehicle 1, the operation ECU 20 instructs the stimulus output device 45 to output a tactile stimulus not to the position of the occupant operation part adjusted by the detection of the driving state detection device 48, but to the occupant operation part actually detected by the operation detection device 44.

[0117] When the operation of the occupant is adjusted and determined by the detection of the driving state detection device 48 in the operation determination of step ST14, the operation ECU 20 outputs a response of a tactile stimulus for the occupant operation to a wider range than the case where the determination is made without adjustment by the detection of the driving state detection device 48 as in step ST16.

[0118] Figure 7 is Figure 6 An explanatory diagram of an example of the operation determination process corresponding to and the response output process using a tactile stimulus.

[0119] In Figure 7 the 3D image projection device 43 projects an image of a cube 70 having a button 71 on the operation surface in front of the occupant.

[0120] The operation ECU 20, which serves as the operation determination unit 52, sets an inputtable area 72 indicated by a dashed line in the position setting diagram of the projected button 71. The operation determination unit 52 determines an operation of an occupant on this inputtable area. For example, when the fingertip of the occupant moves in a way that touches the inputtable area 72, the operation determination unit 52 determines that an operation has been performed on the button 71.

[0121] Then, regarding the occupant, when there is a movement caused by the movement of the vehicle 1, the operation ECU 20 moves the inputtable area 72 from the initial position where it overlaps with the button 71. The offset direction of the inputtable area 72 from the initial position only needs to be the same direction as the movement of the occupant caused by the movement of the vehicle 1 detected by the driving state detection device 48. The offset amount of the inputtable area 72 from the initial position only needs to be the same as the amount of movement of the occupant caused by the movement of the vehicle 1 detected by the driving state detection device 48.

[0122] In Figure 7 In this case, the finger about to operate the button 71 touches in the upper right direction in the figure due to the movement of the vehicle 1. In this situation, the operation ECU 20 moves the inputtable area 72 to offset in the upper right direction from the initial position, and determines the operation of the occupant on the button 71 through the offset-moved inputtable area 73. Thereby, the operation ECU 20 can cancel and suppress the movement of the occupant caused by the movement of the vehicle 1, and determine the operation of the occupant on the button 71 through the inputtable area 73. In addition, when a lateral acceleration acts on the vehicle 1, the offset direction can be set to the lateral direction; when an acceleration in the front-rear direction acts on the vehicle 1, the offset direction can be set to the front-rear direction. In addition, the offset amount can also be proportional to the amount of movement of the occupant caused by the movement of the vehicle 1. Regarding the offset amount, in the maximum acceleration assumed to act during the normal driving of the vehicle 1, an offset amount of about several millimeters can be set.

[0123] As described above, in the present embodiment, the movement of the vehicle 1 or the overall movement of the occupant caused by the movement of the vehicle 1 is detected. And adjustment is made according to the detection of the movement detection device to suppress the movement of the occupant caused by the movement of the vehicle 1, and an operation of the operation part of the occupant on the image projected into the empty space in the passenger compartment 3 is determined based on the detection of the operation detection device 44.

[0124] Thereby, in the present embodiment, it is possible to appropriately determine a non-contact operation of the occupant on the image projected into the empty space that is independent of the movement of the vehicle 1 during driving without being affected by the movement of the vehicle 1 during driving. In the present embodiment, even when the body of the occupant is in a swaying state due to the movement of the vehicle 1, it is possible to correctly determine the operation of the occupant.

[0125] [Second Embodiment]

[0126] Next, the non-contact operation device 40 of the vehicle 1 according to the second embodiment of the present invention will be described. In the present embodiment, components corresponding to those in the above embodiment are denoted by the same reference numerals as in the above embodiment, and their description will be omitted.

[0127] Figure 8 It is a flowchart showing the operation determination process and the response output process as the determination result in the second embodiment.

[0128] In step ST21 where there is movement caused by the movement of the vehicle 1, the operation ECU 20 adjusts the detected operation of the occupant and determines the operation of the occupant on the image.

[0129] Similar to step ST22, the operation ECU 20 uses the inputtable area set for the projected image to determine the operation of the occupant. In addition, the process of step ST22 is the same as that of step ST16. Moreover, particularly in step ST21, the operation ECU 20 corrects the range of the inputtable area of the image and adjusts the movement of the occupant based on the movement of the occupant caused by the movement of the vehicle 1 detected by the driving state detection device 48. The operation ECU 20 expands and adjusts the inputtable area according to the magnitude of the movement caused by the movement of the vehicle 1.

[0130] Furthermore, the operation ECU 20 determines whether the operation of the occupant on the image detected by the operation detection device 44 is being performed on the expanded inputtable area. Thereby, the operation that the occupant wants to perform on the image can be determined. The operation ECU 20 determines the operation on the expanded inputtable area, thereby determining whether the operation part of the occupant is in a state of contacting the image, and further determining the contact shape, position, and movement (direction, speed) of the operation part when the operation part is in a state of operating the image. In addition, when the operation ECU 20 determines that the occupant has operated a specified image part such as the button 71 of the image projected into the space of the passenger compartment 3 based on the operation information, the input information generated by the operation of the occupant is generated.

[0131] In the subsequent step ST15, the operation ECU 20 performs a response output of tactile stimulation on the operation determined in step ST21. Since there is movement caused by the movement of the vehicle 1, the operation ECU 20 instructs the stimulation output device 45 to output tactile stimulation not to the position of the occupant operation part adjusted by the detection of the driving state detection device 48, but to the actual operation part of the occupant detected by the operation detection device 44.

[0132] When adjusting and determining the operation of the occupant through the detection by the driving state detection device 48 in the operation determination of step ST21, the operation ECU 20 outputs a response to the tactile stimulus of the occupant operation to a wider range than the case where the determination is made in the state where the adjustment is not performed through the detection by the driving state detection device 48 as in step ST22.

[0133] Figure 9 Is an explanatory diagram of an example of Figure 8 the operation determination process corresponding to and the response output process using the tactile stimulus.

[0134] In Figure 9 , the 3D image projection device 43 projects an image of the cube 70 having the button 71 on the operation surface in front of the occupant.

[0135] The operation ECU 20, as the operation determination unit 52, sets the inputtable area 72 indicated by the dotted line in the position setting diagram of the projected button 71. The operation determination unit 52 determines the operation of the occupant on the inputtable area. For example, when the fingertip of the occupant moves in a manner of contacting the inputtable area 72, the operation determination unit 52 determines that the button 71 has been operated.

[0136] Then, regarding the occupant, when there is a movement caused by the movement of the vehicle 1, the operation ECU 20 expands the inputtable area 72 from the initial size. The expansion rate of the inputtable area 72 from the initial size can be a ratio corresponding to the amount of movement of the occupant caused by the movement of the vehicle 1 detected by the driving state detection device 48. In addition, in Figure 9 , the inputtable area 72 expands in all directions of up, down, left, and right. In addition, for example, the inputtable area 72 can also expand in the same direction as the movement of the occupant caused by the movement of the vehicle 1 detected by the driving state detection device 48. For example, when the occupant moves only longitudinally, the operation ECU 20 can expand the inputtable area 72 only in the up and down directions.

[0137] In Figure 9 , the finger about to operate the button 71 contacts in the upper right direction in the figure due to the movement of the vehicle 1. In this case, the operation ECU 20 expands the inputtable area 72 from the initial size and determines the operation of the occupant on the button 71 through the expanded inputtable area 74. Thereby, the operation ECU 20 can suppress the influence of the movement of the occupant caused by the movement of the vehicle 1 and determine the operation of the occupant on the button 71 through the inputtable area 74.

[0138] As described above, in the present embodiment, the movement of the vehicle 1 or the overall movement of the occupant caused by the movement of the vehicle 1 is detected. And, adjustment is made based on the detection by the movement detection device to suppress the movement of the occupant caused by the movement of the vehicle 1, and an operation of the occupant's operation part on the image projected into the empty space in the passenger compartment 3 is determined based on the detection by the operation detection device 44.

[0139] Thus, in the present embodiment, it is possible to appropriately determine the non-contact operation of the occupant on the image projected into the empty space that is independent of the movement of the vehicle 1 without being affected by the movement of the vehicle 1 during driving. In the present embodiment, even when the occupant's body is in a swaying state due to the movement of the vehicle 1, it is possible to correctly determine the operation of the occupant.

[0140] [Third Embodiment]

[0141] Next, the non-contact operation device 40 of the vehicle 1 according to the third embodiment of the present invention will be described. In the present embodiment, the components corresponding to those in the above embodiment are denoted by the same reference numerals as in the above embodiment, and the description thereof is omitted.

[0142] In the present embodiment, the operation detection device 44 or the driving state detection device 48 detects the overall movement of the occupant caused by the movement of the vehicle 1 during driving by photographing the occupant or the like.

[0143] Figure 10 It is a flowchart showing the process of operation determination processing and response output processing as a determination result in the third embodiment.

[0144] In step ST31 when there is movement caused by the movement of the vehicle 1, the operation ECU 20 adjusts the detected operation of the occupant and determines the operation of the occupant on the image.

[0145] Similar to step ST32, the operation ECU 20 determines the operation of the occupant based on the movement of the occupant's operation part relative to the projected image. Additionally, the processing of step ST32 is the same as that of step ST16. Moreover, particularly in step ST31, the operation ECU 20 corrects and adjusts the operation of the occupant on the inputtable area of the image detected by the operation detection device 44 to offset the movement of the occupant caused by the movement of the vehicle 1 detected by the operation detection device 44 or the driving state detection device 48. Thus, the operation that the occupant intends to perform on the image can be obtained. The operation ECU 20 determines the operation on the image through the adjusted operation, thereby determining whether the operation part of the occupant is in a state of contacting the image. When the operation part is in a state of operating the image, it further determines the contact shape, position, and movement (direction, speed) of the operation part. Additionally, when the operation ECU 20 determines based on the operation information that the occupant has operated a specified image part such as the button 71 projected into the space of the passenger compartment 3, it generates the input information generated by the operation of the occupant.

[0146] In the subsequent step ST15, the operation ECU 20 performs a response output of tactile stimulation on the operation determined in step ST31. Due to the movement caused by the movement of the vehicle 1, the operation ECU 20 instructs the stimulation output device 45 to output tactile stimulation not to the position of the occupant's operation part adjusted through the detection of the driving state detection device 48, but to the position of the occupant's operation part actually detected by the operation detection device 44.

[0147] When adjusting and determining the operation of the occupant through the detection of the driving state detection device 48 in the operation determination of step ST31, the operation ECU 20 outputs a response of tactile stimulation to the operation of the occupant to a wider range than the case where the determination is made in a state where the detection of the driving state detection device 48 is not passed as in step ST32.

[0148] Figure 11 is Figure 10 An explanatory diagram of an example of the operation determination process corresponding to and the response output process using tactile stimulation.

[0149] In Figure 11 , the 3D image projection device 43 projects an image of the sphere 75 in front of the occupant.

[0150] The operation ECU 20 as the operation determination unit 52 determines the operation of the occupant on the projected sphere 75. The operation determination unit 52 determines that the image has been operated, for example, when the fingertip of the occupant moves in a way that touches the surface of the sphere 75.

[0151] Then, when there is occupant movement caused by the movement of the vehicle 1, the operation ECU 20 corrects and adjusts the fingertip position of the occupant who operates the sphere 75 as an image to cancel out the occupant movement caused by the movement of the vehicle 1.

[0152] For example, the operation ECU 20 calculates the moving direction and moving amount of the occupant's head caused by the movement of the vehicle 1 based on the difference in the shooting positions in the images before and after the operation. Then, the operation ECU 20 corrects the fingertip position of the occupant in the direction opposite to the moving direction of the head according to the moving amount of the head.

[0153] In Figure 11 , the occupant's head moves in a manner of swinging in the forward direction of the vehicle 1, i.e., the left direction in the figure, due to the movement of the vehicle 1. In this case, the operation ECU 20 corrects the fingertip position of the occupant who is to operate the sphere 75 as an image in the right direction in the figure, which is the rear direction of the vehicle 1, by the moving amount of the head. The operation ECU 20 determines the operation on the image based on the adjusted fingertip position.

[0154] As described above, in the present embodiment, the movement of the vehicle 1 or the overall movement of the occupant caused by the movement of the vehicle 1 is detected. And adjustments are made according to the detection of the movement detection device to suppress the occupant movement caused by the movement of the vehicle 1, and the operation of the occupant's operation part on the empty image projected into the passenger compartment 3 is determined based on the detection of the operation detection device 44.

[0155] Thus, in the present embodiment, it is possible to appropriately determine the non-contact operation of the occupant on the image projected into the empty space, which is independent of the movement of the vehicle 1, without being affected by the movement of the vehicle 1 during driving. In the present embodiment, even when the occupant's body is in a swaying state due to the movement of the vehicle 1, the operation of the occupant can be correctly determined.

[0156] The above-described embodiment is an example of a preferred embodiment of the present invention, but the present invention is not limited thereto, and various deformations or changes can be made without departing from the gist of the present invention.

Claims

1. A non-contact operation device for a vehicle, comprising: A projection device that projects an image onto the space of the vehicle's passenger compartment; A generation unit that generates and updates the image projected by the projection device; An operation detection device that detects the operation part of the vehicle occupant who performs a non-contact operation on the inputtable area, and the inputtable area is set for the image projected onto the space of the passenger compartment; An operation determination unit that determines the non-contact operation of the occupant on the image projected onto the space of the passenger compartment according to the position or movement of the occupant's operation part relative to the image; A motion detection device that detects the motion of the vehicle or the motion of the occupant caused by the motion of the vehicle; A stimulation output device that is arranged for the occupant sitting in the passenger compartment and can output a tactile stimulation caused by an ultrasonic field to the occupant who operates on the image; and A stimulation response output unit that outputs the tactile stimulation caused by the ultrasonic field to the operation part of the occupant who operates on the projected image as a response to the operation determined by the operation determination unit, wherein the operation determination unit moves the position of the inputtable area relative to the position of the image from a first position overlapping with the image to a second position offset from the position of the first position relative to the image in the same direction as the motion of the occupant caused by the motion of the vehicle according to the detection of the motion detection device, or expands the size of the inputtable area relative to the size of the image from a first size overlapping with the image to a second size obtained by expanding the first size at an expansion rate corresponding to the amount of motion of the occupant caused by the motion of the vehicle based on the detection of the motion detection device, and determines the operation of the occupant's operation part on the image projected onto the space of the passenger compartment, wherein the stimulation response output unit is configured to: When it is determined by the detection of the motion detection device that there is no motion of the occupant and the operation determination unit makes a determination without adjusting the occupant's operation, output a response of the tactile stimulation to the operation of the occupant for the position of the occupant's operation part detected by the motion detection device that has not been adjusted; When it is determined by the detection of the motion detection device that there is motion of the occupant and the operation determination unit adjusts and determines the occupant's operation through the detection of the motion detection device, output a response of the tactile stimulation to the operation of the occupant not to the position of the occupant's operation part adjusted through the detection of the motion detection device, but to the position of the occupant's operation part actually detected by the operation detection device, and output a response of the tactile stimulation to the operation of the occupant to a wider range than in the case of making a determination without adjusting through the detection of the motion detection device.

2. The non-contact operation device for a vehicle according to claim 1, wherein, The operation determination unit determines the operation of the occupant based on the movement of the occupant with respect to the image projected by the projection device, and adjusts the movement of the occupant detected by the operation detection device in a manner that suppresses the movement of any of the body, upper body, shoulders, and head caused by the movement of the vehicle based on the detection by the movement detection device, thereby determining the operation of the occupant.

3. The non-contact operation device for a vehicle according to claim 1 or 2, wherein the movement detection device detects the driving state of the vehicle.

4. The non-contact operation device for a vehicle according to claim 1 or 2, wherein the movement detection device detects the movement of the occupant caused by the movement of the vehicle based on an image of the occupant being captured.

5. The non-contact operation device for a vehicle according to claim 3, wherein the movement detection device detects the movement of the occupant caused by the movement of the vehicle based on an image of the occupant being captured.

6. A vehicle, comprising: the non-contact operation device according to any one of claims 1 to 5; and a plurality of controlled devices that are connected to the non-contact operation device via a network and control the vehicle, wherein the plurality of controlled devices respectively obtain, via the network, input information generated by the non-contact operation device based on a non-contact operation performed by an occupant on an image projected into the passenger compartment.

Images