Non-contact operation device for vehicle and vehicle

Abstract

The present invention relates to a non-contact operation device for a vehicle and a vehicle, and aims to improve the operation device for the vehicle. The non-contact operation device (40) of the vehicle (1) comprises: a projection device (43) that projects an image onto the vehicle interior (3) of the vehicle (1); a generation unit (51) that generates and updates the image projected by the projection device (43); a stimulus response output unit (53) that outputs tactile stimulation caused by an ultrasonic field as a response to the operation of the operating part of the occupant who operates the image projected on the vehicle interior (3); and a detection device (44) that detects the operating part of the occupant who performs non-contact operation on the image projected on the space of the vehicle interior (3). The generation unit (51) and the stimulus response output unit (53) update the output of the image and the tactile stimulation based on the operation information of the operating part of the occupant detected by the detection device (44).

Description

Technical Field

[0001] The invention relates to a non-contact operation device for a vehicle. Background Art

[0002] Various operating components are provided in the vehicle. For example, the vehicle has a start switch, a steering wheel, a shift lever, pedals, etc. for controlling the driving of the vehicle. In addition, the vehicle is provided with operating components for air conditioning devices, navigation devices, audio devices, image receiving devices, hands-free calling devices, and other equipment and machines. In this way, a variety of operating components are provided in the vehicle. These various operating components are components that are laid out relative to the inner surface of the vehicle cabin, but in recent years, the layout surface of the vehicle cabin has shown a trend of being insufficient. In the future, for example, when considering adding game devices, network communication devices, entertainment devices, etc., it may not be possible to add operating components for these devices.

[0003] In particular, the development of autonomous driving technology is advancing in vehicles. In fact, as the development progresses, the vehicle can drive autonomously even if the occupants do not directly control the driving of the vehicle, and the occupants may ride in a state where the seats are reclined, for example. In this case, when the occupants want to perform a certain operation, they need to stand up and extend their arms to the operating components arranged on the inner surface of the vehicle to operate.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2017-027401 Summary of the invention

[0007] Problems to be solved by the invention

[0008] Therefore, regarding vehicles, as disclosed in Patent Document 1, there is a demand for the development of a new operating device for operating an image projected onto a vehicle cabin.

[0009] In Patent Document 1, an image is presented, and in response to an operation corresponding to the presented image, tactile stimulation by an ultrasonic field is output to the operating part of the occupant. Therefore, the occupant can get a sense of reality that the operation corresponding to the image has been performed.

[0010] However, Patent Document 1 only increases and decreases the tactile stimulation according to the moving direction of the operating part, and the tactile stimulation does not correspond to the image itself or the change of the image. As a result, since the change of the tactile stimulation does not correspond to the change of the image, for example, there may be a sense of disharmony caused by a difference or deviation between the feeling received from the visual sense and the feeling received from the tactile sense.

[0011] Thus, the vehicle needs to have an improved operating device.

[0012] Means of solving problems

[0013] The non-contact operation device of a vehicle of the present invention comprises: a projection device, which projects an image onto a vehicle cabin; a generation unit, which generates and updates the image projected by the projection device; a stimulus response output unit, which outputs tactile stimulation caused by an ultrasonic field as a response to the operation of the operating part of the occupant who operates the image projected on the cabin; and a detection device, which detects the operating part of the occupant who performs non-contact operation on the image projected on the space of the cabin, wherein the generation unit and the stimulus response output unit update the output of the image and the tactile stimulation based on the operation information of the operating part of the occupant detected by the detection device.

[0014] Preferably, when the generating unit updates the image according to the operation, the stimulation response output unit may change the output of the tactile stimulation caused by the ultrasonic field in correspondence with the updated image.

[0015] Preferably, the generating unit can update the image according to the contact position of the image operated by the operating part of the occupant through deformation of the contact position of the image, movement of the image and a combination thereof. When the generating unit updates the image in a manner that deforms the contact position of the image, the stimulus response output unit changes the output of the tactile stimulation caused by the ultrasonic field; when the generating unit updates the image in a manner that moves the image, the stimulus response output unit moves the output of the tactile stimulation caused by the ultrasonic field in the same direction.

[0016] Preferably, the generating unit and the stimulus response output unit may move the output of the image and the tactile stimulus along the movement direction of the operating part of the occupant.

[0017] Preferably, the generation unit and the stimulation response output unit may move the output of the image and tactile stimulation by an amount corresponding to a movement speed of the operating part of the occupant.

[0018] Preferably, the stimulus response output unit may output different tactile stimuli according to at least one of the following: a contact position operated by an operating part of the occupant with respect to the image, a movement direction of the operating part of the occupant, and a movement speed of the operating part of the occupant.

[0019] Preferably, the stimulus response output unit may change the tactile stimulation after the shape change from the tactile stimulation before the shape change according to the shape change of the image caused by the update of the generating unit.

[0020] Preferably, the stimulus response output unit can update the image by the generating unit, so that the tactile stimulation after the image update increases or decreases compared with the tactile stimulation before the image update according to the change in the contact amount between the occupant's operating part and the image.

[0021] Preferably, the contactless operation device of the vehicle may have: a sound output device that outputs sound to the vehicle cabin; and a sound response output unit that outputs different sounds from the sound output device depending on at least one of the movement of the occupant's operating part relative to the image projected onto the vehicle cabin and the contact position of the image.

[0022] The vehicle of the present invention comprises: a contactless operating device as described in any one of the above items; a controlled device, which is connected to the contactless operating device via a network to control the vehicle; and a communication device, which is connected to the contactless operating device via a network to communicate with the outside of the vehicle, wherein the contactless operating device projects the content received by the communication device as an image.

[0023] Effects of the Invention

[0024] In the present invention, the image to be projected onto the vehicle cabin is generated by the generating unit, and the projecting device projects the generated image onto the vehicle cabin. In addition, the detecting device for detecting the operation of the occupant detects the operating part of the occupant who performs non-contact operation on the image projected onto the space of the cabin. The stimulus response output unit outputs tactile stimulation caused by the ultrasonic field as a response to the operation for the operating part of the occupant who operates the image projected onto the cabin.

[0025] Thus, when the passenger performs a non-contact operation on the image projected onto the vehicle interior space, the tactile stimulation caused by the ultrasonic field is applied to the operating part where the operation is performed as a response to the operation. The passenger can obtain a sense of reality as if he or she is actually operating the image projected onto the vehicle interior space. The passenger can operate the image while maintaining the feeling of actual operation.

[0026] Furthermore, the occupant actually operates the operating member in a non-contact manner in the vehicle interior space, and therefore does not need to directly contact a structure such as an inner surface of the vehicle interior and an operating member such as a steering wheel to operate. In the vehicle interior, the occupant does not need to provide an operating member that can be directly contacted by the operating portion in the space itself where the occupant actually operates by the operating portion.

[0027] The operability for the occupant can be improved without newly arranging an operating member that is directly operated by the occupant's operating part near the occupant.

[0028] Furthermore, by changing the display, it is possible to switch and project an operation panel or the like suitable for the operation of each of the plurality of devices.

[0029] In particular, in the present invention, the generation unit and the stimulus response output unit update the output of the image and the tactile stimulation according to the operation information of the operation part of the occupant detected by the detection device. For example, when the generation unit updates the image according to the operation, the stimulus response output unit changes the output of the tactile stimulation caused by the ultrasonic field corresponding to the updated image. When the update of the image changes the surface shape or surface position of the image, the stimulus response output unit uses the changed surface shape or surface position of the image as a reference to update the output mode or position of the tactile stimulation caused by the ultrasonic field.

[0030] Furthermore, the generation unit and the stimulus response output unit update the output of the image and the tactile stimulation based on the operation information of the operating part of the occupant detected by the detection device, wherein the detection device detects the operating part of the occupant who performs non-contact operation on the image projected into the vehicle interior space. The tactile stimulation can change in correspondence with the image itself and the change of the image. For example, the change of the tactile stimulation can correspond to the change of the image. The occupant is less likely to have a sense of disharmony caused by the difference or deviation between the feeling received from vision and the feeling received from touch.

[0031] Therefore, in the present invention, even after the initial operation on the image, the image and the tactile stimulation are changed to be synchronized according to the operation.

[0032] Even after the passenger starts to operate the image, they can maintain the feeling of actually operating the image that changes according to the operation, and continue to operate the image that changes according to the operation. The image and the tactile sense change synchronously, so that even if the passenger continues to operate the non-physical operating parts in mid-air, they can continue to operate with a sense of reality.

[0033] On the other hand, when the change in the image and the change in the tactile sense are not updated synchronously, although the occupant can operate the initial image realistically, it is difficult to operate the subsequent operations realistically. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] 1(A)-(C) are schematic illustrations of a vehicle according to an embodiment of the present invention;

[0035] Figure 2 1 (A) - (C) of the vehicle control system schematic diagram;

[0036] Figure 3 1 (A) - (C) of the automobile provided with a non-contact operating device diagram;

[0037] Figure 4 Yes Description Figure 3 A diagram showing an example of the arrangement of components of a non-contact operation device in a vehicle cabin;

[0038] Figure 5 is a flowchart showing the flow of the non-contact operation process in the first embodiment;

[0039] Figure 6 is a flowchart showing the flow of a main part of the non-contact operation process in the second embodiment;

[0040] Figure 7 Yes means Figure 6 A flowchart of the process of stimulus content change processing in step ST19;

[0041] Figure 8 Yes means Figure 6 Flow chart of the process of the image update process in step ST20.

[0042] Explanation of symbols

[0043] 1…car (vehicle), 2…body, 3…cabin, 4…seat, 10…control system, 19…external communication ECU, 20…operation ECU, 40…contactless operation device, 41…in-vehicle communication unit, 42…timer, 43…3D image projection device, 44…operation detection device, 45…stimulus output device, 46…sound output device, 47…memory, 51…image generation unit, 52…operation determination unit, 53…stimulus response output unit, 54…sound response output unit, 61…display screen, 62…half mirror, 63…stereo camera, 64…quadrilateral frame, 65…ultrasonic element DETAILED DESCRIPTION

[0044] Embodiments of the present invention will be described below based on the drawings.

[0045] [First embodiment]

[0046] 1(A) to (C) are schematic explanatory diagrams of a car 1 according to an embodiment of the present invention.

[0047] FIG. 1(A) is a top view, FIG. 1(B) is a side view, and FIG. 1(C) is a side view.

[0048] The automobile 1 shown in FIG. 1(A) to FIG. 1(C) is an example of a vehicle. The automobile 1 has a body 2 having a cabin 3 in which passengers sit. A seat 4 is provided in the cabin 3 for passengers to sit. A steering wheel 5, a shift lever 6, a brake pedal 7, and an accelerator pedal 8 are provided in the front portion of the cabin 3 so that passengers can operate the driving of the automobile 1. As shown in FIG. 1(B), passengers can operate these operating components such as the steering wheel 5 while sitting in the seat 4.

[0049] Figure 2 1(A) to (C) are schematic explanatory diagrams of a control system 10 of the automobile 1 . Figure 2In the figure, a plurality of control devices constituting the control system 10 are represented and indicated by respectively programmed control ECUs (Electronic Control Units).

[0050] Specifically, Figure 2 The driving ECU 11, steering ECU 12, brake ECU 13, automatic driving / driving support ECU 14, driving operation ECU 15, detection ECU 16, air conditioning ECU 17, occupant monitoring ECU 18, external communication ECU 19, operation ECU 20 of the contactless operation device 40, and system ECU 21 are shown. The plurality of control ECUs are connected to a central gateway (CGW) 27 as a relay device via a vehicle network 26 such as CAN (Controller Area Network) or LIN (Local Interconnect Network) adopted in the automobile 1.

[0051] Furthermore, in each control module, the control ECU is connected to an electronic device used in the automobile 1. The activated control ECU executes various processes to control the operation of the respectively connected electronic devices according to the information (data) acquired from the vehicle network 26. In addition, the control ECU also outputs information (data) such as the operation status of the respectively connected electronic devices to the vehicle network 26.

[0052] For example, the driving operation ECU 15 is connected to operation detection sensors such as the steering wheel 5, the brake pedal 7, the accelerator pedal 8, and the shift lever 6, which are operated by the occupant to control the driving of the automobile 1. The driving operation ECU 15 outputs control information corresponding to the operation amount to the vehicle network 26. The drive ECU 11, the steering ECU 12, and the brake ECU 13 obtain information from the vehicle network 26 to control the driving of the automobile 1.

[0053] The detection ECU 16 is connected to a speed sensor 31 of the vehicle 1, an acceleration sensor 32 capable of detecting acceleration of the vehicle 1 caused by a collision, etc., an external camera 33, etc. The detection ECU 16 outputs the values ​​of the speed sensor 31 and the acceleration sensor 32 of the vehicle 1, the image of the external camera 33, etc. to the vehicle network 26. The detection ECU 16 can also predict a collision based on the image of the external camera 33 and output the prediction result to the vehicle network 26. The central gateway 27 forwards the information. The operation ECU 20 obtains information from the vehicle network 26 and displays the information on a display device (not shown) connected thereto. The operation ECU 20 is connected to a display device and an operation device (not shown) operated by the occupant.

[0054] The occupant monitoring ECU 18 is connected to an interior camera 34. The occupant monitoring ECU 18 performs various processes related to the occupants of the automobile 1 based on the images of the interior camera 34 and information such as acceleration of impact acquired from the vehicle network 26. The occupant monitoring ECU 18 outputs images, sounds, and other information (data) to the vehicle network 26 as needed.

[0055] The external communication ECU 19 wirelessly communicates with, for example, a communication base station 81 or a communication device of another car 82 that exists outside the car 1. The communication base station 81 or the communication device of another car 82 together with the server device 83 constitute the transportation system 80. The external communication ECU 19 wirelessly transmits information acquired from the vehicle network 26 to the communication base station 81 or the communication device of another car 82. The transmitted information can be used, for example, in the server device 83 or in another car 82. In addition, the external communication ECU 19 receives information from the communication base station 81 or the communication device of another car 82 and outputs it to the vehicle network 26. Thus, the occupant monitoring ECU 18 of the car 1, for example, can transmit and receive information (data) to and from the server device 83 or another car 82 outside the car through the external communication ECU 19.

[0056] In addition, the battery 91 installed in the vehicle 1 supplies power to each part, so that Figure 2 The control system 10 shown operates. Power supply lines from the battery 91 to each part are laid in the automobile 1 together with, for example, a communication cable of the vehicle network 26. In addition to the battery 91, power may be supplied to the control system 10 from a generator or a power receiver.

[0057] In this way, the operation ECU 20 is connected to other control devices in the automobile 1 via the vehicle network 26. Figure 2 In the embodiment, ECUs other than the operation ECU 20 are controlled devices (controlled devices) for controlling the automobile 1. However, the external communication ECU 19 is a communication device for communicating with the outside of the automobile 1.

[0058] In addition, as described above, various devices and their operating components are provided in the automobile 1. For example, the automobile 1 has a start switch, a steering wheel 5, a shift lever 6, pedals, etc. for controlling the driving of the automobile 1. In addition, in addition to the above-mentioned devices, the automobile 1 is also provided with operating components for a navigation device, an audio device, an image receiving device, a hands-free calling device, and other equipment and machines. In this way, a variety of operating components are provided in the automobile 1. The various operating components are components that are laid out relative to the inner surface of the vehicle interior 3, but in recent years, the layout surface of the vehicle interior 3 has shown a trend of being insufficient. In the future, for example, when considering the addition of game devices, network communication devices, entertainment devices, etc., it may not be possible to add operating components for these devices.

[0059] In particular, the development of autonomous driving technology is being advanced in the automobile 1. In fact, as the development progresses, if the automobile 1 can be driven autonomously without the passenger directly controlling the driving of the automobile 1, the passenger may ride in a state where the seat 4 is folded down, as shown in FIG. 1(C). In this case, when the passenger wants to perform a certain operation, he needs to stand up from the seat 4 and extend his arms to the operating parts such as the steering wheel 5 arranged relative to the inner surface of the vehicle cabin 3 to operate.

[0060] Therefore, regarding the automobile 1 , there is a demand for development of a new operating device for operating, for example, an image projected onto the interior 3 of the automobile 1 .

[0061] Thus, the operating device of the automobile 1 needs to be further improved.

[0062] Figure 3 1(A) to (C) are diagrams showing the structure of a non-contact operating device 40 provided in the automobile 1 .

[0063] Figure 3 The non-contact operation device 40 includes an in-vehicle communication unit 41, a timer 42, a 3D image projection device 43, an operation detection device 44, a stimulation output device 45, a sound output device 46, a memory 47, and an operation ECU 20 connected to these.

[0064] The memory 47 is, for example, a nonvolatile memory. The memory 47 records programs and data for contactless operation. The program may be a program that performs processing by AI. The program may include a learning program for AI processing. The data includes three-dimensional model data of an image projected during contactless operation, etc. The three-dimensional model data is image data for contactless operation, and has, for example, a plurality of polygonal data constituting the surface of the model.

[0065] The operation ECU 20 may be, for example, a microcomputer such as a central processing unit, an ASIC, or a DSP. The operation ECU 20 reads a program for contactless operation from the memory 47 and executes it. Thus, in the operation ECU 20, a control unit of the contactless operation device 40 is implemented. The control unit of the contactless operation device 40 controls the operation of the entire contactless operation device 40 to implement various functions for contactless operation in the contactless operation device 40. For example, in the operation ECU 20, as various functions for contactless operation, an image generation unit 51, an operation determination unit 52, a stimulus response output unit 53, and a sound response output unit 54 are implemented.

[0066] The in-vehicle communication unit 41 is connected to the vehicle network 26. The in-vehicle communication unit 41 transmits and receives information (data) between other control ECUs such as the air conditioning ECU 17 and the automatic driving / driving support ECU 14 shown in the figure through the vehicle network 26. For example, the in-vehicle communication unit 41 obtains a display instruction of an air conditioning operation image from the air conditioning ECU 17 and outputs it to the operation ECU 20. The in-vehicle communication unit 41 obtains a display instruction of a setting operation image for automatic driving / driving support from the automatic driving / driving support ECU 14 and outputs it to the operation ECU 20.

[0067] The timer 42 measures elapsed time or the time.

[0068] The image generation unit 51 obtains three-dimensional model data from the memory 47 or the in-vehicle communication unit 41 according to the display instruction so as to generate data of the projected image. The image generation unit 51 generates a three-dimensional model based on the obtained three-dimensional model data. The image generation unit 51 determines the projection position and direction of the three-dimensional model in the vehicle cabin 3 seen by the occupant, and generates image data for projection based on the three-dimensional model (stereoscopic model). The image generation unit 51 outputs the image data for projection to the 3D image projection device 43. The image generation unit 51 can also generate a two-dimensional model (plane model) based on the two-dimensional model data to generate image data for projection. In addition, even in the absence of a display instruction, the image generation unit 51 can read the display data of the content such as dynamic images or static images from the memory 47 or the in-vehicle communication unit 41, generate the image data of the three-dimensional image (stereoscopic image) or two-dimensional image (plane image) of the content, and output it to the 3D image projection device 43.

[0069] The 3D image projection device 43 projects a 3D (three-dimensional) image or a 2D (two-dimensional) image into the space of the vehicle cabin 3. The 3D image projection device 43 can project the image into the hollow space of the vehicle cabin 3 by, for example, a hologram method or a mirror method. Thus, a three-dimensional image is projected at the projection position of the vehicle cabin 3 so that the passengers can visually confirm the three-dimensional model.

[0070] The operation detection device 44 detects the predetermined operation position of the occupant. The operation detection device 44 may be, for example, a stereo camera 63 having two imaging elements arranged side by side. In this case, the operation detection device 44 detects the operation performed by the occupant on the image projected onto the hollow part of the vehicle interior 3 based on the images of the two imaging elements.

[0071] The operation determination unit 52 obtains detection information such as the image of the stereo camera 63 from the operation detection device 44, and determines the operation of the passenger on the image projected into the space of the vehicle interior 3 based on the detection information. The operation determination unit 52 determines the position and movement of the passenger's operating part. The movement includes movement information of the operating part such as the direction, speed, and acceleration of the movement. The operation determination unit 52 obtains the pixel position containing the characteristics of the passenger's fingertips from the image through AI processing, and generates the position information of the fingertips through the correlation triangulation method of the image of the stereo camera 63. The operation determination unit 52 generates the movement of the fingertips, such as the direction, speed, and acceleration of the movement, based on the results of the time-shifted shooting. The operation determination unit 52 determines the position and movement of the passenger's operating part based on the projection position of the image, and determines the passenger's operation on the image projected into the space of the vehicle interior 3. The operation determination unit 52 determines whether the passenger's operating part is in contact with the projected image, the remaining distance from the contact, the depth of the contact, etc. The operation determination unit 52 outputs the determined operation information to each part of the operation ECU 20. The operation determination unit 52 outputs the operation information to the image generation unit 51, the stimulus response output unit 53, and the sound response output unit 54. The image generation unit 51 updates the image data for projection according to the operation of the occupant, and outputs it to the 3D image projection device 43. As a result, the image projected into the vehicle interior 3 by the 3D image projection device 43 is updated in response to the operation.

[0072] Furthermore, when the operation determination unit 52 determines that the occupant has operated the image projected into the space of the vehicle interior 3 based on the operation information, the input information generated by the occupant's operation is output to each part of the vehicle 1 through the in-vehicle communication unit 41. For example, when an operation button of the projected image is operated, the operation determination unit 52 generates input information generated by the operation and outputs it to the in-vehicle communication unit 41. The in-vehicle communication unit 41 outputs the input information to the controlled device through the vehicle network 26.

[0073] The stimulation output device 45 may be any device that can apply a sense of touch to the operating part of the occupant through an electrical signal. As a device for applying a sense of touch without contact, for example, there is a device that generates an ultrasonic field and applies the ultrasonic field or field changes to the operating part of the occupant to give a sense of touch to the skin of the operating part. The stimulation output device 45 may be, for example, an element array in which a plurality of ultrasonic elements 65 are arranged on a plane away from the hand. By selectively outputting ultrasonic waves from the plurality of ultrasonic elements 65, a sense of touch as if in contact with an object can be applied to a local part of the hand of the occupant, such as a fingertip.

[0074] The stimulus response output unit 53 outputs an electrical signal to the stimulus output device 45, and selectively outputs ultrasonic waves corresponding to the operation from the plurality of ultrasonic elements 65. Therefore, the stimulus response output unit 53 can generate an ultrasonic field in the vehicle interior 3. The stimulus response output unit 53 locally applies an ultrasonic field or a change in the field to the operating part of the occupant determined by the operation determination unit 52 as being in contact with the image. When a person, for example, extends his hand into the ultrasonic field, he can feel the ultrasonic field through the skin on the surface of his hand. Thus, the stimulus response output unit 53 can output tactile stimulation caused by the ultrasonic field as a response to the operation of the operating part of the occupant who performs non-contact operation on the image projected into the space of the vehicle interior 3.

[0075] In addition, the stimulus response output unit 53 may also have, for example, an element array in which a plurality of pressure output elements are arranged. In this case, the stimulus response output unit 53 may apply pressure to the human skin by individually controlling the operation of the plurality of pressure output elements. Thus, the occupant can obtain a tactile sensation corresponding to the operation.

[0076] The sound output device 46 may be, for example, a speaker driven by a sound signal.

[0077] The sound response output unit 54 outputs a sound signal to the sound output device 46, and the sound corresponding to the operation is output from the sound output device 46. The sound response output unit 54 selects and acquires the sound data stored in the memory 47, and outputs the sound signal generated based on the sound data to the speaker as the sound output device 46. Thus, the occupant can hear the sound corresponding to the operation.

[0078] Figure 4 Yes Description Figure 3 FIG. 4 is a diagram showing an example of arrangement of components of the contactless operation device 40 in the vehicle interior 3 . Figure 4 Also shown is a passenger operating the image with his fingertips.

[0079] The contactless operation device 40 is installed in the vehicle interior 3 of the automobile 1 .

[0080] Figure 4 yes Figure 3 A schematic diagram illustrating a specific combination example of a 3D image projection device 43, an operation detection device 44 and a stimulation output device 45.

[0081] exist Figure 4 In the figure, a display screen 61 and a half mirror 62 are shown as the 3D image projection device 43. A stereo camera 63 is shown as the operation detection device 44. An element array is shown as the stimulation output device 45.

[0082] The stimulation output device 45 has a hollow quadrilateral frame 64. An element array in which a plurality of ultrasonic elements 65 are arranged is provided on each of the four faces of the quadrilateral frame 64. By operating the element arrays on the upper, lower, left and right surfaces of the quadrilateral frame 64, as shown in the figure, the ultrasonic field acts on the fingertips entering the quadrilateral frame 64. Thus, a person can get the tactile sensation of the fingertips touching.

[0083] The half mirror 62 is arranged on the opposite side of the person relative to the hollow quadrilateral frame 64 of the stimulation output device 45. The half mirror 62 is arranged to be inclined 45 degrees relative to the axis of the quadrilateral frame 64 indicated by the dotted line. A display screen 61 for displaying a three-dimensional image (stereoscopic image) or a two-dimensional image (plane image) is arranged on the lower side of the half mirror 62. Therefore, the person can visually confirm the three-dimensional image (stereoscopic image) or the two-dimensional image (plane image) inside the hollow quadrilateral frame 64 of the stimulation output device 45. The person can visually confirm the sphere as a three-dimensional image represented by the circle in the figure inside the hollow quadrilateral frame 64.

[0084] The stereo camera 63 can be set, for example, on the rear side of the half mirror 62. Thus, the two imaging elements of the stereo camera 63 are set at the position of forming the object relative to the axis of the quadrilateral frame 64. Thus, the stereo camera 63 can shoot a finger or the like entering the hollow quadrilateral frame 64 of the stimulation output device 45. Hereinafter, as required, the direction along the axis of the quadrilateral frame 64 is referred to as the Z direction, and the directions perpendicular to the Z direction are referred to as the Y direction and the X direction. In addition, the Z direction can usually be regarded as the front-rear direction of the automobile 1.

[0085] in addition, Figure 3 The components of the non-contact operation device 40 do not need to be the same as Figure 4 As shown, they are centrally arranged in one place.

[0086] For example, when the hollow quadrilateral frame 64 of the stimulation output device 45 is arranged in front of the occupant, it may hinder the operation when the occupant operates the steering wheel 5, etc. As shown in Figures 1(A)-(C), the hollow quadrilateral frame 64 can be arranged in a quadrilateral frame shape along the outer peripheral surface of the vehicle body 2. In this case, there is no need to set a structure such as a quadrilateral frame 64 in front of the seat 4 where the occupant sits. Even in this case, as shown by the dotted circle in the figure, the image can be projected into the space of the vehicle cabin 3. In addition, the operating area corresponding to the projected image is located on the inner side of the quadrilateral frame 64. In addition, the hollow quadrilateral frame 64 does not necessarily have to be a square frame shape. The element array can also be arranged along the outer periphery of the vehicle body 2. In addition, the element array can also be arranged as a whole on the inner surface of the vehicle cabin 3.

[0087] The half mirror 62 can be basically set on the side opposite to the occupant's head with the hollow quadrilateral frame 64 of the stimulation output device 45 as a reference. In addition, if it is only for the purpose of projecting an image into the hollow portion inside the quadrilateral frame 64, a full mirror can be used instead of the half mirror 62. In addition, the display screen 61 itself can also be configured on the side opposite to the occupant's head with the hollow quadrilateral frame 64 of the stimulation output device 45 as a reference. In this case, the half mirror 62 or the full mirror is not required. Such a half mirror 62 or the full mirror and the display screen 61 can be configured, for example, on the footrest and the roof.

[0088] The detection device such as the stereo camera 63 can be arranged in an interior mirror, a rearview mirror, etc. In addition, the vehicle interior 3 can be photographed by a camera element of an occupant monitoring device such as a DMS. The detection device such as the stereo camera 63 can be used as long as it can photograph the part to be detected such as the fingertips of the occupant. In addition, the fingertips of the occupant can be detected by scanning the vehicle interior 3 with a laser instead of an image.

[0089] Figure 5 1 is a flowchart showing the flow of the non-contact operation process in the first embodiment.

[0090] In step ST1, the operation ECU 20 determines whether to start the non-contact operation of projecting an image. For example, when there is a display instruction or when the content is displayed, the operation ECU 20 determines that the non-contact operation is started, and the process proceeds to step ST2. In other cases, the operation ECU 20 determines that the non-contact operation is not started, and ends the process. Figure 5 processing.

[0091] In step ST2, the operation ECU20 acts as the image generation unit 51 to generate and display the initial 3D image. The operation ECU20 generates a three-dimensional model based on the three-dimensional model data obtained from the memory 47 or the in-vehicle communication unit 41, and then generates image data for projection. The operation ECU20 generates image data for projection from the three-dimensional model according to the settings of the initial projection position and display direction of the image preset for the three-dimensional model. The operation ECU20 may also temporarily record the generated three-dimensional model in the memory 47. In this case, the operation ECU20 may read the three-dimensional model from the memory 47 and generate image data for projection in the next generation process for updating the image. The operation ECU20 outputs the generated image data for projection to the 3D image projection device 43. The 3D image projection device 43 projects the image generated based on the image data for projection into the space of the vehicle interior 3. Thus, for example, Figure 4As shown, the three-dimensional model is displayed in a predetermined projection position in front of the passenger with a predetermined display direction. For example, as shown in FIG. 1(B), the predetermined projection position may be set to a position that is within the reach of the passenger sitting on the seat 4. The predetermined display direction may be, for example, a direction in which the front of the three-dimensional model faces the passenger, if the three-dimensional model has a direction.

[0092] In step ST3, the operation ECU 20 acts as the operation determination unit 52 to determine whether the occupant's operation on the image is detected. The operation ECU 20 obtains detection information such as the image of the stereo camera 63 from the operation detection device 44, extracts the specified parts of the occupant such as the fingertips, and detects the occupant's operation on the image based on the position change and the presence or absence of movement of the specified operation parts in, for example, the vehicle interior 3. In the operation detection at this time, even if the occupant's operation part does not operate the image, the operation ECU 20 can determine that the occupant's operation on the image is detected. When the occupant's operation on the image is not detected, the operation ECU 20 will repeat the judgment process of this step. When the occupant's operation on the image is detected, the operation ECU 20 causes the processing to enter step ST4.

[0093] In step ST4, the operation ECU 20 acts as the operation determination unit 52 to determine the occupant's operation on the image. The operation ECU 20 determines whether the occupant's operating part is in contact with the image based on the projection position of the surface of the image, and further determines the contact shape, position and movement (direction, speed) of the operating part when the operating part is in the state of operating the image. The contact shape can be, for example, the number of fingers in contact, the position of the hand, etc. Furthermore, the operation ECU 20 can also determine the remaining distance between the operating part and the image, the depth of contact between the operating part and the image, etc. based on the projection position of the surface of the image.

[0094] Furthermore, when the operation ECU 20 determines based on the operation information that the occupant has operated a predetermined image portion such as a button of the image projected into the space of the vehicle interior 3 , the operation ECU 20 generates input information resulting from the occupant's operation.

[0095] In step ST5, the operation ECU 20, as the operation determination unit 52, outputs the operation information related to the determined occupant operation and the input information generated by the operation to the inside and outside of the device. The operation ECU 20 outputs the operation information to the image generation unit 51, the stimulus response output unit 53, and the sound response output unit 54. The operation ECU 20 outputs the input information to each part of the automobile 1 through the in-vehicle communication unit 41.

[0096] In step ST6 , the operation ECU 20 outputs a response to the operation through tactile sensation and sound according to the operation information.

[0097] The operation ECU 20, as the stimulus response output unit 53, determines the position of the occupant's operating part in contact with the image based on the operation information, selects a plurality of ultrasonic elements 65 that output ultrasonic waves to output ultrasonic waves to the position, and outputs an electrical signal to the stimulus output device 45. The stimulus output device 45 outputs ultrasonic waves from the selected plurality of ultrasonic elements 65. The occupant can obtain the tactile sensation of the operation image by using the response of the ultrasonic waves.

[0098] The operation ECU 20, as the sound response output unit 54, selects sound data from the memory 47 according to the movement of the operating part of the occupant in contact with the image and the contact position of the image determined based on the operation information, and outputs a sound signal to the sound output device 46. The sound output device 46 outputs a sound based on the sound signal to the vehicle interior 3. Thus, the occupant can hear different sounds according to the movement of the operating part of the occupant and the contact position of the image as a response sound to the operation.

[0099] In step ST7, the operation ECU 20, as the image generation unit 51, updates the projected 3D image according to the operation information to respond to the operation. The operation ECU 20 reads the three-dimensional model recorded in the memory 47, updates the image data for projection, and outputs it to the 3D image projection device 43. The 3D image projection device 43 projects the updated image to the vehicle interior 3. As a result, the occupant can visually recognize that the image is being operated by the change in the image. The operation ECU 20 is updated according to the operation information of the operation part of the same occupant detected by the operation detection device 44 to synchronize the output of the image and the stimulus.

[0100] In step ST8, the operation ECU 20 determines whether the non-contact operation is terminated. The operation ECU 20 determines that the non-contact operation is terminated when, for example, the input information based on the operation is output, there is no new display instruction, or the content being displayed ends. Figure 5 In other cases, the operation ECU 20 returns the process to step ST3. The operation ECU 20 repeats the process from step ST3 to step ST8 until it is determined in step ST8 that the non-contact operation is terminated. In this repeated process, the operation ECU 20 repeats the response to the stimulation of the occupant's operation part, the response to the sound, and the response based on the image update according to the occupant's operation on the image. That is, when the operation ECU 20 updates the image according to the operation, it changes the output of the stimulation caused by the ultrasonic field corresponding to the updated image.

[0101] As described above, in the present embodiment, the operation ECU 20 generates an image to be projected onto the cabin 3 of the automobile 1, and the operation detection device 44 projects the generated image onto the cabin 3 of the automobile 1. In addition, the operation detection device 44 for detecting the operation of the occupant detects the operation part of the occupant who performs a non-contact operation on the image projected onto the space of the cabin 3.

[0102] The stimulus response output unit 53 has an element array in which a plurality of ultrasonic elements 65 are arranged, and outputs a stimulus caused by an ultrasonic field as a response to the operation of the occupant's operating part that performs a non-contact operation on the image projected into the space of the vehicle interior 3, based on the operation information of the occupant's operating part. Thus, when the occupant performs a non-contact operation on the image projected into the space of the vehicle interior 3, a stimulus caused by an ultrasonic field is applied to the operating part that performed the operation as a response to the operation.

[0103] In addition, the operation ECU 20 updates the image in synchronization with the output of the stimulus based on the operation information of the operation part of the same occupant detected by the operation detection device 44. Thus, when the occupant performs a non-contact operation on the image projected onto the space of the vehicle interior 3, the image on which the operation is performed is changed in response to the operation.

[0104] In addition, the operation ECU 20 outputs different sounds according to the movement of the operating part of the occupant and the contact position of the image from the sound output device 46 to the vehicle interior 3 based on the operation information of the operating part of the same occupant detected by the operation detection device 44. As a result, when the occupant performs a non-contact operation on the image projected on the space of the vehicle interior 3, the occupant can hear different sounds according to the operation.

[0105] Through the combined responses of these visual, tactile, and auditory senses, the passenger can obtain a sense of reality as if he or she is actually operating the image projected in the space of the vehicle interior 3. The passenger can operate the image while maintaining the feeling of actual operation.

[0106] Furthermore, the occupant actually operates the operating components in a non-contact manner in the space of the vehicle interior 3, and therefore does not need to directly contact a structure to operate the operating components such as the inner surface of the vehicle interior 3 and the steering wheel 5. In the vehicle interior 3, the occupant does not need to set an operating component that can be directly contacted by the operating part in the space where the operation is actually performed by the operating part. The operability of the occupant can be improved without newly arranging an operating component that is directly operated by the occupant's operating part near the occupant. In addition, by changing the display, it is possible to switch and project an operating panel suitable for the operation of each of the multiple devices.

[0107] In particular, in the present embodiment, the generation unit and the stimulus response output unit 53 update the output of the image and the stimulus according to the operation information of the operation part of the same occupant detected by the operation detection device 44, so that the two are synchronized. For example, when the generation unit updates the image according to the operation, the stimulus response output unit 53 changes the output of the stimulus caused by the ultrasonic field corresponding to the updated image. When the update of the image changes the surface shape or surface position of the image, the stimulus response output unit 53 uses the changed surface shape or surface position of the image as a reference to update the output mode or position of the stimulus caused by the ultrasonic field. Moreover, the image generation unit 51 and the stimulus response output unit 53 update the output of the image and the stimulus according to the operation information of the operation part of the same occupant detected by the detection device, wherein the detection device detects the operation part of the occupant who performs non-contact operation on the image projected into the space of the vehicle room 3. The stimulus can change in accordance with the image itself and the change of the image. For example, the change of the stimulus can correspond to the change of the image. The occupant is less likely to have a sense of disharmony caused by the difference or deviation between the feeling received from the visual sense and the feeling received from the tactile sense. Thus, even after the initial operation of the image, the image and the stimulus are changed to be synchronized according to the operation. Even after the occupant starts to operate the image, he or she can maintain the feeling of actual operation while continuing to operate the image that changes according to the operation. The image and the stimulus change synchronously, so that even if the occupant continues to operate the non-physical operating parts in mid-air, he or she can continue the operation with a sense of reality.

[0108] On the other hand, when the change of the image and the change of the stimulus are not updated synchronously, the occupant can operate the first image realistically, but it is difficult to operate the subsequent operations realistically.

[0109] [Second embodiment]

[0110] Next, a non-contact operation device 40 for a vehicle 1 according to a second embodiment of the present invention will be described. In this embodiment, components corresponding to those in the above embodiment are denoted by the same reference numerals as those in the above embodiment, and description thereof will be omitted.

[0111] Figure 6 1 is a flowchart showing the flow of a main part of the non-contact operation process in the second embodiment. Figure 6 For example, it can be replaced by Figure 5 The processing inside the dotted box on the lower side of node A.

[0112] In this embodiment, the communication device receives the content data, and the non-contact operation device 40 acquires the content data from the external communication ECU 19 through the vehicle network 26 and projects the image. The user operates the image as the content. When the content data is acquired and projected, the operation determination unit 52 of the non-contact operation device 40 Figure 5 In step ST5, there is no need to output the input information generated by the operation to the outside of the device.

[0113] In step ST11, the operation ECU 20 determines whether the passenger operates the image of the content that has started to be projected. The operation ECU 20 obtains detection information such as the image of the stereo camera 63 from the operation detection device 44, extracts the specified parts of the passenger such as the fingertips, and detects the passenger's operation on the image based on the position change and the presence or absence of movement of the specified operation parts such as in the vehicle interior 3. When the passenger's operation part is close to the projection position of the image, the operation ECU 20 will determine that the operation is detected even if the passenger's operation part does not operate the image. When the passenger's operation on the image is not detected, the operation ECU 20 will repeat the judgment process of this step. When the passenger's operation on the image is detected, the operation ECU 20 causes the processing to enter step ST12.

[0114] In step ST12, the operation ECU 20, as the operation determination unit 52, determines the operation of the occupant on the image and generates the determined operation information. The operation ECU 20 determines whether the occupant's operating part operates the image, the contact shape, position and movement (direction, speed), remaining distance, depth, etc. of the operating part as the operation information. The operation ECU 20 outputs the determined operation information to the image generation unit 51 and the stimulus response output unit 53. When necessary, the operation ECU 20 can determine the input information generated by the operation based on the operation information and output it to the outside of the device through the vehicle network 26. The communication device can also obtain the input information generated by the operation and send it to the outside of the car 1.

[0115] In step ST13, the operation ECU 20, as the operation determination unit 52, determines whether the operation of the image by the operating part of the occupant is an operation on the surface of the image or in the vicinity thereof. If it is not an operation on the surface of the image or in the vicinity thereof, the operation ECU 20 returns the process to step S12. If it is an operation on the surface of the image or in the vicinity thereof, the operation ECU 20 proceeds to step S14.

[0116] In step ST14, the operation ECU 20, as the stimulus response output unit 53, selects the type of stimulus to be applied to the operating part of the occupant based on the operation information related to the operation on the surface of the image or its vicinity. The operation ECU 20 selects one stimulus from a plurality of stimuli classified in advance in the memory 47 corresponding to the plurality of operation information based on the current operation information.

[0117] In step ST15, the operation ECU 20 sets the output position and intensity of the stimulation as the stimulation response output unit 53. The operation ECU 20 sets the position of the operating part of the occupant in contact with the image as the output position of the stimulation based on the operation information. The operation ECU 20 sets the output intensity of the stimulation based on the selected stimulation type.

[0118] In step ST16, the operation ECU 20 starts outputting stimulation as the stimulation response output unit 53. The operation ECU 20 selects a plurality of ultrasonic elements 65 that output ultrasonic waves to the position selected in step ST15, and outputs an electrical signal to the stimulation output device 45 to output ultrasonic waves corresponding to the output intensity of the stimulation. The stimulation output device 45 outputs ultrasonic waves at a power corresponding to the output intensity of the stimulation from the selected plurality of ultrasonic elements 65. The selected stimulation can give the occupant a tactile sensation of touching the image to perform an operation.

[0119] In step ST17, the operation ECU 20 determines whether to end the output of the stimulus. For example, when the content instructs to stop the output of the stimulus or the content ends, the operation ECU 20 determines that the output is ended and ends. Figure 6 In other cases, the operation ECU 20 causes the process to proceed to step ST18.

[0120] In step ST18, the operation ECU 20 as the operation determination unit 52 determines the next operation of the occupant on the image based on the new image of the stereo camera 63 of the operation detection device 44 after executing step ST11, and updates the operation information. The operation ECU 20 outputs the determined operation information to the image generation unit 51 and the stimulus response output unit 53.

[0121] In step ST19, the operation ECU 20 acts as the stimulus response output unit 53 to change the content of the stimulus according to the updated operation information. The operation ECU 20 selects the type of stimulus applied to the operating part of the occupant according to the updated operation information. In addition, the operation ECU 20 sets the output position and intensity of the stimulus, selects a plurality of ultrasonic elements 65 that output ultrasonic waves, and outputs an electrical signal to the stimulus output device 45. The stimulus output device 45 outputs ultrasonic waves corresponding to the output intensity of the stimulus to the operating part of the occupant in contact with the image from the selected plurality of ultrasonic elements 65. The occupant can obtain a change in tactile sensation by utilizing the stimulus updated by the operation of the touch image.

[0122] In step ST20, the operation ECU 20 acts as the image generation unit 51 to update the projected image according to the updated operation information. The operation ECU 20 generates a three-dimensional model of the operation according to the content data, and generates image data for projection according to the three-dimensional model of the operation. The operation ECU 20 outputs the updated image data for projection to the 3D image projection device 43. The 3D image projection device 43 projects the image generated according to the updated image data for projection into the space of the vehicle interior 3. As a result, the image of the three-dimensional model projected in front of the occupant is updated to an image that is deformed or moved corresponding to the image operation of the occupant's operating part. After that, the operation ECU 20 returns the processing to step ST17.

[0123] The operation ECU 20 repeats the processing from step ST17 to step ST20 until it is determined in step ST17 that the output is finished. In this repeated processing, the operation ECU 20 repeats the processing of responding to the stimulation of the occupant's operation part and responding based on the image update according to the occupant's operation on the image. Therefore, when the operation ECU 20 updates the image according to the operation, it also changes the output of the stimulation caused by the ultrasonic field corresponding to the updated image.

[0124] Figure 7 Yes means Figure 6 This is a flowchart of the process of stimulus content change processing in step ST19.

[0125] In step ST31, the operation ECU 20 as the stimulus response output unit 53 determines whether the distance between the occupant's operating part and the image surface has changed compared to before the update based on the updated operation information. If the distance to the image surface has not changed, the operation ECU 20 ends. Figure 7 In this case, the stimulus response to the tactile sensation does not change from the state before the operation information is updated. If the distance to the image surface has changed, the operation ECU 20 advances the process to step ST32.

[0126] In step ST32, the operation ECU 20 determines whether the operating part of the occupant whose distance to the image surface has changed is in contact with the image surface based on the updated operation information. If it is not in contact with the image surface in the updated operation information, the operation ECU 20 advances the processing to step ST33. If it is in contact with the image surface in the updated operation information, the operation ECU 20 advances the processing to step ST34.

[0127] In step ST33, since there is no contact with the image surface, the operation ECU 20 maintains the output of the stimulus as the same stimulus. In addition, the operation ECU 20 changes the intensity to an intensity corresponding to the remaining distance or depth in the updated operation information. In this case, the stimulus response is the same stimulus as before the update, and only its intensity increases or decreases according to the operation information. After that, the operation ECU 20 ends. Figure 7 processing.

[0128] In step ST34, the operation ECU 20 determines whether the occupant's operating part has already touched the image in the operation information before the updated operation information. If it has not touched the image, the operation ECU 20 advances the processing to step ST35. If it has already touched the image, the operation ECU 20 advances the processing to step ST36.

[0129] In step ST35, since the occupant's operating part begins to contact the image in the updated operating information, the operating ECU20 changes the stimulation. The operating ECU20 selects the type of stimulation applied to the occupant's operating part based on the updated operating information. In addition, the operating ECU20 sets the output position and intensity of the stimulation, selects multiple ultrasonic elements 65 that output ultrasonic waves, and outputs electrical signals to the stimulation output device 45. The stimulation output device 45 outputs ultrasonic waves corresponding to the output intensity of the stimulation from the selected multiple ultrasonic elements 65 to the occupant's operating part that is in contact with the image. The occupant's body can feel the change in the state of operation from not contacting the image to contacting the image through the stimulation that changes due to the start of contacting the image. The occupant can get the tactile sensation before and after the contact. After that, the operating ECU20 ends Figure 7 processing.

[0130] In step ST36, since the occupant's operating part has already been in contact with the image, the operation ECU 20 continues to output the same stimulus. The operation ECU 20 outputs the same stimulus as last time through the stimulus output device 45. The occupant can obtain a certain tactile sensation regarding the state of contact with the image surface. After that, the operation ECU 20 ends. Figure 7 processing.

[0131] Figure 8 Yes means Figure 6 Flow chart of the process of the image update process in step ST20.

[0132] In step ST41, the operation ECU 20 as the image generation unit 51 determines whether the occupant's operating part is in contact with the image based on the updated operation information. If the occupant's operating part is not in contact with the image, the operation ECU 20 ends the operation. Figure 8In this case, the image does not change from the state before the operation information is updated. In other cases, the operation ECU 20 advances the process to step ST42.

[0133] In step ST42, the operation ECU 20 determines whether the contact position of the image touched by the operating part of the occupant is in the image changeable area based on the updated operation information. If the contact position of the image touched is in the image changeable area, the operation ECU 20 causes the process to proceed to step ST43. If the contact position of the image is not in the image changeable area, the operation ECU 20 causes the process to proceed to step ST44.

[0134] In step ST43, the operation ECU 20 updates the image based on the updated operation information so that the contact position of the image as the image variable area is deformed according to the operation. The 3D image projection device 43 projects the image generated based on the updated image data for projection into the space of the vehicle cabin 3. As a result, the shape of the three-dimensional model image projected in front of the occupant is deformed corresponding to the image operation of the occupant's operating part. The shape of the projected image changes according to the contact operation. After that, the operation ECU 20 causes the processing to enter step ST44.

[0135] In step ST44, the operation ECU 20 determines whether it is necessary to move the position and direction of the image based on the updated operation information. If it is not necessary to move the position and direction of the image, the operation ECU 20 ends. Figure 8 If the position and direction of the image need to be moved, the ECU 20 is operated to advance the process to step ST45.

[0136] In step ST45, the operation ECU 20 updates the image based on the updated operation information to move or rotate the image according to the operation. The 3D image projection device 43 projects the image generated based on the updated projection image data into the space of the vehicle cabin 3. As a result, the position and direction of the three-dimensional model image projected in front of the occupant are updated to correspond to the image operation of the occupant's operating part. The position and direction of the projected image change according to the contact operation. After that, the operation ECU 20 ends. Figure 8 processing.

[0137] Through such processing, the operation ECU 20 can achieve the following visually and tactilely coordinated operation response.

[0138] For example, the operation ECU 20 updates the image according to the contact position of the image operated by the operating part of the occupant, by deformation of the image contact position, movement of the image, and a combination thereof. Furthermore, when the image is updated in a manner that deforms the contact position of the image, the operation ECU 20 can change the output level or output mode of the stimulation caused by the ultrasonic field according to the same operation information as the operation information used for the image response. In addition, when the image is updated in a manner that moves the image, the operation ECU 20 can move the output range or output position of the stimulation caused by the ultrasonic field in the same direction. For example, the operation ECU 20 can move the output of the image and the stimulation along the direction of movement of the operating part of the occupant. For example, the operation ECU 20 can move the output of the image and the stimulation by an amount corresponding to the movement speed of the operating part of the occupant.

[0139] In addition, for example, the operation ECU20 can output different stimuli based on the contact position (area) of the image operated by the operating part of the occupant, which can be determined based on the projection position of the image and the operation information. The operation ECU20 can output different stimuli based on the movement direction and movement speed of the operating part of the occupant, which can be determined based on the operation information. The operation ECU20 can output different stimuli based on the gesture when contacting the image. For example, the operation ECU20 can change the stimulus after the shape change from the stimulus before the shape change in combination with the shape change caused by the image update. In addition, the operation ECU20 can increase or decrease the stimulus after the image update from the stimulus before the image update by updating the image in combination with the change in the amount of contact between the operating part of the occupant and the image.

[0140] Next, a specific example of a combination of an image displayed in the space of the vehicle interior 3 and an operation and a response thereto will be described.

[0141] (Specific example 1)

[0142] The operation ECU 20 projects a 3D image of the dog onto the space above the instrument panel in the vehicle interior 3 using the 3D image projection device.

[0143] The passenger reaches out his hand toward the image above the dashboard and strokes the dog in the 3D image with his palm.

[0144] The operation ECU 20 detects the palm by the operation detection device 44 and determines the position and movement of the palm that touches the dog in the 3D image based on the position of the surface of the dog in the 3D image.

[0145] The operation ECU 20 outputs ultrasonic waves from the stimulation output device 45 to the palm to apply stimulation, and outputs the barking sound of a dog from the sound output device 46 to respond to the operation information related to the determined palm stroking action.

[0146] In addition, the operation ECU 20 changes the projected 3D image of the dog according to the same operation information. The operation ECU 20 updates the image according to the contact position of the image touched by the palm. The operation ECU 20 changes the projected 3D image of the dog to an image that responds to the contact at the contact position. The operation ECU 20 updates the projected 3D image of the dog so that it moves in the operation direction of the palm.

[0147] In addition, in this series of control, the operation ECU 20 can also use the surface of the projected dog as a reference, and when the palm of the passenger approaches within a specified distance, output ultrasonic waves from the stimulation output device 45 to the palm to apply stimulation. In this case, the shorter the distance from the palm of the passenger to the image surface, the stronger the stimulation output from the stimulation output device 45 to the palm by the operation ECU 20. The stimulation can be set to the maximum on the surface of the image.

[0148] In this way, the operation ECU 20 can detect the operation of the passenger's palm on the image, and synchronously update the 3D image of the dog and the response to the tactile stimulation according to the operation. The 3D image and the sound wave field can move synchronously.

[0149] (Specific example 2)

[0150] The operation ECU 20 projects a 3D image of a person into the space of the vehicle interior 3 using the 3D image projection device.

[0151] The passenger reaches out his hand toward the image and presses the belly of the 3D image character with his fingers.

[0152] The operation ECU 20 detects the finger through the operation detection device 44 and determines the position and movement of the finger based on the surface position of the 3D image.

[0153] The operation ECU 20 changes the 3D image of the projected person in response to the operation information related to the determined pressing action of the finger. The operation ECU 20 changes the 3D image to an image in which the contact position of the finger operation is recessed.

[0154] The operation ECU 20 outputs ultrasonic waves from the stimulation output device 45 to the fingertips to apply stimulation based on the same operation information. The operation ECU 20 uses the initial surface position before pressing the 3D image as a reference, and outputs ultrasonic waves that increase according to the penetration depth (the amount of depression of the image) of the front end of the finger from the surface position of the image from the stimulation output device 45. In this way, the tactile sensation of the image being elastically deformed by the finger can be applied to the finger.

[0155] (Concrete example 3)

[0156] The operation ECU 20 projects a 3D image of a person having hard parts and soft parts into the space of the vehicle interior 3 using the 3D image projection device.

[0157] In addition, the property parameters of each part related to the projected image may be stored as one attribute in the polygon data, for example. In addition, the operation ECU 20 may independently determine the property of each part by AI processing or the like based on the position of each part in the 3D model.

[0158] The passenger reaches out his hand toward the image and presses the 3D image character with his fingers.

[0159] The operation ECU 20 detects the finger through the operation detection device 44 and determines the position and movement of the finger based on the surface position of the 3D image.

[0160] The operation ECU 20 changes the 3D image of the projected person in response to the operation information related to the determined pressing action of the finger.

[0161] If the image portion pressed by the finger is a hard portion, the operation ECU 20 moves the image operated by the finger in the operation direction of the finger. The operation ECU 20 moves the image by a movement amount corresponding to the pressing speed and pressing amount of the finger.

[0162] If the image part pressed by the finger is a soft part, the operation ECU 20 depresses the contact position of the finger operation in the operation direction of the finger. The operation ECU 20 deforms the image by a depressing amount corresponding to the pressing speed and pressing amount of the finger.

[0163] The operation ECU 20 outputs ultrasonic waves from the stimulation output device 45 to the fingertips to apply stimulation based on the same operation information.

[0164] If the image part pressed by the finger is a hard part, the operation ECU20 uses the last surface position of the 3D image as a reference, and outputs ultrasonic waves from the stimulation output device 45 only to the front end of the finger, which increases sharply according to the amount of intrusion of the front end of the finger from the surface position of the image.

[0165] If the image part pressed by the finger is a soft part, the operation ECU 20 uses the last surface position of the 3D image as a reference, and outputs ultrasonic waves that slowly increase according to the amount of intrusion of the tip of the finger from the surface position of the image to the entire finger from the stimulation output device 45.

[0166] This allows the occupant to feel the texture of each part of the image through the touch of their fingers.

[0167] For example, when operating a hard part of a projected person, the person deforms by slightly sinking and moving greatly at a relatively fast speed. The operation ECU 20 can output stimulation corresponding to the deformation and movement only to the fingertips of the occupant.

[0168] In addition, when the soft part of the character is operated, the image is deformed to be greatly concave while moving slightly at a slow speed. The operation ECU 20 can output stimulation corresponding to the deformation and movement to the entire finger of the occupant.

[0169] In this way, the deformation and movement of the image are switched according to the passenger's contact position on the 3D image and the movement of the operating part (operation direction, operation speed and operation amount), and then switched to make the tactile stimulation different, thereby enabling the passenger to recognize the difference in texture of each part of the image.

[0170] (Concrete example 4)

[0171] The operation ECU 20 projects a 3D image of a person's head into the space of the vehicle interior 3 using the 3D image projection device.

[0172] In addition, the attributes of the eyes, nose, mouth, ears, cheeks, hair and other parts of the head of the projected image can be stored in polygonal data. In addition, the operation ECU 20 can also independently determine the properties of each part through AI processing based on the position of each part in the 3D model.

[0173] The passenger reaches out his hand toward the image and touches the head of the 3D image with his palm.

[0174] The operation ECU 20 detects the palm through the operation detection device 44, and determines the position and movement of the palm based on the surface position of the 3D image.

[0175] The operation ECU 20 changes the 3D image of the projected person in response to the operation information related to the determined palm movement.

[0176] For example, when the hair is stroked with the palm of the hand, the ECU 20 is operated to update the image so that the image of the stroked hair shakes within the curved surface along the head surface.

[0177] When the cheek is pressed with a finger, the operation ECU 20 updates the image so that the pressed cheek becomes concave.

[0178] The operation ECU 20 outputs ultrasonic waves from the stimulation output device 45 to the palm to apply stimulation according to the same operation information.

[0179] When the passenger touches the hair along the curved surface of the head surface with the palm of the hand, the operation ECU 20 outputs weakly fluctuating ultrasonic waves to the entire palm from the stimulation output device 45. This allows the passenger to feel the smooth touch of the swaying hair.

[0180] When the cheek is pressed with a finger approximately perpendicularly to the surface of the head, the operation ECU 20 outputs a strong ultrasonic wave that changes and increases according to the amount of intrusion from the surface position to the front end of the finger from the stimulation output device 45, based on the initial surface position of the head. The ultrasonic wave can be output to apply a sense of pressure to the entire periphery of the front end of the finger. The ultrasonic wave can be intensified steplessly according to the volume of the finger or fist entering the image. The ultrasonic wave can change according to the change of the image shape. As a result, the occupant can get a tactile feeling that the pressed cheek is shaking and concave.

[0181] In this way, by providing different stimulations depending on the combination of the contact position of the character's head and the contact speed and contact direction of the operation part, the passenger can obtain a tactile sensation as if touching a real head.

[0182] The operation ECU 20 may also obtain a pattern that classifies a plurality of gestures (operation methods) according to each part of the head from the memory 47, and determine the type of operation on each part of the head according to the pattern. In addition, the contact positions such as skin, hair, clothing, and fixed objects may also be classified.

[0183] In addition, the operation ECU 20 can make the ultrasonic wave output period, ultrasonic wave attenuation speed, etc. different according to the gesture such as the operation mode. In this way, the stimulus update mode can be made different in each case of quick contact and long contact. For a strong and short stimulus like a tap, a stimulus with a large increase or decrease in impact can be output in response; at the same time, for a weak and long stimulus like a touch, a stimulus with a slow increase or decrease in output can be output in response.

[0184] In this way, the deformation and movement of the image are switched according to the passenger's contact position with the 3D image and the movement of the operating part (operation direction, operation speed and operation amount), and then switched to make the tactile stimulation different, thereby enabling the passenger to recognize the difference in texture of each part of the image and the content of the operation.

[0185] (Concrete example 5)

[0186] The operation ECU 20 projects a 3D image of the person's arm into the space of the vehicle interior 3 via the 3D image projection device.

[0187] In addition, the properties such as the movable range of the arm of the projected image can be stored in the polygon data, for example. In addition, the operation ECU 20 can also independently determine the properties of each part based on the position of each part in the 3D model through AI processing or the like.

[0188] The passenger reaches out his hand toward the image and moves by holding the image's arm.

[0189] The operation ECU 20 detects the hand of the occupant through the operation detection device 44 and determines the position and movement of the hand based on the surface position of the 3D image.

[0190] The operation ECU 20 changes the 3D image of the projected person in response to the operation information related to the determined hand movement.

[0191] For example, when the arm in the image is held by the hand and moved, the operation ECU 20 updates the image so as to move the image of the arm according to the position and movement of the hand. The operation ECU 20 updates the image so as to move the image of the arm within the movable range.

[0192] The operation ECU 20 outputs ultrasonic waves from the stimulation output device 45 to the hand to apply stimulation based on the same operation information.

[0193] When the arm moves within the movable range, the operation ECU 20 outputs ultrasonic waves corresponding to the weight of the arm and the reaction force pressing the arm to the hand, so that the occupant can feel as if holding an arm with weight or an arm with reaction force.

[0194] When the arm moves beyond the movable range, the operation ECU 20 outputs ultrasonic waves larger than those output within the movable range to the hand, thereby allowing the occupant to feel a strong reaction force when the arm is about to move beyond the movable range.

[0195] In this way, the deformation and movement of the image are switched according to the passenger's contact position with the 3D image and the movement of the operating part (operation direction, operation speed and operation amount), and then switched to make the tactile stimulation different, thereby enabling the passenger to recognize the difference in texture of each part of the image and the content of the operation.

[0196] (Concrete example 6)

[0197] The operation ECU 20 projects a 3D image of a character such as a person or an animal that can be sunken when pressed by a finger, or a structure such as a switch that does not sunken even when pressed by a finger, into the space of the vehicle interior 3 using a 3D image projection device.

[0198] In addition, the attributes of the types of people and structures of the projected image may be stored in polygon data, for example. In addition, the operation ECU 20 may independently determine the type based on the 3D model by AI processing or the like.

[0199] The occupant reaches out toward the image and presses it with his fingers.

[0200] The operation ECU 20 detects the passenger's finger through the operation detection device 44 and determines the position and movement of the finger based on the surface position of the 3D image.

[0201] The operation ECU 20 changes the 3D image of the projected person according to the type of the displayed image, in response to the operation information related to the determined finger movement.

[0202] For example, when displaying a person, the operation ECU 20 updates the image according to the position and movement of the finger so that the 3D image of the person is recessed at the operation site.

[0203] When displaying a structure such as a switch, the operation ECU 20 updates the image so that the 3D image of the structure does not sink at the operation site due to the position and movement of the finger. The operation ECU 20 updates the image to change the color of the switch, etc. In this case, the operating fingertip will enter the inner side of the 3D image surface. The operation ECU 20 can also make the color of the switch fade and become translucent when the fingertip enters the inner side of the image, and display the skeleton of the internal structure of the structure. The operation ECU 20 can also move the 3D image of the structure according to the movement of the finger.

[0204] The operation ECU 20 outputs ultrasonic waves from the stimulation output device 45 to the finger to apply stimulation according to the same operation information.

[0205] For example, when a person is displayed, the operation ECU 20 outputs ultrasonic waves of intensity corresponding to the amount of penetration of the finger to the vicinity of the fingertip, based on the initial projection position of the person.

[0206] When a structure such as a switch is displayed, the operation ECU 20 outputs ultrasonic waves of a certain intensity to the fingertip.

[0207] By switching the deformation of the image according to the object projected by the 3D image in this way, and further switching to different stimulations to the tactile sense, the occupant can recognize the difference in the operational feeling corresponding to the type of the projected image.

[0208] As described above, in the embodiment of the present invention, when the passenger operates the image projected into the space of the vehicle interior 3, the projected image is updated based on the operation information, ultrasonic waves are output to the operation part, and sound is output. As a result, the passenger who has performed non-contact operation on the image projected into the space of the vehicle interior 3 can receive the response corresponding to the operation content through vision, touch, and hearing, thereby obtaining a realistic operation feeling.

[0209] For example, for a passenger lying comfortably in the vehicle cabin 3 as shown in FIG1(C), the non-contact operation device 40 can be applied not only to providing a good user interface based on the image of the operation component, but also to providing highly entertaining content. By receiving different responses according to the operation content, the passenger and other users can maintain the operational reality of each operation and can perform the operation continuously.

[0210] The above-described embodiment is an example of a preferred embodiment of the present invention, but the present invention is not limited thereto, and various modifications and changes can be made without departing from the gist of the present invention.

[0211] For example, in the above-described embodiment, the contactless operation device 40 is provided in a vehicle such as the automobile 1 .

[0212] In addition, for example, the contactless operating device 40 can also be set in furniture and home appliances used by the user. In addition, the contactless operating device 40 can be set separately from the vehicle, furniture or home appliances and set in the vehicle, furniture or home appliances by the user.

Claims

1. A non-contact operation device for a vehicle, comprising: A projection device that projects an image into a space in a cabin of the vehicle; a generating unit, which generates and updates the image projected by the projection device; a stimulus response output unit that outputs a tactile stimulus caused by an ultrasonic field as a response to an operation performed by an occupant of the vehicle who operates the image projected in the space of the vehicle compartment; as well as a detection device for detecting the operation part of the occupant who performs a non-contact operation on the image projected in the space of the vehicle cabin, The generating unit and the stimulus response output unit update the output of the image and the tactile stimulus according to the operation information of the operation part of the occupant detected by the detection device. The stimulus response output unit outputs different tactile stimuli according to the contact position operated by the operating part of the occupant with respect to the image, the movement direction of the operating part of the occupant, and the movement speed of the operating part of the occupant, When the generation unit and the stimulation response output unit update the output of the image and the tactile stimulation according to the operation information of the operation part of the occupant detected by the detection device, determining whether the distance from the operating part of the occupant to the surface of the image has changed, and if the distance to the surface of the image has changed, further determining whether the operating part of the occupant is in contact with the surface of the image, and whether the operating part of the occupant has already been in contact with the image, If the operating part of the occupant is not in contact with the surface of the image, the output of the stimulation is maintained while changing the intensity to correspond to the remaining distance or depth to the surface of the image. If the operating part of the occupant is in contact with the surface of the image but has not been in contact with the image for a long time, the type of stimulation is changed, If the operating part of the occupant is in contact with the surface of the image and has already been in contact with the image, the same output is continued.

2. The non-contact operation device for a vehicle according to claim 1, wherein: The stimulation response output unit changes the output of the tactile stimulation caused by the ultrasonic field in accordance with the updated image when the generation unit updates the image according to the operation.

3. The non-contact operation device for a vehicle according to claim 1 or 2, wherein: The generating unit updates the image by deformation of the contact position of the image, movement of the image, or a combination thereof, according to the contact position of the image operated by the operating part of the occupant. When the generating unit updates the image in such a way that the contact position of the image is deformed, the stimulus response output unit changes the output of the tactile stimulation caused by the ultrasonic field; when the generating unit updates the image in such a way that the image is moved, the stimulus response output unit moves the output of the tactile stimulation caused by the ultrasonic field in the same direction.

4. The non-contact operation device for a vehicle according to claim 3, wherein: The generation unit and the stimulation response output unit move the output of the image and the tactile stimulation along the movement direction of the operation part of the occupant.

5. The non-contact operation device for a vehicle according to claim 3, wherein: The generation unit and the stimulation response output unit shift the output of the image and the tactile stimulation by an amount corresponding to the movement speed of the operation part of the occupant.

6. The non-contact operation device for a vehicle according to claim 1 or 2, wherein: The stimulus response output unit changes the tactile stimulation after the shape change from the tactile stimulation before the shape change according to the shape change of the image caused by the update of the generator.

7. The non-contact operation device for a vehicle according to claim 1 or 2, wherein: The stimulus response output unit updates the image by the generating unit, and increases or decreases the tactile stimulation after the image is updated compared with the tactile stimulation before the image is updated, according to the change in the contact amount between the operating part of the occupant and the image.

8. The non-contact operation device for a vehicle according to claim 1 or 2, comprising: a sound output device that outputs sound to the vehicle interior; and A sound response output unit outputs, from the sound output device, a sound that varies according to at least one of a movement of an operating part of an occupant with respect to an image projected onto the vehicle interior and a contact position of the image.

9. A vehicle comprising: The non-contact operating device according to any one of claims 1 to 8; A controlled device, which is connected to the contactless operating device via a network to control the vehicle; as well as a communication device connected to the contactless operation device via a network and communicating with the outside of the vehicle, The non-contact operation device projects the content received by the communication device as an image.

Images