User interface device, Vehicle having the user interface device and method for controlling the vehicle

Abstract

The present invention discloses a user interface device, a vehicle having the same, and a method for controlling the same. The user interface device of the present invention includes an image acquisition unit, a haptic unit, and an image output unit. A vehicle having a user interface device outputs an image of a reference point through an image output unit, generates a circle based on the position information of the reference point, acquires an image of a hand within the generated circle, recognizes a gesture based on the acquired image of the hand, recognizes a user command corresponding to the recognized gesture, controls the operation of a steering device based on the recognized user command, and controls the operation of a haptic unit in response to the recognition of the gesture.

Description

Technology Field

[0001] The present invention relates to a user interface device for receiving driving commands of a vehicle and user commands of various devices, a vehicle having the same, and a method for controlling the same. Background Technology

[0002] In addition to basic driving functions, it includes additional features for user convenience, such as audio, video, navigation, climate control, seat control, and lighting control.

[0003] Such a vehicle further includes a user interface device for inputting operation commands for various functions and outputting the operation status of various functions, and this user interface device enables interaction between the user and various devices provided in the vehicle.

[0004] User interface devices include physical user interface (PUI) devices and graphical user interface (GUI) devices.

[0005] A physical user interface device (PUI) is a device that receives user commands through physical means, such as keypads, remote controls, and touchpads, while a graphical user interface device (GUI) is a device that receives user commands by selecting icons or menus displayed on a screen.

[0006] In other words, the user can move the cursor by referring to menus, lists, icons, etc. displayed through a graphical user interface device and select the item where the cursor is located; at this time, the user can select the item they wish to select by moving the cursor through a physical user interface device.

[0007] Physical user interface devices can be designed to be large in size depending on the target of operation or the operation function.

[0008] When such physical user interface devices were placed in vehicles with limited internal space, they occupied a large amount of space inside the vehicle. The problem to be solved

[0009] One aspect provides a user interface device that outputs a tactile signal based on an image of a hand and outputs an image of the operation of an electronic device, a vehicle having the same, and a method for controlling the same.

[0010] Another aspect provides a vehicle and a control method thereof that acquires a steering command based on an image of a hand and displays a holographic image or a forward display image corresponding to the acquired steering command. means of solving the problem

[0011] A user interface device according to one aspect comprises: an image acquisition unit for acquiring an image of a user; an image output unit for outputting an image of a control system; a control unit for recognizing a gesture based on the acquired image, recognizing a user command corresponding to the recognized gesture, and transmitting the recognized user command to at least one electronic device; and a haptic unit for outputting a tactile signal for gesture recognition in response to a control command of the control unit.

[0012] A control unit of a user interface device according to one aspect controls an image output unit to output an image of a reference point, and if it is determined that an image of a user acquired by an image acquisition unit is an image of touching the output image of a reference point, the control unit controls the image output unit to change the image of the reference point into an image of a control system.

[0013] According to one aspect, the image output unit of the user interface device outputs a holographic image, and the control unit sets the output position of a tactile signal output through the haptic unit based on the location information where the holographic image is displayed.

[0014] A control unit of a user interface device according to one aspect controls an image output unit to output an image of a reference point, generates a circle image based on position information of the reference point and a preset radius, controls the image output unit to output the generated circle image, acquires an area of ​​a palm within a circle image based on an image of a palm within a circle image acquired through an image acquisition unit, and recognizes a gesture based on the area of ​​the circle image and the area of ​​the palm.

[0015] A haptic portion of a user interface device according to one aspect includes an array of multiple piezoelectric elements that output an ultrasonic signal.

[0016] A control unit of a user interface device according to one aspect recognizes the number of fingers based on an image of a finger among the images of a user acquired through an image acquisition unit, and recognizes the recognized number of fingers as a gesture.

[0017] A control unit of a user interface device according to one aspect controls an image output unit to output an image of a reference point, outputs a circular image based on position information of the reference point and a preset radius, recognizes the number of fingers crossing the edge of the circular image based on the number of brightness contrasts of the edge of the circular image acquired by an image acquisition unit, and recognizes the recognized number of fingers as a gesture.

[0018] A vehicle according to another aspect includes: a steering device; an image acquisition unit for acquiring an image of a user; an image output unit for outputting an image of a control system; a control unit for recognizing a gesture based on the acquired image, recognizing a user command corresponding to the recognized gesture, and controlling the operation of the steering device based on the recognized user command; and a haptic unit for outputting a tactile signal for gesture recognition in response to the control command of the control unit.

[0019] A vehicle according to another aspect further includes a seat adjustment unit for adjusting the position of the seat, and a control unit controls the operation of an image output unit to output an image of a reference point, and controls the operation of the seat adjustment unit based on an image of the hand and an image of the reference point among the images of the user acquired by an image acquisition unit.

[0020] According to another aspect, the image output unit of the vehicle outputs a holographic image, and the control unit controls the image output unit to output an image of a reference point, and if it is determined that the image of a user acquired by the image acquisition unit is an image of touching the outputted image of the reference point, the image output unit controls the image output unit to change the image of the reference point into a holographic control system image.

[0021] According to another aspect, the image output unit of the vehicle includes a forward display unit, and the control unit sets the output position of the tactile signal output from the haptic unit based on the image of the user acquired by the image acquisition unit.

[0022] A vehicle according to another aspect further comprises an input unit; and at least one of a sound output unit and a display unit, and the control unit controls the operation of the image output unit to output an image of a reference point, controls the operation of at least one of the sound output unit and the display unit to output guidance information for seat adjustment, and controls the operation of the seat to adjust the position of the seat based on a seat adjustment command received by the input unit.

[0023] A vehicle according to another aspect further includes an input unit that receives setting information of at least one of an image output unit and a haptic unit, and the setting information includes at least one of a type of control system image, a size of a control system image, a shape of a control system image, location information where a control system image is displayed, usage information of a control system, and location information where a tactile signal of a haptic unit is output, and the control unit controls at least one of the image output unit and the haptic unit based on at least one setting information input to the input unit.

[0024] According to another aspect, the haptic part of the vehicle includes a plurality of piezoelectric element arrays that output ultrasonic signals and an ultrasonic speaker, and a plurality of pressure element arrays output a tactile signal as vibration pressure corresponding to an input frequency corresponding to a control command of the control unit, and the ultrasonic speaker outputs a tactile signal as sound corresponding to a control command of the control unit.

[0025] The haptic part of the vehicle according to another aspect includes an array of multiple piezoelectric elements that output an ultrasonic signal.

[0026] The control unit of the vehicle according to another aspect controls the image output unit to output an image of a reference point, generates a circular image based on the position information of the reference point and a preset radius, controls the image output unit to output the generated circular image, acquires the area of ​​a palm within a circular shape based on the image acquired through the image acquisition unit, and recognizes a gesture based on the area of ​​the generated circle and the area of ​​the palm.

[0027] The control unit of the vehicle according to another aspect controls the image output unit to output an image of a reference point, generates a circular image based on the position information of the reference point and a preset radius, controls the image output unit to output the generated circular image, recognizes the number of fingers crossing the edge of the circular image based on the number of contrast values ​​of the edge of the circular image acquired by the image acquisition unit, and recognizes the recognized number of fingers as a gesture.

[0028] A vehicle control method according to another aspect outputs an image of a reference point through an image output unit, generates a circle based on the position information of the reference point, acquires an image of a hand within the generated circle, recognizes a gesture based on the acquired image of the hand, recognizes a user command corresponding to the recognized gesture, controls the operation of a steering device based on the recognized user command, and controls the operation of a haptic unit in response to the recognition of the gesture.

[0029] A vehicle control method according to another aspect further includes controlling parking, reverse, forward, and neutral movements based on recognized user commands.

[0030] Controlling the operation of the haptic part includes outputting an ultrasonic signal. Effects of the invention

[0031] The present invention allows for the removal of the steering wheel, which was an essential component of the vehicle, thereby reducing the cost of the vehicle, decreasing the weight of the vehicle, and improving the space utilization of the driver's seat area, which can increase the freedom of interior design.

[0032] The present invention enables the design of ergonomic next-generation mobility by incorporating sensory technologies such as sight, hearing, and touch into a vehicle, and can also appeal to a sense of luxury.

[0033] The present invention can provide a new experience to the user by fusing a virtual image, such as a holographic image, with haptic feedback, and can create different values ​​in a limited space with an unconventional user interface device.

[0034] As such, the present invention can improve the quality and marketability of user interface devices and vehicles, further increase user satisfaction, and secure product competitiveness. Brief explanation of the drawing

[0035] FIG. 1 is an example of the interior of a vehicle having a user interface device according to an embodiment, and is an example of the interior viewed from the rear seat of the vehicle. FIGS. 2 and FIGS. 3 are interior example views of a vehicle having a user interface device according to an embodiment, and are interior example views viewed from the passenger seat of the vehicle. FIG. 4 is an example diagram of a haptic part of a user interface device according to an embodiment. FIG. 5 is an example diagram of the operation of a haptic part of a user interface device according to an embodiment. FIG. 6 is an example diagram of an image output section of a user interface device according to an embodiment. FIG. 7 is an example diagram of a front display unit among the image output units of a user interface device according to an embodiment. FIG. 8 is an example diagram of a user interface device according to an embodiment. FIG. 9 is a control configuration diagram of a vehicle having a user interface device according to an embodiment. FIG. 10 is an example of gesture recognition in a vehicle having a user interface device according to an embodiment. FIG. 11 is a control flowchart of a vehicle having a user interface device according to an embodiment. Specific details for implementing the invention

[0036] The present invention will be described in detail below with reference to the attached drawings.

[0037] Throughout the specification, the same reference numerals refer to the same components. This specification does not describe all elements of the embodiments, and general content in the art to which the invention pertains or content that overlaps between embodiments is omitted. The terms 'part, module, device' used in the specification may be implemented in software or hardware, and depending on the embodiments, a plurality of 'parts, modules, devices' may be implemented as a single component, or a single 'part, module, device' may include a plurality of components.

[0038] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0039] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0040] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0041] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0042] In each step, identification codes are used for convenience of explanation and do not describe the order of the steps; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0043] The operating principle and embodiments of the present invention will be described below with reference to the attached drawings.

[0044] FIG. 1 is an example interior view of a vehicle according to an embodiment, which is an example interior view seen from the rear seat of the vehicle, and FIG. 2 is an example interior view of a vehicle according to an embodiment, which is an example interior view seen from the passenger seat of the vehicle.

[0045] The vehicle (1) includes a body having an interior and an exterior, and a chassis on which mechanical devices necessary for driving are installed in the remaining parts excluding the body.

[0046] As illustrated in FIG. 1, the interior of the vehicle body consists of a seat (101) on which a passenger sits, a dashboard (102), a cluster such as a speedometer, fuel gauge, automatic transmission selector lever indicator, tachometer, and trip meter that is placed on the dashboard and outputs driving-related information, and a center fascia (103) having an audio and air conditioning control panel.

[0047] Here, the seat (101) includes a driver's seat (101a) where the driver sits, a passenger seat (101b) where a passenger sits, and a rear seat located at the rear of the vehicle.

[0048] The center fascia (103) is located between the driver's seat (101a) and the passenger seat (101b) of the dashboard (102) and is a part where a head unit (104) is provided, and may be equipped with an audio device, a control unit for adjusting the air conditioner and heater, an air vent, a cigarette lighter socket, etc.

[0049] The head unit (104) may be provided with an input unit (105) for receiving user commands and a display unit (106) for displaying operation information for various functions.

[0050] The input unit (105) can receive on commands, off commands, pause commands, etc. of various functions. The input unit (105) may also include a button, key, switch, pedal, lever, jog dial (not shown), or touch pad.

[0051] The input unit (105) can receive a command to select a driving mode, such as an autonomous driving mode or a manual driving mode. The display unit (106) can display operation information of various functions performed in the vehicle.

[0052] The display unit (106) can display manual driving mode and autonomous driving mode.

[0053] The display unit (106) may be provided as a display panel such as an LCD (Liquid Crystal Display), PDP (Plasma Display Panel), or OLED (Organic Light Emitting Diode), or as a touch screen.

[0054] The vehicle chassis includes a power generation unit, power transmission unit, driving unit, steering unit, braking unit, suspension unit, transmission unit, fuel unit, front, rear, left, and right wheels, etc.

[0055] The vehicle may further include a brake pedal for transmitting user commands to a braking device, and an accelerator pedal for transmitting user commands to a power generation device and a power transmission device.

[0056] Vehicles are equipped with various safety and convenience features for the safety of the driver and passengers.

[0057] Examples of convenience devices include hands-free devices, GPS, audio devices and Bluetooth devices, and rear cameras. Additionally, examples of convenience devices may include autonomous driving control devices.

[0058] There may be a vehicle terminal (107) that performs audio, video, and navigation functions as a convenience device.

[0059] The vehicle terminal (107) can display menu items such as navigation, DMB (broadcast output function), radio, call, media, and options, and can display a video of the function selected by the user.

[0060] When performing a navigation function, the vehicle terminal (107) can display at least one of map information, route information, current location information, driving speed information, destination information, and traffic information as an image. The vehicle terminal (107) can display driving time, time remaining to the destination, current time, etc.

[0061] The vehicle terminal (107) may include a graphical user interface (GUI) device and may also include a physical user interface (PUI) device.

[0062] As illustrated in FIGS. 1 and 2, the vehicle (1) further includes a user interface device (200) for inputting and outputting various information.

[0063] The user interface device (200) can receive user commands corresponding to steering operations of the vehicle.

[0064] The user interface device (200) may receive user commands for the operation of the vehicle's audio, air conditioner, vehicle terminal, etc.

[0065] The user interface device (200) can be provided in the activity area and field of view of the driver's hand.

[0066] The user interface device (200) may include an image acquisition unit (210) for acquiring user commands, a haptic unit (220) for outputting a feedback signal for the acquired user commands, and an image output unit (230) for outputting operation information for the acquired user commands.

[0067] The image acquisition unit (210) may be provided within the activity area of ​​the user's hand and adjacent to the haptic unit (220).

[0068] The image acquisition unit (210) may be provided at the end of the armrest (108) or on the center fascia (103) and adjacent to the gear lever.

[0069] The user can input user commands for operating the steering wheel by making a gesture with their hand (H) while resting their arm on the armrest (108). At this time, the image acquisition unit (210) can acquire an image of the user's hand.

[0070] The image acquisition unit (210) is a camera and may include a CCD or CMOS image sensor.

[0071] The image acquisition unit (210) may be an infrared camera.

[0072] The image acquisition unit (210) is a device that detects object information and converts it into an electrical image signal, and transmits the image signal for the acquired image information to the control unit ().

[0073] The haptic part (220) can be provided between the driver's seat (101a) and the passenger seat (101b).

[0074] The haptic part (220) can be placed within the activity area of ​​the user's hand.

[0075] The haptic unit (220) can be provided in the center fascia (103), which is the central area of ​​the dashboard (102).

[0076] The haptic part (220) may be provided in a form that extends from one end of the arm rest (108).

[0077] The surface of the haptic part (220) may be at the same height as the surface of the armrest (108), or may be at a certain height lower than the surface of the armrest.

[0078] As illustrated in FIG. 3, a part of the armrest (108) provided in the vehicle and the image acquisition unit (210) and haptic unit (220) of the user interface device (200) may be provided to be movable up and down. Additionally, the image acquisition unit (210) may be provided in the haptic unit (220), but the camera's field of view may be directed upward (i.e., toward the headlining of the vehicle).

[0079] The haptic unit (220) generates a tactile signal that the user can feel with their body.

[0080] The haptic unit (220) may include a vibration generating unit.

[0081] The haptic unit (220) may include an ultrasonic generating unit.

[0082] The haptic unit (220) may include a plurality of piezoelectric element arrays that output ultrasonic signals and an ultrasonic speaker that outputs ultrasonic sound.

[0083] The haptic part (220) can be provided in the shape of a small steering wheel, a wheel mouse, a spherical shape, or a flat shape.

[0084] As illustrated in FIG. 4, the haptic unit (220) may include a body (221) and a plurality of piezoelectric elements (222) disposed in the body and generating vibrations so that the user can feel the vibrations tactilely. Here, the plurality of piezoelectric elements (222) may generate ultrasonic waves and may be arranged in an array form.

[0085] For example, the haptic part (220) may include a substrate on which a plurality of pixels are arranged, a plurality of thin-film transistors arranged on the substrate and each of which is arranged in each of the plurality of pixels, a planarization layer arranged on the thin-film transistors and having a plurality of contact holes, a pixel electrode arranged on the planarization layer and each of which is arranged in each of the plurality of pixels, a piezoelectric material arranged on the pixel electrode; and a common electrode arranged on the piezoelectric material.

[0086] The haptic unit (220) can generate ultrasonic waves by vibrating a piezoelectric element by a voltage applied to a pixel electrode and a common electrode placed in a thin-film transistor array.

[0087] As shown in FIG. 5, a plurality of piezoelectric element arrays generate ultrasonic waves with a vibration pressure corresponding to an input frequency corresponding to a control command of the control unit.

[0088] The haptic unit may further include an ultrasonic speaker, and the ultrasonic speaker may output an ultrasonic sound corresponding to a control command of the control unit to provide user feedback.

[0089] Depending on the frequency of the ultrasonic signals generated by each piezoelectric element and ultrasonic speaker, the user receives feedback through vibration and sound via touch and hearing. User feedback can be adjusted according to the driver's settings.

[0090] Users can receive not only holographic images but also vibration pressure and sound simultaneously as feedback on steering movements.

[0091] In addition, users can receive holographic images and vibration pressure as feedback on steering movements.

[0092] Users can also receive holographic images and sound simultaneously as feedback on steering movements.

[0093] The video output unit (230) can be provided within the user's field of view and can also be provided adjacent to the haptic unit (220).

[0094] The image output unit (230) can display steering information corresponding to a user command obtained from the image acquisition unit (210).

[0095] The video output unit (230) can display a video corresponding to the operation of the steering wheel corresponding to the steering information.

[0096] As illustrated in FIG. 2, the image output unit (230) may include a holographic image output unit that displays an image corresponding to the operation of the steering wheel as a holographic image.

[0097] When the image output unit (230) is a holographic image output unit, the configuration of the image output unit (230) is explained with reference to FIG. 6.

[0098] As illustrated in FIG. 6, the image output unit (230) includes an image projection unit (231) that projects image light onto a steering wheel, a reflection unit (232) that reflects the image light projected from the image projection unit (231), and a transparent screen unit (233) that transmits the image light reflected from the reflection unit (232) so that the transmitted image light forms a holographic image.

[0099] The image projection unit (231) is a projector and may include a holographic projector for generating a three-dimensional (3D) holographic image. This image projection unit (231) outputs a three-dimensional image of the steering wheel as image light.

[0100] A holographic image of the steering wheel can be formed on one side of this transparent screen part (233).

[0101] The image projection unit (231) may be provided in the area of ​​the center fascia around the head unit or inside the center fascia.

[0102] The reflective portion (232) may be provided to protrude from the center fascia area, around the head unit, or from the inside of the center fascia to the outside. This reflective portion (232) may be provided adjacent to the image projection portion (231) so as to receive image light from the image projection portion (231) provided inside the center fascia.

[0103] The transparent screen part (233) transmits the received image light so that a holographic image is formed in the direction of ultrasonic generation of the haptic part.

[0104] The transparent screen part can be omitted here.

[0105] That is, the user can see a holographic image formed on one side of the transparent screen part (233). In addition, the transparent screen and the reflective part are not visible to the user's eyes.

[0106] As illustrated in FIG. 7, the image output unit (230) may include a forward-viewing image output unit that displays an image corresponding to the operation of the steering wheel as a projected image on the front windshield. That is, the image output unit (230) may be a head-up display (HUD).

[0107] The video output unit (230) may include a floating display provided on the dashboard.

[0108] The video output unit (230) may include a display unit of the vehicle terminal.

[0109] When the image output unit (230) is a holographic image output unit, the user interface device (200) can be provided as shown in FIG. 8.

[0110] An image output unit (230) may be provided on one side of the image acquisition unit (210).

[0111] A haptic unit (220) may be provided at a location adjacent to the image output unit (230) so that ultrasound can reach the location of the holographic image formed by the image output unit (230).

[0112] The position information of the haptic unit (220) can be determined based on the image formation position information of the image output unit (230).

[0113] When the use of the user interface device (200) is unnecessary, such as when performing autonomous driving mode, the user interface device may be turned off. At this time, the vehicle may provide a sense of unity between the user interface device (200) and the armrest (108).

[0114] FIG. 9 is a control configuration diagram of a vehicle having a user interface device according to an embodiment.

[0115] The vehicle may further include an input unit (105), a display unit (106), a control unit (110), a storage unit (111), a communication unit (120), a sound output unit (130), a steering drive unit (140), a lamp drive unit (150), a seat adjustment unit (160), and a user interface device (200).

[0116] The input unit (105) can receive user identification information. For example, the user identification information may be a number, an alphabet, or the user's name.

[0117] The input unit (105) can receive location information of the sheet.

[0118] Here, the seat position information may include information regarding the front-rear and up-down positions of the driver's seat.

[0119] The input unit (105) receives user input regarding the settings of the user interface device (200).

[0120] The settings of the user interface device may include the size of the holographic image, the shape (i.e., type) of the holographic image, the intended use of the user interface device using the holographic image, image location information regarding the location where the holographic image is displayed, image settings by use, and tactile location information regarding the output location of the tactile signal output from the haptic unit.

[0121] The settings of the user interface device can be configured on a per-user basis.

[0122] In addition, the position information regarding the output location of the tactile signal output from the haptic unit can be automatically set according to the position information regarding the location where the image of the hologram is displayed.

[0123] The control system image types for the steering control system shape may include button types such as up and down buttons, steering wheel types, wheel mouse types, sphere types, and flat types.

[0124] The input unit (105) can receive location information of the tactile sensation that can be felt.

[0125] The input unit (105) can receive information regarding the use of the user interface device. Here, the use of the user interface device may include selecting a function of the vehicle and selecting the operation of an electronic device within the vehicle.

[0126] For example, the uses of the user interface device may include steering operation, character input, operation of the gear lever (P, N, R, D), volume operation, etc.

[0127] Video settings by purpose may include steering control system videos, volume button videos, etc.

[0128] The input unit (105) can receive driving commands for the vehicle.

[0129] Here, the driving command includes a steering command in a first direction, a steering command in a second direction different from the first direction, an acceleration command, and a deceleration command.

[0130] A steering command to the first direction may include a move command to the left, and a steering command to the second direction may include a move command to the right.

[0131] The acceleration command includes a speed increase command when driving forward, and the deceleration command includes a speed decrease command when driving forward.

[0132] The display unit (106) can display information about the user command received in the input unit as an image.

[0133] The display unit (106) can display steering information about the steering action input through the user interface device as an image.

[0134] The control unit (110) controls various functions performed in the vehicle in response to user commands input into the input unit (105).

[0135] The control unit (110) can change the settings of the user interface device in response to a user command input into the input unit (105).

[0136] The control unit (110) can control the display unit to display information corresponding to a user command entered into the input unit (105) through the display unit.

[0137] When the control unit (110) receives an image signal from the image acquisition unit (210), it performs clearing processing and noise removal processing on the received image signal to generate image information, and applies vision technologies that decompose objects in the image information to extract hand gestures as results from the image information.

[0138] The control unit (110) can acquire steering information corresponding to a hand gesture and control the operation of the steering device based on the acquired steering information.

[0139] The control unit (110) can control the activation of the user interface device (200) when a vehicle start signal is received.

[0140] The control unit (110) can output an image of a reference point through the image output unit (230) when the user interface device (200) is activated, and can automatically adjust the position of the sheet based on the image of the user's hand acquired through the image acquisition unit (210).

[0141] The control unit (110) can adjust the vertical height and forward / backward distance of the sheet based on the hand position where the user's hand is located and the output position of the point.

[0142] In addition, the control unit (110) can output an image of a reference point through the image output unit (230) when the user interface device (200) is activated, and can also adjust the position of the sheet based on the received sheet adjustment command when a sheet adjustment command is received through the input unit (105).

[0143] The control unit (110) may also control the sound output unit to output seat adjustment guide information so that the position of the seat can be adjusted based on the position information where the holographic image is output, and may also control the display unit.

[0144] The control unit (110) can control the image output unit (230) to change the reference point image into a steering control system image when the image of the hand acquired by the image acquisition unit (210) is recognized as a touch gesture touching the reference point image.

[0145] Here, the steering control system video may include a steering wheel video, a button video, etc.

[0146] When the control unit (110) determines that the seat adjustment is complete, it recognizes the movement of the user's hand through the image acquisition unit (210) and can recognize a hand gesture based on the recognized hand movement. This will be explained in more detail with reference to FIG. 10.

[0147] The control unit (110) sets the location information of the reference point image as reference location information and sets the coordinate value for the reference location information as reference coordinate value.

[0148] As illustrated in FIG. 10 (a), the control unit (110) sets the radius (M) of the image acquired by the image acquisition unit (210) based on the reference coordinate value (RP) and generates a circle based on the set radius. Here, the set radius may be a preset radius.

[0149] The reference coordinate value (RP) is (0.0), which can be the location information of the reference point.

[0150] The image acquired by the image acquisition unit (210) based on the reference coordinate value (RP) may be an image of the user's hand based on the reference point.

[0151] The control unit (110) can control the image output unit to output an image of the generated circle. Here, the image of the circle can be output as an infrared circle image.

[0152] The control unit (110) recognizes a gesture for an image within a generated circle.

[0153] As illustrated in Fig. 10(b), the control unit (110) can identify the palm area occupied by the palm of the entire area of ​​the generated circle and recognize a gesture based on the identified palm area.

[0154] A < Palm area / Total area <B, A와 B는 0 내지 1 범위이다.

[0155] As illustrated in FIG. 10 (c), the control unit (110) checks the number of fingers crossing the edge of a circle created around a reference point and recognizes a gesture based on the number of fingers identified. Here, the fingers may be a number of contrast ratios.

[0156] C < number of fingers (or contrast ratio) <D, C와 D는 0 내지 6이다.

[0157] In addition, the control unit can also recognize operation commands for the gear shift lever based on the recognized gesture. For example, if the recognized gesture corresponds to a single finger touching the edge of a circle, the operation command for the gear shift lever is recognized as a parking command; if the recognized gesture corresponds to two fingers touching the edge of a circle, the operation command for the gear shift lever is recognized as a reverse command; if the recognized gesture corresponds to three fingers touching the edge of a circle, the operation command for the gear shift lever is recognized as a neutral command; and if the recognized gesture corresponds to four fingers touching the edge of a circle, the operation command for the gear shift lever is recognized as a forward command.

[0158] The control unit (110) can also recognize steering commands based on the area of ​​the palm within the generated circle and the number of fingers crossing the edge of the generated circle.

[0159] The control unit (110) can control the operation of the haptic unit (220) in response to gesture recognition. When controlling the operation of the haptic unit (220), the control unit (110) can control the generation of ultrasonic waves for each piezoelectric element array in response to gesture recognition. Accordingly, the user can be made to feel a tactile sensation in the air by means of multiple ultrasonic signals output from the piezoelectric element array.

[0160] This embodiment can make gesture control in the air more realistic by using tactile feedback from a plurality of piezoelectric element arrays.

[0161] That is, while the user presses a steering control button formed in the air as a holographic image, tactile feedback is provided by ultrasonic haptics, and the user can control the steering of the vehicle.

[0162] The control unit (110) can determine a steering command corresponding to a recognized gesture and control the operation of the steering device based on the determined steering command.

[0163] When the video output unit is a front display unit, the control unit (110) can control the output of the image of the steering wheel corresponding to the steering motion so that the image of the steering wheel is projected onto the front window glass. At this time, the user can see the image of the steering wheel turning through the front window glass.

[0164] Accordingly, the image of the reference point output through the image output unit (230) can be positioned in the user's hand.

[0165] The control unit (110) can control the operation of the image output unit to output an image of the steering wheel in front of the driver's seat or to output an image of the steering wheel to the front windshield.

[0166] The control unit (110) controls the generation of a tactile signal by the haptic unit based on acquired steering information and controls the output of a holographic image by the image output unit, and can control changes in the holographic image based on the steering information.

[0167] The control unit (140) may be implemented as a memory (not shown) that stores data for an algorithm or a program that reproduces the algorithm for controlling the operation of components within the vehicle (1), and a processor (not shown) that performs the aforementioned operation using the data stored in the memory. In this case, the memory and the processor may each be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

[0168] The storage unit (111) stores steering motion information corresponding to gesture information.

[0169] The steering motion information may include information regarding the vehicle's rotation direction and the vehicle's rotation angle.

[0170] The storage unit (111) may be implemented as at least one of a non-volatile memory device such as a cache, ROM (Read Only Memory), PROM (Programmable ROM), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and flash memory, a volatile memory device such as RAM (Random Access Memory), or a storage medium such as a hard disk drive (HDD) and CD-ROM, but is not limited thereto. The storage unit (141) may be a memory implemented as a separate chip from the processor described above in relation to the control unit (140), or it may be implemented as a single chip with the processor.

[0171] The control unit (110) can also control communication between the image acquisition unit (210) and the haptic unit (220) within the user interface device.

[0172] The communication unit (120) may include one or more components that enable communication between internal components of the vehicle, and may include, for example, at least one of a short-range communication module, a wired communication module, and a wireless communication module.

[0173] The short-range communication module may include various short-range communication modules that transmit and receive signals using a wireless communication network at short range, such as a Bluetooth module, an infrared communication module, an RFID (Radio Frequency Identification) communication module, a WLAN (Wireless Local Access Network) communication module, an NFC communication module, and a Zigbee communication module.

[0174] Wired communication modules may include various wired communication modules such as Controller Area Network (CAN) communication modules, Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as various cable communication modules such as Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), Digital Visual Interface (DVI), RS-232 (recommended standard 232), power line communication, or plain old telephone service (POTS).

[0175] The wired communication module may further include a LIN (Local Interconnect Network).

[0176] In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, the wireless communication module may include wireless communication modules that support various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), and LTE (Long Term Evolution).

[0177] The sound output unit (130) outputs a sound corresponding to the generation of ultrasonic waves.

[0178] The sound output unit (130) can be provided in the haptic unit (220).

[0179] The sound output unit (130) outputs a sound corresponding to a function being performed in the vehicle (1) in response to a control command from the control unit (110). For example, the sound output unit (130) can output navigation information as sound when performing a navigation function.

[0180] The sound output unit (130) can output sound at a sound volume corresponding to the control command of the control unit (110).

[0181] The sound output unit (130) may include one or more speakers. Here, the speakers convert an amplified low-frequency voice signal into an original sound wave and generate compressional waves in the air to radiate the sound wave, thereby outputting audio data as sound that a user can hear.

[0182] The steering drive unit (140) controls the driving direction of the vehicle by driving the steering device in response to a control command from the control unit (110).

[0183] The lamp driving unit (150) turns the lamp on or turns the lamp off in response to a control command from the control unit (110).

[0184] Here, the lamp may include a UV-C wavelength lamp and may be provided adjacent to the image output unit (230).

[0185] The seat adjustment unit (160) adjusts the forward and backward directions and the up and down directions of the driver's seat (101) in response to a control command from the control unit (110).

[0186] The vehicle may include a first moving member that moves the seat in the forward and backward directions, and a second moving member that moves the seat in the up and down directions.

[0187] The forward direction can be the driving direction, and the rear direction can be the reverse direction.

[0188] The upper direction may be the direction toward the headliner, and the lower direction may be the direction toward the ground.

[0189] The user interface device (200) activates or deactivates the image acquisition unit, haptic unit, and image output unit in response to a control command from the control unit (110).

[0190] The user interface device (200) transmits the image acquired from the image acquisition unit (210) to the control unit (110).

[0191] The user interface device (200) can operate the haptic unit (220) and the image output unit (230) in response to a control command from the control unit (110). A detailed description of the user interface device has been explained with reference to FIGS. 2 to 7, so it will be omitted.

[0192] In FIG. 8, at least one component may be added or removed in response to the performance of the vehicle's components. Additionally, it will be readily understood by those skilled in the art that the relative positions of the components may be changed in response to the performance or structure of the system.

[0193] Meanwhile, each component illustrated in Fig. 8 refers to a software and / or hardware component such as a Field Programmable Gate Array (FPGA) and an Application Specific Integrated Circuit (ASIC).

[0194] Although an example of operating steering using a user interface device has been described, it is also possible to use the user interface device as an input unit to operate various functions performed within the vehicle or to operate various devices.

[0195] FIG. 11 is a control flowchart of a vehicle having a user interface device according to an embodiment.

[0196] When the vehicle receives a start signal, it activates the user interface device (200) (171).

[0197] When the user interface device (200) is activated, the vehicle outputs an image of a reference point (172) through the image output unit (230) of the user interface device.

[0198] The vehicle then acquires an image of the user's hand acquired through the image acquisition unit (210) and automatically adjusts the position of the seat (173) based on the position of the acquired user's hand and the position of the image of the reference point.

[0199] The vehicle can acquire an image of a reference point and an image of the user's hand together through an image acquisition unit (210), and can automatically adjust the position of the seat based on the position of the reference point and the position of the user's hand within the acquired image.

[0200] In addition, the vehicle may output an image of a reference point through the image output unit (230) and then request the user to adjust the seat based on the position of the reference point.

[0201] At this time, the vehicle can display guidance information on how to adjust the seat as a video through the display unit or output guidance sound through the sound output unit, depending on the position of the reference point and the position of the hand.

[0202] That is, when a seat adjustment command is received through the input unit (105), the vehicle can also adjust the position of the seat based on the received seat adjustment command.

[0203] The user can directly adjust the sheet to a position where it is easy to position the hand based on the location of the reference point.

[0204] When adjusting the position of the seat, the vertical height and the front-to-back distance of the seat can be adjusted.

[0205] The vehicle recognizes a gesture based on the image of a hand acquired by the image acquisition unit (210) of the user interface device, and if the recognized gesture is recognized as a touch gesture that touches a reference point image, the reference point image is changed into a steering control system image. At this time, the image output through the image output unit (230) can be changed.

[0206] That is, the vehicle can output a holographic image of the steering wheel (174).

[0207] Here, the steering control system video may include a steering wheel video, a button video, etc.

[0208] After the vehicle outputs a steering control system image, it acquires an image of the user's hand (175) through an image acquisition unit (210), recognizes a gesture based on the acquired image, recognizes a steering command based on the recognized gesture (176), and drives a steering device (177) based on the recognized steering command.

[0209] The vehicle sets the location information of the reference point image as reference location information and sets the coordinate value for the reference location information as reference coordinate value. Then, the vehicle sets the radius (M) of the image acquired by the image acquisition unit (210) based on the reference coordinate value (RP) and generates a circle based on the set radius.

[0210] The vehicle can control the image output unit to generate a circle with a preset radius based on the reference coordinate value (RP) and output an image of the generated circle.

[0211] The image acquired by the image acquisition unit (210) based on the reference coordinate value (RP) may be an image of a circle having the reference coordinate value as the center value of the circle and an image of the user's hand.

[0212] When recognizing a gesture, the vehicle recognizes the gesture on the image within the generated circle.

[0213] For example, the vehicle can identify the palm area occupied by the palm within the total area of ​​the generated circle and recognize a gesture based on the identified palm area.

[0214] As another example, the vehicle checks the number of fingers crossing the edge of a circle generated around a reference point and recognizes the gesture based on the identified number of fingers. Here, the number of fingers may be the contrast ratio of the image relative to the edge of the circle.

[0215] As another example, the vehicle checks the palm area occupied by the hand within the total area of ​​the generated circle, checks the number of fingers crossing the circle generated around a reference point, and recognizes the gesture based on the identified number of fingers and the palm area.

[0216] For example, the vehicle can control left turns if the area of ​​the palm relative to the total area is less than or equal to a first reference value, and control right turns if the area of ​​the palm relative to the total area exceeds the first reference value.

[0217] As another example, a vehicle can control a left turn if the recognized gesture is a bent thumb gesture, and a right turn if the recognized gesture is a bent little finger gesture.

[0218] As another example, a vehicle can also adjust the steering angle in response to the number of bent fingers in the recognized gesture.

[0219] As another example, when the recognized gesture is a bent finger, the vehicle can also adjust the steering angle based on the duration of the bend.

[0220] The vehicle can control the operation of the haptic unit (220) in response to gesture recognition.

[0221] That is, when the vehicle controls the operation of the haptic unit (220), it can control the generation of ultrasonic waves for each piezoelectric element array in response to gesture recognition.

[0222] Here, generating ultrasound includes outputting a tactile signal (178).

[0223] Accordingly, the user can be made to feel a sense of touch in the air by multiple ultrasonic signals output from the piezoelectric element array.

[0224] This embodiment can make gesture control in the air more realistic by using tactile feedback from a plurality of piezoelectric element arrays.

[0225] That is, while the user presses a steering control button formed in the air as a holographic image, tactile feedback is provided by ultrasonic haptics, and the user can control the steering of the vehicle.

[0226] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. The instructions may be stored in the form of program code and, when executed by a processor, may generate a program module to perform the operation of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0227] Computer-readable recording media include all types of recording media that store instructions that can be decoded by a computer. Examples include ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, optical data storage devices, etc.

[0228] As described above, the disclosed embodiments have been explained with reference to the attached drawings. Those skilled in the art will understand that the present invention may be practiced in forms different from the disclosed embodiments without changing the technical spirit or essential features of the invention. The disclosed embodiments are illustrative and should not be interpreted restrictively. Explanation of the symbols

[0229] 1: Vehicle 105: Input section 106: Display unit 108: Armrest 110: Control unit 200: User interface device 210: Image Acquisition Unit 220: Haptic Unit 230: Video output section

Claims

Claim 1 A user interface device for transmitting user commands for controlling at least one electronic device, comprising: an image acquisition unit for acquiring an image of a user; an image output unit for outputting an image of a control system; a control unit for recognizing a gesture based on the acquired image, recognizing a user command corresponding to the recognized gesture, and transmitting the recognized user command to the at least one electronic device; and a haptic unit for outputting a tactile signal for the gesture recognition in response to a control command of the control unit, wherein the control unit controls the image output unit to output an image of a reference point inside a vehicle, and the output image of the reference point is changed into the image of the control system in response to the gesture. Claim 2 A user interface device according to claim 1, wherein the gesture is recognized based on an image of a user acquired by an image acquisition unit touching an image of an output reference point. Claim 3 A user interface device according to claim 1, wherein the image output unit outputs a holographic image, and the control unit sets the output position of a tactile signal output through the haptic unit based on location information where the image of the hologram is displayed. Claim 4 A user interface device according to claim 1, wherein the control unit generates a circular image based on the position information of the reference point and a preset radius, controls the image output unit so that the generated circular image is output, obtains the area of ​​the palm within the circular image based on the image of the palm within the circular image obtained through the image acquisition unit, and recognizes the gesture based on the area of ​​the circular image and the area of ​​the palm. Claim 5 In claim 1, the haptic part is a user interface device comprising a plurality of piezoelectric element arrays that output an ultrasonic signal. Claim 6 In claim 1, the control unit is a user interface device that recognizes the number of fingers based on an image of a finger among the images of the user acquired through the image acquisition unit and recognizes the recognized number of fingers as a gesture. Claim 7 A user interface device according to claim 1, wherein the control unit outputs a circular image based on the position information of the reference point and a preset radius, recognizes the number of fingers crossing the edge of the circular image based on the number of brightness contrasts of the edge of the circular image acquired by the image acquisition unit, and recognizes the recognized number of fingers as a gesture. Claim 8 A vehicle comprising: a steering device; an image acquisition unit for acquiring an image of a user; an image output unit for outputting an image of a control system; a control unit for recognizing a gesture based on the acquired image, recognizing a user command corresponding to the recognized gesture, and controlling the operation of the steering device based on the recognized user command; and a haptic unit for outputting a tactile signal for the gesture recognition in response to a control command of the control unit, wherein the control unit controls the image output unit to output an image of a reference point inside the vehicle, and the output image of the reference point is changed into the image of the control system in response to the gesture. Claim 9 A vehicle according to claim 8, further comprising a seat adjustment unit for adjusting the position of a seat, wherein the control unit controls the operation of the seat adjustment unit based on an image of a hand and an image of a reference point among the images of a user acquired by the image acquisition unit. Claim 10 In claim 8, the image output unit outputs a holographic image, and the gesture is recognized based on an image of a user acquired by the image acquisition unit touching the image of the output reference point. Claim 11 A vehicle according to claim 8, wherein the image output unit includes a forward display unit, and the control unit sets the output position of a tactile signal output from the haptic unit based on the image of a user acquired by the image acquisition unit. Claim 12 A vehicle according to claim 8, further comprising an input unit; and at least one of a sound output unit and a display unit, wherein the control unit controls the operation of at least one of the sound output unit and the display unit to output guidance information for seat adjustment, and controls the operation of the seat to adjust the position of the seat based on a seat adjustment command received by the input unit. Claim 13 A vehicle according to claim 8, further comprising an input unit receiving setting information of at least one of the image output unit and the haptic unit, wherein the setting information includes at least one of the type of the control system image, the size of the control system image, the shape of the control system image, location information where the control system image is displayed, usage information of the control system, and location information where a tactile signal of the haptic unit is output, and wherein the control unit controls at least one of the image output unit and the haptic unit based on at least one setting information input to the input unit. Claim 14 In claim 8, the haptic unit comprises a plurality of piezoelectric element arrays that output an ultrasonic signal and an ultrasonic speaker, wherein the plurality of piezoelectric element arrays output a tactile signal as a vibration pressure corresponding to an input frequency corresponding to a control command of the control unit, and the ultrasonic speaker outputs a tactile signal as a sound corresponding to a control command of the control unit. Claim 15 In claim 8, the haptic unit comprises a plurality of piezoelectric element arrays that output an ultrasonic signal, a vehicle. Claim 16 In claim 8, the control unit generates a circular image based on the position information of the reference point and a preset radius, controls the image output unit so that the generated circular image is output, acquires the area of ​​a palm within the circular image based on the image acquired through the image acquisition unit, and recognizes the gesture based on the area of ​​the generated circle and the area of ​​the palm. Claim 17 In claim 8, the control unit generates a circular image based on the position information of the reference point and a preset radius, controls the image output unit so that the generated circular image is output, and recognizes the number of fingers crossing the edge of the circular image based on the number of contrast values ​​of the edge of the circular image acquired by the image acquisition unit, and recognizes the recognized number of fingers as a gesture. Claim 18 A method for controlling a vehicle, wherein an image of a reference point is output through an image output unit, the output image of the reference point is changed into a control system image corresponding to a gesture, a circle is generated based on the position information of the reference point, an image of a hand within the generated circle is obtained, a gesture is recognized based on the obtained image of the hand, a user command corresponding to the recognized gesture is recognized, the operation of a steering device is controlled based on the recognized user command, and the operation of a haptic unit is controlled in response to the recognition of the gesture. Claim 19 A method for controlling a vehicle according to claim 18, further comprising controlling parking, reverse, forward, and neutral movements based on the recognized user command. Claim 20 In claim 19, a method for controlling a vehicle, wherein controlling the operation of the above-mentioned haptic part includes outputting an ultrasonic signal.

Images