User interface system, control device, and non-transitory computer-readable medium

The user interface system addresses convenience issues by notifying users of operation detection, ensuring seamless transitions and preventing errors, thereby enhancing the reliability and usability of gesture-based interactions.

WO2024237228A1PCT designated stage expired Publication Date: 2024-11-21KK TOKAI RIKA DENKI SEISAKUSHO
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017607
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-15
Filing Date
2024-05-13
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

User interface systems that allow gesture input face challenges in convenience, particularly in determining whether detected non-contact operations are successfully executed, leading to user hesitation or undetected operation termination.

Method used

A user interface system with a detection device, control device, and notification mechanism that notifies users of the start of non-contact operations, ensuring seamless transition from a first to a second screen state based on detected actions, thereby enhancing user experience by preventing operation errors.

Benefits of technology

The system improves user interface convenience by clearly notifying users of operation detection, allowing continuous gesture input without hesitation and preventing operation failures, thus enhancing the reliability and usability of gesture-based interactions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017607_21112024_PF_FP_ABST
    Figure JP2024017607_21112024_PF_FP_ABST
Patent Text Reader

Abstract

A detection device (14) detects a non-contact operation performed by a user in order to execute a controlled operation. On the basis of the non-contact operation, a control device (12) causes a screen image displayed on a display device (22) to transition from a first state corresponding to a state before the controlled operation is executed to a second state corresponding to a state after the controlled operation has been executed. The control device (12) causes a notification device (13) to notify the user that detection of the non-contact operation has started.
Need to check novelty before this filing date? Find Prior Art

Description

User interface system, controller, and non-transitory computer-readable medium

[0001] The present disclosure relates to a user interface system that accepts gesture input, a control device that may be included in the system, and a non-transitory computer-readable medium having stored thereon a computer program executable by a processor included in the control device.

[0002] Japanese Patent Application Publication No. 2001-306243 discloses a user interface system that detects a non-contact action (gesture) associated with a predetermined controlled action based on the movement of a user's hand captured by a camera. When the predetermined gesture is detected, the device executes the controlled action associated with the gesture.

[0003] There is a demand for improving the usability of user interface systems that allow gesture input.

[0004] One example aspect provided by the present disclosure is a user interface system comprising: a detection device that detects a non-contact action performed by a user to perform a controlled action; and a control device that, based on the non-contact action, transitions a screen displayed on a display device from a first state corresponding to a state before the controlled action is performed to a second state corresponding to a state after the controlled action is performed, and the control device causes a notification device to notify the user that detection of the non-contact action has begun.

[0005] One example aspect provided by the present disclosure is a control device comprising: an interface that receives a detection signal corresponding to a non-contact action performed by a user to execute a controlled action; and a processor that, based on the detection signal, transitions a screen displayed on a display device from a first state corresponding to a state before the controlled action is executed to a second state corresponding to a state after the controlled action is executed, and the processor causes a notification device to notify the user that detection of the non-contact action has begun.

[0006] One example aspect provided by the present disclosure is a non-transitory computer-readable medium storing a computer program executable by a processor mounted on a control device, wherein execution of the computer program causes the control device to: receive a detection signal corresponding to a non-contact action performed by a user to execute a controlled action; based on the detection signal, transition a screen displayed on a display device from a first state corresponding to the state before the controlled action is executed to a second state corresponding to the state after the controlled action is executed; and cause a notification device to notify the user that detection of the non-contact action has begun.

[0007] For example, if a user performs a predetermined contactless motion to execute a specific controlled motion, but the controlled motion is not executed, it is difficult for the user to determine whether the cause is the detection device or the user's own motion. However, with the configurations according to the above-described exemplary embodiments, the user is notified that detection of the contactless motion has begun, allowing the user to continue the contactless motion without hesitation. Alternatively, by restarting the initial stage of the contactless motion so that the notification is received, the occurrence of a situation in which the contactless motion ends without detection can be reduced. Therefore, the convenience of a user interface system that allows gesture input can be improved.

[0008] 1 illustrates an example of the configuration of a user interface system according to an embodiment. 2 illustrates an example of a vehicle equipped with the user interface system of FIG. 1. 3 illustrates an example of a screen displayed on the display device of FIG. 1. 4 illustrates another example of a screen displayed on the display device of FIG. 1. 5 illustrates another example of a screen displayed on the display device of FIG. 1.

[0009] The following detailed description of exemplary embodiments will be given with reference to the accompanying drawings. In the drawings used in the following description, the scale has been changed appropriately so that each element can be clearly seen.

[0010] Fig. 1 illustrates an example of the functional configuration of a user interface system 10 according to an embodiment. The user interface system 10 according to this example is configured to be mounted on a vehicle 20 illustrated in Fig. 2. The user interface system 10 is used to control the operation of controlled devices mounted on the vehicle 20 and to acquire information related to the driving environment of the vehicle 20. Examples of controlled devices mounted on the vehicle 20 include audiovisual equipment, air conditioning equipment, lighting equipment, and seat devices. The vehicle 20 is an example of a moving object.

[0011] A display device 22 is installed in a passenger compartment 21 of the vehicle 20. The display device 22 is disposed in a center cluster 23 of the vehicle 20.

[0012] The display device 22 is configured to display a portal screen P as exemplified in Fig. 3. The portal screen P includes a first tile image T1, a second tile image T2, a third tile image T3, and a fourth tile image T4. The first tile image T1 is associated with a function for acquiring map information and traffic information. The second tile image T2 is associated with a music playback function. The third tile image T3 is associated with a function for acquiring weather information. The fourth tile image T4 is associated with a function for acquiring remaining battery power and mileage information.

[0013] 1 , the display device 22 is configured to include a touch panel device 221. The touch panel device 221 is configured to output a detection signal DT corresponding to a change in capacitance caused by a finger of an occupant of the vehicle 20 touching or approaching a detection surface. The detection signal DT may be an analog signal or a digital signal depending on the specifications of the touch panel device 221.

[0014] The user interface system 10 includes a control device 12. The control device 12 includes an input interface 121, a processor 122, and an output interface 123.

[0015] The input interface 121 is configured as a hardware interface that receives the detection signal DT. If the detection signal DT is an analog signal, the input interface 121 includes an appropriate conversion circuit including an A / D converter.

[0016] The processor 122 is configured to output a display control signal DC, which changes the screen displayed on the display device 22 in response to a touch operation corresponding to the detection signal DT, from an output interface 123 configured as a hardware interface. The display control signal DC may be an analog signal or a digital signal depending on the specifications of the display device 22. When the display control signal DC is an analog signal, the output interface 123 includes an appropriate conversion circuit including a D / A converter. This description also applies to other signals that can be output from the output interface 123, which will be described later.

[0017] In the example of FIG. 3, the touch panel device 221 is configured to accept touch operations on each of the first tile image T1, the second tile image T2, the third tile image T3, and the fourth tile image T4.

[0018] For example, when a touch operation is performed on second tile image T2, a detection signal DT corresponding to the position where the touch operation was performed is output from touch panel device 221. When input interface 121 of control device 12 receives detection signal DT, processor 122 outputs, from output interface 123, a display control signal DC that causes display device 22 to make a change to enlarge second tile image T2, as illustrated in FIG.

[0019] The enlarged second tile image T2 provides more detailed information about the music playback function and can accept operation inputs related to more detailed function selection. The occupant of the vehicle 20 can cause the audio equipment to perform a desired operation by appropriately touching the button images or thumbnail images displayed in the enlarged second tile image T2. This description also applies to other tile images.

[0020] As a result of the second tile image T2 being enlarged, the other tile images that were not initially selected disappear from the screen of the display device 22, leaving only the third tile image T3. The third tile image T3 is displayed in a grayed-out state to indicate that it is not selected.

[0021] The processor 122 of the control device 12 is configured to be able to detect non-contact actions (gestures) made by an occupant of the vehicle 20 based on a detection signal DT corresponding to a change in capacitance on the detection surface of the touch panel device 221 as the occupant's hand approaches and moves. In this example, the touch panel device 221 is an example of a detection device.

[0022] 5 and 6 , when a sweeping motion is performed with the occupant's hand approaching the screen of the display device 22, the enlarged display of the second tile image T2 is canceled, and the state returns to the state where the portal screen P is displayed. The sweeping motion is an example of a non-contact motion. Returning to the portal screen P is an example of a controlled motion.

[0023] That is, when a detection signal DT corresponding to a sweeping action is received by input interface 121, processor 122 outputs a display control signal DC from output interface 123, which transitions the screen displayed on display device 22 from the state illustrated in Fig. 4 to the state illustrated in Fig. 6. The screen state illustrated in Fig. 4 is an example of a first state corresponding to the state before the controlled action is executed. The screen state illustrated in Fig. 6 is an example of a second state corresponding to the state after the controlled action is executed.

[0024] More specifically, when the occupant brings their hand close to the screen of the display device 22 prior to the sweeping motion, the processor 122 determines that the detection of the sweeping motion has started based on the detection signal DT corresponding to that state. In this case, the processor 122 outputs, from the output interface 123, a display control signal DC that causes the display device 22 to cancel the grayed-out state of the third tile image T3.

[0025] As a result, the grayed-out state of the third tile image T3 is canceled, as illustrated in Figures 4 and 5. This allows the occupant to recognize that detection of a swatting motion has started. In other words, the processor 122 causes the display device 22 to notify the occupant that detection of a swatting motion has started. In this example, the display device 22 is an example of a notification device. The screen state illustrated in Figure 5 is an example of a third state indicating that detection of a non-contact motion has started.

[0026] For example, if an occupant performs a sweeping motion to return to the portal screen P but the return does not occur, it is difficult for the occupant to determine whether the cause lies with the detection device or with the occupant's own actions. However, with the configuration according to this embodiment, the occupant is notified that detection of the sweeping motion has begun, so the occupant can continue the sweeping motion without hesitation. Alternatively, by repeating the initial stage of the sweeping motion so that the notification is received, it is possible to prevent the occupant from ending the sweeping motion without detection. Therefore, the convenience of the user interface system 10 that allows gesture input can be improved.

[0027] The processor 122 of the control device 12 having the various functions described above can be realized by a dedicated integrated circuit such as a microcontroller, ASIC, FPGA, or the like, which includes a storage element in which a computer program for realizing the functions is pre-installed. In this case, the storage element is an example of a non-transitory computer-readable medium on which a computer program is stored.

[0028] Alternatively, the processor 122 may be realized by a general-purpose microprocessor operating in cooperation with general-purpose memory. Examples of the general-purpose microprocessor include a CPU, an MPU, and a GPU. Examples of the general-purpose memory include a ROM and a RAM. In this case, a computer program for realizing the function may be stored in the ROM. The general-purpose microprocessor specifies at least a portion of the program stored in the ROM, expands it on the RAM, and executes the above-described processing in cooperation with the RAM. In this case, the general-purpose memory is an example of a non-transitory computer-readable medium on which a computer program is stored.

[0029] Processor 122 may be implemented by a combination of a general-purpose microprocessor and a dedicated integrated circuit.

[0030] The configurations described above are merely examples to facilitate understanding of the present disclosure. Each configuration example can be appropriately modified and combined with other configuration examples without departing from the spirit of the present disclosure.

[0031] In the above embodiment, the start of detection of a sweeping motion is notified by canceling the grayed-out state of the third tile image T3. However, any appropriate display mode can be selected as long as it is different from the state before and after the sweeping motion. For example, the enlarged second tile image T2 may be grayed out, or an appropriate mark, an image of a hand performing a sweeping motion, or the like may be additionally displayed.

[0032] In the above embodiment, the display device 22 functions as a notification device. By sharing the function with the display device 22, an increase in the number of devices constituting the user interface system 10 can be suppressed.

[0033] 1, the user interface system 10 may include an independent notification device 13. The notification device 13 may be installed at an appropriate position within the passenger compartment 21 of the vehicle 20 so as to notify the occupant that a brushing operation has been initiated through at least one of a visual notification, an audible notification, and a tactile notification. In this case, the processor 122 of the control device 12 outputs a control signal CT from the output interface 123, based on the detection signal DT received by the input interface 121, to cause the notification device 13 to perform a predetermined notification operation.

[0034] In the above embodiment, the touch panel device 221 functions as a detection device. By sharing the function with the touch panel device 221 as an input device, an increase in the number of devices constituting the user interface system 10 can be suppressed.

[0035] However, in addition to or instead of the touch panel device 221, as illustrated in FIG. 1 , the user interface system 10 may include an independent detection device 14. For example, the detection device 14 may be realized by an imaging device installed at an appropriate position in the passenger compartment 21 of the vehicle 20. The detection device 14 is configured to output a detection signal DT corresponding to an image capturing the occupant's hand movement. In this case, the processor 122 of the control device 12 detects that a sweeping motion has been started and performed based on the detection signal DT received by the input interface 121. The detection device 14 may also detect the occupant's hand movement using infrared rays or radio waves.

[0036] In the above embodiment, the touch panel device 221 is disposed in the center cluster 23 of the vehicle 20. That is, the touch panel device 221 is disposed between the driver's seat and the passenger seat in the left-right direction of the vehicle 20. With such a configuration, non-contact actions by not only the driver but also the passenger sitting in the passenger seat can be detected, thereby further improving the convenience of the user interface system 10 that allows gesture input. From this perspective, it is preferable that the detection device 14 described above be disposed in a position where it can detect non-contact actions by the passenger sitting in the passenger seat.

[0037] The user interface system 10 may be installed in a moving body other than the vehicle 20. Examples of such moving bodies include trains, airplanes, and ships. The moving body may not require a driver. The moving body may have only one seat for the passengers.

[0038] The user interface system 10 does not necessarily need to be mounted on a mobile object, but can be configured as a system that allows input of a predetermined controlled action to cause an appropriate device installed in a home or facility to execute a desired controlled action.

[0039] The contents of Japanese Patent Application No. 2023-080051 filed on May 15, 2023 are incorporated by reference as part of this disclosure.

Claims

1. A user interface system comprising: a detection device that detects a non-contact action performed by a user to execute a controlled action; and a control device that transitions a screen displayed on a display device from a first state corresponding to the state before the controlled action is executed to a second state corresponding to the state after the controlled action is executed based on the non-contact action, wherein the control device causes a notification device to notify the user that detection of the non-contact action has begun.

2. The user interface system according to claim 1, wherein the control device causes the display device to function as the notification device by transitioning the screen to a third state, which indicates that detection of the non-contact motion has begun.

3. The user interface system according to claim 1 or 2, wherein the detection device includes a touch panel device that detects an operation on the display device based on a change in electrostatic capacitance, and the non-contact operation is detected based on the change in electrostatic capacitance.

4. A user interface system according to any one of claims 1 to 3, configured to be mounted on a moving object.

5. The user interface system according to claim 4, wherein the moving body is a vehicle equipped with a driver's seat and a passenger seat, and the detection device is disposed between the driver's seat and the passenger seat in the left-right direction of the vehicle.

6. A control device comprising: an interface that receives a detection signal corresponding to a non-contact action performed by a user to execute a controlled action; and a processor that transitions a screen displayed on a display device from a first state corresponding to the state before the controlled action is executed to a second state corresponding to the state after the controlled action is executed based on the detection signal, wherein the processor causes a notification device to notify the user that detection of the non-contact action has begun.

7. A non-transitory computer-readable medium storing a computer program executable by a processor mounted on a control device, wherein, when the computer program is executed, the control device: receives a detection signal corresponding to a non-contact action performed by a user to execute a controlled action; based on the detection signal, transitions a screen displayed on a display device from a first state corresponding to a state before the controlled action is executed to a second state corresponding to a state after the controlled action is executed; and causes a notification device to notify the user that detection of the non-contact action has begun.

Citation Information

Patent Citations

  • Information processor

    JP2016153250A

  • Gesture input operation processing device

    WO2013136776A1