control device

Abstract

The present invention relates to a control device. A control device for providing a user interface to a user has a display control section that displays a selectable object on a display device, a detection section that detects a position of a hand by analyzing an image of the hand of the user, a determination section that determines an indication position in the display device based on the position of the hand, and a processing section that performs processing associated with the selectable object based on the indication position overlapping the selectable object. The display control section also displays a cursor indicating the indication position on the display device.

Description

Technical Field

[0001] This invention relates to control devices. Background Technology

[0002] Various technologies have been proposed for receiving input from users of vehicles. Patent Document 1 proposes a technology for inputting data into in-vehicle devices by users of vehicles through contactless gestures (so-called air gestures).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-132824 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] In order to input data through operations such as specific hand shapes and hand movement patterns, these operations must be pre-stored. Therefore, intuitive input operations are sometimes not possible for the user of the control device (hereinafter simply referred to as the user). Part of the object of this invention is to provide a technique that enables intuitive input operations for the user.

[0008] means for solving problems

[0009] According to one embodiment, a control device is provided for providing a user interface to a user. The control device includes: a display control unit that displays a selectable object on a display device; a detection unit that detects the position of the user's hand by analyzing an image of the user's hand; a determination unit that determines an indication position on the display device based on the position of the hand; and a processing unit that performs processing to establish an association with the selectable object based on the case where the indication position overlaps with the selectable object. The display control unit also displays a cursor indicating the indication position on the display device.

[0010] Invention Effects

[0011] According to one implementation method, users can perform intuitive input operations. Attached Figure Description

[0012] Figure 1 This diagram illustrates an example of the configuration of a vehicle and control device based on a partial implementation method.

[0013] Figure 2 This is a diagram illustrating a vehicle layout example based on a partial implementation method.

[0014] Figure 3 This diagram illustrates an example of feature points of a hand based on a partial implementation method.

[0015] Figure 4 This diagram illustrates the transition of the display screen based on a portion of the implementation method.

[0016] Figure 5 This diagram illustrates the transition of the display screen based on a portion of the implementation method.

[0017] Figure 6 This diagram illustrates the transition of the display screen based on a portion of the implementation method.

[0018] Figure 7 This diagram illustrates the transition of the display screen based on a portion of the implementation method.

[0019] Figure 8 This diagram illustrates the transition of the display screen based on a portion of the implementation method.

[0020] Figure 9 This diagram illustrates the transition of the display screen based on a portion of the implementation method.

[0021] Figure 10 This diagram illustrates an example of the operation of a control device based on a partial implementation method.

[0022] Explanation of reference numerals in the attached figures

[0023] 100: Vehicle; 101: Control device; 105: Display device; 501: Icon. Detailed Implementation

[0024] The embodiments will now be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments are not intended to limit the invention, and that not all combinations of features described in the embodiments are necessarily essential to the invention. Any combination of two or more features described in the embodiments may be used. Furthermore, identical or identical components will be labeled with the same reference numerals, and repeated descriptions will be omitted.

[0025] A portion of this disclosure relates to a control device for providing a user interface to a user. The control device can be any type of device used to control power management devices (static devices), vehicles, lawnmowers, aircraft (dynamic devices). Hereinafter, an example of the control device controlling a vehicle will be described. The user of the vehicle can be the driver or a passenger. According to a portion of this disclosure, the user can perform input operations simply by moving their hand without touching any specific device. Furthermore, since the location pointed to by the user on the display device is represented by a cursor, the user can intuitively identify which location on the display device they have pointed to.

[0026] Reference Figure 1 The configuration examples of the hardware mounted on the vehicle 100 and the functional configuration examples of the control device 101 will be described below. The following description will focus on the components used in the implementation description. Although used in the implementation of the vehicle 100, descriptions of components not used in the implementation description (e.g., drive devices such as engines and motors, braking devices such as brakes, etc.) are omitted.

[0027] Vehicle 100 includes a control device 101, a camera 104, a display device 105, a speaker 106, and a microphone 107. The control device 101 provides a user interface to the user of vehicle 100 by controlling the camera 104, display device 105, speaker 106, and microphone 107. Hereinafter, the user of vehicle 100 will be referred to simply as the user. The control device 101 may be, for example, an ECU (Electronic Control Unit). The control device 101 may be implemented using a single device or multiple devices capable of communicating with each other. The control device 101 includes a processor 102 and a memory 103. The memory 103 stores programs and data for processing the control device 101. Additionally, the memory 103 may store temporary data generated during the execution of processing by the control device 101. The operation of the control device 101 is performed by the processor 102 executing the program stored in the memory 103. The processor 102 may be a single processor or multiple processors. The memory 103 can be a single memory or multiple memories.

[0028] The capturing device 104 captures images of the user (especially images of the user's hand). The capturing device 104 can be a visible light camera that captures images of the user using visible light. Alternatively, the capturing device 104 can be a near-infrared camera that captures images of the user using near-infrared light. Furthermore, the capturing device 104 can also be a ToF (Time of Flight) camera that captures images of the user using laser light. The capturing device 104 can also be any other type of sensor that captures images of the user. The vehicle 100 may have only one capturing device 104 or multiple capturing devices 104. Furthermore, if the vehicle 100 has multiple capturing devices 104, it may have only one of each type of capturing device 104 or multiple types.

[0029] Display device 105 is a device for providing visual information to a user. For example, display device 105 may be a dot matrix display, a segmented display, an indicator, or any other arbitrary display device. Vehicle 100 may have only one display device 105 or multiple display devices 105.

[0030] Speaker 106 is a device for providing auditory information to a user. Vehicle 100 may have only one speaker 106 or multiple speakers 106. Microphone 107 is a device for receiving sound input from a user. Vehicle 100 may have only one microphone 107 or multiple microphones 107. Vehicle 100 may have input devices other than those mentioned above, such as physical buttons, touch panels, joysticks, etc.

[0031] Reference Figure 2 ,right Figure 1 The example of the installation location of the hardware element described in (a) is used to illustrate this. Figure 2 This is a view taken from the center of the vehicle compartment of vehicle 100, looking forward (towards the windshield 201). Vehicle 100 is a right-hand drive vehicle, facing the direction of travel and having a steering wheel 202 on the right side. This disclosure can also be applied to left-hand drive vehicles.

[0032] The camera device 104 is mounted, for example, near the upper center of the windshield 201. Alternatively, the camera device 104 may be mounted on the rearview mirror (not shown) inside the vehicle 100. The camera device 104 is positioned to capture the hands of both the user seated in the driver's seat (i.e., the driver) and the user seated in the front passenger seat (i.e., the passenger). The control device 101 can identify whether the hand captured by the camera device 104 is a right or left hand. The range within which the user's hand can be captured can be the range considered to be the area in front of the body when the user is seated. The vehicle 100 may have one camera device 104 directly in front of the driver's seat and another camera device 104 directly in front of the front passenger seat.

[0033] The display device 105 is, for example, mounted near the lower center of the windshield 201. Alternatively, the display device 105 may be mounted on the instrument panel (not shown) of the vehicle 100. Figure 2 In the example, vehicle 100 has only one display device 105. Instead, in addition to the display device 105 mounted near the lower center of the windshield 201, vehicle 100 may have another display device 105 directly in front of the passenger seat. The display device 105 can be a device with its own display surface, such as a liquid crystal display or an organic EL (electro-luminescence) display, or it can be a device that projects images onto an object outside the device, such as a head-up display. The display device 105 can be a device that generates three-dimensional (3D) images, holograms, virtual reality (VR) images, or augmented reality (AR) images.

[0034] exist Figure 2 In the example shown, vehicle 100 has a pair of speakers 106 on the left and right sides below display device 105, and a microphone 107 between these speakers 106. The configuration of these devices is not limited to this location.

[0035] Reference Figure 1 Section (b) describes the function of the control device 101. The control device 101 has... Figure 1 The functional units shown in (b) are as follows. These functional units can be implemented by the processor 102 executing a program stored in the memory 103. Alternatively, some or all of these functional units can be implemented by application-specific integrated circuits such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays).

[0036] The image analysis unit 111 acquires and analyzes an image of the user captured by the imaging device 104. The image of the user captured by the imaging device 104 is referred to as a user image. The user image includes the user, and in particular, the user's hands. The image analysis unit 111 detects the position and shape of the user's hands by analyzing the user image.

[0037] Reference Figure 3 This section provides specific examples of methods for analyzing user images. Figure 3 This represents a hand 300 contained in a user image. The image analysis unit 111 detects the position and shape of the hand 300 based on the positions of multiple feature points contained in the hand 300. Figure 3 In the example, 20 feature points of hand 300 are set. Figure 3 In the diagram, each of the multiple feature points is represented by a circle, with one of them designated as feature point 301 (the fingertip of the index finger). For each of the index, middle, ring, and little fingers, the fingertip, first joint, second joint, and base are designated as feature points. For the thumb, the fingertip, first joint, and base are designated as feature points. Furthermore, the center of the thumb ball of the hand 300 is designated as a feature point.

[0038] The image analysis unit 111 detects the position of the hand 300 based on the center positions of multiple feature points of the hand 300. For example, the image analysis unit 111 can determine the two-dimensional coordinate values ​​of multiple feature points in the user image, and determine the two-dimensional coordinate value of the hand 300 by a simple arithmetic mean of the two-dimensional coordinate values ​​of the multiple feature points. When the user image is a two-dimensional image, the coordinate values ​​of the two-dimensional image become the two-dimensional coordinate values ​​of the feature points. When the user image is a three-dimensional image, the coordinate values ​​of a specific plane (e.g., a plane orthogonal to the vehicle length direction) in the three-dimensional image become the two-dimensional coordinate values ​​of the feature points. When there are feature points that cannot be identified based on the user image, the image analysis unit 111 can detect the position of the hand 300 without considering the unidentifiable feature points. Alternatively, the image analysis unit 111 can also estimate the position of the unidentifiable feature points based on the identified feature points, and detect the position of the hand 300 based on the estimated position and the position of the identified feature points. By detecting the position of the hand based on multiple feature points in this way, the position of the hand can be detected with high accuracy.

[0039] The image analysis unit 111 can detect the shape of the hand 300 based on the relative positions of multiple feature points. For example, if the distance between the fingertips of the index and middle fingers is greater than the distance between the bases of the index and middle fingers, the image analysis unit 111 detects that the index and middle fingers are separated. If the distance between the fingertips of the index and middle fingers is equal to the distance between the bases of the index and middle fingers, the image analysis unit 111 detects that the index and middle fingers are in contact. The shape of the hand 300 can be detected using a supervised learning model that takes the two-dimensional coordinates of multiple feature points as input and the actual shape of the hand as output.

[0040] Refer again Figure 1 In (b), the indication position determination unit 112 determines the position indicated by the user in the display device 105 based on the position of the user's hand. The position indicated by the user is called the indication position. For example, the indication position determination unit 112 may store a mapping between the two-dimensional coordinates of the hand's position in the user's image and the two-dimensional coordinates of the screen of the display device 105. According to this mapping, the indication position determination unit 112 establishes a correspondence between the position of the hand 300 with specific two-dimensional coordinates in the image and the specific two-dimensional coordinates of the screen of the display device 105, and determines the two-dimensional coordinates as the indication position.

[0041] The position determination unit 112 can align the position of the user's hand with the center of the screen on the display device 105. For example, the position determination unit 112 can use the display device 105 or the speaker 106 to request the user to move their hand to a position indicating the center of the screen on the display device 105. After the user's hand comes to rest, the position determination unit 112 detects the position of the user's hand and establishes a correspondence between the position of the hand and the center of the screen on the display device 105.

[0042] The indicator position determination unit 112 moves the indicator position in the display device 105 based on the movement of the user's hand. For example, if the user moves their hand upwards from their perspective, the indicator position determination unit 112 moves the indicator position upwards accordingly. If the user moves their hand to the right from their perspective, the indicator position determination unit 112 moves the indicator position to the right accordingly. The relationship between the distance the hand moves and the distance the indicator position moves can be predetermined and stored in the memory 103. For example, if the hand moves 50mm, the indicator position determination unit 112 can move the indicator position by the same amount of 50mm, or it can move the indicator position a shorter distance than the hand's movement (e.g., 30mm), or it can move the indicator position a longer distance than the hand's movement (e.g., 100mm). If the user moves their hand and the indicator position deviates from the screen of the display device 105, the indicator position determination unit 112 can determine that the user is not pointing at the display device 105. The ratio of the indicated position to the distance the hand has moved is stored, for example, in memory 103. The user can change this ratio.

[0043] The processing unit 113 performs various processes corresponding to input from the user. These processes can be anything, such as screen transitions, music playback, video playback, navigation, web browsing, games, etc. The display control unit 114 controls the content displayed on the display device 105. Specific examples of the operation of the display control unit 114 will be described later. The time measurement unit 115 measures various times during the operation of the control device 101. The user identification unit 116 identifies the user using the user interface provided by the control device 101. The control device 101 can provide a separate user interface for each identified individual user.

[0044] Reference Figure 4 This describes an example of how the control device 101 controls the content displayed on the display device 105. The control device 101 (specifically, its display control unit 114) displays a screen 400 on the display device 105. The screen 400 includes multiple icons (in...) Figure 4 In the example, there are six icons labeled "1-1" through "1-6". An icon is an example of a selectable object (also called a selectable object). Selectable objects can be content other than icons. For example, a selectable object could be a string with a link or an image. Hereinafter, icons are mentioned as examples of selectable objects, but the techniques described below can also be applied to other selectable objects besides icons. Specific actions can be associated with selectable objects (e.g., screen transition, music playback start, video playback start, volume increase, etc.).

[0045] The control device 101 determines the indicated position on the display device 105 based on the position of the user's hand, and a cursor 402 representing the indicated position is also displayed on the screen 400 of the display device 105. Figure 4 In the example, cursor 402 has an arrow shape, with the tip of the arrow indicating the indicated position. The shape of cursor 402 is not limited to an arrow; it can be any other shape or representation. For example, cursor 402 can be a circle or a star.

[0046] Suppose that the indicated position is moved along the dotted line 403 by the user moving their hand. In this case, the control device 101 also moves the cursor 402 along the dotted line 403. If the indicated position overlaps with a specific icon 401, it is considered that the user has requested the process of establishing an association with icon 401. Therefore, the control device 101 (specifically, its processing unit 113) can execute the process of establishing an association with icon 401 based on the case where the indicated position overlaps with icon 401. In this way, the user can indicate the execution of the process of establishing an association with icon 401 simply by moving their hand to make the cursor 402 overlap with icon 401. Thus, the user can perform intuitive input operations.

[0047] When the indicated position overlaps with icon 401, the control device 101 can immediately begin processing. However, as in embodiments of this disclosure, in the case of input operations performed by air gestures, there is a possibility that the user may accidentally move their hand, causing the indicated position to overlap with icon 401. Therefore, in some embodiments, the control device 101 (specifically, its time measuring unit 115) measures the duration of the overlap between the indicated position and icon 401. Furthermore, if the duration reaches a threshold time (e.g., 3 seconds), the control device 101 (specifically, its processing unit 113) performs the process of establishing an association with icon 401. On the other hand, if the duration does not reach the threshold time (e.g., if the indicated position has previously deviated from icon 401), the control device 101 (specifically, its processing unit 113) does not perform the process of establishing an association with icon 401. Thus, it is possible to suppress the execution of unintentional user processing. The duration reaching the threshold time can be interpreted as the completion of a timer set to the threshold time. The threshold time is stored, for example, in memory 103. The user can change the threshold time. The same applies to the other threshold times described below.

[0048] The control device 101 begins timing based on the overlap between the indicated position and the icon 401. Simultaneously, the control device 101 switches the display of screen 400 to screen 410, which includes the timer indicator 411. By displaying the timer indicator 411 in this way, the user can easily identify the overlap between the indicated position and the icon 401. Therefore, if such movement of the indicated position is intentional on the part of the user, it is considered that the user has moved their hand.

[0049] The control device 101 changes the appearance of the timer indicator 411 based on the duration of the overlap between the indicated position and the icon 401. For example, as shown in screen 420, the control device 101 can shorten the shaded portion of the timer indicator 411 as time passes. This allows the user to monitor the processing progress over time.

[0050] exist Figure 4 In this example, the timer indicator 411 is displayed as associated with the icon 401 that overlaps with the indicated position (e.g., by overlapping with icon 401). Alternatively, the timer indicator 411 may also be displayed at a position independent of the icon 401 that overlaps with the indicated position (e.g., the upper right of the screen, or a predetermined position). Furthermore, the timer indicator 411 may represent the elapsed time since the point in time when the indicated position overlaps with icon 401, or it may represent the remaining time until the start of the process associated with icon 401. Additionally, the timer indicator 411 may be an illustration, or it may be text or a symbol.

[0051] If the indicator position moves outward from icon 401 before the duration of the state where the indicator position overlaps with icon 401 reaches a threshold time, the control device 101 does not perform the process of establishing an association with icon 401, but instead restores icon 401 to its normal display as shown in screen 400. If the duration of the state where the indicator position overlaps with icon 401 reaches the threshold time, the control device 101 performs the process of establishing an association with icon 401. Based on the execution of this process, the control device 101 can switch the display of screen 420 to the display of screen 430 inherent to the process. The control device 101 can apply a timer until the start of execution to all displayed selectable objects as described above, or it can apply a timer to only a portion of the selectable objects.

[0052] Reference Figure 5 This describes another example of control by the control device 101 over the content displayed on the display device 105. The control device 101 displays a screen 500 on the display device 105. The screen 500 contains multiple icons (in... Figure 5In the example, there are six icons labeled "2-1" to "2-6". Furthermore, screen 500 includes a cursor 402 indicating the user's pointing position. Since the pointing position overlaps with icon 501, a timer indicator 502 is also displayed.

[0053] When the duration of the state in which the indicated position overlaps with icon 501 reaches a threshold time, the control device 101 executes a process to establish an association with icon 501. This process is configured to include switching the display of screen 500 to screen 510. Screen 510 contains multiple icons (in... Figure 5 In the example, there are 6 icons labeled "3-1" to "3-6". During the transition from screen 500 to screen 510, if the position indicated by the user does not change, the control device 101 continuously displays the cursor 402 at the same position on the display device 105.

[0054] The screen 510 after the jump is configured to include icon 511 at a position overlapping with the indicated position. In this case, although the indicated position overlaps with icon 511, the user may not intend to select icon 511. Therefore, if the indicated position at the start of the display time on screen 510 overlaps with icon 511, the control device 101 considers that the indicated position does not overlap with icon 511 until a predetermined condition is met. Consequently, the control device 101 does not start measuring elapsed time and does not display the time indicator. This prevents unintentional user processing from being performed.

[0055] The aforementioned predetermined condition could be that the duration of the state in which the indicator position overlaps with icon 511 reaches a threshold time (e.g., 5 seconds). If the user does not move their hand for a certain period of time, it is considered that the user intends to select icon 511. Alternatively, or based on this, the aforementioned predetermined condition could be that the indicator position moves outward from icon 511 and overlaps with icon 511 again. If the user intentionally changes the indicator position in this way, it is considered that the user intends to select icon 511.

[0056] When the predetermined conditions described above are met, the control device 101 assumes that the indicated position overlaps with the icon 511 and begins to measure the elapsed time, displaying the timing indicator 521 as shown on screen 520.

[0057] Reference Figure 6 Another control example will be described where the indicated position overlaps with the icon on the screen after the jump. Screen 600 is the same as screen 500. When the indicated position overlaps with icon 501, a timer set to initiate the process of establishing an association with icon 501 completes. In this case, control device 101 displays a screen containing multiple icons (in...). Figure 5In the example, there is a screen 610 containing 6 icons labeled “3-1” to “3-6”.

[0058] Since the indicator position of the start display time on screen 610 overlaps with icon 511, regardless of the user's hand movement, control device 101 considers the indicator position on screen 610 to have moved to a position that does not overlap with any icon, and cursor 402 also moves to indicate the moved indicator position. This suppresses unintentional user actions. In this case, because the actual position of the user's hand and the indicator position are offset, control device 101 also offsets the mapping between the hand position in the image captured by imaging device 104 and the indicator position on display device 105.

[0059] Reference Figure 7 Another control example regarding the overlap between the indicated position and the icon on the subsequent screen will be explained. Screen 700 is the same as screen 500. When the indicated position overlaps with icon 501, a timer set to initiate the process of establishing an association with icon 501 is completed. In this case, the control device 101 displays a screen containing multiple icons (in...). Figure 5 In the example, there is a screen 710 containing 6 icons labeled “3-1” to “3-6”.

[0060] The indicator position for the start display time on screen 710 is set to overlap with the original position of the displayed icon 511. In this case, as shown on screen 710, the control device 101 displays icon 511 on screen 710 at a position that does not overlap with the indicator position. As a result, the indicator position does not overlap with any icon. Therefore, it is possible to suppress unintentional user processing. Figure 7 In this example, the control device 101 may display only icon 511 in other locations, but it may also be reconfigured to include icons other than 511.

[0061] Reference Figure 8 This describes another example of control by the control device 101 over the content displayed on the display device 105. The control device 101 displays a screen 800 on the display device 105. The screen 800 contains multiple icons (in... Figure 8 In the example, there are six icons labeled "4-1" to "4-6". Furthermore, screen 800 includes a cursor 402 that indicates the location indicated by the user. Additionally, in screen 800, at the location shown by the dotted line, a switching area 802 is provided for switching the current screen to another screen. Figure 8In this example, switching areas 802 are provided on the left and right sides of the screen 800. Alternatively, or based on this, the switching areas 802 can also be provided on the top or bottom of the screen 800. In addition, the control device 101 can display the screen 800 in a way that allows the user to recognize the presence of the switching areas 802.

[0062] If the duration of the overlap between the indicated position and the switching area 802 reaches a threshold time, the control device 101 switches the display of the current screen 800 to the screen associated with the switching area 802 (e.g., screen 810). This screen switching via the switching area 802 can be used for example, for screen scrolling. Similarly, as with the case where the indicated position overlaps with the icon 801, the control device 101 can display a timer indicator showing the duration of the overlap between the indicated position and the switching area 802.

[0063] The switching area 802 can be divided into multiple switching areas 802a to 802c. Switching area 802c is furthest from the edge of the screen 800, switching area 802b is closer to the edge of the screen 800 than switching area 802c, and switching area 802a is closest to the edge of the screen 800. The control device 101 can independently set a threshold time for screen switching for each of the multiple switching areas 802a to 802c. For example, the control device 101 can shorten the threshold time for areas closer to the edge of the screen 800. For example, the threshold time for switching area 802a can be 0.5 seconds, the threshold time for switching area 802b can be 1 second, and the threshold time for switching area 802c can be 3 seconds. The further a switching area is from the edge of the screen, the higher the possibility of accidental user input. Therefore, the further a switching area is from the edge of the screen, the longer the threshold time is set by the control device 101. This suppresses unintentional user screen switching.

[0064] Reference Figure 9 Another example of control of the content displayed on the display device 105 by the control device 101 will be described. The control device 101 displays a screen 900 on the display device 105. The screen 900 includes multiple icons (in... Figure 9In the example above, there are six icons labeled "6-1" to "6-6". Additionally, screen 900 includes a cursor 402 indicating the location indicated by the user. Furthermore, screen 900 includes an icon 902 indicating the display of the overview screen 910. Similar to the example above, the control device 101 can switch screens based on a timer indicating the completion of this process. The icon 902 indicating the display of the overview screen 910 may be included in only a portion of the screen displayed by the display device 105, or it may be included in the entire screen displayed by the display device 105.

[0065] By selecting icon 902, the control device 101 switches the display of screen 900 to the display of overview screen 910. In overview screen 910, the control device 101 switches the display of screen 900 to the overview screen 910 via icons (in...). Figure 9 In the overview screen 910, multiple functions are represented by labels from "1" to "12". In each of the multiple areas 911 to 913 of the overview screen 910, highly relevant functions are displayed in a summary manner. The overview screen 910 includes an indicator 914 that represents icons of functions related to the previous screen 900.

[0066] Reference Figure 10 An example of the operation of the control device 101 will be explained. Figure 10 The steps of the described method can be performed, for example, by the processor 102 executing a program stored in the memory 103. Figure 10 The method can begin either when the power to vehicle 100 is turned on, or according to explicit instructions from the user. The details described above will not be repeated below.

[0067] In step S1001, the control device 101 (e.g., its image analysis unit 111) detects the user's hand by analyzing the user's image. To perform this detection, the control device 101 may request the user to move their hand to the center of the screen on the display device 105. At this time, the control device 101 can align the position of the user's hand with the center of the screen on the display device 105.

[0068] In step S1002, the control device 101 (e.g., its user identification unit 116) determines whether the user is a specific person based on the detected image of the hand. A specific person may refer to someone who has previously used the user interface provided by the control device 101. To make such a determination, the control device 101 may store usage history records in the memory 103 for each user. The usage history records may include, for example, settings such as the frequency of use of each function, the ratio of the distance the indicated position is moved to the distance the hand is moved, and the setting of the threshold time for performing processing.

[0069] If it is determined in step S1002 that the person is a specific person ("Yes" in step S1002), the control device 101 transfers the process to step S1003; otherwise ("No" in step S1002), the control device 101 transfers the process to step S1004.

[0070] In step S1003, the control device 101 reads the inherent settings of the identified person and controls the display of subsequent screens according to these settings. For example, the control device 101 can prioritize displaying frequently used functions based on the identified person. In step S1004, the control device 101 reads the default settings and controls the display of subsequent screens according to these settings.

[0071] In step S1005, the control device 101 begins tracking the position of the user's hand. Thereafter, the control device 101 continuously detects the position of the user's hand. Additionally, the control device 101 displays a start screen (or home screen) on the display device 105 and displays a cursor indicating the user's indicated position on the display device 105.

[0072] In step S1006, the control device 101 determines whether a selectable object (e.g., an icon) has been selected by the user. This determination can also be based on whether the duration of the state in which the indicated position overlaps with the selectable object has reached a threshold time, as described above.

[0073] If a selectable object is determined to be selected in step S1006 (Yes in step S1006), the control device 101 transfers the process to step S1007; otherwise (No in step S1006), the control device 101 transfers the process to step S1008. In step S1007, the control device 101 performs a process to establish an association with the selectable object.

[0074] In step S1008, the control device 101 determines whether a screen switch has been requested by the user. This determination can also be based on whether the duration of the overlap between the indicated position and the switching area has reached a threshold time, as described above.

[0075] If a screen switching request is determined in step S1008 (yes in step S1008), the control device 101 transfers the processing to step S1009; otherwise (no in step S1008), the control device 101 transfers the processing to step S1010. In step S1008, the control device 101 switches the screen according to the request.

[0076] In step S1010, the control device 101 determines whether the user has performed a sleep gesture. A sleep gesture is a gesture performed by the user to put the display device 105 into a sleep state, and it is predetermined. For example, a sleep gesture could be a gesture of swinging an outstretched hand horizontally.

[0077] If step S1010 determines that the user has performed a sleep gesture ("Yes" in step S1010), the control device 101 transfers the processing to step S1011; otherwise ("No" in step S1010), the control device 101 transfers the processing to step S1006. In step S1011, the control device 101 puts the display device 105 into a sleep state.

[0078] In step S1012, the control device 101 determines whether the user has performed a wake-up gesture. A wake-up gesture is a gesture performed by the user to release the display device 105 from sleep mode, and it is predetermined. For example, a wake-up gesture could be raising a thumb.

[0079] If it is determined in step S1012 that the user has performed a wake-up gesture ("Yes" in step S1012), the control device 101 transfers the processing to step S1013; otherwise ("No" in step S1012), the control device 101 repeats step S1012. In step S1013, the control device 101 activates the display device 105.

[0080] In this way, although the control device 101 detects the wake-up gesture while the display device 105 is in a sleep state, it does not perform the determination in steps S1006 and S1008. Therefore, the user can suppress unintentional processing by putting the display device 105 into a sleep state.

[0081] <Summary of Implementation Methods>

[0082] <Project 1>

[0083] A control device (101) is used to provide a user interface to a user, wherein,

[0084] The control device (101) includes:

[0085] Display control unit (114) which displays selectable objects (501, etc.) on display device (105);

[0086] The detection unit (111) detects the position of the user's hand (300) by analyzing an image of the user's hand;

[0087] Determining unit (112), which determines the indication position in the display device based on the position of the hand; and

[0088] The processing unit (113) performs a process to establish an association with the selectable object based on the fact that the indicated position overlaps with the selectable object.

[0089] The display control unit also displays a cursor (402) indicating the indicated position on the display device.

[0090] According to this project, users can perform desired actions simply by moving their hand to overlap the cursor with the icon. This allows for intuitive input.

[0091] <Project 2>

[0092] According to the control device described in Project 1, the detection unit detects the position of the hand based on the center position of a plurality of feature points (301) contained in the hand.

[0093] According to this project, the hand position can be detected with high accuracy because it is based on multiple feature points.

[0094] <Project 3>

[0095] The control device according to item 1 or 2

[0096] The control device also includes a metering unit (115) that measures the duration of the state in which the indicated position overlaps with the selectable object.

[0097] If the duration reaches a threshold time, the processing unit performs a process to establish an association with the selectable object.

[0098] If the duration does not reach the threshold time, the processing unit does not perform the process of establishing an association with the selectable object.

[0099] According to this project, since no processing is performed before a threshold time has elapsed, it is possible to suppress unintentional user processing.

[0100] <Project 4>

[0101] The control device according to any one of items 1 to 3

[0102] The control device also includes a metering unit (115) that measures the duration of the state in which the indicated position overlaps with the selectable object.

[0103] The display control unit changes the appearance of the screen containing the selectable object according to the duration.

[0104] According to this project, it is possible to grasp the situation and perform processing based on the passage of time.

[0105] <Project 5>

[0106] The control device according to any one of items 1 to 4

[0107] The process of establishing an association with the selectable object includes switching the display of a first screen containing the selectable object to the display of a second screen containing the second selectable object (511).

[0108] If the indicated position at the start display time of the second screen overlaps with the second selectable object, the processing unit considers the indicated position not to overlap with the second selectable object until a predetermined condition is met.

[0109] According to this project, it is possible to suppress unintentional processing by users.

[0110] <Project 6>

[0111] The control device according to any one of items 1 to 4

[0112] The process of establishing an association with the selectable object includes switching the display of a first screen containing the selectable object to the display of a second screen containing a second selectable object.

[0113] If the indicated position at the start display time of the second screen overlaps with the second selectable object, the processing unit moves the indicated position to a position in the second screen that does not overlap with any selectable object.

[0114] According to this project, it is possible to suppress unintentional processing by users.

[0115] <Project 7>

[0116] The control device according to any one of items 1 to 4

[0117] The process of establishing an association with the selectable object includes switching the display of a first screen containing the selectable object to the display of a second screen containing a second selectable object.

[0118] If the indicated position at the start of the second screen overlaps with the position where the second selectable object is displayed, the processing unit displays the second selectable object at a position in the second screen that does not overlap with the indicated position.

[0119] According to this project, it is possible to suppress unintentional processing by users.

[0120] <Project 8>

[0121] The control device according to any one of items 1 to 7

[0122] The first screen containing the selectable object includes a first switching position (802b) and a second switching position (802a) that is closer to the end of the first screen than the first switching position.

[0123] If the duration of the overlap between the indicated position and the first switching position reaches a second threshold time, the display control unit switches the display of the first screen to the display of the third screen.

[0124] If the duration of the overlap between the indicated position and the second switching position reaches a third threshold time, the display control unit switches the display of the first screen to the display of the third screen.

[0125] The third threshold time is shorter than the second threshold time.

[0126] According to this project, unintentional screen switching by users is prevented.

[0127] <Project 9>

[0128] The control device according to any one of items 1 to 8,

[0129] The display control unit displays a first screen containing the selectable object and the third selectable object (902) on the display device.

[0130] The process of establishing an association with the third selectable object includes displaying a fourth screen (910) on the display device, representing a plurality of functions that the user can perform.

[0131] According to this project, users can easily master multiple functions.

[0132] <Project 10>

[0133] The control device according to any one of items 1 to 9

[0134] The control device also includes a recognition unit (116) that identifies the user as a specific person based on the image of the hand.

[0135] The display control unit controls the display of the screen according to the inherent settings of the specific character.

[0136] According to this project, user convenience is improved by implementing display controls that are customized by each user.

[0137] <Project 11>

[0138] A program that enables a computer to function as a unit of a control device as described in any one of items 1 to 10.

[0139] According to this project, a program with the aforementioned effects is provided.

[0140] This invention is not limited to the embodiments described above, and various modifications and alterations can be made within the scope of the spirit of this invention.

Claims

1. A control device for providing a user interface to a user, wherein, The control device includes: The display control unit displays selectable objects on the display device; The detection unit detects the position of the user's hand by analyzing an image of the user's hand; A determining unit that determines the indication position in the display device based on the position of the hand; as well as The processing unit, based on the fact that the indicated location overlaps with the selectable object, performs a process to establish an association with the selectable object. The display control unit also displays a cursor on the display device indicating the indicated position. The first screen containing the selectable object includes a first switching position and a second switching position that is closer to the end of the first screen than the first switching position. If the duration of the overlap between the indicated position and the first switching position reaches a second threshold time, the display control unit switches the display of the first screen to the display of the third screen. If the duration of the overlap between the indicated position and the second switching position reaches a third threshold time, the display control unit switches the display of the first screen to the display of the third screen. The third threshold time is shorter than the second threshold time.

2. The control device according to claim 1, wherein, The detection unit detects the position of the hand based on the center position of multiple feature points contained in the hand.

3. The control device according to claim 1, wherein, The control device also includes a metering unit that measures the duration of the overlap between the indicated position and the selectable object. If the duration reaches a threshold time, the processing unit performs a process to establish an association with the selectable object. If the duration does not reach the threshold time, the processing unit does not perform the process of establishing an association with the selectable object.

4. The control device according to claim 1, wherein, The control device also includes a metering unit that measures the duration of the overlap between the indicated position and the selectable object. The display control unit changes the appearance of the screen containing the selectable object according to the duration.

5. The control device according to claim 1, wherein, The process of establishing an association with the selectable object includes switching the display of a first screen containing the selectable object to the display of a second screen containing a second selectable object. If the indicated position at the start display time of the second screen overlaps with the second selectable object, the processing unit considers the indicated position not to overlap with the second selectable object until a predetermined condition is met.

6. The control device according to claim 1, wherein, The process of establishing an association with the selectable object includes switching the display of a first screen containing the selectable object to the display of a second screen containing a second selectable object. If the indicated position at the start display time of the second screen overlaps with the second selectable object, the processing unit moves the indicated position to a position in the second screen that does not overlap with any selectable object.

7. The control device according to claim 1, wherein, The process of establishing an association with the selectable object includes switching the display of a first screen containing the selectable object to the display of a second screen containing a second selectable object. If the indicated position at the start of the second screen overlaps with the position where the second selectable object is displayed, the processing unit displays the second selectable object at a position in the second screen that does not overlap with the indicated position.

8. The control device according to claim 1, wherein, The display control unit displays a first screen containing the selectable object and the third selectable object on the display device. The process of establishing an association with the third selectable object includes displaying a fourth screen on the display device, representing multiple functions that the user can perform.

9. The control device according to claim 1, wherein, The control device also includes a recognition unit that identifies the user as a specific person based on the image of the hand. The display control unit controls the display of the screen according to the inherent settings of the specific character.

10. A storage medium storing a program for enabling a computer to function as a unit of the control device according to any one of claims 1 to 9.

Images