Computer program, image processing device, image processing method, and in-vehicle interface device

Abstract

To provide a computer program, image processing device, image processing method and vehicle-mounted interface device, which provide a contactless user interface tailored to the subjectivity of a user.SOLUTION: A computer program disclosed herein makes a computer perform processing for acquiring an image capturing multiple indicators, and setting a pointer for specifying a position represented by an end of a first indicator and an end of a second indicator at a position between the end of the first indicator and the end of the second indicator identified on the basis of the acquired image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a computer program, an image processing apparatus, an image processing method, and an in-vehicle interface apparatus.

Background Art

[0002] In recent years, research and development of non-contact user interfaces that respond to operations using a human hand or body have been active. In Patent Document 1, when a user approaches a display with a finger, an image is captured by a camera, and based on the captured image, the operation intention of the user is estimated based on the gesture of the user's finger approaching the display and the position of the finger with respect to the display, and an apparatus that controls an image output unit to change a display image according to the estimated operation intention is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Although the apparatus of Patent Document 1 provides a non-contact user interface function, the user needs to perform an operation of approaching a finger to the display, and there is a limitation that the user must be close enough to touch the display. Further, when the operation intention of the user is estimated based on the gesture of the finger and the position of the finger with respect to the display, the movement of the finger and the position of the finger may not match the user's subjectivity, and there is a possibility that the user may feel uncomfortable.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to provide a computer program, an image processing apparatus, an image processing method, and an in-vehicle interface apparatus that can realize a non-contact user interface adapted to the user's subjectivity. [Means for solving the problem]

[0006] The present invention includes several means for solving the above-mentioned problems, but to give one example, the computer program causes the computer to acquire an image in which multiple indicators are photographed, and to execute a process that sets a pointer to specify a position represented by the ends of the first and second indicators at a position between the end of a first indicator and the end of a second indicator, which are identified based on the acquired image. [Effects of the Invention]

[0007] According to the present invention, a non-contact user interface tailored to the user's subjective experience can be realized. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the configuration of a contactless user interface system. [Figure 2] This figure shows an example of hand movements. [Figure 3] This figure shows an example of the configuration of an image processing device. [Figure 4] This figure shows an example of the configuration of the image processing unit. [Figure 5] This figure shows an example of an image of fingers. [Figure 6] This diagram shows how to explore the fingertips of your right hand. [Figure 7] This diagram shows how to explore the fingertips of your left hand. [Figure 8] This figure shows an example of a classification of finger morphology. [Figure 9] This figure shows an example of a combination of fingertip position and pointer position. [Figure 10] This figure shows an example of the configuration of the target detection unit in Embodiment 2. [Figure 11] This diagram shows how to determine the midpoint between two fingers. [Figure 12] This figure shows an example of the processing procedure of the image processing unit in Embodiment 2. [Figure 13] It is a diagram showing an example of the configuration of the target detection unit of Embodiment 3. [Figure 14] It is a diagram showing a method of determining the abdominal part of the thumb. [Figure 15] It is a diagram showing an example of the processing procedure of the image processing unit of Embodiment 3. [Figure 16] It is a diagram showing the setting of the fingertip position of Application Form 1 of Embodiment 3. [Figure 17] It is a diagram showing the selection of the thumb in Application Form 2 of Embodiment 3. [Figure 18] It is a diagram showing the adjustment of the fingertip position in Application Form 3 of Embodiment 2 or 3. [Figure 19] It is a diagram showing an example of the configuration of the target detection unit of Embodiment 4. [Figure 20] It is a diagram showing an example of the configuration of the pointer information generation unit of Embodiment 5. [Figure 21] It is a diagram showing an example of the range adjustment of the moving average. [Figure 22] It is a diagram showing an example of the processing procedure of the image processing unit of Embodiment 5. [Figure 23] It is a diagram showing an example of the configuration of the image processing unit of Embodiment 6. [Figure 24] It is a diagram showing an example of the configuration of the search axis correction unit. [Figure 25] It is a diagram showing a method of correcting the search axis. [Figure 26] It is a diagram showing an example of the configuration of the image processing unit of Embodiment 7. [Figure 27] It is a diagram showing an example of the configuration of the tracking detection unit. [Figure 28] It is a diagram showing an example of the detection omission of the fingertip tip. [Figure 29] It is a diagram showing an example of the configuration of the target tip detection unit of Embodiment 8. [Figure 30] It is a diagram showing a method of detecting the fingertip tip. [Figure 31] It is a diagram showing an example of an in-vehicle interface device.

Modes for Carrying Out the Invention

[0009] Embodiments of the present invention will be described below with reference to the drawings.

[0010] (Embodiment 1) Figure 1 shows an example of the configuration of a non-contact user interface system 100. The non-contact user interface system 100 comprises a camera 10, a display device 20 having a display screen, and an image processing device 50. The image processing device 50 includes an image processing unit 60 that functions as a non-contact user interface device. The image processing device 50 can be configured as, for example, a personal computer. The image processing unit 60 is not limited to a configuration built into the image processing device 50, and may be configured as a separate device from the image processing device 50. The camera 10 is not limited to general images obtained by visible light or infrared light, and may also be a TOF (Time Of Flight) camera that can also measure distance.

[0011] Camera 10 is positioned to photograph the black plate. The position of camera 10 can be set as appropriate, as long as it is in a position that can photograph the black plate. At an appropriate position between camera 10 and the black plate, the user performs actions such as pinching by bringing their thumb and index finger together, or separating their thumb and index finger. Camera 10 can photograph the movements and state of the user's fingers. The black plate corresponds to the background portion of the image of the fingers other than the fingers, but in this embodiment, the black plate is not essential. Also, the color of the plate is not limited to black and may be any color. However, in order to make the portion of the image corresponding to the fingers stand out against the background, it is preferable that the plate be a single color and not have a complex pattern.

[0012] The image processing device 50 can display images of fingers captured by the camera 10 as they are, or images of fingers that have been processed, on the display screen of the display device 20. When the user moves their fingers horizontally relative to the black plate, the fingers displayed on the display screen can also move horizontally (vertically and horizontally) on the display screen. Also, when the user moves their fingers vertically relative to the black plate, the fingers displayed on the display screen change to virtually move vertically relative to the display screen. Images of fingers that have been processed are, for example, images that display symbols such as dots or circles instead of fingers.

[0013] As shown in Figure 1, when the user pinches (grabs) their thumb and index finger over the number key they want to input, the image processing device 50 determines that the number key has been selected by the pinch of the thumb and index finger and displays the number in the number input field on the display screen of the display device 20. As shown in Figure 1, in addition to the number keys from 0 to 9, the numeric keypad also has a delete key indicated by "DEL" and a clear key indicated by "CLR".

[0014] Figure 2 shows an example of finger movement. Figure 2 shows the action of selecting a number key by pinching with the thumb and index finger. When the thumb and index finger are separated, the pointer position (for example, the green indicator) is displayed between the thumb and index finger. By placing the pointer position over the key to be selected (the delete key in the figure) and pinching with the thumb and index finger, the delete key is selected, and the "2" in the number "9632" in the number input field on the display screen of Figure 1 can be deleted. By placing the pointer position between the thumb and index finger, even small keys can be easily selected.

[0015] Figure 2 shows an example where numbers are displayed in the number input field on the display screen of the display device 20 by pinching with the thumb and index finger of the right hand. However, numbers may also be displayed in the number input field on the display screen of the display device 20 by pinching with the thumb and index finger of the left hand.

[0016] The action of pinching with the fingertips by bringing the thumb and index finger into contact is a familiar action to everyone and is suitable for selection operations. Therefore, this embodiment uses the thumb and index finger to perform a pinching motion as a selection operation. In this embodiment, the object represented by the fingertips is called a referent, and as an example of a referent, the fingers are given as an example, with the thumb and index finger (first finger and second finger) being used as a particular example. In this case, both the fingertips of the thumb and the index finger, which correspond to the ends of the referent, are treated as a single unit of fingertips and are designated as a fingertip position representing the fingertips. The pointer position is determined based on this fingertip position.

[0017] The pointer position can be determined based on the fingertip position. The pointer position may be determined, for example, within the range from the tips of two fingers (e.g., the thumb and index finger) to the first joint, or more preferably, within the range on the inside of the two fingers up to the base of the nail, and the two fingers may be fingers other than the thumb and index finger. The pointer position will be described in detail below.

[0018] Figure 3 shows an example of the configuration of the image processing device 50. The image processing device 50 comprises a control unit 51 that controls the entire device, a communication unit 52, a memory 53, an operation unit 54, a storage unit 55, a recording medium reading unit 58, and an image processing unit 60. The storage unit 55 can be configured as, for example, a hard disk or semiconductor memory, and stores a computer program 56, a learning model 57, and necessary information. The image processing device 50 may be configured as multiple devices by distributing the processing functions.

[0019] The control unit 51 is configured with the required number of CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), etc. The control unit 51 can execute the processing defined in the computer program 56. In other words, the processing performed by the control unit 51 is also the processing performed by the computer program 56. By executing the computer program 56, the control unit 51 can perform the functions of the image processing unit 60. The image processing unit 60 may be configured as hardware, implemented as software, or implemented as a combination of hardware and software. The control unit 51 and the image processing unit 60 can perform processing using the learning model 57.

[0020] The communication unit 52 may, for example, include a communication module and be able to communicate with the display device 20. The communication unit 52 may also have a communication interface function with the camera 10 or other interface devices.

[0021] The memory 53 can be composed of semiconductor memory such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory. By loading the computer program 56 into the memory 53, the control unit 51 can execute the computer program 56.

[0022] The recording medium 59 on which the computer program 56 is recorded can be read by the recording medium reading unit 58. Alternatively, the computer program 56 may be downloaded from an external device via the communication unit 52 and stored in the storage unit 55.

[0023] The control unit 54 is equipped with an interface function for input devices such as a mouse and a keyboard.

[0024] Figure 4 shows an example of the configuration of the image processing unit 60. The image processing unit 60 is equipped with functions for realizing the functionality of a non-contact user interface, and specifically includes a target detection unit 61, an image cropping unit 62, a shape recognition unit 63, a pointer selection unit 64, a pointer information generation unit 65, and a pointer information transmission unit 66. The processing by the image processing unit 60 is performed similarly on general images obtained by visible light or infrared light, as well as on distance images obtained by a TOF camera.

[0025] The target detection unit 61 acquires an image of the fingers (an image in which multiple points are captured) taken by the camera 10. Based on the image of the fingers, the target unit 61 detects the thumb and index finger, which are the target areas, and outputs the position of the fingertips on the image and the size of the region including the fingertips.

[0026] Figure 5 shows an example of a finger image. While Figure 5 illustrates a right hand, a left hand image may also be used. The area outside the fingers is the background and appears black when a black plate is used. If a plate is not used, the image may be processed to make the area outside the fingers a single color (e.g., white, gray, etc.).

[0027] The target detection unit 61 searches the image of the fingers along two diagonal search axes in different directions to detect the fingertips. By searching using two search axes, the unit scans downwards to the right for the right hand and downwards to the left for the left hand, and the first point detected can be considered the fingertip.

[0028] Figure 6 shows a method for searching for the fingertips of the right hand, and Figure 7 shows a method for searching for the fingertips of the left hand. The first search axis is an axis that points diagonally downward to the right and has an angle θ1 with the vertical direction on the image. The second search axis is an axis that points diagonally downward to the left and has an angle θ2 with the vertical direction on the image. As shown in Figure 6, when searching the image of the right hand using the first search axis, it is determined whether or not fingertips exist along the direction perpendicular to the search point of the first search axis. If fingertips do not exist, the search point is moved along the first surge axis, and the same process is repeated to detect the fingertips of the right hand. Similarly, as shown in Figure 7, when searching the image of the left hand using the second search axis, it is determined whether or not fingertips exist along the direction perpendicular to the search point of the second search axis. If fingertips do not exist, the search point is moved along the second surge axis, and the same process is repeated to detect the fingertips of the left hand. By performing the search using these two search axes, it is possible to detect fingertips on both the right and left hands (in Figures 6 and 7, the detected fingertips are indicated by black dots).

[0029] On the other hand, when searching the image of the right hand using the second search axis, or the image of the left hand using the first search axis, areas other than the fingertips (indicated by black dots) are detected (false detection). However, these areas other than the fingertips can be excluded by the processing of the pointer selection unit 64, which will be described later. To search for fingertips along the search axis, the angles θ1 and θ2 of the search axis should be set to the required range including 60° (for example, about 45° to 75°).

[0030] The target detection unit 61 searches using two search axes, the first search axis and the second search axis, and outputs the detected position of the fingertip on the image and the size of the region containing the fingertip to the image cropping unit 62. The size is a predetermined value and defines the region to be cropped by the image cropping unit 62. The first cropped rectangular region and the second cropped rectangular region shown in Figures 6 and 7 correspond to the region to be cropped by the image cropping unit 62. Note that there may be three or more search axes; for example, there may be three search axes consisting of 60° from the left and right plus 0° from directly above.

[0031] The image extraction unit 62 extracts a partial image including the fingertip from the hand image output from the camera 10, based on the detected fingertip position and the size of the area including the fingertip output from the target detection unit 61. The two extracted partial images are partial images that include the first extracted rectangular area and the second extracted rectangular area in Figure 6 or Figure 7, respectively. The image extraction unit 62 outputs the partial image to the shape recognition unit 63.

[0032] The shape recognition unit 63 performs shape recognition processing using the learning model 57. When a partial image is input, the shape recognition unit 63 classifies the shapes of the fingers shown in the partial image and outputs the classified shapes and the detected positions of the fingertips. The learning model 57 is a model that uses a machine learning algorithm, and can use, for example, SVM (Support Vector Machine), decision tree, random forest, neural network, etc. Deep learning can be used, for example, CNN (Convolutional Neural Network), and structures such as AlexNet, VGG, GoogLeNet, and ResNet can be used.

[0033] Figure 8 shows an example of classifying the morphology of fingers. Figure 8 shows the morphology classification performed by the morphology recognition unit 63. As shown in Figure 8, the morphology can be classified into 13 classes, such as class 0, 1, 2, ..., 12. Note that the number of class classifications and the content of the morphology are not limited to the example in Figure 8.

[0034] In Class 0, the finger configuration is such that the thumb and index finger are separated, indicating that the pointer is not selected. In Class 1, the finger configuration is such that the fingertips of the thumb and index finger are in contact, indicating that the pointer is selected. The thumb and index finger are the indicators, with the unselected state being the configuration before indication and the selected state being the configuration during indication.

[0035] Classes 2 and 3 indicate a hand shape consisting of one finger (either the thumb or index finger) and is not a pointer. Classes 4 and 5 indicate a hand shape consisting of two fingers (index and middle fingers) and is not a pointer. Note that in Class 4, there is a gap between the two fingers, while in Class 5, there is no gap between the two fingers. Classes 6 and 7 indicate a hand shape consisting of three fingers (index, middle, and ring fingers) and is not a pointer. In Class 6, there is a gap between the three fingers, while in Class 7, there is no gap between the three fingers. Class 8 indicates a hand shape consisting of four fingers (index, middle, ring, and little fingers) and is not a pointer. Class 9 shows a lateral angle of the base of the fingers and is not a pointer. Class 10 indicates a hand shape consisting of a clenched fist and is not a pointer. Class 11 shows a hand shape other than those in Classes 0-10 and is not a pointer. Class 12 indicates a shape other than fingers, signifying that it is not a pointer. Note that while Figure 8 shows partial images of the right thumb and index finger for Classes 0 and 1, partial images of the left thumb and index finger can be similarly classified.

[0036] The pointer selection unit 64 extracts only those shapes that correspond to pointer shapes from the partial image, based on the classified shapes and fingertip detection positions output by the shape recognition unit 63, and outputs the corresponding class and fingertip position. In this embodiment, only classes 0 and 1 are used as pointers, and if shapes of other classes are output from the shape recognition unit 63, the pointer selection unit 64 ignores the output of the shape recognition unit 63 and outputs nothing. This allows all fingertips that were erroneously detected by the target detection unit 61 to be excluded.

[0037] The pointer information generation unit 65 generates information indicating whether the pointer is selected or not (information indicating the pointer state) based on whether the class output from the pointer selection unit 64 is class 0 or class 1, and outputs the pointer position and information indicating the pointer state based on the fingertip position (detection position). The pointer position illustrated in Figure 2 (the position of the green indicator between the thumb and index finger) can be determined by the pointer information generation unit 65. The pointer information transmission unit 66 outputs pointer information indicating the pointer position and pointer state to the control unit 51. The control unit 51 provides the information indicating the pointer state to an application, etc. This generates display information such as symbols. Alternatively, the control unit 51 can output the pointer information to a host device or the like outside the contactless user interface.

[0038] As described above, when an image of fingers is input, the control unit 51 (or image processing unit 60) inputs an image of fingers to a learning model (form recognition unit 63) that has been generated to output classification information including whether it is a first form (class 1) in which the first finger and the second finger are in contact, or a second form (class 0) in which the first finger and the second finger are separated, and can switch the display mode of the pointer located between the fingertips of the first finger and the second finger according to the classification information output by the learning model.

[0039] (Embodiment 2) In the aforementioned Embodiment 1, the tip of the fingertip detected by the target detection unit 61 was defined as the fingertip position, and the pointer position was determined from there. Embodiment 2 describes the detection of the fingertip position in accordance with the subjective perception of the user (human).

[0040] Figure 9 shows an example of a combination of fingertip position and pointer position. Figure 9A shows the thumb and index finger separated, and Figure 9B shows the thumb and index finger in contact. Both Figures 9A and 9B show the state where the tip of one of the two fingertips is designated as the fingertip position, and the pointer is superimposed on it (referred to as the pointer position). In Figures 9A and 9B, the pointer position is at the tip of one of the fingertips, which can feel unnatural to the user. Furthermore, the pointer position changes relatively significantly depending on which of the two fingertips is selected as the fingertip position, which can further cause discomfort to the user.

[0041] On the other hand, as shown in Figures 9C and 9D, by placing the fingertip between two fingers, the problem of the pointer position changing relatively significantly can be resolved, eliminating user discomfort. Furthermore, compared to using the tip of one of the two fingertips as the fingertip position, discomfort is significantly reduced, and the fingertip position can be adjusted to suit the user's subjective preference. A detailed explanation follows below.

[0042] Figure 10 shows an example of the configuration of the target detection unit 61 of Embodiment 2, and Figure 11 shows a method for determining the midpoint of two fingers. The target detection unit 61 comprises a target tip detection unit 611, an offset calculation unit 612, a fingertip area end-edge detection unit 613, an area midpoint calculation unit 614, and a fingertip position calculation unit 615.

[0043] The target tip detection unit 611 searches the image of the fingers acquired from the camera 10 in the direction of the search axis. If multiple fingertips are present in the image, it detects the fingertip corresponding to the upstream side of the search axis as the fingertip tip and outputs the detected fingertip tip position (coordinates on the image) to the fingertip region end detection unit 613. In the example in Figure 11, there are two fingertips, the thumb and the index finger, and of these, the fingertip of the index finger, which is the upstream side of the search axis, is detected as the fingertip tip.

[0044] The offset calculation unit 612 calculates an offset vector (the vector shown as "Offset" in Figure 11) by multiplying the unit vector in the search axis direction by a predetermined coefficient, based on the information in the search axis direction used by the target tip detection unit 611.

[0045] The fingertip region end detection unit 613 sets a vertical axis in a direction perpendicular to the search axis (perpendicular direction), passing through the endpoint position obtained by adding an offset vector starting from the fingertip position, and detecting the outer boundaries of the thumb and index finger (the boundary of both ends of the finger region including the thumb and index finger) on the set vertical axis. In Figure 11, the outer boundary of the thumb and the outer boundary of the index finger are detected on the vertical axis.

[0046] The region midpoint calculation unit 614 calculates the position (coordinates) of the midpoint of the outer boundary (both end boundaries).

[0047] The fingertip position calculation unit 615 determines a position represented by the ends of the first and second indicators as a designated position, which is located between the ends of the first and second indicators identified based on the acquired image. The pointer can then be set based on this fingertip position.

[0048] Specifically, the fingertip position calculation unit 615 calculates the fingertip position by subtracting the offset vector calculated by the offset calculation unit 612 from the midpoint calculated by the area midpoint calculation unit 614 as the starting point. In Figure 11, the fingertip position is set to a position between the fingertip of the thumb and the fingertip of the index finger. Specifically, the fingertip position is set to an intermediate position between the fingertip of the thumb and the fingertip of the index finger. The pointer position may also be set to the set fingertip position.

[0049] As described above, the control unit 51 (or image processing unit 60) can acquire an image in which multiple indicators are captured, and set a pointer that specifies a position represented by the ends of the first and second indicators at a position between the end of the first indicator and the end of the second indicator, which are identified based on the acquired image. In this case, the image in which multiple indicators are captured may be an image captured by a TOF camera.

[0050] More specifically, the image captured by the multiple indicators is an image of a hand, the ends of the first indicator and the ends of the second indicator are the fingertips of the first and second fingers, and the control unit 51 (or image processing unit 60) can set a pointer at an intermediate position between the fingertips of the first and second fingers.

[0051] By positioning the fingertip between two fingers, the problem of the pointer position changing relatively drastically depending on which of multiple fingers is selected as the pointer can be resolved, thus eliminating user discomfort. Furthermore, compared to using the tip of one of the two fingertips as the fingertip position, discomfort is significantly reduced, and the fingertip position can be adjusted to the user's subjective preference.

[0052] Furthermore, the control unit 51 (or image processing unit 60) can search the acquired image of the fingers along a predetermined search axis to detect the tip of the fingertip, set an orthogonal axis perpendicular to the search axis at a predetermined distance from the detected tip, search along the set orthogonal axis to detect the outer boundaries of the first and second fingers, identify the midpoint of the detected outer boundaries, and set a pointer based on the identified midpoint.

[0053] For example, the control unit 51 (or image processing unit 60) may set the pointer at a position offset by a predetermined distance from the identified midpoint in a direction parallel to the search axis. This allows for setting a pointer position that feels more natural to the user.

[0054] The control unit 51 (or image processing unit 60) can display the set fingertip position on the application screen displayed on the display device 20. In this case, the fingertip position may be displayed on the application screen by superimposing symbols such as points or circles representing the fingertip position onto the image of the fingertip portion of the image acquired from the camera 10, or only symbols such as points or circles representing the fingertip position may be displayed on the application screen. Alternatively, the entire image of the fingers acquired from the camera 10 may be overlaid and made semi-transparent. The image to be overlaid may not be the image of the fingers acquired from the camera 10, but rather a representative fingertip image (real or illustrated) prepared in advance, selected according to the shape of the recognized fingertip. The image of the fingers acquired from the camera 10 may also be displayed as a wipe image in a small size on a part of the display screen.

[0055] Figure 12 shows an example of the processing procedure of the image processing unit 60 in Embodiment 2. In the following description, the main processing unit will be referred to as the image processing unit 60 for convenience. The image processing unit 60 acquires an image of the fingers from the camera 10 (S11), and searches the acquired image in the direction of the search axis to detect the fingertip (S12). The image processing unit 60 calculates an offset vector along the direction of the search axis (S13). The image processing unit 60 sets a vertical axis perpendicular to the search axis from the position obtained by adding the offset vector to the fingertip position (S14).

[0056] The image processing unit 60 searches along the set vertical axis to detect the outer boundary (external boundary) of the finger (for example, two fingers, the thumb and index finger) (S15). The image processing unit 60 identifies the midpoint of the outer boundary of the two fingers and calculates the fingertip position (S16). The image processing unit 60 extracts a partial image of the fingertip position from the image of the hand (S17).

[0057] The image processing unit 60 classifies the partial image (S18) and determines whether the classified shape corresponds to a pointer shape (shape) (S19). If it does not correspond to a pointer shape (NO in S19), the image processing unit 60 continues processing from step S11 onwards. If it corresponds to a pointer shape (YES in S19), the image processing unit 60 generates information indicating the state of the pointer (information indicating the pointer's position and state) (S20), outputs the generated pointer information (S21), and terminates processing.

[0058] As described above, according to Embodiment 2, in a state where two fingers are close together but not touching (a state where they are separated), the fingertip position representing the fingertips of the two fingers can be set to a position between the two fingers, for example, in the middle of the two fingers, thereby creating a fingertip position that matches the user's subjective experience. This makes it possible to realize a non-contact user interface that matches the user's subjective experience.

[0059] (Embodiment 3) When performing a pinch gesture, the movement of the index finger tends to be greater than that of the thumb. To match the user's sense of movement, it is desirable that the fingertip position does not change significantly before and after a pinch gesture. Therefore, in Embodiment 3, instead of setting the fingertip position between the two fingers as in Embodiment 2, a configuration in which the fingertip position is set on the thumb, which has relatively smaller movement, will be described.

[0060] Figure 13 shows an example of the configuration of the target detection unit 61 in Embodiment 3, and Figure 14 shows a method for determining the thumb pad portion. Comparing Embodiment 3 with Embodiment 2, the target detection unit 61 includes a fingertip edge detection unit 616, a thumb selection unit 617, and a thumb pad position calculation unit 618, instead of the fingertip area edge detection unit 613 and the area midpoint calculation unit 614. The target tip detection unit 611, offset calculation unit 612, and fingertip position calculation unit 615 are the same as in Embodiment 2.

[0061] The fingertip end detection unit 616 scans the vertical axis in both directions by a predetermined width to detect the boundary of each of the two fingers (thumb and index finger). As shown in Figure 14, the fingertip end detection unit 616 detects the boundary P1 and P2 of one finger (thumb) and the boundary P3 and P4 of the other finger (index finger).

[0062] The thumb selection unit 617 compares the distance between the boundaries of both ends of each of the two fingers and selects the finger with the longer distance as the thumb. Generally, the thumb is thicker than the index finger, so it is possible to select the thumb based on the distance between the boundaries of the two fingers. In the example in Figure 14, the distance between the boundaries P1 and P2 is longer than the distance between the boundaries P3 and P4, so the finger on the P1-P2 side can be selected as the thumb.

[0063] The thumb pad position calculation unit 618 selects the boundary between the two ends of the finger corresponding to the thumb, which is closer to the other finger (P2 in the example in Figure 14), from among the boundary between both ends of the fingertip detected by the fingertip end detection unit 616, and sets it as the thumb pad position.

[0064] The fingertip position calculation unit 615 calculates the fingertip position by subtracting an offset vector from the position of the thumb pad (P2 in the example in Figure 14).

[0065] The control unit 51 (or image processing unit 60) can set the pointer to the pad of the thumb. By setting the fingertip position to the thumb, which has relatively small movement, it can be matched to the user's sense of movement.

[0066] Figure 15 shows an example of the processing procedure of the image processing unit 60 in Embodiment 3. In the following description, the main processing unit will be referred to as the image processing unit 60 for convenience. The image processing unit 60 acquires an image of the fingers from the camera 10 (S31), and searches the acquired image in the direction of the search axis to detect the fingertip (S32). The image processing unit 60 calculates an offset vector along the direction of the search axis (S33). The image processing unit 60 sets a vertical axis perpendicular to the search axis from the position obtained by adding the offset vector to the fingertip position (S34).

[0067] The image processing unit 60 searches along the set vertical axis to detect the boundary positions of both ends of each of the two fingers (S35). The image processing unit 60 compares the distance between the boundary positions of both ends of each of the two fingers, selects the finger with the longer distance as the thumb, calculates the boundary position of the thumb pad from the boundary position of the thumb that is closer to the other finger, and calculates the fingertip position by subtracting an offset vector from the thumb pad position (S36). The image processing unit 60 extracts a partial image of the fingertip position from the image of the fingers (S37).

[0068] The image processing unit 60 classifies the partial image (S38) and determines whether the classified shape corresponds to a pointer shape (shape) (S39). If it does not correspond to a pointer shape (NO in S39), the image processing unit 60 continues processing from step S31 onwards. If it corresponds to a pointer shape (YES in S39), the image processing unit 60 generates information indicating the state of the pointer (information indicating the pointer's position and state) (S40), outputs the generated pointer information (S41), and terminates processing.

[0069] As described above, according to Embodiment 3, when two fingers are close together but not touching (or separated), the fingertip position representing the fingertips of the two fingers is set based on the position of the thumb pad. This makes it possible to relatively reduce the change in fingertip position before and after a pinch motion, and to set the fingertip position to match the user's subjective perception. This makes it possible to realize a non-contact user interface that matches the user's subjective perception.

[0070] (Applications of Embodiments 2 and 3) Figure 16 shows the setting of the fingertip position in Application Form 1 of Embodiment 3. The fingertip position setting method shown in Figure 14 was configured to calculate the fingertip position by subtracting an offset vector from the thumb pad position (P2 in the example in Figure 14). In the example shown in Figure 16, the fingertip position is calculated by subtracting an offset vector from a desired position in the fingertip position placement interval between the midpoint on the vertical axis between the boundary P1 of both ends of the thumb and the boundary P3 of both ends of the index finger, and the thumb pad position P2. This allows the fingertip position to be set while balancing the thumb pad position and the midpoint according to the user's subjective preference.

[0071] Figure 17 shows the thumb selection method for application form 2 of Embodiment 3. The thumb selection method shown in Figure 14 involved comparing the thickness (distance between the boundaries of both ends) of two fingers. In the example shown in Figure 17, the curvature of the tips of the two fingers is compared, and the finger with the larger curvature (radius of curvature) is selected as the thumb.

[0072] The thumb selection unit 617 can be configured using a learning model. This learning model is generated by machine learning using images of the tip of the thumb, images of the tip of the index finger, and training data indicating whether an image is a thumb or an index finger. Alternatively, the system may perform skeletal recognition of the entire palm before selecting the thumb.

[0073] Figure 18 shows the adjustment of the fingertip position in application form 3 of Embodiment 2 or 3. Starting from the initial fingertip position, an adjustment vector (offset vector) is added to offset the position by a required distance in the search axis direction, and the final position is the offset fingertip position. By offsetting the fingertip position by a required distance from the initial fingertip position, a pointer position that feels more natural to the user can be set. The required offset distance may be a predetermined fixed value, but it may also be increased or decreased according to the thickness of the finger or the size of the cropped image (partial image). Furthermore, to reflect the user's preference, the system may accept an operation to set the required distance.

[0074] The control unit 51 (or image processing unit 60) may accept an operation to set the amount of movement required to move the set pointer position to the desired position. This allows the user to set a pointer position that feels more natural.

[0075] Although not shown in the diagram, the control unit 51 (or image processing unit 60) may also display multiple pointer positions as selectable candidate positions and accept an operation to select a pointer position from the displayed candidate positions. This allows users to set a pointer position that feels more natural according to their individual preferences.

[0076] When adjusting the fingertip position, the addition of the adjustment vector may be performed by the target detection unit 61 or by the pointer information generation unit 65. In this case, the target detection unit 61 outputs information regarding the direction of the search axis to the pointer information generation unit 65. The pointer information generation unit 65 obtains the class classification result of whether the shape of the fingers is class 0 or class 1, so it becomes possible to fine-tune the fingertip position according to the shape, for example, by reducing the adjustment amount (movement distance) in the case of a shape where the thumb and index finger are in contact.

[0077] (Embodiment 4) In the above-described embodiment, a cropped image (partial image) including the recognized fingertip position is input to the learning model used to recognize the shape of the fingers. From the viewpoint of improving recognition accuracy, it is desirable that the size of the input partial image is the same as the size of the training partial image used to train the learning model. For example, if the range is extremely narrow and the entire fingers are not captured, it becomes difficult to distinguish them as fingertips. Conversely, if the range is too wide, the fingers (especially the fingertips) that are the target of recognition will appear small, and will differ from the pattern of the partial images used in the training data. On the other hand, the fingertips that appear in the camera's view change greatly depending on the positional relationship between the fingertips and the camera. If the size of the cropped image is fixed, the spatial range of the recognizable fingertips may fluctuate depending on the positional relationship between the camera and the fingertips (fingers). Embodiment 4 describes a method for improving the accuracy of fingertip shape recognition.

[0078] Figure 19 shows an example of the configuration of the target detection unit 61 of Embodiment 4. The difference from the target detection unit 61 of Embodiment 2 shown in Figure 10 is that it includes a tip distance detection unit 619 and a size calculation unit 610. The tip distance detection unit 619 acquires a distance image from a TOF camera (camera 10). The distance image contains distance information for each pixel. When using a TOF camera, the learning model used by the shape recognition unit 63 can be machine-learned using the distance image instead of the image of the fingers. Alternatively, a distance measuring sensor using a laser, sound waves, etc., can be used instead of a TOF camera, as long as it can measure distance. However, when using such sensors, it is necessary to use a camera to obtain an image of the fingers in conjunction with them.

[0079] The tip distance detection unit 619 acquires information regarding the tip position detected by the target tip detection unit 611. The tip distance detection unit 619 calculates the sum of the distances of each pixel in the region near the tip position on the depth image, and calculates the average value of the distances in the region by dividing the calculated sum by the number of pixels in the region. The calculated average value is then output to the size calculation unit 610 as the tip distance.

[0080] The size calculation unit 610 calculates the size of the cropped image based on the tip distance obtained from the tip distance detection unit 619. The size of the cropped image can be calculated, for example, by multiplying the reciprocal of the tip distance by a predetermined value. By making the size of the cropped image inversely proportional to the distance from the camera, the accuracy of recognizing the shape of the fingertip can be maintained at a high level even if the apparent size changes depending on the distance of the fingers from the camera. In addition, the accuracy of fingertip shape recognition can be improved by approximating with a straight line instead of making the size of the cropped image inversely proportional to the distance.

[0081] As described above, the control unit 51 (or image processing unit 60) can determine the distance to the fingers, adjust the size of the image to be input to the learning model (form recognition unit 63) based on the determined distance, and input the adjusted image to the learning model. This improves the accuracy of form recognition of the fingertips.

[0082] The offset calculation unit 612 obtains the cropped image size from the size calculation unit 610 and calculates the cropped image size based on the information regarding the direction of the search axis used by the target tip detection unit 611. The offset vector is calculated by multiplying by a predetermined coefficient. The subsequent processing is the same as in Embodiment 2, so the explanation is omitted.

[0083] The fingertip position calculation unit 615 may calculate the fingertip position by subtracting an offset vector from the midpoint calculated by the region midpoint calculation unit 614, and then adding an adjustment vector obtained by multiplying the cropped image size by a predetermined coefficient in the search axis direction.

[0084] (Embodiment 5) Due to recognition errors, the fingertip position tends to vary from frame to frame in the hand image (camera image), and the pointer displayed on the screen may tremble slightly (fluctuate slightly). As a countermeasure, it is conceivable to calculate a moving average of the pointer position and correct the pointer position. However, because the fingertip position in past frames is reflected in the process of calculating the moving average, the display of the pointer may lag behind the actual finger movement, which may cause discomfort for the user. This discomfort is more pronounced the faster the fingertip movement. Embodiment 5 describes a method for eliminating factors that cause discomfort due to slight fluctuations in the pointer.

[0085] Figure 20 shows an example of the configuration of the pointer information generation unit 65 of Embodiment 5. The pointer information generation unit 65 comprises a coordinate storage unit 651, a fingertip speed calculation unit 652, a moving average range adjustment unit 653, a moving average calculation unit 654, and a pointer state information generation unit 655. In order to make the pointer movement smooth, the pointer information generation unit 65 uses the history of past pointer coordinates to calculate a moving average of the pointer position and correct the pointer position (coordinates). Furthermore, when calculating the moving average, the pointer information generation unit 65 adjusts the range of the moving average according to the fingertip movement speed, thereby suppressing minute fluctuations in the pointer position and reducing the delay in pointer movement. A detailed explanation follows below.

[0086] The pointer information generation unit 65 receives the fingertip position and the pointer type (the class corresponding to the pointer) output from the pointer selection unit 64.

[0087] The coordinate storage unit 651 sequentially stores the fingertip position output from the pointer selection unit 64 as a history of pointer coordinates.

[0088] The fingertip velocity calculation unit 652 calculates the fingertip velocity for each frame based on the change in the fingertip position between different frames, using the fingertip position for each frame stored in the coordinate storage unit 651.

[0089] The moving average range adjustment unit 653 maintains the range of the moving average and adjusts the range based on the input fingertip speed. The range is the number of frames used when calculating the moving average. To adjust the range, for example, if the fingertip speed is greater than a predetermined threshold, 1 is subtracted from the number of ranges used in the previous frame to create a new range, and if the fingertip speed is less than a predetermined threshold, 1 is added to the number of ranges used in the previous frame to create a new range. In addition, an upper limit and a lower limit may be set for the range when calculating the moving average. The range to be adjusted is set to ±1, but if the change in fingertip speed is large, the range to be adjusted does not have to be limited to ±1, and for example, the range to be adjusted may be set to ±2.

[0090] When the pointer (finger tip) position (current coordinates) is input from the pointer selection unit 64, the moving average calculation unit 654 obtains the adjusted range from the moving average range adjustment unit 653, reads the history of pointer coordinates stored in the coordinate storage unit 651, calculates the moving average, and outputs the corrected pointer (finger tip) coordinates.

[0091] The pointer state information generation unit 655 generates information (pointer state information) indicating whether the pointer is selected or not, based on the form of the pointer output from the pointer selection unit 64 (whether the classified class is class 0 or class 1), and outputs it to the pointer information transmission unit 66.

[0092] Figure 21 shows an example of adjusting the range of a moving average. Assume that the range for calculating the moving average in the previous frame (frame 3) is set to 3, and that the moving average of fingertip position is calculated for the three frames 3, 2, and 1, working backward from frame 3, based on the fingertip position in each frame. If the fingertip velocity in the next frame (frame 4) is greater than the threshold, 1 is subtracted from the range used in the previous frame, so the range becomes 2 (=3-1), and the moving average of fingertip position is calculated based on the fingertip positions in frames 4 and 3. Also, if the fingertip velocity in the next frame is less than the threshold, 1 is added to the range used in the previous frame, so the range becomes 4 (=3+1), and the moving average of fingertip position is calculated based on the fingertip positions in frames 4, 3, 2, and 1.

[0093] As described above, the control unit 51 (or image processing unit 60) may store the history of the set pointer position in association with frames, adjust the number of frames for calculating the moving average according to the movement speed of the fingertip (end of the pointer), calculate the moving average of the pointer position over the adjusted number of frames, and set the pointer position based on the calculated moving average. This eliminates factors that cause discomfort due to minute fluctuations in the pointer position or delays in the pointer's movement.

[0094] Figure 22 is a diagram showing an example of the processing procedure of the image processing unit 60 of Embodiment 5. In the following description, the main processing unit will be referred to as the image processing unit 60 for convenience. The image processing unit 60 acquires the position and shape of the pointer (S51) and stores the pointer position as historical information (S52). The image processing unit 60 calculates the speed of the fingertip (pointer) (S53) and adjusts the range for calculating the moving average of the pointer position (fingert position) according to the speed of the fingertip (S54).

[0095] The image processing unit 60 calculates a moving average of the pointer position within the adjusted range (S55). The image processing unit 60 outputs information indicating the calculated pointer position and state (S56), and then terminates the process.

[0096] As described above, by adjusting the range used to calculate the moving average of the fingertip position according to the fingertip speed, it is possible to eliminate the factor that causes discomfort to the user due to the pointer display lagging behind the actual finger movement. In addition, since the range adjustment is ±1 for each frame, discontinuous changes in the pointer position can be suppressed.

[0097] (Embodiment 6) If the direction of the search axis used to detect the fingertip position is misaligned with the direction the fingertip is pointing, the detected fingertip position is likely to deviate from the user's subjective perception. Embodiment 6 describes a method for correcting the direction of the search axis to match the direction the fingertip is pointing.

[0098] Figure 23 shows an example of the configuration of the image processing unit 60 of Embodiment 6. The difference from the previously described embodiment is the inclusion of a search axis correction unit 67. The search axis correction unit 67 acquires the initial search axis direction (information regarding the direction of the search axis before correction) from the target detection unit 61 and outputs the search axis direction (information regarding the direction of the search axis after correction) to the target detection unit 61.

[0099] Figure 24 shows an example of the configuration of the search axis correction unit 67, and Figure 25 shows a method for correcting the search axis. As shown in Figure 24, the search axis correction unit 67 includes a target tip detection unit 671, an offset calculation unit 672, a fingertip area end-edge detection unit 673, a tangent detection unit 674, and a bisection direction calculation unit 675. The target tip detection unit 671, the offset calculation unit 672, and the fingertip area end-edge detection unit 673 are the same as the target tip detection unit 611, the offset calculation unit 612, and the fingertip area end-edge detection unit 613 of the target detection unit 61 of Embodiment 2.

[0100] The target tip detection unit 671 searches the image of the fingers acquired from the camera 10 in the direction of the initial search axis. If multiple fingertips are present in the image, it detects the fingertip corresponding to the upstream side of the initial search axis as the fingertip tip and outputs the detected fingertip tip position (coordinates on the image) to the fingertip region end detection unit 673.

[0101] The offset calculation unit 672 calculates an offset vector by multiplying the unit vector in the initial search axis direction by a predetermined coefficient, based on the initial search axis direction information used by the target tip detection unit 671.

[0102] The fingertip area end detection unit 673 sets a vertical axis in a direction perpendicular to the initial search axis (perpendicular direction), passing through the endpoint position obtained by adding an offset vector starting from the fingertip position, and detects the boundary points at both ends of the thumb and index finger (boundary points in Figure 25) on the set vertical axis.

[0103] The tangent detection unit 674 traces the edges of nearby fingers to each of the two boundary points and detects the tangents of the edges at the boundary points. In Figure 25, the tangents of the finger edges passing through the boundary point of the thumb (the edge of the finger opposite the index finger) and the tangents of the finger edges passing through the boundary point of the index finger (the edge of the finger opposite the thumb) are detected.

[0104] The bisection direction calculation unit 675 derives a straight line that bisects the angle between the two detected tangent lines, and calculates the corrected search axis direction by the unit vector in the opposite direction to the fingertip direction of the derived straight line. The bisection direction calculation unit 675 outputs information regarding the corrected search axis direction to the target detection unit 61.

[0105] If there is a significant misalignment between the initial search axis direction and the direction of the fingertip, the misalignment may remain even after performing the above procedure only once. In such cases, the corrected search axis direction can be used as the initial search axis direction, and by repeating each of the above processes to sequentially update the search axis, the direction of the search axis can be brought closer to the direction of the fingertip.

[0106] As described above, the control unit 51 (or image processing unit 60) may search the acquired image of the fingers along a predetermined search axis to detect the tip of the fingertip, set an orthogonal axis (vertical axis) perpendicular to the search axis at a predetermined distance from the detected tip, search along the set orthogonal axis to detect the first outer boundary point and the second outer boundary point of the first finger (e.g., thumb) and the second finger (e.g., index finger), respectively, and correct the direction of the search axis based on the first tangent line (e.g., tangent line of the thumb edge) that touches the boundary of the first finger at the detected first outer boundary point and the second tangent line (e.g., tangent line of the index finger edge) that touches the boundary of the second finger at the detected second outer boundary point. This can suppress deviations in the detected fingertip position from the user's subjective perception.

[0107] According to Embodiment 6, before detecting the fingertip position, the direction of the search axis can be brought closer to the direction the fingertip is pointing, thereby suppressing deviation from the user's subjective perception.

[0108] (Embodiment 7) Embodiment 6 was configured to correct the direction of the search axis based on the direction of the fingertip, but the search axis can be set by other methods. For example, when the frame rate of camera 10 is high, the image changes in adjacent frames are small, and it is expected that the direction of the fingertip will hardly change. Embodiment 7 describes a method of setting the search axis using information from the previous frame.

[0109] Figure 26 shows an example of the configuration of the image processing unit 60 of Embodiment 7. The difference from the image processing unit 60 of Embodiment 6 shown in Figure 23 is that it includes a tracking detection unit 68 and a search setting storage unit 69.

[0110] The search setting storage unit 69 records the fingertip position for each fingertip detected in the previous frame, and the search axis direction (information regarding the direction of the search axis) used in the previous frame. However, if the search setting storage unit 69 obtains a class that does not correspond to a pointer from the pointer selection unit 64, it deletes the record of the fingertip position and search axis direction for the corresponding fingertip. The search setting storage unit 69 outputs the recorded fingertip position for each fingertip detected in the previous frame and the search axis direction used in the previous frame to the tracking detection unit 68 in the next frame. Note that if the fingertip position and search axis direction for each fingertip are not recorded, the tracking detection unit 68 will not operate. In other words, the tracking detection unit 68 operates if the information from the previous frame (fingertip position, search axis direction) exists, and does not operate if it does not exist.

[0111] Figure 27 shows an example of the configuration of the tracking detection unit 68. The tracking detection unit 68 comprises a target tip detection unit 681, an offset calculation unit 682, a fingertip area end-edge detection unit 683, an area midpoint calculation unit 684, and a fingertip position calculation unit 685. The target tip detection unit 681, the offset calculation unit 682, the fingertip area end-edge detection unit 683, the area midpoint calculation unit 684, and the fingertip position calculation unit 685 are the same as the target tip detection unit 611, the offset calculation unit 612, the fingertip area end-edge detection unit 613, the area midpoint calculation unit 614, and the fingertip position calculation unit 615 of the target detection unit 61.

[0112] The tracking detection unit 68 detects the fingertip position in the current frame by referring to the fingertip position detected in the previous frame and the search axis direction used in the previous frame. Specifically, when searching for the tip of the fingertip in the direction of the fingertip recognized in the previous frame, the search range is limited to the vicinity of the fingertip position detected in the previous frame. When searching, the search axis direction used in the previous frame can be used. The reason for limiting the search range to the vicinity of the fingertip position is that the position and direction of the fingertip are not expected to change much between adjacent frames.

[0113] If the search setting memory unit 69 does not contain information from the previous frame (finger tip position, search axis direction), the target detection unit 61 will detect the finger tip position instead of the tracking detection unit 68.

[0114] As described above, the control unit 51 (or image processing unit 60) may store the fingertip position and search axis direction identified when the pointer position was set, associating them with frames. If the fingertip position and search axis direction associated with the most recent frame are stored, the pointer position may be set by searching along the search axis based on the fingertip position to detect the tip of the fingertip.

[0115] According to Embodiment 7, regardless of whether it is the right or left hand, only one search axis is required, eliminating the need to use two search axes as in the previously described embodiment, thus reducing the processing time and effort required for the search.

[0116] Furthermore, in order to detect newly appearing fingertips in the camera's field of view, it is necessary to also use a search that utilizes two or more search axes. In this case, a search using information from the previous frame can be performed first, followed by a search using two or more search axes. When using a search with two or more search axes, the area near the fingertips detected in the search using information from the previous frame should be excluded from the search target to avoid duplicate detection.

[0117] (Embodiment 8) Figure 28 shows an example of a missed detection of the fingertip tip. The target tip detection unit 611 of the above embodiment searches the image of the fingers acquired from the camera 10 in the direction of the search axis, and if multiple fingertips are present in the image, it detects the fingertip corresponding to the upstream side of the search axis as the fingertip tip and outputs the detected fingertip tip position to the fingertip region end detection unit 613. However, as shown in Figure 28, there are cases where multiple fingertips are captured in the image of the fingers, and depending on the state of the multiple fingertips (such as the user's finger movements and posture), it may be possible to detect the upstream point in the direction of the search axis, but to overlook necessary parts such as the pinched shape. Embodiment 8 describes a method for detecting multiple fingertips.

[0118] Figure 29 is a diagram showing an example of the configuration of the target tip detection unit 611 of Embodiment 8, and Figure 30 is a diagram showing a method for detecting the tip of a fingertip. As shown in Figure 29, the target tip detection unit 611 includes a contour extraction unit 6111, a search axis projection unit 6112, and a local upstream point detection unit 6113.

[0119] The contour extraction unit 6111 extracts the contour of the object (fingers) present in the camera image (image of fingers) and generates a coordinate list for each pixel of the extracted contour (pixels on the outer perimeter of the fingers). The contour extraction unit 6111 outputs the generated coordinate list to the search axis projection unit 6112.

[0120] The search axis projection unit 6112 projects each pixel in the acquired coordinate list onto the search axis, converting it into coordinates on the search axis. The projection onto the search axis is performed perpendicular to the search axis. By projecting the coordinates on the contour (outer circumference) onto the search axis, the position on the outer circumference of the fingers and the position in the search axis direction are associated by a function, as shown in Figure 30. In Figure 30, a graph representing the function is drawn in a two-dimensional coordinate system where the horizontal axis represents the position on the outer circumference and the vertical axis represents the position in the search axis direction.

[0121] The local upstream point detection unit 6113 searches for a position on the outer circumference to determine the position in the search axis direction and detects the valley portion where the difference exceeds a preset threshold as the position of the fingertip. As shown in Figure 30, in the graph representing the function, the valley portion corresponds to the upstream point indicating the fingertip. By setting an appropriate threshold, all valley portions where there are peak portions exceeding the threshold on both sides of the valley portion are detected as local upstream points.

[0122] As shown in Figure 30, the valley P corresponding to the tip of the thumb is detected as a local upstream point because there are peaks above the threshold on both sides. The tip of the index finger (point A) is excluded from the local upstream point because it is integrated with the valley of the thumb. The reason the tip of the index finger is excluded instead of the tip of the thumb is that the tip of the thumb is further upstream in the direction of the search axis. The valley P′ corresponding to the tip of the middle finger (point B) is detected as a local upstream point because there are peaks above the threshold on both sides. The valley corresponding to the tip of the ring finger is excluded from the local upstream point because there are no peaks above the threshold on either side.

[0123] As described above, the control unit 51 (or image processing unit 60) may identify the contour of the fingers based on the acquired image of the fingers, associate the distance between each point on the identified contour and the search axis in a predetermined direction, and identify the point on the contour corresponding to the minimum point of the associated distance (local upstream point) as the fingertip.

[0124] According to Embodiment 8, even if the necessary location, such as a pinched shape, is not at the upstream end of the search axis, it can be reliably detected as a fingertip. Furthermore, it can suppress the detection of locations that are integrated with other fingertips, such as point A, and minute protrusions that are accidentally generated by noise.

[0125] The contactless user interfaces described in the embodiments above can be used as contactless user interfaces in various fields. Examples of such fields include applications and operating devices used in information processing equipment, in-vehicle equipment, and medical devices. The following section will describe the case of use in in-vehicle equipment.

[0126] Figure 31 shows an example of an in-vehicle interface device. The in-vehicle interface device includes a contactless user interface function for operating the navigation screen without physical contact. The in-vehicle interface device may also include the image processing device 50 and camera 10 described above.

[0127] As shown in Figure 31, when the user (driver) moves their thumb and index finger between the steering wheel and themselves, an image of the fingers captured by a camera 10 (not shown) is transmitted to the image processing device 50. The camera 10 can be installed in an appropriate location inside the vehicle. The image processing device 50 overlays the image of the fingers onto the navigation screen. In this case, the visibility of symbols such as characters on the navigation screen can be maintained by making the image of the fingers semi-transparent, for example. For example, by moving the thumb and index finger while keeping them separated, the user can move the image of the fingers on the navigation screen over the "history" icon. The user can then operate the "history" icon by pinching (grabbing) with their thumb and index finger. The same applies to other icons on the navigation screen.

[0128] The image of the fingers may be the image captured by camera 10 itself, or it may be an image of the fingers that has been processed. The processed image may include, for example, a representative image of the fingers prepared in advance (which may be a real photograph or a diagram) and a symbol representing a pointer. Alternatively, instead of displaying the image of the fingers as a semi-transparent overlay, it may be displayed as a wipe image in a small area of ​​the navigation screen.

[0129] The control unit 51 (or image processing unit 60) may combine and display the acquired image (image of fingers) or an image obtained by processing the acquired image with a symbol representing a pointer on the display screen.

[0130] The control unit 51 (or image processing unit 60) may display an image of the fingers in a separated state on the display screen, display a pointer between the fingertips of the first and second fingers, and, upon receiving a response to bring the fingertips of the first and second fingers into contact, execute the operation on the display screen indicated by the pointer. The image of the fingers may be displayed on the display screen semi-transparently, for example. This can improve the visibility of various operation icons on the display screen.

[0131] As described above, the in-vehicle interface device includes a control unit, which acquires an image of the fingers, sets a pointer at a position between the fingertips of the first and second fingers identified based on the acquired image, which represents a position represented by the fingertips of the first and second fingers, and accepts operation by the pointer displayed on the in-vehicle device operation screen.

[0132] Furthermore, when the image of the fingers moves over the "History" icon, a voice message such as "You can select history by pinching" is output, allowing the user to operate the navigation system without taking their eyes off the road ahead, thus contributing to safer driving. In the example shown in Figure 31, a navigation screen was used as an example, but the in-vehicle equipment is not limited to navigation devices; other in-vehicle equipment such as AV equipment may also be used.

[0133] In the embodiments described above, fingers were used as an example of a reference object, but the reference object is not limited to fingers. For example, it could be a surgical instrument such as forceps, or a robotic hand that simulates fingers. Furthermore, the same effect can be obtained even if the fingers are not bare, such as when wearing gloves.

[0134] (Note 1) The computer program causes the computer to acquire an image in which multiple indicators are photographed, and to set a pointer that specifies a position represented by the ends of the first and second indicators at a position between the end of the first indicator and the end of the second indicator, which are identified based on the acquired image.

[0135] (Note 2) In the computer program of Note 1, the image in which the plurality of indicators are captured is an image of a hand, the end of the first indicator and the end of the second indicator are the fingertips of the first and second fingers, and the computer is instructed to execute a process to set the pointer at an intermediate position between the fingertips of the first and second fingers.

[0136] (Note 3) In the computer program of Note 2, either the first finger or the second finger is a thumb, and the computer is instructed to perform a process of setting the pointer on the pad of the thumb.

[0137] (Note 4) The computer program, in the computer program of Note 2 or Note 3, causes the computer to perform the following processes: search the acquired image of the fingers along a search axis in a predetermined direction to detect the tip of the fingertip; set an orthogonal axis that is a predetermined distance away from the tip and perpendicular to the search axis; search along the set orthogonal axis to detect the outer boundaries of the first finger and the second finger, identify the midpoint of the detected outer boundaries; and set the pointer based on the identified midpoint.

[0138] (Note 5) The computer program in Note 4 causes the computer to execute a process that sets the pointer at a position offset by a predetermined distance from the midpoint in a direction parallel to the search axis.

[0139] (Note 6) The computer program, in any of the computer programs described in Notes 1 to 5, will execute a process that accepts an operation to set the amount of movement required to move the set pointer to the desired position.

[0140] (Note 7) The computer program, in any of the computer programs described in Notes 1 to 6, causes the computer to display multiple positions of the pointer as selectable candidate positions and to execute a process that accepts an operation to select a position of the pointer from among the displayed candidate positions.

[0141] (Note 8) The computer program inputs an image of a hand into a learning model generated in any of the computer programs described in Notes 2 to 5, which outputs classification information including whether the first finger and the second finger are in contact or whether the first finger and the second finger are separated, and then executes a process to switch the display mode of the pointer located between the fingertip of the first finger and the fingertip of the second finger according to the classification information output by the learning model.

[0142] (Note 9) The computer program in Note 8 causes the computer to perform the following processes: determine the distance to the fingers, adjust the size of the image to be input to the learning model based on the determined distance, and input the adjusted image to the learning model.

[0143] (Note 10) The computer program, in any of the computer programs described in Notes 1 to 9, causes the computer to perform a process that combines and displays the acquired image or an image obtained by processing the said image with the symbol representing the pointer on the display screen.

[0144] (Note 11) In any of the computer programs described in Notes 2 to 10, the computer program is instructed to display an image of fingers with the first finger and the second finger separated on a display screen, to display a pointer between the fingertip of the first finger and the fingertip of the second finger, and to execute an operation on the display screen indicated by the pointer when it receives an action to bring the fingertip of the first finger and the fingertip of the second finger into contact.

[0145] (Note 12) The computer program, in any of the computer programs described in Notes 1 to 11, causes the computer to store the history of the set pointer's position in association with frames, adjust the number of frames for calculating the moving average according to the movement speed of the end of the indicator, calculate the moving average of the pointer's position over the adjusted number of frames, and set the pointer's position based on the calculated moving average.

[0146] (Note 13) The computer program, in any of the computer programs described in Notes 2 to 5, causes the computer to perform the following processing: to search the acquired image of the fingers along a search axis in a predetermined direction to detect the tip of the fingertip; to set an orthogonal axis perpendicular to the search axis and separated from the tip by a predetermined distance; to search along the set orthogonal axis to detect the first and second outer boundary points of the first and second fingers, respectively; and to correct the direction of the search axis based on a first tangent line that touches the boundary of the first finger at the detected first outer boundary point and a second tangent line that touches the boundary of the second finger at the detected second outer boundary point.

[0147] (Note 14) In any of the computer programs described in Notes 2 to 5, the computer is instructed to store the position of the fingertip and the direction of the search axis identified when the pointer position was set, associating them with frames, and if the position of the fingertip and the direction of the search axis associated with the most recent frame are stored, the computer is instructed to set the pointer position by searching along the search axis based on the position of the fingertip and detecting the tip of the fingertip.

[0148] (Note 15) The computer program, in any of the computer programs described in Notes 2 to 5, causes the computer to perform the following processes: identify the contour of the fingers based on the acquired image of the fingers; associate the distance between each point on the identified contour and a search axis in a predetermined direction; and identify the point on the contour corresponding to the minimum point of the distance as the fingertip.

[0149] (Note 16) The image processing device includes a control unit, which acquires an image of a plurality of indicators, and sets a pointer that specifies a position represented by the ends of the first and second indicators at a position between the end of a first indicator and the end of a second indicator, which are identified based on the acquired image.

[0150] (Note 17) In the image processing device described in Note 16, the control unit acquires images of the plurality of indicators captured by the TOF camera.

[0151] (Note 18) The image processing method acquires an image in which multiple indicators are photographed, and sets a pointer that specifies a position represented by the ends of the first and second indicators at a position between the end of the first indicator and the end of the second indicator, which are identified based on the acquired image.

[0152] (Note 19) The in-vehicle interface device includes a control unit, which acquires an image of a plurality of indicators, sets a pointer at a position between the end of a first indicator and the end of a second indicator, which are identified based on the acquired image, to specify a position represented by the ends of the first and second indicators, and accepts operations by the pointer displayed on the in-vehicle device operation screen.

[0153] The matters described in each embodiment can be combined with each other. Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, the claims use a form in which claims referencing two or more other claims (multi-claim form), but are not limited to this. A form in which multi-claims referencing at least one multi-claim (multi-multi-claim) may also be used. [Explanation of Symbols]

[0154] 10 Cameras 20 Display device 50 Image Processing Devices 51 Control Unit 52 Communications Department 53 memory 54 Operation section 55 Storage section 56 Computer Programs 57 Learning Models 58 Recording medium reading unit 59 Recording media 60 Image Processing Unit 61 Target detection unit 610 Size Calculation Section 611 Target tip detection unit 6111 Contour extraction unit 6112 Search axis projection section 6113 Local upstream point detection unit 612 Offset Calculation Unit 613 Fingertip area detection unit at both ends 614 Area midpoint calculation part 615 Fingertip position calculation section 616 Fingertip detection unit at both ends 617 Thumb selection section 618 Thumb pad position calculation unit 619 Tip distance detection unit 62 Image cropping section 63 Form recognition part 64 Pointer Selection Section 65 Pointer Information Generation Unit 651 Coordinate storage unit 652 Fingertip speed calculation section 653 Moving Average Range Adjustment Section 654 Moving average calculation section 655 Pointer state information generation unit 66 Pointer Information Transmission Unit 67 Search axis correction unit 671 Target tip detection unit 672 Offset Calculation Unit 673 Fingertip area detection unit at both ends 674 Tangential detection unit 675 Bisection direction calculation unit 68 Tracking and detection unit 681 Target tip detection unit 682 Offset Calculation Unit 683 Fingertip area detection unit at both ends 684 Area midpoint calculation part 685 Fingertip position calculation section 69 Search setting memory unit 100 Contactless User Interface Systems

Claims

1. On the computer, Images of fingers with multiple indicators were captured, A pointer is set at a position between the fingertip of the first finger, which is the end of the first indicator identified based on the acquired image, and the fingertip of the second finger, which is the end of the second indicator, to specify the position represented by the ends of the first and second indicators. Furthermore, the acquired images of the fingers are searched along a predetermined search axis to detect the fingertips. An orthogonal axis is set that is separated by a predetermined distance from the tip and perpendicular to the search axis, The system searches along the set orthogonal axis to detect the outer boundaries of the first and second fingers, Identify the midpoint of the detected outer boundary, Set the pointer based on the identified midpoint. A computer program that executes a process.

2. Either the first finger or the second finger is the thumb. On the computer, The pointer is set on the pad of the thumb. A computer program according to claim 1 that causes a process to be executed.

3. On the computer, The pointer is set at a position offset by a predetermined distance from the midpoint in a direction parallel to the search axis. A computer program according to claim 1 that causes a process to be executed.

4. On the computer, It accepts an operation to set the amount of movement required to move the set pointer position to the desired position. A computer program according to any one of claims 1 to 3 that causes a process to be executed.

5. On the computer, Multiple positions of the aforementioned pointer are displayed as selectable candidate positions. It accepts an operation to select the pointer position from the displayed candidate positions. A computer program according to any one of claims 1 to 3 that causes a process to be executed.

6. On the computer, When an image of fingers is input, a learning model is generated that outputs classification information including whether it is a first form in which the first and second fingers are in contact, or a second form in which the first and second fingers are separated. Depending on the classification information output by the learning model, the display mode of the pointer located between the fingertip of the first finger and the fingertip of the second finger is switched. A computer program according to claim 1 that causes a process to be executed.

7. On the computer, Determine the distance to the fingers, The size of the image input to the learning model is adjusted based on the identified distance. The adjusted image is input to the learning model. A computer program according to claim 6 that causes a process to be executed.

8. On the computer, The acquired image or an image processed from the said image, and the symbol representing the pointer are combined and displayed on the display screen. A computer program according to any one of claims 1 to 3 that causes a process to be executed.

9. On the computer, An image of fingers with the first and second fingers separated is displayed on the screen. A pointer is displayed between the fingertip of the first finger and the fingertip of the second finger. When an action is detected in which the fingertip of the first finger and the fingertip of the second finger come into contact, the operation on the display screen indicated by the pointer is executed. A computer program according to any one of claims 1 to 3 that causes a process to be executed.

10. On the computer, The history of the set pointer positions is stored and associated with each frame. The number of frames for calculating the moving average is adjusted according to the moving speed of the end of the indicator, Calculate the moving average of the pointer position over the number of frames adjusted, The position of the pointer is set based on the calculated moving average. A computer program according to any one of claims 1 to 3 that causes a process to be executed.

11. On the computer, The acquired image of the fingers is searched along a predetermined search axis to detect the tip of the fingertip. A predetermined distance from the tip is set, and an orthogonal axis is set perpendicular to the search axis, The system searches along the set orthogonal axis to detect the first and second outer boundary points of the first and second fingers, respectively. The direction of the search axis is corrected based on a first tangent line that touches the boundary of the first finger at the detected first outer boundary point, and a second tangent line that touches the boundary of the second finger at the detected second outer boundary point. A computer program according to claim 1 that causes a process to be executed.

12. On the computer, The fingertip position and search axis direction identified when setting the pointer position are stored in association with the frame. If the fingertip position and search axis direction associated with the most recent frame are stored, the pointer position is set by searching along the search axis based on the fingertip position to detect the tip of the fingertip. A computer program according to claim 1 that causes a process to be executed.

13. On the computer, Based on the acquired images of the fingers, the contours of the fingers are identified. The distance between each point on the identified contour and the search axis in a predetermined direction is associated, The point on the contour corresponding to the minimum point of the aforementioned distance is identified as the tip of the fingertip. A computer program according to claim 1 that causes a process to be executed.

14. Equipped with a control unit, The control unit, Images of fingers with multiple indicators were captured, A pointer is set at a position between the fingertip of the first finger, which is the end of the first indicator identified based on the acquired image, and the fingertip of the second finger, which is the end of the second indicator, to specify the position represented by the ends of the first and second indicators. Furthermore, the acquired images of the fingers are searched along a predetermined search axis to detect the fingertips. An orthogonal axis is set that is separated by a predetermined distance from the tip and perpendicular to the search axis, The system searches along the set orthogonal axis to detect the outer boundaries of the first and second fingers, Identify the midpoint of the detected outer boundary, Set the pointer based on the identified midpoint. Image processing device.

15. The control unit, The aforementioned multiple indicators acquire images captured by the TOF camera. The image processing apparatus according to claim 14.

16. Images of fingers with multiple indicators were captured, A pointer is set at a position between the fingertip of the first finger, which is the end of the first indicator identified based on the acquired image, and the fingertip of the second finger, which is the end of the second indicator, to specify the position represented by the ends of the first and second indicators. Furthermore, the acquired images of the fingers are searched along a predetermined search axis to detect the fingertips. An orthogonal axis is set that is separated by a predetermined distance from the tip and perpendicular to the search axis, The system searches along the set orthogonal axis to detect the outer boundaries of the first and second fingers, Identify the midpoint of the detected outer boundary, Set the pointer based on the identified midpoint. An image processing method that uses a computer to perform image processing.

17. Equipped with a control unit, The control unit, Images of fingers with multiple indicators were captured, A pointer is set at a position between the fingertip of the first finger, which is the end of the first indicator identified based on the acquired image, and the fingertip of the second finger, which is the end of the second indicator, to specify the position represented by the ends of the first and second indicators. Furthermore, the acquired images of the fingers are searched along a predetermined search axis to detect the fingertips. An orthogonal axis is set that is separated by a predetermined distance from the tip and perpendicular to the search axis, The system searches along the set orthogonal axis to detect the outer boundaries of the first and second fingers, Identify the midpoint of the detected outer boundary, The pointer is set based on the identified midpoint, The system accepts operations using the pointer displayed on the in-vehicle device operation screen. In-vehicle interface device.

Images