Input information confirmation method, readable storage medium and head-mounted display

Abstract

The invention provides an input information confirmation method which comprises the following steps: acquiring a first hand image, extracting position information of each first key point in the first hand image in a display interface, and confirming the position of a virtual keyboard in the display interface according to the position information of the first key points, the virtual keyboard comprises a plurality of non-overlapped input areas; tracking an eyeball fixation point at the current moment, and determining an input region where the fixation point is located in the plurality of input regions as an effective input region; and acquiring a second hand image and extracting position information of each first key point in the second hand image in the effective input area to confirm input information. The invention also provides a readable storage medium and a head-mounted display.

Description

Technical Field

[0001] The present application relates to an input information confirmation method, a readable storage medium storing a computer program of the method, and a head mounted display. Background Art

[0002] In recent years, head-mounted displays have been listed as an important tool for future human social connection. The input system in the head-mounted display is an important tool for users to communicate with machines. In addition, the input system needs to be used in conjunction with the input method so that the user can see the results of the operation after operating the input system according to the input method. Many input methods determine the position of a virtual keyboard in three-dimensional space, and then confirm the input information based on the position of the fingertips relative to the virtual keyboard. However, in three-dimensional space, the fingertips are easily blocked by the back of the hand, making it difficult for the camera module of the head-mounted display to obtain the position information of the fingertips, making it impossible to confirm the input information. Summary of the invention

[0003] The first aspect of the present application provides an input information confirmation method, comprising: Acquire a first hand image and extract position information of each first key point in the first hand image in a display interface, and determine a position of a virtual keyboard in the display interface according to the position information of the first key point, wherein the virtual keyboard includes a plurality of non-overlapping input areas; Tracking the eye gaze point at the current moment, and determining an input area where the gaze point is located in the multiple input areas as a valid input area; and A second hand image is acquired and position information of each of the first key points in the second hand image in the valid input area is extracted to confirm input information.

[0004] The above-mentioned input information confirmation method tracks the eye gaze point through eye tracking technology, and determines an input area where the gaze point is located in multiple input areas as a valid input area. After confirming the valid input area of ​​the virtual keyboard, when the user inputs information, the fingertip moves in the valid input area. The shape of the area surrounded by the outer contour of the virtual keyboard is rectangular. Since the extension direction of the valid input area is the same as the extension direction of the long side of the virtual keyboard, when the fingertip moves in the valid input area, it is difficult for the user's palm to block the fingertip. When it is detected that the fingertip moves into the valid input area, a second hand image is obtained, and the key area where the fingertip is located is determined according to the position information of the first key point in the second hand image to confirm the information that the user needs to input. In the second hand image, the fingertips located in the valid input area are not blocked by the palm. Therefore, the key area where the first key point in the valid input area is located can be accurately determined to confirm the input information, thereby improving the accuracy of information input.

[0005] A second aspect of the present application provides a readable storage medium storing a computer program, which, when executed, implements the steps of the calculation method for confirming input information as described above.

[0006] The above-mentioned readable storage medium integrates the above-mentioned input information confirmation method, and can achieve all the beneficial effects of the above-mentioned calculation method for confirming input information.

[0007] A third aspect of the present application provides a head mounted display, comprising: A display module, used for displaying a virtual keyboard, wherein the virtual keyboard includes a plurality of input information areas; A camera module, used to obtain a hand image of the user; An eye tracking module, used for tracking the current position of the eye gaze point on the virtual keyboard; and A computing module is electrically connected to the camera module and the eye tracking module, respectively, and is used to receive the multiple hand images and the position information of the gaze point. The computing module includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the computing module executes the steps of the input information confirmation method as described above.

[0008] The head mounted display stores the input information confirmation method, and all the beneficial effects of the input information confirmation method can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic diagram of a module of a head mounted display according to an embodiment of the present application.

[0010] Figure 2 Flow chart of the input information confirmation method according to an embodiment of the present application.

[0011] Figure 3 is a schematic diagram of the first hand image.

[0012] Figure 4 For virtual keyboard and Figure 3 Schematic diagram of the positional relationship of the first hand image in .

[0013] Figure 5 for Figure 4 Schematic diagram of a virtual keyboard divided into multiple input areas.

[0014] Figure 6 for Figure 5 Schematic diagram of the positional relationship between the virtual keyboard and the second hand image.

[0015] Main component symbols Head mounted display: 100 Display modules: 10 - Camera modules: 20 Eye tracking module: 30 Operation modules: 40 Memory: 41 - Processor: 42 Virtual keyboard: 50 Input area: 51 Valid input area: 51a Key area: 52, 52a First hand image: 60 Second hand image: 70 The first key point: F1, F2, ..., F10 The second key point: Q1, Q2, Q3, Q4 Midpoint: F11 Focus: Ef The following specific implementation methods will further illustrate the present application in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0016] Head-mounted display is an important tool for future human social connection. When using the head-mounted display, users generally use a virtual keyboard displayed by a display module in the head-mounted display to input information.

[0017] See also Figure 1 The head mounted display 100 provided in the embodiment of the present application includes a display module 10, a camera module 20, an eye tracking module 30 and a computing module 40. The computing module 40 is electrically connected to the display module 10, the camera module 20 and the eye tracking module 30 respectively.

[0018] The display module 10 is used to display a virtual keyboard. The virtual keyboard displayed by the display module 10 includes multiple non-overlapping input areas, each of which includes multiple key areas. When inputting information, the user's hands need to operate the key areas on the virtual keyboard to achieve the purpose of inputting information. The camera module 20 is used to track the user's hands in real time and obtain multiple hand images of the user. The eye tracking module 30 is used to track the position of the user's eye gaze point on the virtual keyboard at the current moment when the user wears the head-mounted display 100, and is used to transmit the position information of the eye gaze point to the operation module 40.

[0019] The operation module 40 is used to extract the position information of each first key point on the user's hand image in the display interface, extract the position information of the second key point of each key area in the virtual keyboard in the display interface, and is also used to determine an input area where the gaze point is located in multiple input areas as a valid input area based on the position information of the gaze point. When the operation module 40 extracts the position information of each first key point on the user's hand image in the display interface, it projects the fingertips of the user's fingers onto the plane where the virtual keyboard is located, thereby identifying the position information of each first key point in the display interface. That is, each first key point is located in the same plane as the virtual keyboard. When the camera module 20 detects that the fingertips of the user's fingers move into the valid input area, the operation module 40 is also used to combine the position information of the first key point in the valid input area and the position information of the second key point in the valid input area to determine the key area where the fingertips of the fingers are located, thereby determining the information input by the user.

[0020] The present application also provides an input information confirmation method, which is applied to the head mounted display 100. Figure 2 The input information confirmation method of the embodiment of the present application includes the following steps: S1, obtaining a first hand image and extracting position information of each first key point in the first hand image in a display interface, confirming the position of a virtual keyboard in the display interface according to the position information of the first key point, wherein the virtual keyboard includes multiple non-overlapping input areas.

[0021] S2, tracking the eye gaze point at the current moment, and determining an input area where the gaze point is located among the multiple input areas as a valid input area.

[0022] S3, acquiring a second hand image and extracting position information of each of the first key points in the second hand image in the valid input area to confirm input information.

[0023] See also Figure 3 In step S1, the camera module 20 acquires a first hand image 60, which is an image including both hands of the user. After receiving the first hand image 60, the computing module 40 extracts the position information of each first key point F1-F10 in the first hand image 60 in the display interface, including the following steps: extracting the position information of the first key point F1-F10 at the fingertip of each finger in the first hand image 60 in the display interface.

[0024] Figure 3In the figure, the first hand image 60 includes ten first key points F1-F10, which are respectively located at the fingertips of the user's ten fingers. The change process of the position information of each first key point in the display interface corresponds to the movement process of the fingertip of each finger in the display interface.

[0025] In this embodiment, the “display interface” refers to the image viewed by the user's eyes through the head mounted display 100 , and the meanings of all “display interfaces” appearing below are the same as explained above.

[0026] After receiving the first hand image, the operation module 40 extracts the position information of each first key point in the first hand image in the display interface through Mediapipe, and confirms the position of the virtual keyboard in the display interface according to the position information of the first key point. Mediapipe is actually an integrated tool library of machine learning visual algorithms, which includes various models such as face detection, face key points, gesture recognition, avatar segmentation and posture recognition. The operation module 40 can directly output the position information of each first key point in the first hand image in the display interface by using Mediapipe. Among them, the position information can be the coordinate information of the first key point in the display interface.

[0027] See also Figure 4 The step of confirming the position of the virtual keyboard 50 in the display interface according to the position information of the first key point includes: Extracting position information of first key points F5 and F6 at the fingertips of two thumbs in the first hand image 60 in the display interface; Extracting the position information of the middle point F11 of the line segment formed by the first key points F5 and F6 at the fingertips of the two thumbs in the display interface; The position of the virtual keyboard 50 in the display interface is confirmed according to the position information of the middle point F11.

[0028] The middle point F11 is located on the symmetry axis of the virtual keyboard 50 , and all the first key points F1 - F10 in the first hand image 60 are located within the virtual keyboard 50 .

[0029] After the step of confirming the position of the virtual keyboard in the display interface according to the position information of the first key point, the method further includes: The position information of each second key point of each key area in the display interface is extracted, each second key point is located at the edge of the key area, and the area enclosed by each second key point connected in sequence is equal to or smaller than the area of ​​the key area.

[0030] See also Figure 5In this embodiment, after the computing module 40 transmits the position information of the virtual keyboard 50 to the display module 10, the display module 10 is used to display the virtual keyboard 50. The virtual keyboard 50 includes a plurality of non-overlapping input areas 51, and each input area 51 includes a plurality of key areas 52 arranged at intervals. When different key areas 50 are touched, different information can be input to the head mounted display 100. In this embodiment, when some key areas 52 are touched, letter information is input, when some key areas 52 are touched, numeric information is input, and when some key areas 52 are touched, symbol information is input.

[0031] In this embodiment, the virtual keyboard 50 has a rectangular outer contour, and each input area 51 also has a rectangular outer contour. The input areas 51 are arranged in sequence along the extension direction of the short side of the virtual keyboard 50, and the long side direction of each input area 51 is parallel to the long side direction of the virtual keyboard 50, and the length of the long side of each input area 51 is equal to the length of the long side of the virtual keyboard 50. In other embodiments, the length of the long side of each input area 51 may be less than the length of the long side of the virtual keyboard 50.

[0032] Each input area 51 includes a plurality of key areas 52 arranged in sequence along the extension direction of the long side of the virtual keyboard 50. In the present embodiment, each key area 52 is rectangular. Each key area 52 has four first key points, each of which is located at a rectangular vertex of the key area 52. Taking a key area 52a as an example, the key area 52a has four second key points, namely, second key points Q1, Q2, Q3 and Q4, and the second key points Q1, Q2, Q3 and Q4 are all located at the rectangular vertex of the key area 52a.

[0033] In step S2, the eye tracking module 30 uses the eye tracking technology to track the current position of the eye gaze point on the virtual keyboard, and transmits the position information of the gaze point to the calculation module 40. The calculation module 40 combines the position information of the gaze point and the position information of the virtual keyboard to determine that an input area where the gaze point is located among the multiple input areas is a valid input area.

[0034] Please refer to Figure 5For example, in at least one embodiment of the present application, if the eye tracking module 30 tracks that the user's eye gaze point Ef is currently located in the input area 51 where the key area 52a is located, the operation module 40 determines that the input area 51 where the gaze point Ef is located is the valid input area 51a based on the received position information of the gaze point Ef and the positions of the multiple input areas 51 on the virtual keyboard 50 in the display interface. When the user's finger tip moves into the valid input area 51a, the operation module 40 confirms the input information based on the position information of the first key point corresponding to the finger tip in the valid input area 51a. When the user's finger tip is located in other input areas 51, the operation module 40 does not extract the position information of the first key point corresponding to the finger.

[0035] In step S3, after the camera module 20 detects that the fingertips of the user's fingers move into the valid input area, a second hand image is acquired, and the second hand image is an image including both hands of the user. After receiving the second hand image, the computing module 40 extracts the position information of each first key point in the second hand image in the display interface, including the following steps: extracting the position information of the first key point at the fingertips of each finger in the second hand image in the display interface.

[0036] The process of extracting the first key point at the fingertip of each finger in the second hand image is the same as the process of extracting the first key point at the fingertip of each finger in the first hand image in step S1.

[0037] The operation module 40 extracts the position information of each first key point in the second hand image in the valid input area to confirm the input information, including the following steps: Determine position information of a first key point in a second hand image located within the valid input area; According to the position information of the first key point located in the valid input area and the position information of multiple second key points corresponding to multiple key areas in the valid input area, the key area where the fingertip corresponding to the first key point located in the valid input area is located is determined, and the key area where the fingertip is located is the information for confirming the input.

[0038] Among them, if there are two or more of the first key points located in the valid input area, the order of the input information is confirmed according to the order in which the fingertips corresponding to each first key point move into the valid input area.

[0039] See also Figure 6For example, in at least one embodiment of the present application, when the camera module 20 detects that there is a fingertip located in the effective input area 51a, the second hand image 70 is acquired. After receiving the second hand image 70, the computing module 40 extracts the position information of the plurality of first key points F1-F10 in the second hand image 70, and confirms the key area where the fingertip of the finger corresponding to the first key point located in the effective input area 51a among the plurality of first key points F1-F10 is located. Figure 6 It can be seen that the first key points F2-F4 and F7-F9 are located in the effective input area 51a, that is, there are multiple first key points located in the effective input area 51a. At this time, the camera module 20 detects the order in which the fingertips corresponding to each first key point located in the effective input area 51a move into the effective input area to confirm the order of the input information.

[0040] The method for confirming information input of the present application first determines the position of the virtual keyboard in the display interface according to the position information of multiple first key points in the first hand image, and then divides the virtual keyboard into multiple non-overlapping input areas, and the multiple input areas are arranged in sequence along the extension direction of the short side of the virtual keyboard. Then, the eye gaze point is tracked by eye tracking technology, and an input area where the gaze point is located in the multiple input areas is determined as a valid input area. When the fingertip of the user moves into the valid input area, since the multiple key areas in the valid input area are arranged in sequence along the extension direction of the long side of the virtual keyboard, after determining the valid input area of ​​the virtual keyboard, when the user inputs information, the user's palm is difficult to block the fingertip during the movement of the fingertip in the valid input area. At this time, the second hand image is obtained, and the key area where the fingertip is located is accurately determined according to the position information of the first key point in the second hand image to confirm the information that the user needs to enter, thereby improving the accuracy of information input.

[0041] An embodiment of the present application also provides a non-volatile computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above-mentioned input information confirmation methods when executed by a processor.

[0042] When the above-mentioned input information confirmation method is implemented in the form of a separate product for sale or use, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned implementation method, and can also be completed by instructing the relevant hardware through a computer program, which can be stored in a computer-readable storage medium, and the computer program can be executed by the processor to implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes a computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electrical carrier signal, a telecommunication signal, and a software distribution medium.

[0043] Please refer to Figure 1 The computing module 40 of the head mounted display of the embodiment of the present application includes a memory 41 and a processor 42. The memory 41 stores a computer program. When the processor 42 executes the computer program, the computing module 40 executes an input information confirmation method.

[0044] The processor 42 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the sample processing system / sample processing device, and uses various interfaces and lines to connect various parts of the entire sample processing system / sample processing device.

[0045] The memory 41 is used to store the computer program and / or module. The processor 42 realizes various functions of the sample processing system / sample processing device by running or executing the computer program and / or module stored in the memory 71 and calling the data stored in the memory 41. The memory 41 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 41 may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0046] Those skilled in the art should recognize that the above embodiments are only used to illustrate the present application and are not intended to be limiting of the present application. As long as they are within the spirit and scope of the present application, appropriate changes and modifications to the above embodiments are within the scope of protection claimed in the present application.

Claims

1. A method for confirming input information, characterized in that: include: Acquire a first hand image and extract position information of each first key point in the first hand image in a display interface, and determine a position of a virtual keyboard in the display interface according to the position information of the first key point, wherein the virtual keyboard includes a plurality of non-overlapping input areas; Tracking the eye gaze point at the current moment, and determining an input area where the gaze point is located in the multiple input areas as a valid input area; and A second hand image is acquired and position information of each of the first key points in the second hand image in the valid input area is extracted to confirm input information.

2. The input information confirmation method according to claim 1, characterized in that: The step of confirming the position of the virtual keyboard in the display interface according to the position information of the first key point includes: Extracting position information of first key points at the fingertips of two thumbs in the first hand image in the display interface; Extracting the position information of the middle point of the line segment formed by the first key points at the fingertips of the two thumbs in the display interface; The position of the virtual keyboard in the display interface is determined according to the position information of the middle point.

3. The input information confirmation method according to claim 2, characterized in that: In the step of confirming the position of the virtual keyboard in the display interface according to the position information of the middle point, the middle point is located on the symmetry axis of the virtual keyboard, and all the first key points in the first hand image are located within the virtual keyboard.

4. The input information confirmation method according to claim 1, characterized in that: Each of the input areas includes a plurality of non-overlapping key areas, and after the step of determining the position of the virtual keyboard in the display interface according to the position information of the first key point, the step further includes: The position information of each second key point of each button area in the display interface is extracted, each second key point is located at the edge of the button area, and the area enclosed by each second key point connected in sequence is equal to or smaller than the area of ​​the button area.

5. The input information confirmation method according to claim 4, characterized in that: The step of extracting position information of each of the first key points in the second hand image in the valid input area to confirm input information includes: Determining position information of the first key point in the second hand image located within the valid input area; According to the position information of the first key point located in the valid input area and the position information of multiple second key points corresponding to multiple key areas in the valid input area, the key area where the fingertip corresponding to the first key point in the valid input area is located is determined, and the key area where the fingertip is located is the information for confirming the input.

6. The input information confirmation method according to claim 5, characterized in that: If there are two or more of the first key points located in the valid input area, the order of the input information is determined according to the order in which the fingertips corresponding to each of the first key points move into the valid input area.

7. The input information confirmation method according to claim 1, characterized in that: The first hand image and the second hand image are both images including both hands of the user; The step of extracting the position information of each first key point in the first hand image in the display interface includes: extracting the position information of the first key point at the fingertip of each finger in the first hand image in the display interface; The step of extracting position information of each of the first key points in the second hand image in the valid input area includes: extracting position information of the first key point at the fingertip of each finger in the second hand image in the valid input area.

8. The input information confirmation method according to claim 1, characterized in that: In the step of tracking the eye gaze point at the current moment, the eye gaze point at the current moment is tracked using eye tracking technology.

9. A readable storage medium storing a computer program, characterized in that: When the computer program is executed, the input information confirmation method according to any one of claims 1 to 8 is implemented.

10. A head mounted display, characterized in that: include: A display module, configured to display a virtual keyboard, wherein the virtual keyboard includes a plurality of non-overlapping input areas; A camera module, used to obtain a hand image of the user; An eye tracking module, used for tracking the position of the eye gaze point on the virtual keyboard at the current moment; as well as A computing module is electrically connected to the camera module and the eye tracking module, respectively, and is used to receive the multiple hand images and the position information of the gaze point. The computing module includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the computing module executes the steps of the input information confirmation method as described in any one of claims 1 to 8.

Images