Virtual keyboard display method and device, equipment and storage medium

Abstract

The invention provides a virtual keyboard display method and device, equipment and a storage medium, and belongs to the field of image display.The method comprises the steps that a hand posture image of a wearer of the head-mounted display equipment is obtained, and a keyboard layout mode of a virtual keyboard is determined according to the hand posture image; determining display parameters of the virtual keyboard in the palm of the wearer according to the hand posture image and the keyboard layout mode; and projecting and displaying the virtual keyboard in the palm of the wearer according to the display parameters. According to the method and the device, the accuracy of the display position of the virtual keyboard and the suitability of the display size are greatly improved, so that the accuracy and the convenience of character input in the virtual keyboard by a wearer are improved, and the use experience of the wearer is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of image display technology, and in particular to a virtual keyboard display method, device, equipment and storage medium. Background Art

[0002] With the continuous development of science and technology, the functions and application scenarios of augmented reality (AR) and virtual reality (VR) technologies are increasing, and users are paying more and more attention to the human-computer interaction experience, among which text input is a particularly important part of human-computer interaction. At present, most of the text input is performed through a virtual keyboard. For example, by projecting a virtual keyboard on the palm of the user, the user enters text on the virtual keyboard on the palm. However, in actual application, due to the user's personal habits or different input content, there will be different keyboard layout requirements. For example, when inputting pure Chinese content, it is usually more convenient to use a 9-key keyboard; however, when inputting English or a mixture of Chinese, English, and numbers, it is more convenient to use a 26-key full keyboard. Therefore, how to accurately display the virtual keyboard on the user's palm is a problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the present application is to provide a virtual keyboard display method, device, equipment and storage medium, aiming to accurately display a virtual keyboard on the palm of a user to improve the user experience.

[0004] In a first aspect, the present application provides a virtual keyboard display method, which is applied to a head mounted display device, and the virtual keyboard display method comprises the following steps:

[0005] In some embodiments, a hand gesture image of a wearer of the head mounted display device is obtained, and a keyboard layout of a virtual keyboard is determined according to the hand gesture image;

[0006] Determining display parameters of the virtual keyboard in the palm of the wearer according to the hand gesture image and the keyboard layout;

[0007] The virtual keyboard is projected and displayed in the palm of the wearer according to the display parameters.

[0008] The determining of the keyboard layout of the virtual keyboard according to the hand gesture image includes:

[0009] Performing gesture recognition on the hand gesture image through a preset hand recognition model to obtain the hand gesture, wherein the preset hand recognition model is a pre-trained neural network model;

[0010] The keyboard layout of the virtual keyboard is determined according to the hand gesture and the preset association relationship, wherein the preset association relationship is used to describe the relationship between the keyboard layout and the hand gesture.

[0011] In some embodiments, determining display parameters of the virtual keyboard in the palm of the wearer according to the hand gesture image and the keyboard layout includes:

[0012] Determining a plurality of display key points of the virtual keyboard displayed in the hand gesture image according to the keyboard layout mode;

[0013] Display parameters of the virtual keyboard in the palm of the wearer are determined according to the multiple display key points.

[0014] In some embodiments, determining a plurality of display key points of the virtual keyboard displayed in the hand gesture image according to the keyboard layout mode includes:

[0015] Performing palm key point recognition on the hand posture image to obtain a plurality of palm key points;

[0016] The multiple display key points are determined according to a preset relationship between a keyboard layout mode and display key points and the multiple palm key points.

[0017] In some embodiments, determining display parameters of the virtual keyboard in the palm of the wearer according to the plurality of display key points includes:

[0018] Determine the minimum bounding rectangle of the plurality of display key points;

[0019] The display parameters are determined according to the circumscribed rectangle.

[0020] In some embodiments, after projecting and displaying the virtual keyboard in the palm of the wearer according to the display parameters, the method further includes:

[0021] When a change in the hand posture of the wearer is detected, the current keyboard layout of the virtual keyboard is switched to a keyboard layout corresponding to the changed hand posture.

[0022] In some embodiments, the method further comprises:

[0023] In response to a virtual keyboard closing instruction triggered by the wearer, closing the virtual keyboard.

[0024] In a second aspect, the present application further provides a virtual keyboard display device, the virtual keyboard display device comprising an acquisition module, a determination module and a display module, wherein:

[0025] The acquisition module is used to acquire a hand posture image of a wearer of the head mounted display device, and determine a keyboard layout of the virtual keyboard according to the hand posture image;

[0026] The determination module is used to determine display parameters of the virtual keyboard in the palm of the wearer according to the hand posture image and the keyboard layout mode;

[0027] The display module is used to project and display the virtual keyboard in the palm of the wearer according to the display parameters.

[0028] In a third aspect, the present application also provides a head-mounted display device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the virtual keyboard display method as described above are implemented.

[0029] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the virtual keyboard display method as described above are implemented.

[0030] The present application provides a virtual keyboard display method, device, equipment and storage medium. The present application obtains a hand posture image of a wearer of a head-mounted display device, and determines a keyboard layout mode of a virtual keyboard according to the hand posture image; then, according to the hand posture image and the keyboard layout mode, determines the display parameters of the virtual keyboard in the palm of the wearer; and projects and displays the virtual keyboard in the palm of the wearer according to the display parameters. In the present application, the keyboard layout mode of the virtual keyboard required by the wearer can be determined based on the hand posture image of the wearer, and then the display parameters of the virtual keyboard displayed in the palm of the wearer can be accurately determined according to the hand posture image and the keyboard layout mode, and the virtual keyboard is projected and displayed in the palm of the wearer based on the display parameters, which greatly improves the accuracy of the display position of the virtual keyboard and the suitability of the display size, thereby improving the accuracy and convenience of the wearer's text input in the virtual keyboard, and greatly improving the wearer's user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 A flowchart of a virtual keyboard display method provided in an embodiment of the present application;

[0033] Figure 2 for Figure 1 A schematic diagram of the sub-step flow of the virtual keyboard display method in FIG.

[0034] Figure 3 for Figure 1 A schematic diagram of the sub-step flow of the virtual keyboard display method in FIG.

[0035] Figure 4 A schematic diagram of a scenario in which a palm displays key points provided in an embodiment of the present application;

[0036] Figure 5 A schematic diagram of a scene of a virtual keyboard display provided in an embodiment of the present application;

[0037] Figure 6 A schematic diagram of another scenario of a virtual keyboard display provided in an embodiment of the present application;

[0038] Figure 7 A schematic block diagram of a virtual keyboard display device provided in an embodiment of the present application;

[0039] Figure 8 for Figure 7 A schematic block diagram of a submodule of a virtual keyboard display device;

[0040] Fig. 9 A schematic block diagram of the structure of a head mounted display device provided in an embodiment of the present application.

[0041] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0044] With the continuous development of science and technology, the functions and application scenarios of augmented reality (AR) and virtual reality (VR) technologies are increasing, and users are paying more and more attention to the human-computer interaction experience, among which text input is a particularly important part of human-computer interaction. At present, most of the text input is performed through a virtual keyboard. For example, by projecting a virtual keyboard on the palm of the user, the user enters text on the virtual keyboard on the palm. However, in actual application, due to the user's personal habits or different input content, there will be different keyboard layout requirements. For example, when inputting pure Chinese content, it is usually more convenient to use a 9-key keyboard; however, when inputting English or a mixture of Chinese, English, and numbers, it is more convenient to use a 26-key full keyboard. Therefore, how to accurately display the virtual keyboard on the user's palm is a problem that needs to be solved urgently.

[0045] To solve the above problems, the embodiments of the present application provide a virtual keyboard display method, device, equipment and storage medium. The hand posture image of the wearer of the head mounted display device is obtained, and the keyboard layout of the virtual keyboard is determined according to the hand posture image; the display parameters of the virtual keyboard in the palm of the wearer are determined according to the hand posture image and the keyboard layout; and the virtual keyboard is projected and displayed in the palm of the wearer according to the display parameters.

[0046] Among them, the virtual keyboard display method can be applied to a head-mounted display device, and the head-mounted display device can be an augmented reality glasses, an AR helmet, a VR glasses, a mixed reality (Mixed Reality, MR) glasses, an MR helmet and other devices.

[0047] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0048] Please refer to Figure 1 , Figure 1 A flowchart of a virtual keyboard display method provided in an embodiment of the present application.

[0049] like Figure 1 As shown, the virtual keyboard display method includes steps S101 to S104.

[0050] Step S101: Acquire a hand gesture image of a wearer of the head mounted display device, and determine a keyboard layout of a virtual keyboard according to the hand gesture image.

[0051] Among them, the hand posture image is the posture image of the wearer's hand, and the hand posture image is an image of the wearer's palm facing the head-mounted display device. The hand posture image includes a horizontal hand posture image and a vertical hand posture image, etc., wherein the palm direction is generally the palm direction.

[0052] In some embodiments, in response to a virtual keyboard start instruction triggered by the wearer, a hand gesture image of the wearer is collected through a camera, and a keyboard layout of the virtual keyboard is determined according to the hand gesture image. The camera may be a camera disposed in the head mounted display device.

[0053] It should be noted that the way in which the wearer triggers the virtual keyboard activation command can be set according to actual conditions, and the embodiment of the present invention does not specifically limit this. For example, the wearer's voice virtual keyboard activation command and the wearer's touch virtual keyboard activation command on the head-mounted display device, etc.

[0054] In some embodiments, Figure 2 As shown, step S101 includes sub-steps S1011 to S1012.

[0055] Sub-step S1011: performing gesture recognition on the hand gesture image through a preset hand recognition model to obtain the hand gesture, wherein the preset hand recognition model is a pre-trained neural network model.

[0056] Among them, the preset hand recognition model is a pre-trained neural network model, which includes but is not limited to a convolutional neural network model, a recurrent neural network model and a recurrent convolutional neural network model.

[0057] In some embodiments, a first sample data set is obtained, wherein the first sample data set includes multiple first sample data, each first sample data includes a hand posture image and a labeled hand posture; any one of the first sample data from the first sample data set is selected as the first target sample data, and the hand posture image in the first target sample data is input into a preset neural network model for posture recognition to obtain a predicted hand posture; based on the predicted hand posture and the labeled hand posture, it is determined whether the preset neural network model converges; if the preset neural network model has not converged, the step of selecting any one of the first sample data from the first sample data set as the first target sample data is continued until the preset neural network model converges to obtain a preset hand recognition model.

[0058] In some embodiments, the method of determining whether the preset neural network model has converged based on the predicted hand posture and the labeled hand posture can be: determining the loss value of the preset neural network model based on the predicted hand posture and the labeled hand posture, and determining that the preset neural network model has not converged when the loss value of the preset neural network model is greater than or equal to the preset loss value; and determining that the preset neural network model has converged when the loss value of the preset neural network model is less than the preset loss value. The preset loss value can be set according to actual conditions, and the present invention does not specifically limit this. For example, the preset loss value can be set to 0.02.

[0059] In some embodiments, the loss value of the preset neural network model can be determined based on the predicted hand posture and the labeled hand posture by: calculating the cosine similarity of the predicted hand posture and the labeled hand posture to obtain a similarity value; subtracting the similarity value from the unit 1 to obtain a parameter value to be determined as the loss value.

[0060] In some embodiments, the hand gesture is recognized by a preset hand recognition model on the hand gesture image to obtain the hand gesture. By performing gesture recognition on the hand gesture image by a preset hand recognition model, the hand gesture of the wearer can be accurately obtained, which greatly improves the efficiency and accuracy of the virtual keyboard display.

[0061] Sub-step S1012: determining a keyboard layout of the virtual keyboard according to the hand gesture and a preset association relationship, wherein the preset association relationship is used to describe the relationship between the keyboard layout and the hand gesture.

[0062] Among them, the keyboard layout includes a 9-key keyboard layout and a 26-key keyboard layout. Under different hand posture conditions, there are a vertical hand posture 9-key keyboard layout, a horizontal hand posture 9-key keyboard layout, and a horizontal hand posture 26-key keyboard layout. It should be noted that the present application is also applicable to the handwriting keyboard layout, that is, the handwriting keyboard layout and the 9-key keyboard layout can be regarded as keyboard layouts of the same size.

[0063] In some embodiments, a preset association relationship between hand gestures and keyboard layout modes is obtained, and the keyboard layout mode of the virtual keyboard is determined based on the preset association relationship and the hand gesture, wherein the preset association relationship is used to describe the relationship between the keyboard layout mode and the hand gesture. The preset association relationship can be set according to actual conditions, and the embodiments of the present invention do not make specific limitations on this. For example, the keyboard layout mode corresponding to the hand gesture of a vertical hand gesture is a 9-key keyboard layout mode of a vertical hand gesture.

[0064] Step S102: Determine display parameters of the virtual keyboard in the palm of the wearer according to the hand gesture image and the keyboard layout.

[0065] The display parameters include a display position and a display size. The display position is a display area of ​​the virtual keyboard in the palm of the wearer, and the display size includes a length and a width of the virtual keyboard.

[0066] In some embodiments, Figure 3 As shown, step S102 includes sub-steps S1021 and S1022.

[0067] Sub-step S1021: determining a plurality of display key points of the virtual keyboard displayed in the hand gesture image according to the keyboard layout mode.

[0068] Among them, the display key points are the key points set for each position point of the palm, such as Figure 4 As shown, the palm diagram included in the figure includes 21 palm key points, wherein key point 0 is a key point close to the wrist, and the other palm key points are points at each finger joint.

[0069] In some embodiments, palm key points are recognized on the hand gesture image to obtain multiple palm key points; multiple display key points are determined based on the preset relationship between the keyboard layout mode and the display key points and the multiple palm key points. By recognizing the hand gesture image, the palm key points can be accurately obtained, and the preset relationship can be used to determine multiple display key points, which greatly improves the efficiency and accuracy of the virtual keyboard display.

[0070] In some embodiments, the palm key point recognition of the hand gesture image can be performed to obtain multiple palm key points by: recognizing the hand gesture image through a preset palm key point recognition model to obtain multiple palm key points. The preset palm key point recognition model is a pre-trained neural network model, which includes but is not limited to a convolutional neural network model, a recurrent neural network model, and a recurrent convolutional neural network model. By performing recognition through the preset palm key point recognition model, multiple palm key points can be accurately obtained, which greatly improves the efficiency and accuracy of the virtual keyboard display.

[0071] In some embodiments, the method for training a preset palm key point recognition model can be: obtaining a second sample data set, wherein the second sample data set includes multiple second sample data, each second sample data includes a hand posture image and annotated palm key points; selecting any second sample data from the second sample data set as the second target sample data, inputting the hand posture image in the second target sample data into the preset neural network model for palm key points, and obtaining predicted palm key points; determining whether the preset neural network model converges based on the predicted palm key points and the annotated palm key points; if the preset neural network model has not converged, continuing to execute the step of selecting any second sample data from the second sample data set as the second target sample data until the preset neural network model converges to obtain the preset palm key point recognition model.

[0072] In some embodiments, the method of determining whether the preset neural network model has converged based on the predicted palm key points and the labeled palm key points can be: determining the loss value of the preset neural network model based on the predicted palm key points and the labeled palm key points, and determining that the preset neural network model has not converged when the loss value of the preset neural network model is greater than or equal to the preset threshold; determining that the preset neural network model has converged when the loss value of the preset neural network model is less than the preset threshold. The preset threshold can be set according to actual conditions, and the present invention does not specifically limit this. For example, the preset threshold can be set to 0.02.

[0073] In some embodiments, the loss value of the preset neural network model can be determined based on the predicted palm key points and the labeled palm key points by: calculating the cosine similarity of the predicted palm key points and the labeled palm key points to obtain a similarity value; subtracting the similarity value from the unit 1 to obtain a parameter value to be determined as the loss value.

[0074] In some embodiments, according to the preset relationship between the keyboard layout mode and the display key points and the multiple palm key points, the method of determining the multiple display key points can be: obtaining the preset relationship between the preset keyboard layout mode and the display key points, and determining the display key points matched by the keyboard layout mode according to the preset relationship. Among them, the preset relationship is established in advance according to the keyboard layout mode and the display key points, and the preset relationship can be set according to the actual situation, and the embodiment of the present invention does not specifically limit this. Through the preset relationship, the display key points matched by the keyboard layout mode can be accurately obtained.

[0075] Exemplarily, when the keyboard layout is a vertical hand gesture 9-key keyboard layout, a preset relationship between the preset keyboard layout and the display key points is obtained, and the display key points matched by the vertical hand gesture 9-key keyboard layout are determined according to the preset relationship. Figure 4Key point 0, key point 1, key point 2, key point 5, key point 9, key point 13 and key point 17 in.

[0076] Exemplarily, when the keyboard layout is a horizontal hand gesture 9-key keyboard layout, a preset relationship between the preset keyboard layout and the display key points is obtained, and the display key points matched by the horizontal hand gesture 9-key keyboard layout are determined according to the preset relationship. Figure 4 Key point 0, key point 1, key point 2, key point 5, key point 9, key point 13 and key point 17 in.

[0077] Exemplarily, when the keyboard layout is a horizontal hand posture 26-key keyboard layout, a preset relationship between the preset keyboard layout and the display key points is obtained, and the display key points matched by the horizontal hand posture 26-key keyboard layout are determined according to the preset relationship. Figure 4 Key points 0, 1, 6 and 18 in .

[0078] Exemplarily, when the keyboard layout is a horizontal hand posture 26-key keyboard layout, a preset relationship between the preset keyboard layout and the display key points is obtained, and the display key points matched by the horizontal hand posture 26-key keyboard layout are determined according to the preset relationship. Figure 4 Key points 0, 1, 7 and 19 in .

[0079] Sub-step S1022: determining display parameters of the virtual keyboard in the palm of the wearer according to the multiple display key points.

[0080] In some embodiments, the minimum bounding rectangle of multiple display key points is determined; and the display parameters are determined based on the bounding rectangle. The display parameters in the palm of the wearer can be accurately determined by the minimum bounding rectangle of multiple display key points, which greatly improves the efficiency and accuracy of the virtual keyboard display.

[0081] In some embodiments, the display parameters can be determined according to the circumscribed rectangle by obtaining the center point of the circumscribed rectangle, obtaining the center point of the matrix, and using the center point of the matrix as the keyboard center point of the virtual keyboard, scaling the standard virtual keyboard with the keyboard center point, and when any point of the standard virtual keyboard touches any of the display key points, obtaining the size of the virtual keyboard at this time, and determining the size of the virtual keyboard at this time as the display size. The center point of the keyboard is used as the display position for displaying the virtual keyboard. Among them, the standard virtual keyboard is a virtual keyboard with a preset length and width, and the standard virtual keyboard can be set according to actual conditions, and the embodiments of the present invention do not specifically limit this.

[0082] Step S103: projecting and displaying the virtual keyboard in the palm of the wearer according to the display parameters.

[0083] The virtual keyboard is projected and displayed in the palm of the wearer according to the display parameters. By displaying the parameters, the virtual keyboard can be accurately displayed in the palm of the wearer, which greatly improves the user experience.

[0084] Exemplarily, a hand posture image of a wearer of a head mounted display device is obtained, and according to the hand posture image, a keyboard layout of a virtual keyboard is determined to be a vertical hand posture 9-key keyboard layout; according to the hand posture image and the vertical hand posture 9-key keyboard layout, display parameters of the virtual keyboard in the palm of the wearer are determined; and according to the display parameters, a projection display is performed in the palm of the wearer to obtain the following Figure 5 Virtual keyboard.

[0085] Exemplarily, a hand posture image of a wearer of a head mounted display device is obtained, and according to the hand posture image, a keyboard layout of a virtual keyboard is determined to be a horizontal hand posture 26-key keyboard layout; according to the hand posture image and the horizontal hand posture 26-key keyboard layout, display parameters of the virtual keyboard in the palm of the wearer are determined; and according to the display parameters, a projection display is performed in the palm of the wearer to obtain the following Figure 6 Virtual keyboard.

[0086] In some embodiments, when a change in the wearer's hand posture is detected, the current keyboard layout of the virtual keyboard is switched to a keyboard layout corresponding to the changed hand posture. By changing the hand posture, the keyboard layout can be efficiently changed, greatly improving the wearer's experience.

[0087] It should be noted that the changes in hand posture include but are not limited to the horizontal and vertical transformation of the palm, the vertical and horizontal transformation of the palm and the palm flipping, among which the horizontal and vertical transformation of the palm represents the transformation of the vertical hand posture 9-key keyboard layout mode to the horizontal hand posture 26-key keyboard layout mode; the vertical and horizontal transformation of the palm represents the transformation of the horizontal hand posture 26-key keyboard layout mode to the vertical hand posture 9-key keyboard layout mode; palm flipping represents the transformation of the horizontal hand posture 26-key keyboard layout mode to the horizontal hand posture 9-key keyboard layout mode, or the transformation of the horizontal hand posture 9-key keyboard layout mode to the horizontal hand posture 26-key keyboard layout mode.

[0088] Exemplarily, when it is detected that the wearer's hand posture changes from a horizontal hand posture to a vertical hand posture, the current virtual keyboard is closed, and the keyboard layout mode of the virtual keyboard is determined according to the hand posture image; the display parameters of the virtual keyboard in the palm of the wearer are determined according to the hand posture image and the keyboard layout mode; the virtual keyboard is projected and displayed in the palm of the wearer according to the display parameters to complete the switching of the virtual keyboard.

[0089] Exemplarily, when it is detected that the wearer's hand posture is a palm flip, and the current keyboard layout is a 26-key keyboard layout in a horizontal hand posture, it is necessary to change the display to a 9-key keyboard layout in a horizontal hand posture, and the display size of the 9-key keyboard layout in the horizontal hand posture in the wearer's palm is determined according to multiple display key points, and the virtual keyboard is projected and displayed in the wearer's palm according to the display parameters.

[0090] In some embodiments, the virtual keyboard is closed in response to a virtual keyboard closing instruction triggered by the wearer. By responding to the virtual keyboard closing instruction, the virtual keyboard can be closed in a timely manner, which greatly improves the efficiency and accuracy of virtual keyboard control.

[0091] It should be noted that the virtual keyboard closing instruction can be set according to actual conditions, and the embodiment of the present invention does not specifically limit this. For example, the virtual keyboard closing instruction can be the wearer's virtual keyboard closing voice, virtual keyboard closing touch case instruction, virtual keyboard closing gesture, etc.

[0092] The virtual keyboard display method provided in the above embodiment obtains the hand posture image of the wearer of the head mounted display device, and determines the keyboard layout of the virtual keyboard according to the hand posture image; then determines the display parameters of the virtual keyboard in the palm of the wearer according to the hand posture image and the keyboard layout; and projects and displays the virtual keyboard in the palm of the wearer according to the display parameters. In the present application, the keyboard layout of the virtual keyboard required by the wearer can be determined based on the hand posture image of the wearer, and then the display parameters of the virtual keyboard displayed in the palm of the wearer can be accurately determined according to the hand posture image and the keyboard layout, and the virtual keyboard is projected and displayed in the palm of the wearer based on the display parameters, which greatly improves the accuracy of the display position of the virtual keyboard and the suitability of the display size, thereby improving the accuracy and convenience of the wearer's text input in the virtual keyboard, and greatly improving the wearer's user experience.

[0093] See also Figure 7 , Figure 7 A schematic block diagram of a virtual keyboard display device provided in an embodiment of the present application.

[0094] like Figure 7 As shown, the virtual keyboard display device 200 includes an acquisition module 210, a determination module 220 and a display module 230, wherein:

[0095] The acquisition module 210 is used to acquire a hand posture image of a wearer of the head mounted display device, and determine a keyboard layout of the virtual keyboard according to the hand posture image;

[0096] The determination module 220 is used to determine display parameters of the virtual keyboard in the palm of the wearer according to the hand posture image and the keyboard layout mode;

[0097] The display module 230 is used to project and display the virtual keyboard in the palm of the wearer according to the display parameters.

[0098] In some embodiments, the determining module 220 is further configured to:

[0099] Performing gesture recognition on the hand gesture image through a preset hand recognition model to obtain the hand gesture, wherein the preset hand recognition model is a pre-trained neural network model;

[0100] The keyboard layout of the virtual keyboard is determined according to the hand gesture and the preset association relationship, wherein the preset association relationship is used to describe the relationship between the keyboard layout and the hand gesture.

[0101] In some embodiments, Figure 8 As shown, the determination module 220 includes a first determination submodule 221 and a second determination submodule 222, wherein:

[0102] The first determination submodule 221 is used to determine a plurality of display key points of the virtual keyboard displayed in the hand gesture image according to the keyboard layout mode;

[0103] The second determination submodule 222 is used to determine display parameters of the virtual keyboard in the palm of the wearer according to the multiple display key points.

[0104] In some embodiments, the first determining submodule 221 is further configured to:

[0105] Performing palm key point recognition on the hand posture image to obtain a plurality of palm key points;

[0106] The multiple display key points are determined according to a preset relationship between a keyboard layout mode and display key points and the multiple palm key points.

[0107] In some embodiments, the second determining submodule 222 is further configured to:

[0108] Determine the minimum bounding rectangle of the plurality of display key points;

[0109] The display parameters are determined according to the circumscribed rectangle.

[0110] In some embodiments, the virtual keyboard display device 200 is further used for:

[0111] When a change in the hand posture of the wearer is detected, the current keyboard layout of the virtual keyboard is switched to a keyboard layout corresponding to the changed hand posture.

[0112] In some embodiments, the virtual keyboard display device 200 is further used for:

[0113] In response to a virtual keyboard closing instruction triggered by the wearer, closing the virtual keyboard.

[0114] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned virtual keyboard display device can refer to the corresponding process in the aforementioned virtual keyboard display method embodiment, and will not be repeated here.

[0115] See also Fig. 9 , Fig. 9 A schematic block diagram of the structure of a head mounted display device provided in an embodiment of the present application.

[0116] like Fig. 9 As shown, the head mounted display device 200 includes a processor 202 and a memory 203 connected via a system bus 201, wherein the memory 203 may include a storage medium and an internal memory.

[0117] The storage medium may store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor 202 may execute any virtual keyboard display method.

[0118] The processor 202 is used to provide computing and control capabilities to support the operation of the entire head-mounted display device.

[0119] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any virtual keyboard display method.

[0120] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the head-mounted display device to which the scheme of the present application is applied. The specific head-mounted display device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] It should be understood that the processor 202 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0122] In one embodiment, the processor 202 is used to run a computer program stored in the memory to implement the following steps:

[0123] Acquire a hand gesture image of a wearer of the head mounted display device, and determine a keyboard layout of a virtual keyboard according to the hand gesture image;

[0124] Determining display parameters of the virtual keyboard in the palm of the wearer according to the hand gesture image and the keyboard layout;

[0125] The virtual keyboard is projected and displayed in the palm of the wearer according to the display parameters.

[0126] In one embodiment, when the processor 202 implements the determining of the keyboard layout of the virtual keyboard according to the hand gesture image, it is configured to implement:

[0127] Performing gesture recognition on the hand gesture image through a preset hand recognition model to obtain the hand gesture, wherein the preset hand recognition model is a pre-trained neural network model;

[0128] The keyboard layout of the virtual keyboard is determined according to the hand gesture and the preset association relationship, wherein the preset association relationship is used to describe the relationship between the keyboard layout and the hand gesture.

[0129] In one embodiment, when the processor 202 determines the display parameters of the virtual keyboard in the palm of the wearer according to the hand gesture image and the keyboard layout, it is used to implement:

[0130] Determining a plurality of display key points of the virtual keyboard displayed in the hand gesture image according to the keyboard layout mode;

[0131] Display parameters of the virtual keyboard in the palm of the wearer are determined according to the multiple display key points.

[0132] In one embodiment, when the processor 202 determines the multiple display key points of the virtual keyboard displayed in the hand gesture image according to the keyboard layout mode, it is used to implement:

[0133] Performing palm key point recognition on the hand posture image to obtain a plurality of palm key points;

[0134] The multiple display key points are determined according to a preset relationship between a keyboard layout mode and display key points and the multiple palm key points.

[0135] In one embodiment, when the processor 202 determines the display parameters of the virtual keyboard in the palm of the wearer according to the multiple display key points, it is used to implement:

[0136] Determine the minimum bounding rectangle of the plurality of display key points;

[0137] The display position and the display size are determined according to the circumscribed rectangle.

[0138] In one embodiment, after implementing the projecting and displaying of the virtual keyboard in the palm of the wearer according to the display parameters, the processor 202 is further configured to implement:

[0139] When a change in the hand posture of the wearer is detected, the current keyboard layout of the virtual keyboard is switched to a keyboard layout corresponding to the changed hand posture.

[0140] In one embodiment, the processor 202 is further configured to implement:

[0141] In response to a virtual keyboard closing instruction triggered by the wearer, closing the virtual keyboard.

[0142] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the virtual keyboard display method described above can refer to the corresponding process in the aforementioned virtual keyboard display method embodiment, and will not be repeated here.

[0143] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the virtual keyboard display method of the present application.

[0144] The computer-readable storage medium may be an internal storage unit of the head-mounted display device described in the foregoing embodiment, such as a hard disk or memory of the head-mounted display device. The computer-readable storage medium may be non-volatile or volatile. The computer-readable storage medium may also be an external storage device of the head-mounted display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the head-mounted display device.

[0145] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0146] It should also be understood that the term "and / or" used in the present specification refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0147] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application.

Claims

1. A virtual keyboard display method, characterized in that: Applied to a head mounted display device, the method comprises: Acquire a hand gesture image of a wearer of the head mounted display device, and determine a keyboard layout of a virtual keyboard according to the hand gesture image; Determining display parameters of the virtual keyboard in the palm of the wearer according to the hand gesture image and the keyboard layout; The virtual keyboard is projected and displayed in the palm of the wearer according to the display parameters.

2. The virtual keyboard display method according to claim 1, characterized in that: The determining of the keyboard layout of the virtual keyboard according to the hand gesture image includes: Performing gesture recognition on the hand gesture image through a preset hand recognition model to obtain the hand gesture, wherein the preset hand recognition model is a pre-trained neural network model; The keyboard layout of the virtual keyboard is determined according to the hand gesture and the preset association relationship, wherein the preset association relationship is used to describe the relationship between the keyboard layout and the hand gesture.

3. The virtual keyboard display method according to claim 2, characterized in that: The step of determining display parameters of the virtual keyboard in the palm of the wearer according to the hand gesture image and the keyboard layout includes: Determining a plurality of display key points of the virtual keyboard displayed in the hand gesture image according to the keyboard layout mode; Display parameters of the virtual keyboard in the palm of the wearer are determined according to the multiple display key points.

4. The virtual keyboard display method according to claim 3, characterized in that: The step of determining a plurality of display key points of the virtual keyboard displayed in the hand gesture image according to the keyboard layout mode includes: Performing palm key point recognition on the hand posture image to obtain a plurality of palm key points; The multiple display key points are determined according to a preset relationship between a keyboard layout mode and display key points and the multiple palm key points.

5. The virtual keyboard display method according to claim 3, characterized in that: The step of determining display parameters of the virtual keyboard in the palm of the wearer according to the plurality of display key points comprises: Determine the minimum bounding rectangle of the plurality of display key points; The display parameters are determined according to the circumscribed rectangle.

6. The virtual keyboard display method according to any one of claims 1 to 5, characterized in that: After projecting and displaying the virtual keyboard in the palm of the wearer according to the display parameters, the method further includes: When a change in the hand posture of the wearer is detected, the current keyboard layout of the virtual keyboard is switched to a keyboard layout corresponding to the changed hand posture.

7. The virtual keyboard display method according to any one of claims 1 to 5, characterized in that: The method further comprises: In response to a virtual keyboard closing instruction triggered by the wearer, closing the virtual keyboard.

8. A virtual keyboard display device, characterized in that: The virtual keyboard display device includes an acquisition module, a determination module and a display module, wherein: The acquisition module is used to acquire a hand posture image of a wearer of the head mounted display device, and determine a keyboard layout of the virtual keyboard according to the hand posture image; The determination module is used to determine display parameters of the virtual keyboard in the palm of the wearer according to the hand posture image and the keyboard layout mode; The display module is used to project and display the virtual keyboard in the palm of the wearer according to the display parameters.

9. A head mounted display device, characterized in that: The head mounted display device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the virtual keyboard display method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the virtual keyboard display method according to any one of claims 1 to 7 are implemented.

Images