Virtual keyboard display method and device, equipment and storage medium

Abstract

The invention provides a virtual keyboard display method and device, equipment and a storage medium, and belongs to the field of image display, and the method comprises the steps: obtaining a hand image of a wearer of head-mounted display equipment, and determining image coordinates and relative depth information of a plurality of palm key points according to the hand image; according to the image coordinates and the relative depth information of each palm key point, a virtual space plane is fitted, and the distances between all points in the virtual space plane and the palm are equal; determining space coordinate information of an area occupied by the virtual keyboard in a virtual space plane according to the plurality of palm key points; and projecting and displaying the virtual keyboard in the palm of the wearer according to the space coordinate information. Each position of the virtual keyboard displayed through projection can be uniformly attached to the palm, so that a user has the same touch feedback in the use process, the accuracy and convenience of character input in the virtual keyboard by a wearer are improved, and the use experience of the wearer is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of image display technology, and in particular to a virtual keyboard display method, device, equipment and storage medium. Background Art

[0002] With the continuous development of science and technology, the functions and application scenarios of augmented reality (AR) and virtual reality (VR) technologies are increasing, and users are paying more and more attention to the human-computer interaction experience, among which text input is a particularly important part of human-computer interaction. At present, most of the text input is performed through a virtual keyboard. For example, by projecting a virtual keyboard on the palm of the user, the user inputs text on the virtual keyboard on the palm. However, each position of the current virtual keyboard cannot fit the palm evenly, and then the user lacks tactile feedback during use, which weakens the interactive immersion and makes the user experience poor. Summary of the invention

[0003] The main purpose of the present application is to provide a virtual keyboard display method, device, equipment and storage medium, aiming to accurately display a virtual keyboard on the palm of a user to improve the user experience.

[0004] In a first aspect, the present application provides a virtual keyboard display method, the virtual keyboard display method comprising the following steps:

[0005] Acquire a hand image of a wearer of the head mounted display device, and determine image coordinates and relative depth information of a plurality of palm key points according to the hand image;

[0006] Fitting a virtual space plane according to the image coordinates and relative depth information of each key point of the palm, wherein the distance between each point in the virtual space plane and the palm is equal;

[0007] Determining spatial coordinate information of an area occupied by a virtual keyboard in the virtual space plane according to the multiple palm key points;

[0008] The virtual keyboard is displayed in the palm of the wearer according to the spatial coordinate information.

[0009] In a second aspect, the present application further provides a virtual keyboard display device, the virtual keyboard display device comprising an acquisition module, a determination module, a fitting module and a projection display module, wherein:

[0010] The acquisition module is used to acquire a hand image of a wearer of the head mounted display device;

[0011] The determination module is used to determine the image coordinates and relative depth information of multiple palm key points according to the hand image;

[0012] The fitting module is used to fit a virtual space plane according to the image coordinates and relative depth information of each key point of the palm, wherein the distance between each point in the virtual space plane and the palm is equal;

[0013] The determination module is further used to determine the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the multiple palm key points;

[0014] The projection display module is used to display the virtual keyboard in the palm of the wearer according to the spatial coordinate information.

[0015] In a third aspect, the present application also provides a head-mounted display device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the virtual keyboard display method as described above are implemented.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the virtual keyboard display method as described above are implemented.

[0017] The present application provides a method, device, equipment and storage medium for displaying a virtual keyboard. The present application obtains a hand image of a wearer of a head-mounted display device, and determines the image coordinates and relative depth information of multiple palm key points according to the hand image; according to the image coordinates and relative depth information of each palm key point, a virtual space plane is fitted, and the distance between each point and the palm is equal; according to multiple palm key points, the spatial coordinate information of the area occupied by the virtual keyboard is determined in the virtual space plane; according to the spatial coordinate information, the virtual keyboard is displayed in the palm of the wearer. In the present application, a virtual space plane is fitted in which each point on a plane is equidistant from the palm through the image coordinates and relative depth information of the palm key points, and a virtual keyboard is displayed on the virtual space plane, so that each position of the virtual keyboard in the virtual space plane is also equidistant from the palm, and the user has the same tactile feedback during use, thereby improving the accuracy and convenience of the wearer's text input in the virtual keyboard, and greatly improving the wearer's use experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a virtual keyboard display method provided in an embodiment of the present application;

[0020] Figure 2 for Figure 1 A schematic diagram of the sub-step flow of the virtual keyboard display method in FIG.

[0021] Figure 3 A schematic diagram of a scenario for implementing the virtual keyboard display method provided in this embodiment;

[0022] Figure 4 for Figure 1 A schematic diagram of the sub-step flow of the virtual keyboard display method in FIG.

[0023] Figure 5 A schematic diagram of another scenario for implementing the virtual keyboard display method provided in this embodiment;

[0024] Figure 6 A schematic diagram of another scenario for implementing the virtual keyboard display method provided in this embodiment;

[0025] Figure 7 A schematic diagram of another scenario for implementing the virtual keyboard display method provided in this embodiment;

[0026] Figure 8 A schematic block diagram of a virtual keyboard display device provided in an embodiment of the present application;

[0027] Fig. 9 for Figure 8 A schematic block diagram of a submodule of a virtual keyboard display device;

[0028] Fig.10 A schematic block diagram of the structure of a head mounted display device provided in an embodiment of the present application.

[0029] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0032] With the continuous development of science and technology, the functions and application scenarios of augmented reality (AR) and virtual reality (VR) technologies are increasing, and users are paying more and more attention to the human-computer interaction experience, among which text input is a particularly important part of human-computer interaction. At present, most of the text input is performed through a virtual keyboard. For example, by projecting a virtual keyboard on the palm of the user, the user inputs text on the virtual keyboard on the palm. However, each position of the current virtual keyboard cannot fit the palm evenly, and then the user lacks tactile feedback during use, which weakens the interactive immersion and makes the user experience poor.

[0033] To solve the above problems, the embodiments of the present application provide a virtual keyboard display method, device, equipment and storage medium. The virtual keyboard display method includes obtaining a hand image of a wearer of a head mounted display device, and determining the image coordinates and relative depth information of multiple palm key points according to the hand image; fitting a virtual space plane according to the image coordinates and relative depth information of each palm key point, wherein the distance between each point in the virtual space plane and the palm is equal; determining the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the multiple palm key points; and displaying the virtual keyboard in the palm of the wearer according to the spatial coordinate information.

[0034] Among them, the virtual keyboard display method can be applied to a head-mounted display device, which can be an augmented reality glasses, an AR helmet, a VR glasses, a mixed reality (Mixed Reality, MR) glasses, an MR helmet and other devices.

[0035] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0036] Please refer to Figure 1 , Figure 1 A flowchart of a virtual keyboard display method provided in an embodiment of the present application.

[0037] like Figure 1 As shown, the virtual keyboard display method includes steps S101 to S104.

[0038] Step S101: Acquire a hand image of a wearer of a head mounted display device, and determine image coordinates and relative depth information of a plurality of palm key points according to the hand image.

[0039] The hand image is a posture image of the wearer's hand, and the hand image is an image with the wearer's palm facing the head-mounted display device.

[0040] In some embodiments, in response to a virtual keyboard activation instruction triggered by the wearer, a hand image of the wearer is captured by a camera, and image coordinates and relative depth information of multiple palm key points are determined based on the hand image. The camera may be a camera disposed in a head mounted display device.

[0041] It should be noted that the way in which the wearer triggers the virtual keyboard opening instruction can be set according to actual conditions, and the embodiment of the present invention does not specifically limit this. For example, the wearer triggers the virtual keyboard opening instruction through voice.

[0042] In some embodiments, the hand image is input into a preset palm key point recognition model to obtain multiple palm key points and image coordinates and relative depth information of the palm key points. The preset palm key point recognition model is a pre-trained neural network model. By recognizing the hand image through the preset palm key point recognition model, the image coordinates and relative depth information of the palm key points can be accurately obtained.

[0043] In some embodiments, a sample data set is obtained, the sample data set includes multiple sample data, each sample data includes a sample hand image, annotated image coordinates and annotated relative depth information; a sample data is selected from the sample data set as target sample data; the target hand image in the target sample data is input into a preset neural network model to obtain predicted image coordinates and predicted relative depth information; based on the annotated image coordinates, annotated relative depth information, predicted image coordinates and predicted relative depth information in the target sample data, it is determined whether the preset neural network model converges; if the preset neural network model does not converge, the model parameters of the preset neural network model are updated, and the step of selecting a sample data from the sample data set as the target sample data is continued until the preset neural network model converges to obtain a preset palm key point recognition model.

[0044] It should be noted that the preset neural network model includes but is not limited to a convolutional neural network model, a recurrent neural network model and a recurrent convolutional neural network model.

[0045] In some embodiments, based on the annotated image coordinates, annotated relative depth information, predicted image coordinates, and predicted relative depth information in the target sample data, a method for determining whether the preset neural network model has converged may be: determining the target loss value of the preset neural network model based on the annotated image coordinates, the annotated relative depth information, the predicted image coordinates, and the predicted relative depth information; if the target loss value is less than or equal to the preset loss value, determining that the preset neural network model has converged; if the target loss value is greater than the preset loss value, determining that the preset neural network model has not converged. Among them, the preset loss value can be set according to actual conditions, and the embodiments of the present invention do not specifically limit this. For example, the preset loss value can be set to 0.02.

[0046] In some embodiments, according to the annotated image coordinates, the annotated relative depth information, the predicted image coordinates and the predicted relative depth information, the method for determining the target loss value of the preset neural network model can be: calculating the similarity between the annotated image coordinates and the predicted image coordinates to obtain a first similarity; calculating the similarity between the annotated relative depth information and the predicted relative depth information to obtain a second similarity; performing a mean operation on the first similarity and the second similarity to obtain a target similarity, and taking the value of unit 1 minus the target similarity as the target loss value. Among them, the method for calculating the similarity can be selected according to the actual situation, and the embodiment of the present invention does not specifically limit this. For example, the method for calculating the similarity can be cosine similarity.

[0047] Step S102: fitting a virtual space plane according to the image coordinates and relative depth information of each key point of the palm, wherein the distance between each point in the virtual space plane and the palm is equal.

[0048] Among them, the distance between each point in the virtual space plane and the palm is equal. It can be understood that when the wearer clicks the virtual space plane with a finger, the distance between each click point and the palm is equal, and the tactile feedback of the wearer clicking different click points is the same. For example, the distance between the virtual space plane and the palm is zero, that is, the distance between each area of ​​the virtual space plane and the palm is zero, and the tactile feedback of the wearer clicking different click points is the same.

[0049] In some embodiments, Figure 2 As shown, step S102 includes sub-steps S1021 and S1022.

[0050] Sub-step S1021, determining the spatial coordinate matrix of the palm key points in the camera coordinate system according to the image coordinates and the relative depth information, wherein the spatial coordinate matrix includes determining the spatial coordinates of each of the palm key points in the camera coordinate system.

[0051] The spatial coordinate matrix includes determining the spatial coordinates of each palm key point in the camera coordinate system.

[0052] In some embodiments, the camera intrinsic parameters of the camera that captures the hand image are obtained, and the spatial coordinate matrix of each palm key point in the camera coordinate system is calculated based on the camera intrinsic parameters, image coordinates and relative depth information. The camera intrinsic parameters include focal length, principal point coordinates and deformity coefficient, etc. By calculating the camera intrinsic parameters, image coordinates and relative depth information, the spatial coordinate matrix of each palm key point can be accurately obtained.

[0053] It should be noted that the method for calculating the spatial coordinate matrix of each palm key point in the camera coordinate system can be selected according to actual conditions, and the embodiment of the present invention does not make any specific limitation to this.

[0054] For example, Figure 3 As shown, Figure 3 The palm key point 11, palm key point 12, palm key point 13, palm key point 14, palm key point 15, palm key point 16 and palm key point 17 are included. The spatial coordinate matrix of each palm key point in the camera coordinate system is calculated, and the spatial coordinate matrix of palm key point 11 is (x1, y1, z1), the spatial coordinate matrix of palm key point 12 is (x2, y2, z2), the spatial coordinate matrix of palm key point 13 is (x3, y3, z3), and the spatial coordinate matrix of palm key point 14 is The spatial coordinate matrix of palm key point 15 is (x4, y4, z4), the spatial coordinate matrix of palm key point 15 is (x5, y5, z5), the spatial coordinate matrix of palm key point 16 is (x6, y6, z6), and the spatial coordinate matrix of palm key point 17 is (x7, y7, z7). The spatial coordinate matrices of palm key point 11, palm key point 12, palm key point 13, palm key point 14, palm key point 15, palm key point 16 and palm key point 17 are superimposed to obtain the spatial coordinates of the palm key points in the camera coordinate system:

[0055] Sub-step S1022: fitting the virtual space plane according to the space coordinate matrix.

[0056] In some embodiments, a first matrix is ​​constructed according to a spatial coordinate matrix and a preset constant, and a second matrix is ​​constructed according to the spatial coordinate matrix; a preset plane parameter calculation formula is obtained, and the first matrix and the second matrix are parameterized according to the preset plane parameter calculation formula to obtain virtual space plane parameters; the parameters of the preset space plane are updated according to the virtual space plane parameters to obtain the virtual space plane. The preset constant can be set according to actual conditions, and the embodiment of the present invention does not specifically limit this. For example, the preset constant can be set to 1.

[0057] In some embodiments, based on the spatial coordinate matrix and preset constants, the first matrix can be constructed by: using the first column of the spatial coordinate matrix as the first column of the first matrix; using the second column of the spatial coordinate matrix as the second column of the first matrix; and filling the third column of the first matrix with preset constants to obtain the first matrix.

[0058] For example, the spatial coordinates of the palm key points in the camera coordinate system are The preset constant is 1, and the first column of the spatial coordinate matrix is ​​used as the first column of the first matrix; the second column of the spatial coordinate matrix is ​​used as the second column of the first matrix; the third column of the first matrix is ​​filled with the preset constant, and the first matrix is ​​obtained as

[0059] In some embodiments, the second matrix may be constructed according to the spatial coordinate matrix by using the third column of the spatial coordinate matrix as the second matrix. For example, the spatial coordinates of the palm key points in the camera coordinate system are The third column of the spatial coordinate matrix is ​​used as the second matrix, and the second matrix is ​​obtained as

[0060] In some embodiments, the plane equation of the virtual space plane is preset to ax+by+d=z, and the plane equation is solved by the least squares method, wherein the least squares method solves the linear equation as follows: Among them, M is the first matrix, Q is the second matrix, is the plane method parameter to be solved. The preset plane parameter calculation formula is: Based on the preset plane parameter calculation formula, the parameters of the first matrix and the second matrix are solved to obtain the virtual space plane parameters a, b, d. The virtual space plane parameters a, b, d are used to update the plane equation ax+by+d=z to obtain the virtual space plane as ax+by-z=d.

[0061] Step S103: determining the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the multiple palm key points.

[0062] The space coordinate information of the area occupied by the virtual keyboard is the coordinate information of the position coordinates of the area occupied by the virtual keyboard in space.

[0063] In some embodiments, Figure 4 As shown, step S103 includes sub-steps S1031 and S1032.

[0064] Sub-step S1031, determining the spatial coordinates of the projection point of each palm key point on the virtual space plane, and obtaining a plurality of key point projection spatial coordinates.

[0065] Among them, the projection point is the projection position point of the palm key point on the virtual space plane, and the projection spatial coordinates of the key point are the projection point of the palm key point on the virtual space plane, and the spatial coordinates of the projection point.

[0066] In some embodiments, a preset projection point parameter formula is obtained, and the preset parameter formula is: Wherein, t is the projection point parameter, x, y, z are the spatial coordinates of the palm key point, a, b and d are the plane parameters of the virtual space plane; the spatial coordinates of the palm key point are substituted into the preset projection point parameter formula for calculation to obtain the projection point parameter t of the palm key point, and the first preset dimension coordinate formula, the second preset dimension coordinate formula and the third preset dimension coordinate formula are obtained, wherein the first preset dimension coordinate formula is x′=x+ta, x′ is the first dimension coordinate value of the spatial coordinate of the projection point of the palm key point in the virtual space plane, x is the first dimension coordinate value of the spatial coordinate of the palm key point in the virtual space plane, and t is the palm key point Projection point parameter, a is the virtual space plane parameter; the second preset dimensional coordinate formula is y′=y+tb, y′ is the second dimensional coordinate value of the spatial coordinates of the projection point of the palm key point in the virtual space plane, y is the second dimensional coordinate value of the spatial coordinates of the palm key point in the virtual space plane, t is the palm key point projection point parameter, b is the virtual space plane parameter; the third preset dimensional coordinate formula is z′=zt, z′ is the third dimensional coordinate value of the spatial coordinates of the projection point of the palm key point in the virtual space plane, z is the third dimensional coordinate value of the spatial coordinates of the palm key point in the virtual space plane, and t is the palm key point projection point parameter.

[0067] It can be understood that by calculating the spatial coordinates of the palm key points based on the preset parameter formula, the projection point parameters can be accurately obtained, and by calculating the spatial coordinates of the palm key points based on the first preset dimension coordinate formula, the second preset dimension coordinate formula and the third preset dimension coordinate formula, the key point projection spatial coordinates can be accurately obtained.

[0068] Exemplarily, the spatial coordinates of the palm key point are (x0, y0, z0). Based on the aforementioned preset parameter formula and the plane parameters of the virtual space plane, the palm key point projection point parameter can be calculated as t. The spatial coordinates of the palm key point and the palm key point projection point parameter are respectively substituted into the first preset dimension coordinate formula, the second preset dimension coordinate formula and the third preset dimension coordinate formula for calculation, and the first dimension coordinate value is x0+ta, the second dimension coordinate value is y0+tb, and the third dimension coordinate value is z0-t. The key point projection spatial coordinates of the palm key point are obtained as (x0+ta, y0+tb, z0-t).

[0069] Sub-step S1032: determining the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the spatial coordinates of each key point projection.

[0070] In some embodiments, a two-dimensional coordinate system is established in the virtual space plane, and the key point coordinates of the projected spatial coordinates of each key point in the two-dimensional coordinate system are determined; the two-dimensional coordinate information of the area occupied by the virtual keyboard is determined based on the coordinates of each key point; and the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane is determined based on the two-dimensional coordinate information. By determining the two-dimensional coordinate information of the area occupied by the virtual keyboard in the virtual space plane and then performing spatial coordinate conversion on the two-dimensional coordinate information, the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane can be accurately obtained, which greatly improves the efficiency and accuracy of the virtual keyboard reality.

[0071] In some embodiments, a point is selected in the virtual space plane as the origin of the newly created two-dimensional coordinate system, and a rectangular coordinate system is established based on the origin, and the key point coordinates of each key point projection space coordinate in the two-dimensional coordinate system are determined.

[0072] For example, Figure 5 As shown, the straight line from the midpoint of palm key point 12 and palm key point 13 to palm key point 14 is used as the x-axis, and the straight line 90 degrees perpendicular to the x-axis is used as the y-axis to determine the unit vector of the two-dimensional coordinate system, and the key point coordinates of the projected space coordinates of each key point in the two-dimensional coordinate system are determined based on the unit vector and the key point projected space coordinates.

[0073] In some embodiments, the method for determining the unit vector of the two-dimensional coordinate system may be: obtaining a first preset formula and a second preset formula, wherein the first preset formula is Among them, X u is the first direction vector of the unit vector, P 12 is the key point projection space coordinate of the palm key point 12, P 13 is the key point projection space coordinate of the palm key point 13, P 14is the key point projection space coordinate of the palm key point 14, 0.5 represents P 12 and P 13 The second preset formula is Y u =X u *n, where Y u is the second direction vector of the unit vector, a and b are the virtual space plane parameters. 11 Establish a rectangular coordinate system for the origin and get Figure 6 The two-dimensional coordinate system shown. Obtain the preset first horizontal coordinate formula, which is p=(PP 11 )X u , p is the horizontal coordinate of the palm key point in the two-dimensional coordinate system, P is the projection space coordinate of the key point, P 11 is the spatial coordinate of the origin in the two-dimensional coordinate system, q u is the first direction vector of the unit vector. The preset first horizontal coordinate formula is q=(PP 11 )Y u , q is the ordinate of the palm key point in the two-dimensional coordinate system, P is the projection space coordinate of the key point, P 11 is the spatial coordinate of the origin in the two-dimensional coordinate system, Y u is the second direction vector of the unit vector. Based on the preset first horizontal coordinate formula and the preset second vertical coordinate formula, the projection space coordinates of each key point are calculated to obtain the key point coordinates of each palm key point in the two-dimensional coordinate system.

[0074] In some embodiments, according to the coordinates of each key point, the method of determining the two-dimensional coordinate information of the area occupied by the virtual keyboard can be: establishing a bounding rectangle in the virtual space plane according to the coordinates of each key point, and taking the area occupied by the bounding rectangle as the area occupied by the virtual keyboard; obtaining the coordinates of each corner of the bounding rectangle to obtain the two-dimensional coordinate information of the area occupied by the virtual keyboard. By selecting the bounding rectangle of the palm key point from the virtual space plane, the area occupied by the virtual keyboard can be accurately determined, and then obtaining the coordinates of each corner of the bounding rectangle to obtain the two-dimensional coordinate information of the area occupied by the virtual keyboard, which greatly improves the accuracy of determining the area occupied by the virtual keyboard.

[0075] For example, Figure 6 As shown, palm key point 12, palm key point 14, palm key point 16 and palm key point 17 are selected as the positioning points of the virtual keyboard, the circumscribed rectangles of palm key point 12, palm key point 14, palm key point 16 and palm key point 17 are determined, the target rectangle is obtained, the coordinates of each corner of the circumscribed rectangle are obtained, and the two-dimensional coordinate information of the area occupied by the virtual keyboard is obtained.

[0076] In some embodiments, the coordinates of each corner of the circumscribed rectangle are obtained to obtain the two-dimensional coordinate information of the area occupied by the virtual keyboard, and the two-dimensional coordinate information is converted into the spatial coordinates in the camera coordinate system to obtain the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane. The occupied area of ​​the virtual keyboard can be accurately located by the coordinates of each corner of the circumscribed rectangle, and the coordinates of each corner of the circumscribed rectangle are converted into the spatial coordinate information, so that the head mounted display device can project and display the virtual keyboard according to the spatial coordinate information.

[0077] Exemplarily, the coordinates of each corner of the circumscribed rectangle are obtained, the upper left corner coordinates (m1, n1), the lower left corner coordinates (m1, n2), the upper right corner coordinates (m2, n1), and the upper right corner coordinates (m2, n2), and the preset space conversion formula is obtained. The preset space conversion formula is R=P 11 +[X u Y u ]N, where R is the spatial coordinate of the virtual keyboard positioning, P 11 is the key point projection space coordinate of palm key point 11, X u is the first direction vector of the unit vector, Y u is the second direction vector of the unit vector, N is the coordinates of the corners of the circumscribed rectangle, and the key point projection space coordinates, the first direction vector, the second direction vector and the coordinates of the corners of the circumscribed rectangle are calculated based on the preset space conversion formula to obtain multiple space coordinates of the area occupied by the virtual keyboard.

[0078] Step S104: displaying the virtual keyboard in the palm of the wearer according to the spatial coordinate information.

[0079] In some embodiments, after obtaining the spatial coordinate information, the virtual keyboard is projected and displayed on the palm of the wearer according to the spatial coordinate information. Based on the spatial coordinate information, the spatial coordinates of the virtual keyboard projection display can be accurately located, and based on the spatial coordinates of each corner of the virtual keyboard, a virtual keyboard that can evenly fit the palm at each position is accurately projected, so that the user has the same tactile feedback during use, thereby improving the accuracy and convenience of the wearer's text input in the virtual keyboard, and greatly improving the wearer's use experience.

[0080] Exemplarily, a hand image of a wearer of a head mounted display device is obtained, and image coordinates and relative depth information of a plurality of palm key points are determined according to the hand image; a virtual space plane is fitted according to the image coordinates and relative depth information of each palm key point, and the distance between each point in the virtual space plane and the palm is equal; spatial coordinate information of an area occupied by a virtual keyboard is determined in the virtual space plane according to the plurality of palm key points; and the spatial coordinate information is displayed in the palm of the wearer to generate a virtual keyboard as shown in FIG. Figure 7The virtual keyboard is shown.

[0081] The virtual keyboard display method provided in the above embodiment obtains the hand image of the wearer of the head mounted display device, and determines the image coordinates and relative depth information of multiple palm key points according to the hand image; according to the image coordinates and relative depth information of each palm key point, fits a virtual space plane, in which each point in the virtual space plane is equidistant from the palm; determines the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to multiple palm key points; and displays the virtual keyboard in the palm of the wearer according to the spatial coordinate information. In the present application, a virtual space plane in which each point on a plane is equidistant from the palm is fitted through the image coordinates and relative depth information of the palm key points, and a virtual keyboard is displayed on the virtual space plane, so that each position of the virtual keyboard in the virtual space plane is also equidistant from the palm, and the user has the same tactile feedback during use, thereby improving the accuracy and convenience of the wearer's text input in the virtual keyboard, and greatly improving the wearer's use experience.

[0082] See also Figure 8 , Figure 8 A schematic block diagram of a virtual keyboard display device provided in an embodiment of the present application.

[0083] like Figure 8 As shown, the virtual keyboard display device 200 includes an acquisition module 210, a determination module 220, a fitting module 230 and a projection display module 240, wherein:

[0084] The acquisition module 210 is used to acquire a hand image of a wearer of the head mounted display device;

[0085] The determination module 220 is used to determine the image coordinates and relative depth information of multiple palm key points according to the hand image;

[0086] The fitting module 230 is used to fit a virtual space plane according to the image coordinates and relative depth information of each key point of the palm, and the distance between each point in the virtual space plane and the palm is equal;

[0087] The determination module 220 is further configured to determine the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the multiple palm key points;

[0088] The projection display module 240 is used to display the virtual keyboard in the palm of the wearer according to the spatial coordinate information.

[0089] In some embodiments, the fitting module 230 is further configured to:

[0090] Determine, according to the image coordinates and the relative depth information, a spatial coordinate matrix of the palm key points in the camera coordinate system, wherein the spatial coordinate matrix includes determining the spatial coordinates of each of the palm key points in the camera coordinate system;

[0091] The virtual space plane is fitted according to the space coordinate matrix.

[0092] In some embodiments, the fitting module 230 is further configured to:

[0093] Constructing a first matrix according to the spatial coordinate matrix and preset constants, and constructing a second matrix according to the spatial coordinate matrix;

[0094] Obtaining a preset plane parameter calculation formula, and solving the first matrix and the second matrix for parameters according to the preset plane parameter calculation formula to obtain virtual space plane parameters;

[0095] The parameters of the preset space plane are updated according to the virtual space plane parameters to obtain the virtual space plane.

[0096] In some embodiments, Fig. 9 As shown, the determination module 220 further includes a first determination submodule 221 and a second determination submodule 222, wherein:

[0097] A first determination submodule 221 is used to determine the spatial coordinates of the projection point of each palm key point on the virtual space plane to obtain a plurality of key point projection spatial coordinates;

[0098] The second determining submodule 222 determines the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the spatial coordinates of each key point projection.

[0099] In some embodiments, the second determining submodule 222 is further configured to:

[0100] Establishing a two-dimensional coordinate system in the virtual space plane, and determining the key point coordinates of each key point projection space coordinate in the two-dimensional coordinate system;

[0101] Determining two-dimensional coordinate information of the area occupied by the virtual keyboard according to the coordinates of each key point;

[0102] The spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane is determined according to the two-dimensional coordinate information.

[0103] In some embodiments, the second determining submodule 222 is further configured to:

[0104] Establishing a circumscribed rectangle in the virtual space plane according to the coordinates of each key point, and using the area occupied by the circumscribed rectangle as the area occupied by the virtual keyboard;

[0105] The coordinates of each corner of the circumscribed rectangle are obtained to obtain the two-dimensional coordinate information of the area occupied by the virtual keyboard.

[0106] In some embodiments, the second determining submodule 222 is further configured to:

[0107] The two-dimensional coordinate information is converted into spatial coordinates in a camera coordinate system to obtain spatial coordinate information of an area occupied by the virtual keyboard in a virtual space plane.

[0108] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned virtual keyboard display device can refer to the corresponding process in the aforementioned virtual keyboard display method embodiment, and will not be repeated here.

[0109] See also Fig.10 , Fig.10 A schematic block diagram of the structure of a head mounted display device provided in an embodiment of the present application.

[0110] like Fig.10 As shown, the head mounted display device 300 includes a processor 302 and a memory 303 connected via a system bus 301, wherein the memory may include a storage medium and an internal memory.

[0111] The storage medium can store a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any virtual keyboard display method.

[0112] The processor 302 is used to provide computing and control capabilities to support the operation of the entire head-mounted display device.

[0113] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any virtual keyboard display method.

[0114] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the head-mounted display device to which the scheme of the present application is applied. The specific head-mounted display device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] It should be understood that the processor 302 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0116] In one embodiment, the processor 302 is used to run a computer program stored in the memory to implement the following steps:

[0117] Acquire a hand image of a wearer of the head mounted display device, and determine image coordinates and relative depth information of a plurality of palm key points according to the hand image;

[0118] Fitting a virtual space plane according to the image coordinates and relative depth information of each key point of the palm, wherein the distance between each point in the virtual space plane and the palm is equal;

[0119] Determining spatial coordinate information of an area occupied by a virtual keyboard in the virtual space plane according to the multiple palm key points;

[0120] The virtual keyboard is displayed in the palm of the wearer according to the spatial coordinate information.

[0121] In one embodiment, when implementing the fitting of the virtual space plane according to the image coordinates and relative depth information of each palm key point, the processor 302 is used to implement:

[0122] Determine, according to the image coordinates and the relative depth information, a spatial coordinate matrix of the palm key points in the camera coordinate system, wherein the spatial coordinate matrix includes determining the spatial coordinates of each of the palm key points in the camera coordinate system;

[0123] The virtual space plane is fitted according to the space coordinate matrix.

[0124] In one embodiment, when implementing the fitting of the virtual space plane according to the space coordinate matrix, the processor 302 is used to implement:

[0125] Constructing a first matrix according to the spatial coordinate matrix and preset constants, and constructing a second matrix according to the spatial coordinate matrix;

[0126] Obtaining a preset plane parameter calculation formula, and solving the first matrix and the second matrix for parameters according to the preset plane parameter calculation formula to obtain virtual space plane parameters;

[0127] The parameters of the preset space plane are updated according to the virtual space plane parameters to obtain the virtual space plane.

[0128] In one embodiment, when the processor 302 determines the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the multiple palm key points, it is used to implement:

[0129] Determine the spatial coordinates of the projection point of each palm key point on the virtual space plane to obtain a plurality of key point projection spatial coordinates;

[0130] The spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane is determined according to the spatial coordinates of each key point projection.

[0131] In one embodiment, when the processor 302 determines the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the projection spatial coordinates of each key point, it is used to implement:

[0132] Establishing a two-dimensional coordinate system in the virtual space plane, and determining the key point coordinates of each key point projection space coordinate in the two-dimensional coordinate system;

[0133] Determining two-dimensional coordinate information of the area occupied by the virtual keyboard according to the coordinates of each key point;

[0134] The spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane is determined according to the two-dimensional coordinate information.

[0135] In one embodiment, when the processor 302 determines the two-dimensional coordinate information of the area occupied by the virtual keyboard according to the coordinates of each key point, it is used to implement:

[0136] Establishing a circumscribed rectangle in the virtual space plane according to the coordinates of each key point, and using the area occupied by the circumscribed rectangle as the area occupied by the virtual keyboard;

[0137] The coordinates of each corner of the circumscribed rectangle are obtained to obtain the two-dimensional coordinate information of the area occupied by the virtual keyboard.

[0138] In one embodiment, when the processor 302 determines the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the two-dimensional coordinate information, it is configured to implement:

[0139] The two-dimensional coordinate information is converted into spatial coordinates in a camera coordinate system to obtain spatial coordinate information of an area occupied by the virtual keyboard in a virtual space plane.

[0140] It should be noted that technicians in the relevant field can clearly understand that for the convenience and brevity of description, the specific working process of the head-mounted display device described above can refer to the corresponding process in the aforementioned virtual keyboard display method embodiment, and will not be repeated here.

[0141] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the virtual keyboard display method of the present application.

[0142] The computer-readable storage medium may be an internal storage unit of the head-mounted display device described in the foregoing embodiment, such as a hard disk or memory of the head-mounted display device. The computer-readable storage medium may be non-volatile or volatile. The computer-readable storage medium may also be an external storage device of the head-mounted display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the head-mounted display device.

[0143] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0144] It should also be understood that the term "and / or" used in the present specification refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0145] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application.

Claims

1. A virtual keyboard display method, characterized in that: include: Acquire a hand image of a wearer of the head mounted display device, and determine image coordinates and relative depth information of a plurality of palm key points according to the hand image; Fitting a virtual space plane according to the image coordinates and relative depth information of each key point of the palm, wherein the distance between each point in the virtual space plane and the palm is equal; Determining spatial coordinate information of an area occupied by a virtual keyboard in the virtual space plane according to the multiple palm key points; The virtual keyboard is displayed in the palm of the wearer according to the spatial coordinate information.

2. The virtual keyboard display method according to claim 1, characterized in that: The step of fitting a virtual space plane according to the image coordinates and relative depth information of each palm key point comprises: Determine, according to the image coordinates and the relative depth information, a spatial coordinate matrix of the palm key points in the camera coordinate system, wherein the spatial coordinate matrix includes determining the spatial coordinates of each of the palm key points in the camera coordinate system; The virtual space plane is fitted according to the space coordinate matrix.

3. The virtual keyboard display method according to claim 2, characterized in that: The step of fitting the virtual space plane according to the space coordinate matrix comprises: Constructing a first matrix according to the spatial coordinate matrix and preset constants, and constructing a second matrix according to the spatial coordinate matrix; Obtaining a preset plane parameter calculation formula, and solving the first matrix and the second matrix for parameters according to the preset plane parameter calculation formula to obtain virtual space plane parameters; The parameters of the preset space plane are updated according to the virtual space plane parameters to obtain the virtual space plane.

4. The virtual keyboard display method according to claim 1, characterized in that: The step of determining the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the multiple palm key points includes: Determine the spatial coordinates of the projection point of each palm key point on the virtual space plane to obtain a plurality of key point projection spatial coordinates; According to the projected spatial coordinates of each key point, the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane is determined.

5. The virtual keyboard display method according to claim 4, characterized in that: Determining the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the spatial coordinates of each key point projection includes: Establishing a two-dimensional coordinate system in the virtual space plane, and determining the key point coordinates of each key point projection space coordinate in the two-dimensional coordinate system; Determining two-dimensional coordinate information of the area occupied by the virtual keyboard according to the coordinates of each key point; The spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane is determined according to the two-dimensional coordinate information.

6. The virtual keyboard display method according to claim 5, characterized in that: Determining the two-dimensional coordinate information of the area occupied by the virtual keyboard according to the coordinates of each key point includes: Establishing a circumscribed rectangle in the virtual space plane according to the coordinates of each key point, and using the area occupied by the circumscribed rectangle as the area occupied by the virtual keyboard; The coordinates of each corner of the circumscribed rectangle are obtained to obtain the two-dimensional coordinate information of the area occupied by the virtual keyboard.

7. The virtual keyboard display method according to claim 5, characterized in that: The step of determining the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the two-dimensional coordinate information includes: The two-dimensional coordinate information is converted into spatial coordinates in a camera coordinate system to obtain spatial coordinate information of an area occupied by the virtual keyboard in a virtual space plane.

8. A virtual keyboard display device, characterized in that: The virtual keyboard display device includes an acquisition module, a determination module, a fitting module and a projection display module, wherein: The acquisition module is used to acquire a hand image of a wearer of the head mounted display device; The determination module is used to determine the image coordinates and relative depth information of multiple palm key points according to the hand image; The fitting module is used to fit a virtual space plane according to the image coordinates and relative depth information of each key point of the palm, wherein the distance between each point in the virtual space plane and the palm is equal; The determination module is further used to determine the spatial coordinate information of the area occupied by the virtual keyboard in the virtual space plane according to the multiple palm key points; The projection display module is used to display the virtual keyboard in the palm of the wearer according to the spatial coordinate information.

9. A head mounted display device, characterized in that: The head mounted display device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the virtual keyboard display method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the virtual keyboard display method according to any one of claims 1 to 7 are implemented.

Images