Display method, device, equipment and storage medium of near-eye display device

Abstract

The application relates to the technical field of gesture interaction, and provides a display method and device of a near-eye display equipment, equipment and a storage medium, the method comprises the following steps: obtaining a hand image obtained by the near-eye display equipment shooting a user's hand; determining position information of a plurality of preset key points of the user's hand on the hand image; determining a preset plane corresponding to the plurality of preset key points according to the plurality of position information; when it is determined that the user's hand meets a preset condition according to the preset plane, displaying a virtual interactive object on the preset plane, so that the display accuracy of the virtual interactive object by the near-eye display equipment is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of gesture interaction, and particularly relates to a display method and device of a near-eye display apparatus, an apparatus and a storage medium. BACKGROUND

[0002] In the related art, a near-eye display apparatus can display virtual interaction objects, such as a virtual keyboard, a virtual special effect, and the like, on a user's hand. However, in a process in which the user can interact with the near-eye display apparatus through the hand, the user often needs to place the hand in a shooting area of the near-eye display apparatus. In a case in which the user does not need the near-eye display apparatus to display the virtual interaction object, if the near-eye display apparatus erroneously determines that the virtual interaction object needs to be displayed on the user's hand according to the user's hand in the shooting area, the display accuracy of the virtual interaction object by the near-eye display apparatus is poor, thereby affecting the user's interaction experience with the near-eye display apparatus. SUMMARY

[0003] The main purpose of the present application is to provide a display method and device of a near-eye display apparatus, an apparatus and a storage medium, aiming at solving the technical problem of poor display accuracy of a virtual interaction object by a near-eye display apparatus due to the near-eye display apparatus erroneously determining that the virtual interaction object needs to be displayed on a user's hand according to the user's hand in a shooting area.

[0004] In a first aspect, the present application provides a display method of a near-eye display apparatus, the display method comprising:

[0005] obtaining a hand image obtained by the near-eye display apparatus shooting a user's hand;

[0006] determining position information of a plurality of preset key points of the user's hand on the hand image;

[0007] determining a preset plane corresponding to the plurality of preset key points according to the plurality of position information;

[0008] when it is determined according to the preset plane that the user's hand meets a preset condition, displaying a virtual interaction object on the preset plane.

[0009] In a second aspect, the present application provides a display device of a near-eye display apparatus, the display device comprising:

[0010] an image obtaining module, configured to obtain a hand image obtained by the near-eye display apparatus shooting a user's hand;

[0011] a position determining module, configured to determine position information of a plurality of preset key points of the user's hand on the hand image;

[0012] A plane determination module is configured to determine a preset plane corresponding to the plurality of preset key points according to the plurality of position information.

[0013] A display module is configured to display a virtual interactive object on the preset plane when it is determined that the user's hand meets a preset condition according to the preset plane.

[0014] In a third aspect, the present application provides a near-eye display device, comprising a memory and a processor.

[0015] The memory is configured to store a computer program.

[0016] The processor is configured to execute the computer program and implement the steps of the display method of the near-eye display device as described above when executing the computer program.

[0017] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the display method of the near-eye display device as described above are implemented.

[0018] The present application provides a display method, device, equipment and storage medium of a near-eye display device. The display method comprises: obtaining a hand image of a user's hand captured by a near-eye display device; determining position information of a plurality of preset key points of the user's hand on the hand image; determining a preset plane corresponding to the plurality of preset key points according to the plurality of position information; and displaying a virtual interactive object on the preset plane when it is determined that the user's hand meets a preset condition according to the preset plane.

[0019] In the case that the user places the hand in the shooting area of the near-eye display device, the near-eye display device can capture the hand image of the user's hand. In the case that the hand image is obtained, the near-eye display device can determine the preset plane according to the position information of the plurality of preset key points on the hand image. Correspondingly, the near-eye display device can determine whether the user's hand meets the preset condition according to the preset plane, so as to determine that the user has a display requirement for the virtual interactive object in the case that the user's hand meets the preset condition. Then, the near-eye display device can display the virtual interactive object in response to the display requirement of the user for the virtual interactive object, thereby facilitating to improve the display accuracy of the virtual interactive object by the near-eye display device. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is a flowchart of a display method of a near-eye display device according to an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of a hand image according to an embodiment of the present application;

[0023] Figure 3 is a display diagram of a virtual interactive object according to an embodiment of the present application;

[0024] Figure 4 is a display diagram of a virtual interactive object according to another embodiment of the present application;

[0025] Figure 5 is a schematic block diagram of a display device of a near-eye display device according to an embodiment of the present application;

[0026] Figure 6 is a schematic block diagram of a near-eye display device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0028] The flowchart shown in the drawings is only an example and does not necessarily include all the contents and operations / steps, nor does it necessarily execute in the order described. For example, some operations / steps can be further divided, combined or partially merged, so the actual execution order may be changed according to the actual situation.

[0029] Embodiments of the present application provide a display method, device, and equipment of a near-eye display device and a storage medium. The display method of the near-eye display device can be applied to the near-eye display device. The near-eye display device can include a virtual reality (VR) glasses, an augmented reality (AR) glasses, a mixed reality (MR) glasses, a VR helmet, an AR helmet, an MR helmet, and the like, which are not limited herein. The display method of the near-eye display device can also be applied to a server, which can be a separate server or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms.

[0030] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0031] Please refer to Figure 1 , Figure 1 is a flowchart of a display method of a near-eye display device provided by embodiments of the present application. It should be noted that the hand recognition method provided by embodiments of the present application can be used in a near-eye display device, or can be used in a server, which is not limited herein.

[0032] As Figure 1 shown, the display method of the near-eye display device includes steps S101 to S104.

[0033] S101, obtaining a hand image of a user hand captured by the near-eye display device.

[0034] For example, the near-eye display device can capture the user hand of the user wearing the near-eye display device to obtain a corresponding hand image. For example, the near-eye display device can be provided with a shooting module. When it is detected that the user hand is within the shooting range corresponding to the near-eye display device, the near-eye display device can capture the user hand through the shooting module to obtain the hand image.

[0035] In some embodiments, the near-eye display device can perform image preprocessing on the hand image to remove the background image region as much as possible when the hand image is obtained. For example, the near-eye display device can perform cropping processing on the hand image to reduce the background image region and retain the hand image region, which is not limited herein.

[0036] In this way, in a case where the near-eye display device acquires the hand image, the hand image can be used by the near-eye display device to determine the display requirement of the user for the virtual interactive object, so as to subsequently display the virtual interactive object in a case where the user has the display requirement for the virtual interactive object.

[0037] S102, determine position information of a plurality of preset key points of a user's hand on the hand image.

[0038] For example, in a case where the hand image is acquired, the near-eye display device can perform key point detection on the hand image. For example, the near-eye display device can determine the position information of a plurality of preset key points of the user's hand on the hand image according to a preset key point detection algorithm or a deep learning model. The number of preset key points can include 5, 6, 7, etc., which is not limited herein.

[0039] In some embodiments, when the near-eye display device displays the virtual interactive object in response to the display requirement of the user for the virtual interactive object, the virtual interactive object can be displayed in the palm region of the user's hand. Based on this, in a case where the hand image is acquired, the near-eye display device can determine the display requirement of the user for the virtual interactive object according to the palm image corresponding to the palm region in the hand image. As shown in Figure 2 The near-eye display device can determine the palm image corresponding to the palm region from the hand image. Accordingly, the preset key points can be determined from the key points included in the palm region. For example, the near-eye display device can perform key point detection on the palm image corresponding to the palm region in the hand image to obtain the position information of a plurality of preset key points on the hand image.

[0040] In this way, the near-eye display device can determine the position information of a plurality of preset key points on the hand image according to the hand image, which is beneficial to improve the convenience of determining the position information of the preset key points on the hand image. In a case where the near-eye display device determines the position information of a plurality of preset key points, the position information of the plurality of preset key points can be used by the near-eye display device to determine the display requirement of the user for the virtual interactive object, so as to subsequently display the virtual interactive object in a case where the user has the display requirement for the virtual interactive object.

[0041] S103, determine a preset plane corresponding to the plurality of preset key points according to the plurality of position information.

[0042] In a case where the near-eye display device determines the position information of a plurality of preset key points, the near-eye display device can fit a preset plane corresponding to the plurality of preset key points according to the plurality of position information. The preset plane can be as close as possible to the plurality of preset key points, for example, the preset plane can satisfy that the sum of squares of the perpendicular distances of the plurality of preset key points to the preset plane is minimum.

[0043] For example, the preset plane equation can be represented as ax+by+d=z. a, b, and d are used to indicate coefficients of the plane equation, and a, b, and d are constants. The coefficients of the plane equation can include different term coefficients of the plane equation, which can be different or the same, without limitation.

[0044] For example, the position information of the plurality of preset key points can include 2.5D coordinates, and the position information can include horizontal coordinates (x coordinates), vertical coordinates (y coordinates), and depth coordinates (z coordinates).

[0045] For example, the position information of the plurality of preset key points can include 2.5D coordinates, and the position information can include horizontal coordinates (x coordinates), vertical coordinates (y coordinates), and depth coordinates (z coordinates).

[0046] The preset plane equation is a general representation of different planes. In the case of different coefficients of different plane equations, the planes indicated by different plane equations can be different. Based on this, the near-eye display device can solve the coefficients of the plane equation according to the plurality of position information to determine the preset plane corresponding to the plurality of preset key points.

[0047] In some embodiments, the preset position matrix corresponding to the plurality of position information is determined according to the horizontal coordinates of the plurality of position information, the vertical coordinates of the plurality of position information, and a preset constant; the preset position vector corresponding to the plurality of position information is determined according to the depth coordinates of the plurality of position information; the association between the preset position matrix, the preset position vector, and the coefficients of the plane equation is determined according to the preset position matrix, the preset position vector, and the plane equation; the coefficients of the plane equation are determined according to the least squares method and the plurality of position information and the association; and the preset plane is determined according to the plane equation and the coefficients of the plane equation.

[0048] For example, when the number of the plurality of preset key points is 7, the horizontal coordinates of the plurality of position information can include coordinates x1, x2, …, and x7, the vertical coordinates of the plurality of position information can include coordinates y1, y2, …, and y7, and the depth coordinates of the plurality of position information can include coordinates z1, z2, …, and z7. The preset constant can include 1. The preset position matrix determined by the near-eye display device according to the horizontal coordinates of the plurality of position information, the vertical coordinates of the plurality of position information, and the preset constant can be represented as matrix P:

[0049]

[0050] The preset position vector determined by the near-eye display device according to the depth coordinates of the plurality of position information can be represented as vector

[0051]

[0052] Near-eye display devices can be based on matrix P and vector Determine matrix P and vector The relationship between the coefficients of the plane equation and the equation can be expressed as:

[0053]

[0054] Where a, b, and d are used to indicate the coefficients of the plane equation.

[0055] In determining matrix P and vector Vectors corresponding to the coefficients of the plane equation In the case of the correlation between them, near-eye display devices can use the least squares method to analyze the vectors. Solve the equations to determine the coefficients of the plane equations.

[0056] For example, near-eye display devices can determine the results based on the least squares method. Based on this, near-eye display devices can combine matrix P and vector The coefficients of the plane equation, namely the specific constant values ​​corresponding to a, b, and d, are determined. Correspondingly, the near-eye display device can substitute the specific constant values ​​corresponding to a, b, and d into the plane equation to obtain the plane equation corresponding to the preset plane, thereby determining the preset plane. Among these, matrix P and vector... The corresponding row number is determined based on the number of preset key points. When the number of preset key points changes, the matrix P and the vector... The corresponding row number will also change.

[0057] Based on the plane equation and its coefficients, the plane equation corresponding to the preset plane is obtained. In other words, given a predetermined preset plane, the preset plane minimizes the sum of the squares of the vertical distances between each of the multiple preset keypoints and the preset plane. Accordingly, the multiple preset keypoints can all lie within the preset plane, partially within the preset plane, or all outside the preset plane; no restrictions are imposed here.

[0058] In this way, the near-eye display device can determine the preset plane corresponding to multiple preset key points based on multiple location information, which helps to improve the convenience of determining the preset plane. When the near-eye display device determines the preset plane corresponding to multiple preset key points, the preset plane can be used by the near-eye display device to judge the user's display needs for virtual interactive objects, so as to display the virtual interactive objects when the user has display needs for them.

[0059] S104. When it is determined that the user's hand meets the preset condition according to the preset plane, display the virtual interactive object on the preset plane.

[0060] For example, when the user wears the near-eye display device, the shooting direction of the near-eye display device can be consistent with the line-of-sight direction of the user. Therefore, the near-eye display device can set the preset condition according to the direction of the preset plane and the direction of the reference plane to determine whether the user has a display requirement for the virtual interactive object. The direction of the reference plane is perpendicular to the shooting direction of the near-eye display device.

[0061] In some embodiments, the near-eye display device can determine the included angle between the direction of the preset plane and the direction of the reference plane, and determine whether the user's hand meets the preset condition by judging whether the included angle between the direction of the preset plane and the direction of the reference plane is within a preset angle range. For example, when the included angle between the direction of the preset plane and the direction of the reference plane is within the preset angle range, the near-eye display device can determine that the user's hand meets the preset condition. For example, when the angle corresponding to the direction of the reference plane is 0 degrees, the preset angle range can include 0-15 degrees, 0-20 degrees, 0-30 degrees, etc. The preset angle range can be pre-set or set by the user, and is not limited herein.

[0062] For example, when the included angle between the direction of the preset plane and the direction of the reference plane is within the preset angle range, the near-eye display device can determine that the deviation angle of the direction of the preset plane relative to the direction of the reference plane is less than or equal to a preset deviation angle threshold. Based on this, the near-eye display device can determine that the user's hand is directly opposite the shooting direction of the near-eye display device, or that the user's hand is approximately directly opposite the shooting direction of the near-eye display device. For example, when the included angle between the direction of the preset plane and the direction of the reference plane is within the preset angle range, the near-eye display device can determine that the deviation angle of the direction of the preset plane relative to the direction of the reference plane is less than or equal to a preset deviation angle threshold. Figure 3As shown, in a case where the deviated angle of the preset plane relative to the direction of the reference plane is less than the preset deviated angle threshold, if the virtual interactive object, such as a virtual keyboard, a virtual special effect, or the like, is displayed on the preset plane, when the user interacts with different regions of the virtual interactive object, visual occlusion is less likely to occur between the different regions. For example, when the user interacts with the virtual interactive object displayed on the preset plane by using multiple fingers of another hand, if the user places different fingers on different interactive sub-objects of the virtual interactive object, the near-eye display device can continue to capture the user's hand to obtain a corresponding interactive hand image. Based on the fact that the included angle between the direction of the preset plane and the direction of the reference plane is within the preset angle range, the user's hand is directly opposite the capturing direction of the near-eye display device, or the user's hand is approximately directly opposite the capturing direction of the near-eye display device, and the position difference between different fingers of another hand in the interactive hand image in the capturing direction is large, the position of the interactive sub-object corresponding to each finger can be obtained based on the interactive hand image to determine the user's interaction demand. Taking the virtual interactive object as a nine-grid keyboard corresponding to a nine-grid input method, and the interactive sub-objects including virtual keys in the nine-grid keyboard, and the user placing the index finger, middle finger, and ring finger of another hand on three adjacent virtual keys of the nine-grid keyboard as an example. Since the preset plane is approximately opposite the capturing direction of the near-eye display device, even if the three adjacent virtual keys are in the same row or the same column of the nine-grid keyboard, the position difference between the index finger, middle finger, and ring finger in the interactive hand image captured by the near-eye display device is large. At this time, the user's interaction with the virtual interactive object is more convenient. Based on this, the near-eye display device can determine that the user's hand meets the preset condition based on the preset plane. Accordingly, the near-eye display device can determine that the user has a display demand for the virtual interactive object, and thus display the virtual interactive object on the preset plane.

[0063] For example, in a case where the included angle between the direction of the preset plane and the direction of the reference plane is not within the preset angle range, the near-eye display device can determine that the deviated angle of the preset plane relative to the direction of the reference plane is greater than the preset deviated angle threshold. Based on this, the near-eye display device can determine that the user's hand is approximately opposite the capturing direction of the near-eye display device. For example, Figure 4As shown, in a case where the deviation angle of the direction of the preset plane relative to the direction of the reference plane is greater than the preset deviation angle threshold, if the virtual interactive object, such as a virtual keyboard, a virtual special effect, or the like, is displayed on the preset plane, when the user interacts with different regions of the virtual interactive object, visual occlusion between different regions is prone to occur. For example, when the user interacts with the virtual interactive object displayed on the preset plane by using multiple fingers of another hand, if the user respectively places different fingers on different interactive sub-objects of the virtual interactive object, the near-eye display device can continue to capture the user's hand to obtain a corresponding interactive hand image. Based on the fact that the included angle between the direction of the preset plane and the direction of the reference plane is not within the preset angle range and the user's hand is approximately in the shooting direction of the near-eye display device, the position difference between different fingers of another hand in the interactive hand image in the shooting direction is small, and thus the position of the interactive sub-object corresponding to each finger cannot be obtained based on the interactive hand image, and the user's interaction demand cannot be determined. Taking the virtual interactive object as a nine-grid keyboard corresponding to a nine-grid input method, and the interactive sub-objects as virtual keys in the nine-grid keyboard, and taking the case where the user respectively places the index finger, the middle finger, and the ring finger of another hand on three adjacent virtual keys of the nine-grid keyboard as an example. Since the preset plane can be approximately in the shooting direction of the near-eye display device, in a case where the three adjacent virtual keys are in the same row or the same column of the nine-grid keyboard, the position difference between the index finger, the middle finger, and the ring finger in the interactive hand image captured by the near-eye display device is small. For example, the index finger, the middle finger, and the ring finger are prone to overlap and occlude each other, thereby causing the appearance of a blind area of the near-eye display device. At this time, the user's interaction with the virtual interactive object is not convenient. Based on this, the near-eye display device can determine that the user's hand does not meet the preset condition. Accordingly, the near-eye display device can determine that the user does not have a display demand for the virtual interactive object, and thus it is determined that the virtual interactive object does not need to be displayed on the preset plane.

[0064] For example, since the direction indicated by the normal vector of the preset plane is perpendicular to the direction of the preset plane, the near-eye display device can determine that the included angle between the normal vector and the shooting direction of the near-eye display device and the included angle between the direction of the preset plane and the shooting direction of the near-eye display device belong to a reciprocal relationship. Accordingly, the near-eye display device can also determine that the included angle between the normal vector and the shooting direction of the near-eye display device and the included angle between the normal vector and the direction of the reference plane belong to a reciprocal relationship. Based on this, the near-eye display device can set the preset condition based on the direction indicated by the normal vector of the preset plane in combination with the direction of the reference plane, to determine whether the user has a display demand for the virtual interactive object.

[0065] In some implementations, the normal vector of a preset plane is obtained; the angle between the normal vector and a preset reference vector is determined; when the angle is less than or equal to a preset angle threshold, the user's hand is determined to meet the preset conditions.

[0066] The near-eye display device can obtain the normal vector of a preset plane. The near-eye display device can determine a preset reference vector based on the direction of a reference plane, wherein the direction of the reference plane is perpendicular to the direction of the near-eye display device. Having obtained the normal vector of the preset plane and the preset reference vector, the near-eye display device can determine the angle between the normal vector and the preset reference vector.

[0067] For example, the normal vector of the preset plane is determined based on the coefficients of the plane equation corresponding to the preset plane.

[0068] Taking the plane equation corresponding to the preset plane as ax + by + d = z as an example, the coefficients of the plane equation include a, b, and d. Therefore, the normal vector of the preset plane determined by the near-eye display device based on the coefficients of the plane equation corresponding to the preset plane can be expressed as a vector.

[0069]

[0070] The normal vector of the preset plane can be used by near-eye display devices in conjunction with the preset reference vector to determine whether the user's hand meets the preset conditions.

[0071] For example, by substituting the normal vector and the preset reference vector into the preset angle formula, the angle between the normal vector and the preset reference vector can be obtained.

[0072] The preset reference vector can be represented as a vector.

[0073]

[0074] The angle between the normal vector and the preset reference vector can be expressed as angle θ. The preset angle formula can be expressed as:

[0075]

[0076] in, Used for indicator vector Dot product of vectors Used for indicator vector The modulus, Used for indicator vector The modulus.

[0077] vector and vectors Substituting into the preset angle formula, we can determine:

[0078]

[0079] Correspondingly, in a case where the cosine value cos θ of the included angle θ is determined, the near-eye display device can determine the size of the included angle θ through the correlation between the cosine value and the angle. The near-eye display device can also determine the corresponding included angle θ according to the cosine value cos θ through an inverse cosine function. This is not limited here.

[0080] In another exemplary embodiment, in a case where the normal vector of the preset plane is acquired, the near-eye display device can also determine the preset reference vector according to the shooting direction of the near-eye display device.

[0081] Taking a plane equation ax + by + d = z corresponding to the preset plane as an example. In order to maintain the direction consistency of the normal vector of the preset plane and the shooting direction of the near-eye display device, the normal vector of the preset plane can be expressed as vector

[0082]

[0083] The preset reference vector determined according to the shooting direction of the near-eye display device can be expressed as vector

[0084]

[0085] The preset included angle formula can be expressed as:

[0086]

[0087] wherein, is used to indicate the vector dot product vector is used to indicate the vector the modulus of vector is used to indicate the modulus of vector .

[0088] Substituting the vector and the vector into the preset included angle formula, it can be determined that:

[0089]

[0090] Correspondingly, in a case where the cosine value cos θ of the included angle θ is determined, the near-eye display device can determine the size of the included angle θ through the correlation between the cosine value and the angle. The near-eye display device can also determine the corresponding included angle θ according to the cosine value cos θ through an inverse cosine function. This is not limited here.

[0091] When the included angle between the normal vector and the preset reference vector is less than or equal to a preset angle threshold, the near-eye display device can determine that the deviation angle of the preset plane relative to the direction of the reference plane is less than or equal to a preset deviation angle threshold. Accordingly, the near-eye display device can determine that the user's hand is facing the shooting direction of the near-eye display device or is approximately facing the shooting direction of the near-eye display device, thereby determining that the user's hand meets the preset condition. Taking an angle of 0 degrees indicated by the preset reference vector as an example, the preset angle threshold includes, for example, 5 degrees, 10 degrees, 15 degrees, 20 degrees, 25 degrees, 30 degrees, and the like. The preset angle threshold can be pre-set or set by the user, which is not limited herein.

[0092] In some embodiments, when the included angle is greater than the preset angle threshold, it is determined that the user's hand does not meet the preset condition.

[0093] When the included angle between the normal vector and the preset reference vector is greater than the preset angle threshold, the near-eye display device can determine that the deviation angle of the preset plane relative to the direction of the reference plane is greater than the preset deviation angle threshold. Accordingly, the near-eye display device can determine that the user's hand is approximately facing the shooting direction of the near-eye display device, thereby determining that the user's hand does not meet the preset condition.

[0094] In this way, the near-eye display device can determine the normal vector and determine the included angle between the normal vector and the preset reference vector, and then evaluate whether the user's hand meets the preset condition according to whether the included angle between the normal vector and the preset reference vector is less than or equal to the preset angle threshold, which is equivalent to evaluating whether the user has a display requirement for the virtual interactive object, thereby facilitating the determination convenience and accuracy of the user's display requirement for the virtual interactive object. Based on the improvement of the determination convenience and accuracy of the user's display requirement for the virtual interactive object, the near-eye display device can display the virtual interactive object when it is determined that the user's hand meets the preset condition, and not display the virtual interactive object when it is determined that the user's hand does not meet the preset condition. Based on this, the near-eye display device can adapt to the user's display requirement for the virtual interactive object when displaying the virtual interactive object, thereby facilitating the display accuracy of the virtual interactive object by the near-eye display device.

[0095] The display method of the near-eye display device provided in the above embodiments includes: obtaining a hand image of a user's hand captured by a near-eye display device; determining position information of a plurality of preset key points of the user's hand on the hand image; determining a preset plane corresponding to the plurality of preset key points according to the plurality of position information; and displaying a virtual interactive object on the preset plane when it is determined that the user's hand meets a preset condition according to the preset plane.

[0096] In a case where the user places the hand in a shooting area of the near-eye display device, the near-eye display device can shoot the hand of the user to obtain a hand image. In a case where the hand image is obtained, the near-eye display device can determine a preset plane according to position information of a plurality of preset key points in the hand image. Accordingly, the near-eye display device can determine whether the hand of the user meets a preset condition according to the preset plane, so as to determine that the user has a display requirement for the virtual interactive object in a case where the hand of the user meets the preset condition. Then, the near-eye display device can display the virtual interactive object in response to the display requirement of the user for the virtual interactive object, thereby facilitating to improve display accuracy of the virtual interactive object by the near-eye display device.

[0097] Referring to Figure 5 , Figure 5 is a schematic block diagram of a display device of a near-eye display device provided by an embodiment of the present application. The display device of the near-eye display device can be configured in the near-eye display device or a server, and is used to execute the display method of the near-eye display device described above. The near-eye display device can include a VR glasses, an AR glasses, an MR glasses, a VR helmet, an AR helmet, an MR helmet, etc., without limitation. The server can be a separate server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks, and basic cloud computing services such as big data and artificial intelligence platforms.

[0098] As shown in Figure 5 , the display device of the near-eye display device includes an image acquisition module 110, a position determination module 120, a plane determination module 130, and a display module 140.

[0099] The image acquisition module 110 is configured to acquire a hand image of a hand of a user shot by a near-eye display device.

[0100] The position determination module 120 is configured to determine position information of a plurality of preset key points of the hand of the user on the hand image.

[0101] The plane determination module 130 is configured to determine a preset plane corresponding to the plurality of preset key points according to the plurality of position information.

[0102] The display module 140 is configured to display a virtual interactive object on the preset plane when it is determined that the hand of the user meets a preset condition according to the preset plane.

[0103] Exemplarily, the plane determination module 130 includes a first plane determination sub-module.

[0104] The first plane determining sub-module is configured to determine a preset plane corresponding to the plurality of preset key points based on a preset plane equation and the plurality of position information.

[0105] The first plane determining sub-module includes a matrix determining sub-module, a vector determining sub-module, a relationship determining sub-module, a coefficient determining sub-module, and a second plane determining sub-module.

[0106] The matrix determining sub-module is configured to determine a preset position matrix corresponding to the plurality of position information based on horizontal coordinates of the plurality of position information, vertical coordinates of the plurality of position information, and a preset constant.

[0107] The vector determining sub-module is configured to determine a preset position vector corresponding to the plurality of position information based on depth coordinates of the plurality of position information.

[0108] The relationship determining sub-module is configured to determine a relationship between the preset position matrix, the preset position vector, and coefficients of the plane equation based on the preset position matrix, the preset position vector, and the plane equation.

[0109] The coefficient determining sub-module is configured to determine the coefficients of the plane equation based on a least square method and the plurality of position information and the relationship.

[0110] The second plane determining sub-module is configured to determine the preset plane based on the plane equation and the coefficients of the plane equation.

[0111] The display module 140 includes a normal vector obtaining sub-module, an included angle determining sub-module, and a first determining sub-module.

[0112] The normal vector obtaining sub-module is configured to obtain a normal vector of the preset plane.

[0113] The included angle determining sub-module is configured to determine an included angle between the normal vector and a preset reference vector.

[0114] The first determining sub-module is configured to determine that the user's hand meets a preset condition when the included angle is less than or equal to a preset angle threshold.

[0115] The normal vector obtaining sub-module includes a normal vector determining sub-module.

[0116] The normal vector determining sub-module is configured to determine the normal vector of the preset plane based on coefficients of a plane equation corresponding to the preset plane.

[0117] The included angle determining sub-module includes an included angle calculating sub-module.

[0118] The included angle calculation submodule is configured to substitute the normal vector and the preset reference vector into a preset included angle formula to obtain an included angle between the normal vector and the preset reference vector.

[0119] The display device further includes a second judgment submodule.

[0120] The second judgment submodule is configured to determine that the user's hand does not meet a preset condition when the included angle is greater than a preset angle threshold.

[0121] It should be noted that, for the convenience and brevity of description, the specific working processes of the above-described device and each module and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0122] The method of the present application can be used in a variety of general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0123] The above-described method and device can be implemented as a computer program in the form of a computer program, which can run on a near-eye display device or a server to control the near-eye display device. The near-eye display device can include VR glasses, AR glasses, MR glasses, VR headgear, AR headgear, MR headgear, and the like, without limitation. The server can be a separate server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks, and basic cloud computing services such as big data and artificial intelligence platforms.

[0124] Please refer to Figure 6 , Figure 6 is a structural schematic block diagram of a near-eye display device provided by an embodiment of the present application.

[0125] As Figure 6As shown, the near-eye display device includes a memory and a processor. Wherein the memory and the processor can be connected through a system bus, the memory can include a storage medium and an internal memory.

[0126] The storage medium can store an operating system and a computer program. The computer program is executed to enable the processor to perform any one of the display methods of the near-eye display device.

[0127] The processor is used to provide computing and control capabilities to support the operation of the entire near-eye display device.

[0128] The internal memory provides an environment for the execution of the computer program in the storage medium, which is executed by the processor to enable the processor to perform any one of the display methods of the near-eye display device.

[0129] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0130] It should be understood that the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Wherein the general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0131] Wherein in one embodiment, the processor is used to execute a computer program and implement the following steps when executing the computer program:

[0132] Obtaining a hand image of a user's hand captured by the near-eye display device;

[0133] Determining position information of a plurality of preset key points of the user's hand on the hand image;

[0134] According to a plurality of position information, determining a plurality of preset planes corresponding to a plurality of preset key points;

[0135] When it is determined that the user's hand meets the preset condition according to the preset plane, a virtual interactive object is displayed on the preset plane.

[0136] It should be noted that, for the convenience and brevity of description, the specific working process of the display of the near-eye display device is described above, and the corresponding process in the display method of the near-eye display device can be referred to, which will not be described here.

[0137] The embodiments of the present application further provide a computer readable storage medium, and the computer readable storage medium stores a computer program. The method implemented by the computer program executed by the processor can refer to each embodiment of the display method of the near-eye display device.

[0138] The computer readable storage medium can be an internal storage unit of the near-eye display device, such as a hard disk or a memory of the near-eye display device. The computer readable storage medium can also be an external storage device of the near-eye display device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0139] It should be understood that the terms used herein in the specification and the appended claims are merely for the purpose of describing particular embodiments and do not intend to limit the present application. As used in the specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0140] It should also be understood that the term "and / or" used in the specification and the appended claims means one or more of the associated listed items as well as all possible combinations of the items and includes the combinations. It should be noted that the terms "comprise", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or systems including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or systems. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or system including the element.

[0141] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A display method of a near-eye display device, characterized by, The display method comprises: obtaining a hand image of a user's hand captured by a near-eye display device; determining position information of a plurality of preset key points of the user's hand on the hand image; determining a preset plane corresponding to the plurality of preset key points according to the plurality of position information; when it is determined that the user's hand meets a preset condition according to the preset plane, displaying a virtual interactive object on the preset plane; the determining of the preset plane corresponding to the plurality of preset key points according to the plurality of position information comprises: determining the preset plane corresponding to the plurality of preset key points according to the plurality of position information based on a preset plane equation; the determining of the preset plane corresponding to the plurality of preset key points according to the plurality of position information based on the preset plane equation comprises: determining a preset position matrix corresponding to the plurality of position information according to horizontal coordinates of the plurality of position information, vertical coordinates of the plurality of position information, and a preset constant; determining a preset position vector corresponding to the plurality of position information according to depth coordinates of the plurality of position information; determining a correlation between the preset position matrix, the preset position vector, and coefficients of the plane equation according to the preset position matrix, the preset position vector, and the plane equation; determining the coefficients of the plane equation according to the plurality of position information and the correlation based on a least square method; determining the preset plane according to the plane equation and the coefficients of the plane equation.

2. The display method according to claim 1, wherein the determining of the user's hand meeting the preset condition according to the preset plane comprises: obtaining a normal vector of the preset plane; determining an included angle between the normal vector and a preset reference vector; when the included angle is less than or equal to a preset angle threshold, determining that the user's hand meets the preset condition.

3. The display method according to claim 2, wherein the obtaining of the normal vector of the preset plane comprises: determining the normal vector of the preset plane according to the coefficients of the plane equation corresponding to the preset plane.

4. The display method according to claim 2, wherein the determining of the included angle between the normal vector and the preset reference vector comprises: substituting the normal vector and the preset reference vector into a preset included angle formula to obtain the included angle between the normal vector and the preset reference vector.

5. The display method according to claim 2, wherein after the determining of the included angle between the normal vector and the preset reference vector, the display method further comprises: when the included angle is greater than the preset angle threshold, determining that the user's hand does not meet the preset condition.

6. A display device of a near-eye display apparatus, characterized by comprising: the display device comprises: an image acquisition module configured to obtain a hand image of a user's hand captured by a near-eye display device; a position determination module configured to determine position information of a plurality of preset key points of the user's hand on the hand image; a plane determination module configured to determine a preset plane corresponding to the plurality of preset key points according to the plurality of position information; a display module configured to, when it is determined that the user's hand meets a preset condition according to the preset plane, display a virtual interactive object on the preset plane; the determining of the preset plane corresponding to the plurality of preset key points according to the plurality of position information comprises: Determine, based on the preset plane equation, a preset plane corresponding to the plurality of preset key points according to the plurality of position information; Determine, based on the preset plane equation, a preset plane corresponding to the plurality of preset key points according to the plurality of position information, including: Determine, according to the horizontal coordinate of each of the plurality of position information, the vertical coordinate of each of the plurality of position information, and a preset constant, a preset position matrix corresponding to the plurality of position information; Determine, according to the depth coordinate of each of the plurality of position information, a preset position vector corresponding to the plurality of position information; Determine, according to the preset position matrix, the preset position vector, and the plane equation, a correlation between the preset position matrix, the preset position vector, and the coefficients of the plane equation; Determine, based on the least square method, the coefficients of the plane equation according to the plurality of position information and the correlation; Determine, according to the plane equation and the coefficients of the plane equation, the preset plane.

7. A near-eye display device, comprising: The near-eye display device comprises a memory and a processor; The memory is configured to store a computer program; The processor is configured to execute the computer program and implement the steps of the display method of the near-eye display device according to any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the display method of the near-eye display device according to any one of claims 1 to 5.

Images