Display method and device of near-to-eye display equipment, equipment and storage medium

Abstract

The invention relates to the technical field of gesture interaction, and provides a display method and device of a near-eye display device, the device and a storage medium, and the method comprises the steps: obtaining a hand image obtained by shooting a user hand through the near-eye display device; determining position information of a plurality of preset key points of the hand of the user on the hand image; determining a preset plane corresponding to the plurality of preset key points according to the plurality of pieces of position information; when it is determined that the hand of the user meets the preset condition according to the preset plane, the virtual interaction object is displayed on the preset plane, so that the display accuracy of the near-eye display device on the virtual interaction object is improved.

Description

Technical Field

[0001] The present application relates to the field of gesture interaction technology, and in particular to a display method, apparatus, device and storage medium for a near-eye display device. Background Art

[0002] In the related art, a near-eye display device can display virtual interactive objects on the user's hands, such as a virtual keyboard, virtual special effects, and so on. However, in the process where the user can interact with the near-eye display device through the hand, the user often needs to place the hand in the shooting area of ​​the near-eye display device. In the case where the user does not need the near-eye display device to display the virtual interactive object, if the near-eye display device mistakenly determines that the virtual interactive object needs to be displayed on the user's hand based on the user's hand being in the shooting area, it is easy to cause the near-eye display device to display the virtual interactive object with poor accuracy, thereby affecting the user's interactive experience with the near-eye display device. Summary of the invention

[0003] The main purpose of the present application is to provide a display method, apparatus, device and storage medium for a near-eye display device, aiming to solve the technical problem of poor display accuracy of virtual interactive objects by the near-eye display device due to the near-eye display device erroneously judging that the virtual interactive object needs to be displayed on the user's hand based on the user's hand in the shooting area.

[0004] In a first aspect, the present application provides a display method of a near-eye display device, the display method comprising:

[0005] Acquire a hand image of a user's hand captured by a near-eye display device;

[0006] Determine position information of a plurality of preset key points of the user's hand on the hand image;

[0007] Determine, according to the plurality of position information, a preset plane corresponding to the plurality of preset key points;

[0008] When it is determined according to the preset plane that the user's hand meets the preset condition, a virtual interactive object is displayed on the preset plane.

[0009] In a second aspect, the present application provides a display device of a near-eye display device, the display device comprising:

[0010] An image acquisition module, used to acquire a hand image of a user's hand captured by a near-eye display device;

[0011] A position determination module, used to determine position information of a plurality of preset key points of the user's hand on the hand image;

[0012] A plane determination module, used to determine a preset plane corresponding to a plurality of the preset key points according to the plurality of the position information;

[0013] The display module is used to display a virtual interactive object on the preset plane when it is determined according to the preset plane that the user's hand meets a preset condition.

[0014] In a third aspect, the present application provides a near-eye display device, the near-eye display device comprising a memory and a processor;

[0015] The memory is used to store computer programs;

[0016] The processor is used to execute the computer program and implement the steps of the display method of the near-eye display device as described above when executing the computer program.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the display method of the near-eye display device as described above are implemented.

[0018] The present application provides a display method, apparatus, device and storage medium for a near-eye display device. The display method includes: obtaining a hand image obtained by photographing a user's hand by a near-eye display device; determining position information of multiple preset key points of the user's hand on the hand image; determining a preset plane corresponding to the multiple preset key points based on the multiple position information; and displaying a virtual interactive object on the preset plane when it is determined that the user's hand meets preset conditions based on the preset plane.

[0019] When the user places his hand in the shooting area of ​​the near-eye display device, the near-eye display device can shoot the user's hand to obtain a hand image. When the hand image is obtained, the near-eye display device can determine the preset plane based on the position information of multiple preset key points in the hand image. Accordingly, the near-eye display device can determine whether the user's hand meets the preset conditions based on the preset plane, so as to determine that the user has a display demand for the virtual interactive object when the user's hand meets the preset conditions. The near-eye display device can display the virtual interactive object in response to the user's display demand for the virtual interactive object, thereby facilitating the improvement of the display accuracy of the virtual interactive object by the near-eye display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a flowchart of a display method of a near-eye display device provided in an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of a hand image involved in an embodiment of the present application;

[0023] Figure 3 is a display schematic diagram of a virtual interactive object involved in an embodiment of the present application;

[0024] Figure 4 is a display schematic diagram of a virtual interactive object involved in another embodiment of the present application;

[0025] Figure 5 is a schematic block diagram of a display device of a near-eye display device provided in an embodiment of the present application;

[0026] Figure 6 It is a schematic block diagram of a near-eye display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0029] The embodiments of the present application provide a display method, device, equipment and storage medium for a near-eye display device. The display method of the near-eye display device can be applied to a near-eye display device. The near-eye display device may include virtual reality (VR) glasses, augmented reality (AR) glasses, mixed reality (MR) glasses, VR helmets, AR helmets, MR helmets, etc., without limitation. The display method of the near-eye display device can also be applied to a server, which can be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0030] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0031] See also Figure 1 , Figure 1 It is a flow chart of a display method of a near-eye display device provided in an embodiment of the present application. It should be noted that the hand recognition method provided in the embodiment of the present application can be used in a near-eye display device or in a server, which is not limited here.

[0032] like Figure 1 As shown, the display method of the near-eye display device includes steps S101 to S104.

[0033] S101: Acquire a hand image of a user's hand captured by a near-eye display device.

[0034] For example, the near-eye display device can capture the hand of a user wearing the near-eye display device to obtain a corresponding hand image. For example, the near-eye display device can be provided with a shooting module. When it is detected that the user's hand is within the shooting range corresponding to the near-eye display device, the near-eye display device can capture the user's hand through the shooting module to obtain a hand image.

[0035] In some embodiments, when the near-eye display device acquires a hand image, since the hand image may include a hand image area and a background image area, the near-eye display device may perform image preprocessing on the hand image to remove the background image area as much as possible. For example, the near-eye display device may perform cropping on the hand image to reduce the background image area and retain the hand image area, which is not limited here.

[0036] In this way, when the near-eye display device obtains the hand image, the hand image can be used by the near-eye display device to determine the user's display demand for the virtual interactive object, so that the virtual interactive object can be displayed later when the user has a display demand for the virtual interactive object.

[0037] S102: Determine position information of a plurality of preset key points of the user's hand on the hand image.

[0038] For example, when a hand image is acquired, the near-eye display device can perform key point detection on the hand image. For example, the near-eye display device can determine the position information of multiple preset key points of the user's hand on the hand image based on a preset key point detection algorithm or a deep learning model. The number of preset key points can include 5, 6, 7, etc., which is not limited here.

[0039] In some embodiments, when the near-eye display device displays the virtual interactive object in response to the user's display demand for the virtual interactive object, the virtual interactive object can be displayed in the palm area of ​​the user's hand. Based on this, when the near-eye display device obtains the hand image, it can determine the user's display demand for the virtual interactive object based on the palm area corresponding to the hand image. Figure 2 As shown, the near-eye display device can determine the palm image corresponding to the palm area from the hand image. Accordingly, the preset key points can be determined from the key points included in the palm area. For example, the near-eye display device can perform key point detection on the palm image corresponding to the palm area in the hand image to obtain position information of multiple preset key points on the hand image.

[0040] In this way, the near-eye display device can determine the position information of multiple preset key points on the hand image based on the hand image, which is conducive to improving the convenience of determining the position information of the preset key points on the hand image. In the case where the near-eye display device determines the position information of multiple preset key points, the position information of the multiple preset key points can be used by the near-eye display device to determine the user's display requirements for the virtual interactive object, so as to display the virtual interactive object in the subsequent case where the user has a display requirement for the virtual interactive object.

[0041] S103: Determine preset planes corresponding to multiple preset key points according to the multiple position information.

[0042] When the near-eye display device determines the position information of multiple preset key points, it can fit the preset planes corresponding to the multiple preset key points based on the multiple position information. The preset plane can be as close to the multiple preset key points as possible, such as the preset plane can satisfy the minimum sum of the squares of the vertical distances between the multiple preset key points and the preset plane.

[0043] Exemplarily, based on a preset plane equation and according to a plurality of position information, a preset plane corresponding to a plurality of preset key points is determined.

[0044] For example, the preset plane equation can be expressed as ax+by+d=z. a, b, d are used to indicate the coefficients of the plane equation, and a, b, d are constants. The coefficients of the plane equation may include coefficients of different items of the plane equation, and the coefficients of different items may be different or the same, which is not limited here.

[0045] For example, the position information of the plurality of preset key points may include 2.5D coordinates, and the position information may include a horizontal coordinate (x coordinate), a vertical coordinate (y coordinate), and a depth coordinate (z coordinate).

[0046] The preset plane equation is equivalent to a universal representation of different planes. When the coefficients of different plane equations are different, the planes indicated by different plane equations may be different. Based on this, the near-eye display device can solve the coefficients of the plane equation according to multiple position information to determine the preset planes corresponding to multiple preset key points.

[0047] In some embodiments, a preset position matrix corresponding to the multiple position information is determined based on the horizontal coordinates of each of the multiple position information, the vertical coordinates of each of the multiple position information, and a preset constant; a preset position vector corresponding to the multiple position information is determined based on the depth coordinates of each of the multiple position information; a correlation between the preset position matrix and the preset position vector and the coefficients of the plane equation is determined based on the preset position matrix, the preset position vector, and the plane equation; based on the least squares method, the coefficients of the plane equation are determined based on the multiple position information and the correlation; and a preset plane is determined based on the plane equation and the coefficients of the plane equation.

[0048] Taking the number of the preset key points as 7 as an example, the horizontal coordinates of the plurality of position information may include the coordinate x 1 , coordinate x 2 To coordinate x 7 , the vertical coordinates of each of the plurality of position information may include the coordinate y 1 , coordinate y 2 To coordinate y 7 , the depth coordinates of each of the multiple position information may include the coordinate z 1 , coordinate z 2 To coordinate z 7 The preset constant may include 1. The preset position matrix determined by the near-eye display device according to the horizontal coordinates of each of the plurality of position information, the vertical coordinates of each of the plurality of position information and the preset constant may be expressed as a matrix P:

[0049]

[0050] The near-eye display device determines the preset position vector according to the depth coordinates of each of the multiple position information, which can be expressed as a vector

[0051]

[0052] The near-eye display device can be based on the matrix P and the vector Determine the matrix P and the vector The relationship between the coefficients of the plane equation can be expressed as:

[0053]

[0054] Where a, b, and d are used to indicate the coefficients of the plane equation.

[0055] After determining the matrix P and the vector The vector corresponding to the coefficients of the plane equation In the case of the correlation relationship between the vector Solve to determine the coefficients of the plane equation.

[0056] For example, the near-eye display device can determine based on the least squares method Based on this, the near-eye display device can combine the matrix P and the vector The coefficients of the plane equation, i.e., the specific constant values ​​corresponding to a, b, and d, are determined. Accordingly, the near-eye display device can substitute the specific constant values ​​corresponding to a, b, and d into the plane equation to obtain the plane equation corresponding to the preset plane, thereby determining the preset plane. Among them, the matrix P and the vector The corresponding number of rows is determined according to the number of preset key points. When the number of preset key points changes, the matrix P and the vector The corresponding number of rows will also change.

[0057] When the plane equation corresponding to the preset plane is obtained according to the plane equation and the coefficient of the plane equation, that is, when the preset plane is determined, the preset plane can minimize the sum of the squares of the vertical distances between the multiple preset key points and the preset plane. Accordingly, the multiple preset key points can all be located in the preset plane, can also be partially located in the preset plane, or can all be located outside the preset plane, without limitation here.

[0058] In this way, the near-eye display device can determine the preset planes corresponding to the multiple preset key points based on the multiple position information, which is conducive to improving the convenience of determining the preset planes. When the near-eye display device determines the preset planes corresponding to the multiple preset key points, the preset planes can be used by the near-eye display device to determine the user's display requirements for the virtual interactive object, so as to display the virtual interactive object in the subsequent case where the user has a display requirement for the virtual interactive object.

[0059] S104: When it is determined according to the preset plane that the user's hand meets the preset condition, display the virtual interactive object on the preset plane.

[0060] For example, when a user wears a near-eye display device, the shooting direction of the near-eye display device may be consistent with the direction of the user's line of sight, and the near-eye display device may set the preset conditions according to the direction of the preset plane combined with the direction of the reference plane to determine whether the user has a display demand for the virtual interactive object. The direction of the reference plane is perpendicular to the shooting direction of the near-eye display device.

[0061] In some embodiments, the near-eye display device can determine the angle between the direction of the preset plane and the direction of the reference plane, and determine whether the user's hand meets the preset conditions by determining whether the angle between the direction of the preset plane and the direction of the reference plane is within a preset angle range. For example, when the angle between the direction of the preset plane and the direction of the reference plane is within a preset angle range, the near-eye display device can determine that the user's hand meets the preset conditions. Taking the angle corresponding to the direction of the reference plane as 0 degrees as an example, the preset angle range includes, for example, 0 to 15 degrees, 0 to 20 degrees, 0 to 30 degrees, and so on. The preset angle range can be pre-set or set by the user, and is not limited here.

[0062] For example, when the angle between the direction of the preset plane and the direction of the reference plane is within a preset angle range, the near-eye display device can determine that the deviation angle of the preset plane relative to the direction of the reference plane is less than or equal to a preset deviation angle threshold. Based on this, the near-eye display device can determine that the user's hand is facing the shooting direction of the near-eye display device, or that the user's hand is approximately facing the shooting direction of the near-eye display device. Figure 3As shown, when the deviation angle of the preset plane relative to the direction of the reference plane is less than the preset deviation angle threshold, if a virtual interactive object, such as a virtual keyboard, virtual special effects, etc., is displayed on the preset plane, when the user interacts with different areas of the virtual interactive object, visual occlusion is not likely to occur between different areas. For example, when the user interacts with the virtual interactive object displayed in the preset plane through multiple fingers of the other hand, if the user places different fingers on different interactive sub-objects of the virtual interactive object, the near-eye display device can continue to shoot the user's hand to obtain a corresponding interactive hand image. Based on the fact that the angle between the direction of the preset plane and the direction of the reference plane is within the preset angle range, the user's hand is facing the shooting direction of the near-eye display device, or the user's hand is approximately facing the shooting direction of the near-eye display device, the position difference between different fingers of the other hand in the interactive hand image under the shooting direction can be large, and the positions of the interactive sub-objects corresponding to different fingers can be obtained based on the interactive hand image to determine the user's interaction needs. Take the virtual interactive object as the nine-square grid keyboard corresponding to the nine-square grid input method, and the interactive sub-object includes the virtual keys in the nine-square grid keyboard. For example, the user places the index finger, middle finger, and ring finger of the other hand on three adjacent virtual keys of the nine-square grid keyboard respectively. Since the preset plane is similar to the shooting direction of the near-eye display device, even if the three adjacent virtual keys are in the same row or column of the nine-square grid keyboard, the position difference between the index finger, middle finger, and ring finger in the interactive hand image captured by the near-eye display device is also large. At this time, the user's interaction convenience with the virtual interactive object is high. Based on this, the near-eye display device can determine that the user's hand meets the preset conditions according to the preset plane. Accordingly, the near-eye display device can determine that the user has a display demand for the virtual interactive object, thereby displaying the virtual interactive object on the preset plane.

[0063] For example, when the angle between the direction of the preset plane and the direction of the reference plane is not within the preset angle range, the near-eye display device can determine that the deviation angle of the preset plane relative to the direction of the reference plane is greater than a preset deviation angle threshold. Based on this, the near-eye display device can determine that the user's hand is close to the shooting direction of the near-eye display device. Figure 4As shown, when the deviation angle of the preset plane relative to the direction of the reference plane is greater than the preset deviation angle threshold, if a virtual interactive object, such as a virtual keyboard, virtual special effects, etc., is displayed on the preset plane, when the user interacts with different areas of the virtual interactive object, visual occlusion is likely to occur between different areas. For example, when the user interacts with the virtual interactive object displayed in the preset plane through multiple fingers of the other hand, if the user places different fingers on different interactive sub-objects of the virtual interactive object, the near-eye display device can continue to shoot the user's hand to obtain the corresponding interactive hand image. Based on the fact that the angle between the direction of the preset plane and the direction of the reference plane is not within the preset angle range, the user's hand is close to the shooting direction of the near-eye display device, so that the position difference between different fingers of the other hand in the interactive hand image under the shooting direction can be small, then the positions of the interactive sub-objects corresponding to different fingers cannot be obtained based on the interactive hand image, and thus the user's interaction needs cannot be determined. Take the virtual interactive object as the nine-square grid keyboard corresponding to the nine-square grid input method, the interactive sub-object includes the virtual keys in the nine-square grid keyboard, and the user places the index finger, middle finger and ring finger of the other hand on three adjacent virtual keys of the nine-square grid keyboard respectively. As an example. Since the preset plane can be approximated to the shooting direction of the near-eye display device, when the three adjacent virtual keys are in the same row or column of the nine-square grid keyboard, the position difference between the index finger, middle finger and ring finger in the interactive hand image captured by the near-eye display device is small. For example, the index finger, middle finger and ring finger are prone to overlap and occlusion, resulting in the appearance of a shooting blind spot of the near-eye display device. At this time, the user's interaction convenience with the virtual interactive object is low. Based on this, the near-eye display device can determine that the preset plane determines that the user's hand does not meet the preset conditions. Accordingly, the near-eye display device can determine that the user has no display demand for the virtual interactive object, thereby determining that the virtual interactive object does not need to be displayed on the preset plane.

[0064] Exemplarily, since the direction indicated by the normal vector of the preset plane is perpendicular to the direction of the preset plane, the near-eye display device can determine that the angle between the normal vector and the shooting direction of the near-eye display device and the angle between the direction of the preset plane and the shooting direction of the near-eye display device are in a complementary relationship. Correspondingly, the near-eye display device can also determine that the angle between the normal vector and the shooting direction of the near-eye display device and the angle between the normal vector and the direction of the reference plane are in a complementary relationship. Based on this, the near-eye display device can set the preset conditions according to the direction indicated by the normal vector of the preset plane combined with the direction of the reference plane to determine whether the user has a display demand for the virtual interactive object.

[0065] In some embodiments, a normal vector of a preset plane is obtained; an angle between the normal vector and a preset reference vector is determined; and when the angle is less than or equal to a preset angle threshold, it is determined that the user's hand meets a preset condition.

[0066] The near-eye display device may obtain a normal vector of a preset plane. The near-eye display device may determine a preset reference vector according to a direction of a reference plane, wherein the direction of the reference plane is perpendicular to the direction of the near-eye display device. When the near-eye display device obtains a normal vector of a preset plane and a preset reference vector, the near-eye display device may determine an angle between the normal vector and the preset reference vector.

[0067] For example, the normal vector of the preset plane is determined according to the coefficients of the plane equation corresponding to the preset plane.

[0068] Taking the plane equation corresponding to the preset plane as ax+by+d=z as an example, the coefficients of the plane equation include a, b and d. Then, the near-eye display device determines the normal vector of the preset plane according to the coefficients of the plane equation corresponding to the preset plane, which can be expressed as a vector

[0069]

[0070] The normal vector of the preset plane can be used by the near-eye display device in combination with a preset reference vector to determine whether the user's hand meets the preset conditions.

[0071] For example, the normal vector and the preset reference vector are substituted into the preset angle formula to obtain the angle between the normal vector and the preset reference vector.

[0072] The preset reference vector can be expressed as a vector

[0073]

[0074] The angle between the normal vector and the preset reference vector can be expressed as angle θ. The preset angle formula can be expressed as:

[0075]

[0076] in, To indicate vector Dot product vector To indicate vector The modulus value, To indicate vector The modulus value of .

[0077] Vector And vector Substituting into the preset angle formula, we can determine:

[0078]

[0079] Accordingly, when the cosine value cosθ of the angle θ is determined, the near-eye display device can determine the size of the angle θ by the correlation between the cosine value and the angle. The near-eye display device can also determine the corresponding angle θ according to the cosine value cosθ by an inverse cosine function. This is not limited here.

[0080] In another exemplary embodiment, when the near-eye display device obtains the normal vector of a preset plane, it can also determine a preset reference vector according to the shooting direction of the near-eye display device.

[0081] Take the plane equation corresponding to the preset plane as ax+by+d=z as an example. In order to maintain the consistency of the normal vector of the preset plane with the shooting direction of the near-eye display device, the normal vector of the preset plane can be expressed as a vector

[0082]

[0083] The preset reference vector determined according to the shooting direction of the near-eye display device can be expressed as a vector

[0084]

[0085] The preset angle formula can be expressed as:

[0086]

[0087] in, To indicate vector Dot product vector To indicate vector The modulus value, To indicate vector The modulus value of .

[0088] Vector And vector Substituting into the preset angle formula, we can determine:

[0089]

[0090] Accordingly, when the cosine value cosθ of the angle θ is determined, the near-eye display device can determine the size of the angle θ by the correlation between the cosine value and the angle. The near-eye display device can also determine the corresponding angle θ according to the cosine value cosθ by an inverse cosine function. This is not limited here.

[0091] When the angle between the normal vector and the preset reference vector is less than or equal to the preset angle threshold, the near-eye display device can determine that the deviation angle of the preset plane relative to the reference plane is less than or equal to the preset deviation angle threshold. Accordingly, the near-eye display device can determine that the user's hand is facing the shooting direction of the near-eye display device, or the user's hand is approximately facing the shooting direction of the near-eye display device, thereby determining that the user's hand meets the preset conditions. Taking the angle indicated by the preset reference vector as 0 degrees as an example, the preset angle thresholds include 5 degrees, 10 degrees, 15 degrees, 20 degrees, 25 degrees, 30 degrees, etc. The preset angle thresholds can be pre-set or set by the user, and are not limited here.

[0092] In some implementations, when the included angle is greater than a preset angle threshold, it is determined that the user's hand does not meet the preset condition.

[0093] When the angle between the normal vector and the preset reference vector is greater than the preset angle threshold, the near-eye display device can determine that the deviation angle of the preset plane relative to the reference plane is greater than the preset deviation angle threshold. Accordingly, the near-eye display device can determine that the user's hand is close to the shooting direction of the near-eye display device, thereby determining that the user's hand does not meet the preset condition.

[0094] In this way, the near-eye display device can determine the normal vector and the angle between the normal vector and the preset reference vector, and then evaluate whether the user's hand meets the preset conditions based on whether the angle between the normal vector and the preset reference vector is less than or equal to the preset angle threshold, which is equivalent to evaluating whether the user has the display requirements for the virtual interactive object, which is conducive to improving the convenience and accuracy of determining the user's display requirements for the virtual interactive object. Based on the improvement of the convenience and accuracy of determining the user's display requirements for the virtual interactive object, the near-eye display device can display the virtual interactive object when it is determined that the user's hand meets the preset conditions, and not display the virtual interactive object when it is determined that the user's hand does not meet the preset conditions. Based on this, the near-eye display device can adapt to the user's display requirements for the virtual interactive object when displaying the virtual interactive object, which is conducive to improving the display accuracy of the near-eye display device for the virtual interactive object.

[0095] The display method of the near-eye display device provided in the above embodiment includes: obtaining a hand image obtained by photographing the user's hand by the near-eye display device; determining position information of multiple preset key points of the user's hand on the hand image; determining a preset plane corresponding to the multiple preset key points based on the multiple position information; when it is determined that the user's hand meets the preset conditions based on the preset plane, displaying a virtual interactive object on the preset plane.

[0096] When the user places his hand in the shooting area of ​​the near-eye display device, the near-eye display device can shoot the user's hand to obtain a hand image. When the hand image is obtained, the near-eye display device can determine the preset plane based on the position information of multiple preset key points in the hand image. Accordingly, the near-eye display device can determine whether the user's hand meets the preset conditions based on the preset plane, so as to determine that the user has a display demand for the virtual interactive object when the user's hand meets the preset conditions. The near-eye display device can display the virtual interactive object in response to the user's display demand for the virtual interactive object, thereby facilitating the improvement of the display accuracy of the virtual interactive object by the near-eye display device.

[0097] See also Figure 5 , Figure 5 It is a schematic block diagram of a display device of a near-eye display device provided in an embodiment of the present application. The display device of the near-eye display device can be configured in a near-eye display device or a server to execute the display method of the aforementioned near-eye display device. The near-eye display device may include VR glasses, AR glasses, MR glasses, VR helmets, AR helmets, MR helmets, and the like, without limitation. The server may be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms.

[0098] like Figure 5 As shown, the display device of the near-eye display apparatus includes an image acquisition module 110 , a position determination module 120 , a plane determination module 130 and a display module 140 .

[0099] The image acquisition module 110 is used to acquire a hand image of a user's hand captured by a near-eye display device.

[0100] The position determination module 120 is used to determine the position information of a plurality of preset key points of the user's hand on the hand image.

[0101] The plane determination module 130 is used to determine the preset planes corresponding to the plurality of preset key points according to the plurality of position information.

[0102] The display module 140 is configured to display a virtual interactive object on the preset plane when it is determined that the user's hand meets a preset condition according to the preset plane.

[0103] Exemplarily, the plane determination module 130 includes a first plane determination submodule.

[0104] The first plane determination submodule is used to determine the preset planes corresponding to the plurality of preset key points based on a preset plane equation and according to the plurality of position information.

[0105] Exemplarily, the first plane determination submodule includes a matrix determination submodule, a vector determination submodule, a relationship determination submodule, a coefficient determination submodule and a second plane determination submodule.

[0106] The matrix determination submodule is used to determine the preset position matrix corresponding to the plurality of position information according to the respective horizontal coordinates of the plurality of position information, the respective vertical coordinates of the plurality of position information and the preset constant.

[0107] The vector determination submodule is used to determine the preset position vectors corresponding to the plurality of position information according to the respective depth coordinates of the plurality of position information.

[0108] The relationship determination submodule is used to determine the association relationship between the preset position matrix and the preset position vector and the coefficients of the plane equation according to the preset position matrix, the preset position vector and the plane equation.

[0109] The coefficient determination submodule is used to determine the coefficients of the plane equation based on the least square method according to the multiple position information and the association relationship.

[0110] The second plane determination submodule is used to determine the preset plane according to the plane equation and the coefficients of the plane equation.

[0111] Exemplarily, the display module 140 includes a normal vector acquisition submodule, an angle determination submodule, and a first judgment submodule.

[0112] The normal vector acquisition submodule is used to obtain the normal vector of the preset plane.

[0113] The angle determination submodule is used to determine the angle between the normal vector and a preset reference vector.

[0114] The first judgment submodule is used to determine that the user's hand meets a preset condition when the angle is less than or equal to a preset angle threshold.

[0115] Exemplarily, the normal vector acquisition submodule includes a normal vector determination submodule.

[0116] The normal vector determination submodule is used to determine the normal vector of the preset plane according to the coefficients of the plane equation corresponding to the preset plane.

[0117] Exemplarily, the angle determination submodule includes an angle calculation submodule.

[0118] The angle calculation submodule is used to substitute the normal vector and the preset reference vector into a preset angle formula to obtain the angle between the normal vector and the preset reference vector.

[0119] Exemplarily, the display device further includes a second judgment submodule.

[0120] The second judgment submodule is used to determine that the user's hand does not meet the preset condition when the angle is greater than a preset angle threshold.

[0121] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0122] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0123] Exemplarily, the above-mentioned method and apparatus may be implemented in the form of a computer program, which may be run on a near-eye display device or a server to control the near-eye display device. Exemplarily, the near-eye display device may include VR glasses, AR glasses, MR glasses, VR helmets, AR helmets, MR helmets, etc., without limitation. The server may be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms.

[0124] See also Figure 6 , Figure 6 It is a schematic block diagram of the structure of a near-eye display device provided in an embodiment of the present application.

[0125] like Figure 6As shown, the near-eye display device includes a memory and a processor. The memory and the processor may be connected via a system bus, and the memory may include a storage medium and an internal memory.

[0126] The storage medium may store an operating system and a computer program. When the computer program is executed, the processor may execute any display method of a near-eye display device.

[0127] The processor is used to provide computing and control capabilities to support the operation of the entire near-eye display device.

[0128] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any display method of the near-eye display device.

[0129] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0130] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0131] In one embodiment, the processor is used to execute a computer program and implement the following steps when executing the computer program:

[0132] Acquire a hand image of a user's hand captured by a near-eye display device;

[0133] Determine position information of a plurality of preset key points of the user's hand on the hand image;

[0134] Determine, according to the plurality of position information, a preset plane corresponding to the plurality of preset key points;

[0135] When it is determined according to the preset plane that the user's hand meets the preset condition, a virtual interactive object is displayed on the preset plane.

[0136] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the display of the near-eye display device described above can refer to the corresponding process in the display method embodiment of the aforementioned near-eye display device, and will not be repeated here.

[0137] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The method implemented when the computer program is executed by a processor can refer to the various embodiments of the display method of the near-eye display device of the present application.

[0138] The computer-readable storage medium may be an internal storage unit of the near-eye display device described in the aforementioned embodiment, such as a hard disk or memory of the near-eye display device. The computer-readable storage medium may also be an external storage device of the near-eye display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the near-eye display device.

[0139] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0140] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0141] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A display method for a near-eye display device, characterized in that: The display method comprises: Acquire a hand image of a user's hand captured by a near-eye display device; Determine position information of a plurality of preset key points of the user's hand on the hand image; Determine, according to the plurality of position information, a preset plane corresponding to the plurality of preset key points; When it is determined according to the preset plane that the user's hand meets the preset condition, a virtual interactive object is displayed on the preset plane.

2. The display method according to claim 1, characterized in that: The step of determining the preset planes corresponding to the plurality of preset key points according to the plurality of position information includes: Based on a preset plane equation and according to the plurality of position information, a preset plane corresponding to the plurality of preset key points is determined.

3. The display method according to claim 2, characterized in that: The step of determining the preset planes corresponding to the plurality of preset key points based on the preset plane equation and according to the plurality of position information includes: Determine a preset position matrix corresponding to the plurality of position information according to the respective horizontal coordinates of the plurality of position information, the respective vertical coordinates of the plurality of position information and a preset constant; Determining preset position vectors corresponding to the plurality of position information according to respective depth coordinates of the plurality of position information; Determine, according to the preset position matrix, the preset position vector and the plane equation, a correlation relationship between the preset position matrix and the preset position vector and coefficients of the plane equation; Determine the coefficients of the plane equation based on the least squares method according to the plurality of position information and the association relationship; The preset plane is determined according to the plane equation and the coefficients of the plane equation.

4. The display method according to any one of claims 1 to 3, characterized in that: The determining, according to the preset plane, that the user's hand meets a preset condition includes: Obtaining a normal vector of the preset plane; Determining the angle between the normal vector and a preset reference vector; When the included angle is less than or equal to a preset angle threshold, it is determined that the user's hand meets a preset condition.

5. The display method according to claim 4, characterized in that: The obtaining of the normal vector of the preset plane includes: The normal vector of the preset plane is determined according to the coefficients of the plane equation corresponding to the preset plane.

6. The display method according to claim 4, characterized in that: The determining the angle between the normal vector and a preset reference vector comprises: Substitute the normal vector and the preset reference vector into a preset angle formula to obtain the angle between the normal vector and the preset reference vector.

7. The display method according to claim 4, characterized in that: After determining the angle between the normal vector and the preset reference vector, the display method further includes: When the included angle is greater than a preset angle threshold, it is determined that the user's hand does not meet a preset condition.

8. A display device of a near-eye display device, characterized in that: The display device comprises: An image acquisition module, used to acquire a hand image of a user's hand captured by a near-eye display device; A position determination module, used to determine position information of a plurality of preset key points of the user's hand on the hand image; A plane determination module, used to determine a preset plane corresponding to a plurality of the preset key points according to the plurality of the position information; The display module is used to display a virtual interactive object on the preset plane when it is determined according to the preset plane that the user's hand meets a preset condition.

9. A near-eye display device, characterized in that: The near-eye display device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the steps of the display method of the near-eye display device according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the display method of the near-eye display device according to any one of claims 1 to 7 are implemented.

Images