Gesture degree-of-freedom pose determination method and device and electronic equipment

Through the combination of monocular camera image and range measurement module, the calculation of the hand 3-degree of freedom translation posture is simplified, the problem of high calculation complexity in the prior art is solved, and more efficient posture determination is achieved.

CN118915920APending Publication Date: 2024-11-08BEIJING CHEHEJIA AUTOMOBILE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310511623.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-08
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the field of virtual reality and reality augmentation, the calculation complexity of determining the hand 3-degree of freedom translation posture is high in the prior art, and a multi-eye camera is required to acquire multi-eye images for comprehensive calculation, resulting in an increase in the calculation complexity.

Method used

By obtaining a monocular camera image containing a hand, measuring the distance between the hand and the center of the monocular camera, calculating the distance between the hand position and the center of the image and the radius of the circumference, and calculating the field angle and proportion of the monocular camera, and calculating the The angle between the line between the hand and the center of light and the offset distance of the field of view are used to determine the 3-degree-of-freedom translation position of the hand by using the proportional relationship between the field of view and the image during imaging.

Benefits of technology

The calculation process of the hand 3-degree-of-freedom translation posture is simplified, the calculation complexity is reduced, and the calculation efficiency is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915920A_ABST
    Figure CN118915920A_ABST
Patent Text Reader

Abstract

The invention provides a gesture freedom degree pose determination method and device and electronic equipment, and the method comprises the steps: obtaining a target image containing a hand, obtaining a first distance between the hand and the optical center of a monocular camera, and calculating a second distance between the hand position in the target image and the center of the image, calculating an included angle between a connecting line between the hand and the optical center of the monocular camera and the optical axis of the monocular camera based on the field angle of the monocular camera and the first proportion, and calculating a second proportion between the horizontal coordinate and the vertical coordinate of the hand position based on the included angle, a third distance between the optical center of the monocular camera and the field plane and the horizontal coordinate and the vertical coordinate of the hand position; and the transverse offset distance and the longitudinal offset distance between the hand and the center of the view field plane are calculated, the transverse offset distance, the longitudinal offset distance and the first distance serve as the three-degree-of-freedom translation pose of the hand, and the calculation complexity can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of virtual reality technology, and in particular to a method, device and electronic device for determining the degree of freedom of gesture. Background Art

[0002] Object pose estimation plays an important role in fields such as Virtual Reality (VR) and Augmented Reality (AR). In practical applications, VR / AR devices usually use the gyroscope of the handle to determine the 3 degrees of freedom (3dof) rotation pose of the hand, and further combine the multi-eye camera to determine the 3dof translation pose, that is, 6dof pose. Among them, in the related technology, when determining the 3dof translation pose, it is necessary to use a multi-eye camera to collect multi-eye images, and combine multiple images to calculate the 3dof translation pose. Therefore, the computational complexity of the 3dof translation pose needs to be improved. Summary of the invention

[0003] The present disclosure proposes a method, device and electronic device for determining the degree of freedom of gesture, aiming to solve one of the technical problems in the related art at least to a certain extent.

[0004] The first aspect of the present disclosure proposes a method for determining a gesture degree of freedom posture, including: obtaining a target image including a hand, and obtaining a first distance from the hand to the optical center of a monocular camera; calculating a second distance from the hand position in the target image to the center of the image, and determining a first ratio of the second distance to the radius of the circumscribed circle of the target image; calculating an angle between a line connecting the hand and the optical center of the monocular camera and the optical axis of the monocular camera based on the field of view angle of the monocular camera and the first ratio; calculating a lateral offset distance and a longitudinal offset distance from the hand to the center of the field of view plane based on the angle, a third distance from the optical center of the monocular camera to the field of view plane, and a second ratio of the horizontal and vertical coordinates of the hand position; and using the lateral offset distance, the longitudinal offset distance, and the first distance as the three-degree-of-freedom translation posture of the hand.

[0005] The second aspect of the present disclosure proposes a gesture degree of freedom posture determination device, including: an acquisition module, used to acquire a target image including a hand, and acquire a first distance from the hand to the optical center of a monocular camera; a first calculation module, used to calculate a second distance from the hand position in the target image to the center of the image, and determine a first ratio of the second distance to the radius of the circumscribed circle of the target image; a second calculation module, used to calculate the angle between the line connecting the hand and the optical center of the monocular camera and the optical axis of the monocular camera based on the field of view angle of the monocular camera and the first ratio; a third calculation module, used to calculate the lateral offset distance and the longitudinal offset distance of the hand from the center of the field of view plane based on the angle, the third distance from the optical center of the monocular camera to the field of view plane, and the second ratio of the horizontal and vertical coordinates of the hand position; and a first determination module, used to use the lateral offset distance, the longitudinal offset distance and the first distance as the three-degree-of-freedom translation posture of the hand.

[0006] An embodiment of the third aspect of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the gesture degree of freedom posture determination method of the embodiment of the present disclosure.

[0007] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the gesture degree of freedom posture determination method disclosed in the embodiment of the present disclosure.

[0008] A fifth aspect of the present disclosure provides a virtual reality device, which includes the electronic device of the first aspect.

[0009] In this embodiment, a target image including a hand is obtained, and a first distance between the hand and the optical center of the monocular camera is obtained, and a second distance between the hand position in the target image and the image center is calculated, and a first ratio of the second distance to the radius of the circumscribed circle of the target image is determined, based on the field of view angle of the monocular camera and the first ratio, the angle between the line connecting the hand and the optical center of the monocular camera and the optical axis of the monocular camera is calculated, based on the angle, the third distance between the optical center of the monocular camera and the field of view plane, and the second ratio of the horizontal and vertical coordinates of the hand position, the lateral offset distance and the longitudinal offset distance between the hand and the center of the field of view plane are calculated, and the lateral offset distance, the longitudinal offset distance and the first distance are used as the three-degree-of-freedom translation posture of the hand, the proportional relationship between the field of view and the image during imaging can be utilized, and the three-degree-of-freedom translation posture of the hand can be determined according to the monocular image and the distance between the hand and the monocular camera light. Compared with the existing multi-eye image translation posture calculation, the calculation process of this scheme is simple, so the complexity of the translation posture calculation can be reduced.

[0010] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0012] Figure 1 It is a flowchart of a method for determining the degree of freedom of gesture provided in accordance with an embodiment of the present disclosure;

[0013] Figure 2 is a structural schematic diagram of the stereoscopic field of view of a monocular camera provided according to an embodiment of the present disclosure;

[0014] Figure 3 is a schematic diagram of a structure of a distance measurement module for measuring hand distance according to an embodiment of the present disclosure;

[0015] Figure 4 is a schematic diagram of the structure of a target image and a plane field of view of a monocular camera provided according to an embodiment of the present disclosure;

[0016] Figure 5 is a flow chart of a method for determining the degree of freedom of gesture provided in accordance with another embodiment of the present disclosure;

[0017] Figure 6 is a schematic diagram of the structure of multiple bone points of a hand provided according to an embodiment of the present disclosure;

[0018] Figure 7 is a schematic diagram of the structure of the pitch angle, yaw angle and roll angle provided according to an embodiment of the present disclosure;

[0019] Figure 8 is a schematic diagram of a gesture degree of freedom position determination device provided according to another embodiment of the present disclosure;

[0020] Fig. 9 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0021] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0022] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0023] It should be noted that the executor of the gesture degree of freedom posture determination method of this embodiment may be a gesture degree of freedom posture determination device, which may be implemented by software and / or hardware, and which may be configured in an electronic device, which may include but is not limited to a terminal, a server, etc.

[0024] Figure 1 is a flow chart of a method for determining the degree of freedom of gesture according to an embodiment of the present disclosure, such as Figure 1 As shown, the method includes:

[0025] S101: Acquire a target image including a hand, and acquire a first distance between the hand and the optical center of a monocular camera.

[0026] In order to implement the gesture freedom posture determination method, the embodiment of the present disclosure also provides a posture determination system, which includes a computing unit, a monocular camera (such as an RGB camera), a ranging module, and any other possible device, without limitation. The ranging module can, for example, use binocular ranging, infrared structured light ranging, radar ranging, or any other possible type of ranging device, and the output of the ranging module is pre-configured to be aligned with the output of the monocular camera.

[0027] Specifically, Figure 2 is a structural schematic diagram of the stereoscopic field of view of a monocular camera provided according to an embodiment of the present disclosure, Figure 3 is a schematic diagram of a structure of measuring hand distance according to a distance measurement module provided in an embodiment of the present disclosure, such as Figure 2 and Figure 3 As shown, this embodiment first uses a monocular camera to capture an image containing a hand (for example, an RGB image), which can be called a target image; and while capturing the target image, this embodiment uses a ranging module to measure the distance between the hand in space and the optical center of the monocular camera, which can be called a first distance, in meters.

[0028] In some embodiments, a ranging module may be used to measure the distance between the wrist and the optical center of the monocular camera, and the distance may be used as the first distance; alternatively, a ranging module (such as radar ranging) may be used to measure the closest distance between the surface of the hand and the optical center of the monocular camera in space, and the closest distance may be used as the first distance, that is, the distance output by the ranging module is the first distance.

[0029] After the target image is captured by the monocular camera and the first distance is measured, the target image and the first distance can be transmitted to the computing unit. In this case, the computing unit can directly obtain the target image and the first distance, that is, the gesture degree of freedom position determination method of the embodiment of the present disclosure can be executed by the computing unit.

[0030] In some other embodiments, the target image may include multiple objects, such as hands, heads, and any possible objects in the imaging environment, and the objects are distributed in different areas of the target image. In this case, the present embodiment may use a distance measurement matrix device as a distance measurement module to obtain a distance matrix corresponding to the target image, and send the distance matrix to the calculation unit, that is, the output of the distance measurement module of the present embodiment is a distance matrix. Among them, the distance matrix corresponds to a plurality of image blocks divided by the target image, and the distance data of each position in the distance matrix represents the distance of the object contained in each image block from the optical center of the monocular camera. In this case, when the calculation unit obtains the first distance, it first obtains the distance matrix; further, the image block position (i.e., the hand position) of the hand in the target image is determined, and the target distance data corresponding to the hand position in the distance matrix is ​​determined as the first distance.

[0031] For example, the target image can be divided into 3*3 image blocks, then the distance matrix is ​​3*3, and the distance data at each position in the matrix respectively represents the distance between the object contained in the corresponding image block and the optical center of the monocular camera. Among them, the hand is in the middle position image block of the 3*3 image block, for example, the distance data at the middle position of the 3*3 distance matrix can be used as the target distance data, and the target distance data can be used as the first distance.

[0032] S102: Calculate a second distance between the hand position in the target image and the image center, and determine a first ratio between the second distance and the radius of the circumscribed circle of the target image.

[0033] Specifically, Figure 4 is a schematic diagram of the structure of the target image and the plane field of view of the monocular camera provided according to an embodiment of the present disclosure, such as Figure 4 As shown, the width of the target image is represented by W (pixels), and the height is represented by H (pixels), and W and H are known quantities. In addition, the disclosed embodiment can construct a plane coordinate system for the target image with the image center of the target image as the origin, the width direction as the X-axis, and the height direction as the Y-axis. The calculation unit of this embodiment can determine the position coordinates of the hand in the plane coordinate system, that is, the hand position, which can be represented as (x, y), and x and y are known quantities.

[0034] Among them, Figure 4 As shown, the distance between the hand position (x, y) in the target image and the image center can be called the second distance, which can be represented by r, that is,

[0035] like Figure 4 As shown, the radius of the circumscribed circle of the target image in this embodiment can be represented by R.

[0036] The ratio of the second distance r to the radius R of the circumscribed circle of the target image may be referred to as a first ratio, that is, r / R.

[0037] S103: Calculate an angle between a line connecting the hand and the optical center of the monocular camera and an optical axis of the monocular camera based on the field of view of the monocular camera and the first ratio.

[0038] like Figure 4 As shown, the field of view (FOV) of the monocular camera can be represented by θ (a known quantity), the width of the field of view plane of the monocular camera can be represented by W' (meters), and the height can be represented by H' (meters). In the embodiment of the present disclosure, a plane coordinate system can be constructed for the field of view plane with the center of the field of view plane as the origin, the width direction as the X-axis, and the height direction as the Y-axis. The position coordinates of the hand in the plane coordinate system of the field of view plane can be represented by (x', y'), where x' represents the lateral offset distance of the hand from the center of the field of view plane, in meters, and y' represents the longitudinal offset distance of the hand from the center of the field of view plane, in meters.

[0039] Among them, Figure 4 As shown, the distance between the hand position in the field of view plane and the center of the field of view can be represented by r'; and the radius of the circumscribed circle of the field of view plane can be represented by R', and according to the imaging principle, r / R=r' / R'.

[0040] In the embodiment of the present disclosure, the angle between the hand and the optical center of the monocular camera and the optical axis of the monocular camera can be calculated based on the field of view angle θ of the monocular camera and the first ratio r / R. The angle can be represented by θ'.

[0041] Specifically, Figure 4 As shown, according to the imaging principle, the optical axis (represented by z), the line connecting the optical center of the monocular camera and the hand (represented by z'), and the line r' connecting the center of the hand and the field of view plane form a right triangle, then tan(θ') = r' / z, tan(θ / 2) = R' / z.

[0042] tan(θ')=r' / z and tan(θ / 2)=R' / z, eliminating z, we can get:

[0043] θ'=arctan(r' / R'*tan(θ / 2))

[0044] According to the above imaging principle r / R=r' / R', we can get:

[0045]

[0046] S104: Calculate a lateral offset distance and a longitudinal offset distance of the hand from the center of the field of view plane based on the included angle, a third distance between the optical center of the monocular camera and the field of view plane, and a second ratio of the horizontal and vertical coordinates of the hand position.

[0047] Among them, the distance between the optical center of the monocular camera and the field of view plane can be called the third distance, that is, the third distance is the optical axis length Z.

[0048] Some embodiments, such as Figure 4 As shown, the calculation unit can calculate the radius R' of the circumscribed circle of the field of view plane according to the width W' and height H' (known quantities) of the field of view plane, that is: Furthermore, the third distance z is calculated according to the radius R' of the circumscribed circle of the viewing plane and the viewing angle θ', that is, tan(θ / 2)=R' / z, and z can be obtained in meters.

[0049] The second ratio is used to describe the ratio of the horizontal coordinate to the vertical coordinate of the hand position in the target image. The second ratio is x / y. According to the imaging principle, x / y=x' / y'.

[0050] After obtaining the angle θ' between the optical center line of the monocular camera and the optical axis of the monocular camera, the third distance z and the second ratio x / y, the lateral offset distance x' and the longitudinal offset distance y' can be calculated according to θ', z and x / y.

[0051] Specifically, Figure 4 As shown, we can get:

[0052] sinθ'=r' / z

[0053]

[0054] According to x'=x / y*y', we can get:

[0055]

[0056] Therefore, when θ', z, x and y are known, the actual values ​​of the lateral offset distance x' and the longitudinal offset distance y' can be calculated.

[0057] S105: Using the lateral offset distance, the longitudinal offset distance, and the first distance as the three-degree-of-freedom translation posture of the hand.

[0058] That is to say, in this embodiment, the lateral offset distance x', the longitudinal offset distance y' and the first distance z' are used as the 3Dof translation posture of the hand in the three-dimensional (3D) space coordinate system. It should be noted that the 3D space coordinate system can be called a second 3D space coordinate system, and its origin is, for example, the position of the ranging module.

[0059] In this embodiment, a target image including a hand is obtained, and a first distance between the hand and the optical center of the monocular camera is obtained, and a second distance between the hand position in the target image and the image center is calculated, and a first ratio of the second distance to the radius of the circumscribed circle of the target image is determined, based on the field of view angle of the monocular camera and the first ratio, the angle between the line connecting the hand and the optical center of the monocular camera and the optical axis of the monocular camera is calculated, based on the angle, the third distance between the optical center of the monocular camera and the field of view plane, and the second ratio of the horizontal and vertical coordinates of the hand position, the lateral offset distance and the longitudinal offset distance between the hand and the center of the field of view plane are calculated, and the lateral offset distance, the longitudinal offset distance and the first distance are used as the three-degree-of-freedom translation posture of the hand, the proportional relationship between the field of view and the image during imaging can be utilized, and the three-degree-of-freedom translation posture of the hand can be determined according to the monocular image and the distance between the hand and the monocular camera light. Compared with the existing multi-eye image translation posture calculation, the calculation process of this scheme is simple, so the complexity of the translation posture calculation can be reduced.

[0060] Figure 5 is a flow chart of a method for determining the degree of freedom of gesture according to another embodiment of the present disclosure, such as Figure 5 As shown, the method includes:

[0061] S501: Acquire a target image including a hand, and acquire a first distance between the hand and the optical center of a monocular camera.

[0062] S502: Calculate a second distance between the hand position in the target image and the image center, and determine a first ratio of the second distance to the radius of the circumscribed circle of the target image.

[0063] S503: Calculate an angle between a line connecting the hand and the optical center of the monocular camera and an optical axis of the monocular camera based on the field of view of the monocular camera and the first ratio.

[0064] S504: Calculate a lateral offset distance and a longitudinal offset distance of the hand from the center of the field of view plane based on the included angle, a third distance between the optical center of the monocular camera and the field of view plane, and a second ratio of the horizontal and vertical coordinates of the hand position.

[0065] S505: Using the lateral offset distance, the longitudinal offset distance, and the first distance as the three-degree-of-freedom translation posture of the hand.

[0066] The specific description of S501-S505 refers to the above embodiment and will not be repeated here.

[0067] S506: Calculate the target image using a preset neural network model to determine the three-dimensional coordinates of multiple bone points of the hand.

[0068] In the disclosed embodiment, after acquiring the target image, the computing unit may also use a preset neural network model to compute the target image to determine the three-dimensional coordinates of multiple bone points of the hand, that is, the target image is input into the neural network model, and the three-dimensional coordinates of multiple bone points are output. The neural network model may be any network model for extracting three-dimensional coordinates from a single frame image, and there is no limitation on this.

[0069] For example, Figure 6 is a schematic diagram of the structure of multiple bone points of the hand provided according to an embodiment of the present disclosure, such as Figure 6 As shown, the multiple bone points of the hand in this embodiment include bone points 0-10, and the neural network model outputs the three-dimensional coordinates corresponding to each bone point. It should be noted that the three-dimensional (3D) space coordinate system corresponding to the three-dimensional coordinates of the bone points can be called a first 3D space coordinate system, and the origin of the first 3D space coordinate system is, for example, the wrist position.

[0070] S507: Calculate the yaw angle, pitch angle, and roll angle of the hand based on the three-dimensional coordinates of the multiple skeleton points, wherein the yaw angle, pitch angle, and roll angle are used as a three-degree-of-freedom rotation posture.

[0071] Figure 7 is a schematic diagram of the structure of the pitch angle, yaw angle and roll angle provided according to an embodiment of the present disclosure, such as Figure 7 As shown, this embodiment can further calculate the yaw angle (Yaw), pitch angle (Roll) and roll angle (Pitch) of the hand based on the three-dimensional coordinates of multiple bone points, and use the yaw angle (Yaw), pitch angle (Roll) and roll angle (Pitch) as the 3Dof rotation posture of the hand in the first 3D space coordinate system.

[0072] Specifically, Figure 7 As shown, in the operation of determining the yaw angle (Yaw) and the pitch angle (Pitch), this embodiment can determine the first skeleton point and the second skeleton point representing the horizontal direction of the hand from multiple skeleton points, such as Figure 6 As shown, for example, skeleton point No. 2 and skeleton point No. 5 are selected as the first skeleton point and the second skeleton point, and there is no limitation to this.

[0073] Further, based on the three-dimensional coordinates of the first skeleton point (skeleton point No. 2) and the second skeleton point (skeleton point No. 5), the projection of the line connecting the first skeleton point and the second skeleton point on the XOZ plane is determined, and the projection can be called the first projection; further, the angle θ1 between the first projection and the Z axis is determined as the yaw angle Yaw; in the process of determining the roll angle Pitch, this embodiment determines the projection of the line connecting the first skeleton point and the second skeleton point on the XOY plane, and the projection can be called the second projection; further, the angle θ3 between the second projection and the Y axis is determined as the roll angle Pitch.

[0074] Some other embodiments, such as Figure 7 As shown, in the operation of determining the pitch angle (Roll), this embodiment can determine the third skeleton point and the fourth skeleton point representing the vertical direction of the hand from multiple skeleton points, such as Figure 6 As shown, for example, skeletal point No. 0 and skeletal point No. 3 are selected as the third skeletal point and the fourth skeletal point respectively, without limitation; further, based on the three-dimensional coordinates of the third skeletal point (for example, skeletal point No. 0) and the fourth skeletal point (skeletal point No. 3), the projection of the line connecting the third skeletal point and the fourth skeletal point on the XOZ plane is calculated, and the projection can be called the third projection; further, the angle θ2 between the third projection and the line connecting the third skeletal point and the fourth skeletal point is determined as the pitch angle Roll.

[0075] In some embodiments, after the yaw angle θ1, the pitch angle θ2, and the roll angle θ3 are determined, Kalman filtering may be performed on the yaw angle θ1, the pitch angle θ2, and the roll angle θ3.

[0076] Specifically, this embodiment can define an initial state covariance matrix P, a state transfer matrix F, a state transfer covariance matrix Q, an observation matrix H, and an observation noise covariance R to perform Kalman filtering on the yaw angle θ1, the pitch angle θ2, and the roll angle θ3, and the filtered yaw angle θ1, the pitch angle θ2, and the roll angle θ3 are used as the 3Dof rotation posture. Among them, this embodiment does not specifically limit the specific process of Kalman filtering.

[0077] It should be noted that S506 and S507 may be executed after S501 , and this embodiment does not impose any specific limitation on the actual execution order of S506 and S507 .

[0078] S508: Using the three-degree-of-freedom translation posture and the three-degree-of-freedom rotation posture as the six-degree-of-freedom posture of the hand.

[0079] That is to say, this embodiment can combine the 3Dof translation posture (x', y', z') with the 3Dof rotation posture (yaw angle Yaw, pitch angle Roll, roll angle Pitch) to determine the 6Dof posture (x', y', z', Yaw, Roll, Pitch) of the hand in three-dimensional space. Among them, since the 3Dof translation posture is the posture in the second 3D space coordinate system, and the 3Dof rotation posture is the posture in the first 3D space coordinate system, the 6Dof posture of the embodiment of the present disclosure is the 6Dof posture of the first 3D space coordinate system relative to the second 3D space coordinate system.

[0080] Therefore, the embodiment of the present disclosure can also determine the three-degree-of-freedom rotational posture of the hand based on the monocular image, and can also determine the six-degree-of-freedom posture of the hand.

[0081] In this embodiment, a target image including a hand is obtained, and a first distance between the hand and the optical center of the monocular camera is obtained, and a second distance between the hand position in the target image and the image center is calculated, and a first ratio of the second distance to the radius of the circumscribed circle of the target image is determined, based on the field of view angle of the monocular camera and the first ratio, the angle between the line connecting the hand and the optical center of the monocular camera and the optical axis of the monocular camera is calculated, based on the angle, the third distance between the optical center of the monocular camera and the field of view plane, and the second ratio of the horizontal and vertical coordinates of the hand position, the lateral offset distance and the longitudinal offset distance between the hand and the center of the field of view plane are calculated, and the lateral offset distance, the longitudinal offset distance and the first distance are used as the three-degree-of-freedom translation posture of the hand, the proportional relationship between the field of view and the image during imaging can be utilized, and the three-degree-of-freedom translation posture of the hand can be determined according to the monocular image and the distance between the hand and the monocular camera light. Compared with the existing multi-eye image translation posture calculation, the calculation process of this scheme is simple, so the complexity of the translation posture calculation can be reduced. Furthermore, the embodiments of the present disclosure can also determine the three-degree-of-freedom rotational posture of the hand based on the monocular image, and can also determine the six-degree-of-freedom posture of the hand.

[0082] In order to implement the above embodiments, the present disclosure also proposes a gesture degree of freedom position determination device.

[0083] Figure 8 It is a schematic diagram of a gesture degree of freedom position determination device provided according to another embodiment of the present disclosure.

[0084] like Figure 8 As shown, the gesture freedom degree posture determination device 80 includes:

[0085] An acquisition module 801 is used to acquire a target image including a hand, and acquire a first distance between the hand and the optical center of the monocular camera;

[0086] A first calculation module 802, used to calculate a second distance between the hand position in the target image and the image center, and determine a first ratio of the second distance to the radius of the circumscribed circle of the target image;

[0087] A second calculation module 803 is used to calculate the angle between the hand and the optical center of the monocular camera and the optical axis of the monocular camera based on the field of view of the monocular camera and the first ratio;

[0088] A third calculation module 804 is used to calculate the lateral offset distance and the longitudinal offset distance of the hand from the center of the field of view plane based on the angle, the third distance between the optical center of the monocular camera and the field of view plane, and the second ratio of the horizontal and vertical coordinates of the hand position; and

[0089] The first determination module 805 is used to use the lateral offset distance, the longitudinal offset distance and the first distance as the three-degree-of-freedom translation posture of the hand.

[0090] In some embodiments, the third calculation module 804 is specifically used to: calculate the radius of the circumscribed circle of the field of view plane according to the width and height of the field of view plane; and calculate the third distance according to the radius of the circumscribed circle of the field of view plane and the field of view angle.

[0091] In some embodiments, the device 80 also includes: a fourth calculation module, used to calculate the target image using a preset neural network model to determine the three-dimensional coordinates of multiple bone points of the hand; and a second determination module, used to calculate the yaw angle, pitch angle and roll angle of the hand based on the three-dimensional coordinates of the multiple bone points, wherein the yaw angle, pitch angle and roll angle are used as three-degree-of-freedom rotation posture; and the device also includes: a third determination module, used to use the three-degree-of-freedom translation posture and the three-degree-of-freedom rotation posture as the six-degree-of-freedom posture of the hand.

[0092] In some embodiments, the second determination module is specifically used to: determine a first skeletal point and a second skeletal point representing the horizontal direction of the hand from a plurality of skeletal points; based on the three-dimensional coordinates of the first skeletal point and the second skeletal point, determine an angle between a first projection of a line connecting the first skeletal point and the second skeletal point on the XOZ plane and the Z axis as the yaw angle, and calculate an angle between a second projection of a line connecting the first skeletal point and the second skeletal point on the XOY plane and the Y axis as the roll angle.

[0093] In some embodiments, the second determination module is specifically used to: determine a third skeletal point and a fourth skeletal point representing the vertical direction of the hand from multiple skeletal points; calculate a third projection of the line connecting the third skeletal point and the fourth skeletal point on the XOZ plane based on the three-dimensional coordinates of the third skeletal point and the fourth skeletal point; and use the angle between the third projection and the line connecting the third skeletal point and the fourth skeletal point as the pitch angle.

[0094] In some embodiments, the device 80 further includes: a filtering module, configured to perform Kalman filtering on the yaw angle, the pitch angle, and the roll angle.

[0095] In some embodiments, the acquisition module 801 is specifically used to: determine the shortest distance between the surface of the hand and the optical center of the monocular camera as the first distance; or obtain a distance matrix corresponding to the target image, and determine the target distance corresponding to the hand position in the distance matrix as the first distance.

[0096] In this embodiment, a target image including a hand is obtained, and a first distance between the hand and the optical center of the monocular camera is obtained, and a second distance between the hand position in the target image and the image center is calculated, and a first ratio of the second distance to the radius of the circumscribed circle of the target image is determined, based on the field of view angle of the monocular camera and the first ratio, the angle between the line connecting the hand and the optical center of the monocular camera and the optical axis of the monocular camera is calculated, based on the angle, the third distance between the optical center of the monocular camera and the field of view plane, and the second ratio of the horizontal and vertical coordinates of the hand position, the lateral offset distance and the longitudinal offset distance between the hand and the center of the field of view plane are calculated, and the lateral offset distance, the longitudinal offset distance and the first distance are used as the three-degree-of-freedom translation posture of the hand, the proportional relationship between the field of view and the image during imaging can be utilized, and the three-degree-of-freedom translation posture of the hand can be determined according to the monocular image and the distance between the hand and the monocular camera light. Compared with the existing multi-eye image translation posture calculation, the calculation process of this scheme is simple, so the complexity of the translation posture calculation can be reduced.

[0097] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, a virtual reality device, and a computer program product.

[0098] In order to implement the above embodiments, the present disclosure further proposes a computer program product. When an instruction processor in the computer program product is executed, the method for determining the degree of freedom of gesture proposed in the above embodiments of the present disclosure is executed.

[0099] Fig. 9 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Fig. 9 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0100] like Fig. 9 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0101] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0102] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0103] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Fig. 9 Not shown, often called a "hard drive").

[0104] although Fig. 9 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present disclosure.

[0105] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0106] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0107] The processing unit 16 executes various functional applications by running programs stored in the system memory 28, such as implementing the gesture degree of freedom position determination method mentioned in the above embodiment.

[0108] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0109] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0110] It should be noted that, in the description of the present disclosure, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "plurality" is two or more.

[0111] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0112] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0113] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0114] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0115] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0116] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0117] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. A method for determining the degree of freedom of gestures, It is characterized in that The method comprises: Acquire a target image including a hand, and acquire a first distance between the hand and the optical center of the monocular camera; Calculating a second distance between the hand position in the target image and the image center, and determining a first ratio of the second distance to the radius of the circumscribed circle of the target image; Based on the field of view angle of the monocular camera and the first ratio, calculating the angle between the line connecting the hand and the optical center of the monocular camera and the optical axis of the monocular camera; Calculate the lateral offset distance and the longitudinal offset distance of the hand from the center of the field of view plane based on the angle, the third distance between the optical center of the monocular camera and the field of view plane, and the second ratio of the horizontal and vertical coordinates of the hand position; and The lateral offset distance, the longitudinal offset distance and the first distance are used as the three-degree-of-freedom translation posture of the hand.

2. The method according to claim 1, It is characterized in that Before calculating the lateral offset distance and the longitudinal offset distance of the hand from the center of the field of view plane, the method further includes: Calculating the radius of the circumscribed circle of the field of view plane according to the width and height of the field of view plane; and The third distance is calculated according to the radius of the circumscribed circle of the field of view plane and the field of view angle.

3. The method according to claim 1, It is characterized in that The method further comprises: Calculating the target image using a preset neural network model to determine the three-dimensional coordinates of the plurality of bone points of the hand; and Calculating the yaw angle, pitch angle and roll angle of the hand based on the three-dimensional coordinates of the multiple skeleton points, wherein the yaw angle, the pitch angle and the roll angle are used as a three-degree-of-freedom rotation posture; Furthermore, after taking the lateral offset distance, the longitudinal offset distance and the first distance as the three-degree-of-freedom translation posture of the hand, the method further includes: The three-degree-of-freedom translation posture and the three-degree-of-freedom rotation posture are used as the six-degree-of-freedom posture of the hand.

4. The method according to claim 3, It is characterized in that The calculating the yaw angle, the pitch angle and the roll angle of the hand based on the three-dimensional coordinates of the multiple skeleton points includes: Determine a first skeleton point and a second skeleton point representing the horizontal direction of the hand from the multiple skeleton points; Based on the three-dimensional coordinates of the first bone point and the second bone point, the angle between the first projection of the line connecting the first bone point and the second bone point on the XOZ plane and the Z axis is calculated as the yaw angle, and the angle between the second projection of the line connecting the first bone point and the second bone point on the XOY plane and the Y axis is calculated as the roll angle.

5. The method according to claim 3, It is characterized in that The calculating the yaw angle, the pitch angle and the roll angle of the hand based on the three-dimensional coordinates of the multiple skeleton points includes: Determine a third skeleton point and a fourth skeleton point representing the vertical direction of the hand from the multiple skeleton points; Calculate a third projection of the line connecting the third skeletal point and the fourth skeletal point on the XOZ plane based on the three-dimensional coordinates of the third skeletal point and the fourth skeletal point; and The angle between the third projection and the line connecting the third skeleton point and the fourth skeleton point is used as the pitch angle.

6. The method according to claim 3, It is characterized in that After determining the yaw angle, pitch angle and roll angle of the hand, the method further includes: Kalman filtering is performed on the yaw angle, the pitch angle, and the roll angle.

7. The method according to claim 1, It is characterized in that The obtaining of a first distance between the hand and the optical center of the monocular camera includes: Determine the shortest distance between the surface of the hand and the optical center of the monocular camera as the first distance; or A distance matrix corresponding to the target image is obtained, and target distance data corresponding to the hand position in the distance matrix is ​​determined as a first distance.

8. A gesture freedom position determination device, It is characterized in that include: An acquisition module, used to acquire a target image including a hand, and acquire a first distance between the hand and the optical center of the monocular camera; A first calculation module, used for calculating a second distance between the hand position in the target image and the image center, and determining a first ratio of the second distance to the radius of the circumscribed circle of the target image; A second calculation module, configured to calculate an angle between a line connecting the hand and the optical center of the monocular camera and an optical axis of the monocular camera based on the field of view of the monocular camera and the first ratio; A third calculation module is used to calculate the lateral offset distance and the longitudinal offset distance of the hand from the center of the field of view plane based on the angle, the third distance between the optical center of the monocular camera and the field of view plane, and the second ratio of the horizontal and vertical coordinates of the hand position; as well as The first determination module is used to use the lateral offset distance, the longitudinal offset distance and the first distance as the three-degree-of-freedom translation posture of the hand.

9. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

11. A virtual reality device, It is characterized in that The virtual reality device includes the electronic device as claimed in claim 9.