Method and device for determining three-dimensional posture of gesture, equipment and medium

By obtaining the initial and historical gestures of gestures, combining constraints, and using a nonlinear optimization algorithm to determine the three-dimensional gestures of gestures, the accuracy and reliability problems of gesture gesture determination in the prior art are solved, and efficient posture optimization and cost reduction are achieved.

CN119002695APending Publication Date: 2024-11-22GEER TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411029001.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-22

Smart Images

  • Figure CN119002695A_ABST
    Figure CN119002695A_ABST
Patent Text Reader

Abstract

The invention discloses a method and device for determining a three-dimensional posture of a gesture, equipment and a medium, and relates to the field of data processing, and the method comprises the steps: obtaining a current initial three-dimensional posture of the gesture in a current frame image and a historical target three-dimensional posture of the gesture in a previous frame image; determining a current target three-dimensional attitude of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional attitude and the historical target three-dimensional attitude; wherein the gesture constraint conditions comprise at least two of a time constraint condition, a space constraint condition and a shape constraint condition, and the time constraint condition is a first maximum deviation value between a current target three-dimensional attitude and a historical target three-dimensional attitude; the spatial constraint condition is a second maximum deviation value between a two-dimensional coordinate obtained by projecting a current target three-dimensional attitude and a two-dimensional coordinate of a gesture in a current frame image, and the shape constraint condition is a third maximum deviation value between the current target three-dimensional attitude and a gesture shape corresponding to a historical target three-dimensional attitude.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing, and more specifically, to a method, device, equipment and medium for determining a three-dimensional posture of a hand gesture. Background Art

[0002] With the development of science and technology and economy, XR head-mounted display devices such as AR, VR, and MR are being used more and more widely. As an intuitive, efficient and natural way of interaction, gestures have become an important way for users to interact with virtual scenes presented by XR head-mounted display devices. For example, gestures can be used to simulate user interface operations such as mouse and keyboard, and can also be used to control virtual objects, such as grabbing virtual objects. Therefore, determining the three-dimensional posture of the user's gesture has become a key technology for users to interact with virtual scenes presented by XR head-mounted display devices.

[0003] At present, deep learning models are usually used to predict the three-dimensional posture of user gestures. However, the prediction results of deep learning models themselves have certain unreliability. In actual applications, user gestures are diverse and easy to change. The hands have high degrees of freedom and there are self-occlusion and occlusion phenomena of both hands. These problems make it difficult to guarantee the accuracy and reliability of the three-dimensional posture of gestures predicted by deep learning models. In order to improve the accuracy of deep learning model predictions, it is often necessary to collect and annotate more and more accurate training data sets for model training, and continuously optimize the model structure to improve the model accuracy. This results in more manpower and time required to train and optimize deep learning models, and the final result is still difficult to guarantee the accuracy and reliability of the three-dimensional posture of gestures of different hand shapes in multiple different scenarios.

[0004] Therefore, how to provide a method for determining the three-dimensional posture of a user's hand gesture with low manpower and time cost and high accuracy and reliability has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] One object of the present application is to provide a new technical solution for determining the three-dimensional posture of a hand gesture.

[0006] According to a first aspect of the present application, a method for determining a three-dimensional posture of a gesture is provided, comprising:

[0007] Obtaining the current initial 3D posture of the gesture in the current frame image and the historical target 3D posture of the gesture in the previous frame image;

[0008] Determining a current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture;

[0009] Among them, the gesture constraint condition includes at least two of the time constraint condition, the space constraint condition and the shape constraint condition. The time constraint condition is the first maximum deviation value between the current target three-dimensional posture and the historical target three-dimensional posture. The space constraint condition is the second maximum deviation value between the two-dimensional coordinates obtained by projection of the current target three-dimensional posture and the two-dimensional coordinates of the gesture in the current frame image. The shape constraint condition is the third maximum deviation value between the gesture shapes corresponding to the current target three-dimensional posture and the historical target three-dimensional posture.

[0010] Optionally, after acquiring the current initial three-dimensional posture of the gesture in the current frame image, the method further includes:

[0011] Perform filtering processing on the current initial three-dimensional posture to obtain the filtered current initial three-dimensional posture, and update the current initial three-dimensional posture to the filtered current initial three-dimensional posture.

[0012] Optionally, before determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture, the method further includes:

[0013] According to the current initial three-dimensional posture, determining whether the current gesture shape satisfies the normal gesture shape;

[0014] If the condition is not satisfied, the step of determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture is triggered.

[0015] Optionally, the method further comprises:

[0016] In the case where it is determined that the current gesture shape satisfies a normal gesture shape, determining a current gesture movement amount according to the current initial three-dimensional posture and the historical target three-dimensional posture;

[0017] When the current gesture movement amount is greater than the preset movement amount, triggering the step of determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture;

[0018] When the current gesture movement amount is less than or equal to the preset movement amount, the current initial three-dimensional posture is used as the current target three-dimensional posture.

[0019] Optionally, determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture includes:

[0020] Determine the target loss value according to the gesture constraints;

[0021] Input the current initial 3D posture and the historical target 3D posture into the LM algorithm using the ceres nonlinear optimization library to obtain the current predicted 3D posture;

[0022] Determining a current loss value according to the current predicted three-dimensional posture and the historical target three-dimensional posture;

[0023] According to the current loss value and the target loss value, adjusting the parameters of the LM algorithm using the ceres nonlinear optimization library, and repeating the step of inputting the current initial three-dimensional posture and the historical target three-dimensional posture into the LM algorithm using the ceres nonlinear optimization library to obtain the current predicted three-dimensional posture, until a preset convergence condition is reached;

[0024] The current predicted three-dimensional pose that meets the convergence condition is used as the current target three-dimensional pose.

[0025] Optionally, the preset convergence condition is that after repeating the step of inputting the current initial three-dimensional posture and the historical target three-dimensional posture into the LM algorithm to obtain the current predicted three-dimensional posture for a preset number of times, the current gesture movement amount corresponding to the current predicted three-dimensional posture relative to the historical target three-dimensional posture is less than or equal to the preset movement amount.

[0026] Optionally, the method further comprises:

[0027] In the absence of the historical target three-dimensional posture, determining whether the current gesture shape satisfies a normal gesture shape according to the current initial three-dimensional posture;

[0028] If the above conditions are met, the current initial three-dimensional posture of the gesture in the current frame image is used as the current target three-dimensional posture of the gesture in the current frame image.

[0029] According to a second aspect of the present application, a device for determining a three-dimensional posture of a gesture is provided, comprising:

[0030] An acquisition module, used to acquire the current initial three-dimensional posture of the gesture in the current frame image and the historical target three-dimensional posture of the gesture in the previous frame image;

[0031] A determination module, configured to determine a current target three-dimensional posture of the gesture in the current frame image according to a gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture;

[0032] Among them, the gesture constraint condition includes at least two of the time constraint condition, the space constraint condition and the shape constraint condition. The time constraint condition is the first maximum deviation value between the current target three-dimensional posture and the historical target three-dimensional posture. The space constraint condition is the second maximum deviation value between the two-dimensional coordinates obtained by projection of the current target three-dimensional posture and the two-dimensional coordinates of the gesture in the current frame image. The shape constraint condition is the third maximum deviation value between the gesture shapes corresponding to the current target three-dimensional posture and the historical target three-dimensional posture.

[0033] According to a third aspect of the present application, an electronic device is provided, the electronic device comprising the device for determining the three-dimensional posture of the gesture as described in the second aspect; or,

[0034] The electronic device includes a memory and a processor, the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the method for determining the three-dimensional posture of a gesture as described in any one of the first aspects.

[0035] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for determining the three-dimensional posture of a gesture according to any one of the first aspects is implemented.

[0036] The present application provides a method for determining the three-dimensional posture of a gesture, the method comprising: obtaining the current initial three-dimensional posture of the gesture in the current frame image and the historical target three-dimensional posture of the gesture in the previous frame image; determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture; wherein the gesture constraint condition includes at least two of the time constraint condition, the space constraint condition and the shape constraint condition, the time constraint condition is the first maximum deviation value between the current target three-dimensional posture and the historical target three-dimensional posture, the space constraint condition is the second maximum deviation value between the two-dimensional coordinates obtained by projecting the current target three-dimensional posture and the two-dimensional coordinates of the gesture in the current frame image, and the shape constraint condition is the third maximum deviation value between the current target three-dimensional posture and the gesture shape corresponding to the historical target three-dimensional posture. The method for determining the three-dimensional posture of the gesture provided by the present application optimizes the current initial three-dimensional posture of the gesture in the current frame image through various dimensions, so that a more accurate three-dimensional posture can be obtained, that is, the present application provides a method for determining the three-dimensional posture of the user's gesture with high accuracy and high reliability. In addition, the method for determining the three-dimensional posture of the gesture provided by the present application does not involve a deep learning model, so there is no need to collect and label samples, which can greatly reduce manpower and time costs.

[0037] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments of the present application with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0039] Figure 1 A hardware configuration of an electronic device for implementing a method for determining a three-dimensional posture of a gesture provided by an embodiment of the present application is provided. Figure 1 ;

[0040] Figure 2 The following is a flow chart of a method for determining a three-dimensional posture of a gesture according to an embodiment of the present application. Figure 1 ;

[0041] Figure 3 is a schematic diagram of a gesture provided according to an embodiment of the present application;

[0042] Figure 4 The following is a flow chart of a method for determining a three-dimensional posture of a gesture according to an embodiment of the present application. Figure 2 ;

[0043] Figure 5 The structure of a device for determining a three-dimensional posture of a gesture provided in accordance with an embodiment of the present application is schematically shown. Figure 1 ;

[0044] Figure 6 A hardware configuration of an electronic device for implementing a method for determining a three-dimensional posture of a gesture provided by an embodiment of the present application is provided. Figure 2 . DETAILED DESCRIPTION

[0045] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0046] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0047] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0048] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0049] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0050] Figure 1 A hardware configuration of an electronic device for implementing a method for determining a three-dimensional posture of a gesture provided by an embodiment of the present application is provided. Figure 1 .

[0051] The electronic device 1000 may be a terminal or a server. Further, the terminal may be a head mounted device (such as an AR device, an MR device, and a VR device), a portable computer, a tablet computer, a PDA, etc. The server may be a cloud server, etc.

[0052] The electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, an image sensor 1900, and the like. The processor 1100 may be a central processing unit CPU, a microprocessor MCU, and the like. The memory 1200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, and the like. The interface device 1300 may include, for example, a USB interface, an earphone interface, and the like. The communication device 1400 may, for example, be capable of wired or wireless communication. The display device 1500 may be, for example, a liquid crystal display screen, a touch display screen, and the like. The input device 1600 may include, for example, a touch screen, a keyboard, and the like. The user may input / output voice information through the speaker 1700 and the microphone 1800. The image sensor 1900 is used to capture images of the user's hands.

[0053] Despite Figure 1 Multiple devices are shown for the electronic device 1000, but the present application may only involve some of the devices, for example, the electronic device 1000 only involves the memory 1200, the processor 1100 and the image sensor 1900.

[0054] In the embodiment of the present application, the memory 1200 of the electronic device 1000 is used to store instructions, and the instructions are used to control the processor 1100 to execute the method for determining the three-dimensional posture of the gesture provided in the embodiment of the present application.

[0055] In the above description, the technician can design instructions according to the scheme disclosed in this application. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0056] The present application provides a method for determining the three-dimensional posture of a hand gesture, the method being as follows Figure 2As shown, the following steps S2100 and S2200 are included.

[0057] Step S2100, obtaining the current initial three-dimensional posture of the gesture in the current frame image and the historical target three-dimensional posture of the gesture in the previous frame image.

[0058] In this embodiment, the current initial three-dimensional posture of the gesture in the current frame image is the three-dimensional posture of the gesture in the current frame image obtained by using traditional technology, wherein the three-dimensional posture of the gesture can be exemplarily as follows: Figure 3 As shown, through multiple gesture feature point descriptions, a gesture feature point includes corresponding three-dimensional coordinates and three-dimensional rotation angles, and the traditional technology can be specifically a deep learning model. Based on this, the acquisition of the current initial three-dimensional posture of the gesture in the current frame image in the above step S2100 is specifically implemented through the following steps S2110 and S2111.

[0059] Step S2110, obtaining the current frame image.

[0060] Step S2111, determining a current initial three-dimensional posture of the gesture in the current frame image according to the current frame image and a preset mapping relationship;

[0061] The preset mapping relationship is data reflecting the corresponding relationship between the image and the three-dimensional posture of the gesture in the image, or data reflecting the corresponding relationship between the image and the two-dimensional coordinates of the gesture in the image.

[0062] In one embodiment of the present application, the current frame image includes depth information. The preset mapping relationship is data in the traditional technology that reflects the correspondence between the image and the two-dimensional coordinates of the gesture in the image, and an exemplary first deep learning model is used to obtain the two-dimensional coordinates of the gesture in the image based on the image. When the two-dimensional coordinates of the gesture in the current frame image are obtained based on the first deep learning model, the three-dimensional posture of the gesture in the current frame image is obtained in combination with the depth information corresponding to the two-dimensional coordinates of the gesture.

[0063] In another embodiment of the present application, the preset mapping relationship is data in traditional technology that reflects the correspondence between an image and the three-dimensional posture of the gesture in the image. Exemplarily, a second deep learning model of the three-dimensional posture of the gesture in the image is obtained based on the image.

[0064] It should be noted that, since the preset mapping relationship is relevant data in traditional technology, the method for obtaining the preset mapping relationship will not be described in detail in this application, and the specific acquisition method is not limited.

[0065] Based on the above content, it can be understood that the method for determining the three-dimensional posture of a gesture provided by the present application is a method for optimizing the three-dimensional posture of a gesture in a current frame image obtained by using traditional techniques to obtain a more accurate three-dimensional posture of the gesture in the current frame image. Among them, in the present application, the three-dimensional posture of the gesture in the current frame image obtained by using traditional techniques is optimized to obtain a more accurate three-dimensional posture of the gesture in the current frame image, which is recorded as the current target three-dimensional posture of the gesture in the current frame image.

[0066] The historical target three-dimensional posture of the gesture in the previous frame image is specifically the current target three-dimensional posture of the gesture in the current frame image obtained when the method for determining the three-dimensional posture of the gesture provided in this application is executed on the previous frame image when the previous frame image is used as the current frame image.

[0067] Step S2200, determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture.

[0068] Among them, the gesture constraint conditions include at least two of the time constraint conditions, the space constraint conditions and the shape constraint conditions. The time constraint condition is the first maximum deviation value between the current target three-dimensional posture and the historical target three-dimensional posture. The space constraint condition is the second maximum deviation value between the two-dimensional coordinates of the current target three-dimensional posture obtained by projection and the two-dimensional coordinates of the gesture in the current frame image. The shape constraint condition is the third maximum deviation value between the gesture shapes corresponding to the current target three-dimensional posture and the historical target three-dimensional posture.

[0069] In the present application, since the current frame image and the previous frame image have time continuity, and the corresponding sampling time interval between the two is short, the current target three-dimensional posture is close to the historical target three-dimensional posture, the gesture shape corresponding to the current target three-dimensional posture is close to the gesture shape corresponding to the historical target three-dimensional posture, and the two-dimensional image coordinates of the current target three-dimensional posture obtained by forward projection according to the camera projection model are close to the two-dimensional image coordinates of the gesture in the current frame image. On this basis, under normal gesture conditions, based on experience or experiments, the maximum deviation value between the current target three-dimensional posture and the historical target three-dimensional posture is determined to be recorded as the first maximum deviation value, the maximum deviation value between the current target three-dimensional posture and the gesture shape corresponding to the historical target three-dimensional posture is determined to be recorded as the third maximum deviation value, and the maximum deviation value between the two-dimensional coordinates obtained by projection of the current target three-dimensional posture and the two-dimensional coordinates of the gesture in the current frame image is determined to be recorded as the second maximum deviation value. Among them, it can be understood that the first maximum deviation value can be associated with the hand speed or hand acceleration corresponding to the current frame image and the previous frame image. Specifically, the greater the hand speed or hand acceleration corresponding to the current frame image and the previous frame image, the greater the value of the first maximum deviation value.

[0070] Further, taking the current initial three-dimensional posture as the initial value, the current initial three-dimensional posture is optimized according to the gesture constraint conditions consisting of at least two of the time constraint conditions corresponding to the first maximum deviation value, the space constraint conditions corresponding to the second maximum deviation value, and the shape constraint conditions corresponding to the third maximum deviation value, so as to obtain a more accurate three-dimensional posture of the gesture in the current frame image, which is recorded as the current target three-dimensional posture of the gesture in the current frame image. That is to say, the method for determining the three-dimensional posture of the gesture provided in the present application optimizes the current initial three-dimensional posture of the gesture in the current frame image in various dimensions, so that a more accurate three-dimensional posture can be obtained, that is, the present application provides a method for determining the three-dimensional posture of the user's gesture with high accuracy and high reliability. In addition, the method for determining the three-dimensional posture of the gesture provided in the present application does not involve a deep learning model, so there is no need for sample collection and annotation, which can greatly reduce manpower and time costs.

[0071] Based on the above content, the present application provides a method for determining the three-dimensional posture of a gesture, the method comprising: obtaining the current initial three-dimensional posture of the gesture in the current frame image and the historical target three-dimensional posture of the gesture in the previous frame image; determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture; wherein the gesture constraint condition includes at least two of the time constraint condition, the space constraint condition and the shape constraint condition, the time constraint condition is the first maximum deviation value between the current target three-dimensional posture and the historical target three-dimensional posture, the space constraint condition is the second maximum deviation value between the two-dimensional coordinates obtained by projecting the current target three-dimensional posture and the two-dimensional coordinates of the gesture in the current frame image, and the shape constraint condition is the third maximum deviation value between the current target three-dimensional posture and the gesture shape corresponding to the historical target three-dimensional posture. The method for determining the three-dimensional posture of a gesture provided by the present application optimizes the current initial three-dimensional posture of the gesture in the current frame image by various dimensions, so that a more accurate three-dimensional posture can be obtained, that is, the present application provides a method for determining the three-dimensional posture of a user's gesture with high accuracy and high reliability. In addition, the method for determining the three-dimensional posture of a gesture provided in the present application does not involve a deep learning model, so there is no need for sample collection and labeling, which can greatly reduce manpower and time costs.

[0072] In one embodiment of the present application, the method for determining the three-dimensional posture of a gesture provided by the present application further includes the following step S2120 after obtaining the current initial three-dimensional posture of the gesture in the current frame image in the above step S2100.

[0073] Step S2120, performing filtering processing on the current initial three-dimensional posture to obtain the current initial three-dimensional posture after filtering, and updating the current initial three-dimensional posture to the current initial three-dimensional posture after filtering.

[0074] In one embodiment of the present application, filtering of the current initial three-dimensional gesture may be implemented according to the OneEuroFilter filtering algorithm. The OneEuroFilter filtering algorithm may improve the accuracy of the current initial three-dimensional gesture by adjusting the jitter and delay of the gesture.

[0075] It can be further understood that when the hand moves at a low speed, the human eye is more sensitive to the jitter of the gesture, and when the hand moves at a high speed, the human eye is more sensitive to the delay of the gesture. Therefore, in an embodiment of the present application, the method for determining the three-dimensional posture of the gesture provided by the present application may also include, before the above step S2120: estimating the hand speed based on the current initial three-dimensional posture and the historical target three-dimensional posture; adjusting the minimum cutoff frequency and speed coefficient corresponding to the OneEuroFilter filtering algorithm based on the estimated hand speed. On this basis, the above step S2120 can balance the jitter and delay of the gesture in the current frame image.

[0076] In one embodiment of the present application, the method for determining the three-dimensional posture of a gesture provided by the present application further includes the following steps S2210 and S2211 before the above-mentioned step S2200.

[0077] Step S2210: Determine whether the current gesture shape satisfies a normal gesture shape according to the current initial three-dimensional posture.

[0078] In this embodiment, the current gesture shape refers to the shape of the gesture reflected by the current initial three-dimensional posture. And, according to the current initial three-dimensional posture, at least one of the lengths of the finger joints, the angles between the fingers, and the angles between the main joints of the fingers corresponding to the gesture in the current frame image is determined, and the current gesture shape is represented by at least one of the lengths of the finger joints, the angles between the fingers, and the angles between the main joints of the fingers corresponding to the gesture in the current frame image.

[0079] Furthermore, no matter how the gesture changes, for a normal gesture shape, at least one of the following characteristics is satisfied: the length of each finger joint is less than a threshold, the angles between each finger are less than a corresponding threshold, and the angles of each finger main joint are within a corresponding threshold range. Therefore, whether the current gesture shape reflected by the current initial three-dimensional posture satisfies a normal gesture shape can be reflected based on the degree to which the current gesture shape reflected by the current initial three-dimensional posture satisfies at least one of the following characteristics: the length of each finger joint is less than a threshold, the angles between each finger are less than a corresponding threshold, and the angles of each finger main joint are within a corresponding threshold range.

[0080] Among them, when the length of each finger joint is less than the threshold value 1 (such as 6cm), specifically, Figure 3For example, for the thumb, gesture feature point 1 and gesture feature point 2 form a joint, gesture feature point 2 and gesture feature point 3 form a joint, and gesture feature point 3 and gesture feature point 4 form a joint; for the index finger, gesture feature point 5 and gesture feature point 6 form a joint, gesture feature point 6 and gesture feature point 7 form a joint, and gesture feature point 7 and gesture feature point 8 form a joint; for the middle finger, gesture feature point 9 and gesture feature point 10 form a joint, gesture feature point 10 and gesture feature point 11 form a joint, and gesture feature point 11 and gesture feature point 12 form a joint; for the ring finger, gesture feature point 13 and gesture feature point 14 form a joint, gesture feature point 14 and gesture feature point 15 form a joint, and gesture feature point 15 and gesture feature point 16 form a joint; for the little finger, gesture feature point 17 and gesture feature point 18 form a joint, gesture feature point 18 and gesture feature point 19 form a joint, and gesture feature point 19 and gesture feature point 20 form a joint; the lengths of all joints are less than the threshold value 1. In this embodiment, a joint length is represented by the Euclidean distance between the three-dimensional coordinates included in the gesture characteristics corresponding to the two joint endpoints;

[0081] Furthermore, the angles between the fingers are all smaller than the corresponding thresholds. Specifically, the angle between the thumb and index finger is smaller than the threshold 2 (such as 90°), and the angles between the index finger and middle finger, the middle finger and ring finger, and the ring finger and little finger are all smaller than the threshold 3 (such as 60°). Among them, the calculation principle of each angle is based on the cosine theorem; for example, for the angle between the thumb and the index finger, the gesture feature point 0, the gesture feature point 4 and the gesture feature point 8 form a triangle, and the angle between the line segment formed by the gesture feature point 4 and the gesture feature point 0, and the angle between the line segment formed by the gesture feature point 0 and the gesture feature point 8 are calculated according to the cosine theorem to serve as the angle between the thumb and the index finger; for the angle between the index finger and the middle finger, it is obtained by calculating the angle between the line segment formed by the gesture feature point line segment 8 and the gesture feature point 0, and the angle between the line segment formed by the gesture feature point 0 and the gesture feature point 1; for the angle between the middle finger and the ring finger, it is obtained by calculating the angle between the line segment formed by the gesture feature point 12 and the gesture feature point 0, and the angle between the line segment formed by the gesture feature point 0 and the gesture feature point 16; for the angle between the ring finger and the little finger, it is obtained by calculating the angle between the line segment formed by the gesture feature point 16 and the gesture feature point 0, and the angle between the gesture feature point 0 and the gesture feature point 20.

[0082] And, for each main joint angle within the corresponding threshold range, specifically, for the index finger, the angle between the joint formed by gesture feature point 6 and gesture feature point 5, and the angle between the joint formed by gesture feature point 6 and gesture feature point 7 is greater than threshold 4 (such as 10°) and less than threshold 5 (such as 180°); for the middle finger, the angle between the joint formed by gesture feature point 10 and gesture feature point 9, and the angle between the joint formed by gesture feature point 10 and gesture feature point 11 is greater than threshold 6 (such as 80°) and less than threshold 5; for the ring finger, the angle between gesture feature point 14 and gesture feature point 15 is greater than threshold 6 (such as 80°) and less than threshold 5. Point 13 forms a joint, and the angle between the gesture feature points 14 and 15 is greater than the threshold 6 (such as 80°) and less than the threshold 5; for the little finger, the gesture feature points 18 and 17 are calculated to form a joint, and the angle between the gesture feature points 18 and 19 is greater than the threshold 6 (such as 80°) and less than the threshold 5; for the thumb, the gesture feature points 3 and 2 are calculated to form a joint, and the angle between the gesture feature points 3 and 4 is greater than the threshold 6 (such as 80°) and less than the threshold 5.

[0083] For the above content, taking the feature that the length of each finger joint is less than the threshold value 1 as an example, for the current initial three-dimensional posture, it is determined whether the length of each finger joint is less than the threshold value 1. Each joint length corresponds to a weight. If for a certain joint, its length is less than the threshold value 1, then the weight corresponding to the joint is accumulated. On this basis, according to the judgment result of whether the length of each finger joint is less than the threshold value 1, the cumulative weight value of the feature that the length of each finger joint is less than the threshold value 1 is obtained. It can be understood that for the two features that the angles between each finger are less than the corresponding threshold value and the angles between the main joints of each finger are within the corresponding threshold range, the weights are determined in a similar manner. Finally, the cumulative weight values ​​corresponding to each feature are accumulated to obtain the final weight, and it is determined whether the final weight is greater than or equal to the threshold value 7 (for example, 0.8), where the threshold value 7 is the minimum value of the final weight corresponding to the normal gesture shape, and the maximum value of the final weight is 1. In the case of yes, it is determined that the current gesture shape reflected by the current initial three-dimensional posture satisfies the normal gesture shape. Otherwise, it is determined that the current gesture shape reflected by the current initial three-dimensional posture does not satisfy the normal gesture shape.

[0084] In addition, it should be noted that the specific values ​​of the above thresholds 1 to 7 can be obtained based on experience or implementation, and this application does not limit this.

[0085] Step S2211, when the conditions are not met, triggers the step of determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint conditions, the current initial three-dimensional posture and the historical target three-dimensional posture.

[0086] In this embodiment, when it is determined based on the above step S2210 that the current gesture shape reflected by the current initial three-dimensional posture does not satisfy the normal gesture shape, it means that the current initial three-dimensional posture is inaccurate. At this time, the above step S2200 is triggered to determine the accurate three-dimensional posture of the gesture in the current frame image, that is, the current target three-dimensional posture of the gesture in the current frame image.

[0087] Corresponding to the above step S2211, in one embodiment of the present application, when it is determined that the current gesture shape satisfies the normal gesture shape, the current initial 3D posture is considered accurate. At this time, the current initial 3D posture is used as the current target 3D posture.

[0088] Corresponding to the above step S2211, in another embodiment of the present application, the method for determining the three-dimensional posture of a gesture provided in the embodiment of the present application also includes the following steps S2212 to S2214.

[0089] Step S2212: When it is determined that the current gesture shape satisfies the normal gesture shape, the current gesture movement amount is determined according to the current initial three-dimensional posture and the historical target three-dimensional posture.

[0090] In this embodiment, the Euclidean distance between the corresponding gesture feature points in the current initial 3D posture and the historical target 3D posture is calculated, and the average Euclidean distance between the Euclidean distances calculated based on the above is recorded as the current gesture movement amount.

[0091] Step S2213, when the current gesture movement amount is greater than the preset movement amount, triggering the step of determining the current target 3D posture of the gesture in the current frame image according to the gesture constraint conditions, the current initial 3D posture and the historical target 3D posture.

[0092] In this embodiment, the preset movement amount is the minimum current gesture movement amount that causes the current initial three-dimensional posture to be inaccurate, which can be determined based on experience or experiments. This application does not limit the specific value of the preset movement amount.

[0093] In this embodiment, when the current gesture movement amount is greater than the preset movement amount, it means that the hand speed is too fast, which makes the current initial 3D posture prone to abnormality. At this time, the above step S2200 is triggered to determine the accurate 3D posture of the gesture in the current frame image, that is, the current target 3D posture of the gesture in the current frame image.

[0094] Step S2214: When the current gesture movement amount is less than or equal to the preset movement amount, the current initial three-dimensional posture is used as the current target three-dimensional posture.

[0095] In this embodiment, corresponding to the above step S2213, when the current gesture movement amount is less than or equal to the preset movement amount, it means that the hand speed is relatively slow and the current initial 3D posture is not prone to abnormality. At this time, the current initial 3D posture is used as the current target 3D posture.

[0096] In one embodiment of the present application, a specific implementation method of the above-mentioned step S2200 is adopted, and the specific implementation method specifically includes the following steps S2220 to S2224.

[0097] Step S2220, determining the target loss value according to the gesture constraint condition.

[0098] In this embodiment, a weighted sum is performed for each constraint condition included in the gesture constraint condition to obtain a target loss value, wherein the weight corresponding to each constraint condition in the gesture constraint condition can be set based on experience.

[0099] In one example, when the gesture constraint condition includes a time constraint condition, a space constraint condition, and a shape constraint condition, a weighted sum is performed on the first maximum deviation value, the second maximum deviation value, and the third maximum deviation value to obtain a target loss value.

[0100] Step S2221, input the current initial 3D posture and the historical target 3D posture into the LM algorithm using the ceres nonlinear optimization library to obtain the current predicted 3D posture.

[0101] Step S2222, determining the current loss value according to the current predicted three-dimensional posture and the historical target three-dimensional posture.

[0102] In this embodiment, taking the gesture constraint conditions including time constraint conditions, space constraint conditions and shape constraint conditions as an example, the specific implementation of the above step S2222 is: according to the current predicted three-dimensional gesture and the historical target three-dimensional gesture, calculate the first deviation value between the current predicted three-dimensional gesture and the historical target three-dimensional gesture; perform forward projection of the camera projection model on the current predicted three-dimensional gesture to obtain the two-dimensional coordinates obtained by the projection of the current predicted three-dimensional gesture, and calculate the second deviation value between the two-dimensional coordinates of the gesture in the current frame image obtained by the projection of the current predicted three-dimensional gesture; calculate the third deviation value between the gesture shape corresponding to the current predicted three-dimensional gesture and the gesture shape corresponding to the historical target three-dimensional gesture; according to the weighted sum of the first deviation value, the second deviation value and the third deviation value calculated in the above content, obtain the current loss value. Among them, the weight corresponding to the first deviation value is the same as the weight corresponding to the first maximum deviation value when the target loss value is calculated in the above step S2220. Similarly, the weight corresponding to the second deviation value is the same as the weight corresponding to the second maximum deviation value when the target loss value is calculated in the above step S2220, and the weight corresponding to the third deviation value is the same as the weight corresponding to the third maximum deviation value when the target loss value is calculated in the above step S2220.

[0103] Step S2223, according to the current loss value and the target loss value, adjust the parameters of the LM algorithm using the ceres nonlinear optimization library, and repeatedly input the current initial three-dimensional posture and the historical target three-dimensional posture into the LM algorithm using the ceres nonlinear optimization library to obtain the current predicted three-dimensional posture, until the preset convergence condition is reached.

[0104] Step S2224: The current predicted three-dimensional posture that meets the convergence condition is used as the current target three-dimensional posture.

[0105] In this embodiment, according to the degree of deviation between the current loss value and the target loss value, the parameters of the ceres nonlinear optimization library LM algorithm, such as the iteration step size, are adjusted. For the ceres nonlinear optimization library LM algorithm after adjusting the parameters, the above step S2221 is repeated until the preset convergence condition is reached. The current predicted three-dimensional posture corresponding to the convergence condition is used as the current target three-dimensional posture.

[0106] In one embodiment of the present application, the preset convergence condition may be repeating the above step S2221 for a preset number of times. The preset number of times may be set based on experience. In one example, the value of the preset number of times is 30.

[0107] In another embodiment of the present application, the preset convergence condition may be that the deviation between the latest current loss value and the target loss value is less than or equal to the preset deviation. The predicted deviation is the maximum deviation between the current loss value and the target loss value when the current loss value is close to the target loss value.

[0108] It is understandable that when the current loss value is close to the target loss value, the latest current predicted 3D posture is a more accurate 3D posture corresponding to the current image. At this time, the latest current predicted 3D posture is used as the current target 3D posture.

[0109] In another embodiment of the present application, the preset convergence condition is that after repeatedly inputting the current initial three-dimensional posture and the historical target three-dimensional posture into the LM algorithm to obtain the current predicted three-dimensional posture within a preset number of steps, the current gesture movement amount between the current predicted three-dimensional posture and the historical target three-dimensional posture is less than or equal to the preset movement amount.

[0110] In view of the preset convergence conditions provided in this embodiment, in the process of repeatedly inputting the current initial three-dimensional posture and the historical target three-dimensional posture into the LM algorithm to obtain the current predicted three-dimensional posture step within the preset number of times, the current gesture movement amount corresponding to the latest current predicted three-dimensional posture and the historical target three-dimensional posture is calculated (as shown in the above step S2212). If the current gesture movement amount corresponding to the latest current predicted three-dimensional posture relative to the historical target three-dimensional posture is less than or equal to the preset movement amount, the latest current predicted three-dimensional posture is used as the current target three-dimensional posture. In this way, the number of times the above step S2221 is repeated can be reduced to obtain the current target three-dimensional posture faster, so as to facilitate the efficiency of determining the current target three-dimensional posture by the method for determining the three-dimensional posture of the gesture provided in this application.

[0111] In one embodiment of the present application, the method for determining the three-dimensional posture of a gesture provided by the present application also includes the following steps S2300 and S2400.

[0112] Step S2300 : in the absence of a historical target three-dimensional gesture, determining whether the current gesture shape satisfies a normal gesture shape according to the current initial three-dimensional gesture.

[0113] In this embodiment, there are three situations for the absence of historical target three-dimensional posture: situation one, the current frame image is the first frame image; situation two, the historical target three-dimensional posture is lost; situation three, there is no hand image in the previous frame image.

[0114] It should be noted that the step S2300 of determining whether the current gesture shape satisfies the normal gesture shape according to the current initial three-dimensional posture can be implemented according to the step S2210, which will not be described in detail here.

[0115] Step S2400: If the conditions are met, the current initial three-dimensional posture of the gesture in the current frame image is used as the current target three-dimensional posture of the gesture in the current frame image.

[0116] In this embodiment, based on the above steps S2300 and S2400 , the current target three-dimensional posture can be obtained for the current initial three-dimensional posture for which there is no historical target three-dimensional posture.

[0117] Corresponding to the above step S2400, if the conditions are not met, the output of the current target three-dimensional posture is not performed.

[0118] Based on the above embodiments, in one embodiment of the present application, the method for determining the three-dimensional posture of the gesture provided by the present application is as follows: Figure 4 As shown, the following steps are included:

[0119] Step S4100, obtaining the current initial three-dimensional posture of the gesture in the current frame image and the historical target three-dimensional posture of the gesture in the previous frame image, and then executing the following step S4200;

[0120] Step S4200, filtering is performed on the current initial three-dimensional posture to obtain the current initial three-dimensional posture after filtering, and the current initial three-dimensional posture is updated to the current initial three-dimensional posture after filtering, and then the following step S4300 is performed.

[0121] Step S4300, determining whether the current gesture shape satisfies a normal gesture shape according to the current initial three-dimensional posture, and if so, executing the following step S4500, and if not, executing the following step S4400;

[0122] Step S4400, determining a current target 3D posture of the gesture in the current frame image according to the gesture constraint condition, the current initial 3D posture and the historical target 3D posture;

[0123] Step S4500, determining the current gesture movement amount according to the current initial 3D gesture and the historical target 3D gesture, and then executing the following step S4600;

[0124] Step S4600, determining whether the current gesture movement amount is greater than a preset movement amount, if yes, executing the above step S4400, if no, executing the following step S4700;

[0125] Step S4700: taking the current initial 3D posture as the current target 3D posture.

[0126] The present application also provides a device 500 for determining a three-dimensional posture of a gesture, such as Figure 5 As shown, including:

[0127] An acquisition module 510 is used to acquire a current initial three-dimensional posture of a gesture in a current frame image and a historical target three-dimensional posture of a gesture in a previous frame image;

[0128] A determination module 520, configured to determine a current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture;

[0129] Among them, the gesture constraint condition includes at least two of the time constraint condition, the space constraint condition and the shape constraint condition. The time constraint condition is the first maximum deviation value between the current target three-dimensional posture and the historical target three-dimensional posture. The space constraint condition is the second maximum deviation value between the two-dimensional coordinates obtained by projection of the current target three-dimensional posture and the two-dimensional coordinates of the gesture in the current frame image. The shape constraint condition is the third maximum deviation value between the gesture shapes corresponding to the current target three-dimensional posture and the historical target three-dimensional posture.

[0130] In one embodiment of the present application, the apparatus 500 for determining the three-dimensional posture of a gesture provided by the present application further includes:

[0131] A filtering module, used for performing filtering processing on the current initial three-dimensional posture to obtain the current initial three-dimensional posture after filtering;

[0132] An updating module is used to update the current initial three-dimensional posture to the current initial three-dimensional posture after filtering.

[0133] In one embodiment of the present application, the determination module 520 is further used to: determine whether the current gesture shape satisfies a normal gesture shape according to the current initial three-dimensional posture;

[0134] In this embodiment, the device 500 for determining the three-dimensional posture of a gesture provided by the present application further includes:

[0135] The trigger module is used to trigger the step of determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture when the condition is not satisfied.

[0136] In one embodiment of the present application, the determination module 520 is further configured to: determine the current gesture movement amount according to the current initial three-dimensional posture and the historical target three-dimensional posture when it is determined that the current gesture shape satisfies the normal gesture shape;

[0137] The trigger module is further configured to trigger the step of determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture when the current gesture movement amount is greater than the preset movement amount;

[0138] The determination module 520 is further configured to: when the current gesture movement amount is less than or equal to the preset movement amount, use the current initial three-dimensional posture as the current target three-dimensional posture.

[0139] In one embodiment of the present application, the determination module 520 is specifically used to:

[0140] Determine the target loss value according to the gesture constraints;

[0141] Input the current initial 3D posture and the historical target 3D posture into the LM algorithm using the ceres nonlinear optimization library to obtain the current predicted 3D posture;

[0142] Determining a current loss value according to the current predicted three-dimensional posture and the historical target three-dimensional posture;

[0143] According to the current loss value and the target loss value, adjusting the parameters of the LM algorithm using the ceres nonlinear optimization library, and repeating the step of inputting the current initial three-dimensional posture and the historical target three-dimensional posture into the LM algorithm using the ceres nonlinear optimization library to obtain the current predicted three-dimensional posture, until a preset convergence condition is reached;

[0144] The current predicted three-dimensional pose that meets the convergence condition is used as the current target three-dimensional pose.

[0145] In one embodiment of the present application, the preset convergence condition is that after repeating the step of inputting the current initial three-dimensional posture and the historical target three-dimensional posture into the LM algorithm to obtain the current predicted three-dimensional posture for a preset number of times, the current gesture movement amount corresponding to the current predicted three-dimensional posture relative to the historical target three-dimensional posture is less than or equal to the preset movement amount.

[0146] In one embodiment of the present application, the determination module 520 is further configured to:

[0147] In the absence of the historical target three-dimensional posture, determining whether the current gesture shape satisfies a normal gesture shape according to the current initial three-dimensional posture;

[0148] If the above conditions are met, the current initial three-dimensional posture of the gesture in the current frame image is used as the current target three-dimensional posture of the gesture in the current frame image.

[0149] The present application also provides an electronic device, which includes a device 500 for determining a three-dimensional posture of any gesture provided in the above device embodiments.

[0150] Or, if Figure 6As shown, the electronic device 600 includes a memory 610 and a processor 620, wherein the memory 610 is used to store computer instructions, and the processor 620 is used to call the computer instructions from the memory 610 to execute the method for determining the three-dimensional posture of a gesture as described in any one of the above method embodiments.

[0151] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program implements a method for determining a three-dimensional posture of a gesture according to any one of the above method embodiments.

[0152] The present application may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0153] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0154] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0155] The computer program instructions for performing the operation of the present application can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of a computer-readable program instruction to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute a computer-readable program instruction, thereby realizing various aspects of the present application.

[0156] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0157] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0158] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0159] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0160] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present application is defined by the appended claims.

Claims

1. A method for determining the three-dimensional posture of a hand gesture, It is characterized in that include: Obtaining the current initial 3D posture of the gesture in the current frame image and the historical target 3D posture of the gesture in the previous frame image; Determining a current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture; Among them, the gesture constraint condition includes at least two of the time constraint condition, the space constraint condition and the shape constraint condition. The time constraint condition is the first maximum deviation value between the current target three-dimensional posture and the historical target three-dimensional posture. The space constraint condition is the second maximum deviation value between the two-dimensional coordinates obtained by projection of the current target three-dimensional posture and the two-dimensional coordinates of the gesture in the current frame image. The shape constraint condition is the third maximum deviation value between the gesture shapes corresponding to the current target three-dimensional posture and the historical target three-dimensional posture.

2. The method according to claim 1, It is characterized in that After acquiring the current initial three-dimensional posture of the gesture in the current frame image, the method further includes: Perform filtering processing on the current initial three-dimensional posture to obtain the filtered current initial three-dimensional posture, and update the current initial three-dimensional posture to the filtered current initial three-dimensional posture.

3. The method according to claim 1, It is characterized in that The method further includes, before determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture: According to the current initial three-dimensional posture, determining whether the current gesture shape satisfies the normal gesture shape; If the condition is not satisfied, the step of determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture is triggered.

4. The method according to claim 3, It is characterized in that The method further comprises: In the case where it is determined that the current gesture shape satisfies a normal gesture shape, determining a current gesture movement amount according to the current initial three-dimensional posture and the historical target three-dimensional posture; When the current gesture movement amount is greater than the preset movement amount, triggering the step of determining the current target three-dimensional posture of the gesture in the current frame image according to the gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture; When the current gesture movement amount is less than or equal to the preset movement amount, the current initial three-dimensional posture is used as the current target three-dimensional posture.

5. The method according to claim 1, It is characterized in that The determining, according to the gesture constraint condition, the current initial three-dimensional gesture and the historical target three-dimensional gesture, of the current target three-dimensional gesture of the gesture in the current frame image comprises: Determine the target loss value according to the gesture constraints; Input the current initial 3D posture and the historical target 3D posture into the LM algorithm using the ceres nonlinear optimization library to obtain the current predicted 3D posture; Determining a current loss value according to the current predicted three-dimensional posture and the historical target three-dimensional posture; According to the current loss value and the target loss value, adjusting the parameters of the LM algorithm using the ceres nonlinear optimization library, and repeating the step of inputting the current initial three-dimensional posture and the historical target three-dimensional posture into the LM algorithm using the ceres nonlinear optimization library to obtain the current predicted three-dimensional posture, until a preset convergence condition is reached; The current predicted three-dimensional pose that meets the convergence condition is used as the current target three-dimensional pose.

6. The method according to claim 5, It is characterized in that The preset convergence condition is that when the step of inputting the current initial 3D posture and the historical target 3D posture into the LM algorithm to obtain the current predicted 3D posture is repeated for a preset number of times, the current gesture movement amount corresponding to the current predicted 3D posture relative to the historical target 3D posture is less than or equal to the preset movement amount.

7. The method according to claim 1, It is characterized in that The method further comprises: In the absence of the historical target three-dimensional posture, determining whether the current gesture shape satisfies a normal gesture shape according to the current initial three-dimensional posture; If the above conditions are met, the current initial three-dimensional posture of the gesture in the current frame image is used as the current target three-dimensional posture of the gesture in the current frame image.

8. A device for determining a three-dimensional posture of a hand gesture, It is characterized in that include: An acquisition module, used to acquire the current initial three-dimensional posture of the gesture in the current frame image and the historical target three-dimensional posture of the gesture in the previous frame image; A determination module, configured to determine a current target three-dimensional posture of the gesture in the current frame image according to a gesture constraint condition, the current initial three-dimensional posture and the historical target three-dimensional posture; Among them, the gesture constraint condition includes at least two of the time constraint condition, the space constraint condition and the shape constraint condition. The time constraint condition is the first maximum deviation value between the current target three-dimensional posture and the historical target three-dimensional posture. The space constraint condition is the second maximum deviation value between the two-dimensional coordinates obtained by projection of the current target three-dimensional posture and the two-dimensional coordinates of the gesture in the current frame image. The shape constraint condition is the third maximum deviation value between the gesture shapes corresponding to the current target three-dimensional posture and the historical target three-dimensional posture.

9. An electronic device, It is characterized in that The electronic device comprises the device for determining the three-dimensional posture of the gesture as claimed in claim 8; or, The electronic device comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the method for determining the three-dimensional posture of a gesture as described in any one of claims 1-7.

10. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method for determining the three-dimensional posture of a gesture according to any one of claims 1-7 is implemented.

Citation Information

Patent Citations

  • Gesture estimation optimization method and system and electronic device

    CN112749512A

  • Gesture recognition method and device based on spinor, electronic equipment and storage medium

    CN115798030A

  • Gesture posture recognition method and device, electronic equipment, medium and program product

    CN116524582A

  • Gesture recognition method and apparatus, and storage medium

    WO2021098543A1

  • Gesture recognition method and apparatus

    WO2022105692A1