Target attitude determination method and device, electronic equipment and storage medium

Abstract

The invention provides a target attitude determination method and device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models, augmented reality and the like, and can be applied to scenes such as digital human and e-commerce live broadcast. Acquiring a hand image of the target object from the video frame; according to the hand image, hand parameters of the target object and first relative position information of both hands are obtained, and the hand parameters at least comprise a hand initial posture and a hand initial shape; according to the hand parameters and the first relative position information, the hand posture is optimized, and the hand target posture of the target object is obtained.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, especially to the technical fields of computer vision, deep learning, large models, augmented reality, etc., and can be applied to scenarios such as digital humans and e-commerce live broadcasts, and relates to a method, device, electronic device and storage medium for determining a target posture. Background Art

[0002] Hands are a vital component of human interaction due to their flexibility and efficiency. The interaction between gestures and objects is not only the main way for people to interact with complex real-world scenes, but also an important way for humans to perceive and operate the environment. Hand gesture reconstruction and tracking technology can be applied to many fields, such as human-computer interaction, augmented reality (AR), robot learning, etc., greatly expanding the possibility of collaboration between people and technology. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device and storage medium for determining a target posture.

[0004] According to a first aspect of the present disclosure, a method for determining a target posture is provided, comprising: acquiring a video frame, and acquiring a hand image of a target object from the video frame; acquiring hand parameters of the target object and first relative position information of both hands based on the hand image, wherein the hand parameters include at least an initial hand posture and an initial hand shape; optimizing the hand posture based on the hand parameters and the first relative position information to obtain a target hand posture of the target object.

[0005] According to a second aspect of the present disclosure, a target posture determination device is provided, comprising: a first acquisition module, used to acquire a video frame, and acquire a hand image of a target object from the video frame; a second acquisition module, used to acquire hand parameters of the target object and first relative position information of both hands based on the hand image, wherein the hand parameters include at least an initial hand posture and an initial hand shape; and a processing module, used to optimize the hand posture based on the hand parameters and the first relative position information, to obtain the target hand posture of the target object.

[0006] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target posture determination method described in the above-mentioned one aspect embodiment.

[0007] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, on which a computer program / instructions are stored, and the computer instructions are used to enable the computer to execute the target posture determination method described in the above-mentioned embodiment.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the target posture determination method described in the above-mentioned embodiment.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.

[0010] The present disclosure performs hand recognition by adding the first relative position information of both hands, which can reduce the impact of complex scenes on hand target posture recognition. At the same time, by optimizing and improving the accuracy and imaging effect of the finally generated hand target posture, the difficulty of hand recognition in jittery scenes is reduced, thereby improving the user experience and the practicality of the present solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 A schematic diagram of a flow chart of a target posture determination method provided in an embodiment of the present disclosure;

[0013] Figure 2 A schematic diagram of a flow chart of another target posture determination method provided by an embodiment of the present disclosure;

[0014] Figure 3 A schematic diagram of a flow chart of another target posture determination method provided by an embodiment of the present disclosure;

[0015] Figure 4 A schematic diagram of a flow chart of another target posture determination method provided by an embodiment of the present disclosure;

[0016] Figure 5 A schematic diagram of a flow chart of another target posture determination method provided by an embodiment of the present disclosure;

[0017] Figure 6 A schematic diagram of a flow chart of another target posture determination method provided by an embodiment of the present disclosure;

[0018] Figure 7 This is a schematic diagram of the overall process of hand gesture recognition according to an embodiment of the present disclosure;

[0019] Figure 8A schematic diagram of the structure of a target posture determination device provided in an embodiment of the present disclosure;

[0020] Fig. 9 The present invention is a block diagram of an electronic device according to a method for determining a target posture according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] The target posture determination method, device and electronic device of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0023] Artificial Intelligence (AI) is a discipline that studies how computers can simulate certain thought processes and intelligent behaviors of human beings (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level technologies and software-level technologies. AI hardware technologies generally include computer vision technology, speech recognition technology, natural language processing technology, as well as learning / deep learning, big data processing technology, knowledge graph technology, and other aspects.

[0024] Computer Vision (CV) is a science that studies how to enable computers to understand and interpret digital images or video sequences. It combines knowledge from multiple disciplines, including image processing, pattern recognition, machine learning, deep learning, statistics, and signal processing. The goal of computer vision is to simulate the capabilities of the human visual system so that computers can "see" and understand the world. Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify, track, and measure targets, and further perform graphics processing so that computers can process images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain "information" from images or multidimensional data, which can be used to help make a "decision". Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data.

[0025] Deep Learning (DL) is a new research direction in the field of Machine Learning (ML). It is introduced into machine learning to make it closer to its original goal - artificial intelligence. Deep learning is the internal laws and representation levels of learning sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far exceed previous related technologies.

[0026] Big models refer to "large parameter" models trained using large-scale data and powerful computing power. These models are usually highly versatile and generalizable, and can be applied to natural language processing, image recognition, speech recognition and other fields. They can be divided into big language models, big visual models, big multimodal models, and basic big models. Big AI models are gradually affecting all walks of life.

[0027] Augmented Reality (AR) technology integrates the computer-generated virtual environment with the real environment around the user by means of optoelectronic display technology, interactive technology, multiple sensor technologies, and computer graphics and multimedia technologies, so that users can be convinced from a sensory perspective that the virtual environment is part of the real environment around them. AR has new features of virtual-real combination, real-time interaction, and three-dimensional registration. AR is a technology that superimposes and integrates virtual scenery or information with the real physical environment, and interactively presents it to the user, thereby creating a space where virtual and real things share the same space. In essence, AR is a new interface technology that integrates software and hardware technologies such as positioning, presentation, and interaction. Its purpose is to allow users to sense the spatial-temporal connection and integration of virtual and real spaces, so as to enhance their perception and cognition of the real environment.

[0028] Figure 1 A flowchart of a method for determining a target posture is provided in an embodiment of the present disclosure.

[0029] like Figure 1 As shown, the target posture determination method may include:

[0030] S101, obtaining a video frame, and obtaining a hand image of a target object from the video frame.

[0031] It should be noted that the video frames in the embodiments of the present disclosure may be acquired in real time by an image acquisition device, or may be acquired by processing historical video data, and no limitation is made here. There are multiple hand images, which may be acquired by sampling, or may be a continuously acquired segment of images containing the hand.

[0032] In the embodiments of the present disclosure, there may be many methods for acquiring the hand image of the target object from the video frame, which are not limited here.

[0033] Optionally, the video frame may be calculated by an image recognition algorithm to determine the hand image from the video frame. The image recognition algorithm may be changed according to actual recognition requirements or design needs, and no limitation is made here.

[0034] Optionally, the hand image may be marked in advance in the video frame, and the hand image in the video frame may be determined according to the marking identifier.

[0035] S102, acquiring hand parameters of the target object and first relative position information of both hands according to the hand image, wherein the hand parameters at least include an initial hand posture and an initial hand shape.

[0036] In the disclosed embodiment, the hand parameters may include a variety of data, which are not limited herein. For example, the hand parameters may include hand joint data, skin color data, etc. in addition to the initial hand posture and initial hand shape.

[0037] It should be noted that the hand posture data is used to characterize the current hand movement posture, for example, it can be grasping, natural hanging, fisting, etc. Different movement postures correspond to different hand parameters.

[0038] The hand shape data is used to characterize the appearance of the hand. For example, the shape data may include the fatness, thinness, length, etc. of the hand. Different shape data correspond to different hand parameters.

[0039] The first relative position information is used to characterize the relative position of the hands in the physical space in the hand image. The gesture perception method in the current technology is difficult to accurately capture the hand gestures under complex backgrounds and various postures, and the hand perception accuracy is greatly limited by the video resolution and occlusion problems. In the present disclosure, the relative position information of the hands is determined based on the hand image, which can reduce the impact of the scene on the hand posture recognition judgment. At the same time, based on the spatial position information of the hands, smoother hand posture tracking can be achieved, providing a natural and smooth interactive experience, which is particularly suitable for application scenarios such as virtual reality and augmented reality.

[0040] It should be noted that the relative position information may be three-dimensional information in a preset coordinate system, used to represent the relative coordinates of the two hands in the three-dimensional coordinate system.

[0041] In the embodiment of the present disclosure, there may be multiple methods for acquiring the hand parameters of the target object and the first relative position information of the two hands based on the hand image, and no limitation is made here.

[0042] In one possible implementation, the hand image can be processed by a gesture detection model to generate hand parameters of the target object and first relative position information of both hands. The gesture detection model is trained in advance and can be stored in a storage space of an electronic device for easy retrieval when needed.

[0043] S103, optimizing the hand posture according to the hand parameters and the first relative position information to obtain a target hand posture of the target object.

[0044] In actual operation, image jitter refers to a slight position shift or instability between adjacent frames in a video sequence or continuously shot images due to various reasons. Image jitter may affect the subsequent recognition and judgment of hand gestures. Therefore, after obtaining the hand parameters and the first relative position information, the hand gesture can be de-jittered by the hand parameters and the first relative position information to solve the above problem, thereby improving the accuracy of subsequent processing.

[0045] In the embodiment of the present disclosure, a video frame is first obtained, and a hand image of the target object is obtained from the video frame. Then, based on the hand image, the hand parameters of the target object and the first relative position information of the hands are obtained. The hand parameters include at least the initial hand posture and the initial hand shape. Finally, based on the hand parameters and the first relative position information, the hand posture is optimized to obtain the target hand posture of the target object. Therefore, the present disclosure can reduce the influence of complex scenes on the recognition of the target hand posture by adding the first relative position information of the hands for hand recognition. At the same time, the accuracy and imaging effect of the finally generated target hand posture are improved by optimization, the difficulty of hand recognition in jittery scenes is reduced, and the user experience and the practicality of the present solution are improved.

[0046] In the above embodiment, the hand image of the target object is captured from the video frame, and the hand image of the target object can also be captured by Figure 2 Explaining further, the method includes:

[0047] S201, performing two-dimensional posture detection on a video frame to obtain a first global posture of a target object.

[0048] It should be noted that two-dimensional posture detection refers to identifying and locating key points (such as joints, fingertips, etc.) of the human body or other objects from a single or multiple images, and outputting the positions of these key points in the form of two-dimensional coordinates.

[0049] In the disclosed embodiment, a two-dimensional posture detection algorithm can be used to perform two-dimensional posture detection calculation on the video frame to generate a first global posture of the target object. The two-dimensional posture detection algorithm is designed in advance and can be changed according to actual design needs, and no limitation is made here.

[0050] S202: Determine a first key point from a first global posture.

[0051] In the disclosed embodiment, the first key point of the hand is a point corresponding to a certain part of the hand that is set in advance. For example, the first key point can be a point corresponding to the wrist, a point corresponding to the center of the palm, etc.

[0052] The first key point can be one or more, and there is no limitation here. The specific limitation can be made according to actual design requirements.

[0053] S203, obtaining a hand detection frame according to the first key point, and obtaining a hand image of the target object from the video frame based on the hand detection frame.

[0054] In the disclosed embodiment, a rectangular area may be expanded with the first key point as the center according to a predetermined ratio as the initial hand detection frame. The size of the hand detection frame may be designed in advance and may be changed according to actual design needs, and no limitation is made here.

[0055] In one possible implementation, the size of the hand detection box can be set according to the proportions of the hand. For example, the hand aspect ratio can be assumed to be a fixed value (such as 3:1), and the bounding box size can be adjusted accordingly.

[0056] In one possible implementation, a first key point is first determined from the first global posture, and then a hand detection frame is obtained according to the first key point, and based on the hand detection frame, a hand image of the target object is obtained from the video frame. Thus, by setting the hand detection frame for hand detection, the accuracy and stability of hand detection and tracking are improved, and the robustness and adaptability of the system are enhanced, which is applicable to a variety of practical application scenarios.

[0057] In the above embodiment, the hand image may be input into a pre-trained hand parameterization model, and the hand parameters may be output through the hand parameterization model, wherein the hand parameters also include the second key point and the camera parameters.

[0058] It should be noted that the hand parameterization model is a model that converts received images into hand parameters. Since the model expression is different from that of two-dimensional detection, the output result of the hand parameterization model may be less accurate than that of the two-dimensional detection output.

[0059] In the disclosed embodiment, the camera parameters are the device parameters of the image acquisition device for acquiring the video frames in the above-mentioned embodiment. The second key point is the key node of the hand for determining the position of the double guard. The second key point is designed in advance and can be changed according to the actual design needs, and no limitation is made here. For example, the second key point can be the point corresponding to the wrist of both hands, the point corresponding to the center of the palm of both hands, etc. Thus, the hand image is processed by the hand parameterization model to obtain the second key point and camera parameters, providing a data basis for the subsequent hand target posture recognition.

[0060] It should be noted that the hand parameterized model in the embodiment of the present disclosure is a method for representing and reconstructing the geometric structure of the hand, which describes the shape, posture and other attributes of the hand through a set of parameters. There can be many types of hand parameterized models, which are not limited here.

[0061] It should be noted that, since in general a person's left hand and right hand generally exist as mirror images, in order to facilitate processing, the left hand or right hand that is pre-specified in the embodiment of the present disclosure is used as the designated hand. In the embodiment of the present disclosure, in response to the hand image not being a designated hand image, the hand parameters are mirrored. For example, when the designated hand is the left hand and the hand image is the right hand, the hand image is mirrored and converted into a left hand image. In this way, the hand image can be normalized, the efficiency of subsequent image processing and analysis can be improved, and the processing cost can be reduced.

[0062] In the above embodiment, the process of obtaining the first relative position information can also be performed by Figure 3 Explaining further, the method includes:

[0063] S301, obtaining spatial position information of each hand according to the hand images and camera parameters of each hand.

[0064] In the disclosed embodiment, the process of obtaining the spatial position of the hand according to the hand image and the camera parameters involves mapping the hand key points in the two-dimensional image into the three-dimensional space to obtain the spatial position information of each hand.

[0065] There can be multiple mapping methods. In one possible implementation, the hand image and camera parameters can be processed through a three-dimensional space mapping model to generate a three-dimensional space corresponding to the hand image and the corresponding spatial position information, providing global information to enhance perception capabilities in complex scenes.

[0066] S302: Obtain first relative position information according to the spatial position information of each hand.

[0067] In the disclosed embodiment, the spatial position information of the designated key points of each hand can be first determined, where the designated key points include at least the key points at the base of the hand. Then, based on the spatial position information of the designated key points, the relative displacement between the two hands of the target object can be determined as the first relative position information.

[0068] It should be noted that the key point at the base of the hand is the base point for determining the position of the hand in the image. The key point at the base of the hand can be set according to actual design needs and is not limited here. In one possible implementation, the center point of the wrist can be used as the key point at the base of the hand. By obtaining the key point at the base of the hand, the position of both hands can be quickly tracked and located.

[0069] In the disclosed embodiment, first, the spatial position information of each hand is obtained according to the hand images and camera parameters of each hand, and then the first relative position information is obtained according to the spatial position information of each hand. Thus, by determining the first relative position information through the spatial position information of the two hands, the accuracy and stability of hand posture estimation can be significantly improved, and the robustness and adaptability of the system are also enhanced, which is applicable to a variety of practical application scenarios.

[0070] In the disclosed embodiment, the hand posture is optimized according to the hand parameters and the first relative position information to obtain the target hand posture of the target object. The hand posture can be de-jittered according to the hand parameters and the first relative position information, and the hand posture after de-jittering is optimized to obtain the target hand posture of the target object.

[0071] In the above embodiment, according to the hand parameters and the first relative position information, the hand posture is de-jittered, and the hand posture after de-jittering is optimized to obtain the hand target posture of the target object. Figure 4 Explaining further, the method includes:

[0072] S401, pay multiple attention to the hand parameters and the first relative position information to eliminate the hand posture and obtain a target output, where the target output includes the hand candidate parameters and the second relative position information.

[0073] In the embodiments disclosed this year, de-jittering the hand posture is a process of optimizing and improving the hand parameters and relative position information, and does not delete the hand parameters and relative position information to affect the number of hand parameters. In this way, the number of samples for subsequent analysis to determine the hand target posture can be guaranteed.

[0074] It should be noted that the multi-attention mechanism can focus on the most relevant features and time steps, reduce the interference of irrelevant information, and thus improve the accuracy of hand image deshaking.

[0075] In one possible implementation method, multiple attentions are paid to the hand parameters and the first relative position information to eliminate the jitter of the hand posture and obtain the target output. First, a time series is obtained based on multiple consecutive historical video frames and the hand parameters and the first relative position information of the video frames. Then, the time series is cross-attended through the multi-attention Transformer mechanism to obtain the characteristic changes of the hand parameters in the time dimension. Then, according to the characteristic changes of the hand parameters in the time dimension, the jitter information of the hand parameters is identified. Finally, the hand parameters are smoothed based on the jitter information to obtain candidate hand parameters and the second relative position information.

[0076] In an embodiment of the present disclosure, the order of all hand images currently being analyzed that correspond to hand parameters and first relative position information can be determined through historical video frames, and a time series can be generated based on the order, hand parameters and first relative position information.

[0077] In the embodiments of the present disclosure, there may be a variety of smoothing methods, which are not limited here.

[0078] Optionally, time series smoothing may be performed. For example, a Kalman filter, a Savitzky-Golay filter or the like may be applied to perform smoothing based on the identified jitter information to eliminate noise caused by camera shake or slight motion.

[0079] Optionally, smoothing can be performed via geometric constraint optimization, for example, reasonable bone length constraints and angle rationality checks can be imposed to correct the pose estimation results.

[0080] Therefore, by generating a time series, a data basis is provided for realizing continuous hand image analysis.

[0081] S402, determining the target key points of the hand, and based on the target key points and the first relative position information, combined with the second relative position information, globally optimizing the candidate hand parameters to obtain the hand target parameters, and reconstructing the hand target posture based on the hand target parameters.

[0082] In the disclosed embodiment, first, multiple attention is paid to the hand parameters and the first relative position information to eliminate the jitter of the hand posture and obtain the target output. The target output includes candidate hand parameters and the second relative position information. Then, the target key points of the hand are determined, and based on the target key points and the first relative position information, combined with the second relative position information, the candidate hand parameters are globally optimized to obtain the hand target parameters. Based on the hand target parameters, the hand target posture is reconstructed.

[0083] In the embodiments disclosed this year, de-jittering the hand posture is a process of optimizing and improving the hand parameters and relative position information, and does not delete the hand parameters and relative position information to affect the number of hand parameters. In this way, the number of samples for subsequent analysis to determine the hand target posture can be guaranteed.

[0084] It should be noted that the multi-attention mechanism can focus on the most relevant features and time steps, reduce the interference of irrelevant information, and thus improve the accuracy of hand image de-shaking. Therefore, by de-shaking and globally optimizing the hand posture, the hand parameters and hand position information can be corrected and optimized, thereby improving the accuracy and imaging effect of the final generated hand target posture and enhancing the user experience.

[0085] In the above embodiment, the hand parameters are smoothed based on the jitter information to obtain the candidate hand parameters of the target object. Figure 5 Explaining further, the method includes:

[0086] S501, determining a time point when jitter occurs according to jitter information.

[0087] In the embodiment of the present disclosure, the timestamp of the corresponding jitter point in the jitter information may be used as the time point when the jitter occurs.

[0088] S502: Obtain hand parameters of N video frames adjacent to the time point, where N is an integer greater than or equal to 1.

[0089] It should be noted that the adjacent N video frames include video frames that appear before the jitter time point and video frames that appear after the jitter time point, and the specific number ratio is random or pre-designed, and is not limited here. This can provide more comprehensive data for subsequent smoothing processing and improve processing effects.

[0090] S503: According to the hand parameters of the N video frames, smoothing processing is performed on the hand parameters corresponding to the time point to obtain candidate hand parameters.

[0091] In the disclosed embodiment, after obtaining the hand parameters of N video frames, the hand parameters corresponding to the time point after the hand parameters of the N video frames can be calculated by a preset smoothing algorithm to generate candidate hand parameters. The smoothing algorithm can be of various types, which are not limited here. For example, it can be a Kalman filter algorithm, a mean algorithm, etc.

[0092] In the disclosed embodiment, firstly, the time point at which the jitter occurs is determined according to the jitter information, then the hand parameters of N video frames adjacent to the time point are obtained, where N is an integer greater than or equal to 1, and finally, the hand parameters corresponding to the time point are smoothed according to the hand parameters of the N video frames to obtain candidate hand parameters. Thus, the hand parameters can be effectively smoothed, thereby improving the stability and accuracy of hand posture estimation, and improving the precision and stability of hand posture estimation.

[0093] In one embodiment of the present disclosure, to determine the target key point of the hand, the confidence of the first key point and the second key point may be first determined, and then the key point with high confidence may be selected from the first key point and the second key point to obtain the target key point of the hand.

[0094] It should be noted that the confidence is an evaluation value for determining whether the first key point and the second key point are closer to the real point. The higher the confidence, the closer the first key point and the second key point are to the real point. By comparing the confidence of the first key point and the second key point of the same time sequence, the key point with high confidence can be selected as the target key point. Therefore, by calculating the confidence of the first key point and the second key point, a better key point can be selected from the first key point and the second key point for subsequent processing, thereby improving the stability and accuracy of subsequent hand posture estimation.

[0095] In the above embodiment, based on the target key point and the first relative position information, combined with the second relative position information, the candidate hand parameters are globally optimized to obtain the hand target parameters. Figure 6 Explaining further, the method includes:

[0096] S601, extracting candidate key points of the hand from the candidate hand parameters, and determining the key point loss based on the candidate key points of the hand and the target key points of the hand.

[0097] In the disclosed embodiment, the key point loss is a reference value for whether the hand candidate key points are synchronized with the target key points. The smaller the key point loss, the more synchronized the hand candidate key points are with the target key points.

[0098] In the disclosed embodiment, the key point loss of the candidate key points of the hand and the target key points of the hand can be calculated by a key point loss function. The key point loss function is designed in advance and can be changed according to actual design needs, and no limitation is made here.

[0099] S602: Determine the position loss according to the first relative position information and the second relative position information.

[0100] In the embodiment of the present disclosure, the position loss is a reference value for whether the corresponding points of the first relative position information and the second relative position information are close. The smaller the position loss, the closer the corresponding points of the first relative position information and the second relative position information are.

[0101] In the embodiment of the present disclosure, the position loss of the first relative position information and the second relative position information can be calculated by a position loss function. The position loss function is designed in advance and can be changed according to actual design needs, and no limitation is made here.

[0102] S603, iteratively optimizing the candidate hand parameters according to the key point loss and the position loss to determine the hand target parameters.

[0103] In the disclosed embodiment, firstly, the candidate key points of the hand are extracted from the candidate hand parameters, and the key point loss is determined based on the candidate key points of the hand and the target key points of the hand, and then the position loss is determined according to the first relative position information and the second relative position information, and finally, according to the key point loss and the position loss, the candidate hand parameters are iteratively optimized to determine the target hand parameters. Thus, by calculating the key point loss and the position loss, the error of the posture estimation can be accurately quantified, so as to better provide direction and accurate reference data for iterative optimization.

[0104] In one possible implementation, the candidate hand parameters are iteratively optimized to determine the hand target parameters. First, the adjacent video frames of the video frame can be determined, and the smoothing loss function can be determined based on the video frame and the adjacent video frames. Then, the hand orientation loss is determined based on the candidate key points of the hand, and then a globally optimized loss function is generated based on the key point loss and position loss, as well as the smoothing loss function and the hand orientation loss. Finally, the candidate hand parameters are iteratively optimized based on the globally optimized loss function to determine the hand target parameters. Thus, by generating a globally optimized loss function in four dimensions, namely, adjacent video frames, hand orientation, key point loss, and position loss, and iteratively optimizing with the globally optimized loss function, the optimization effect of the final generated hand target posture can be improved, thereby improving the user experience.

[0105] Figure 7 FIG. 1 is a schematic diagram of the overall process of hand gesture recognition according to an embodiment of the present disclosure. Figure 7 As shown, the method includes:

[0106] S701, obtaining a video frame.

[0107] S702, processing the video frame through a 2D pose detection module to generate hand posture perception and two-hand relative position information.

[0108] S703, inputting the posture perception and the relative position information of both hands into the posture jitter elimination network, identifying and eliminating the abnormal jitter of the hand posture, and obtaining a smoother and more stable tracking result.

[0109] S704, globally optimizing the result output by the posture jitter elimination network through a global optimization module to generate a target hand posture of the target object.

[0110] Corresponding to the target posture determination method provided by the above-mentioned embodiments, an embodiment of the present disclosure also provides a target posture determination device. Since the target posture determination device provided by the embodiment of the present disclosure corresponds to the target posture determination method provided by the above-mentioned embodiments, the implementation method of the above-mentioned target posture determination method is also applicable to the target posture determination device provided by the embodiment of the present disclosure, and will not be described in detail in the following embodiments.

[0111] Figure 8 Schematic diagram of a target posture determination device provided by an embodiment of the present disclosure. Figure 8 As shown, the target posture determination device 800 includes: a first acquisition module 810, a second acquisition module 820 and a processing module 830.

[0112] The first acquisition module 810 is used to acquire a video frame and obtain a hand image of the target object from the video frame;

[0113] A second acquisition module 820 is used to acquire hand parameters of the target object and first relative position information of both hands according to the hand image, where the hand parameters at least include an initial hand posture and an initial hand shape;

[0114] The processing module 830 is used to optimize the hand posture according to the hand parameters and the first relative position information to obtain the target hand posture of the target object.

[0115] In the embodiment of the present disclosure, the first acquisition module 810 is also used to: perform two-dimensional posture detection on the video frame to obtain the first global posture of the target object; determine the first key point from the first global posture; obtain a hand detection frame according to the first key point, and based on the hand detection frame, obtain a hand image of the target object from the video frame.

[0116] In the disclosed embodiment, the first acquisition module 810 is further used to: input the hand image into a pre-trained hand parameterization model, and output hand parameters through the hand parameterization model, wherein the hand parameters also include a second key point and camera parameters.

[0117] In the embodiment of the present disclosure, the device is also used for: in response to the hand image not being a designated hand image, mirroring the hand parameters.

[0118] In the disclosed embodiment, the second acquisition module 820 is further used to: acquire spatial position information of each hand according to the hand images and camera parameters of each hand; and acquire first relative position information according to the spatial position information of each hand.

[0119] In the embodiment of the present disclosure, the second acquisition module 820 is also used to: determine the spatial position information of the designated key points of each hand, the designated key points at least including the key points of the base of the hand; and determine the relative displacement between the two hands of the target object according to the spatial position information of the designated key points as the first relative position information.

[0120] In the disclosed embodiment, the processing module 830 is further used to: de-jitter the hand posture according to the hand parameters and the first relative position information, and optimize the hand posture after de-jittering to obtain the target hand posture of the target object.

[0121] In the disclosed embodiment, the processing module 830 is also used to: pay multiple attention to the hand parameters and the first relative position information to eliminate the jitter of the hand posture and obtain a target output, the target output including candidate hand parameters and the second relative position information; determine the target key points of the hand, and based on the target key points and the first relative position information, in combination with the second relative position information, globally optimize the candidate hand parameters to obtain the hand target parameters, and reconstruct the hand target posture based on the hand target parameters.

[0122] In the disclosed embodiment, the processing module 830 is further used to: obtain a time series based on multiple consecutive historical video frames and hand parameters and first relative position information of video frames; cross-pay attention to the time series through a multi-attention Transformer mechanism to obtain characteristic changes of the hand parameters in the time dimension; identify jitter information of the hand parameters based on the characteristic changes of the hand parameters in the time dimension; and smooth the hand parameters based on the jitter information to obtain candidate hand parameters and second relative position information.

[0123] In the embodiment of the present disclosure, the processing module 830 is also used to: determine the time point when the shaking occurs based on the shaking information; obtain the hand parameters of N video frames adjacent to the time point, where N is an integer greater than or equal to 1; and smooth the hand parameters corresponding to the time point based on the hand parameters of the N video frames to obtain candidate hand parameters.

[0124] In the disclosed embodiment, the processing module 830 is further used to: determine the confidence of the first key point and the second key point; and select a key point with a high confidence from the first key point and the second key point to obtain a target key point of the hand.

[0125] In the embodiment of the present disclosure, the processing module 830 is also used to: extract candidate hand key points from the candidate hand parameters, and determine the key point loss based on the candidate hand key points and the target key points of the hand; determine the position loss according to the first relative position information and the second relative position information; iteratively optimize the candidate hand parameters according to the key point loss and the position loss to determine the target hand parameters.

[0126] In an embodiment of the present disclosure, the device is also used to: determine adjacent video frames of a video frame, and determine a smoothing loss function based on the video frame and the adjacent video frames; determine a hand orientation loss based on candidate key points of the hand; generate a globally optimized loss function based on the key point loss and position loss, as well as the smoothing loss function and the hand orientation loss; iteratively optimize the candidate hand parameters based on the globally optimized loss function to determine the hand target parameters.

[0127] The present disclosure performs hand recognition by adding the first relative position information of both hands, which can reduce the impact of complex scenes on hand target posture recognition. At the same time, by optimizing and improving the accuracy and imaging effect of the finally generated hand target posture, the difficulty of hand recognition in jittery scenes is reduced, thereby improving the user experience and the practicality of the present solution.

[0128] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0129] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0130] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0131] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program / instruction stored in a read-only memory (ROM) 902 or a computer program / instruction loaded from a storage unit 906 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0132] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906 such as a keyboard, a mouse, etc.; an output unit 907 such as various types of displays, speakers, etc.; a storage unit 908 such as a disk, an optical disk, etc.; and a communication unit 909 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0133] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a target posture determination method. For example, in some embodiments, the target posture determination method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 906. In some embodiments, part or all of the computer program / instructions may be loaded and / or installed on the device 900 via ROM 902 and / or communication unit 909. When the computer program / instructions are loaded into RAM 903 and executed by the computing unit 901, one or more steps of the target posture determination method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the target posture determination method in any other appropriate manner (e.g., by means of firmware).

[0134] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs / instructions that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0136] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0138] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0139] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs / instructions running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0140] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the disclosure can be achieved, and this document does not limit them here.

[0141] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining a target posture, wherein: The method comprises: Acquire a video frame, and acquire a hand image of a target object from the video frame; Acquire hand parameters of the target object and first relative position information of both hands according to the hand image, wherein the hand parameters at least include an initial hand posture and an initial hand shape; The hand posture is optimized according to the hand parameters and the first relative position information to obtain the hand target posture of the target object.

2. The method according to claim 1, wherein: The acquiring the hand image of the target object from the video frame comprises: Performing two-dimensional posture detection on the video frame to obtain a first global posture of the target object; determining a first keypoint from the first global pose; A hand detection frame is acquired according to the first key point, and based on the hand detection frame, a hand image of the target object is acquired from the video frame.

3. The method according to claim 1, wherein: The process of obtaining the hand parameters includes: The hand image is input into a pre-trained hand parameterization model, and the hand parameters are output through the hand parameterization model, wherein the hand parameters also include a second key point and a camera parameter.

4. The method according to claim 3, wherein: The method further comprises: In response to the hand image not being a designated hand image, mirror processing is performed on the hand parameters.

5. The method according to any one of claims 1 to 4, wherein: The process of acquiring the first relative position information includes: Acquire spatial position information of each of the two hands according to the hand images and camera parameters of each of the two hands; The first relative position information is acquired according to the spatial position information of each of the two hands.

6. The method according to claim 5, wherein: The acquiring the first relative position information according to the spatial position information of each of the two hands includes: Determine the spatial position information of the designated key points of each of the two hands, wherein the designated key points at least include the key points of the base of the hands; The relative displacement between the two hands of the target object is determined according to the spatial position information of the designated key point as the first relative position information.

7. The method according to claim 1, wherein: The step of optimizing the hand posture according to the hand parameter and the first relative position information to obtain the hand target posture of the target object includes: The hand posture is de-jittered according to the hand parameters and the first relative position information, and the de-jittered hand posture is optimized to obtain the hand target posture of the target object.

8. The method according to claim 7, wherein: De-jittering the hand posture according to the hand parameter and the first relative position information, and optimizing the hand posture after de-jittering to obtain the hand target posture of the target object, including: Performing multi-attention on the hand parameters and the first relative position information to de-shake the hand posture and obtain a target output, wherein the target output includes the hand candidate parameters and the second relative position information; Determine the target key points of the hand, and based on the target key points and the first relative position information, combine the second relative position information to globally optimize the candidate hand parameters to obtain hand target parameters, and reconstruct the hand target posture based on the hand target parameters.

9. The method according to claim 8, wherein: The multi-attention is paid to the hand parameter and the first relative position information to eliminate the jitter of the hand posture and obtain the target output, including: Obtaining a time series based on a plurality of continuous historical video frames and the hand parameters and the first relative position information of the video frames; The time series is cross-attended through a multi-attention Transformer mechanism to obtain characteristic changes of hand parameters in the time dimension; identifying jitter information of the hand parameters according to characteristic changes of the hand parameters in the time dimension; The hand parameters are smoothed based on the jitter information to obtain the hand candidate parameters and the second relative position information.

10. The method according to claim 9, wherein: The smoothing process of the hand parameters based on the jitter information to obtain the hand candidate parameters includes: Determining a time point when the jitter occurs according to the jitter information; Obtain hand parameters of N video frames adjacent to the time point, where N is an integer greater than or equal to 1; According to the hand parameters of the N video frames, the hand parameters corresponding to the time point are smoothed to obtain the candidate hand parameters.

11. The method according to claim 8, wherein: The step of determining the target key points of the hand includes: Determine the confidence of the first key point and the second key point; A key point with a high confidence level is selected from the first key point and the second key point to obtain a target key point of the hand.

12. The method according to claim 8, wherein: The globally optimizing the candidate hand parameters based on the target key points and the first relative position information in combination with the second relative position information to obtain the hand target parameters includes: Extracting candidate hand key points from the candidate hand parameters, and determining key point losses based on the candidate hand key points and target key points of the hand; determining a position loss according to the first relative position information and the second relative position information; The candidate hand parameters are iteratively optimized according to the key point loss and the position loss to determine the hand target parameters.

13. The method according to claim 12, wherein: The method further comprises: Determining adjacent video frames of the video frame, and determining a smoothing loss function according to the video frame and the adjacent video frames; Determining a hand orientation loss according to the candidate hand key points; Generate a globally optimized loss function according to the key point loss and the position loss, the smoothness loss function and the hand orientation loss; The candidate hand parameters are iteratively optimized according to the global optimization loss function to determine the hand target parameters.

14. A device for determining a target posture, wherein: The device comprises: A first acquisition module, used to acquire a video frame and acquire a hand image of a target object from the video frame; A second acquisition module, configured to acquire hand parameters of the target object and first relative position information of both hands according to the hand image, wherein the hand parameters at least include an initial hand posture and an initial hand shape; The processing module is used to optimize the hand posture according to the hand parameters and the first relative position information to obtain the hand target posture of the target object.

15. The device according to claim 14, wherein: The first acquisition module is further used for: Performing two-dimensional posture detection on the video frame to obtain a first global posture of the target object; determining a first keypoint from the first global pose; A hand detection frame is acquired according to the first key point, and based on the hand detection frame, a hand image of the target object is acquired from the video frame.

16. The device according to claim 14, wherein: The first acquisition module is further used for: The hand image is input into a pre-trained hand parameterization model, and the hand parameters are output through the hand parameterization model, wherein the hand parameters also include a second key point and a camera parameter.

17. The device according to claim 16, wherein: The device is also used for: In response to the hand image not being a designated hand image, mirror processing is performed on the hand parameters.

18. The device according to any one of claims 14 to 17, wherein: The second acquisition module is further used for: Acquire spatial position information of each of the two hands according to the hand images and camera parameters of each of the two hands; The first relative position information is acquired according to the spatial position information of each of the two hands.

19. The device according to claim 18, wherein: The second acquisition module is further used for: Determine the spatial position information of the designated key points of each of the two hands, wherein the designated key points at least include the key points of the base of the hands; The relative displacement between the two hands of the target object is determined according to the spatial position information of the designated key point as the first relative position information.

20. The device according to claim 14, wherein: The processing module is further used for: The hand posture is de-jittered according to the hand parameters and the first relative position information, and the de-jittered hand posture is optimized to obtain the hand target posture of the target object.

21. The device according to claim 20, wherein: The processing module is further used for: Performing multi-attention on the hand parameters and the first relative position information to de-shake the hand posture and obtain a target output, wherein the target output includes the hand candidate parameters and the second relative position information; Determine the target key points of the hand, and based on the target key points and the first relative position information, combine the second relative position information to globally optimize the candidate hand parameters to obtain hand target parameters, and reconstruct the hand target posture based on the hand target parameters.

22. The device according to claim 21, wherein The processing module is further used for: Obtaining a time series based on a plurality of continuous historical video frames and the hand parameters and the first relative position information of the video frames; The time series is cross-attended through a multi-attention Transformer mechanism to obtain characteristic changes of hand parameters in the time dimension; identifying jitter information of the hand parameters according to characteristic changes of the hand parameters in the time dimension; The hand parameters are smoothed based on the jitter information to obtain the hand candidate parameters and the second relative position information.

23. The device according to claim 22, wherein: The processing module is further used for: Determining a time point when the jitter occurs according to the jitter information; Obtain hand parameters of N video frames adjacent to the time point, where N is an integer greater than or equal to 1; According to the hand parameters of the N video frames, the hand parameters corresponding to the time point are smoothed to obtain the candidate hand parameters.

24. The device according to claim 21, wherein The processing module is further used for: Determine the confidence of the first key point and the second key point; A key point with a high confidence level is selected from the first key point and the second key point to obtain a target key point of the hand.

25. The device according to claim 21, wherein The processing module is further used for: Extracting candidate hand key points from the candidate hand parameters, and determining key point losses based on the candidate hand key points and target key points of the hand; determining a position loss according to the first relative position information and the second relative position information; The candidate hand parameters are iteratively optimized according to the key point loss and the position loss to determine the hand target parameters.

26. The device according to claim 25, wherein The device is also used for: Determining adjacent video frames of the video frame, and determining a smoothing loss function according to the video frame and the adjacent video frames; Determining a hand orientation loss according to the candidate hand key points; Generate a globally optimized loss function according to the key point loss and the position loss, the smoothness loss function and the hand orientation loss; The candidate hand parameters are iteratively optimized according to the global optimization loss function to determine the hand target parameters.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target posture determination method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the target posture determination method according to any one of claims 1-13.

29. A computer program product comprising a computer program / instructions, wherein: When the computer program / instruction is executed by a processor, the target posture determination method according to any one of claims 1 to 13 is implemented.

Images