Method, device, equipment and medium for recognizing human body movements

Abstract

A method, apparatus, device and medium for identifying human motion are provided. In one method, in response to receiving an input image including a human object, a target motion associated with the human object in the input image is determined based on a motion model. The motion model describes the association between an image including a human object and the motion of the human object in the image, and the motion is represented by a set of spatial points in a virtual human object corresponding to the human object, and the dimension of the virtual human object matches the dimension of the human object. The input image is updated using the virtual human object that performs the target motion. Using the exemplary implementation of the present disclosure, the accuracy of the motion model can be improved, so that the motion model can identify human motion in a more accurate and effective manner.

Description

Technical Field

[0001] The exemplary implementations of the present disclosure generally relate to machine learning technology, and in particular to methods, devices, apparatuses, and computer-readable storage media for recognizing human motions based on machine learning technology. Background Art

[0002] A large number of augmented reality applications have been developed, and users can use image acquisition devices in devices such as mobile terminals to take pictures of people in real environments. Furthermore, human body movements of people can be recognized based on machine learning to control augmented reality applications and / or other purposes. Generally speaking, the computing resources of mobile devices are relatively limited. At this time, how to improve the accuracy of human body movement recognition has become an urgent problem to be solved. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for recognizing human actions is provided. In the method, in response to receiving an input image including a human object, a target action associated with the human object in the input image is determined based on an action model. The action model describes an association between an image including a human object and an action of the human object in the image, the action being represented by a set of spatial points in a virtual human object corresponding to the human object, and the dimensions of the virtual human object matching the dimensions of the human object. The input image is updated using the virtual human object that performs the target action.

[0004] In a second aspect of the present disclosure, a device for recognizing human actions is provided. The device includes: a determination module configured to, in response to receiving an input image including a human object, determine a target action associated with the human object in the input image based on an action model, wherein the action model describes an association relationship between an image including the human object and an action of the human object in the image, wherein the action is represented by a set of spatial points in a virtual human object corresponding to the human object, and the dimension of the virtual human object matches the dimension of the human object; and an update module configured to update the input image using the virtual human object that performs the target action.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the device executes the method according to the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.

[0007] It should be understood that the content described in this content section is not intended to limit the key features or important features of the implementation of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the following, in conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 Examples of human body motions that can be processed by implementations of the present disclosure are shown;

[0010] Figure 2 A block diagram showing an example of a training image for training an action model according to some technical solutions;

[0011] Figure 3 A block diagram of a process for recognizing human motion according to some implementations of the present disclosure is shown.

[0012] Figure 4 A block diagram showing a coordinate system of various spatial points in a virtual human object according to some implementations of the present disclosure;

[0013] Figure 5 A block diagram showing a process for setting a virtual human object based on a training image according to some implementations of the present disclosure;

[0014] Figure 6 A block diagram illustrating a process for updating a motion model based on a comparison of a training image and a predicted virtual image according to some implementations of the present disclosure;

[0015] Figure 7 A block diagram showing updating a human object in an input image based on a virtual human object according to some implementations of the present disclosure is shown;

[0016] Figure 8 A block diagram showing generation of an intermediate image including a blurred area according to some implementations of the present disclosure is shown;

[0017] Fig. 9 A block diagram showing a process of adjusting the orientation of a spatial point in a virtual human object according to some implementations of the present disclosure;

[0018] Fig.10 A block diagram showing a process of generating a left-hand virtual human object based on a right-hand virtual human object according to some implementations of the present disclosure;

[0019] Fig.11 A flow chart of a method for recognizing human body motion according to some implementations of the present disclosure is shown;

[0020] Fig.12 A block diagram showing an apparatus for recognizing human motion according to some implementations of the present disclosure; and

[0021] Fig.13 A block diagram of a device capable of implementing multiple implementations of the present disclosure is shown. DETAILED DESCRIPTION

[0022] The implementation of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the implementations described herein. On the contrary, these implementations are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and implementations of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] In the description of the implementation methods of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "an implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0024] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0025] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0026] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0027] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0028] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0029] Example Environment

[0030] Currently, many augmented reality applications have been developed. For example, an image acquisition device can be used to capture and recognize the user's body movements. Further, the recognized body movements can be used to control various objects in the augmented reality application, etc. In the following, first refer to Figure 1 Describes the outline of human action recognition. Figure 1 An example 100 of human motion that may be processed by implementations of the present disclosure is shown.

[0031] It will be understood that the human body object may include various parts of the human body, including but not limited to the torso, limbs, head, hands, etc. In the context of the present disclosure, only the hand will be used as an example of the human body object to describe the details of the process of recognizing human body movements. Figure 1 As shown, the hand can perform a variety of actions, for example, actions 110, 120, and 130 relate to writing actions, and actions 140, 150, and 160 relate to other actions. It will be understood that Figure 1 The actions shown in are merely illustrative, and the user can control his or her fingers to perform any other actions.

[0032] At present, a technical solution based on machine learning to identify human body movements has been proposed. For example, multiple key points can be marked at multiple predetermined positions of the hand. Hand images can be collected and the marked hand images can be used to train the action model. Figure 2 A block diagram 200 is shown of an example of training images for training a motion model according to some technical solutions.

[0033] like Figure 2As shown, image 210 shows an image of the back of the right hand, and key points 0 to 20, for example, can be marked in the hand (for example, key points can be set at various joints of the hand), and the hand image can be simplified to include a tree structure defined by key points 0 to 20. Image 220 shows an image of the palm of the right hand, and image 220 also includes predefined key points 0 to 20. It will be understood that the position of the key points in the hand is fixed and does not change with different movements. For example, key point 0 is always located at the wrist joint point, and key point 4 is always located at the end of the thumb, and so on.

[0034] According to a technical solution, a three-dimensional model of a virtual hand object can be pre-established, and a motion model can be used to represent the correspondence between the key points in the hand image and the spatial positions of the corresponding key points in the three-dimensional model. However, there may be differences between the hand in the hand image used as training data and the hand in the three-dimensional model. Assuming that the hand shape of the three-dimensional model is slender, but the hand shape in the training data is short and thick, this will cause the trained motion model to be unable to accurately recognize hand movements. In other words, when using the trained motion model to recognize hand movements, it may be impossible to accurately determine the key points, resulting in the recognition of erroneous movements. At this point, how to recognize human body movements in a more convenient and effective way has become a problem that needs to be solved urgently.

[0035] Highlight process of human action recognition

[0036] In order to solve the deficiencies in the above technical solutions, according to an exemplary implementation of the present disclosure, a method for identifying human body movements is proposed. In summary, an action model describing the association relationship between an image including a human body object and the action of the human body object in the image can be obtained. It will be understood that the action here is represented by a set of spatial points in a virtual human body object corresponding to the human body object, and the dimension of the virtual human body object matches the dimension of the human body object. In the context of the present disclosure, when establishing a virtual object of the hand, the dimension of the hand used as training data needs to be considered. In other words, a three-dimensional virtual human body object of the hand can be established according to the same size, proportion and posture based on the dimensions of the hand in the training image (for example, including size, proportion and posture, etc.). According to an exemplary implementation of the present disclosure, the trained action model can be directly received; alternatively and / or additionally, the action model can be trained based on machine learning technology.

[0037] See also Figure 3 An overview of an exemplary implementation of the present disclosure is described. Figure 3 A block diagram 300 of a process for recognizing human motion according to some implementations of the present disclosure is shown. Figure 3As shown, the input image 330 can be processed based on the action model 320 that describes the association relationship between the image including the human object and the action of the human object in the image. Specifically, when the input image 330 including the human object is received, the target action associated with the human object in the input image 330 can be determined based on the action model 320. Further, the input image can be updated using the virtual human object that performs the target action so as to generate a new output image 340.

[0038] It will be understood that the action model 320 herein can describe the association between an image including a human object and the action of the human object in the image, the action being represented by a set of spatial points in a virtual human object corresponding to the human object, and the dimension of the virtual human object matches the dimension of the human object. In order to obtain the action model 320, a set of image key points 212 in an image 210 including a human object can be obtained. According to an exemplary implementation of the present disclosure, an image 210 can be selected from a set of training images, for example, a front image or a back image including a full view of a hand can be selected. It will be understood that each image in the set includes a set of pre-annotated image key points. In this way, the dimensional information of each part of the hand can be obtained as much as possible, thereby establishing a virtual human object with higher accuracy.

[0039] Furthermore, a virtual human object 310 corresponding to the human object may be generated based on the dimensions of the human object. Figure 3 As shown, the virtual human object 310 may include a set of spatial points 312 that respectively correspond to a set of image key points 212 in the image 210. It will be understood that for simplicity purposes, Figure 3 A set of spatial points 312 in the virtual human object 310 in FIG. 2 only shows thumb-related spatial points 0′ to 4′ (corresponding to key points 0 to 4 in the image 210 , respectively). The virtual human object 310 may further include other spatial points corresponding to other key points 5 to 20 in the image 210 .

[0040] like Figure 3 As shown, when establishing the virtual human object 310, a virtual hand model having the same size, proportion and posture as the real hand can be established based on the dimensions of the hand in the image 210. Specifically, the virtual human object 310 can be made to coincide with the hand in the image 210. That is, when making the virtual human object 310 of the hand, each joint point in the virtual human object 310 can be aligned with each joint point of the hand in the image 210. In this way, it can be ensured that the orientation of the two is consistent and the length ratio is as similar as possible.

[0041] Further, the action model 320 can be determined based on a set of image key points 212 and a set of spatial points 312. Here, the action model 320 can describe the association between the image including the human object and the action of the human object in the image. It will be understood that the action here is represented by a set of spatial points 312 in the virtual human object 310. That is, when a set of spatial points 312 is determined, the action of the hand can be determined. Further, the action model 320 can be used to process the input object 330 including the virtual human object 310. Specifically, the virtual human object 310 can be made to perform a determined action, and the input image 330 can be updated accordingly.

[0042] With the exemplary implementation of the present disclosure, since the virtual human object 310 and the hand in the image 210 have the same dimensions, the positions and orientations of a set of image key points 212 and a set of spatial points 312 can accurately correspond one to one. In this way, the virtual human object 310 can simulate the movements of the hand in the image 210 more realistically, thereby improving the accuracy of the movement model 320.

[0043] Detailed process of recognizing human actions

[0044] Already seen Figure 3 An overview of identifying human actions is described, and more details about identifying human actions will be provided below. According to an exemplary implementation of the present disclosure, in the process of generating a virtual human object, the size of the virtual human object 310 needs to be considered first. Specifically, the training image can be used to obtain the action model 320. For example, the positional relationship of a set of image key points 212 can be determined based on the dimensions of the human object in the image 210. In other words, the distance between each key point can be determined, and then the relative position between each spatial point in the virtual human object 310 can be determined. Using the exemplary implementation of the present disclosure, the action can be described based on a virtual human object that is consistent with the size, proportion and posture of the human object. In this way, in this way, the accuracy of the action model 320 can be improved and the accuracy of subsequent processing can be improved.

[0045] In the image 210, it is assumed that the distance between key point 0 and key point 1 is 4 cm, the distance between key point 1 and key point 2 is 5 cm, the distance between key point 2 and key point 3 is 3.5 cm, and the distance between key point 3 and key point 4 is 2.5 cm. The positions of each spatial point in a set of spatial points 312 can be determined based on the above distances. In other words, in the generated virtual human object 310, the distance between spatial point 0' and spatial point 1' is 4 cm, the distance between spatial point 1' and spatial point 2' is 5 cm, the distance between spatial point 2' and spatial point 3' is 3.5 cm, and the distance between spatial point 3' and spatial point 4' is 2.5 cm. In this way, it can be ensured that the size of the virtual human object 310 is consistent with the real hand in the image 210, thereby improving the accuracy of the action model 320 determined later.

[0046] It will be understood that, in addition to the position of the spatial point, the orientation of each spatial point will also affect the posture of the virtual human object 310. It will be understood that, since the image key points are located at the joint points of the hand of the image 210 and the spatial points are located at the joint points of the virtual human object 310, the hand posture of the virtual human object 310 cannot be uniquely determined based only on the relationship between the spatial points. At this time, it is also necessary to consider the orientation of each image key point, and then determine the orientation of each spatial point based on the orientation.

[0047] According to an exemplary implementation of the present disclosure, a reference coordinate system can be established based on a variety of methods to indicate the orientation of each spatial point. Figure 4 Describes more about determining the orientation of a point in space. Figure 4 A block diagram 400 of a coordinate system of various spatial points in a virtual human object according to some implementations of the present disclosure is shown. Figure 4 As shown, a reference coordinate system of its own can be established for each spatial point. Specifically, the orientation of a group of spatial points 312 can be determined based on the orientation relationship of a group of image key points 212 in the dimension of the human object.

[0048] For example, a coordinate system 410 may be established for the spatial point 0', and the origin of the coordinate system 410 may be located at the position of the spatial point 0'. The coordinate system 410 may be established based on a variety of methods, for example, the direction of the x-axis may be defined as to the right, and the coordinate system 410 may be established based on a left-handed coordinate system and / or a right-handed coordinate system. Further, each spatial point may be processed in a similar manner, and the orientation of each spatial point may be determined. At this time, the directions of the coordinate axes of the various spatial points may be consistent, for example, the x-axis may be to the right, the y-axis may be upward and the y-axis may point to the outside of the screen. Alternatively and / or additionally, the directions of the coordinate axes of the various spatial points may be inconsistent, and the orientation of the spatial point may be determined based on the wrist orientation of the real person in the hand image in the image 210. For example, the x-axis direction of the spatial point 1' may point to the spatial point 0', and the x-axis direction of the spatial point 2' may point to the spatial point 1', and so on.

[0049] According to an exemplary implementation of the present disclosure, the orientation can be represented based on a variety of formats, for example, the orientation can be represented based on attitude angles (i.e., yaw angle, pitch angle, roll angle). Alternatively and / or additionally, the orientation can be represented based on a quadruple, and so on. Using the exemplary implementation of the present disclosure, it can be ensured that the orientation of each finger of the virtual human object 310 is consistent with the real hand, and it can be ensured that the length of each bone segment of the virtual human object 310 is also consistent with the real hand. In this way, the virtual human object 310 can be as consistent as possible with the hand in the image 210, thereby improving the accuracy of the subsequent determination of the action model 320.

[0050] According to an exemplary implementation of the present disclosure, a group of key points 212 respectively include a group of joint points of a human body object, and a group of spatial points 312 include a group of joint points of a virtual human body object corresponding to a group of joint points of the human body object. In this way, the key points and the spatial points are respectively set at the joint points of the real hand and the virtual human body object 310, so that the virtual human body object 310 can simulate the movement of the real hand more accurately. With the movement of the real hand, the spatial point can have a corresponding orientation. According to an exemplary implementation of the present disclosure, the threshold of the orientation of the spatial point can be set according to the range of motion limit of the human skeleton. In this way, the virtual human body object 310 can simulate the movement of the real hand more accurately.

[0051] According to an exemplary implementation of the present disclosure, the distance between two spatial points is determined based on the distance between key points, and the distance between the two spatial points does not change with hand movements. In other words, the skeleton between the two spatial points of the virtual human object 310 can be considered as a rigid body and will not be stretched or compressed. In this way, only the orientation of each spatial point needs to be considered to accurately describe each spatial point, and then describe the overall posture of the virtual human object 310.

[0052] According to an exemplary implementation of the present disclosure, after the virtual human object 310 aligned with the hand in the image 210 has been determined, the action model 320 can be determined using a set of image key points 212 and a set of spatial points 312. In other words, the orientation of each spatial point in the virtual human object 310 can be determined based on the image 210, thereby obtaining the action model 320. According to an exemplary implementation of the present disclosure, the action model 320 can be obtained based on machine learning technology. The training process can be performed based on the initial action model and / or the pre-trained action model.

[0053] According to an exemplary implementation of the present disclosure, a large number of training images in the training image set may be used to optimize the motion model 320. Specifically, the motion model 320 may be used to process a given training image including a human object, and the motion model 320 may be updated based on a comparison between the processing result and the given training image. Figure 5 Describes more details of an exemplary implementation according to the present disclosure, Figure 5 A block diagram 500 is shown of a process for setting a virtual human object 310 based on a training image according to some implementations of the present disclosure. Figure 5 As shown, a training image 510 may be obtained from a training image set. Specifically, the training image 510 may be input into the motion model 320 to determine a predicted motion 530 of a human object in the training image 510. In this process, a set of joint points in the training image 510 may be obtained, and the set of joint points may be input into the motion model 320 to determine the predicted motion 530.

[0054] It will be understood that the predicted action 530 herein is described using a set of spatial points (i.e., joint points) in the virtual human object 310, and the action model 320 can output the orientation 540 of each spatial point in the virtual human object 310. At this time, the orientation of a set of joint points of the virtual human object 310 can be set as the orientation 540 of the predicted spatial point. In the context of the present disclosure, once the orientation of the joint points is determined, the position of each joint point can be automatically determined based on the distance between each joint point. Thus, the posture of the virtual human object 310 in the three-dimensional virtual space can be determined, that is, the predicted action 530 of the virtual human object 310 can be determined.

[0055] According to an exemplary implementation of the present disclosure, the predicted virtual image of the virtual human object 310 may be determined based on the predicted action 530, that is, the predicted virtual image of the virtual human object 310 may be determined based on the orientation of a set of joint points. Figure 6 Describes more details about the predicted virtual image. Figure 6 A block diagram 600 is shown of a process for updating a motion model based on a comparison of a training image and a predicted virtual image according to some implementations of the present disclosure. Figure 6 As shown, the predicted virtual image 630 refers to an image obtained by projecting the virtual human object 310 in the three-dimensional space into the two-dimensional space. Specifically, the positional relationship between the virtual camera and the virtual human object 310 in the three-dimensional space can be determined according to the spatial positional relationship between the camera and the real hand when the training image 510 is collected, thereby obtaining the predicted virtual image 630.

[0056] Further, the predicted virtual image 630 and the part associated with the human body object in the training image 510 can be compared (that is, the positions of the various joints of the hand in the virtual image 630 and the training image 510 are compared), so as to update the action model 320. According to an exemplary implementation of the present disclosure, a loss function can be determined based on the comparison of the two images. In the following, only the joints related to the thumb of the hand will be described as an example.

[0057] like Figure 6As shown, the training image 510 may include key points 612, 614, 616, 618, and 620, and at this time, the connection lines of each key point can form the relevant bones of the thumb. Similarly, there may be predicted image space points (referred to as predicted points) corresponding to the above key points in the predicted virtual image 630. For example, the predicted virtual image 630 may include predicted points 632, 634, 636, 638, and 640 located at the thumb joints of the virtual human object 310. Using the exemplary implementation of the present disclosure, the position of each joint point can describe the outline of the hand action. By using the predicted points associated with the joint points to describe the action, only the main factors affecting the hand action can be considered and the secondary factors (e.g., muscle contour points, etc.) can be ignored. In this way, the amount of calculation involved in determining the action model 320 can be reduced and the performance can be improved.

[0058] According to an exemplary implementation of the present disclosure, the above-mentioned key points and the corresponding predicted points can be compared respectively to determine the distance between the two. For example, the distance between the key point 612 at the wrist and the predicted point 632 can be determined, the distance between the key point 612 at the palm and the predicted point 632 can be determined, the distance between the key point 616 at the base of the thumb and the predicted point 636 can be determined, the distance between the key point 618 at the thumb joint point and the predicted point 638 can be determined, and so on. The loss function 640 of the action model 320 can be set based on these distances. For example, the loss function 640 can be set based on the summation, variance and / or other methods of each distance.

[0059] According to an exemplary implementation of the present disclosure, various parameters in the action model 320 may be updated in a direction that minimizes the loss function 640. It will be understood that although Figure 6 Only the joints related to the thumb in the hand are shown, and the joints related to other fingers in the hand can be processed in a similar manner, and the distance between the key points and the predicted points at each joint can be determined. In this way, the positions of the various joints in the hand can be comprehensively considered, so that the action model 320 can learn the relevant knowledge of the hand movements in the training image 510 more accurately. According to an exemplary implementation of the present disclosure, the action model 320 can be continuously trained in an iterative manner. The training stop condition can be set in advance, for example, the training can be stopped when the number of training times meets a predetermined threshold, the training can be stopped when the action model 320 meets a threshold convergence condition, and so on.

[0060] The training process for generating the motion model 320 has been described above. After the motion model 320 has been obtained, the motion model 320 can be used to recognize the motion of the human object in the input image 330. Figure 7Describes more details of processing the input image using the motion model 320. Figure 7 A block diagram of updating a human object in an input image based on a virtual human object according to some implementations of the present disclosure is shown.

[0061] like Figure 7 As shown, in the case of receiving an input image 330 including a hand, the input image 330 may be provided to the action model 320 so as to determine the action 712 of the hand. Further, the orientation of the joint points of the virtual human object 310 may be set based on the action 712 of the hand. Subsequently, the virtual human object 310 may be projected into a two-dimensional image space, and a corresponding virtual image 714 may be generated (at this time, the virtual human object 310 is set to perform the action 712). Next, the generated virtual image 714 may be used to replace the portion of the input image 330 associated with the hand so as to generate an output image 340.

[0062] like Figure 7 As shown, the real hand in the output image 340 has been replaced by the manipulator of the virtual human object 310, and the action of the manipulator is determined based on the action of the real hand in the input image 330. It will be understood that although the above describes the process of recognizing actions, generating virtual images, and replacing the human object in the input image with the generated virtual images separately, alternatively and / or additionally, the module for performing the above process can be integrated into the action model 320. At this time, the integrated action model 320 can directly receive the input image 330 and provide the corresponding output image 340.

[0063] Using the exemplary implementation of the present disclosure, the virtual human object 310 used by the action model 320 is established based on the dimensions of the hand in the training image, so that the factors such as the bone size, proportion, and posture of each joint point of the virtual human object 310 are consistent with the real hand in the training image. In this way, the action model 320 can accurately obtain the action knowledge of the real hand in the training image, so that the virtual human object 310 can accurately simulate the action of the real hand. As a result, the manipulator in the output image 340 can simulate the hand action in the input image 330 in a more accurate and effective way.

[0064] According to an exemplary implementation of the present disclosure, the size of the manipulator in the output image 340 can be adjusted based on the size of the real hand. Assuming that the virtual human object 310 is generated based on a training image set including an adult hand image, the size of the manipulator generated at this time is consistent with the adult hand. In the case where the input image 330 includes a child's hand, if the child's hand is directly replaced with an adult's manipulator hand image, it will result in a situation where the manipulator and the child's body proportions are inconsistent. At this time, the size of the manipulator can be adjusted based on the size of the child's hand in the input image 330 so that the manipulator in the output image 340 can match the child's body proportions.

[0065] will understand, although Figure 7 An example of processing a single input image 330 and replacing the real hand in the input image 330 with a manipulator is shown. Alternatively and / or additionally, a continuous video captured in real time can be input to the action model 320. At this time, the action model 320 can be used to recognize each action in the continuous video in real time. Further, a continuous video including the manipulator performing each action can be output. At this time, the real hand in the input video has been replaced by the manipulator, and the action of the manipulator is consistent with the action of the real hand.

[0066] According to an exemplary implementation of the present disclosure, when processing continuous videos, the moving distance of the character's hand in the input video may be large. In this case, the character's hand area can be replaced based on the background area in the previous and subsequent video frames. In other words, the real hand can be removed from the input video first. Figure 8 Describes the removal of more details of the real hand. Figure 8 Block diagram 800 for generating an intermediate image including a blurred region according to some implementations of the present disclosure is shown.

[0067] like Figure 8 As shown, each video frame (e.g., video frame 812) in the input video 810 may be processed to generate an intermediate image 820. Specifically, the image in the region including the real hand may be processed based on the background image of the ceiling in the video frame before the video frame 812 to generate a blurred region 822. In the blurred region 822, the image of the real hand has been removed, and the portion originally blocked by the real hand has been updated to the image of the ceiling portion. In this way, the image of the real hand may be removed in advance, thereby avoiding the situation where the generated robotic hand cannot completely block the real hand.

[0068] The process of training and using the motion model 320 has been described above. In the above process, if it is found that the motion determined by the motion model 320 cannot accurately reflect the motion of the real hand, a more accurate motion recognition effect can be obtained by adjusting the joint points of the virtual human object 310. Fig. 9 Describes more details of adjusting the virtual human object 310, Fig. 9 Block diagram 900 illustrates a process of adjusting the orientation of a spatial point in a virtual human object according to some implementations of the present disclosure.

[0069] like Fig. 9 As shown, virtual human object 310 represents the original virtual human object established based on image 210, and virtual human object 910 represents the adjusted virtual human object. In virtual human object 310, the bones of the thumb part are approximately kept in a straight line. In the predicted virtual image obtained using motion model 320, assuming that the thumb part presents an erroneous posture of outward rotation, the part in area 920 can be adjusted (as shown by arrow 930). For example, the joint points of the thumb part of virtual human object 310 can be rotated inward, that is, the orientation of each joint point at the end of the thumb is adjusted so that it rotates inward by a predetermined angle (as shown in area 922).

[0070] According to an exemplary implementation of the present disclosure, the above adjustment process can be performed during the training process. Specifically, the predicted virtual image determined by the action model 320 and the part associated with the human body object in the training image can be compared, and if it is found that the occurrence of wrong postures reaches a certain number or proportion, the above adjustment action can be performed. Alternatively and / or additionally, the above adjustment process can also be performed during the use of the action model 320. In this way, data from more input images can be continuously used to further improve the accuracy of action recognition.

[0071] When the adjusted virtual human object 910 is used, since the thumb portion in the area 922 has been rotated inwardly by a predetermined angle, the inward rotation angle can offset the error in the outward rotation angle of the corresponding joint point identified by the action model 320, thereby making the thumb orientation in the output image match the thumb orientation in the input image as much as possible. Using the exemplary implementation of the present disclosure, the posture of the virtual human object 310 can be adjusted to compensate for the deficiencies in the action identified by the action model 320. In this way, the degree of matching between the action of the manipulator in the output image and the real hand in the input image 330 can be improved.

[0072] It will be understood that due to the symmetry of the human body structure, only the left or right virtual human object can be established. When the human object in the input image 330 is identified to be located on the opposite side, the virtual human object can be mirrored to the other side to perform action recognition. Fig.10 Describes more details about the mirroring process. Fig.10 A block diagram 1000 is shown of a process of generating a left-hand virtual human object based on a right-hand virtual human object according to some implementations of the present disclosure.

[0073] The part of the human body object in the input image 330 may be determined first. If the part (e.g., the left hand) is inconsistent with the direction of the generated virtual human body object 310 (e.g., the right hand), the virtual human body object 1010 in another direction (left hand) may be obtained based on the mirroring operation 1012. At this point, the action model 320 and the virtual human body object 1010 may be used to recognize the action of the left hand.

[0074] It will be understood that in the rendering process of some renderers, due to the configuration attribute of using materials with one-way rendering, the rendering directions of the left and right hand materials are opposite. Therefore, it is necessary to update the rendering direction based on the position of the virtual human object. Fig.10 As shown, assuming that the rendering direction 1020 of the virtual human object 310 for the right hand is “front”, the rendering direction 1022 of the virtual human object 1010 for the left hand can be updated to “back”. In this way, it can be ensured that the material of the manipulator is correctly rendered during the rendering process, thereby generating a correct virtual image.

[0075] It will be understood that although the above describes the action recognition process using the right hand as an example of a human object, the above process can be applied to the recognition of other human objects. For example, the movements of the human torso, limb movements, head movements, and the like can be recognized. In addition, although the above describes a specific application environment using the action model 320 using the example of using the recognized hand movements to control a manipulator, the technical solution described above can also be used in other application environments. For example, recognized human movements can be used to perform device control, such as controlling the speaker volume, display brightness, and the like of an electronic device; for another example, recognized human movements can be used to control virtual characters in a game; for another example, various virtual objects can be controlled in augmented reality applications, and the like.

[0076] Example Process

[0077] Fig.11A flow chart of a method 1100 for identifying human motion according to some implementations of the present disclosure is shown. Specifically, at box 1110, it is determined whether an input image including a human object is received. If the input image is received, the method 1100 proceeds to box 1120, and a target action associated with the human object in the input image is determined based on the action model. Here, the action model describes the association between an image including a human object and the action of the human object in the image, and the action is represented by a set of spatial points in a virtual human object corresponding to the human object, and the dimensions of the virtual human object match the dimensions of the human object. At box 1130, the input image is updated using the virtual human object that performs the target action.

[0078] According to an exemplary implementation of the present disclosure, updating the input image includes: determining a target virtual image of a virtual human object based on a target action, the virtual human object being set as the target action; and replacing a portion of the input image associated with the human object with the target virtual image.

[0079] According to an exemplary implementation of the present disclosure, determining a target virtual image includes: determining a part of a human object in an input image; updating a rendering direction of a material of the virtual object based on the part; and generating a target virtual image based on the updated rendering direction.

[0080] According to an exemplary implementation of the present disclosure, the method further includes obtaining the action model by: representing the dimensions of the human object based on a first group of image key points in a first training image including the human object; generating a virtual human object based on the dimensions of the human object, the virtual human object including a group of spatial points corresponding to the first group of image key points respectively; and determining the action model based on the first group of image key points and the group of spatial points.

[0081] According to an exemplary implementation of the present disclosure, generating a virtual human object includes at least any one of the following: determining the positions of a group of spatial points based on the positional relationship of the first group of image key points; and determining the orientation of a group of spatial points based on the orientation relationship of the first group of image key points.

[0082] According to an exemplary implementation of the present disclosure, a first group of image key points respectively correspond to a group of joint points of a human object, and a group of spatial points includes a group of joint points of a virtual human object respectively corresponding to the group of joint points of the human object.

[0083] According to an exemplary implementation of the present disclosure, obtaining an action model includes: determining a predicted action of a human object in a second training image based on the action model and the second training image including the human object; determining a predicted virtual image of a virtual human object based on the predicted action; and updating the action model based on a comparison result between the predicted virtual image and a portion associated with the human object in the second training image.

[0084] According to an exemplary implementation of the present disclosure, determining the predicted action includes: acquiring a set of joint points of the human object in the second training image; and determining the predicted action based on the set of joint points and the action model of the human object in the second training image.

[0085] According to an exemplary implementation of the present disclosure, the predicted action is represented by a set of predicted spatial points of a virtual human object, and determining the predicted virtual image includes: setting the orientation of a set of joint points of the virtual human object to the orientation of a set of predicted spatial points respectively; and determining the predicted virtual image of the virtual human object based on the set set of joint points.

[0086] According to an exemplary implementation of the present disclosure, updating the action model includes: determining a set of predicted image space points in the predicted virtual image that respectively correspond to a set of joint points; and updating the action model based on the distance between the predicted image space points in the set of predicted image space points and the corresponding image key points in the second set of image key points.

[0087] According to an exemplary implementation of the present disclosure, the method 1100 further includes: adjusting the orientation of a group of spatial points of the virtual human object based on a comparison result of the predicted virtual image and the part associated with the human object in the second training image.

[0088] Example devices and equipment

[0089] Fig.12 A block diagram of an apparatus 1200 for recognizing human motion according to some implementations of the present disclosure is shown. The apparatus 1200 includes: a determination module 1210, configured to determine a target motion associated with the human object in the input image based on a motion model in response to receiving an input image including a human object. Here, the motion model describes the association between an image including a human object and the motion of the human object in the image, the motion being represented by a set of spatial points in a virtual human object corresponding to the human object, and the dimension of the virtual human object matches the dimension of the human object. The apparatus 1200 further includes: and an update module 1220, configured to update the input image using the virtual human object that performs the target motion.

[0090] According to an exemplary implementation of the present disclosure, the update module 1220 includes: an image determination module, configured to determine a target virtual image of a virtual human object based on a target action, the virtual human object being set as the target action; and a replacement module, configured to replace a portion of the input image associated with the human object with the target virtual image.

[0091] According to an exemplary implementation of the present disclosure, the image determination module includes: determining the part of the human body object in the input image; updating the rendering direction of the material of the virtual object based on the part; and generating a target virtual image based on the updated rendering direction.

[0092] According to an exemplary implementation of the present disclosure, the device 1200 further includes an acquisition module, and the acquisition module includes: a representation module, configured to represent the dimensions of a human object based on a first group of image key points in a first training image including the human object; a generation module, configured to generate a virtual human object based on the dimensions of the human object, the virtual human object including a group of spatial points corresponding to the first group of image key points respectively; and an action determination module, configured to determine an action model based on the first group of image key points and a group of spatial points.

[0093] According to an exemplary implementation of the present disclosure, the generation module includes: a position determination module, configured to determine the positions of a group of spatial points based on the positional relationship of a first group of image key points; and an orientation determination module, configured to determine the orientation of a group of spatial points based on the orientation relationship of the first group of image key points.

[0094] According to an exemplary implementation of the present disclosure, a first group of image key points respectively correspond to a group of joint points of a human object, and a group of spatial points includes a group of joint points of a virtual human object respectively corresponding to the group of joint points of the human object.

[0095] According to an exemplary implementation of the present disclosure, an acquisition module includes: a prediction module, configured to determine a predicted action of a human object in a second training image based on the action model and a second training image including the human object; an image prediction module, configured to determine a predicted virtual image of a virtual human object based on the predicted action; and a model updating module, configured to update the action model based on a comparison result of the predicted virtual image and a part associated with the human object in the second training image.

[0096] According to an exemplary implementation of the present disclosure, the prediction module includes: a key point acquisition module, configured to acquire a set of joint points of a human object in a second training image; and a joint point-based prediction module, configured to determine a predicted action based on a set of joint points of the human object in the second training image and an action model.

[0097] According to an exemplary implementation of the present disclosure, the predicted action is represented by a set of predicted spatial points of the virtual human object, and the image prediction module includes: a setting module, configured to set the orientation of a set of joint points of the virtual human object as the orientation of a set of predicted spatial points, respectively; and an orientation-based image prediction module, configured to determine the predicted virtual image of the virtual human object based on the set set of joint points.

[0098] According to an exemplary implementation of the present disclosure, the update module includes: a spatial point prediction module, configured to determine a set of predicted image spatial points in a predicted virtual image that respectively correspond to a set of joint points; and a distance-based update module, configured to update the action model based on the distance between the predicted image spatial points in a set of predicted image spatial points and the corresponding image key points in a second set of image key points.

[0099] According to an exemplary implementation of the present disclosure, the device 1200 further includes: an adjustment module configured to adjust the orientation of a group of spatial points of the virtual human object based on a comparison result of the predicted virtual image and the part associated with the human object in the second training image.

[0100] Fig.13 1 shows a block diagram of a device 1300 capable of implementing multiple implementations of the present disclosure. It should be understood that Fig.13 The computing device 1300 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the implementations described herein. Fig.13 The computing device 1300 shown may be used to implement the methods described above.

[0101] like Fig.13 As shown, computing device 1300 is in the form of a general-purpose computing device. Components of computing device 1300 may include, but are not limited to, one or more processors or processing units 1310, memory 1320, storage device 1330, one or more communication units 1340, one or more input devices 1350, and one or more output devices 1360. Processing unit 1310 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 1320. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device 1300.

[0102] The computing device 1300 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device 1300, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1320 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1330 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the computing device 1300.

[0103] The computing device 1300 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig.13 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 1320 may include a computer program product 1325 having one or more program modules that are configured to perform various methods or actions of various implementations of the present disclosure.

[0104] The communication unit 1340 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the computing device 1300 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 1300 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0105] Input device 1350 may be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1360 may be one or more output devices, such as a display, speaker, printer, etc. Computing device 1300 may also communicate with one or more external devices (not shown) through communication unit 1340 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with computing device 1300, or communicate with any device that allows computing device 1300 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0106] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, the computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0107] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0109] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0110] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0111] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for recognizing human body motion, comprising: determining a dimension of the human object based on an image of the human object, the image being selected from training images used to train the motion model; Based on the dimensions of the human object, generating a virtual human object corresponding to the human object, the dimensions of the virtual human object matching the dimensions of the human object, and the dimensions including at least any of the following: size, proportion, and posture; In response to receiving an input image including the human object, determining a target action associated with the human object in the input image based on an action model, the action model describing an association relationship between an image including the human object and an action of the human object in the image, the action being represented by a set of spatial points in a virtual human object corresponding to the human object; as well as The input image is updated using the virtual human object that performs the target action, including: in response to determining that the target action cannot reflect the real action of the human object in the input image, adjusting the orientation of the spatial points in the set of spatial points in the virtual human object so as to offset the error in the target action determined by the action model.

2. The method of claim 1 , wherein updating the input image comprises: determining a target virtual image of the virtual human object based on the target action, the virtual human object being set as the target action; as well as The target virtual image is used to replace a portion of the input image associated with the human object.

3. The method according to claim 2, wherein determining the target virtual image comprises: Determining a location of the human object in the input image; Updating a rendering direction of a material of the virtual human object based on the part; as well as The target virtual image is generated based on the updated rendering direction.

4. The method according to claim 1, further comprising obtaining the motion model by: representing the dimension of the human object based on a first set of image key points in a first training image including the human object; generating the virtual human object based on the dimensions of the human object, the virtual human object comprising the set of spatial points respectively corresponding to the first set of image key points; and The motion model is determined based on the first set of image key points and the set of spatial points.

5. The method according to claim 4, wherein generating the virtual human object comprises at least any one of the following: Determining the positions of the set of spatial points based on the positional relationship of the first set of image key points; and Based on the orientation relationship of the first group of image key points, the orientation of the group of spatial points is determined.

6. The method according to claim 5, wherein the first group of image key points respectively correspond to a group of joint points of the human body object, and the group of spatial points includes a group of joint points of the virtual human body object respectively corresponding to the group of joint points of the human body object.

7. The method according to claim 6, wherein obtaining the motion model comprises: determining a predicted motion of the human object in the second training image based on the motion model and a second training image including the human object; determining a predicted virtual image of the virtual human object based on the predicted motion; as well as The motion model is updated based on a comparison result of the predicted virtual image and a portion associated with the human object in the second training image.

8. The method of claim 7, wherein determining the predicted action comprises: Acquire a set of joint points of the human object in the second training image; as well as The predicted action is determined based on the set of joint points of the human object in the second training image and the action model.

9. The method of claim 7, wherein the predicted motion is represented by a set of predicted spatial points of the virtual human object, and determining the predicted virtual image comprises: Setting the orientations of the group of joint points of the virtual human object to the orientations of the group of predicted space points respectively; as well as The predicted virtual image of the virtual human object is determined based on the set set of joint points.

10. The method of claim 9, wherein updating the motion model comprises: Determine a group of predicted image space points in the predicted virtual image that respectively correspond to the group of joint points; as well as The motion model is updated based on distances between predicted image space points in the set of predicted image space points and corresponding image key points in the key points of the second training image.

11. The method according to claim 7, further comprising: Based on a comparison result of the predicted virtual image and a portion associated with the human object in the second training image, the orientation of the set of spatial points of the virtual human object is adjusted.

12. A device for recognizing human body movements, comprising: a dimension determination module configured to determine the dimension of the human object based on an image of the human object, the image being selected from training images used to train the motion model; an acquisition module configured to generate a virtual human object corresponding to the human object based on the dimensions of the human object, wherein the dimensions of the virtual human object match the dimensions of the human object, and the dimensions include at least any one of the following: size, proportion, and posture; a determination module configured to, in response to receiving an input image including the human object, determine a target action associated with the human object in the input image based on the action model, wherein the action model describes an association relationship between an image including the human object and an action of the human object in the image, wherein the action is represented by a set of spatial points in a virtual human object corresponding to the human object; as well as An updating module is configured to update the input image using the virtual human object that performs the target action, comprising: in response to determining that the target action cannot reflect the real action of the human object in the input image, adjusting the orientation of the spatial points in the set of spatial points in the virtual human object so as to offset the error in the target action determined by the action model.

13. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.

14. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is caused to implement the method according to any one of claims 1 to 11.

Images