Method and apparatus for recognizing human body action, and device and medium
By establishing a virtual human object matching the real hand dimension in a mobile device and updating the input image using the action model, the problem of insufficient accuracy of human body motion recognition in the prior art is solved, and the control accuracy of augmented reality applications is improved.
Patent Information
- Application Number
- PCT/CN2023/113182
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-08-19
- Filing Date
- 2023-08-15
- Publication Date
- 2024-12-05
AI Technical Summary
In the prior art, when recognizing human movements, especially in mobile devices, it is difficult to improve accuracy due to limited computing resources, resulting in insufficient matching between virtual human objects and real hand movements, affecting the effect of augmented reality applications.
By establishing virtual human objects that match the real hand dimension, using spatial points in the action model to represent the action relationship, updating the input image to improve the accuracy of action recognition, and using machine learning technology to train the action model to adapt to different hand postures and proportions .
It improves the accuracy of human body movement recognition and the accuracy of subsequent processing, so that virtual human objects can simulate real hand movements more realistically, and enhances the control accuracy of augmented reality applications.
Smart Images

Figure CN2023113182_05122024_PF_FP_ABST
Abstract
Description
Method, device, apparatus and medium for recognizing human body motion
[0001] This application claims priority to the Chinese invention patent application entitled “Methods, devices, equipment and media for identifying human body motions” and application number 2022109998963, filed on August 19, 2022. The entire contents of that application are incorporated herein by reference. Technical Field
[0002] Exemplary implementations of the present disclosure generally relate to machine learning technology, and more particularly to methods, devices, apparatuses, and computer-readable storage media for recognizing human motions based on machine learning technology. Background Art
[0003] Numerous augmented reality applications have been developed, allowing users to capture images of people in real-world environments using image capture devices such as mobile terminals. Furthermore, machine learning can be used to identify human motions for controlling augmented reality applications and / or other purposes. Given the limited computing resources of mobile devices, improving the accuracy of human motion recognition has become a pressing issue.
[0004] Summary of the Invention
[0005] In a first aspect of the present disclosure, a method for recognizing human motion is provided. In this method, in response to receiving an input image including a human object, a target motion associated with the human object in the input image is determined based on a motion model. The motion model describes an association between the image including the human object and the motion of the human object in the image, where the motion is represented by a set of spatial points in a virtual human object corresponding to the human object, and the dimensions of the virtual human object match the dimensions of the human object. The input image is updated using the virtual human object performing the target motion.
[0006] In a second aspect of the present disclosure, a device for recognizing human actions is provided. The device includes: a determination module configured to, in response to receiving an input image including a human object, determine a target action associated with the human object in the input image based on a motion model, wherein the motion model describes an association relationship between the image including the human object and the action of the human object in the image, wherein the action is represented by a set of spatial points in a virtual human object corresponding to the human object, and the dimensions of the virtual human object match the dimensions of the human object; and an update module configured to update the input image using the virtual human object performing the target action.
[0007] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform the method according to the first aspect of the present disclosure.
[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.
[0009] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the implementation of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent hereinafter with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:
[0011] FIG1 shows an example of human body motion that can be processed by implementations of the present disclosure;
[0012] FIG2 is a block diagram illustrating an example of a training image for training an action model according to some technical solutions;
[0013] FIG3 shows a block diagram of a process for recognizing human motion according to some implementations of the present disclosure;
[0014] FIG4 illustrates a block diagram of a coordinate system of various spatial points in a virtual human object according to some implementations of the present disclosure;
[0015] 5 illustrates a block diagram of a process for setting a virtual human object based on a training image according to some implementations of the present disclosure;
[0016] 6 illustrates a block diagram of a process for updating a motion model based on a comparison of training images and predicted virtual images, according to some implementations of the present disclosure;
[0017] FIG7 illustrates a block diagram of updating a human object in an input image based on a virtual human object according to some implementations of the present disclosure;
[0018] FIG8 illustrates a block diagram of generating an intermediate image including a blurred area according to some implementations of the present disclosure;
[0019] FIG9 illustrates a block diagram of a process for adjusting the orientation of a spatial point in a virtual human object according to some implementations of the present disclosure;
[0020] FIG10 is a block diagram illustrating a process of generating a left-hand virtual human object based on a right-hand virtual human object according to some implementations of the present disclosure;
[0021] FIG11 shows a flow chart of a method for recognizing human motion according to some implementations of the present disclosure;
[0022] FIG12 shows a block diagram of an apparatus for recognizing human motion according to some implementations of the present disclosure; and
[0023] FIG13 illustrates a block diagram of a device capable of implementing various implementations of the present disclosure. DETAILED DESCRIPTION
[0024] The following describes implementations of the present disclosure in more detail with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the implementations described herein. Rather, these implementations are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and implementations of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.
[0025] In the description of the implementation of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "an implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.
[0026] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.
[0027] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0028] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.
[0029] As an optional but non-limiting implementation, in response to receiving a user's active request, a prompt message may be sent to the user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. Furthermore, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.
[0030] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0031] Sample Environment
[0032] Currently, a variety of augmented reality applications have been developed. For example, an image acquisition device can be used to capture and recognize a user's body movements. Furthermore, the recognized body movements can be used to control various objects in the augmented reality application, etc. Below, an overview of recognizing body movements is first described with reference to FIG1 , which shows an example 100 of body movements that can be processed by implementations of the present disclosure.
[0033] It will be understood that a human body object can include various parts of the human body, including but not limited to the torso, limbs, head, hands, etc. In the context of this disclosure, the details of the process of recognizing human body movements will be described using only the hand as an example of a human body object. As shown in Figure 1, the hand can perform a variety of movements. For example, movements 110, 120, and 130 relate to writing movements, and movements 140, 150, and 160 relate to other movements. It will be understood that the movements shown in Figure 1 are merely illustrative, and the user can control their fingers to perform any other movements.
[0034] Currently, technical solutions for recognizing human motions based on machine learning have been proposed. For example, multiple key points can be marked at predetermined locations on the hand. Hand images can be collected and used to train motion models using the marked hand images. Figure 2 shows a block diagram 200 of an example training image for training a motion model according to some technical solutions.
[0035] As shown in Figure 2, image 210 shows an image of the back of the right hand, and key points 0 to 20 (for example, key points can be set at various joints of the hand) can be marked in the hand, and the hand image can be simplified to include a tree structure defined by key points 0 to 20. Image 220 shows an image of the palm of the right hand, and image 220 also includes predefined key points 0 to 20. It will be understood that the positions of the key points in the hand are fixed and will not change with different movements. For example, key point 0 is always located at the wrist joint, and key point 4 is always located at the end of the thumb, etc.
[0036] According to one technical solution, a three-dimensional model of a virtual hand object can be pre-established, and a motion model can be used to represent the correspondence between the key points in the hand image and the spatial positions of the corresponding key points in the three-dimensional model. However, there may be differences between the hand in the hand image used as training data and the hand in the three-dimensional model. Assuming that the hand shape of the three-dimensional model is slender, but the hand shape in the training data is short and thick, this will cause the trained motion model to be unable to accurately recognize hand movements. In other words, when using the trained motion model to recognize hand movements, it may be impossible to accurately determine the key points, resulting in the recognition of incorrect movements. At this time, how to recognize human body movements in a more convenient and effective way has become a problem that needs to be solved urgently.
[0037] Summary of the process of recognizing human actions
[0038] In order to solve the deficiencies in the above-mentioned technical solutions, according to an exemplary implementation of the present disclosure, a method for recognizing human body movements is proposed. In summary, a motion model that describes the association relationship between an image including a human body object and the movement of the human body object in the image can be obtained. It will be understood that the movement here is represented by a set of spatial points in a virtual human body object corresponding to the human body object, and the dimensions of the virtual human body object match the dimensions of the human body object. In the context of the present disclosure, when establishing a virtual object of the hand, it is necessary to consider the dimensions of the hand used as training data. In other words, a three-dimensional virtual human body object of the hand can be established in the same size, proportion and posture based on the dimensions of the hand in the training image (for example, including size, proportion and posture, etc.). According to an exemplary implementation of the present disclosure, the trained motion model can be directly received; alternatively and / or additionally, the motion model can be trained based on machine learning technology.
[0039] 3 , which illustrates a block diagram 300 of a process for recognizing human motions according to some implementations of the present disclosure, provides an overview of an exemplary implementation of the present disclosure. As shown in FIG3 , an input image 330 may be processed based on a motion model 320 that describes an association between an image including a human object and the motion of the human object in the image. Specifically, upon receiving an input image 330 including a human object, a target motion associated with the human object in the input image 330 may be determined based on the motion model 320. Furthermore, the input image may be updated using a virtual human object performing the target motion to generate a new output image 340.
[0040] It will be understood that the action model 320 herein can describe the association between an image including a human object and the action of the human object in the image, the action being represented by a set of spatial points in a virtual human object corresponding to the human object, and the dimensions of the virtual human object match the dimensions of the human object. In order to obtain the action model 320, a set of image key points 212 in the image 210 including the human object can be obtained. According to an exemplary implementation of the present disclosure, an image 210 can be selected from a set of training images, for example, a front image or a back image including a full view of the hand can be selected. It will be understood that each image in the set includes a set of pre-labeled image key points. In this way, the dimensional information of each part of the hand can be obtained as much as possible, thereby establishing a virtual human object with higher accuracy.
[0041] Furthermore, a virtual human object 310 corresponding to the human object can be generated based on the dimensions of the human object. As shown in FIG3 , the virtual human object 310 can include a set of spatial points 312 that correspond to the set of image key points 212 in the image 210. It will be understood that, for simplicity, the set of spatial points 312 in the virtual human object 310 in FIG3 only shows thumb-related spatial points 0' to 4' (corresponding to key points 0 to 4 in the image 210, respectively). The virtual human object 310 can further include other spatial points corresponding to other key points 5 to 20 in the image 210.
[0042] As shown in Figure 3, when creating a virtual human object 310, a virtual hand model with the same size, proportions, and posture as the real hand can be created based on the dimensions of the hand in image 210. Specifically, the virtual human object 310 can be made to overlap with the hand in image 210. That is, when creating the virtual human object 310 for the hand, each joint in the virtual human object 310 can be aligned with each joint in the hand in image 210. This ensures that the orientation of the two is consistent and the length ratio is as similar as possible.
[0043] Furthermore, an action model 320 can be determined based on a set of image key points 212 and a set of spatial points 312. Here, the action model 320 can describe the association between an image including a human object and the action of the human object in the image. It will be understood that the action here is represented using a set of spatial points 312 in the virtual human object 310. That is, by determining a set of spatial points 312, the action of the hand can be determined. Furthermore, the action model 320 can be used to process an input object 330 including the virtual human object 310. Specifically, the virtual human object 310 can be caused to perform a determined action, and the input image 330 can be updated accordingly.
[0044] Using the exemplary implementation of the present disclosure, because virtual human object 310 and the hand in image 210 have the same dimensions, the positions and orientations of a set of image key points 212 and a set of spatial points 312 can be accurately mapped one-to-one. In this way, virtual human object 310 can more realistically simulate the movements of the hand in image 210, thereby improving the accuracy of movement model 320.
[0045] Detailed process of recognizing human movements
[0046] An overview of human motion recognition has been described with reference to FIG3 , and more details on human motion recognition will be provided below. According to an exemplary implementation of the present disclosure, in the process of generating a virtual human object, the size of the virtual human object 310 needs to be considered first. Specifically, the training image can be used to obtain the motion model 320. For example, the positional relationship of a set of image key points 212 can be determined based on the dimensions of the human object in the image 210. In other words, the distance between each key point can be determined, and then the relative position between each spatial point in the virtual human object 310 can be determined. Using the exemplary implementation of the present disclosure, the action can be described based on a virtual human object that is consistent with the size, proportion and posture of the human object. In this way, the accuracy of the motion model 320 can be improved, and the accuracy of subsequent processing can be improved.
[0047] In image 210, it is assumed that the distance between key point 0 and key point 1 is 4 cm, the distance between key point 1 and key point 2 is 5 cm, the distance between key point 2 and key point 3 is 3.5 cm, and the distance between key point 3 and key point 4 is 2.5 cm. The position of each spatial point in a set of spatial points 312 can be determined based on the above distances. In other words, in the generated virtual human object 310, the distance between spatial point 0' and spatial point 1' is 4 cm, the distance between spatial point 1' and spatial point 2' is 5 cm, the distance between spatial point 2' and spatial point 3' is 3.5 cm, and the distance between spatial point 3' and spatial point 4' is 2.5 cm. In this way, it can be ensured that the size of the virtual human object 310 is consistent with the real hand in image 210, thereby improving the accuracy of the action model 320 determined later.
[0048] It will be appreciated that, in addition to the position of the spatial points, the orientation of each spatial point will also affect the pose of the virtual human object 310. It will be appreciated that, because the image key points are located at the joints of the hand in image 210 and the spatial points are located at the joints of the virtual human object 310, the hand pose of the virtual human object 310 cannot be uniquely determined based solely on the relationship between the spatial points. In this case, the orientation of each image key point must also be considered, and the orientation of each spatial point can be determined based on this orientation.
[0049] According to an exemplary implementation of the present disclosure, a reference coordinate system can be established based on a variety of methods to represent the orientation of each spatial point. Below, see FIG4 for more information on determining the orientation of a spatial point. FIG4 shows a block diagram 400 of a coordinate system for each spatial point in a virtual human object according to some implementations of the present disclosure. As shown in FIG4 , a reference coordinate system can be established for each spatial point. Specifically, the orientation of a set of spatial points 312 can be determined based on the orientation relationship of a set of image key points 212 in the dimensions of the human object.
[0050] For example, a coordinate system 410 can be established for the spatial point 0', and the origin of the coordinate system 410 can be located at the position of the spatial point 0'. The coordinate system 410 can be established based on a variety of methods. For example, the direction of the x-axis can be defined as rightward, and the coordinate system 410 can be established based on a left-handed coordinate system and / or a right-handed coordinate system. Further, each spatial point can be processed in a similar manner, and the orientation of each spatial point can be determined. At this time, the directions of the coordinate axes of the various spatial points can be consistent. For example, the x-axis can be rightward, the y-axis can be upward, and the y-axis points to the outside of the screen. Alternatively and / or additionally, the directions of the coordinate axes of the various spatial points can be inconsistent, and the orientation of the spatial point can be determined based on the wrist orientation of the real person in the hand image in the image 210. For example, the x-axis direction of the spatial point 1' can point to the spatial point 0', and the x-axis direction of the spatial point 2' can point to the spatial point 1', and so on.
[0051] According to an exemplary implementation of the present disclosure, the orientation can be represented based on multiple formats. For example, the orientation can be represented based on attitude angles (i.e., yaw angle, pitch angle, roll angle). Alternatively and / or additionally, the orientation can be represented based on a quadruple, and so on. Utilizing the exemplary implementation of the present disclosure, it is possible to ensure that the orientation of each finger of the virtual human object 310 is consistent with the real hand, and it is possible to ensure that the length of each skeletal segment of the virtual human object 310 is also consistent with the real hand. In this way, the virtual human object 310 can be as consistent as possible with the hand in the image 210, thereby improving the accuracy of the subsequent determination of the action model 320.
[0052] According to an exemplary implementation of the present disclosure, a group of key points 212 respectively include a group of joint points of a human body object, and a group of spatial points 312 include a group of joint points of a virtual human body object that respectively correspond to the group of joint points of the human body object. In this way, by setting the key points and spatial points at the joint points of a real hand and a virtual human body object 310, respectively, the virtual human body object 310 can more accurately simulate the movements of a real hand. As the real hand moves, the spatial points can have a corresponding orientation. According to an exemplary implementation of the present disclosure, the threshold of the orientation of the spatial points can be set according to the range of motion limit of the human skeleton. In this way, the virtual human body object 310 can more accurately simulate the movements of a real hand.
[0053] According to an exemplary implementation of the present disclosure, the distance between two spatial points is determined based on the distance between key points, and this distance does not change with hand movements. In other words, the skeleton between two spatial points of the virtual human object 310 can be considered a rigid body, immune to stretching and compression. In this way, only the orientation of each spatial point needs to be considered to accurately describe each spatial point, and thus the overall posture of the virtual human object 310.
[0054] According to an exemplary implementation of the present disclosure, after determining a virtual human object 310 aligned with a hand in image 210, a motion model 320 can be determined using a set of image key points 212 and a set of spatial points 312. In other words, the orientation of each spatial point in virtual human object 310 can be determined based on image 210, thereby obtaining motion model 320. According to an exemplary implementation of the present disclosure, motion model 320 can be obtained based on machine learning techniques. The training process can be performed based on an initial motion model and / or a pre-trained motion model.
[0055] According to an exemplary implementation of the present disclosure, a large number of training images in a training image set can be used to optimize the motion model 320. Specifically, the motion model 320 can be used to process a given training image including a human object, and the motion model 320 can be updated based on a comparison of the processing result with the given training image. In the following, more details according to an exemplary implementation of the present disclosure are described with reference to FIG5 , which shows a block diagram 500 of a process for setting a virtual human object 310 based on a training image according to some implementations of the present disclosure. As shown in FIG5 , a training image 510 can be obtained from a training image set. Specifically, the training image 510 can be input into the motion model 320 to determine a predicted motion 530 of the human object in the training image 510. In this process, a set of joint points in the training image 510 can be obtained, and the set of joint points can be input into the motion model 320 to determine the predicted motion 530.
[0056] It will be understood that the predicted action 530 herein is described using a set of spatial points (i.e., joint points) in the virtual human object 310, and the action model 320 can output the orientation 540 of each spatial point in the virtual human object 310. At this point, the orientation of a set of joint points of the virtual human object 310 can be set as the orientation 540 of the predicted spatial points. In the context of the present disclosure, once the orientation of the joint points is determined, the position of each joint point can be automatically determined based on the distance between each joint point. Thus, the posture of the virtual human object 310 in the three-dimensional virtual space can be determined, that is, the predicted action 530 of the virtual human object 310 can be determined.
[0057] According to an exemplary implementation of the present disclosure, a predicted virtual image of the virtual human object 310 can be determined based on the predicted action 530, that is, the predicted virtual image of the virtual human object 310 can be determined based on the orientation of a set set of joint points. In the following, more details about the predicted virtual image are described with reference to FIG6 , which shows a block diagram 600 of a process for updating the action model based on the comparison of the training image and the predicted virtual image according to some implementations of the present disclosure. As shown in FIG6 , the predicted virtual image 630 refers to the image obtained by projecting the virtual human object 310 in the three-dimensional space into the two-dimensional space. Specifically, the positional relationship between the virtual camera and the virtual human object 310 in the three-dimensional space can be determined based on the spatial positional relationship between the camera and the real hand when the training image 510 is collected, thereby obtaining the predicted virtual image 630.
[0058] Furthermore, the predicted virtual image 630 can be compared with the portion of the training image 510 associated with the human body object (i.e., the positions of the various hand joints in the virtual image 630 and the training image 510 are compared), thereby updating the motion model 320. According to an exemplary implementation of the present disclosure, a loss function can be determined based on the comparison of the two images. The following description will only use the joints associated with the thumb of the hand as an example.
[0059] As shown in Figure 6, the training image 510 may include key points 612, 614, 616, 618, and 620. At this time, the lines connecting the key points can form the relevant bones of the thumb. Similarly, there may be predicted image space points (referred to as predicted points) corresponding to the above-mentioned key points in the predicted virtual image 630. For example, the predicted virtual image 630 may include predicted points 632, 634, 636, 638, and 640 located at the thumb joints of the virtual human object 310. Using the exemplary implementation of the present disclosure, the position of each joint point can describe the outline of the hand movement. By using the predicted points associated with the joint points to describe the action, only the main factors affecting the hand movement can be considered and the secondary factors (e.g., muscle contour points, etc.) can be ignored. In this way, the amount of calculation involved in determining the action model 320 can be reduced and the performance can be improved.
[0060] According to an exemplary implementation of the present disclosure, the aforementioned key points and corresponding predicted points may be compared to determine the distance between them. For example, the distance between the key point 612 at the wrist and the predicted point 632 may be determined, the distance between the key point 612 at the palm and the predicted point 632 may be determined, the distance between the key point 616 at the base of the thumb and the predicted point 636 may be determined, the distance between the key point 618 at the thumb joint and the predicted point 638 may be determined, and so on. A loss function 640 of the motion model 320 may be set based on these distances. For example, the loss function 640 may be set based on the sum, variance, and / or other methods of the respective distances.
[0061] According to an exemplary implementation of the present disclosure, the various parameters in the motion model 320 can be updated in a direction that minimizes the loss function 640. It will be understood that although Figure 6 only shows the joints related to the thumb in the hand, the joints related to other fingers in the hand can be processed in a similar manner, and the distance between the key points and the predicted points at each joint can be determined. In this way, the positions of the various joints in the hand can be comprehensively considered, so that the motion model 320 can more accurately learn the relevant knowledge of the hand movements in the training image 510. According to an exemplary implementation of the present disclosure, the motion model 320 can be continuously trained in an iterative manner. The training stop conditions can be set in advance. For example, the training can be stopped when the number of training times meets a predetermined threshold, the training can be stopped when the motion model 320 meets a threshold convergence condition, and so on.
[0062] The training process for generating the motion model 320 has been described above. After the motion model 320 has been obtained, the motion model 320 can be used to recognize the motion of the human object in the input image 330. Below, more details of processing the input image using the motion model 320 are described with reference to FIG7 , which shows a block diagram of updating the human object in the input image based on the virtual human object according to some implementations of the present disclosure.
[0063] As shown in FIG7 , upon receiving an input image 330 including a hand, the input image 330 may be provided to the motion model 320 to determine the hand's motion 712. Furthermore, the orientation of the joints of the virtual human object 310 may be set based on the hand's motion 712. Subsequently, the virtual human object 310 may be projected into a two-dimensional image space, and a corresponding virtual image 714 may be generated (in this case, the virtual human object 310 is configured to perform the motion 712). The generated virtual image 714 may then be used to replace the portion of the input image 330 associated with the hand, thereby generating the output image 340.
[0064] As shown in FIG7 , the real hand in the output image 340 has been replaced by the robotic hand of the virtual human object 310, and the movement of the robotic hand is determined based on the movement of the real hand in the input image 330. It will be understood that although the processes of recognizing movement, generating a virtual image, and replacing the human object in the input image with the generated virtual image are described separately above, alternatively and / or additionally, the modules for performing the above processes can be integrated into the movement model 320. In this case, the integrated movement model 320 can directly receive the input image 330 and provide the corresponding output image 340.
[0065] Using the exemplary implementation of the present disclosure, the virtual human object 310 used by the action model 320 is established based on the dimensions of the hand in the training image. Therefore, the skeletal size, proportions, and posture of each joint of the virtual human object 310 are consistent with the real hand in the training image. In this way, the action model 320 can accurately obtain the movement knowledge of the real hand in the training image, and then enable the virtual human object 310 to accurately simulate the movement of the real hand. As a result, the robotic hand in the output image 340 can simulate the hand movement in the input image 330 in a more accurate and effective manner.
[0066] According to an exemplary implementation of the present disclosure, the size of the robotic hand in the output image 340 can be adjusted based on the size of the real hand. Assuming that the virtual human object 310 is generated based on a training image set that includes adult hand images, the size of the robotic hand generated at this time is consistent with the adult hand. In the case where the input image 330 includes a child's hand, if the child's hand is directly replaced with an adult's robotic hand image, the proportions of the robotic hand and the child's body will be inconsistent. In this case, the size of the robotic hand can be adjusted based on the size of the child's hand in the input image 330 so that the robotic hand in the output image 340 can match the child's body proportions.
[0067] It will be understood that although FIG. 7 illustrates an example of processing a single input image 330 and replacing a real hand in the input image 330 with a robotic hand, alternatively and / or additionally, a continuous video captured in real time can be input to the action model 320. In this case, the action model 320 can be used to identify each action in the continuous video in real time. Furthermore, a continuous video including a robotic hand performing each action can be output. In this case, the real hand in the input video has been replaced by the robotic hand, and the actions of the robotic hand are consistent with those of the real hand.
[0068] According to an exemplary implementation of the present disclosure, when processing continuous video, the movement distance of a person's hand in the input video may be considerable. In this case, the hand region can be replaced based on the background region in the preceding and following video frames. In other words, the real hand can be first removed from the input video. Further details on removing the real hand are described below with reference to FIG8 , which shows a block diagram 800 for generating an intermediate image including a blurred region according to some implementations of the present disclosure.
[0069] As shown in FIG8 , each video frame (e.g., video frame 812) in input video 810 can be processed to generate intermediate image 820. Specifically, the image within the region including the real hand can be processed based on the background image of the ceiling in the video frame preceding video frame 812 to generate blurred region 822. In blurred region 822, the image of the real hand has been removed, and the portion originally obscured by the real hand has been updated with the image of the ceiling. In this way, the image of the real hand can be pre-removed, thereby avoiding the situation where the generated robotic hand does not completely obscure the real hand.
[0070] The process of training and using motion model 320 has been described above. During this process, if it is found that the motions determined by motion model 320 do not accurately reflect real-world hand motions, more accurate motion recognition can be achieved by adjusting the joints of virtual human object 310. Further details on adjusting virtual human object 310 are described below with reference to FIG9 , which shows a block diagram 900 of the process of adjusting the orientation of spatial points in a virtual human object according to some implementations of the present disclosure.
[0071] As shown in FIG9 , virtual human object 310 represents the original virtual human object created based on image 210, and virtual human object 910 represents the adjusted virtual human object. In virtual human object 310, the bones of the thumb portion remain approximately straight. In the predicted virtual image obtained using motion model 320, assuming that the thumb portion exhibits an incorrect outwardly rotated posture, the portion within region 920 can be adjusted (as indicated by arrow 930). For example, the joints of the thumb portion of virtual human object 310 can be rotated inward, that is, the orientation of each joint at the end of the thumb can be adjusted so that it rotates inward by a predetermined angle (as indicated by region 922).
[0072] According to an exemplary implementation of the present disclosure, the aforementioned adjustment process can be performed during the training process. Specifically, the predicted virtual image determined using the motion model 320 can be compared with the portion associated with the human body object in the training image. If it is found that the occurrence of incorrect postures reaches a certain number or proportion, the aforementioned adjustment action can be performed. Alternatively and / or additionally, the aforementioned adjustment process can also be performed during the use of the motion model 320. In this way, data from more input images can be continuously utilized to further improve the accuracy of motion recognition.
[0073] When using the adjusted virtual human object 910, since the thumb portion within region 922 has been rotated inward by a predetermined angle, this inward rotation angle can offset the error in the outward rotation angle of the corresponding joint point identified by the motion model 320, thereby ensuring that the thumb orientation in the output image matches the thumb orientation in the input image as closely as possible. Using the exemplary implementation of the present disclosure, the posture of the virtual human object 310 can be adjusted to compensate for deficiencies in the motion identified by the motion model 320. In this way, the degree of motion matching between the robotic hand in the output image and the real hand in the input image 330 can be improved.
[0074] It will be appreciated that due to the symmetry of the human body structure, only the left or right virtual human object can be created. When the human object in the input image 330 is identified as being on the opposite side, the virtual human object can be mirrored to the other side to perform action recognition. More details about the mirroring process are described below with reference to FIG10 , which shows a block diagram 1000 of a process for generating a left-hand virtual human object based on a right-hand virtual human object according to some implementations of the present disclosure.
[0075] First, the part of the human body object in the input image 330 can be determined. If the part (e.g., the left hand) is inconsistent with the orientation of the generated virtual human body object 310 (e.g., the right hand), a virtual human body object 1010 in the other orientation (the left hand) can be obtained based on a mirroring operation 1012. At this point, the motion model 320 and the virtual human body object 1010 can be used to recognize the motion of the left hand.
[0076] It will be appreciated that during the rendering process of some renderers, due to the configuration property of using materials with unidirectional rendering, the rendering directions of the left and right hand materials are opposite. Therefore, it is necessary to update the rendering direction based on the position of the virtual human object. As shown in Figure 10, assuming that the rendering direction 1020 of the virtual human object 310 for the right hand is "front", the rendering direction 1022 of the virtual human object 1010 for the left hand can be updated to "back". In this way, the materials of the manipulator can be correctly rendered during the rendering process, thereby generating a correct virtual image.
[0077] It will be understood that although the above describes the motion recognition process using the right hand as an example of a human object, the above process can be applied to the recognition of other human objects. For example, the movements of the human torso, limb movements, head movements, etc. can be recognized. In addition, although the above describes a specific application environment using the motion model 320 using the example of using the recognized hand movements to control a manipulator, the technical solution described above can also be used in other application environments. For example, the recognized human body movements can be used to perform device control, such as controlling the speaker volume, display brightness, etc. of an electronic device; for another example, the recognized human body movements can be used to control virtual characters in a game; for another example, various virtual objects can be controlled in augmented reality applications, and so on.
[0078] Example Process
[0079] FIG11 shows a flowchart of a method 1100 for recognizing human motion according to some implementations of the present disclosure. Specifically, at box 1110, it is determined whether an input image including a human object is received. If an input image is received, the method 1100 proceeds to box 1120, and a target motion associated with the human object in the input image is determined based on a motion model. Here, the motion model describes the association between an image including a human object and the motion of the human object in the image, and the motion is represented by a set of spatial points in a virtual human object corresponding to the human object, and the dimensions of the virtual human object match the dimensions of the human object. At box 1130, the input image is updated using the virtual human object that performs the target motion.
[0080] According to an exemplary implementation of the present disclosure, updating the input image includes: determining a target virtual image of a virtual human object based on a target action, the virtual human object being set as the target action; and replacing a portion of the input image associated with the human object with the target virtual image.
[0081] According to an exemplary implementation of the present disclosure, determining a target virtual image includes: determining a part of a human object in an input image; updating a rendering direction of a material of the virtual object based on the part; and generating a target virtual image based on the updated rendering direction.
[0082] According to an exemplary implementation of the present disclosure, the method further includes obtaining the action model by: representing the dimensions of the human body object based on a first set of image key points in a first training image including the human body object; generating a virtual human body object based on the dimensions of the human body object, the virtual human body object including a set of spatial points corresponding to the first set of image key points respectively; and determining the action model based on the first set of image key points and the set of spatial points.
[0083] According to an exemplary implementation of the present disclosure, generating a virtual human object includes at least any one of the following: determining the positions of a group of spatial points based on the positional relationship of the first group of image key points; and determining the orientation of a group of spatial points based on the orientation relationship of the first group of image key points.
[0084] According to an exemplary implementation of the present disclosure, a first group of image key points respectively correspond to a group of joint points of a human body object, and a group of spatial points includes a group of joint points of a virtual human body object respectively corresponding to the group of joint points of the human body object.
[0085] According to an exemplary implementation of the present disclosure, obtaining an action model includes: determining a predicted action of a human object in a second training image based on the action model and a second training image including the human object; determining a predicted virtual image of the virtual human object based on the predicted action; and updating the action model based on a comparison result between the predicted virtual image and a part associated with the human object in the second training image.
[0086] According to an exemplary implementation of the present disclosure, determining the predicted action includes: obtaining a set of joint points of the human object in the second training image; and determining the predicted action based on the set of joint points and the action model of the human object in the second training image.
[0087] According to an exemplary implementation of the present disclosure, the predicted action is represented by a set of predicted spatial points of a virtual human object, and determining the predicted virtual image includes: setting the orientation of a set of joint points of the virtual human object to the orientation of a set of predicted spatial points respectively; and determining the predicted virtual image of the virtual human object based on the set set of joint points.
[0088] According to an exemplary implementation of the present disclosure, updating the action model includes: determining a set of predicted image space points in the predicted virtual image that respectively correspond to a set of joint points; and updating the action model based on the distance between the predicted image space points in the set of predicted image space points and the corresponding image key points in the second set of image key points.
[0089] According to an exemplary implementation of the present disclosure, the method 1100 further includes adjusting the orientation of a group of spatial points of the virtual human object based on a comparison result of the predicted virtual image and the portion associated with the human object in the second training image.
[0090] Example devices and equipment
[0091] FIG12 shows a block diagram of an apparatus 1200 for recognizing human motion according to some implementations of the present disclosure. The apparatus 1200 includes: a determination module 1210 configured to, in response to receiving an input image including a human object, determine a target motion associated with the human object in the input image based on a motion model. Here, the motion model describes the association between an image including a human object and the motion of the human object in the image, and the motion is represented by a set of spatial points in a virtual human object corresponding to the human object, and the dimensions of the virtual human object match the dimensions of the human object. The apparatus 1200 further includes: and an update module 1220 configured to update the input image using the virtual human object that performs the target motion.
[0092] According to an exemplary implementation of the present disclosure, the update module 1220 includes: an image determination module, configured to determine a target virtual image of a virtual human object based on a target action, the virtual human object being set as a target action; and a replacement module, configured to replace a portion of the input image associated with the human object with the target virtual image.
[0093] According to an exemplary implementation of the present disclosure, the image determination module includes: determining a part of a human object in an input image; updating a rendering direction of a material of a virtual object based on the part; and generating a target virtual image based on the updated rendering direction.
[0094] According to an exemplary implementation of the present disclosure, the device 1200 further includes an acquisition module, and the acquisition module includes: a representation module, configured to represent the dimensions of a human object based on a first group of image key points in a first training image including the human object; a generation module, configured to generate a virtual human object based on the dimensions of the human object, the virtual human object including a group of spatial points corresponding to the first group of image key points respectively; and an action determination module, configured to determine an action model based on the first group of image key points and a group of spatial points.
[0095] According to an exemplary implementation of the present disclosure, the generation module includes: a position determination module, configured to determine the positions of a group of spatial points based on the positional relationship of a first group of image key points; and an orientation determination module, configured to determine the orientation of a group of spatial points based on the orientation relationship of the first group of image key points.
[0096] According to an exemplary implementation of the present disclosure, a first group of image key points respectively correspond to a group of joint points of a human body object, and a group of spatial points includes a group of joint points of a virtual human body object respectively corresponding to the group of joint points of the human body object.
[0097] According to an exemplary implementation of the present disclosure, an acquisition module includes: a prediction module configured to determine a predicted action of a human object in a second training image based on the action model and a second training image including the human object; an image prediction module configured to determine a predicted virtual image of a virtual human object based on the predicted action; and a model updating module configured to update the action model based on a comparison result of the predicted virtual image and a part associated with the human object in the second training image.
[0098] According to an exemplary implementation of the present disclosure, the prediction module includes: a key point acquisition module, configured to acquire a set of joint points of a human object in a second training image; and a joint point-based prediction module, configured to determine a predicted action based on a set of joint points of the human object in the second training image and an action model.
[0099] According to an exemplary implementation of the present disclosure, the predicted action is represented by a set of predicted spatial points of a virtual human object, and the image prediction module includes: a setting module, configured to set the orientation of a set of joint points of the virtual human object to the orientation of a set of predicted spatial points respectively; and an orientation-based image prediction module, configured to determine the predicted virtual image of the virtual human object based on the set set of joint points.
[0100] According to an exemplary implementation of the present disclosure, the update module includes: a spatial point prediction module, configured to determine a set of predicted image spatial points in a predicted virtual image that respectively correspond to a set of joint points; and a distance-based update module, configured to update the action model based on the distance between the predicted image spatial points in the set of predicted image spatial points and the corresponding image key points in the second set of image key points.
[0101] According to an exemplary implementation of the present disclosure, the device 1200 further includes: an adjustment module configured to adjust the orientation of a group of spatial points of the virtual human object based on a comparison result of the predicted virtual image and the part associated with the human object in the second training image.
[0102] FIG13 shows a block diagram of a device 1300 capable of implementing various implementations of the present disclosure. It should be understood that the computing device 1300 shown in FIG13 is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. The computing device 1300 shown in FIG13 can be used to implement the methods described above.
[0103] As shown in Figure 13, computing device 1300 is in the form of a general-purpose computing device. Components of computing device 1300 may include, but are not limited to, one or more processors or processing units 1310, memory 1320, storage device 1330, one or more communication units 1340, one or more input devices 1350, and one or more output devices 1360. Processing unit 1310 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 1320. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 1300.
[0104] The computing device 1300 typically includes a plurality of computer storage media. Such media can be any available media accessible to the computing device 1300, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1320 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1330 can be a removable or non-removable medium and can include a machine-readable medium such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the computing device 1300.
[0105] The computing device 1300 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 13 , a disk drive for reading from or writing to a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 1320 may include a computer program product 1325 having one or more program modules configured to perform various methods or actions of various implementations of the present disclosure.
[0106] The communication unit 1340 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device 1300 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 1300 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.
[0107] Input device 1350 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 1360 may be one or more output devices, such as a display, speaker, or printer. Computing device 1300 may also communicate with one or more external devices (not shown) via communication unit 1340, as needed, such as storage devices, display devices, or the like, with one or more devices that allow a user to interact with computing device 1300, or with any device that allows computing device 1300 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).
[0108] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.
[0109] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.
[0110] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0111] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0112] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.
[0113] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.
Claims
1. A method for recognizing human body motion, comprising: In response to receiving an input image including the human object, determining a target action associated with the human object in the input image based on an action model, the action model describing an association relationship between an image including the human object and an action of the human object in the image, the action being represented by a set of spatial points in a virtual human object corresponding to the human object, and the dimensions of the virtual human object matching the dimensions of the human object; as well as The input image is updated using the virtual human object performing the target action.
2. The method of claim 1 , wherein updating the input image comprises: determining a target virtual image of the virtual human object based on the target action, the virtual human object being set as the target action; as well as The target virtual image is used to replace a portion of the input image associated with the human object.
3. The method according to claim 2, wherein determining the target virtual image comprises: Determining a location of the human object in the input image; updating a rendering direction of a material of the virtual object based on the part; as well as The target virtual image is generated based on the updated rendering direction.
4. The method according to claim 1, further comprising obtaining the motion model by: representing the dimension of the human object based on a first set of image key points in a first training image including the human object; generating the virtual human object based on the dimensions of the human object, the virtual human object comprising the set of spatial points respectively corresponding to the first set of image key points; and The motion model is determined based on the first set of image key points and the set of spatial points.
5. The method according to claim 4, wherein generating the virtual human object comprises at least any one of the following: Determining the positions of the set of spatial points based on the positional relationship of the first set of image key points; and Based on the orientation relationship of the first group of image key points, the orientation of the group of spatial points is determined.
6. The method according to claim 5, wherein the first group of image key points respectively correspond to a group of joint points of the human body object, and the group of spatial points includes a group of joint points of the virtual human body object respectively corresponding to the group of joint points of the human body object.
7. The method according to claim 6, wherein obtaining the motion model comprises: determining a predicted motion of the human object in the second training image based on the motion model and a second training image including the human object; determining a predicted virtual image of the virtual human object based on the predicted motion; as well as The motion model is updated based on a comparison result of the predicted virtual image and a portion associated with the human object in the second training image.
8. The method of claim 7, wherein determining the predicted action comprises: Acquire a set of joint points of the human object in the second training image; as well as The predicted action is determined based on the set of joint points of the human object in the second training image and the action model.
9. The method of claim 7, wherein the predicted motion is represented by a set of predicted spatial points of the virtual human object, and determining the predicted virtual image comprises: Setting the orientations of the group of joint points of the virtual human object to the orientations of the group of predicted space points respectively; as well as The predicted virtual image of the virtual human object is determined based on the set set of joint points.
10. The method of claim 9, wherein updating the motion model comprises: Determine a group of predicted image space points in the predicted virtual image that respectively correspond to the group of joint points; as well as The motion model is updated based on distances between predicted image space points in the set of predicted image space points and corresponding image key points in the second set of image key points.
11. The method according to claim 7, further comprising: Based on a comparison result of the predicted virtual image and a portion associated with the human object in the second training image, the orientation of the set of spatial points of the virtual human object is adjusted.
12. A device for recognizing human body movements, comprising: a determination module configured to, in response to receiving an input image including the human object, determine a target action associated with the human object in the input image based on the action model, wherein the action model describes an association relationship between an image including the human object and an action of the human object in the image, the action being represented by a set of spatial points in a virtual human object corresponding to the human object, and a dimension of the virtual human object matching a dimension of the human object; as well as An updating module is configured to update the input image using the virtual human object performing the target action.
13. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.
14. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is caused to implement the method according to any one of claims 1 to 11.
