Motion capture method and device

Abstract

The invention provides a motion capture method and device. The method comprises the following steps: acquiring a first image; performing limb key point detection on a target in the first image to obtain at least two limb key points, and determining a target area representing a specific part of the target in the first image; determining at least two region key points in the target region; and based on the at least two limb key points and the at least two region key points, capturing an action of the target in an image acquired after the first image. The invention can reduce the consumption of system resources.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a motion capture method and device. Background Art

[0002] With the rapid development of computer vision and digital technology, motion capture has become a popular technology in digital entertainment, virtual reality, and medical rehabilitation. Motion capture captures human movements in real time and converts them into digital signals. It is primarily used in game production, film and television special effects, and medical rehabilitation.

[0003] In the existing technology, a single red, green, and blue (RGB) camera can be used to collect human motion data. After cleaning and annotating the data to obtain a human body cropped image, the global human joint image, global gesture image, local joint image, and local gesture image are determined based on the cropped image. These images are fed into the mobilenet network training to extract the human facial expression and posture information, body movement information, gesture movement information, and root position 3D information, thereby performing human motion capture.

[0004] However, in the above method, since body movements, left and right hand movements, facial expressions, etc. are captured and detected through human body cutouts, system resources are consumed greatly. Summary of the Invention

[0005] The present invention provides a motion capture method and device, which are used to solve the defect of large system resource consumption in the prior art and achieve the purpose of reducing system resource consumption.

[0006] The present invention provides a motion capture method, comprising:

[0007] acquiring a first image;

[0008] Performing limb key point detection on the target in the first image to obtain at least two limb key points, and determining a target area representing a specific part of the target in the first image;

[0009] Determining at least two key points in the target area;

[0010] The movement of the target in an image collected after the first image is captured based on the at least two limb key points and the at least two region key points.

[0011] According to a motion capture method provided by the present invention, determining a target area representing a specific part of the target in the first image includes:

[0012] Performing region detection on the first image to obtain an initial region representing the specific part;

[0013] Acquire a first limb key point and a second limb key point of the specific part from the at least two limb key points;

[0014] determining a correction factor based on the position of the first limb key point and the position of the second limb key point;

[0015] The initial area is corrected based on the correction factor to obtain the target area.

[0016] According to a motion capture method provided by the present invention, the first limb key point is a non-edge point of the specific part;

[0017] The correcting the initial area based on the correction factor to obtain the target area includes:

[0018] When a first distance between the center point of the initial region and the first limb key point is less than the correction factor, updating the center point based on a position of the center point of the initial region and a position of the first limb key point;

[0019] updating the center point based on the first limb key point when a first distance between the center point of the initial area and the first limb key point is greater than or equal to the correction factor;

[0020] The target area is determined based on the updated center point, the position of the first limb key point, the position of the second limb key point and the correction factor.

[0021] According to a motion capture method provided by the present invention, determining at least two key points in the target area includes:

[0022] Performing key point detection on a sub-image corresponding to the target area in the first image to obtain at least two initial area key points;

[0023] Determining a quality score corresponding to the sub-image based on the sub-image and the at least two initial region key points;

[0024] When the quality score is less than a preset score, searching for a second image corresponding to the sub-image from an image library corresponding to the specific part;

[0025] replacing the sub-image with the second image to obtain an updated target area;

[0026] The at least two region key points are determined based on the updated target region.

[0027] According to a motion capture method provided by the present invention, searching for a second image corresponding to the sub-image from an image library corresponding to a specific part includes:

[0028] For each of the initial region key points, determining a target key point corresponding to the initial region key point in each third image in the image library;

[0029] respectively determining second distances between the initial region key points and the target key points in each of the third images;

[0030] For each of the third images, determining an average distance corresponding to the third image based on a second distance between each of the initial region key points and a corresponding target key point in the third image;

[0031] Based on the average distance corresponding to each of the third images, a second image corresponding to the sub-image is determined.

[0032] According to a motion capture method provided by the present invention, capturing the motion of the target in an image captured after the first image based on the at least two limb key points and the at least two region key points includes:

[0033] Inputting the at least two limb key points and the at least two region key points into a key point tracker, and obtaining predicted regions of each limb key point and each region key point output by the key point tracker in a next frame of the first image; the key point tracker is trained based on sample images and constraints determined based on body feature parameters;

[0034] Performing heat map regression on the predicted area to obtain a key point heat map;

[0035] The action of the target in the frame image after the next frame image is captured based on the key point heat map.

[0036] According to a motion capture method provided by the present invention, capturing the motion of the target in an image captured after the first image based on the at least two limb key points and the at least two region key points includes:

[0037] Determining a parameter value of an evaluation parameter based on a position of a predicted key point predicted in a next frame of the first image, the evaluation parameter comprising at least one of the following: a state evaluation parameter, an action correlation evaluation parameter, and a physiological characteristic evaluation parameter; the predicted key point comprising a limb prediction key point and / or a region prediction key point;

[0038] Determining action evaluation values ​​of the objects represented by all the predicted key points in the next frame of image based on parameter values ​​of the evaluation parameters;

[0039] When the action evaluation value is greater than a preset value, performing action tracking on the target in a frame image subsequent to the next frame image based on the predicted key points in the next frame image;

[0040] When the action evaluation value is less than or equal to the preset value, limb key point detection is performed on the target in a frame image subsequent to the next frame image.

[0041] According to a motion capture method provided by the present invention, the evaluation parameters include state evaluation parameters;

[0042] The step of determining a parameter value of an evaluation parameter based on a position of a predicted key point predicted in a next frame of an image of the first image comprises:

[0043] For each of two adjacent target image frames within a preset window, determining the instantaneous speed of the target in the target image after the two adjacent target image frames based on the positions of corresponding key points in the two adjacent target image frames, and determining the predicted speed of the target in the next target image frame based on the positions of key points in the first image and the positions of predicted key points in the next target image frame after the first image; the target image frame includes the first image and a preset number of image frames acquired before the first image;

[0044] Determining the target speed corresponding to the next frame image based on the instantaneous speed of the target in each of the subsequent target images and the predicted speed of the target in the next frame image;

[0045] The parameter value of the state evaluation parameter is determined based on the target speed corresponding to the next frame image and the target speed corresponding to the first image.

[0046] According to a motion capture method provided by the present invention, the evaluation parameters include motion correlation evaluation parameters;

[0047] The step of determining a parameter value of an evaluation parameter based on a position of a predicted key point predicted in a next frame of an image of the first image comprises:

[0048] For each predicted key point in the next frame image, determining a vector angle corresponding to the key point based on a position of the predicted key point in the next frame image and a position of the corresponding key point in the first image;

[0049] The parameter value of the action correlation evaluation parameter is determined based on the vector angle corresponding to each key point and the length of the specific part in the next frame image.

[0050] The present invention also provides a motion capture device, comprising:

[0051] An acquisition module, configured to acquire at least one frame of a first image;

[0052] A detection module, configured to perform limb key point detection on a target in each frame of the first image to obtain at least two limb key points;

[0053] a determination module, configured to determine a target region representing a specific portion of the target in the first image;

[0054] The determining module is further configured to determine at least two key points within the target area;

[0055] A tracking module is used to capture the movement of the target in the image collected after the first image based on the at least two limb key points and the at least two area key points corresponding to the first image of each frame.

[0056] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described motion capture methods when executing the program.

[0057] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-described motion capture methods when executed by a processor.

[0058] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the motion capture methods described above.

[0059] The motion capture method and device provided by the present invention, after acquiring a first image, performs limb key point detection on the target in the first image to obtain at least two limb key points, and determines a target area representing a specific part of the target in the first image, and determines at least two regional key points within the target area, so as to capture the motion of the target in an image collected after the first image based on the at least two limb key points and the at least two regional key points. Since, after determining the target area representing the specific part of the target, only the pixels within the target area are detected to determine the at least two regional key points within the target area, the phenomenon of needing to perform pixel detection on the entire image in the prior art is avoided, thereby effectively reducing the resource consumption of the system. In addition, since only the pixels within the target area are detected, the real-time performance and accuracy of motion capture can also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 A schematic diagram of a flow chart of a motion capture method provided by an embodiment of the present invention;

[0062] Figure 2 A schematic diagram of a process for performing key point detection on a first image according to an embodiment of the present invention;

[0063] Figure 3 A schematic diagram of facial key points provided by an embodiment of the present invention;

[0064] Figure 4 A schematic diagram of key points of a hand provided by an embodiment of the present invention;

[0065] Figure 5 A schematic diagram of human body spatial constraints provided by an embodiment of the present invention;

[0066] Figure 6 A schematic diagram of motion capture provided by an embodiment of the present invention;

[0067] Figure 7 A schematic structural diagram of a motion capture device provided by an embodiment of the present invention;

[0068] Figure 8 The following is a schematic diagram of the physical structure of an electronic device. DETAILED DESCRIPTION

[0069] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0070] Currently, full-body motion capture is increasingly being applied in digital entertainment, virtual reality, medical rehabilitation, and video surveillance. In existing technologies, full-body motion capture involves cleaning and annotating the motion data collected by a camera to obtain a cropped image of the entire body. This cropped image then determines a global joint map, a global gesture map, local joint maps, and local gesture maps. These maps are then fed into a mobilenet network for training, extracting facial expression and posture information, body movement information, gesture information, and root position 3D information for motion capture.

[0071] However, in the above method, body movements, left and right hand movements, and facial expressions are captured and detected as three subtasks on the entire image. However, since the hands and face occupy a small pixel area in the image, if motion capture is performed by regression on the entire image, the consumption of computing resources is huge, resulting in poor real-time performance of motion capture and the accuracy of motion capture cannot be guaranteed.

[0072] In order to solve the above problems, an embodiment of the present invention proposes a motion capture method, in which at least two limb key points are determined by detecting limb key points of the target in each acquired first image frame. In addition, a target area representing a specific part of the target in the first image can also be detected, and the target area can be detected to determine at least two regional key points within the target area, so as to capture the motion of the above-mentioned target in the image collected after the first image based on the at least two limb key points and at least two regional key points obtained by detection. Since after determining the target area representing the specific part of the target, only the pixels within the target area are detected to determine at least two regional key points within the target area, the phenomenon of needing to perform pixel detection on the entire image in the prior art is avoided, thereby effectively reducing the resource consumption of the system. In addition, since only the pixels within the target area are detected, the real-time performance of motion capture can also be improved, and the accuracy of motion capture is also greatly improved.

[0073] In entertainment scenarios, such as somatosensory games and virtual anchor applications, the motion capture method in the embodiments of the present invention can achieve real-time capture of human body movements and subsequent human body movement reconstruction, with the advantages of low cost, strong real-time performance, strong robustness and smooth movements.

[0074] The following combination Figures 1 to 5The motion capture method provided by an embodiment of the present invention is described. The embodiment of the present invention can be applied to scenarios where the motion of a target is captured, for example, in scenarios such as video surveillance, game production, film and television special effects, and medical rehabilitation. The execution subject of this method can be an electronic device such as an intelligent camera, a computer, a server, a server cluster, or a specially designed motion capture device, or a motion capture device provided in the electronic device. The motion capture device can be implemented by software, hardware, or a combination of both.

[0075] Figure 1 A flow chart of a motion capture method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0076] Step 101: Acquire a first image.

[0077] In this step, a standard monocular camera can be used to capture the first image. Alternatively, the first image can be captured using other cameras, such as a binocular camera. It should be understood that when multiple first images are captured, these multiple first images are sequentially continuous, thereby ensuring the continuity of the captured motion.

[0078] Step 102: Perform limb key point detection on the target in the first image to obtain at least two limb key points, and determine a target area representing a specific part of the target in the first image.

[0079] In this step, the target may include a human body. The specific parts may include hands and / or face. Of course, the specific parts may also be set to other parts, such as arms, legs, or torso, depending on the application scenario of motion capture. In the embodiment of the present invention, considering that the fingers of the hands usually have refined movements and the face has changes in expression, the hands and face play a very important role in capturing the whole body movements of the target. Therefore, the specific parts in the embodiment of the present invention include hands and face. When the specific parts are other parts, the specific implementation method is similar to the implementation method when the specific parts include hands and face, and will not be repeated in the embodiment of the present invention.

[0080] Specifically, Figure 2 A schematic diagram of a process for detecting key points in a first image according to an embodiment of the present invention is shown in FIG. Figure 2 As shown, for each acquired first image, the first image can be input into the backbone network for image feature extraction, and the extracted features are respectively input into the hand and face detector and the posture estimator. It should be noted that in order to meet real-time requirements, the backbone network can use a lightweight network, such as mobileNet.

[0081] The hand-face detector can detect the face and hand in the first image and mark the target area of ​​the face and the target area of ​​the hand. Among them, the face, as the most critical factor in identifying the human body, can be used as a direct judgment standard to determine whether there is a human target in the first image. Therefore, if the face cannot be detected in the first image, the subsequent detection operation of the hand-face area and the limb key points will not be performed on the first image, and face detection will be performed after re-acquiring the next frame of image. This can avoid subsequent invalid detection or false detection when there is no human target in the first image, which can save system resources and performance.

[0082] When a target is detected in the first image, a posture estimator is used to detect limb key points to obtain at least two limb key points. For example, the posture estimator can detect 28 limb key points of the target. These limb key points are used to establish a hinge structure, which can effectively describe the target's body characteristics and posture movements.

[0083] Among them, in order to meet the real-time requirements, the hand-face detector can also be trained using a lightweight detection model.

[0084] Step 103: Determine at least two region key points within the target region.

[0085] In this step, after determining the target area, a sub-image corresponding to the target area can be cropped from the first image, and the sub-image can be input into the key point detection model to obtain at least two regional key points in the target area. For example, if the specific part is the face, the key point detection model can be a facial key point detection model, that is, the sub-image of the target area corresponding to the face is input into the facial key point detection model to obtain at least two facial key points. For example, Figure 3 A schematic diagram of facial key points provided by an embodiment of the present invention, such as Figure 3 As shown in FIG, 40 facial key points can be obtained through the facial key point detection model, and the facial movements and expressions of the human body are represented based on the three-dimensional coordinates of these 40 facial key points.

[0086] If the specific part is a hand, the key point detection model may be a hand key point detection model, that is, a sub-image of the target area corresponding to the hand is input into the hand key point detection model to obtain at least two hand key points. For example, Figure 4 A schematic diagram of the key points of the hand provided by an embodiment of the present invention, such as Figure 4 As shown in FIG, 42 hand key points can be obtained through the hand key point detection model, and the three-dimensional coordinates of these 42 hand key points are used to represent the hand movements of the human body. Among them, the face key points and the hand key points can be understood as the regional key points described in the embodiment of the present invention.

[0087] It should be noted that the facial key points and hand key points can also be calibrated by other methods, as long as the key points in the facial area and the hand area can be calibrated. The embodiment of the present invention does not limit the specific calibration method of the facial key points and the hand key points and the number of calibrated key points.

[0088] In addition, because hands and faces often occupy smaller pixels in the first image, a spatial attention module can be added to the key point detection model to improve the accuracy and stability of regional key points, that is, hand and face key point detection.

[0089] Step 104: Capture the motion of the target in an image collected after the first image based on at least two limb key points and at least two region key points.

[0090] In this step, motion capture can be understood as a technique for capturing the body's posture and movements over a period of time. After determining at least two limb key points and at least two region key points within the target region in each frame of the first image, limb movements can be captured based on the limb key points, and hand movements and facial movements and expressions can be captured based on the at least two region key points. Motion capture can include motion capture using tracking methods or motion capture using key point detection methods.

[0091] The image acquired after the first image may include the next frame of image of the first image, or may include the next frame of image of the next frame of image.

[0092] The motion capture method provided by an embodiment of the present invention performs limb key point detection on a target in the first image after acquiring a first image, obtains at least two limb key points, determines a target area representing a specific part of the target in the first image, and determines at least two regional key points within the target area, so as to capture the motion of the target in an image collected after the first image based on the at least two limb key points and the at least two regional key points. Since, after determining the target area representing the specific part of the target, only the pixels within the target area are detected to determine at least two regional key points within the target area, the phenomenon of needing to perform pixel detection on the entire image in the prior art is avoided, thereby effectively reducing the resource consumption of the system and improving the efficiency of motion capture. In addition, since only the pixels within the target area are detected, the real-time and accuracy of motion capture can also be improved.

[0093] Illustratively, based on the above embodiments, when determining the target area representing a specific part of the target in the first image, area detection can be performed on the first image to obtain an initial area representing the specific part, and the first limb key point and the second limb key point of the specific part can be obtained from at least two limb key points. Based on the position of the first limb key point and the position of the second limb key point, a correction factor is determined, and then the initial area is corrected based on the correction factor to obtain the target area.

[0094] Specifically, the specific parts include hands and faces. Therefore, the first image can be subjected to regional detection by a hand-face detector to obtain an initial region representing the specific parts. The above-mentioned region detection process can be understood as a coarse detection process, which can only detect the approximate areas where the hands and faces are located, that is, the output bounding box cannot completely surround the hand and face areas, or the obtained initial area cannot contain the complete hands and face, which will cause the subsequent detection of regional key points to be abnormal. Therefore, in order to improve the accuracy of regional key points, it is also necessary to optimize the bounding box obtained in the coarse detection process. It should be understood that the area where the above-mentioned bounding box is located is also the initial area of ​​the specific part. Optimizing the bounding box means optimizing the position of the initial area.

[0095] In addition, a first limb key point and a second limb key point of a specific part can also be obtained from at least two limb key points. If the specific part is a hand, the first limb key point can be a non-edge point of the hand, such as a palm key point, and the second limb key point can be an edge point of the hand, such as a wrist key point. If the specific part is a face, the first limb key point can be a neck key point, and the second limb key point can be a head key point. Since the position of each limb key point can be obtained by the posture estimator, after obtaining the first limb key point and the second limb key point, the position of these two limb key points can be obtained. In addition, based on the position of the first limb key point and the position of the second limb key point, the rotation angle of the specific part, such as the rotation angle of the head or the rotation angle of the hand, can also be determined.

[0096] Furthermore, it is necessary to determine a correction factor based on the position of the first limb key point and the position of the second limb key point. The correction factor is used to correct the initial area detected by the hand-face detector. Specifically, taking the first limb key point as the palm key point and the second limb key point as the wrist key point as an example, assuming that the coordinates of the palm key point are (x palm ,y palm ), the coordinates of the wrist key point are (x wrist ,y wrist ), in a possible implementation, the distance D from the wrist key point to the palm key point can be determined according to the following formula (1):distance :

[0097]

[0098] Furthermore, based on D distance Determine the correction factor K corresponding to the initial area of ​​the hand. For example, the correction factor K can be set to K=n*D distance , where n is set according to actual conditions or experience.

[0099] In another possible implementation, the coordinates of the palm key points (x palm ,y palm ) and the coordinates of the wrist key point (x wrist ,y wrist ) is input into the network model, and the correction factor K is obtained through network learning: K=F(x wrist ,y wrist ,x palm ,y wrist ).

[0100] When the specific part is the face, the correction factor corresponding to the initial area of ​​the face can be determined based on the coordinates of the neck key points and the coordinates of the head key points. The specific determination method is similar to the above-mentioned method of determining the correction factor corresponding to the initial area of ​​the hand, and will not be repeated here.

[0101] After the correction factor is determined, the initial region obtained during the rough detection process can be corrected based on the correction factor to obtain a target region, which includes a complete specific part, such as a complete face or hand.

[0102] In this embodiment, after determining the initial area representing the specific part, a correction factor can be determined based on the position of the first limb key point and the position of the second limb key point, and the initial area is corrected based on the correction factor so that the target area obtained after correction can include the complete specific part, thereby improving the accuracy of the target area and ensuring the accuracy of the subsequent regional key points detected based on the target area.

[0103] Exemplarily, based on the above embodiment, the first limb key point is a non-edge point of a specific part. When the initial area is corrected based on the correction factor to determine the target area, the center point may be updated based on the position of the center point of the initial area and the position of the first limb key point when the first distance between the center point of the initial area and the first limb key point is less than the correction factor. When the first distance between the center point of the initial area and the first limb key point is greater than or equal to the correction factor, the center point is updated based on the first limb key point. The target area is determined based on the updated center point, the position of the first limb key point, the position of the second limb key point and the correction factor.

[0104] Specifically, the first limb key point is a non-edge point of a specific part. Taking the specific part as the hand as an example, the first limb key point is the palm key point, and the second limb key point is the wrist key point. Assume that the coordinates of the center point of the detection box corresponding to the initial area are (x center ,y center ), the width of the detection frame is w, and the height is h. The center point coordinates of the detection frame corresponding to the initial area are also the center point coordinates of the initial area. Therefore, the first distance D between the center point of the initial area and the first limb key point can be determined according to the following formula (2): ToCenter :

[0105] D ToCenter =sqrt((x center -x palm ) 2 -(y center -y palm ) 2 ) (2)

[0106] At the first distance D ToCenter When the correction factor K is less than the initial area, the position of the center point (x center ,y center ) and the position of the first limb key point (x palm ,y palm ) Update the position of the center point of the initial area. For example, the midpoint between the coordinates of the center point of the initial area and the coordinates of the first limb key point can be used as the updated center point. Therefore, the coordinates of the updated center point are

[0107]

[0108] At the first distance D ToCenter When the correction factor K is greater than or equal to the correction factor K, the position of the first limb key point (x palm ,y palm ) to update the position of the center point of the initial area. For example, the position of the first limb key point (xpalm ,y palm ) as the updated center point position, so the updated center point coordinate is X new =x palm , Y new =y palm .

[0109] Furthermore, the width w and height h of the detection frame corresponding to the initial area can be corrected based on the correction factor, which can be specifically performed according to the following formulas (3) and (4):

[0110]

[0111]

[0112] Among them, W new Indicates the corrected width, H new Indicates the corrected height, D distance Represents the distance between a first limb keypoint and a second limb keypoint, which can be determined based on the position of the first limb keypoint and the position of the second limb keypoint.

[0113] After the updated center point, the corrected width, and the corrected height are determined, the corrected target area is determined.

[0114] In this embodiment, the first distance between the center point of the initial area and the first limb key point can be determined, and the first distance and the correction factor can be compared to update the center point based on the comparison result. Since this embodiment can update the center point in a targeted manner according to different situations, the accuracy of the updated center point is improved, and the accuracy of the determined target area is also improved, ensuring that the target area includes a complete specific part, providing a basis for subsequent regional key point detection.

[0115] Exemplarily, based on the above embodiments, when determining at least two regional key points within the target area, key point detection can be performed on the sub-image corresponding to the target area in the first image to obtain at least two initial regional key points, and based on the sub-image and the at least two initial regional key points, the quality score corresponding to the sub-image is determined. When the quality score is less than the preset score, a second image corresponding to the sub-image is searched from the image library corresponding to the specific part, and the sub-image is replaced with the second image. After obtaining the updated target area, at least two regional key points are determined based on the updated target area.

[0116] Specifically, continue to refer to Figure 2As shown, the first image can be cropped based on the rotation angle of the specific part and the target area determined in the aforementioned embodiment to obtain a sub-image corresponding to the target area. The cropped sub-image is then input into a key point detection model to obtain at least two initial region key points. Exemplarily, the key point detection model can output the three-dimensional coordinates of the initial region key points. For example, inputting a facial sub-image into the facial key point detection model can obtain at least two initial facial key points, and inputting a hand sub-image into the hand key point detection model can obtain at least two initial hand key points.

[0117] The cropped sub-image and at least two detected initial region key points are input into a pre-trained motion quality evaluation model, and the motion quality of the detected initial region key points is evaluated by the motion quality evaluation model, and a quality score P is output. If the quality score P is less than the preset score σ, it means that the motion quality of the specific part is poor and there may be abnormal motion. At this time, motion association and completion are required. Exemplarily, a second image corresponding to the sub-image can be searched from an image library corresponding to a specific part, wherein the image library corresponding to the specific part can also be understood as an action library, which includes a plurality of pre-stored specific part movements. The image library provided in the embodiment of the present invention is established by learning training samples and hand and facial movements in actual usage scenarios. Among them, the second image corresponding to the sub-image can be understood as an image with a high similarity to the movement of the specific part in the currently detected sub-image.

[0118] After the second image is found, the sub-image is replaced by the second image to obtain an updated target region. In this way, key point detection can be performed on the updated target region to obtain at least two region key points.

[0119] In addition, if the quality score P is greater than or equal to the preset score σ, it means that the detected initial area key points are normal. Therefore, the initial area key points can be used as the final area key points for motion capture. For example, motion capture can be performed through key point detection or key point tracking, or motion capture can be performed in a mode that alternates key point detection and key point tracking.

[0120] It should be noted that the above-mentioned preset score σ needs to be set according to the actual usage scenario and picture quality, and can also be obtained through training, or learned through a pre-trained model. The embodiment of the present invention does not limit the specific method of determining the preset score.

[0121] In this embodiment, when the quality score corresponding to a sub-image is less than a preset score, a second image corresponding to the sub-image is searched from the image library corresponding to the specific part. This second image is used to replace the sub-image to complete and associate the specific part's movements. This ensures the accuracy of regional key points extracted from the target area obtained after movement completion. Furthermore, regional key points detected in low-quality or occluded situations can be distance-matched with the hand and facial movement library, and the closest movement can be added to the detected movement. This makes the hand and facial movements more reasonable and improves anti-interference capabilities and detection effectiveness.

[0122] For example, based on the above embodiments, when searching for a second image corresponding to a sub-image from an image library corresponding to a specific part, the search may be performed in the following manner:

[0123] For each initial region key point, determine the target key point corresponding to the initial region key point in each third image in the image library, and determine the second distance between the initial region key point and the target key point in each third image respectively. For each third image, based on the second distance between each initial region key point and the corresponding target key point in the third image, determine the average distance corresponding to the third image, and then determine the second image corresponding to the sub-image based on the average distance corresponding to each third image.

[0124] Specifically, taking the specific part including the hand as an example, when the quality score P is less than the preset score σ, for each detected initial region keypoint, a target keypoint corresponding to the initial region keypoint in each third image in the image library corresponding to the hand is determined. For example, if the initial region keypoint is the fingertip keypoint of the middle finger, the fingertip keypoint of the middle finger in each third image in the image library corresponding to the hand is determined as the target keypoint.

[0125] Considering that the coordinate systems of the initial region key points and the target key points may be different, it is necessary to align the initial region key points and the target key points by affine transformation according to the following formula (5):

[0126]

[0127] Among them, (x i ,y i ) represents the coordinates of the target key point, (x′ i ,y′ i) represents the coordinates of the target keypoints after alignment, rotation_matrix represents the rotation matrix for the hand alignment operation, and translation_vector represents the translation vector for the hand alignment operation. Both rotation_matrix and translation_vector can be calculated based on the positions of the initial region keypoints and the corresponding target keypoints.

[0128] After the target key points in each third image are aligned, the second distance between the initial region key point and the aligned target key point in each third image is determined, wherein the second distance may be a Euclidean distance. For example, the second distance between the initial region key point and the target key point may be calculated according to the following formula (6):

[0129]

[0130] in, Represents the initial region key point p i The coordinates of (x′ i ,y′ i ) represents the coordinates of the target key points after alignment.

[0131] Since there are multiple initial region keypoints, the target keypoint corresponding to each initial region keypoint can be determined in each third image. For each third image, the average distance value can be determined based on the second distance between each initial region keypoint and the corresponding target keypoint according to formula (7), which is the average distance avg_dis corresponding to the third image:

[0132]

[0133] Among them, l represents the number of key points detected in the initial area, p i represents the initial region key points, Represents the initial region key point p i The corresponding second distance, d p represents the variance of the second distance between each initial region key point and the corresponding target key point in the third image.

[0134] It should be understood that the smaller the average distance, the higher the similarity between the third image and the sub-image corresponding to that average distance. Therefore, after calculating the average distance corresponding to each third image, the third image with the smallest average distance is determined as the second image corresponding to the sub-image. This improves the accuracy of the completion when using this second image to complete the hand gesture in the original first image.

[0135] It is worth noting that when the specific part is the face, the method of determining the second image corresponding to the sub-image from the image library corresponding to the face is similar to the above implementation process and will not be repeated here.

[0136] In this embodiment, the distance between the initial area key point and the target key point corresponding to the initial area key point in each third image of the image library can be determined, and the average distance corresponding to each third image can be determined based on the second distance, so that the third image with the smallest average distance can be determined as the second image closest to or most similar to the sub-image, so that the action of the specific part in the sub-image can be completed based on the second image, which not only improves the accuracy of the second image, but also improves the precision of the action completion of the specific part.

[0137] For example, based on the above embodiments, when capturing the movement of the target in an image captured after the first image based on at least two limb key points and at least two region key points, the method can be as follows:

[0138] At least two limb key points and at least two region key points are input into a key point tracker to obtain predicted regions of each limb key point and each region key point output by the key point tracker in the next frame image of the first image, wherein the key point tracker is trained based on sample images and constraints determined based on body feature parameters; heat map regression is performed on the predicted regions to obtain a key point heat map, and the movement of the target in the frame after the next frame image is captured based on the key point heat map.

[0139] Specifically, key point information of the first N frames of the first image can be recorded. The key point information includes the 3D coordinates of limb key points, the 3D coordinates of regional key points, the bone lengths and angles between bones determined based on the 3D coordinates of the limb key points and the 3D coordinates of the regional key points, and the movement speeds of the limb key points and the regional key points. N is a positive integer, and the value of N can be determined and modified based on actual usage.

[0140] Furthermore, the key point information determined above can be input into the key point tracker. The key point tracker can predict the predicted area of ​​each limb key point and each region key point in the next frame image according to the key point information of the first image of the previous N frames using formula (8):

[0141] Joint_Area i =F(Q 约束先验 ,{Joint i-1 , Joint i-2 , Joint i-3 ,…}) (8)

[0142] Among them, Joint_Areai Indicates the predicted area of ​​each limb key point and each region key point in the next frame image, Q 约束先验 Indicates the length constraints, rotation constraints, and space constraints between bones. i-1 Represents the region of each limb key point and each region key point in the first image of the i-1 frame, Joint i-2 Represents the area of ​​each limb key point and each region key point in the first image of the i-2th frame, etc.

[0143] The keypoint tracker is trained based on sample images and constraints determined by body feature parameters. Specifically, the keypoint tracker is trained by feeding the network with prior vectors, such as the length constraints, rotation constraints, and spatial constraints between bones determined by the body feature parameters, into the training process. This allows the network to learn the skeletal constraint features of the target person in the sample images. These skeletal constraint features contain information such as the target person's bone length, rotational orientation, and motion area. Figure 5 A schematic diagram of human body space constraints provided by an embodiment of the present invention, such as Figure 5 As described above, the human body space constraint is set as the circumscribed circle range with the human body root joint, that is, the hip key point, as the center and the limbs fully extended.

[0144] Since the key point tracker is determined based on sample images and body feature parameters, when the key point tracker predicts the predicted area of ​​each limb key point and each region key point in the next frame image, it can ensure that the predicted key points do not exceed the length constraint, rotation constraint, and space constraint.

[0145] After obtaining the predicted area of ​​each limb key point and each regional key point in the next frame of the first image, the key point tracker performs key point heat map regression on the predicted area to obtain the predicted key point heat map in the next frame. Since the limb key points and regional key points in the first image are detected by a detection algorithm, it is necessary to track each key point in the next frame using the limb key points and regional key points detected in the first image. When tracking, it is necessary to predict the key points in the next frame, obtain the predicted area of ​​each limb key point and each regional key point in the next frame, and then perform heat map regression to obtain the key point heat map. Among them, when performing heat map regression, the predicted area will be tested for heat map Gaussian distribution to obtain the predicted coordinates of each limb key point and each regional key point in the next frame, thereby completing the tracking of the target in the next frame.

[0146] For a key point J i =(x i ,y i ), the value of a certain point (x, y) on the key point heat map is hi (x, y), the formula for heat map regression is shown in the following formula (9):

[0147]

[0148] It should be understood that the above key point heat map is based on J i The 3D coordinates of several possible limb key points and regional key points can be found by sampling from a Gaussian / normal distribution centered on , where σ is a pre-designed fixed value, which is generally proportional to the size of the key point heat map.

[0149] Furthermore, after predicting the three-dimensional coordinates of possible limb key points and regional key points in the next frame based on the key point heat map, these limb key points and regional key points are associated and matched. Specifically, the association and matching can be performed using the Hungarian algorithm or the distance matching algorithm. If a match is successful, the position of the limb key point and regional key point is retained as a new key point. The same tracking method is used for the next frame to capture the target's movement in the frame after the next frame, thereby completing the tracking of the target in the frame after the next frame.

[0150] Among them, the above-mentioned first image can be understood as the current frame image, the next frame image can be understood as the next frame image of the current frame image, that is, the second frame image, and the next frame image after the next frame image can be understood as the next frame image of the next frame image, that is, the third frame image.

[0151] In this embodiment, after the predicted area of ​​each limb key point and each regional key point in the next frame image is determined based on the key point tracker, heat map regression can be performed on the predicted area, so that the movement of the target in the frame after the next frame image can be captured based on the obtained key point heat map. Since the key point tracker is trained based on the constraint conditions determined by the body feature parameters, the determined prediction area is more accurate, thereby improving the accuracy of motion capture.

[0152] Furthermore, based on the above embodiments, in order to improve the accuracy of motion capture, when performing motion capture on a target in an image captured after the first image based on at least two limb key points and at least two regional key points, it is also necessary to evaluate the motion of the target tracked in the next frame of the first image. When the evaluation value is high, the motion capture of the target in the frame after the next frame is then performed. Exemplarily, the parameter value of the evaluation parameter can be determined based on the position of the predicted key points predicted in the next frame of the first image. The evaluation parameter includes at least one of the following: a state evaluation parameter, an action correlation evaluation parameter, and a physiological characteristic evaluation parameter; the key points include limb key points and / or regional key points; based on the parameter value of the evaluation parameter, the evaluation value of the target represented by all the predicted key points in the next frame is determined; when the evaluation value is greater than a preset value, the target in the frame after the next frame is motion tracked based on the predicted key points in the next frame; when the action evaluation value is less than or equal to the preset value, the target in the frame after the next frame is detected for the limb key points.

[0153] Specifically, the tracking action of the target in the next frame image of the first image can be comprehensively evaluated, and the parameter values ​​of the evaluation parameters can be determined based on the positions of the limb prediction key points and / or regional prediction key points in the next frame image, wherein the state evaluation parameters include the position and speed of the limb prediction key points and / or regional prediction key points, the action correlation evaluation parameters include whether there is a correlation between the actions made by the target in two consecutive frames of images, and the physiological characteristic evaluation parameters include bone length, bone rotation angle and spatial state, etc.

[0154] Furthermore, the action evaluation value of the target's action in the next frame can be determined based on the parameter values ​​of the evaluation parameters. For example, the action of the target tracked in the next frame can be evaluated by calculating the comprehensive displacement speed, movement amplitude, and movement frequency of the arm, key joints, and root node over several frames, thereby obtaining the action evaluation value.

[0155] When the action evaluation value is greater than a preset value, it indicates that the tracking action of the target in the next frame image is reasonable. Based on the at least two limb prediction key points and at least two region prediction key points corresponding to the next frame image, the positions of the target in the next frame image of the at least two limb prediction key points and at least two region prediction key points are predicted, thereby tracking the action of the target in the next frame image.

[0156] When the action evaluation value is less than or equal to the preset value, it means that the tracked action may be unreasonable, that is, the positions of the predicted limb prediction key points and regional prediction key points in the next frame image may deviate. Therefore, the tracking algorithm will be switched to the detection algorithm to perform key point detection on the position and action of the target in the frame after the next frame image, so as to capture the action through the key point detection algorithm.

[0157] It should be noted that the tracking algorithm and detection algorithm mentioned above can also be switched according to the actual application scenario.

[0158] In this embodiment, after determining the parameter value of the evaluation parameter by the position of the predicted limb key points and / or regional predicted key points in the next frame of the first image, the action evaluation value of the target in the next frame is determined based on the parameter value of the evaluation parameter. When the action evaluation value is greater than a preset value, the target is tracked. Since the tracking method can be used instead of partial action detection for motion capture, resource consumption can be reduced. In addition, since the target is tracked when the action evaluation value is greater than the preset value, the accuracy of the captured action can be improved, avoiding the deviation of subsequent motion capture caused by continued tracking based on the predicted action when the predicted action produces an error. When the action evaluation value is less than or equal to the preset value, the tracking algorithm is switched to the detection algorithm, so that the limb key points of the target in the next frame are detected by the detection algorithm to perform motion capture, which can improve the accuracy of motion capture.

[0159] For example, based on the above embodiment, when the evaluation parameter includes a state evaluation parameter, the parameter value of the evaluation parameter can be determined based on the position of the predicted key point in the next frame of the first image by the following method:

[0160] For each of two adjacent target image frames within a preset window, the instantaneous speed of the target in the target image after the two adjacent target image frames is determined based on the positions of each corresponding key point in the two adjacent target image frames, and the predicted speed of the target in the next frame image is determined based on the positions of each key point in the first image and the positions of the predicted key points in the next frame image after the first image, and the target speed corresponding to the next frame image is determined based on the instantaneous speed of the target in each subsequent target image and the predicted speed of the target in the next frame image, and then the parameter value of the state assessment parameter is determined based on the target speed corresponding to the next frame image and the target speed corresponding to the first image. Wherein, the target image includes the first image and a preset number of frames of images acquired before the first image.

[0161] Specifically, the preset window a can be selected based on actual conditions or experience. The first image is used as the last frame image in the preset window a, and multiple frames of target images are selected through the preset window a. For any two adjacent target images in the preset window a, based on the positions of the corresponding key points in the two adjacent target images, the distance between the corresponding key points in the two adjacent target images is determined, and then the instantaneous speed of the target in the latter target image is determined using formula (10):

[0162]

[0163] Among them, v t represents the instantaneous velocity of the target, p i Ordered key points representing specific parts of the target, Indicates that the key point p is detected at time t i The Euclidean distance between the key points detected at tn, d p represents the variance of the Euclidean distances corresponding to all key points, k represents the total number of detected key points, where k is the minimum number of detectable points in the two target image frames. n represents the difference in the number of detectable frames between the two frames. The two adjacent target images mentioned above are not continuous at the time of acquisition. Therefore, there may be a certain frame difference between the two adjacent target images, which is n.

[0164] In addition, based on a similar approach as above, the predicted speed of the target in the next frame of image can be determined based on the positions of the key points in the first image and the positions of the predicted key points in the next frame of image of the first image.

[0165] In practical applications, due to the problem of frame drop and overlap when the camera is shooting, it is also necessary to smooth the speed. Assuming that the speed of the target in the target image of the previous and next frames changes little within a period of time, set the preset window a as the smoothing data V = {v t-a+1 ,v t-a+2 +…v t Based on the instantaneous speed of the target in each subsequent first image within the preset window a and the predicted speed of the target in the next frame image, the target speed corresponding to the next frame image can be determined. for: Among them, v t-a+1 represents the instantaneous velocity of the target in the image captured at time t-a+1, v t-a+2 Indicates the instantaneous velocity of the target in the image taken at time t-a+2.

[0166] In the same manner as above, the target speed corresponding to the first image can also be obtained, so that the parameter value of the state assessment parameter can be determined based on the target speed corresponding to the next frame image and the target speed corresponding to the first image. Specifically, it can be determined based on formula (11):

[0167]

[0168] Among them, S1 represents the parameter value of the state assessment parameter, which can also be understood as the state estimation score. represents the target speed corresponding to the first image, Indicates the target speed corresponding to the next frame image, τ v represents the speed limit threshold, τ t The above formula (11) indicates whether the target speed corresponding to the next frame image and the target speed corresponding to the first image are both less than τ v , if less than, then less than τ v Is the duration of t , which in turn serve as the basis for determining the parameter values ​​of the state assessment parameters.

[0169] In this embodiment, the target speed can be determined by smoothing based on the instantaneous speed of the target, which can avoid the problem of inaccurate speed caused by frame drops and overlapping frames when the camera captures images, and improve the accuracy of the parameter values ​​of the state evaluation parameters.

[0170] For example, based on the above embodiments, when the evaluation parameter includes an action correlation evaluation parameter, the parameter value of the evaluation parameter can be determined based on the position of the predicted key point in the next frame of the first image by the following method:

[0171] For each predicted key point in the next frame image, based on the position of the predicted key point in the next frame image and the position of the corresponding key point in the first image, the vector angle corresponding to the key point is determined; based on the vector angle corresponding to each key point and the length of the specific part in the next frame image, the parameter value of the action correlation evaluation parameter is determined.

[0172] Specifically, since the face usually has some subtle changes in expression and the hands have some refined movements, the changes in the movements of the hands and face in two adjacent frames of images are usually not too large, and there will be correlation between the movements of the hands and face in the two frames of images. Therefore, the correlation evaluation can be performed based on the regional prediction key points of specific parts in the next frame of image, or based on the limb prediction key points and regional prediction key points in the next frame of image.

[0173] Specifically, for each predicted key point in the next frame image, a vector graph is established to determine the correlation of the action, and a corresponding vector set can be established. And calculate the angle θ of the key point corresponding to the first image of the previous frame according to formula (12):

[0174]

[0175] In addition, the length of a specific part can be determined based on the key points predicted in the region. For example, the length of a finger can be determined based on the key points predicted in the hand. When the vector angle corresponding to each predicted key point is less than the preset threshold τ θ , and the length of the specific part does not exceed the set threshold τ d When , it is considered that there is a correlation between the action of the target in the next frame image and the first image, and the parameter value of the action correlation evaluation parameter is determined according to formula (13), that is, the evaluation score S2:

[0176] S2=F({θ,d},{τ θ ,τ d}) (13)

[0177] It should be noted that if the specific part includes the hand, a corresponding threshold τ can be set for each finger θ .

[0178] In this embodiment, the vector angle corresponding to the key point can be determined based on the position of the predicted key point in the next frame image and the position of the corresponding key point in the first image, and then the parameter value of the action correlation evaluation parameter can be determined based on the vector angle and the length of the specific part. This not only makes the parameter value of the action correlation evaluation parameter more accurate, but also allows for more dimensions to be used to determine the action evaluation value of the target, thereby improving the comprehensiveness of the evaluation.

[0179] For example, when the evaluation parameters include physiological characteristics evaluation parameters, since the physiological characteristics of the human body are usually fixed parameters, the bone length l, rotation angle a and spatial step s can be determined by predicting the position of the key points, and the length constraint τ can be pre-set. l , angle constraint τ a , spatial constraint τ s As a basis for judgment, the parameter value or evaluation score S3 of the physiological characteristic evaluation parameter is determined according to formula (14):

[0180] S3=F({l,a,s},{τ l ,τ a ,τ s}) (14)

[0181] In one possible implementation, when the evaluation parameters include state evaluation parameters, action correlation evaluation parameters, and physiological characteristic evaluation parameters, after S1, S2, and S3 are determined in the above manner, the comprehensive score S, i.e., the action evaluation value, can be determined according to formula (15):

[0182] S=αS1+βS2+γS3 (15)

[0183] Among them, α, β, and γ represent weights, which can be set according to actual conditions, such as 0.2, 0.3, 0.5, etc. respectively.

[0184] To facilitate the evaluation of the target's movements, the score S can be normalized to (0, 1). The closer the score is to 1, the better the captured human motion is. The score S is compared with a threshold. If the score is greater than the threshold, the tracking algorithm is used to track the target, reducing the computational overhead. If the score is less than or equal to the threshold, the detection algorithm is used to re-detect key points or perform subsequent optimization and correction, improving the accuracy of motion capture.

[0185] Figure 6 A schematic diagram of motion capture provided by an embodiment of the present invention, such as Figure 6 As shown, the images of the first N frames that have completed motion capture and the frames to be captured that need motion capture can be input into the key point tracker. The predicted areas of each limb key point and each regional key point output by the key point tracker in the frame to be captured can be obtained. After heat map regression is performed on the predicted area and a key point heat map is obtained, the target's motion in the next frame of the frame to be captured is captured again based on the key point heat map through key point association matching. When training the key point tracker, it is necessary to add the human skeleton constraint prior to ensure that the predicted positions of each predicted key point meet the human skeleton constraint conditions.

[0186] In addition, after predicting the positions of each limb key point and each region key point in the frame image to be captured, the parameter values ​​of the evaluation parameters can also be determined to determine the motion evaluation value of the target in the frame image to be captured, so as to perform human motion assessment. When the motion evaluation value is greater than the preset value, it indicates that the currently captured motion is reasonable, and the motion capture will continue to be performed on the next frame image after the frame image to be captured through key point tracking. When the motion evaluation value is less than or equal to the preset value, the motion capture will be performed on the next frame image after the frame image to be captured through key point detection.

[0187] The motion capture device provided by the present invention is described below. The motion capture device described below and the motion capture method described above can be referenced to each other.

[0188] Figure 7The schematic diagram of the structure of the motion capture device provided by the embodiment of the present invention is shown in FIG. Figure 7 As shown, the motion capture device 700 includes:

[0189] An acquisition module 701 is configured to acquire a first image;

[0190] A detection module 702 is configured to perform limb key point detection on the target in the first image to obtain at least two limb key points;

[0191] A determination module 703 is configured to determine a target region representing a specific portion of the target in the first image;

[0192] The determining module 703 is further configured to determine at least two key points within the target area;

[0193] The capturing module 704 is configured to capture the movement of the target in an image captured after the first image based on the at least two limb key points and the at least two region key points.

[0194] In an exemplary embodiment, the determining module 703 is specifically configured to:

[0195] Performing region detection on the first image to obtain an initial region representing the specific part;

[0196] Acquire a first limb key point and a second limb key point of the specific part from the at least two limb key points;

[0197] determining a correction factor based on the position of the first limb key point and the position of the second limb key point;

[0198] The initial area is corrected based on the correction factor to obtain the target area.

[0199] In an exemplary embodiment, the first limb key point is a non-edge point of the specific part;

[0200] The determining module 703 is specifically configured to:

[0201] When a first distance between the center point of the initial region and the first limb key point is less than the correction factor, updating the center point based on a position of the center point of the initial region and a position of the first limb key point;

[0202] updating the center point based on the first limb key point when a first distance between the center point of the initial area and the first limb key point is greater than or equal to the correction factor;

[0203] The target area is determined based on the updated center point, the position of the first limb key point, the position of the second limb key point and the correction factor.

[0204] In an exemplary embodiment, the determining module 703 is specifically configured to:

[0205] Performing key point detection on a sub-image corresponding to the target area in the first image to obtain at least two initial area key points;

[0206] Determining a quality score corresponding to the sub-image based on the sub-image and the at least two initial region key points;

[0207] When the quality score is less than a preset score, searching for a second image corresponding to the sub-image from an image library corresponding to the specific part;

[0208] replacing the sub-image with the second image to obtain an updated target area;

[0209] The at least two region key points are determined based on the updated target region.

[0210] In an exemplary embodiment, the determining module 703 is specifically configured to:

[0211] For each of the initial region key points, determining a target key point corresponding to the initial region key point in each third image in the image library;

[0212] respectively determining second distances between the initial region key points and the target key points in each of the third images;

[0213] For each of the third images, determining an average distance corresponding to the third image based on a second distance between each of the initial region key points and a corresponding target key point in the third image;

[0214] Based on the average distance corresponding to each of the third images, a second image corresponding to the sub-image is determined.

[0215] In an exemplary embodiment, the capture module 704 is specifically configured to:

[0216] Inputting the at least two limb key points and the at least two region key points into a key point tracker, and obtaining predicted regions of each limb key point and each region key point output by the key point tracker in a next frame of the first image; the key point tracker is trained based on sample images and constraints determined based on body feature parameters;

[0217] Performing heat map regression on the predicted area to obtain a key point heat map;

[0218] The action of the target in the frame image after the next frame image is captured based on the key point heat map.

[0219] In an exemplary embodiment, the capture module 704 is specifically configured to:

[0220] Determining a parameter value of an evaluation parameter based on a position of a predicted key point predicted in a next frame of the first image, the evaluation parameter comprising at least one of the following: a state evaluation parameter, an action correlation evaluation parameter, and a physiological characteristic evaluation parameter; the predicted key point comprising a limb prediction key point and / or a region prediction key point;

[0221] Determining action evaluation values ​​of the objects represented by all the predicted key points in the next frame of image based on parameter values ​​of the evaluation parameters;

[0222] When the action evaluation value is greater than a preset value, performing action tracking on the target in a frame image subsequent to the next frame image based on the predicted key points in the next frame image;

[0223] When the action evaluation value is less than or equal to the preset value, limb key point detection is performed on the target in a frame image subsequent to the next frame image.

[0224] In one example embodiment, the assessment parameters include state assessment parameters;

[0225] The determination module 703 is specifically configured to:

[0226] For each of two adjacent target image frames within a preset window, determining the instantaneous speed of the target in the target image after the two adjacent target image frames based on the positions of corresponding key points in the two adjacent target image frames, and determining the predicted speed of the target in the next target image frame based on the positions of key points in the first image and the positions of predicted key points in the next target image frame after the first image; the target image frame includes the first image and a preset number of image frames acquired before the first image;

[0227] Determining the target speed corresponding to the next frame image based on the instantaneous speed of the target in each of the subsequent target images and the predicted speed of the target in the next frame image;

[0228] The parameter value of the state evaluation parameter is determined based on the target speed corresponding to the next frame image and the target speed corresponding to the first image.

[0229] In an example embodiment, the evaluation parameter includes an action relevance evaluation parameter;

[0230] The determination module 703 is specifically configured to:

[0231] For each predicted key point in the next frame image, determining a vector angle corresponding to the key point based on a position of the predicted key point in the next frame image and a position of the corresponding key point in the first image;

[0232] The parameter value of the action correlation evaluation parameter is determined based on the vector angle corresponding to each key point and the length of the specific part in the next frame image.

[0233] The device of this embodiment can be used to execute the method of any embodiment in the motion capture method embodiment. Its specific implementation process and technical effects are similar to those in the motion capture method embodiment. For details, please refer to the detailed description in the motion capture method embodiment, which will not be repeated here.

[0234] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a motion capture method, which includes: acquiring a first image; performing limb key point detection on a target in the first image to obtain at least two limb key points, and determining a target area in the first image that represents a specific part of the target; determining at least two regional key points within the target area; and capturing the motion of the target in an image captured after the first image based on the at least two limb key points and the at least two regional key points.

[0235] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0236] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the motion capture method provided by the above methods, which includes: acquiring a first image; performing limb key point detection on the target in the first image to obtain at least two limb key points, and determining a target area in the first image that represents a specific part of the target; determining at least two region key points within the target area; and capturing the motion of the target in an image captured after the first image based on the at least two limb key points and the at least two region key points.

[0237] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the motion capture method provided by the above-mentioned methods, the method comprising: acquiring a first image; performing limb key point detection on the target in the first image to obtain at least two limb key points, and determining a target area in the first image representing a specific part of the target; determining at least two region key points within the target area; and capturing the motion of the target in an image captured after the first image based on the at least two limb key points and the at least two region key points.

[0238] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0239] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0240] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A motion capture method, characterized in that: include: acquiring a first image; Performing limb key point detection on the target in the first image to obtain at least two limb key points, and determining a target area representing a specific part of the target in the first image; Determining at least two key points in the target area; The movement of the target in an image collected after the first image is captured based on the at least two limb key points and the at least two region key points.

2. The motion capture method according to claim 1, wherein: Determining a target area representing a specific portion of the target in the first image includes: Performing region detection on the first image to obtain an initial region representing the specific part; Acquire a first limb key point and a second limb key point of the specific part from the at least two limb key points; determining a correction factor based on the position of the first limb key point and the position of the second limb key point; The initial area is corrected based on the correction factor to obtain the target area.

3. The motion capture method according to claim 2, wherein: The first limb key point is a non-edge point of the specific part; The correcting the initial area based on the correction factor to obtain the target area includes: When a first distance between the center point of the initial region and the first limb key point is less than the correction factor, updating the center point based on a position of the center point of the initial region and a position of the first limb key point; updating the center point based on the first limb key point when a first distance between the center point of the initial area and the first limb key point is greater than or equal to the correction factor; The target area is determined based on the updated center point, the position of the first limb key point, the position of the second limb key point and the correction factor.

4. The motion capture method according to claim 1, wherein: The determining of at least two key points in the target area includes: Performing key point detection on a sub-image corresponding to the target area in the first image to obtain at least two initial area key points; Determining a quality score corresponding to the sub-image based on the sub-image and the at least two initial region key points; When the quality score is less than a preset score, searching for a second image corresponding to the sub-image from an image library corresponding to the specific part; replacing the sub-image with the second image to obtain an updated target area; The at least two region key points are determined based on the updated target region.

5. The motion capture method according to claim 4, characterized in that: The searching for a second image corresponding to the sub-image from an image library corresponding to the specific part includes: For each of the initial region key points, determining a target key point corresponding to the initial region key point in each third image in the image library; respectively determining second distances between the initial region key points and the target key points in each of the third images; For each of the third images, determining an average distance corresponding to the third image based on a second distance between each of the initial region key points and a corresponding target key point in the third image; Based on the average distance corresponding to each of the third images, a second image corresponding to the sub-image is determined.

6. The motion capture method according to any one of claims 1 to 5, characterized in that: The capturing of the movement of the target in an image collected after the first image based on the at least two limb key points and the at least two region key points includes: Inputting the at least two limb key points and the at least two region key points into a key point tracker, and obtaining predicted regions of each limb key point and each region key point output by the key point tracker in a next frame of the first image; the key point tracker is trained based on sample images and constraints determined based on body feature parameters; Performing heat map regression on the predicted area to obtain a key point heat map; The action of the target in the frame image after the next frame image is captured based on the key point heat map.

7. The motion capture method according to claim 6, wherein: The performing motion capture on the target in the image acquired after the first image based on the at least two limb key points and the at least two region key points includes: Determining a parameter value of an evaluation parameter based on a position of a predicted key point predicted in a next frame of the first image, the evaluation parameter comprising at least one of the following: a state evaluation parameter, an action correlation evaluation parameter, and a physiological characteristic evaluation parameter; the predicted key point comprising a limb prediction key point and / or a region prediction key point; Determining action evaluation values ​​of the targets represented by all the predicted key points in the next frame of image based on parameter values ​​of the evaluation parameters; When the action evaluation value is greater than a preset value, performing action tracking on the target in a frame image subsequent to the next frame image based on the predicted key points in the next frame image; When the action evaluation value is less than or equal to the preset value, limb key point detection is performed on the target in a frame image subsequent to the next frame image.

8. The motion capture method according to claim 7, characterized in that: The evaluation parameters include status evaluation parameters; The step of determining a parameter value of an evaluation parameter based on a position of a predicted key point predicted in a next frame of an image of the first image comprises: For each of two adjacent target image frames within a preset window, determining the instantaneous speed of the target in the target image after the two adjacent target image frames based on the positions of corresponding key points in the two adjacent target image frames, and determining the predicted speed of the target in the next target image frame based on the positions of key points in the first image and the positions of predicted key points in the next target image frame after the first image; the target image frame includes the first image and a preset number of image frames acquired before the first image; Determining the target speed corresponding to the next frame image based on the instantaneous speed of the target in each of the subsequent target images and the predicted speed of the target in the next frame image; The parameter value of the state evaluation parameter is determined based on the target speed corresponding to the next frame image and the target speed corresponding to the first image.

9. The motion capture method according to claim 7, wherein: The evaluation parameters include action correlation evaluation parameters; The step of determining a parameter value of an evaluation parameter based on a position of a predicted key point predicted in a next frame of an image of the first image comprises: For each predicted key point in the next frame image, determining a vector angle corresponding to the key point based on a position of the predicted key point in the next frame image and a position of the corresponding key point in the first image; The parameter value of the action correlation evaluation parameter is determined based on the vector angle corresponding to each key point and the length of the specific part in the next frame image.

10. A motion capture device, characterized in that: include: An acquisition module, configured to acquire a first image; A detection module, configured to perform limb key point detection on the target in the first image to obtain at least two limb key points; a determination module, configured to determine a target region representing a specific portion of the target in the first image; The determining module is further configured to determine at least two key points within the target area; A capture module is used to capture the movement of the target in an image collected after the first image based on the at least two limb key points and the at least two area key points.

Images