Three-dimensional human body action capturing method for human body video and related equipment

Abstract

The invention discloses a three-dimensional human body action capturing method for a human body video and related equipment, and relates to the field of image processing. The method comprises the steps of obtaining an image sequence of a human body video; obtaining human body features obtained by fusing any two continuous frames of images in the image sequence; and converting the human body features into three-dimensional human body action parameters. By implementing the technical scheme provided by the invention, the stability of three-dimensional human body motion capture during continuous video processing can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for capturing three-dimensional human body motion in a human body video and related equipment. Background Art

[0002] With the development of computer vision technology, 3D human motion capture has been widely used in virtual reality, animation, human-computer interaction and other fields. At present, most 3D human motion capture methods are mainly used for processing static images. This method can obtain human posture information when processing a single image, but it often faces some challenges when applied to continuous videos.

[0003] When processing continuous videos, existing technologies often have difficulty ensuring the stability of motion capture results. This instability may manifest as sudden changes, jitters, or unnatural transitions in motion. These problems not only affect the visual effect, but may also lead to a decline in the quality of subsequent applications. Summary of the invention

[0004] The present invention provides a method for capturing three-dimensional human motion in human body videos and related equipment, which can improve the stability of three-dimensional human motion capture when processing continuous videos.

[0005] In a first aspect of the present invention, a method for capturing three-dimensional human body motions in a human body video is provided, comprising: Obtain an image sequence of a human body video; Acquire human features obtained by fusing any two consecutive frames of images in the image sequence; The human body features are converted into three-dimensional human body motion parameters.

[0006] Optionally, the acquiring of human features obtained by fusing any two consecutive frames of images in the image sequence includes: Obtaining coding features of any two consecutive frames of images in the image sequence, wherein the coding features are used to represent position information of a human body in the image; Acquire a key position in the image based on the coding feature; Fusion of encoding features of any two consecutive frames of images; The fused coding features and the key positions are used as human body features.

[0007] Optionally, the acquiring coding features of any two consecutive frames of images in the image sequence includes: Obtain any two consecutive frames of images in the image sequence, and split each of the images into image blocks; Performing position coding on the image block to obtain a position coding block; The position coding block is spliced ​​with the body feature block to obtain coding features, wherein the body feature block is obtained from a standard three-dimensional human body model.

[0008] Optionally, the step of acquiring an image sequence of a human body video includes: Acquire a human body video, and split the human body video into multiple frame images; The background part in each frame image is filtered to obtain an image sequence.

[0009] Optionally, converting the human body features into three-dimensional human body motion parameters includes: Inputting the human body features into a regressor to obtain action parameters output by the regressor; The motion parameters are input into an attention module to obtain the three-dimensional human motion parameters output by the attention module.

[0010] Optionally, after converting the human body features into three-dimensional human body motion parameters, the method further includes: A three-dimensional human body corresponding to the three-dimensional human body motion parameters is generated.

[0011] Optionally, before acquiring the image sequence of the human body video, the method further includes: Using sample human body videos to train a three-dimensional human body model, the three-dimensional human body model is used to implement any of the above-mentioned methods for capturing three-dimensional human body movements for human body videos; The step of training the three-dimensional human body model using a sample human body video includes: Inputting the sample human body video into the three-dimensional human body model to obtain a sample three-dimensional human body output by the three-dimensional human body model; The error value of the sample three-dimensional human body model is calculated based on a loss function, and the parameters of the three-dimensional human body model are adjusted according to the error value. The loss function is used to measure the errors of the three-dimensional human body model in human body parameters, three-dimensional joint point coordinates and two-dimensional joint point coordinates.

[0012] In a second aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a method for capturing three-dimensional human motions in a human body video as described in any one of the above is implemented.

[0013] In a third aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for capturing three-dimensional human body motions in a human body video as described in any one of the above is implemented.

[0014] In a fourth aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-described methods for capturing three-dimensional human motions in a human body video.

[0015] In summary, one or more technical solutions provided in the present invention have at least the following technical effects or advantages: By acquiring an image sequence of a human body video, obtaining the human body features obtained by fusing any two consecutive frames of images in the image sequence, and then converting the human body features into three-dimensional human body motion parameters, effective capture of human body motion in continuous videos is achieved. This method utilizes the information association between consecutive frames, and compared with the method in the prior art that only processes static images, it can significantly improve the consistency and stability of motion capture results.

[0016] By fusing the information of consecutive frames, this method effectively reduces the problems of motion mutation and jitter, thereby improving the smoothness and accuracy of 3D human motion reconstruction. This improvement not only enhances the visual effect, but also provides higher quality input data for subsequent applications such as animation production and motion analysis, effectively solving the problems existing in the existing technology and improving the stability of 3D human motion capture when processing continuous videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 The present invention is a flowchart of a method for capturing three-dimensional human body motion in a human body video provided by an embodiment of the present invention.

[0019] Figure 2 It is a flow chart of another method for capturing three-dimensional human body motion in human body video provided by an embodiment of the present invention.

[0020] Figure 3 It is a flow chart of another method for capturing three-dimensional human body motion in human body video provided by an embodiment of the present invention.

[0021] Figure 4 It is a flow chart of another method for capturing three-dimensional human body motion in human body video provided by an embodiment of the present invention.

[0022] Figure 5 It is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Please refer to Figure 1 , Figure 1 This is a flow chart of a method for capturing 3D human motion in human video provided by an embodiment of the present application. The method can be implemented by a computer program, can be implemented by a single-chip microcomputer, or can be run on a 3D human motion capture system for human video based on a von Neumann system. The computer program can be integrated into an application or run as an independent tool application. Specifically, the method can include the following steps: S101, obtaining an image sequence of a human body video.

[0025] Among them, human body video refers to a collection of continuous images containing one or more sequences of human body movements. These videos are composed of multiple continuous image frames, each of which records the posture information of the human body at a specific moment, arranged at fixed time intervals to form a coherent dynamic picture. Human body videos can cover a wide range of content, from single-person full-body movements to multi-person interactive scenes, including specific movements such as dance or gymnastics, as well as various postures in daily activities. Even human body movements in some special environments, such as underwater movements or movements in low-gravity environments, can also be classified as human body videos processed by the present invention.

[0026] Correspondingly, an image sequence refers to an ordered set of a series of continuous static images. These images are arranged in chronological order, and each image represents the state of the scene at a specific moment. An image sequence can be regarded as a discretized representation of a video, which retains the temporal continuity and dynamic information in the video, while converting continuous video data into discrete units that can be processed separately.

[0027] In the embodiment of the present application, the image sequence can be understood as a series of key frames extracted from a human body video. These key frames are extracted from the original video at fixed time intervals or specific selection rules to form a set of images that are continuous in time but discrete in data structure. Each frame of the image contains complete scene information, especially key features such as the posture, position and appearance of the human body. The image sequence may include all video frames sampled at equal intervals, or it may be a set of key frames dynamically selected according to the degree of action change.

[0028] Based on the above embodiment, as an optional embodiment, please refer to Figure 2 , Figure 2 : is a flow chart of another method for capturing three-dimensional human body motion in human body video provided by an embodiment of the present application. The specific method for obtaining the image sequence may also include the following steps: S201, obtaining a human body video, and splitting the human body video into multiple frame images.

[0029] In a feasible implementation, the input human body video can be segmented into continuous frame images using the ffmpeg tool. ffmpeg is a powerful open source multimedia processing tool that can efficiently convert videos into a series of static images. For example, for a 30-frame / second video, ffmpeg can convert it into 30 continuous images per second. The advantage of this method is that each frame can be processed independently, making it easier for subsequent modules to accurately analyze the human body posture at each time point.

[0030] In another feasible implementation, the video stream may be directly read, which is more suitable for real-time processing scenarios, can reduce the storage requirements of intermediate data, and improve processing efficiency.

[0031] S202: Filter the background part in each frame image to obtain an image sequence.

[0032] Specifically, background filtering is a crucial preprocessing step. In practical applications, human videos often contain complex and changing background information, which may interfere with the motion capture system. For example, other moving objects in the background may be mistaken for part of the human body, or complex textures may affect the accurate recognition of the human body outline. By filtering the background, the system can focus on the human body itself, greatly reducing the source of error and improving the accuracy of subsequent processing.

[0033] Specifically, for each image I containing a human body, a human body detection method D can be applied to locate the human body in the image. The human body detection method D here can select advanced algorithms based on deep learning such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector). These algorithms can quickly and accurately identify the location of the human body in the image and output a minimum detection box.

[0034] Furthermore, the detection process can be expressed as: box = D(I); Among them, box is usually a four-tuple (x, y, width, height), which represents the x and y coordinates of the upper left corner of the detection box and the width and height of the detection box. This detection box accurately locates the position of the human body in the image. After obtaining the detection box, the image cropping operation begins. The cropping operation can be implemented by a cropping function Crop: I' = Crop(I, box); Among them, the Crop function uses the coordinate information of the detection box box to extract a sub-image I' that contains only the human body as much as possible from the original image I, thereby reducing the background information.

[0035] S102: Acquire human features obtained by fusing any two consecutive frames of images in the image sequence.

[0036] Human features refer to a set of high-dimensional vectors extracted from image sequences, which are used to comprehensively describe the state and changes of the human body in consecutive video frames. Human features are a multimodal data representation that integrates spatial, temporal and semantic information, effectively encoding the visual appearance, posture information and temporal dynamics of the human body.

[0037] Specifically, human features capture the relative positions and overall contours of various parts of the human body, including spatial structural information such as joint positions, limb orientations, and overall body posture. It is worth noting that human features obtained by fusing continuous frame images can reflect the changes in human posture and position over time, which is crucial for understanding continuous action sequences and predicting future motion trends.

[0038] Based on the above embodiment, as an optional embodiment, please refer to Figure 3 , Figure 3 : This is a flow chart of another method for capturing 3D human motion in human body video provided by an embodiment of the present application. Specifically, the step of obtaining human features obtained by fusion of any two consecutive frames of images in an image sequence may further include the following steps: S301, obtaining coding features of any two consecutive frames of images in an image sequence, where the coding features are used to represent position information of a human body in the image.

[0039] Among them, the encoded feature is a key data representation, which is extracted from the input image through a deep neural network, especially an encoder based on the Transformer architecture. The encoded feature is essentially a set of high-dimensional vectors or tensors that transform the original image pixels into a more abstract and semantic representation through complex nonlinear transformations. The encoded feature not only contains the visual content and spatial structure of the image, but also pays special attention to encoding key information such as the position, posture and appearance of the human body in the image.

[0040] Furthermore, the coded features are mainly used to accurately locate the position of the human body in the image and the relative position relationship of each part, providing rich input information for the subsequent posture parameter regression. By comparing the coded features of consecutive frames, the system can analyze the continuity and changes of human motion, which is crucial for capturing smooth action sequences. In addition, the coded features are also used as basic features to be fused with features of other modalities to provide a more comprehensive description of the action.

[0041] Based on the above embodiment, as an optional embodiment, please refer to Figure 4 , Figure 4 3 is a flow chart of another method for capturing three-dimensional human body motion in human body video provided by an embodiment of the present application. The specific method for obtaining coding features may also include the following steps: S401, obtaining any two consecutive frames of images in an image sequence, and splitting each image into image blocks.

[0042] Among them, the image block refers to the sub-image unit obtained by dividing the complete image according to specific rules. In the embodiment of the present application, it can be understood as a fixed-size, non-overlapping rectangular area obtained by dividing the original image by a uniform grid. Each image block contains a local area of ​​the original image, retaining the pixel information and spatial structure in the area. These image blocks are used to convert large-size, high-resolution images into a series of smaller data units that are easier to process while maintaining spatial information.

[0043] Exemplarily, the segmented image blocks are recorded as , each image patch is a rectangular area of ​​fixed size. For example, if the image size is 256×256 pixels, it may be split into 16×16 small patches, and each image patch size is 16×16 pixels.

[0044] S402: Perform position encoding on the image block to obtain a position encoding block.

[0045] The position coding block refers to a feature representation containing the visual content of the image block and its corresponding spatial position information. In the embodiment of the present application, it can be understood as a high-dimensional feature vector or tensor obtained by fusing the original image block with its position coding vector.

[0046] Specifically, the position encoding block can be expressed as: ; Where ζ represents a convolutional layer, which is used to perform linear projection on each image block. The role of linear projection is to convert the two-dimensional image block into a high-dimensional feature vector. The fused position encoding block It not only contains the visual content of the image block, but also contains its position information in space.

[0047] S403, splicing the position coding block with the body feature block to obtain coding features, wherein the body feature block is obtained from a standard three-dimensional human body model.

[0048] Among them, the body feature block refers to a set of feature representations that encode prior knowledge of the human body extracted from a standard three-dimensional human body model. In the embodiment of the present application, it can be understood as a high-dimensional vector or tensor generated based on a parametric human body model such as SMPL-X, which contains prior knowledge such as the structural information of the human body, the relationship between joint connections, and motion constraints.

[0049] Specifically, in order to further enhance the effect of feature extraction, the system uses prior knowledge of the human body. Prior knowledge of the human body can be obtained through learnable body feature blocks. These body feature blocks are obtained by the system during the training process by learning the prior information of human posture and structure.

[0050] Feature Block With body feature blocks Spliced ​​together to form a new encoding feature. The purpose of splicing is to allow the network to integrate the prior information of the human body during the feature extraction process, so as to better capture the global characteristics of the human body.

[0051] The concatenated features are input into the Transformer-based encoder γ. The Transformer encoder consists of multiple Transformer blocks, each of which includes the following key components: multi-head self-attention, feedforward network, and double-layer normalization, where: Multi-head self-attention mechanism: Through the self-attention mechanism, the system can capture the relationship between different parts of the image and help the network understand the overall structure of the human body.

[0052] Feedforward network: used to further process features and enhance the nonlinear ability of feature representation.

[0053] Double-layer normalization: used to stabilize the training process and prevent gradient explosion or disappearance.

[0054] After being processed by multiple Transformer blocks, the features are updated and encoded features are generated, representing the updated image features and body features respectively: ; S302: Acquire key positions in the image based on the coding features.

[0055] Among them, key positions refer to specific points or areas in the human body structure that have important anatomical and kinematic significance. In the embodiments of the present application, they can be understood as the main joints in the human skeletal structure and other body parts with significant characteristics, such as the center of the head, neck, shoulders, elbows, wrists, hips, knees, ankles, etc. These key positions can be represented by precise coordinates in three-dimensional space, and are usually expressed as areas with high response values ​​in two-dimensional images.

[0056] Specifically, using the updated image features and physical characteristics , through a series of fully connected layers Perform preliminary body motion parameter regression and obtain . It contains the rough posture information of the human body, laying the foundation for subsequent refined processing.

[0057] However, considering the complexity of areas such as hands and head, it is difficult to meet the needs of high-precision motion capture by relying solely on such rough estimates. Therefore, the present invention cleverly introduces the prior knowledge of the SMPL-X model. As a widely used parametric human body model, SMPL-X not only describes the overall posture of the human body, but also provides detailed information about the hands and head. Combined with the prior knowledge of SMPL-X and further leveraging the vertex information of the previous frame, the system is able to more accurately deduce key positions such as the head and hands.

[0058] Among them, the SMPL-X model is a parameterized 3D human body model, which generates a 3D mesh of the human body through posture parameters and shape parameters. This mesh is composed of multiple vertices, each of which has a corresponding 3D coordinate, i.e., vertex information.

[0059] S303: Fuse the coding features of any two consecutive image frames.

[0060] Specifically, in actual motion capture tasks, relying solely on a single frame image is often difficult to handle complex situations such as fast motion, occlusion or blur. By fusing information from adjacent frames, the system's ability to adapt to these challenges can be greatly enhanced.

[0061] Specifically, it is fused with the current frame features through a designed interaction mechanism. Usually, a temporal attention mechanism or a feature fusion module is used to align and fuse the feature blocks of the previous frame with the feature blocks of the current frame to generate updated encoding features. and .

[0062] By fusing features from consecutive frames, the system is able to better capture the temporal dynamic consistency of human motion. This is particularly important for processing fast movements or complex postures, as it can effectively avoid incoherent prediction results in these situations. For example, when processing fast waving movements, a single frame of information may not be able to accurately capture the position and posture of the hand, and by fusing the information from the previous frame, the system can better understand and predict the movement trajectory of the hand.

[0063] In addition, this feature fusion can also improve the system's robustness to occlusion and image quality issues. When part of the human body in a frame is occluded or the image quality is poor, the information in the previous frame can provide valuable supplements to help the system more accurately infer the position and posture of the occluded part. This not only improves the accuracy of motion capture, but also significantly enhances the system's adaptability in various complex scenarios.

[0064] S304: Use the fused coding features and key positions as human body features.

[0065] S103: Convert the human body features into three-dimensional human body motion parameters.

[0066] Among them, three-dimensional human motion parameters refer to a set of numerical representations used to describe the posture, shape and motion state of the human body in three-dimensional space. In the embodiment of the present application, it can be understood as a parameterized representation based on the SMPL-X model. The three-dimensional human motion parameters are mainly used to drive the three-dimensional human model to realize the visualization of human posture. By inputting the parameters into models such as SMPL-X, a corresponding three-dimensional human mesh model can be generated to intuitively display the captured movements. At the same time, this parameterized representation provides a compact and information-rich input for motion analysis and recognition, which is convenient for subsequent tasks such as motion classification or anomaly detection.

[0067] Based on the above embodiment, as an optional embodiment, the conversion process of the three-dimensional human body motion parameters may further include the following steps: S501: Input human body features into a regressor to obtain action parameters output by the regressor.

[0068] S502: Input the motion parameters into the attention module to obtain the three-dimensional human motion parameters output by the attention module.

[0069] Specifically, in the process of converting human features into 3D human motion parameters, the embodiment of the present invention adopts an innovative two-stage method to improve the accuracy and stability of motion capture, especially when dealing with complex structures such as hands and heads. The method first uses a regressor to obtain preliminary motion parameters, then refines them through an attention module, and finally outputs high-quality 3D human motion parameters.

[0070] Specifically, in the first stage, the fused human features are input into a regressor to obtain preliminary motion parameters. This regressor is designed to extract key information of human posture from high-dimensional features. However, considering the complexity of human motion, especially the fine motion of hands and head, it is often difficult to meet the needs of high-precision motion capture by relying solely on this preliminary estimate. Therefore, the present invention introduces the processing of the second stage.

[0071] In the second stage, the action parameters can be input into a specially designed attention module. It can focus on the hand and head areas for more accurate parameter estimation. The above process can be expressed as: ; in, are the precise movement parameters of the left and right hands respectively. These are the precise movement parameters of the head. These parameters contain the fine angle information of the finger joints and the detailed rotation information of the head, and can accurately describe complex hand postures and subtle head movements.

[0072] Based on the above embodiment, as an optional embodiment, a three-dimensional human body corresponding to the three-dimensional human body motion parameters may also be generated.

[0073] Specifically, the generation process mainly relies on the SMPL-X layer, which is a differentiable layer of the SMPL-X model that can convert the parameterized human body representation into a detailed three-dimensional mesh model. The input of the SMPL-X layer includes full body posture parameters, shape parameters, facial expression parameters, and hand posture parameters. In the present invention, these parameters are respectively obtained from the previous steps. .

[0074] Based on the above embodiment, as an optional embodiment, the present invention further provides a three-dimensional human body model for implementing the above-mentioned method for capturing three-dimensional human body movements for human body videos. Specifically, the training process of the three-dimensional human body model may include the following steps: S601: Input a sample human body video into a three-dimensional human body model, and obtain a sample three-dimensional human body output by the three-dimensional human body model.

[0075] Specifically, in the embodiment of the present invention, the 3D human body model is a carefully designed deep learning system, whose structure and function are closely integrated with the 3D human motion capture method for human body video described in the above embodiment. The model is composed of multiple key components, including a video preprocessing module, a feature extraction network, a key point positioning module, a temporal feature fusion module, a parameter regressor, an attention refinement module, and an SMPL-X Layer. These components work together to realize the conversion process from sample human body video to accurate 3D human body representation.

[0076] When a sample human video is input to the 3D human model, it is first processed by the video preprocessing module. This module uses tools such as ffmpeg to segment the video into consecutive frames and applies human detection algorithms such as YOLO or SSD to crop out the image area that mainly contains the human body.

[0077] The preprocessed image sequence then enters the feature extraction network. This network based on the Transformer architecture is one of the core components of the model. It first divides the image into multiple image blocks, performs linear projection through the convolution layer, and fuses with the position encoding information. These features are then concatenated with the pre-learned body feature blocks and deeply processed through multiple Transformer blocks. This process not only extracts the visual features of a single frame, but also enhances the understanding of the human body structure by fusing it with prior knowledge of the body.

[0078] Next, the key point location module uses the encoded features output by the feature extraction network and combines the prior knowledge of the SMPL-X model to accurately locate the key positions of the human body, especially complex areas such as the hands and head. This lays the foundation for subsequent refined processing. Subsequently, the temporal feature fusion module comes into play, fusing the features of consecutive frames through the designed interaction mechanism. This enhances the model's understanding of action continuity and helps capture complex action sequences.

[0079] The fused features are fed into the parameter regressor to preliminarily estimate the human motion parameters. These parameters describe the overall posture and shape of the human body. However, considering that areas such as the hands and head require more refined control, the attention refinement module further processes these preliminary parameters to generate more accurate motion parameters.

[0080] Finally, these refined parameters are input into the SMPL-X Layer, which is the last component of the model and is responsible for converting the parameterized human body representation into a detailed 3D mesh model, i.e., the final sample 3D human body.

[0081] S602: Calculate the error value of the sample three-dimensional human body model based on the loss function, and adjust the parameters of the three-dimensional human body model according to the error value.

[0082] Among them, the loss function refers to a function used to quantify the deviation between the model output and the expected output, which maps the difference between the model's prediction result and the true label to a non-negative real number. In the embodiment of the present application, it can be understood as a metric for evaluating the difference between the generated three-dimensional human body model and the real human body model. The loss function is used to guide the training process of the model, and the model parameters are optimized by minimizing the value of the loss function, thereby improving the prediction accuracy of the model.

[0083] In the embodiment of the present invention, three loss functions are used in the process of training the 3D human body model, which respectively measure the errors of the model in human body parameters, 3D joint point coordinates and 2D joint point coordinates. The network guides its learning through these loss functions, gradually approaching the real human body motion parameters and joint point positions.

[0084] The first loss function is used to calculate the error of human body parameters .

[0085] First, the network will regress the human body parameters, which usually include: posture parameters: control the rotation of human joints; shape parameters: control the overall body shape of the human body (such as height, weight, etc.); during the training process, the network regression parameters P will be different from the actual parameters Compare them and calculate the L1 error between them. The definition of L1 error is the sum of the absolute differences between two vectors. The formula is as follows: ; In the formula, represents the L1 norm, which is the sum of the absolute values ​​of each element in the vector.

[0086] The second loss function is used to calculate the error of the three-dimensional joint points .

[0087] The posture parameters of the human body will eventually affect the position of each joint in the 3D human body model. Through the SMPL-X model, the human body parameters obtained by regression can generate the entire 3D human body model, thereby obtaining the 3D spatial coordinates of each joint of the human body. The positions of these joints are a direct expression of the human body posture. In order to measure the accuracy of the network's prediction of 3D joints, the relationship between the 3D joints generated by the network and the real 3D joints is calculated. The formula is as follows: .

[0088] The third loss: the error of the two-dimensional joint point .

[0089] In actual scenes, 3D annotation data may not be easy to obtain, but the 2D joint coordinates can be obtained through camera projection. Therefore, the network also needs to be supervised in 2D space. By projecting the 3D joint points onto the image plane, the corresponding 2D joint points are obtained, and then compared with the real 2D joint points. The formula is expressed as follows: .

[0090] In order to comprehensively consider these three losses, a weighted total loss function L is used during training to weight each loss. The formula for the total loss is as follows: ; In the formula, , are weight parameters respectively.

[0091] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a method for capturing three-dimensional human motions for human video, the method comprising: obtaining an image sequence of human video; obtaining human features obtained by fusion of any two consecutive frames of images in the image sequence; and converting the human features into three-dimensional human motion parameters.

[0092] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0093] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional human body motion capture method for human body video provided by the above-mentioned methods. The method includes: obtaining an image sequence of human body video; obtaining human body features obtained by fusion of any two consecutive frames of images in the image sequence; and converting the human body features into three-dimensional human body motion parameters.

[0094] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the method for capturing three-dimensional human body motion for human body video provided by the above-mentioned methods. The method includes: obtaining an image sequence of human body video; obtaining human body features obtained by fusion of any two consecutive frames of images in the image sequence; and converting the human body features into three-dimensional human body motion parameters.

[0095] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for capturing three-dimensional human body motion in human body video, characterized in that: include: Obtain an image sequence of a human body video; Acquire human features obtained by fusing any two consecutive frames of images in the image sequence; The human body features are converted into three-dimensional human body motion parameters.

2. The method for capturing three-dimensional human body motion in human body video according to claim 1, characterized in that: The step of obtaining the human body features obtained by fusing any two consecutive frames of images in the image sequence comprises: Obtaining coding features of any two consecutive frames of images in the image sequence, wherein the coding features are used to represent position information of a human body in the image; Acquire a key position in the image based on the coding feature; Fusion of encoding features of any two consecutive frames of images; The fused coding features and the key positions are used as human body features.

3. The method for capturing three-dimensional human body motions in human body video according to claim 2, characterized in that: The obtaining of coding features of any two consecutive frames of images in the image sequence includes: Obtain any two consecutive frames of images in the image sequence, and split each of the images into image blocks; Performing position coding on the image block to obtain a position coding block; The position coding block is spliced ​​with the body feature block to obtain coding features, wherein the body feature block is obtained from a standard three-dimensional human body model.

4. The method for capturing three-dimensional human body motion in human body video according to claim 1, characterized in that: The step of obtaining an image sequence of a human body video comprises: Acquire a human body video, and split the human body video into multiple frame images; The background part in each frame image is filtered to obtain an image sequence.

5. The method for capturing three-dimensional human body motion in human body video according to claim 1, characterized in that: The converting the human body features into three-dimensional human body motion parameters comprises: Inputting the human body features into a regressor to obtain action parameters output by the regressor; The motion parameters are input into an attention module to obtain the three-dimensional human motion parameters output by the attention module.

6. The method for capturing three-dimensional human motion in human body video according to claim 1, characterized in that: After converting the human body features into three-dimensional human body motion parameters, the method further includes: A three-dimensional human body corresponding to the three-dimensional human body motion parameters is generated.

7. The method for capturing three-dimensional human body motions in human body video according to claim 1, characterized in that: Before obtaining the image sequence of the human body video, the method further includes: Using a sample human body video to train a three-dimensional human body model, the three-dimensional human body model is used to implement the method for capturing three-dimensional human body motion for a human body video as described in any one of claims 1 to 6; The step of training the three-dimensional human body model using a sample human body video includes: Inputting the sample human body video into the three-dimensional human body model to obtain a sample three-dimensional human body output by the three-dimensional human body model; The error value of the sample three-dimensional human body model is calculated based on a loss function, and the parameters of the three-dimensional human body model are adjusted according to the error value. The loss function is used to measure the errors of the three-dimensional human body model in human body parameters, three-dimensional joint point coordinates and two-dimensional joint point coordinates.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for capturing three-dimensional human motion in a human body video as described in any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for capturing three-dimensional human motion in a human body video as claimed in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for capturing three-dimensional human motion in a human body video as claimed in any one of claims 1 to 7 is implemented.

Images