A semantic controllable human behavior posture video data augmentation system and method

Abstract

The application relates to a semantic controllable human behavior posture video data augmentation system and method, which performs standardized definition and collection on target human behaviors in an application scene, performs 3D human posture estimation through frame-by-frame analysis of a video, represents an estimation model by using human posture kinematics knowledge, constrains key points and degrees of freedom, constructs human posture key point embedding expression, generates as much as possible realistic new data by using a pre-trained adversarial generation network to game extraction information, performs 3D posture reconstruction on the generated data, realizes human posture migration on different target persons by using a posture migration algorithm, and realizes human behavior data augmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human posture analysis and data augmentation, and particularly relates to a semantic controllable human behavior posture video data augmentation system and method, which can be applied to public safety, health monitoring, smart education and the like application scenarios, constructs corresponding data resource pool, and serves diversified downstream detection tasks. BACKGROUND

[0002] The method of deep neural network as one of powerful tools needs sufficient training data to support. How to effectively train the network with insufficient samples has been a big challenge in deep learning. In some special application scenarios, it is difficult to collect abnormal behavior data; the cost of behavior data labeling is high, the labeled behavior data is less, and the sample quantity is unevenly distributed, which makes it difficult to obtain sufficient effective information from the data for the detection and classification tasks in the scene, so as to get satisfactory classification results. The video classification network trained based on insufficient data samples uses deep feature extraction network to extract the deep information of video data, which is easy to cause overfitting phenomenon.

[0003] In actual scenarios, such as public safety, health monitoring, smart education and the like application scenarios, human behavior information is often constrained by scenes, and sampling parameters, field of view range, data format and data quality are all affected to different degrees. The distribution of data set often follows long-tail distribution, and abnormal human posture only accounts for a small part of all data, which further leads to the lack of diversity of rare posture and the generalization ability of existing posture estimation network needs to be improved. Therefore, the rare abnormal behavior has the phenomena of small sampling quantity, low data richness and single behavior characteristics. The above facts together cause the problems of small data quantity, low data richness and difficult data collection of low-frequency abnormal human behavior information in specific application scenarios, and it is difficult to solve the problems by using general data set and directly migrating model method. Some abnormal detection models are developed based on target detection technology, and the information on single frame image is used as the basis for abnormal behavior recognition, which is not applicable in some scenarios, causing false detection. The difficulty of video data detection and video data generation is greater than that of target detection and image generation for single frame static image, so there is research significance for the augmentation of human behavior data.

[0004] Data augmentation techniques artificially generate different versions of the same dataset, increasing its data volume and data-related information. Data augmentation strategies are commonly used in the field of machine intelligence to address data scarcity and lack of data diversity. Among the various data augmentation strategies for different tasks, some proven effective methods for image data augmentation include geometric transformation, color enhancement, and cropping. However, there is little research on data augmentation for high-dimensional information with both spatial and temporal characteristics, such as human pose sequences. Meanwhile, most data augmentation algorithms aim to generalize existing data, and there are few data augmentation methods based on controlled editing of data meaning. SUMMARY

[0005] To address the problems in the prior art, the purpose of the present application is to provide a semantic controllable human behavior pose video data augmentation method that can autonomously generate new data samples corresponding to the behavior data, thereby improving the content richness and pose richness of human behavior data.

[0006] To achieve the above-mentioned purpose, the technical solution adopted by the present application is:

[0007] A semantic controllable human behavior pose video data augmentation system, the system comprising a data normalization module, a pose estimation module, a behavior generation module, a pose reconstruction module, and a pose transfer module;

[0008] The data normalization module is used to initialize and define the behavior information in the original video and then crop according to the defined segment length to obtain a behavior video data segment. The initialization and definition includes defining its category, behavior start video frame, behavior end video frame, behavior segment actual length, and normalized behavior video data segment length L, adjusting the data frame size to (r, r), where the value of L can be adjusted according to experience, i.e., adjusting the features of each batch of augmented data;

[0009] The pose estimation module is used to process the normalized behavior video data segment as follows:

[0010] (1) J key points are used to label and define human poses;

[0011] (2) A pose estimation network is used to label and define the behavior video data segment, obtaining labeled 3D pose key points and their coordinates, as well as estimated pose data, which is considered as a real human pose behavior sequence;

[0012] The labeled 3D pose key points and their coordinates are represented as:

[0013] {Joint i(X1, Y1, Z1), (X2, Y2, Z2)...(X J , Y J , Z J ), i = 0, 1, 2...J};

[0014] The pose estimation data represents a real action sequence of a given semantic label, and is specifically represented as:

[0015]

[0016] wherein J is the number of 3D human pose estimation key points, and L is the sequence length;

[0017] (3) Setting a mapping encoder P e , the human pose behavior sequence is projected into a one-dimensional vector , and then combined according to the number of frames, and finally the reconstructed behavior sequence data is mapped H is the dimension of the embedding space, and the transformed reconstructed behavior sequence data is obtained; the one-dimensional vector is represented as:

[0018]

[0019] wherein i is the index of the sequence S on the x-axis;

[0020] The behavior generation module takes the transformed reconstructed behavior sequence data in the pose estimation module as input data, judges whether the data needs to be edited according to the expansion requirement, and generates semantic attributes; if the judgment is no, the reconstructed behavior sequence data is taken as the input data of the pre-trained model generator G1, and new sequence sample data is generated; if the judgment is yes, the reconstructed behavior sequence data is taken as the input data of the attribute encoder, the semantic attribute of the action sequence is encoded, and the sub-vectors are combined and controlled according to the expansion requirement, to obtain the hidden space vector after the semantic attribute is regulated, and the hidden space vector is input into the pre-trained semantic controllable model generator G2 to generate new sequence sample data;

[0021] The pose reconstruction module is used for parsing and reconstructing the new sample data generated by the behavior generation module, constructing a decoder P d from the mapping encoder P e , and restoring to obtain the reconstructed human pose behavior sequence;

[0022] The pose transfer module is used for transferring the reconstructed human pose behavior sequence obtained by the pose reconstruction module to a target character by using a pose projection algorithm and a pose transfer algorithm, adjusting the target character parameter char, mapping the pose on different characters, and achieving the purpose of data augmentation.

[0023] The pre-training process of the model generator G1 is as follows:

[0024] (1) Pre-training the generative adversarial network using a public data set;

[0025] Specifically, the data of the public data set is subjected to data normalization and pose estimation processing to obtain a reconstructed behavior sequence and is taken as real data; random noise z is taken as the input of the model generator G1 to map to generated data G(z); and the real data and the generated data G(z) are taken as the input of the discriminator to determine whether the data generation effect is real by the discriminator;

[0026] (2) In the pre-training process of the model generator G1, the discriminator is used to measure the distribution difference between the reconstructed data and the original data, and the generator loss and the discriminator loss are fed back to the generator and the discriminator through back propagation for iterative training. The generator and the discriminator are alternately trained in the training process, and the loss function of the discriminator training is as follows:

[0027]

[0028] The loss function of the model generator G1 is as follows:

[0029]

[0030] (3) The perceptual loss in the training process of the model generator G1 pre-training module is as follows: the generated sample and the real value are respectively sent into the VGG16 network to extract features, and the L1 distance of M feature values in N layers of the VGG16 network is calculated as formula (1-4):

[0031]

[0032] Wherein, z refers to the model input, x refers to the corresponding real data G(z) refers to the generated data;

[0033] (4) The error L limb of the key point pair set ε = {i, j} is taken as a constraint condition to optimize the objective function of the generative adversarial network, and the error L limb is as follows:

[0034]

[0035] The pre-training of the semantic controllable model generator G2 includes a first training phase and a second training phase: the first training phase is as follows:

[0036] (1) For real data The data latent space is structured and semantically partitioned; the latent spaces Z and W are divided into multiple subspaces, each encoding a different semantic attribute; the partitioning rule is as follows: assuming each data sequence has N semantic attributes, the latent space is divided into N+1 subspaces, and the last subspace includes all the remaining attributes; the latent spaces Z and W are represented as a concatenation of subvectors in the two latent spaces, and the combined latent space vector w is input into the semantically controllable model generator G2;

[0037] (2) Based on the combined latent space vector w, the semantically controllable model generator G2 generates generated data containing each combined semantic attribute; the latent space is decoupled by the contrastive learning method. Two different latent vectors are taken in a contrastive training branch, and the contrastive loss of the two vectors on the k-th attribute is calculated so that the images generated by the two vectors are close to each other when they have the same attribute, and far away from each other when they have different attributes.

[0038] At the same time, the error L between key point pairs limb As constraints, the objective function Loss and error L of the generative adversarial network are optimized. limb as follows:

[0039]

[0040] (3) In the comparative training process, in order to make a specific attribute of the generated data controllable, a differentiable function M is defined for attribute k. k Mapping two-dimensional image data I onto a D-dimensional space, data with similar attributes are close to each other in the D-dimensional space, while data with different attributes are far apart. This means that two images I... i and I j The distance between attributes k is defined as their distance in the corresponding latent space as shown in equation (1-6):

[0041] d k (I i I j ) = dist(M k (I i M k (I j (1-6)

[0042] (4) Through the training in the first stage described above, entanglement is resolved in the hidden space of GAN;

[0043] The second phase of training is as follows:

[0044] (1) Design a multilayer perceptron attribute encoder to map each semantic attribute to a corresponding latent space sub-vector, and perform explicit control on each attribute; train an attribute encoder E for each semantic attribute k. k y k →wk where y k is the interpretable representation of the attribute, W k is the sub-vector representation of the semantic attribute in the latent space;

[0045] (2) generating the corresponding latent space sub-vector through the trained mapping encoder E k : y k → w k , and any combination of sub-vectors can be used to control the generation of latent space vectors containing different semantic attribute content

[0046] A semantic controllable human behavior posture video data augmentation method, the method is implemented by using the system as described above, the method comprises the following steps:

[0047] Step 1, uniformly normalizing the original video data.

[0048] Specifically, after initializing and defining the behavior information in the original video to be augmented, the defined segment length is cut to obtain a behavior video data segment; the initialization and definition includes defining its category, behavior starting video frame, behavior ending video frame, behavior segment actual length, and normalized behavior video data segment length L, adjusting the data frame size to (r, r), wherein the L value can be adjusted according to experience, that is, adjusting the features of each batch of augmented data;

[0049] Step 2, according to the initialization and definition information and the beginning and end video frames of each unit data, the original video data is cut to obtain a video data segment X i ={X1, X2,...X t}, i = 0, 1, 2...L (L is the total number of video segments), that is, it contains the complete expression of the behavior;

[0050] Step 3, based on the normalized behavior video data segment, the above data is subjected to human posture estimation and extraction of posture key point coordinate data;

[0051] Before performing human 3D posture representation, the behavior video data segment is labeled and defined through the posture estimation network, and the output is labeled with 3D posture key points and their coordinates, and the estimated posture data is regarded as a real human posture behavior sequence; the posture estimation module outputs labeled 3D posture key points and their coordinates {Joint i |(X1, Y1, Z1), (X2, Y2, Z2)...(X J , Y J , Z J ), i = 0, 1, 2...L}; this posture estimation data represents a real action sequence with a given semantic label where J is the number of human 3D pose estimation key points, and L is the length of the sequence.

[0052] Step 4, define the mapping encoder P, and map the human pose behavior sequence S to a one-dimensional vector sequence Frame-by-frame projection to a one-dimensional vector According to the number of frames, it is combined, as shown in formula (1-1), where i is the index of the sequence S on the x-axis; and finally, the mapping is performed to obtain the reconstructed behavior sequence data H is the dimension of the embedding space, and the transformed reconstructed behavior sequence data is obtained.

[0053]

[0054] Step 5, use the trained model generator G1 and the semantic controllable model generator G2 to generate data, with the transformed reconstructed behavior sequence data in the pose estimation module as input data, and output generated data;

[0055] Specifically as follows:

[0056] Step 5-1, according to the expansion requirement, it is judged whether the data needs to be edited and generated in semantic attribute; if it is judged as no, the transformed reconstructed behavior sequence data in step 4 is taken as the input data of the pre-trained generator G1, and new sequence sample data is generated;

[0057] Step 5-2, if it is judged as yes, the transformed reconstructed behavior sequence data in step 4 is taken as the input data of the attribute encoder, the semantic attribute of the action sequence is encoded, and the sub-vectors are combined and controlled according to the expansion requirement, to obtain the hidden space vector after semantic attribute regulation;

[0058] Step 5-3, input the hidden space vector in step 5-2 into the pre-trained semantic controllable model generator G2 to generate new sequence sample data;

[0059] Step 6, in the pose sequence reconstruction module, the new sequence sample obtained in step 5 needs to be parsed and reconstructed, and for the encoder P e in step 4, the decoder P is constructed according to the key point label and coordinate format standard in step 2 and step 4, and the reconstructed human pose behavior sequence is obtained by decoding; d

[0060] Step 7, the reconstructed human pose behavior sequence obtained in step 6 is used to realize the conversion between the camera coordinate system, the pixel coordinate system and the image coordinate system by using the camera focal length f, to obtain the corresponding 2D human pose representation, and the conversion formula is shown in formula 1-9:

[0061] ​

[0062] Step 8, using the pose migration network generator G3 to generate the 2D human body pose sequence representation converted in step 7 The corresponding real sequence According to the target character parameter char, map to the generated video image frame group The generation model is shown in formula (1-10):

[0063]

[0064] Wherein, G corresponds to the sequence generation function, the generated image of the current frame Based on the generated image of the previous frame And semantic labels Output; by changing the corresponding character data parameters, realizing the migration of the generated pose on different characters, finally obtaining the generated data, realizing data augmentation.

[0065] After adopting the above scheme, the target character behavior in the application scene is standardized and collected, 3D human body pose estimation is performed through video frame-by-frame analysis; the estimation model is represented by using human body pose kinematics knowledge, the key points and degrees of freedom are constrained, and the human body pose key point embedding expression is constructed; the extracted information is generated by using the pre-trained adversarial generation network, and new data as close to reality as possible is generated; the generated data is reconstructed to 3D pose, and the human body pose migration on different target characters is realized through the pose migration technology, and the human body behavior data augmentation is realized. Compared with the prior art, the present application has the following beneficial effects:

[0066] 1、The present application uses the method of constructing a sequence encoder to map the action sequence S into a two-dimensional embedding of the motion sequence that can be directly processed by a convolutional neural network, as the input of the generation model, to generate completely new action sequence data. Relieve the problem of lack of spatial continuity of the generated sequence when generating a continuous frame image from a single frame in the generation model.

[0067] 2、The present application uses an additional contrast learning hidden space editing method to construct a semantic controllable generation module, which divides different semantic attributes in different human body pose sequence data through a structured data hidden space, and finally achieves the effect of controllable editing of different motion feature attributes in the behavior sequence, and further generates semantic controllable behavior sequence data. This method can supervise the division of semantic attributes of motion sequence data to a certain extent, decouples the data hidden space, trains explicit control parameters for each defined attribute, and adjusts the parameters to achieve the purpose of semantic controllable pose sequence data augmentation.

[0068] 3、The application mainly adopts the adversarial generative network technology to generate and augment the 3D human body posture and action data, and simultaneously combines the posture migration technology to realize the style migration of the character content, and systematically improves the richness of the human behavior in the content level and the action level.

[0069] 4、The method can improve the content richness and posture richness of the human behavior data, autonomously generates new training samples, constructs a corresponding data resource pool, and serves the downstream detection task. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 The total flow structure of the application;

[0071] Figure 2 The specific flow structure of the application.

[0072] The application will be further described in detail below with reference to the accompanying drawings. DETAILED DESCRIPTION

[0073] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0074] As shown in the figure, Figure 1 The application discloses a semantic controllable human behavior posture video data augmentation system, which comprises a data normalization module, a posture estimation module, a behavior generation module, a posture reconstruction module and a posture migration module.

[0075] The data normalization module is used for defining and cutting the behavior information in the original video according to the defined segment length to obtain a behavior video data segment. The initialization definition includes defining the category, the behavior starting video frame, the behavior ending video frame, the behavior segment actual length, and the normalized behavior video data segment length L, and adjusting the data frame size to (r, r). Wherein, the value of L can be adjusted according to experience, that is, the characteristics of each batch of augmented data are adjusted. In this embodiment, the original video data is cut in the data normalization module, and the frame size is adjusted to (512, 512) fixed size to obtain the behavior video data segment.

[0076] The posture estimation module is used for processing the normalized behavior video data segment as follows:

[0077] (1) Adopt J key points to label and define human body posture. In this embodiment, the global human body 3D posture estimation key point is defined as J = 32, and the local human body 3D posture estimation key point is defined as J = 13.

[0078] (2) Use the posture estimation network to label and define the behavior video data segment, obtain the label with 3D posture key points and the coordinates thereof, and estimate the posture data, which is regarded as a real human body posture behavior sequence.

[0079] The label with 3D posture key points and the coordinates thereof are expressed as:

[0080] {Joint i |(X1, Y1, Z1), (X2, Y2, Z2)...(X J , Y J , ZJ), i = 0, 1, 2...J};

[0081] The posture estimation data represents a real action sequence with a given semantic label, and is specifically expressed as:

[0082]

[0083] Wherein, J is the number of human body 3D posture estimation key points, and L is the sequence length.

[0084] (3) Set a mapping encoder P e , project the human body posture behavior sequence frame by frame into a one-dimensional vector , then combine according to the number of frames, and finally map to obtain the reconstructed behavior sequence data H is the dimension of the embedding space, and the transformed reconstructed behavior sequence data is obtained. The one-dimensional vector is expressed as:

[0085]

[0086] Wherein, i is the index of the sequence S on the x-axis.

[0087] The behavior generation module takes the reconstructed behavior sequence data converted in the pose estimation module as input data, judges whether the data needs to be edited and generated in terms of semantic attributes according to the augmentation requirement; if the judgment is no (no need to control the generation of semantic attributes), the reconstructed behavior sequence data is taken as the input data of the pre-trained model generator G1 to generate new sequence sample data; if the judgment is yes (need to control the generation of semantic attributes), the reconstructed behavior sequence data is taken as the input data of the attribute encoder to perform semantic attribute coding on the action sequence, and according to the augmentation requirement, the sub-vectors are combined and controlled to obtain the hidden space vector after the semantic attribute regulation, and the hidden space vector is input into the pre-trained semantic controllable model generator G2 to generate new sequence sample data.

[0088] The pre-training process of the model generator G1 is as follows:

[0089] (1) Pre-training the generative adversarial network using public data sets.

[0090] Specifically, the reconstructed behavior sequence data of the public data set obtained through data normalization and pose estimation processing is taken as real data. Random noise z is taken as the input of the model generator G1 to generate data G(z). The real data and the generated data G(z) are taken as the input of the discriminator to judge whether the data generation effect is real.

[0091] (2) In the pre-training process of the model generator G1, the discriminator is used to measure the distribution difference between the reconstructed data and the original data, and the generator loss and the discriminator loss are fed back to the generator and the discriminator through back propagation for iterative training. The generator and the discriminator are alternately trained in the training process, and the loss function of the discriminator training is:

[0092]

[0093] The loss function of the model generator G1 is:

[0094]

[0095] (3) The perceptual loss in the training process of the model generator G1 pre-training module is to send the generated sample and the real value into the VGG16 network to extract features, and calculate the L1 distance of M feature values in N layers of the VGG16 network as shown in formula (1-4):

[0096]

[0097] Wherein, z refers to the model input, x is the corresponding real data G(z) is the generated data.

[0098] ​(4) Since the human pose structure can be regarded as a rigid body with independent and stable internal kinematic chains, the relative distance between these semantically important key points should remain stable. Therefore, the error L limb will be used as a constraint condition to optimize the objective function of the adversarial generative network. The error L limb is as follows:

[0099]

[0100] The pre-training of the semantic controllable model generator G2 includes a first training phase and a second training phase: the first training phase is as follows:

[0101] (1) The data latent space of the real data is structured and semantically divided. The latent space Z and W are divided into multiple subspaces, each of which encodes different semantic attributes. The division rule is as follows: assuming that each data sequence has N semantic attributes, the latent space is divided into N+1 subspaces, and the last subspace (i.e., the N+1 subspace) includes all the remaining attributes. The latent space Z and W are represented as a concatenation of sub-vectors in the two latent spaces, and the combined latent space vector w is input into the semantic controllable model generator G2.

[0102] (2) Based on the combined latent space vector w, the semantic controllable model generator G2 generates generated data containing each combined semantic attribute. The method of contrastive learning is used to decouple the latent space, and two different latent vectors are taken in one branch of the contrastive training. The contrastive loss of the two vectors for the kth attribute is calculated, and the purpose is to make the generated pictures of the two vectors close to each other (higher similarity) when the attribute is the same, and far away from each other when the attributes are different.

[0103] At the same time, since the human pose structure can be regarded as a rigid body with independent and stable internal kinematic chains, the relative distance between these semantically important key points should remain stable. Therefore, the error L limb between key point pairs will be used as a constraint condition to optimize the objective function Loss of the adversarial generative network. The error L limb is as follows:

[0104]

[0105] (3) In the process of contrastive training, in order to make a specific attribute of the generated data controllable, a differentiable function M k is defined for attribute k, which maps the two-dimensional image data I to a D-dimensional space, and data with similar attributes are close to each other in the D-dimensional space, and data with different attributes are far away from each other. The attribute distance k between two images I i and I j is defined as the distance between them in the corresponding latent space as shown in equation (1-6):

[0106] d k (I i I j ) = dist(M k (I i M k (I j (1-6)

[0107] (4) Through the training in the first stage mentioned above, deentanglement is achieved in the hidden space of GAN.

[0108] The second phase of training is as follows:

[0109] (1) A multilayer perceptron attribute encoder is designed to map each semantic attribute to a corresponding latent space sub-vector, thereby enabling explicit control of each attribute. An attribute encoder E is trained for each semantic attribute k. k y k →w k , where y k For an interpretable representation of an attribute, w k This is the sub-vector representation of the semantic attribute in the latent space.

[0110] (2) Through the mapping encoder E of each trained attribute k y k →w k This generates corresponding latent space sub-vectors, and any combination of these sub-vectors can be used to control the generation of latent space vectors containing different semantic attributes.

[0111] The pose reconstruction module is used to parse and reconstruct new sample data generated in the behavior generation module, based on the human keypoint label and coordinate data format and the mapping encoder P. e Construct decoder P d The sequence of human posture and behavior is then restored to obtain a reconstructed sequence.

[0112] The pose transfer module is used to transfer the reconstructed human pose sequence obtained from the pose reconstruction module to the target person using the pose projection algorithm and the pose transfer algorithm. By adjusting the target person parameter char, the pose is mapped to different people, thereby achieving the purpose of data augmentation.

[0113] like Figure 2 As shown, based on the above system, this invention also discloses a semantically controllable method for augmenting human behavior and posture video data, which includes the following steps:

[0114] Step 1: Standardize the raw video data.

[0115] Specifically, the behavior information in the original video to be amplified is initialized and defined, and then the defined segment length is cropped to obtain a behavior video data segment. The initialization and definition includes defining its category, behavior start video frame, behavior end video frame, behavior segment actual length, and normalized behavior video data segment length L, and adjusting the data frame size to (r, r). The value of L can be adjusted according to experience, that is, adjusted by the characteristics of each batch of amplified data.

[0116] Step 2, according to the initialization and definition information and the beginning and end video frames of each unit data, the original video data is cropped to obtain a video data segment X i = {X1, X2,... X t}, i = 0, 1, 2... L (L is the total number of video segments), that is, the complete expression of the behavior.

[0117] In this embodiment, the original video data is cropped in the data normalization module, and the frame size is adjusted to (512, 512) fixed size to obtain a behavior video data segment.

[0118] Step 3, based on the normalized behavior video data segment, the data is subjected to human pose estimation and extraction of pose key point coordinate data in the present application. Before human 3D pose representation, J key points are used to label and define the human pose. The global human 3D pose estimation key point is J = 32, and the local human 3D pose estimation key point is defined as J = 13.

[0119] The behavior video data segment is labeled and defined by the pose estimation network, and the output is labeled with 3D pose key points and coordinates, and the estimated pose data is regarded as a real human pose behavior sequence. The pose estimation module outputs labeled 3D pose key points and coordinates {Joint i |(X1, Y1, Z1), (X2, Y2, Z2)... (X J , Y J , Z J ), i = 0, 1, 2... L}. The pose estimation data represents a real action sequence with a given semantic label Where J is the number of human 3D pose estimation key points, and L is the sequence length.

[0120] Step 4, define a mapping encoder P to project the human pose behavior sequence frame by frame into a one-dimensional vector and then combined according to the number of frames. As shown in equation (1-1), where i is the index of sequence S on the x-axis. Finally, the reconstructed behavior sequence data H is the dimension of the embedding space, and the reconstructed behavior sequence data after transformation is obtained.

[0121]

[0122] Step 5, using the trained model generator G1 and the semantic controllable model generator G2, input the reconstructed behavior sequence data after transformation in the pose estimation module as input data, and output generated data.

[0123] Specifically as follows:

[0124] Step 5-1, according to the expansion requirement, it is judged whether the data needs to be edited and generated with semantic attributes. If it is judged as no (no need to control the semantic attribute generation), the reconstructed behavior sequence data after transformation in step 4 is input as the input data of the pre-trained generator G1 to generate new sequence sample data.

[0125] Step 5-2, if it is judged as yes (need to control the semantic attribute generation), the reconstructed behavior sequence data after transformation in step 4 is input as the input data of the attribute encoder, the action sequence is encoded with semantic attributes, and the sub-vectors are combined and controlled according to the expansion requirement to obtain the hidden space vector after semantic attribute regulation.

[0126] Step 5-3, input the hidden space vector in step 5-2 into the pre-trained semantic controllable model generator G2 to generate new sequence sample data.

[0127] The training process of the model generator G1 is as follows:

[0128] (1) First, pre-train the adversarial generation network using public data sets;

[0129] Specifically, the public data is processed through steps 1-4 to obtain the reconstructed behavior sequence data after data normalization and pose estimation Take random noise z as the input mapping of the adversarial generation network G1 to generate data G(z). Take the real data and the generated data as the input of the discriminator to judge whether the data generation effect is real.

[0130] (2) In the training process of the model generator G1 pre-training module, the distribution difference between the reconstructed data and the original data needs to be measured by the discriminator. The generator loss and the discriminator loss are fed back to the generator and the discriminator through back propagation for iterative training. The generator and the discriminator are alternately trained in the training process. The loss function of the discriminator training is:

[0131]

[0132] The loss function of the model generator is:

[0133]

[0134] (2) The perceptual loss during the training process of the model generator G1 pre-training module is shown in Equation 1-4. The generated samples and real values ​​are respectively fed into the VGG16 network to extract features. The L1 distance between them in the network with M feature values ​​in N layers is calculated as follows, where z refers to the model input and x is the corresponding real data S. e G(z) represents the generated data.

[0135]

[0136] Where z refers to the model input, and x is the corresponding real data. G(z) represents the generated data.

[0137] (4) Simultaneously, since the human body's posture structure can be considered as a rigid body with independent and stable internal kinematic chains, the relative distances between these semantically important key points should remain stable. Therefore, the error L between key points and the set ε = {i, j} limb This will be used as a constraint to optimize the objective function of the generative adversarial network.

[0138]

[0139] The training of the semantically controllable model generator G2 consists of a first training phase and a second training phase. The first training phase is as follows:

[0140] (1) Regarding real data The latent space of the data is structurally semantically partitioned. The latent spaces Z and W are divided into multiple subspaces, each encoding a different semantic attribute. The partitioning rule is as follows: assuming each data sequence has N semantic attributes, the latent space is divided into N+1 subspaces, with the last subspace (i.e., the N+1th subspace) including all remaining attributes. The latent spaces Z and W are represented as a concatenation of subvectors from the two latent spaces. The combined latent space vector w is input into the semantically controllable model generator G2.

[0141] (2) Based on the combined latent space vector w, the semantically controllable model generator G2 generates generated data containing the semantic attributes of each combination. The latent space is decoupled using a contrastive learning method, where the loss function for contrastive learning is:

[0142]

[0143] Among them, l k The contrastive loss component is for semantic attribute k. Two distinct latent vectors z are taken in a contrastive training branch. i and z j Calculate the contrastive loss of two vectors with respect to the k-th attribute, with the aim of making z...i and z j generated picture I i and I j in the same attribute Close to each other, different attributes are far away from each other.

[0144] The contrast loss for each attribute is:

[0145]

[0146] where Ii=G(zi) represents the image generated by the latent space vector zi, is the k-th sub-vector of zi, d k is the distance function of the k-th attribute, is the threshold of each attribute related to the same and different sub-vectors, is a constant for normalizing the loss according to the number of same and different loss components. Through discrimination, the similarity between the generated image Ii, Ij and other images can be gradually fed back and punished during the training process.

[0147] (3) At the same time, since the human body posture structure can be regarded as a rigid body with independent and stable internal kinematic chain, the relative distance between these key points which are semantically important should be kept stable. Therefore, the error L limb of the key point pair ε={i, j} will be taken as a constraint condition to optimize the objective function of the generative adversarial network.

[0148]

[0149] (4) In the process of contrast training, in order to make a specific attribute of the generated image controllable, a differentiable function M k is defined for attribute k, which maps two-dimensional image data I into D-dimensional space, and data with similar attributes are close to each other in D-dimensional space, while data with different attributes are far away from each other. The attribute distance k between two images I i and I j is defined as their distance in the corresponding latent space as shown in equation (1-6):

[0150] d k (I i , I j ) = dist(M k (I i ), M k (I j )) (1-6)

[0151] (4) Through the above first stage training, the entanglement in the latent space of GAN is realized.

[0152] The second training stage is as follows:

[0153] (1) Design a multi-layer perceptron attribute encoder to map each semantic attribute to a corresponding hidden space sub-vector, and then explicitly control each attribute. Train an attribute encoder E k : y k → w k for each semantic attribute k, where y k is the interpretable representation of the attribute, and w k is the sub-vector representation of the semantic attribute in the hidden space.

[0154] (2) Generate the corresponding hidden space sub-vector by training the mapping encoder E k : y k → w k for each attribute, and use any combination of sub-vectors to control the generation of hidden space vectors containing different semantic attribute content

[0155] Step 6, in the pose sequence reconstruction module, the new sequence samples obtained in step 5 need to be parsed and reconstructed. For the encoder P e in step 4, follow the key point label and coordinate format standard in steps 2 and 4 to build the decoder P d , and obtain the reconstructed human pose behavior sequence by decoding.

[0156] Step 7, use the camera focal length f to realize the conversion between the camera coordinate system, pixel coordinate system and image coordinate system, and obtain the corresponding 2D human pose representation. The conversion formula is shown in 1-9:

[0157]

[0158] Step 8, use the pose transfer network generator G3 to generate the 2D human pose sequence representation converted in step 7 to the corresponding real sequence according to the target character parameter char, and map it to the generated video image frame group The generation model is shown in formula (1-10):

[0159]

[0160] Where G is the sequence generation function, the generated image of the current frame is based on the generated image of the previous frame and the semantic label output. By changing the corresponding character data parameters, the generated pose is transferred to different characters, and the generated data is finally obtained, realizing data augmentation.

[0161] To sum up, the application is aimed at the target person behavior in the application scene, and the 3D human body posture estimation is performed through the video frame-by-frame analysis; the estimated model is represented by using the human body posture kinematics knowledge, the key points and the degrees of freedom are constrained, and the human body posture key point embedding expression is constructed; the extracted information is generated by using the pre-trained adversarial generation network, and the new data as close to reality as possible is generated; the generated data is subjected to 3D posture reconstruction, the human body posture migration on different target persons is realized through the posture migration technology, and the human body behavior data expansion is realized.

[0162] 1、The application uses the method of constructing a sequence encoder to map the action sequence S into a motion sequence two-dimensional embedding that can be directly processed by a convolutional neural network, as the input of the generation model, to generate brand-new action sequence data. This alleviates the problem of lack of spatial continuity of the generated sequence when generating continuous frame images in the generation model.

[0163] 2、The application uses the hidden space editing method with additional contrast learning to construct a semantic controllable generation module, and different semantic attributes in different human body posture sequence data are compared and divided through the structured data hidden space, so that the effect of controllable editing of different motion feature attributes in the behavior sequence is finally achieved, and semantic controllable behavior sequence data is generated. This method can supervise the division of semantic attributes of motion sequence data, to a certain extent, decouples the data hidden space, trains explicit control parameters for each defined attribute, and adjusts the parameters to achieve the purpose of semantic controllable posture sequence data expansion.

[0164] 3、The application mainly uses the adversarial generation network technology to generate and expand the 3D human body posture and action data, and combines the posture migration technology to realize the style migration of the person content, and systematically improves the richness of the human body behavior in the content level and the action level.

[0165] 4、The method can improve the content richness and posture richness of the human body behavior data, autonomously generate new training samples, construct a corresponding data resource pool, and serve downstream detection tasks. The application has the characteristics of reducing manual labor, optimizing resources, generating diversified data, and small running load.

[0166] The above is only an embodiment of the application, and does not limit the technical scope of the application in any way, so any slight modification, equivalent change and modification made according to the technical essence of the application to the above embodiment are still within the scope of the technical solution of the application.

Claims

1. A semantically controllable human behavior and posture video data augmentation system, characterized in that: The system includes a data normalization module, a pose estimation module, a behavior generation module, a pose reconstruction module, and a pose transfer module; The data normalization module is used to initialize and define the behavioral information in the original video, and then trim it according to the defined segment length to obtain behavioral video data segments. The initialization definition includes defining its category, the starting video frame of the behavior, the ending video frame of the behavior, the actual length of the behavioral segment, and the normalized behavioral video data segment length. Adjust the data frame size to Among them, the length of video data segments The value is adjusted based on the characteristics of each batch of amplified data; The pose estimation module is used to process the normalized behavioral video data segments as follows: (1) Adopt The key points are marked and defined to represent human posture; (2) Use a pose estimation network to label and define the behavioral video data segments to obtain the annotations and coordinates of 3D pose key points, as well as the estimated pose data, which is regarded as a series of real human pose behavior sequences. The annotations and coordinate representations of 3D pose keypoints are as follows: ； Pose estimation data represents the true action sequence given a semantic label, specifically as follows: ； in, The number of key points for 3D human pose estimation. The sequence length; (3) Set the mapping encoder The sequence of human postures and behaviors is projected frame by frame into a one-dimensional vector. Then, the data is combined according to the number of frames to finally map and reconstruct the behavior sequence data. H is the dimension of the embedding space, resulting in the transformed reconstructed behavioral sequence data; a one-dimensional vector. Represented as: Where i is the index of sequence S on the x-axis; The behavior generation module takes the reconstructed behavior sequence data transformed in the pose estimation module as input data. Based on the amplification requirements, it determines whether semantic attribute editing of the data is needed. If the determination is no, the reconstructed behavior sequence data is used as input data to the pre-trained model generator G1 to generate new sequence sample data. If the determination is yes, the reconstructed behavior sequence data is used as input data to the attribute encoder to encode the action sequence semantic attributes. Based on the amplification requirements, the sub-vectors are combined and controlled to obtain the semantically attribute-tuned latent space vector. The latent space vector is then input to the pre-trained semantically controllable model generator G2 to generate new sequence sample data. The pose reconstruction module is used to parse and reconstruct the new sample data generated in the behavior generation module, based on the human keypoint label and coordinate data format and the mapping encoder. Build Decoder Then, the sequence of human posture and behavior is restored to obtain the reconstructed sequence. The pose transfer module is used to transfer the reconstructed human pose sequence obtained from the pose reconstruction module to the target person using pose projection and pose transfer algorithms, by adjusting the target person's parameters. This allows for the mapping of the pose onto different characters, thus achieving the goal of data augmentation.

2. The semantically controllable human behavior posture video data augmentation system according to claim 1, characterized in that: The pre-training process of the model generator G1 is as follows: (1) Pre-train adversarial generative networks using publicly available datasets; Specifically, the reconstructed behavior sequence is obtained by processing publicly available datasets through data normalization and pose estimation. And treat it as real data; with random noise The input mapping to the model generator G1 is used to generate data. ; Using real data and generated data As input to the discriminator, the discriminator determines whether the generated data is realistic; (2) During the pre-training of the model generator G1, the discriminator is used to measure the distribution difference between the reconstructed data and the original data. The generator loss and discriminator loss are fed back to the generator and discriminator through backpropagation for iterative training. The generator and discriminator are trained alternately during the training process. The loss function for training the discriminator is: The loss function of model generator G1 is: (3) The perceptual loss during the training process of the model generator G1 pre-training module is to feed the generated samples and real values ​​into the VGG16 network to extract features, and calculate the L1 distance of the M feature values ​​in the N layers of the VGG16 network as shown in equation (1-4): in, Refers to the model input. To correspond to real data , To generate data; (4) Set of key point pairs error As a constraint, the objective function of the generative adversarial network is optimized, and the error... as follows: 。 3. The semantically controllable human behavior posture video data augmentation system according to claim 1, characterized in that: The pre-training of the semantically controllable model generator G2 includes a first training phase and a second training phase: The first training phase is as follows: (1) Regarding real data The data latent space is structurally semantically partitioned; the latent space is... and The data sequence is divided into multiple subspaces, each encoding different semantic attributes. The partitioning rules are as follows: assuming each data sequence has... Each semantic attribute divides the latent space into: Each subspace contains all remaining properties; the implicit space... and Represented as a concatenation of subvectors in two latent spaces, the combined latent space vectors Input semantically controllable model generator G2; (2) Based on the combined latent space vector The semantically controllable model generator G2 generates generated data containing various combined semantic attributes; a contrastive learning method is used to decouple the latent space. In a contrastive training branch, two different latent vectors are taken, and the contrastive loss of the two vectors with respect to the k-th attribute is calculated. This allows the images generated by the two vectors to be close to each other when they have the same attribute, and far apart when they have different attributes. At the same time, the error between key point pairs As constraints, the objective function Loss and error of the generative adversarial network are optimized. as follows: (3) In the process of comparative training, in order to make a specific attribute of the generated data controllable, a differentiable function is defined for attribute k. Mapping two-dimensional image data I to a D-dimensional space, data with similar attributes are close to each other in the D-dimensional space, while data with different attributes are far apart. This allows us to map two images... and The k-attribute distance between them is defined as their distance in the corresponding latent space as shown in equation (1-6): (4) Through the training in the first stage described above, deentanglement is achieved in the latent space of GAN; The second phase of training is as follows: (1) Design a multilayer perceptron attribute encoder to map each semantic attribute to a corresponding latent space sub-vector, and perform explicit control on each attribute; for each semantic attribute Training an attribute encoder ,in An interpretable representation of an attribute. This is the sub-vector representation of the semantic attribute in the latent space; (2) Through the mapping encoder of each attribute after training This generates corresponding latent space sub-vectors, and any combination of these sub-vectors can be used to control the generation of latent space vectors containing different semantic attributes. .

4. A semantically controllable method for augmenting human behavior and posture video data, characterized in that: The method is implemented using the system described in any one of claims 1-3, and the method includes the following steps: Step 1: Standardize the raw video data. Specifically, after initializing and defining the behavioral information in the original video to be augmented, the data is trimmed according to the defined segment length to obtain behavioral video data segments. The initialization definition includes defining its category, the starting video frame of the behavior, the ending video frame of the behavior, the actual length of the behavioral segment, and the normalized behavioral video data segment length. Adjust the data frame size to ,in, The value can be adjusted based on experience, that is, adjusted according to the characteristics of each batch of amplified data; Step 2: Based on the initialization definition information and the beginning and end video frames of each unit's data, trim the original video data to obtain video data segments. , The length of the video data segment, i.e., the complete representation of the behavior; Step 3: Based on the normalized behavior video data segments, perform human pose estimation on the above data and extract the pose key point coordinate data; Before representing the human body in 3D pose, a pose estimation network is used to annotate and define the behavioral video data segments, outputting annotations and coordinates of 3D pose key points. The resulting estimated pose data is considered as a sequence of real human pose behaviors. The pose estimation module will output annotations and coordinates of 3D pose key points. This pose estimation data represents the true action sequence given a semantic label. ,in The number of key points for 3D human pose estimation. The sequence length; Step 4: Define the mapping encoder Human posture behavior sequence Frame-by-frame projection into a one-dimensional vector Then, the sequences are combined according to the number of frames, as shown in equation (1-1), where i is the index of sequence S on the x-axis; finally, the reconstructed behavioral sequence data is obtained. H is the dimension of the embedding space, and the transformed reconstructed behavior sequence data is obtained. Step 5: Using the trained model generator G1 and semantically controllable model generator G2, take the reconstructed behavior sequence data transformed in the pose estimation module as input data and output generated data. Specifically as follows: Step 5-1: Based on the amplification requirements, determine whether semantic attribute editing of the data is required; if the determination is no, the reconstructed behavioral sequence data transformed in step 4 is used as the input data of the pre-trained generator G1 to generate new sequence sample data. Step 5-2: If the judgment is yes, the reconstructed behavior sequence data after transformation in step 4 is used as the input data of the attribute encoder to encode the semantic attributes of the action sequence. According to the amplification requirements, the sub-vectors are combined and controlled to obtain the latent space vector after semantic attribute regulation. Step 5-3: Input the latent space vector from Step 5-2 into the pre-trained semantically controllable model generator G2 to generate new sequence sample data; Step 6: In the pose sequence reconstruction module, the new sequence samples obtained in step 5 need to be parsed and reconstructed. This involves the encoder from step 4. The decoder is built following the keypoint label and coordinate format standards in steps 2 and 4. The reconstructed human posture and behavior sequence is obtained through decoding; Step 7: Utilize the camera focal length to reconstruct the human pose sequence obtained in Step 6. The transformation between the camera coordinate system, pixel coordinate system, and image coordinate system is implemented to obtain the corresponding 2D human pose representation. The transformation formula is shown in 1-9: Step 8: Represent the 2D human pose sequence transformed in Step 7 using the pose transfer network generator G3. , the corresponding real sequence Based on the target person's parameters Mapped to the generated video image frame group The generative model is shown in equation (1-10): in, The corresponding sequence generation function generates the image for the current frame. Image generated based on the previous few frames and semantic tags Output: By changing the corresponding character data parameters, the generated pose is transferred to different characters, and finally the generated data is obtained, thus realizing data augmentation.

Images