A video sample generation method based on multi-attribute synthesis

Abstract

The invention discloses a video sample generation method based on multivariate attribute synthesis, comprising the following steps: multivariate attribute model construction: firstly, the multivariate static attributes and dynamic attributes of the foreground object frame by frame are decomposed from the video according to the pre-trained video encoder and attribute decomposition network, and then the static attributes and dynamic attributes are respectively processed for consistency and smoothness according to the consistency constraint and the smoothness constraint; finally, the multivariate attribute model is generated by vector splicing; multivariate attribute embedding space generation: an autoencoder based on a neural network is constructed, the autoencoder is trained and a multivariate attribute embedding space is generated; multivariate attribute synthesis: sampling from the multivariate attribute embedding space, firstly the multivariate attributes of each frame are generated by using the trained decoder, and then the three-dimensional geometry and texture of the target object of each frame are calculated by using the shape synthesis and illumination synthesis methods, and finally, the video sample is rendered. Repeating the multivariate attribute synthesis process can obtain multiple video samples specified by the user.

Description

Technical Field

[0001] The invention relates to a video sample generation method, in particular to a video sample generation method based on multi-attribute synthesis. Background Art

[0002] With the rapid development of the information age, video data is showing an explosive growth trend, and the demand for downstream applications based on video is increasing, such as video object detection, video prediction, etc. Thanks to the development of artificial intelligence, artificial intelligence technology represented by deep learning has achieved great success in various applications, and its success is inseparable from the support of a large number of training samples. However, despite the large amount of video data on the Internet, it is still very difficult to collect a variety of video samples.

[0003] Although a large number of methods for synthesizing video data have emerged in recent years, for example, Yang C, Wang Z, Zhu X, et al. Pose guided human video generation [C] / / Proceedings of the European Conference on Computer Vision (ECCV). 2018: 201-216., Dorkenwald M, Milbich T, Blattmann A, et al. Stochastic image-to-video synthesis using cinns [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 3742-3753. However, the goal of these methods is to improve the quality of synthesized videos and cannot meet the diversity requirements of video samples.

[0004] The data diversity required for video samples is mainly reflected in the variety of geometry, texture, lighting, posture and deformation of the target in the video. Nowadays, most of the literature focuses on solving the diversity problem in image synthesis, such as: Literature 1: Z.Yu and C.Zhang, "Image based static facial expression recognition with multiple deep network learning," in Proceedings of the 2015ACM onInternational Conference on Multimodal Interaction.ACM, 2015, pp.435-442. The orientation and scale diversity problems are solved by traditional cropping, flipping and other enhancement methods. Literature 2: Zhou H, Liu J, Liu Z, et al. Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2020:5911-5920. First, a single image is fitted using a 3D model, and then the 3D model is rotated and rendered to obtain the initial rotated face image. Finally, a realistic rotated face image is obtained through the GAN framework to solve the posture diversity problem. Reference 3: I.Goodfellow, J.Pouget-Abadie, M.Mirza, B.Xu, D.Warde-Farley, S.Ozair, A.Courville, and Y.Bengio, "Generative adversarialnets," in Advances in neural information processing systems, 2014, pp.2672-2680. The deformation diversity problem is solved by GAN.Reference 4: M. Shin, M. Kim, and D.-S. Kwon, "Baseline CNN structure analysis for facial expression recognition," in Robot and Human Interactive Communication (RO-MAN), 2016 25th IEEE International Symposium on. IEEE, 2016, pp. 724-729. The face image is decomposed into high-frequency components and low-frequency components by Fourier transform, and the low-frequency components are suppressed. Finally, a new face image is reconstructed by inverse Fourier transform. However, this method can only improve the contrast of the image and cannot solve the problem of illumination diversity. Reference 5: Feng Li. Face illumination regularization method based on SFS algorithm [D]. 2005. The face image is decomposed into three components: albedo, normal and illumination by using the SFS algorithm. The illumination component is replaced by the forward light source and then reconstructed into a face image to achieve image illumination regularization. However, this method simply replaces the ambient light source in the image with the forward light source and cannot solve the problem of illumination diversity. Reference 6: Wang C, Wang C, Xu C, et al. Tag disentangled generative adversarial networks for object image re-rendering [C]. 2017. By decomposing the face image into a series of labels such as identity, posture, and illumination, and then generating a new face image by manipulating the illumination label. Their method has a certain ability to generate images with diverse illumination. Some of the above methods are mainly aimed at the expansion of image samples. In actual situations, the application occasions of video are often more than those of single image applications, such as video surveillance, live broadcast, etc. Therefore, it is necessary to propose an expansion solution for video data. The existing video data expansion solutions include: Patent 1:. Chen Li , Jia Gaoguo ,A data enhancement method for video classification tasks with a small amount of data: China, 10743444.7[P]. 2019-12-24. However, they simply merged the existing data sets without making special designs for diversity. Based on some of the above methods, this paper provides a method for expanding video data to meet diversity requirements. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a method for generating video samples based on multi-attribute synthesis in view of the shortcomings of the prior art.

[0006] In order to solve the above technical problems, the present invention discloses a method for generating video samples based on multi-attribute synthesis, comprising the following steps:

[0007] Step 1, constructing a multivariate attribute model, the method comprising: given an initial video data set, preprocessing each video in the data set, and decomposing each video into multivariate static attributes and dynamic attributes of a foreground object frame by frame according to a pre-trained video encoder and an attribute decomposition network; performing consistency and smoothness processing on the multivariate static attributes and dynamic attributes according to consistency constraints and smoothness constraints respectively; generating a multivariate attribute model by vector concatenation;

[0008] Step 2, generating a multivariate attribute embedding space, the method comprising: constructing an autoencoder network based on a neural network; mapping the multivariate attribute model constructed in step 1 to a low-dimensional embedding space through an encoder in the autoencoder network; restoring the multivariate attribute model to a decoder in the autoencoder network, and training the autoencoder network through a constrained reconstruction loss; generating a multivariate attribute embedding space by recording the numerical value range of the low-dimensional embedding space;

[0009] Step 3, synthesizing multi-attributes, the method includes: sampling from the multi-attribute embedding space, using the decoder trained in step 2 to generate the multi-attributes of each frame; using shape synthesis and lighting synthesis methods to calculate the three-dimensional geometry and texture of the target object in each frame; rendering to generate video samples; repeating the multi-attribute synthesis process to obtain the specified multiple video samples, and finally completing the generation of video samples based on multi-attribute synthesis.

[0010] In the present invention, step 1 comprises the following steps:

[0011] Step 1-1, video preprocessing;

[0012] Step 1-2, video attribute decomposition;

[0013] Steps 1-3, attribute preprocessing;

[0014] Steps 1-4, multi-attribute vector concatenation;

[0015] Step 1-5: for each video in a given initial video data set, repeat steps 1-1 to 1-3 until every video in the video data set is processed and the construction of the multivariate attribute model is completed.

[0016] In the present invention, the video preprocessing method in step 1-1 includes:

[0017] The pre-trained target detector is used to detect the target frame by frame in the video, and then the target in the video is cut out according to the bounding box obtained by the detection. Finally, each cut target is scaled to a size of 224×224 to obtain the processed video frame sequence I i, i=1,…,T, i represents the i-th frame, and T represents the total number of frames in the video.

[0018] The video attribute decomposition method in step 1-2 of the present invention includes:

[0019] Use the pre-trained video encoder to convert the video sequence I obtained by step 1-1 into i Encoded into frame-by-frame feature vector f i ; The video encoder includes T residual network networks with shared parameters, whose input is a 224×224 three-channel image and whose output is an n-dimensional feature vector;

[0020] Use the pre-trained attribute decomposition network to decompose the video into frame-by-frame geometric f i g , Texture i a 、Posture i p 、Light i l and deformation f i m ; The attribute decomposition network contains 5 sub-networks, including: geometry estimation network, texture estimation network, posture estimation network, illumination estimation network and deformation estimation network; wherein, the geometry estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is an m-dimensional feature vector. g dimensional feature vector for geometric estimation; the texture estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is an m-dimensional feature vector. a dimensional feature vector for texture estimation; the illumination estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is a 27-dimensional feature vector for illumination estimation; the posture estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is a 6-dimensional feature vector for posture estimation; the deformation estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is an m-dimensional feature vector. g dimensional feature vector, used for deformation estimation;

[0021] The attribute preprocessing method in steps 1-3 of the present invention includes static attribute consistency processing and dynamic attribute smoothness processing;

[0022] The static attribute consistency processing method includes: using the following objective function to solve the consistency vector f con :

[0023]

[0024] The frame-by-frame geometry f obtained in step 1-1 is ih and texture f i a Input them into the static attribute consistency processing method to obtain the geometry after consistency processing And texture

[0025] The method for processing the smoothness of dynamic attributes includes: splicing T column vectors in the column direction in time order into an n×T matrix; splitting the matrix by row to obtain n T-dimensional vectors f′ j , where j = 1, ..., n, and each vector is processed as follows:

[0026] f″ j =kernel*f′ j

[0027] Where * represents discrete convolution operation, kernel = [0.0545, 0.244, 0.403, 0.244, 0.0545] represents a one-dimensional convolution kernel, and for vector f′ j Use kernel to perform convolution operation with a step size of 1 to obtain the smoothed result f″ j ; After smoothing all n T-dimensional vectors in the above method, reassemble them into an n×T matrix in row order, and then split the matrix by column to obtain T smoothed n-dimensional column vectors; the frame-by-frame posture f obtained in step 1-1 is i p 、Light i l and deformation f i m Input dynamic attribute smoothness processing method to obtain smoothed posture illumination And deformation

[0028] The multivariate attribute vector splicing method in step 1-4 of the present invention comprises: for the geometric vector obtained in step 1-2 Texture attitude illumination And deformation Perform vector concatenation operations to build a multivariate attribute model:

[0029]

[0030] where <·,·,…,·> represents the concatenation operation of multiple vectors, S i Represents the result of concatenating the multi-attribute vectors of the i-th frame.

[0031] In the present invention, step 2 comprises:

[0032] Step 2-1, construct an autoencoding network;

[0033] Step 2-2, self-encoding training;

[0034] Step 2-3, generate multi-attribute embedding space.

[0035] The method for constructing an autoencoding network in step 2-1 of the present invention comprises:

[0036] Construct an autoencoder network based on a neural network, which includes an encoder and a decoder;

[0037] The encoder contains m cells, which share parameters. The τth cell accepts the cell state C of the τ-1th cell. τ-1 , hidden state h τ-1 And the current input x τ As input; output is the cell state C of the current cell τ And the hidden state h τ , the update rule of the autoencoder network is as follows:

[0038] f τ =σ(W f · <h τ-1 ,x τ >+b f )

[0039] p τ =σ(W p · <h τ-1 ,x τ >+b p )

[0040]

[0041]

[0042] o τ =σ(W o · <h τ-1 ,x τ >+b o )

[0043] h τ =o τ *tanh(C τ )

[0044] Where <·,·> represents the concatenation operation of two vectors, σ is the sigmoid function, tanh is the tanh function, (W f ,b f ),(W p ,bp ),(W C ,b C ),(W o ,b o ) represent the weights and biases of the four fully connected layers respectively; the input of the encoder is the multivariate attribute model, and the output is a t-dimensional feature code f attr ;

[0045] The structure of the decoder is consistent with the dynamic mode encoding network. The input of the decoder is a t-dimensional vector f attr , the output is a multivariate attribute model; in the update rule, the cell state C1 and hidden state h1 of the encoder at time 1 are set to all zero vectors, and the input x of the decoder at all times τ , are set to f attr , where τ=1,…,m.

[0046] The autoencoder training method described in step 2-2 of the present invention comprises:

[0047] The autoencoder network described in step 2-1 is trained using the stochastic gradient descent method. The loss function consists of three parts: a reconstruction loss l sigrec , a smooth loss l smooth and a consistency loss l consis ; Among them, the reconstruction loss calculation method is as follows:

[0048]

[0049] Among them, S i represents the i-th vector of the multivariate attribute model, Represents the i-th vector of the reconstructed multivariate attribute model;

[0050] The smoothing loss calculation method is as follows:

[0051]

[0052] The calculation method of the multivariate attribute matrix S is as follows: all vectors in the reconstructed multivariate attribute model are In the column direction, they are stacked into a matrix. [:,:] represents a slice operation. [m g +m a :,:] means to select the mth a +m g Go to the last row of elements, It means to find the second-order derivative of the matrix in the x direction;

[0053] The consistency loss is calculated as follows:

[0054]

[0055] Finally, the total training loss of the entire network is as follows:

[0056] l=λ1l sigrec +λ2l smoo +λ3l consis

[0057] Among them, λ1,λ2,λ3 are balance factors.

[0058] The method for generating a multi-attribute embedding space described in step 2-3 of the present invention includes:

[0059] For all multi-attribute models, use the encoder in step 2-1 to generate feature vectors Calculate the encoding range of each dimension and store it in two vectors r l ,r h , where the vector r l Each dimension records the minimum value of the feature vector in that dimension, and the vector r h Each dimension records the maximum value of the feature vector in that dimension, and the multi-attribute embedding space consists of all vectors r that satisfy the following conditions: {r|r l,α ≤r α ≤r h,α ,α=1,…,t}, where r l,α Represents r l The value of the αth dimension, r α Represents the value of the αth dimension of vector r, r h,a Represents r h The value of the αth dimension.

[0060] In the present invention, step 3 comprises:

[0061] Step 3-1, sample a feature vector from the multi-attribute embedding space, and decode the feature vector into a multi-attribute model according to the decoder trained in step 2-2

[0062] Step 3-2, generate dynamic attributes and static attributes: extract the static attribute part from the decoded multivariate attribute model and the dynamic properties section It is further divided into geometric Texture illumination attitude And deformation Generate representative static properties, geometry Texture

[0063] Step 3-3, Synthesize geometric shapes: 3D mesh model of frame t The calculation method is as follows:

[0064]

[0065] Among them, G avg It is the average geometric shape defined in the three-dimensional deformable model 3DMM, with a dimension of 3n, where n represents the number of mesh vertices; each 3 dimensions represents the x, y, and z coordinate values ​​of a three-dimensional vertex; G ne It is the neutral shape basis defined in 3DMM, with the same dimension as G avg , each 3 dimensions represent the offset of a 3D vertex coordinate value; G mor It is the deformation basis of the shape defined in 3DMM, with the same dimensions and meaning as G ne The final G mesh is a vector with a dimension of 3n. By reshaping it, it is normalized into an n×3 matrix, where each row of the matrix represents the coordinate value of a vertex of the three-dimensional mesh of the model;

[0066] Step 3-4, synthesize texture lighting: First synthesize the texture and restore the calculation formula of the vertex color value as follows:

[0067]

[0068] Among them, T avg It is a vertex-by-vertex texture of the average geometric shape defined by 3DMM, with a dimension of 3n, where each 3 dimensions represent the R, G, and B values ​​of a 3D vertex; T ne It is the texture base defined in 3DMM, with the same dimensions as T avg , each 3 dimensions represents the offset of the R, G, B values ​​of a 3D vertex; the final T mesh is a vector of dimension 3n, which is normalized into an n×3 matrix by reshaping it, where each row of the matrix represents the R, G, B value of a vertex of the model's three-dimensional mesh;

[0069] Synthetic lighting: lighting at frame t The synthesis method is as follows:

[0070]

[0071] in, Represents the color value of the i-th vertex of the synthesized texture, represents the value of the jth dimension of the tth frame illumination, and Y(j) represents the jth basis of the spherical harmonic function; perform the above operation on each vertex to obtain the vertex-by-vertex texture color after synthetic illumination

[0072] Step 3-5, rendering: The synthesized 3D geometric shape of the tth frame Composite lit per-vertex textures And posture Send it to the renderer to draw the tth frame image I t And the mask image M of the tth frame t , randomly cropped from another background image and I t Images of the same size I b , synthesize the final video frame, the synthesis method is as follows:

[0073]

[0074] Where ⊙ represents element-by-element multiplication;

[0075] Step 3-6, write the generated T frame image into the video stream to obtain the final video sample;

[0076] Step 3-7, repeat steps 3-1 to 3-6 until the set number is reached, and the generation of video samples based on multi-attribute synthesis is completed.

[0077] Beneficial effects:

[0078] By constructing a multi-attribute model from the initial video dataset, generating a multi-attribute embedding space, and generating new samples through multi-attribute synthesis, the scale of the initial video dataset is effectively expanded and the diversity of the initial video dataset is increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0080] Figure 1 It is a schematic diagram of the processing flow of the present invention.

[0081] Figure 2 This is a diagram of a picture after some feature vectors are manually selected and rendered after random sampling from the multi-attribute embedding space.

[0082] Figure 3 This is a schematic diagram of the picture after lighting synthesis.

[0083] Figure 4 It is a schematic diagram of the mask diagram.

[0084] Figure 5 This is a schematic diagram of the generated image.

[0085] Figure 6 This is another set of pictures generated. DETAILED DESCRIPTION

[0086] A method for generating video samples based on multi-attribute synthesis comprises the following steps:

[0087] Step 1, constructing a multivariate attribute model, the method comprising: given an initial video data set, preprocessing each video in the data set, and decomposing each video into multivariate static attributes and dynamic attributes of a foreground object frame by frame according to a pre-trained video encoder and an attribute decomposition network; performing consistency and smoothness processing on the multivariate static attributes and dynamic attributes according to consistency constraints and smoothness constraints respectively; generating a multivariate attribute model by vector concatenation;

[0088] The following steps are involved:

[0089] Step 1-1, video preprocessing, the method includes:

[0090] The pre-trained target detector is used to detect the target frame by frame in the video, and then the target in the video is cut out according to the bounding box obtained by the detection. Finally, each cut target is scaled to a size of 224×224 to obtain the processed video frame sequence I i , i=1,…,T, i represents the i-th frame, and T represents the total number of frames in the video.

[0091] Step 1-2, video attribute decomposition; the method includes:

[0092] Use the pre-trained video encoder to convert the video sequence I obtained by step 1-1 into i Encoded into frame-by-frame feature vector f i ; The video encoder includes T residual network networks with shared parameters, whose input is a 224×224 three-channel image and whose output is an n-dimensional feature vector;

[0093] Use the pre-trained attribute decomposition network to decompose the video into frame-by-frame geometric f i g , Texture i a 、Posture i p 、Light i l and deformation f i m ; The attribute decomposition network contains 5 sub-networks, including: geometry estimation network, texture estimation network, posture estimation network, illumination estimation network and deformation estimation network; wherein, the geometry estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is an m-dimensional feature vector. g dimensional feature vector for geometric estimation; the texture estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is an m-dimensional feature vector. adimensional feature vector for texture estimation; the illumination estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is a 27-dimensional feature vector for illumination estimation; the posture estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is a 6-dimensional feature vector for posture estimation; the deformation estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is an m-dimensional feature vector. g dimensional feature vector, used for deformation estimation;

[0094] Step 1-3, attribute preprocessing; the method includes static attribute consistency processing and dynamic attribute smoothness processing;

[0095] The static attribute consistency processing method includes: considering T n-dimensional vectors {f i |i=1,…,T}, which express the same object, then these T n-dimensional vectors should be as consistent as possible, and the following objective function is used to solve the consistency vector f con :

[0096]

[0097] The frame-by-frame geometry f obtained in step 1-1 is i g and texture f i a Input them into the static attribute consistency processing method to obtain the geometry after consistency processing And texture

[0098] The method for processing the smoothness of dynamic attributes includes: considering T n-dimensional column vectors {f i |i=1,…,T}, which expresses the state of the same object at T consecutive moments, then these T n-dimensional vectors should change as smoothly as possible. First, concatenate the T column vectors in time order in the column direction into an n×T matrix; split the matrix by row to obtain n T-dimensional vectors f′ j , where j = 1, ..., n, and each vector is processed as follows:

[0099] f″ j =kernel*f′ j

[0100] Where * represents discrete convolution operation, kernel = [0.0545, 0.244, 0.403, 0.244, 0.0545] represents a one-dimensional convolution kernel, and for vector f′ j Use kernel to perform convolution operation with a step size of 1 to obtain the smoothed result f″ j; After smoothing all n T-dimensional vectors in the above method, reassemble them into an n×T matrix in row order, and then split the matrix by column to obtain T smoothed n-dimensional column vectors; the frame-by-frame posture f obtained in step 1-1 is i p 、Light i l and deformation f i m Input dynamic attribute smoothness processing method to obtain smoothed posture illumination And deformation

[0101] Step 1-4, multi-attribute vector concatenation; the method includes: for the geometric vector obtained by step 1-2 Texture attitude illumination And deformation Perform vector concatenation operations to build a multivariate attribute model:

[0102]

[0103] where <·,·,…,·> represents the concatenation operation of multiple vectors, S i Represents the result of concatenating the multi-attribute vectors of the i-th frame.

[0104] Step 1-5: for each video in a given initial video data set, repeat steps 1-1 to 1-3 until every video in the video data set is processed and the construction of the multivariate attribute model is completed.

[0105] Step 2, generating a multivariate attribute embedding space, the method includes: constructing an autoencoder network based on a neural network; mapping the multivariate attribute model constructed in step 1 to a low-dimensional embedding space through an encoder in the autoencoder network; restoring the multivariate attribute model to a decoder in the autoencoder network, and training the autoencoder network through constrained reconstruction loss; generating a multivariate attribute embedding space by recording the numerical value range of the low-dimensional embedding space; including:

[0106] Step 2-1, constructing an autoencoding network; the method includes:

[0107] Construct an autoencoder network based on a neural network, which includes an encoder and a decoder;

[0108] The encoder contains m cells, which share parameters. The τth cell accepts the cell state C of the τ-1th cell. τ-1 , hidden state h τ-1 And the current input x τAs input; output is the cell state C of the current cell τ And the hidden state h τ , the update rule of the autoencoder network is as follows:

[0109] f τ =σ(W f · <h τ-1 ,x τ >+b f )

[0110] p τ =σ(W p · <h τ-1 ,x τ >+b p )

[0111]

[0112]

[0113] o τ =σ(W o · <h τ-1 ,x τ >+b o )

[0114] h τ =o τ *tanh(C τ )

[0115] Where <·,·> represents the concatenation operation of two vectors, σ is the sigmoid function, tanh is the tanh function, (W f ,b f ),(W p ,b p ),(W C ,b C ),(W o ,b o ) represent the weights and biases of the four fully connected layers respectively; the input of the encoder is the multivariate attribute model, and the output is a t-dimensional feature code f attr ;

[0116] The structure of the decoder is consistent with the dynamic mode encoding network. The input of the decoder is a t-dimensional vector f attr , the output is a multivariate attribute model; in the update rule, the cell state C1 and hidden state h1 of the encoder at time 1 are set to all zero vectors, and the input x of the decoder at all times τ , are set to f attr , where τ=1,…,m.

[0117] Step 2-2, self-encoding training; the method includes:

[0118] The autoencoder network described in step 2-1 is trained using the stochastic gradient descent method. The loss function consists of three parts: a reconstruction loss l sigrec , a smooth loss l sm and a consistency loss l consis ; Among them, the reconstruction loss calculation method is as follows:

[0119]

[0120] Among them, S i represents the i-th vector of the multivariate attribute model, Represents the i-th vector of the reconstructed multivariate attribute model;

[0121] The smoothing loss calculation method is as follows:

[0122]

[0123] The calculation method of the multivariate attribute matrix S is as follows: all vectors in the reconstructed multivariate attribute model are In the column direction, they are stacked into a matrix. [:,:] represents a slice operation. [m g +m a :,:] means to select the mth a +m g Go to the last row of elements, It means to find the second-order derivative of the matrix in the x direction;

[0124] The consistency loss is calculated as follows:

[0125]

[0126] Finally, the total training loss of the entire network is as follows:

[0127] l=λ1l sigrec +λ2l smoo +λ3l consis

[0128] Among them, λ1,λ2,λ3 are balance factors.

[0129] Step 2-3, generate a multi-attribute embedding space. The method includes:

[0130] For all multi-attribute models, use the encoder in step 2-1 to generate feature vectors Calculate the encoding range of each dimension and store it in two vectors r l ,r h , where the vector r lEach dimension records the minimum value of the feature vector in that dimension, and the vector r h Each dimension records the maximum value of the feature vector in that dimension, and the multi-attribute embedding space consists of all vectors r that satisfy the following conditions: {r|r l,α ≤r α ≤r h,α ,α=1,…,t}, where r l,α Represents r l The value of the αth dimension, r α Represents the value of the αth dimension of vector r, r h,a Represents r h The value of the αth dimension.

[0131] Step 3, synthesizing multi-attributes, the method includes: sampling from the multi-attribute embedding space, using the decoder trained in step 2 to generate the multi-attributes of each frame; using shape synthesis and lighting synthesis methods to calculate the three-dimensional geometry and texture of the target object in each frame; rendering to generate video samples; repeating the multi-attribute synthesis process to obtain the specified multiple video samples, and finally completing the generation of video samples based on multi-attribute synthesis.

[0132] Step 3-1, sample a feature vector from the multi-attribute embedding space, and decode the feature vector into a multi-attribute model according to the decoder trained in step 2-2

[0133] Step 3-2, generate dynamic attributes and static attributes: extract the static attribute part from the decoded multivariate attribute model and the dynamic properties section It is further divided into geometric Texture illumination attitude And deformation Generate representative static properties, geometry Texture

[0134] Step 3-3, Synthesize geometric shapes: 3D mesh model of frame t The calculation method is as follows:

[0135]

[0136] Among them, G avg It is the average geometric shape defined in the three-dimensional deformable model 3DMM, with a dimension of 3n, where n represents the number of mesh vertices; each 3 dimensions represents the x, y, and z coordinate values ​​of a three-dimensional vertex; G ne It is the neutral shape basis defined in 3DMM, with the same dimension as G avg, each 3 dimensions represent the offset of a 3D vertex coordinate value; G mor It is the deformation basis of the shape defined in 3DMM, with the same dimensions and meaning as G ne The final G mesh is a vector with a dimension of 3n. By reshaping it, it is normalized into an n×3 matrix, where each row of the matrix represents the coordinate value of a vertex of the three-dimensional mesh of the model;

[0137] Step 3-4, synthesize texture lighting: First synthesize the texture and restore the calculation formula of the vertex color value as follows:

[0138]

[0139] Among them, T avg It is a vertex-by-vertex texture of the average geometric shape defined by 3DMM, with a dimension of 3n, where each 3 dimensions represent the R, G, and B values ​​of a 3D vertex; T ne It is the texture base defined in 3DMM, with the same dimensions as T avg , each 3 dimensions represents the offset of the R, G, B values ​​of a 3D vertex; the final T mesh is a vector of dimension 3n, which is normalized into an n×3 matrix by reshaping it, where each row of the matrix represents the R, G, B value of a vertex of the model's three-dimensional mesh;

[0140] Synthetic lighting: lighting at frame t The synthesis method is as follows:

[0141]

[0142] in, Represents the color value of the i-th vertex of the synthesized texture, represents the value of the jth dimension of the tth frame illumination, and Y(j) represents the jth basis of the spherical harmonic function; perform the above operation on each vertex to obtain the vertex-by-vertex texture color after synthetic illumination

[0143] Step 3-5, rendering: The synthesized 3D geometric shape of the tth frame Composite lit per-vertex textures And posture Send it to the renderer to draw the tth frame image I t And the mask image M of the tth frame t , randomly cropped from another background image and I t Images of the same size I b , synthesize the final video frame, the synthesis method is as follows:

[0144]

[0145] Where ⊙ represents element-by-element multiplication;

[0146] Step 3-6, write the generated T frame image into the video stream to obtain the final video sample;

[0147] Step 3-7, repeat steps 3-1 to 3-6 until the set number is reached, and the generation of video samples based on multi-attribute synthesis is completed.

[0148] Example

[0149] like Figure 1 As shown, the present invention discloses a method for generating video samples based on multi-attribute synthesis, which is specifically implemented according to the following steps:

[0150] Step 1, multi-attribute model construction: first, decompose the multi-attribute static attributes (geometry, texture) and dynamic attributes (posture, lighting, deformation) of the foreground object frame by frame from the video according to the pre-trained video encoder and attribute decomposition network, and then perform consistency and smoothness processing on the static attributes and dynamic attributes respectively according to the consistency constraints and smoothness constraints; finally, generate the multi-attribute model through vector splicing.

[0151] Step 2, multi-attribute embedding space generation: construct an autoencoder based on a neural network, map the multi-attribute model constructed in step 1 to a low-dimensional embedding space through the encoder of the autoencoder network, and then restore it to a multi-attribute model through the decoder, and train the autoencoder network through constrained reconstruction loss. Generate a multi-attribute embedding space by recording the numerical value range of the low-dimensional embedding space.

[0152] Step 3, multi-attribute synthesis: Sampling from the multi-attribute embedding space, first using the trained decoder to generate the multi-attributes of each frame, then using shape synthesis and lighting synthesis methods to calculate the 3D geometry and texture of the target object in each frame, and finally, rendering to generate video samples. Repeating the multi-attribute synthesis process can obtain multiple video samples specified by the user.

[0153] The following is a detailed description of the main processes of each step

[0154] Wherein, step 1 comprises the following steps:

[0155] Step 1.1, video attribute decomposition: Use the pre-trained video encoder to decompose the video {I i |i=1,…,T} is encoded into a frame-by-frame feature vector {f i |1,…,T}, and then use the attribute decomposition network to decompose the frame-by-frame feature vector to obtain the frame-by-frame geometry {f i g |i=1,…,T}, texture {f i a|i=1,…,T}, posture {f i p |i=1,…,T}, illumination {f i l |i=1,…,T} and deformation {f i m |i,…,T}.

[0156] Step 1.2, attribute preprocessing, consider T n-dimensional vectors {f i |i=1,…,T}, which expresses the same object, then these T n-dimensional vectors should be as consistent as possible, and the geometry and texture of each frame are processed for static attribute consistency to obtain the geometry after consistency processing And texture Consider T n-dimensional column vectors {f i |i=1,…,T}, which expresses the state of the same object at T consecutive moments, then these T n-dimensional vectors should change as smoothly as possible, and the smoothed posture is obtained by smoothing the posture, illumination and deformation of each frame illumination And deformation

[0157] Step 1.3, multi-attribute vector concatenation, for the geometry obtained in step 1.2 Texture attitude illumination And deformation Using vector concatenation operations to obtain a multivariate attribute model

[0158] Step 1.4: For each video in the original video dataset, repeat steps 1-1 to 1-3 until every video in the video dataset is processed.

[0159] Step 2 includes the following steps:

[0160] Step 2.1, construct an autoencoder network, where the encoder contains m cell units, these m cell units share parameters, and the i-th cell unit accepts the cell state C of the i-1-th cell unit i-1 , hidden state h i-1 And the current input x i As input, and output the cell state C of the current cell i And the hidden state h i The structure of the decoder is consistent with that of the dynamic mode encoding network, except that the input of the decoder is a t-dimensional vector f attr, the output is a multivariate attribute model. In the update rule, the difference is that the cell state C1 and hidden state h1 of the encoder at time 1 are set to all zero vectors, and the input x of the decoder at all times i ,i=1,…,m are all set to f attr .

[0161] Step 2.2, autoencoder network training, the autoencoder network constructed in step 2.1 is trained using back propagation and stochastic gradient descent, and minimizes reconstruction loss, smoothing loss, and consistency loss.

[0162] Step 2.3, generate a multi-attribute embedding space, and use the encoder in step 2-1 to generate feature vectors for all multi-attribute models Calculate the encoding range of each dimension and store it in two vectors r l ,r h , where r l Each dimension records the minimum value of the feature vector in that dimension, r h Each dimension records the maximum value of the feature vector in that dimension.

[0163] Step 3 contains the following steps

[0164] Step 3.1: Sample a feature vector from the multi-attribute embedding space and decode the feature vector into a multi-attribute model according to the decoder trained in step 2.2.

[0165] Step 3.2: Generate dynamic attributes and static attributes, and extract the static attribute part from the decoded multivariate attribute model and the dynamic properties section And further split it into geometric Texture illumination attitude And deformation Next, generate representative static attributes, geometry Texture

[0166] Step 3.3, synthesizing geometric shapes, synthesizing geometry and deformation into three-dimensional shapes according to the 3DMM model.

[0167] Step 3.4, synthesize texture lighting, synthesize the texture into vertex color values ​​according to the 3DMM model, and then obtain the vertex texture color after synthesized lighting according to the spherical harmonic lighting formula

[0168] Step 3.5, rendering, the synthesized 3D geometric shape of the tth frame Composite lit per-vertex textures And posture Send it to the renderer and draw the tth frame rendering image I t And the mask image M of the tth frame t , randomly cropped from another background image and I t Background image of the same size I b , and blends the rendered image and the background image according to the mask image.

[0169] Step 3.6, write the generated T frame image into the video stream to obtain the final video sample.

[0170] Step 3.7: Repeat steps 3.1 to 3.6 until the number of samples required by the user is met.

[0171] In this embodiment, a human face is used as an example for illustration. The specific implementation process is as follows:

[0172] The video dataset in step 1 mainly adopts the video dataset described in the literature Nagrani A, Chung JS, Zisserman A. Voxceleb: a large-scale speaker identification dataset [J]. arXiv preprint arXiv: 1706.08612, 2017. Then, step 1.1 is executed to perform attribute decomposition. The residual network described in step 1.1 is constructed according to the method described in the literature He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778. The video coding network and attribute decomposition network are pre-trained on the CelebA dataset in the manner described in Deng Y, Yang J, Xu S, et al. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops. 2019: 0-0. Then, the multi-attribute model is constructed according to steps 1.2 and 1.3.

[0173] In step 2, the multi-attribute model obtained in step 1 is trained with an autoencoder network to generate a multi-attribute embedding space. The stochastic gradient descent method in step 2.1 is performed according to the method described in the document Bottou L. Stochastic gradient descent tricks [M] / / Neural networks: Tricks of the trade. Springer, Berlin, Heidelberg, 2012: 421-436.

[0174] In order to illustrate some intermediate results in step 3, the present invention first takes the generation of a single frame as an example. The multi-attribute embedding space generated in step 2 is sampled and representative static attributes are generated according to steps 3.1 and 3.2. The present invention sets the deformation and illumination to all zero vectors and uses steps 3.3 and 3.4 to obtain the three-dimensional model of the face and the color value of each vertex. Then, the rendering step in step 3.5 is used to render the results of multiple samplings to obtain the following: Figure 2 The result shown in FIG. 1 is then obtained by the present invention according to steps 3.1 and 3.2 to generate illumination, and according to step 3.4 to re-synthesize the vertex color value, and then using the rendering of step 3.5 to obtain the following Figure 3 Results shown. Figure 4 is the corresponding mask image, where the white area represents the face area and the black area represents the background area.

[0175] Next, the present invention shows how to generate multiple frames of a video. First, the multi-attribute embedding space generated in step 2 is sampled and representative static attributes and t dynamic attributes are generated according to steps 3.1 and 3.2. Then, the static attributes and the dynamic attributes at each moment are used to obtain the rendered image of each frame according to steps 3.3, 3.4, and 3.5. Since the number of rendered video frames is large, the present invention takes a picture every 3 frames and stitches them together to obtain the following: Figure 5 After that, the present invention samples the multi-attribute embedding space generated in step 2 and generates representative static attributes and t dynamic attributes according to steps 3.1 and 3.2. We replace the second representative attributes with the representative attributes obtained after the first sampling calculation, and obtain the rendered image of each frame according to steps 3.3, 3.4, and 3.5. Take one picture every 3 frames and stitch them together to obtain the following: Figure 6 of the image. Figure 6 are several frames in a new video, and the sampling interval is 3 frames. Figure 6 and Figure 5 The identities of the people in the painting are the same, but the differences are in the lighting, deformation and posture.

[0176] The 3DMM model described in steps 3.3 and 3.4 adopts the model described in the document Paysan P, Knothe R, Amberg B, et al. A 3D face model for pose and illumination invariant face recognition [C] / / 2009 sixth IEEE international conference on advanced video and signal based surveillance. Ieee, 2009: 296-301.

[0177] The present invention provides a method and idea for generating a video sample based on multi-attribute synthesis. There are many methods and approaches to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A video sample generation method based on multi-attribute synthesis, characterized in that: The steps include: Step 1, constructing a multivariate attribute model, the method comprising: given an initial video data set, preprocessing each video in the data set, and decomposing each video into multivariate static attributes and dynamic attributes of a foreground object frame by frame according to a pre-trained video encoder and an attribute decomposition network; performing consistency and smoothness processing on the multivariate static attributes and dynamic attributes according to consistency constraints and smoothness constraints respectively; generating a multivariate attribute model by vector concatenation; Step 2, generating a multivariate attribute embedding space, the method comprising: constructing an autoencoder network based on a neural network; mapping the multivariate attribute model constructed in step 1 to a low-dimensional embedding space through an encoder in the autoencoder network; restoring the multivariate attribute model to a decoder in the autoencoder network, and training the autoencoder network through a constrained reconstruction loss; generating a multivariate attribute embedding space by recording the numerical value range of the low-dimensional embedding space; Step 3, synthesizing multivariate attributes, the method includes: sampling from the multivariate attribute embedding space, using the decoder trained in step 2 to generate the multivariate attributes of each frame; using shape synthesis and illumination synthesis methods to calculate the three-dimensional geometry and texture of the target object in each frame; rendering to generate video samples; repeating the multivariate attribute synthesis process to obtain a specified number of video samples, and finally completing the generation of video samples based on multivariate attribute synthesis; Wherein, step 2 comprises: Step 2-1, construct an autoencoding network; Step 2-2, self-encoding training; Step 2-3, generate multi-attribute embedding space; The method for constructing the autoencoding network described in step 2-1 includes: Construct an autoencoder network based on a neural network, which includes an encoder and a decoder; The encoder contains m cells, which share parameters. The τth cell accepts the cell state C of the τ-1th cell. τ-1 , hidden state h τ-1 And the current input x τ As input; output is the cell state C of the current cell τ And the hidden state h τ , the update rule of the autoencoder network is as follows: f τ =σ(W f ·<h τ-1 ,x τ >+b f ) p τ =σ(W p · <h τ-1 ,x τ >+b p ) the τ =σ(W o · <h τ-1 ,x τ >+b o ) h τ =o τ *tanh(C τ ) Where <·,·> represents the concatenation operation of two vectors, σ is the sigmoid function, tanh is the tanh function, (W f , b f ), (W p , b p ), (W C , b C ), (W o , b o ) represent the weights and biases of the four fully connected layers respectively; the input of the encoder is the multivariate attribute model, and the output is a t-dimensional feature code f attr ; The structure of the decoder is consistent with the dynamic mode encoding network. The input of the decoder is a t-dimensional vector f attr , the output is a multivariate attribute model; in the update rule, the cell state C1 and hidden state h1 of the encoder at time 1 are set to all zero vectors, and the input x of the decoder at all times τ , are set to f attr , where τ = 1,…,m.

2. The method for generating video samples based on multi-attribute synthesis according to claim 1, characterized in that: Step 1 includes the following steps: Step 1-1, video preprocessing, the method includes: The pre-trained target detector is used to detect the target frame by frame in the video, and then the target in the video is cut out according to the bounding box obtained by the detection. Finally, each cut target is scaled to a size of 224×224 to obtain the processed video frame sequence I i , i=1,...,T,i represents the i-th frame, T represents the total number of frames in the video; Step 1-2, video attribute decomposition; Steps 1-3, attribute preprocessing; Steps 1-4, multi-attribute vector concatenation; Step 1-5: for each video in a given initial video data set, repeat steps 1-1 to 1-3 until every video in the video data set is processed and the construction of the multivariate attribute model is completed.

3. The method for generating video samples based on multi-attribute synthesis according to claim 2, characterized in that: The video attribute decomposition method in step 1-2 includes: Use the pre-trained video encoder to convert the video sequence I obtained by step 1-1 into i Encoded into frame-by-frame feature vector f i ; The video encoder includes T residual network networks with shared parameters, whose input is a 224×224 three-channel image and whose output is an n-dimensional feature vector; Use the pre-trained attribute decomposition network to decompose the video into frame-by-frame geometric f i g , Texture i a 、Posture i p 、Light i l and deformation f i m The attribute decomposition network contains five sub-networks, including: a geometry estimation network, a texture estimation network, a posture estimation network, an illumination estimation network, and a deformation estimation network; wherein the geometry estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is an m-dimensional feature vector. g dimensional feature vector for geometric estimation; the texture estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is an m-dimensional feature vector. a dimensional feature vector for texture estimation; the illumination estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is a 27-dimensional feature vector for illumination estimation; the posture estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is a 6-dimensional feature vector for posture estimation; the deformation estimation network is a single-layer fully connected layer, whose input is an n-dimensional feature vector and whose output is an m-dimensional feature vector. g dimensional feature vector for deformation estimation.

4. The method for generating video samples based on multi-attribute synthesis according to claim 3, characterized in that: The attribute preprocessing method in steps 1-3 includes static attribute consistency processing and dynamic attribute smoothness processing; The static attribute consistency processing method includes: using the following objective function to solve the consistency vector f con : The frame-by-frame geometry f obtained in step 1-1 is i g and texture f i a Input them into the static attribute consistency processing method to obtain the geometry after consistency processing And texture The method for processing the dynamic attribute smoothness includes: splicing T column vectors in the column direction in time order into an n×T matrix; splitting the matrix by row to obtain n T-dimensional vectors f j ′, where j = 1, ..., n, and each vector is processed as follows: f j ″=kernel*f j ′ Where * represents discrete convolution operation, kernel = [0.0545, 0.244, 0.403, 0.244, 0.0545] represents a one-dimensional convolution kernel, and the vector f j ′Use kernel to perform convolution operation with a step size of 1 to obtain the smoothed result f j ″; smooth all n T-dimensional vectors in the above way and then splice them back into an n×T matrix in row order, and then split the matrix by column to obtain T smoothed n-dimensional column vectors; the frame-by-frame posture f obtained in step 1-1 is i p 、Light i l and deformation f i m Input dynamic attribute smoothness processing method to obtain smoothed posture illumination and deformation 5. The method for generating video samples based on multi-attribute synthesis according to claim 4, characterized in that: The multivariate attribute vector concatenation method in step 1-4 includes: for the geometric vector obtained in step 1-2 Texture attitude illumination and deformation Perform vector concatenation operations to build a multivariate attribute model: where <·,·,...,·> represents the concatenation operation of multiple vectors, S i Represents the result of concatenating the multi-attribute vectors of the i-th frame.

6. The method for generating video samples based on multi-attribute synthesis according to claim 5, characterized in that: The autoencoder training method described in step 2-2 includes: The autoencoder network described in step 2-1 is trained using the stochastic gradient descent method. The loss function consists of three parts: a reconstruction loss l sigrec , a smooth loss l smooth and a consistency loss l consis ; Among them, the reconstruction loss calculation method is as follows: Among them, S i represents the i-th vector of the multivariate attribute model, Represents the i-th vector of the reconstructed multivariate attribute model; The smoothing loss calculation method is as follows: The calculation method of the multivariate attribute matrix S is as follows: all vectors in the reconstructed multivariate attribute model are In the column direction, they are stacked into a matrix. [:,:] represents a slice operation. [m g +m a :,:] indicates selecting the mth a +m g Go to the last row of elements, It means to find the second-order derivative of the matrix in the x direction; The consistency loss is calculated as follows: Finally, the total training loss of the entire network is as follows: l=λ1l sigrec +λ2l smooth +λ3l consis Among them, λ1, λ2, λ3 are balancing factors.

7. The method for generating video samples based on multi-attribute synthesis according to claim 6, characterized in that: The method for generating a multi-attribute embedding space described in step 2-3 includes: For all multi-attribute models, use the encoder in step 2-1 to generate feature vectors Calculate the encoding range of each dimension and store it in two vectors r l , r h , where the vector r l Each dimension records the minimum value of the feature vector in that dimension, and the vector r h Each dimension records the maximum value of the feature vector in that dimension, and the multi-attribute embedding space consists of all vectors r that satisfy the following conditions: {r|r l，α ≤r α ≤r h，α , α=1,…,t}, where r l，α Represents r l The value of the αth dimension, r α Represents the value of the αth dimension of vector r, r h，α Represents r h The value of the αth dimension.

8. The method for generating video samples based on multi-attribute synthesis according to claim 7, characterized in that: Step 3 includes: Step 3-1, sample a feature vector from the multi-attribute embedding space, and decode the feature vector into a multi-attribute model according to the decoder trained in step 2-2 Step 3-2, generate dynamic attributes and static attributes: extract the static attribute part from the decoded multivariate attribute model and the dynamic properties section It is further divided into geometric Texture illumination attitude and deformation Generate representative static properties, geometry Texture Step 3-3, Synthesize geometric shapes: 3D mesh model of frame t The calculation method is as follows: Among them, G avg It is the average geometric shape defined in the three-dimensional deformable model 3DMM, with a dimension of 3n, where n represents the number of mesh vertices; each 3 dimensions represents the x, y, and z coordinate values ​​of a three-dimensional vertex; G ne It is the neutral shape basis defined in 3DMM, with the same dimension as G avg , each 3 dimensions represent the offset of a 3D vertex coordinate value; G mor It is the deformation basis of the shape defined in 3DMM, with the same dimensions and meaning as G ne The final G mesh is a vector with a dimension of 3n. By reshaping it, it is normalized into an n×3 matrix, where each row of the matrix represents the coordinate value of a vertex of the three-dimensional mesh of the model; Step 3-4, synthesize texture lighting: First synthesize the texture and restore the calculation formula of the vertex color value as follows: Among them, T avg It is a vertex-by-vertex texture of the average geometric shape defined by 3DMM, with a dimension of 3n, where each 3 dimensions represent the R, G, and B values ​​of a 3D vertex; T ne It is the texture base defined in 3DMM, with the same dimensions as T avg , each 3 dimensions represents the offset of the R, G, B values ​​of a 3D vertex; the final T mesh is a vector of dimension 3n, which is normalized into an n×3 matrix by reshaping it, where each row of the matrix represents the R, G, and B values ​​of a vertex of the three-dimensional mesh of the model; Synthetic lighting: lighting at frame t The synthesis method is as follows: in, Represents the color value of the i-th vertex of the synthesized texture, represents the value of the jth dimension of the tth frame illumination, and Y(j) represents the jth basis of the spherical harmonic function; perform the above operation on each vertex to obtain the vertex-by-vertex texture color after synthetic illumination Step 3-5, rendering: The synthesized 3D geometric shape of the tth frame Composite lit per-vertex textures And posture Send it to the renderer to draw the tth frame image I t And the mask image M of the tth frame t , randomly cropped from another background image and I t Images of the same size I b , synthesize the final video frame, the synthesis method is as follows: Where ⊙ represents element-by-element multiplication; Step 3-6, write the generated T frame image into the video stream to obtain the final video sample; Step 3-7, repeat steps 3-1 to 3-6 until the set number is reached, and the generation of video samples based on multi-attribute synthesis is completed.

Images