A video translation method and apparatus

Abstract

The application relates to a video translation method and device, which comprises the following steps: inputting image features and coordinate features of original data into a fusion input generator; calculating a three-frame loss according to continuous frames of a video of the input generator, and optimizing the generator according to the three-frame loss; and performing video translation based on the optimized generator, so as to improve the visual effect of the converted video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video conversion, in particular to a video translation method and device. BACKGROUND

[0002] At present, video translation technology, especially face translation technology, is often applied as a function in camera software or social media software. Efficient and beautiful unsupervised face video translation technology is of great significance to enrich the playability of social software and promote the commercialization process of technology.

[0003] The related technology is based on a cycle generative adversarial network, which sets up two sets of generator-discriminator architecture and introduces cycle consistency loss to ensure the connection between the two conversion domains, without the need to use a paired picture dataset. However, the cycle consistency loss has a negative effect in many application scenarios. Since it always requires the picture before conversion and the picture after conversion-inversion to be completely consistent, the generator may "slack off" during the conversion process, so it will leave a residual image of the picture before conversion, thus reducing the conversion effect.

[0004] In addition, the related technology is for converting a single image. Since it does not consider the continuity between video frames, relying only on single-frame image conversion methods in actual video conversion tasks results in low continuity and stability of the generated result video. SUMMARY

[0005] The embodiments of the present application provide a video translation method and device to improve the visual effect after video conversion.

[0006] In one aspect, the embodiments of the present application provide a video translation method, characterized in that it comprises the steps of:

[0007] inputting the image features and coordinate features of the input original data into the fusion generator;

[0008] calculating a three-frame loss according to the video continuous frames of the input generator, and optimizing the generator according to the three-frame loss;

[0009] performing video translation based on the optimized generator.

[0010] In some embodiments, the calculation of the three-frame loss according to the video continuous frames of the input generator comprises the steps of:

[0011] calculating the three-frame loss by using a first algorithm;

[0012] The first algorithm is to make the distance between the first frame and the second frame less than the distance between the first frame and the third frame for any input generator video continuous frame, and the distance is the structural similarity index.

[0013] In some embodiments, the first algorithm includes:

[0014] The three-frame loss is defined as max(d(x′1, x′2) - d(x′1, x′3) + margin, 0), where x′1 represents the first frame, x′2 represents the second frame, and x′3 represents the third frame in the three-frame loss [x′1, x′2, x′3] obtained for the input video consecutive frames [x1, x2, x3] of the generator, and d(·) represents the SSIM distance.

[0015] In some embodiments, the steps further include:

[0016] The image features and coordinate features are processed by a predictor before being input into the generator of the fused video frames.

[0017] The prediction loss is calculated for the consecutive frames of the fused video, and the predictor is constrained based on the calculation results;

[0018] The method for calculating the prediction loss is as follows:

[0019] The predicted loss = d(x3, P(x1, x2)), where,

[0020] [x1, x2, x3] are the consecutive video frames obtained after fusion, P(·) represents the predictor, and d(·) represents the SSIM distance.

[0021] In some embodiments, before the image features and coordinate features of the input raw data are fused, the following steps are also included:

[0022] Calculate the identity loss on the input raw data, and constrain the generator according to the identity loss;

[0023] The method for calculating the identity loss includes:

[0024] Where Lidt is the identity loss, x s The original input data for the s-domain, x t The input raw data for the t field, For x s Input generator G s The image obtained from it, For x t Input generator G t The image obtained from G, and G s The generation direction is from the t domain to the s domain.

[0025] On the other hand, embodiments of the present invention also provide a video translation device, characterized in that it includes:

[0026] a feature fusion module configured to fuse image features and coordinate features of input raw data to generate a fused input;

[0027] a loss constraint module configured to calculate a triple-frame loss based on video consecutive frames of the input generator, and optimize the generator based on the triple-frame loss;

[0028] a video generation module configured to generate a video based on the optimized generator.

[0029] In some embodiments, the loss constraint module is further configured to:

[0030] calculate the triple-frame loss by using a first algorithm;

[0031] the first algorithm is to make the distance between the first frame and the second frame less than the distance between the first frame and the third frame for any video consecutive frames of the input generator, and the distance is a structural similarity index.

[0032] In some embodiments, the first algorithm comprises:

[0033] the triple-frame loss = max(d(x'1, x'2) - d(x'1, x'3) + margin, 0), wherein, for video consecutive frames [x1, x2, x3] of the input generator, the obtained triple-frame [x'1, x'2, x'3] is obtained, x'1 represents the first frame, x'2 represents the second frame, x'3 represents the third frame, and d(·) represents the SSIM distance.

[0034] In some embodiments, further comprising a prediction loss module configured to:

[0035] input the video consecutive frames obtained after the fusion into the generator through a predictor first;

[0036] calculate a prediction loss based on the video consecutive frames obtained after the fusion, and constrain the predictor based on the calculation result;

[0037] the calculation method of the prediction loss is:

[0038] the prediction loss = d(x3, P(x1, x2)), wherein,

[0039] [x1, x2, x3] is the video consecutive frames obtained after the fusion, P(·) represents the predictor, and d(·) represents the SSIM distance.

[0040] In some embodiments, further comprising an identity loss constraint module configured to

[0041] calculate an identity loss based on the input raw data, and constrain the generator based on the identity loss;

[0042] The calculation method of the identity loss comprises:

[0043] Wherein, Lidt is the identity loss, x s is the input original data in the s domain, x t is the input original data in the t domain, is the x s image obtained in the input generator G s , is the x t image obtained in the input generator G t , and the generation direction of G s is from the t domain to the s domain.

[0044] The technical scheme provided by the present application has the beneficial effects that:

[0045] The video translation method and device provided by the embodiments of the present application can effectively utilize the connection between the frames close in time in the video to train the generator by calculating the three-frame loss, so that the continuity and stability of the generated result video are improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0047] Figure 1 The flowchart of the video translation method provided by the embodiments of the present application is shown in the figure;

[0048] Figure 2 The frame diagram of the video translation method provided by the embodiments of the present application is shown in the figure;

[0049] Figure 3 The structure diagram of the video translation device provided by the embodiments of the present application is shown in the figure;

[0050] Figure 4 The structure diagram of the video translation device provided by the embodiments of the present application is shown in the figure;

[0051] Figure 5 The structure diagram of the video translation device provided by the embodiments of the present application is shown in the figure;

[0052] Figure 6 The prediction loss calculation effect diagram provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0053] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0054] As shown in Figure 1 , the embodiments of the present application provide a video translation method, which comprises the steps of:

[0055] S100: inputting image features and coordinate features of input original data into a fusion generator;

[0056] S200: calculating a three-frame loss according to continuous frames of a video of the input generator, and optimizing the generator according to the three-frame loss;

[0057] S300: performing video translation based on the optimized generator.

[0058] It should be noted that the embodiments of the present application are a video translation method based on an adversarial generative network, and the video can include a dynamic video or image of a biological face. Before S100, a data set also needs to be constructed, and the data set can include continuous frames of video images of a biological face domain, corresponding face images of an animation domain, and face coordinates obtained after coordinate extraction of the biological face domain images. The design and composition of the data set can be in an unsupervised manner. Preferably, each image in the data set is a three-frame, i.e., [x1, x2, x3], [x4, x5, x6]......, which is continuous in time. After face coordinate extraction, [y1, y2, y3] is obtained.

[0059] When extracting the face coordinates, a face coordinate detection technology can be used to obtain the face coordinates of each frame of the video image. The formula for extracting the coordinates from the face image is:

[0060] F L = Face_alignment(I), wherein Face_alignment(·) represents a coordinate extraction sampling operator, I represents an input frame image, and F L represents the extracted coordinate information.

[0061] In the feature fusion process, the face continuous frame image and the corresponding face coordinates can be input into a convolutional neural network, transposed and multiplied respectively, normalized by a Softmax function to obtain a cross-domain attention matrix, and then the cross-domain attention matrix and the original features are multiplied respectively to obtain the fused face appearance feature and face geometry feature.

[0062] It can be understood that the face appearance feature and the face geometry feature are different and complementary. Figure 2 As shown, y i and x i After being extracted by the convolutional layer, the coordinate features F L and image features F c are obtained, and are defined as query (Q) and key (K), the cross-attention matrix is calculated, and is multiplied with the two features respectively to obtain the fused face appearance feature F a and face geometry coordinate feature F p . The joint learning of the face feature and the coordinate feature is promoted by the attention mechanism, which can pay attention to the relationship between the deep features of the two, so that the face geometry feature can be better preserved during image translation, and the result is more identifiable.

[0063] In the embodiment of the present application, the time factor and the face position information in the video conversion task are considered, the continuity of the conversion result is ensured, and the geometry feature of the conversion result is also preserved. By calculating the triple frame loss, the relationship between the frames adjacent in time in the video can be effectively utilized to train the generator, so that the continuity and stability of the generated result video are improved.

[0064] In some embodiments, the triple frame loss is calculated according to the input video continuous frames of the generator in S200, including the steps of: using a first algorithm to calculate the triple frame loss, wherein the first algorithm is to make the distance between the first frame and the second frame less than the distance between the first frame and the third frame for any input video continuous frames of the generator, and the distance is the structural similarity index.

[0065] Preferably, the first algorithm is:

[0066] The triple frame loss = max (d (x'1, x'2) - d (x'1, x'3) + margin, 0), wherein for the input video continuous frames [x1, x2, x3] of the generator, the obtained triple frame [x'1, x'2, x'3], x'1 represents the first frame, x'2 represents the second frame, x'3 represents the third frame, and d(·) represents the SSIM distance.

[0067] It can be understood that the introduction of the triple frame loss calculation to optimize the generator considers x1 as an anchor point, x2 as a positive sample, and x3 as a negative sample, and restricts the distance between x1 and x2 and x1 and x3. Therefore, the morphological consistency between consecutive frames of the video can be guaranteed, thereby obtaining more beautiful and stable face video translation results.

[0068] In some embodiments, the image features and coordinate features obtained after fusion are input into the generator after being processed by a predictor; a prediction loss is calculated for the video consecutive frames obtained after fusion, and the predictor is constrained based on the calculation result;

[0069] The calculation method of the prediction loss is:

[0070] The prediction loss = d(x3, P(x1, x2)), wherein,

[0071] [x1, x2, x3] is the video consecutive frames obtained after fusion, P(·) represents the predictor, and d(·) represents the SSIM distance.

[0072] It should be noted that the predictor can be a prediction network U-net, and for an input image [x1, x2, x3], it is input into the generator in turn to obtain [x'1, x'2, x'3]. Then [x1, x2] and [x'1, x'2] are input into the prediction network U-net to obtain and As shown in Figure 6 , for two consecutive frames t0 and t1, after passing through the predictor PA, two real and false images of t2 are obtained, and the real degree of the image can be effectively improved through the constraint of the prediction loss.

[0073] It can be understood that the predictor has two directions: t domain to s domain and s domain to t domain, t domain represents target domain, and s domain represents source domain. It corresponds to two generators Gs and Gt. The two generators Gs and Gt are respectively input with real face data (such as real face) and computer face data (such as animation face). The s domain image is converted into a t domain image through the generator Gt, and then the s domain image is restored through Gs, and the reverse direction is the same.

[0074] In some embodiments, before the image features and coordinate features of the input original data are fused, the method further includes the steps of: calculating an identity loss for the input original data, and constraining the generator according to the identity loss;

[0075] The calculation method of the identity loss includes:

[0076] Wherein, Lidt is the identity loss, x sThe original input data for the s-domain, x t The input raw data for the t field, For x s Input generator G s The image obtained from it, For x t Input generator G t The image obtained from G, and G s The generation direction is from the t domain to the s domain.

[0077] It is understandable that there are two generators, Gs and Gt, where generator Gs generates from the t domain to the s domain, and generator Gt generates from the s domain to the t domain. The generators in the two directions are computed in parallel. The process described in this embodiment includes the input and output of one generator, and the process in the other direction (the reverse generator) is similar and will not be described again.

[0078] In this embodiment, if G s Given an image in the s-domain as input, the generator's conversion capability is improved by ensuring the generated image closely resembles the original. Applying identity loss to constrain the generator effectively avoids losing the original image's identity information, thus optimizing video translation results.

[0079] In some embodiments, after the image features and coordinate features of the input raw data enter the fused input generator, the following steps are also included:

[0080] The adversarial consistency loss is calculated based on consecutive video frames from the input generator, and the generator is optimized based on the adversarial consistency loss.

[0081] The method for calculating the resistance to consistency loss is as follows:

[0082] in,

[0083] x represents the original input data. This represents the extended field of the input raw data. This represents the extended domain of the image data obtained after transformation and inverse transformation by the generator. Let E represent the discriminator used to distinguish between the two domains, where E is the mathematical expectation.

[0084] In this embodiment, to avoid the generator excessively catering to the inversely converted image and the original image during the video conversion process... Figure 1 Consistency requirements lead to ghosting; by introducing constraints to counteract consistency loss, the generator is given more flexibility.

[0085] like Figure 3 As shown, this embodiment of the invention also provides a video translation device, characterized in that it includes:

[0086] a feature fusion module configured to fuse image features and coordinate features of input raw data into a fused input generator;

[0087] a loss constraint module configured to calculate a triple-frame loss based on video consecutive frames of the input generator, and optimize the generator based on the triple-frame loss;

[0088] a video generation module configured to generate a video based on the optimized generator.

[0089] In some embodiments, the loss constraint module is further configured to:

[0090] calculate the triple-frame loss by using a first algorithm;

[0091] the first algorithm is to make a distance between a first frame and a second frame smaller than a distance between the first frame and a third frame for video consecutive frames of the input generator, and the distance is a structural similarity index.

[0092] In some embodiments, the first algorithm comprises:

[0093] the triple-frame loss = max(d(x'1, x'2) - d(x'1, x'3) + margin, 0), wherein, for video consecutive frames [x1, x2, x3] of the input generator, a triple frame [x'1, x'2, x'3] obtained therefrom, x'1 represents the first frame, x'2 represents the second frame, x'3 represents the third frame, and d(·) represents the SSIM distance.

[0094] As shown in Figure 4 , further comprising a prediction loss module configured to:

[0095] input video consecutive frames obtained after the fusion of the image features and the coordinate features into the generator through a predictor first;

[0096] calculate a prediction loss based on the video consecutive frames obtained after the fusion, and constrain the predictor based on a calculation result;

[0097] the calculation method of the prediction loss is:

[0098] the prediction loss = d(x3, P(x1, x2)), wherein,

[0099] [x1, x2, x3] are the video consecutive frames obtained after the fusion, P(·) represents the predictor, and d(·) represents the SSIM distance.

[0100] As shown in Figure 5 , in some embodiments, further comprising an identity loss constraint module configured to

[0101] calculating an identity loss on the input raw data, and constraining the generator according to the identity loss;

[0102] The method for calculating the identity loss comprises:

[0103] wherein, Lidt is the identity loss, x s is the input raw data in the s domain, x t is the input raw data in the t domain, is x s the image obtained in the input generator G s is the image obtained in the input generator G t t and the generation direction of G s is from the t domain to the s domain.

[0104] In some embodiments, the loss constraint module is further configured to:

[0105] calculating an adversarial consistency loss based on consecutive frames of the video input to the generator, and optimizing the generator according to the adversarial consistency loss;

[0106] The method for calculating the adversarial consistency loss comprises:

[0107] wherein,

[0108] x represents the input raw data, represents the extended domain of the input raw data, represents the extended domain of the image data obtained after conversion and inverse conversion by the generator, represents a discriminator for distinguishing between two domains, and E is the mathematical expectation.

[0109] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof. In the hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable storage medium, which can include computer-readable storage media (or non-transitory media) and communication media (or transitory media). ​​

[0110] It should be noted that, in the present application, the relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another, without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... " does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0111] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of video translation, characterized by, It comprises the steps of: inputting image features and coordinate features of input raw data into a fusion generator; calculating a triple-frame loss according to video continuous frames of the input generator, and optimizing the generator according to the triple-frame loss; video translation based on the optimized generator; the step of calculating the triple-frame loss according to the video continuous frames of the input generator comprises the steps of: calculating the triple-frame loss by using a first algorithm; the first algorithm is that for any input generator video continuous frame, the distance between the first frame and the second frame is less than the distance between the first frame and the third frame, and the distance is a structural similarity index; the first algorithm comprises: The triple consecutive frame loss = wherein the video consecutive frames for the input generator ], the resulting triple consecutive frame , denotes the first frame, denotes the second frame, denotes the third frame, denotes the SSIM distance; the video translation method further comprises the steps of: the image features and coordinate features obtained after fusion are input into the generator through a predictor first; calculating a prediction loss for the video continuous frames obtained after fusion, and constraining the predictor based on the calculation result; the calculation method of the prediction loss is: The prediction loss = -log p(y | x) = -log softmax(f(x))y denotes a predictor, denotes SSIM distance.

2. The video translation method of claim 1, wherein, before the image features and coordinate features of the input raw data are input into the fusion, the method further comprises the steps of: calculating an identity loss for the input raw data, and constraining the generator according to the identity loss; the calculation method of the identity loss comprises: Lidt = wherein, Lidt is an identity loss, is an input generator yields an image, is an input generator yields an image, and the generation direction of is from the t-domain to the s-domain.

3. A video translation device, characterized by, it comprises: a feature fusion module for inputting image features and coordinate features of input raw data into a fusion generator; a loss constraint module for calculating a triple-frame loss according to video continuous frames of the input generator, and optimizing the generator according to the triple-frame loss; a video generation module for video translation based on the optimized generator; the loss constraint module is further used for: calculating the triple-frame loss by using a first algorithm; the first algorithm is that for any input generator video continuous frame, the distance between the first frame and the second frame is less than the distance between the first frame and the third frame, and the distance is a structural similarity index; the first algorithm comprises: The triple consecutive frame loss = wherein the video consecutive frames for the input generator ] the resulting triple consecutive frame wherein, denotes the first frame, denotes the second frame, denotes the third frame, denotes the SSIM distance; the video translation device further comprises a prediction loss module for: the image features and coordinate features obtained after fusion are input into the generator through a predictor first; calculating a prediction loss for the video continuous frames obtained after fusion, and constraining the predictor based on the calculation result; the calculation method of the prediction loss is: The prediction loss = -log p(y | x) = -log softmax(f(x))y denotes a predictor, denotes SSIM distance.

4. A video translation device as claimed in claim 3, characterized in that further comprising an identity loss constraint module for calculating an identity loss for the input raw data, and constraining the generator according to the identity loss; the calculation method of the identity loss comprises: Lidt = wherein, Lidt is an identity loss, is an input generator yields an image, is an input generator yields an image, and the generation direction is from the t-domain to the s-domain.

Images