A character processing method and system applied to video conversion

Abstract

The application provides a character processing method and system applied to video conversion. The method comprises the following steps: obtaining a to-be-converted video, performing language conversion on to-be-converted audio of the to-be-converted video according to a second language, inputting target audio and a first character image into a lip movement conversion model, obtaining a second character image output by the lip movement conversion model, and generating a target video according to the target audio and the second character image. The application adjusts the lip movement in the first character image through the translated audio data, so that the lip movement of a target character in the target video corresponds to the pronunciation of the translated target audio, thereby improving the user experience.

Description

[0001] This application claims priority to Chinese patent application No. 202311479579.X, filed on November 8, 2023, with the State Intellectual Property Office of China. The entire contents of the latter are incorporated herein by reference. Technical Field

[0002] This application relates to the field of speech conversion technology, and in particular to a method and system for processing people in video conversion. Background Technology

[0003] When watching films from different countries, users cannot directly understand the meaning of the dialogue from the audio alone because the film's language differs from their own. Therefore, users need to continuously read subtitles while watching the film to comprehend the meaning of the characters' lines.

[0004] To address this, speech-to-text technology can be used to convert the film's language to the user's desired language while preserving the characters' vocal characteristics, thus achieving real-time translation. However, during this process, because the language of the dialogue changes, the original lip movements of the characters in the film cannot correspond to the converted dialogue, reducing the user's viewing experience. Summary of the Invention

[0005] To address the issue of mismatch between lip movements and language pronunciation in video footage after language conversion, some embodiments of this application provide a method and system for processing characters during video conversion.

[0006] Firstly, this application provides a method for processing people in video conversion, including:

[0007] The video to be converted is obtained, which includes audio to be converted and a first person image. The language information of the audio to be converted is a first language, and the first person image includes at least the first lip-syncing action of the target person speaking in the first language.

[0008] The audio to be converted is converted into a different language than the first language to obtain the target audio.

[0009] The target audio and the first person image are input into a lip-sync model to obtain a second person image output by the lip-sync model. The second person image includes at least a second lip-sync action of the target person speaking in the second language. The lip-sync model includes a transformation module, which is configured to calculate the transformation coefficients corresponding to each feature channel in the image features of the first person image based on the target audio, and perform feature transformation on the image feature channels of the first person image based on the transformation coefficients to obtain the image features of the first person image.

[0010] A target video is generated based on the target audio and the image of the second person.

[0011] In some embodiments, before the step of inputting the target audio and the first person image into the lip-sync model, the method further includes:

[0012] Acquire training data; the training data includes a first training image, the first training image including at least one training figure with lip movements;

[0013] The training data is input into the model to be trained so as to extract the training audio features and the first training image of the training data through the model to be trained. The model to be trained is a lip-sync model that has not been trained to convergence.

[0014] Based on the training audio features, the first training image is transformed using the model to be trained to generate the second training image;

[0015] The training loss between the second training image and the lip shape image label is calculated based on the loss function;

[0016] If the training loss is greater than the loss threshold, then the model to be trained is trained iteratively.

[0017] If the training loss is less than or equal to the loss threshold, then the lip-sync model is input according to the model parameters of the model to be trained.

[0018] In some embodiments, the model to be trained includes an encoder, a transform module, and a decoder. The step of performing feature channel transformation on a first training image using the model to be trained, based on the training audio features, includes:

[0019] The encoder encodes the training audio features to obtain the training audio code, and the encoder encodes the first training image to obtain the first training image code;

[0020] The transformation module determines the feature channel calculation relationship based at least on the trained audio encoding and the first trained image encoding.

[0021] Based on the feature channel calculation relationship, feature channel conversion is performed on the first training image encoding to obtain the second training image encoding;

[0022] The second training image is obtained by decoding the encoding of the second training image through the decoder.

[0023] In some embodiments, the step of determining the feature channel calculation relationship by the transformation module, at least based on the trained audio code and the first trained image code, includes:

[0024] A reference image is obtained based on the encoding of the first training image, wherein the reference image is a person image that is the same as the training person in the first training image;

[0025] The reference image is stitched together with the first training image to obtain a training stitched image;

[0026] The transformation coefficients are calculated based on the training audio encoding and the stitched training image to determine the feature channel calculation relationship, wherein the transformation coefficients are used to indicate the transformation relationship of the first training image relative to the reference image.

[0027] In some embodiments, the step of calculating the transform coefficients based on the stitched training image includes:

[0028] The encoder extracts the alignment features of the stitched training images;

[0029] Transformation coefficients are calculated based on the alignment features and the training audio features.

[0030] In some embodiments, the transform coefficients include a first transform coefficient, a second transform coefficient, and a third transform coefficient, and the step of performing feature channel transformation on the encoding of the first training image according to the feature channel calculation relationship includes:

[0031] The rotation angle of the feature channel encoded in the first training image is calculated based on the first transformation coefficient;

[0032] The translation distance of the feature channels encoded in the first training image is calculated based on the second transformation coefficients;

[0033] The scaling factor of the feature channels encoded in the first training image is calculated based on the third transformation coefficient;

[0034] The first training image code is converted into a second training image code based on the rotation angle, the translation distance, and the scaling factor.

[0035] In some embodiments, the encoder includes a hierarchical structure formed by a preset number of coding units, wherein the input of a coding unit is the output of the previous coding unit, and the output of a coding unit is the input of the next coding unit. The step of decoding the second training image encoded by the decoder includes:

[0036] The second training image is encoded and divided into a preset number of first image segments;

[0037] The first image segment is input into the encoder so that attention calculation is performed independently on each of the first image segments through a certain encoding unit to obtain the first attention encoding result;

[0038] The first image segment is adjusted in a preset manner to divide the second training image encoding into a preset number of second image segments; wherein, there is partial overlap between the first image segment and the corresponding second image segment;

[0039] The first attention encoding result is input into the next level encoding unit to perform attention calculation on the second image segment through the encoding unit, so as to obtain the second attention encoding result;

[0040] Iterate the above operations to obtain the second feature encoding based on the second attention encoding result;

[0041] Decoding the second feature encoding yields the second training image.

[0042] In some embodiments, the decoder includes a preset number of upsampling layers; the step of decoding the second feature encoding to obtain a second training image includes:

[0043] The second feature encoding is input into the decoder, and the second feature encoding is decoded according to the upsampling layer to obtain the second training image;

[0044] In this module, a preset number of encoding units are distributed with the upsampling layer according to a corresponding structure.

[0045] In the decoder, the second feature encoding of the upsampling layer performing the decoding operation is output by a non-corresponding encoding unit.

[0046] In some embodiments, the feature dimensions of the upsampling layer and the coding unit increase with the increase of the layer, and the number of feature channels of the upsampling layer and the coding unit decreases with the increase of the layer;

[0047] The encoding unit and the upsampling layer are connected at the corresponding level.

[0048] Secondly, some embodiments of this application provide a person processing system applied to video conversion, a controller, and a lip-syncing model. The lip-syncing model includes a transformation module, which is configured to calculate transformation coefficients corresponding to each feature channel in the image features of a first person image based on at least the target audio, and perform feature transformation on the image feature channels of the first person image based on the transformation coefficients to obtain the image features of the first person image; the controller is configured to:

[0049] The video to be converted is obtained, which includes audio to be converted and a first person image. The language information of the audio to be converted is a first language, and the first person image includes the first lip-syncing action of the target person speaking in the first language.

[0050] The audio to be converted is converted into a different language than the first language to obtain the target audio.

[0051] The target audio and the first person image are input into the lip-sync model to obtain a second person image output by the lip-sync model. The second person image includes the second lip-sync action of the target person speaking in the second language.

[0052] A target video is generated based on the target audio and the image of the second person.

[0053] As can be seen from the above technical solutions, this application provides a method and system for character processing in video conversion. The method involves acquiring a video to be converted, performing language conversion on the audio of the video to be converted in a second language, inputting the target audio and a first character image into a lip-syncing model, obtaining a second character image output by the lip-syncing model, and then generating a target video based on the target audio and the second character image. This application adjusts the lip movements in the first character image using translated audio data, ensuring that the lip movements of the target character in the target video correspond to the pronunciation of the translated target audio, thereby improving the user experience. Attached Figure Description

[0054] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A flowchart illustrating a method for processing people in video conversion, provided as an embodiment of this application;

[0056] Figure 2 This is a flowchart illustrating the training process of the lip-sync model according to an embodiment of this application.

[0057] Figure 3 This is a flowchart illustrating the training process of the model to be trained in this embodiment of the application.

[0058] Figure 4 This is a flowchart illustrating the calculation relationship of the feature channels in the embodiments of this application;

[0059] Figure 5 This is a flowchart illustrating the calculation of transform coefficients based on training spliced ​​images and training audio features in an embodiment of this application.

[0060] Figure 6 This is a flowchart illustrating the feature transformation performed on the feature channel based on the transformation coefficients in an embodiment of this application. Detailed Implementation

[0061] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0062] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0063] The terms "first," "second," "third," etc., used in the specification and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms can be used interchangeably where appropriate.

[0064] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0065] Users can play videos through smart devices, such as movies, TV series, operas, or variety shows. When watching films from different countries, because the film's language is different from the user's native language, the user cannot directly understand the meaning of the dialogue spoken by the characters in the film. For example, when a Chinese user watches an English film, if the user cannot understand English, they cannot directly know the meaning of the dialogue spoken by the characters in the film from the audio.

[0066] Therefore, some video resources can include subtitle resources, that is, displaying subtitles in the corresponding language on the screen while the video is playing. For example, when playing a video in English, Chinese subtitles can be added on the screen so that Chinese users can understand the meaning of the dialogue. This requires users to watch the video and subtitles simultaneously to understand the meaning of the characters' dialogue. Although the subtitles are on the screen, repeatedly watching content in different positions affects the user's viewing experience.

[0067] To improve the viewing experience, the dialogue in a film can be translated into a different language before being dubbed in post-production, allowing users to intuitively understand the meaning of the lines based on the dubbing. However, this post-production dubbing method is time-consuming and needs to be completed before the film is played, making it impossible to translate and play the dialogue in real time.

[0068] To address this, speech-to-text technology can be used to convert the film's language to the user's desired language while preserving the characters' vocal characteristics, thus achieving real-time translation. However, during this process, because the language of the spoken dialogue changes, the original lip movements in the film cannot correspond to the converted dialogue. For example, when the audio is in the first language, the film plays the audio in that language based on preset sound characteristics, and the characters' lip movements in the film correspond to the pronunciation of the audio in the first language. After real-time translation, the audio is converted from the first language to the second language. Due to the differences in pronunciation between languages, the characters' lip movements in the film do not correspond to the audio in the second language, resulting in a discrepancy between the lip movements the user sees and the actual audio pronunciation, negatively impacting the user's viewing experience.

[0069] To address the issue of mismatch between lip movements and language pronunciation in video footage after language conversion, some embodiments of this application provide a method for processing characters during video conversion. This method can be applied to display devices used for playing audio and video or to lip-syncing systems based on language conversion. Figure 1 A flowchart illustrating a method for processing people in video conversion, as provided in this application. See also... Figure 1 Taking a person processing system applied to video conversion as an example, the method includes:

[0070] S100: Get the video to be converted.

[0071] The lip-syncing system can acquire the video to be converted input by the user. The video to be converted includes the audio to be converted and an image of the first person. The audio to be converted is the original audio of the video, which is also the audio data for which language conversion will be performed. The language information of the audio to be converted is the first language. The first language is the language of the audio before translation; for example, if a user is playing an English movie on a display device, then the first language of the English movie is English.

[0072] In some embodiments, the video to be converted may also be a dubbed film, meaning the language of the audio to be converted is different from the original language. In this case, the first language is the dubbed language. For example, if the video to be converted is an English film dubbed in Chinese, then the first language of the audio to be converted is Chinese.

[0073] The first person image is the person speaking in the video to be converted. The lip-syncing system can process the video to be converted into frames to obtain the first person image from the frames. The first person image includes the first lip-sync action of the target person speaking in the first language, wherein the target person is the person in the first person image who is speaking.

[0074] In some embodiments, the first person image may also include facial images of other people. For example, the frontal view of the target person and the frontal images of other people in the video to be converted may be displayed simultaneously in the video frames of the video to be converted. During the subsequent video conversion process, the lip movements of other people can be converted simultaneously. In addition to facial images or lip movements, the first person image may also include information such as the target person's body movements, posture, and expression, so as to keep the body movements, posture, and expression unchanged during the video conversion process, or to change the target person's posture and movement in the video to be converted according to preset action information.

[0075] In some embodiments, the lip-syncing system can detect the number of target individuals in a frame and acquire a first image of each individual based on that number. When a frame contains only one target individual, the image of the target individual speaking in the frame can be used as the first image. When a frame contains multiple target individuals, first images of different target individuals speaking can be acquired separately in the frame, and a set of individual images can be generated based on the first images of the corresponding target individuals. For example, when a frame contains individuals A and B speaking simultaneously, the lip-syncing system can acquire the first images of individuals A and B respectively, generate a set of individual images A based on the first image of individual A, and generate a set of individual images B based on the first image of individual B, so that lip-syncing can be performed separately for each speaking individual in subsequent processing.

[0076] It should be understood that a person in a non-speaking state in a framed picture is not counted as a target person. For example, when in a framed picture, person A is in a speaking state and person B is in a non-speaking state, at this time, person A is the target person and person B is not the target person. Therefore, the lip movement conversion system can also detect the speaking state of the person in the video to be converted and update the number of target persons according to the speaking state. For example, in the above example, when person B changes from a non-speaking state to a speaking state, both person A and person B in the framed picture are in a speaking state. At this time, person B is the target person in the current framed picture, and at this time, the number of target persons changes from 1 to 2.

[0077] S200: Perform language conversion on the audio to be converted according to the second language to obtain the target audio.

[0078] After obtaining the video to be converted, the lip movement conversion system can extract the audio to be converted in the video to be converted to facilitate subsequent processing of the audio to be converted. After completing the extraction of the audio to be converted, the lip movement conversion system can perform language conversion on the audio to be converted according to the language conversion instruction input by the user according to the second language or to obtain the target audio, where the second language is different from the first language. For example, when the audio to be converted is in Chinese, a second language different from English can be used to perform language conversion on the audio to be converted, such as English, Russian, Spanish, Portuguese, etc. Taking the second language as English as an example, when the audio to be converted is the voice audio of "Hello", the target audio after performing language conversion is the voice audio of "Hello". After performing language conversion on the audio to be converted, the obtained target audio maintains the voice characteristics of the target person, but the language information is switched from the first language to the second language.

[0079] In some embodiments, the audio to be converted is the voice audio spoken by the target person. After the lip movement conversion system extracts the audio to be converted, in order to reduce the interference of other audio in the video to be converted on the voice audio, the lip movement conversion system can perform noise reduction processing on the audio to be converted to reduce the influence of other audio on the audio to be converted.

[0080] S300: Input the target audio and the first person image into the lip movement conversion model to obtain the second person image output by the lip movement conversion model.

[0081] After the lip movement conversion system performs language conversion on the audio to be converted to obtain the target audio, at this time, due to the change in language, the first lip movement action of the target person speaking in the first language is not the same as the pronunciation action of the target audio. Therefore, when the user watches the video to be converted, there will be a problem that the lip movement action of the target person in the video to be converted does not match the pronunciation lip movement action of the target audio.

[0082] To this end, the lip-syncing system can input the target audio and a first person image into a lip-syncing model. The lip-syncing model can extract the audio features of the target audio and, based on the audio features, switch the first lip movement of the target person in the first person image to a second lip movement, thereby outputting a second person image. The lip-syncing model is obtained by performing a specific training process based on training data, which includes at least one training person image with lip movements. The lip-syncing model includes a transformation module, which calculates transformation coefficients corresponding to each feature channel in the image features of the first person image based on at least the target audio. The transformation module can perform feature transformation on each feature channel of the first person image based on the calculated transformation coefficients, thereby obtaining the image features of the first person image.

[0083] The second character image includes the target character's second lip movements when speaking in a second language. The second lip movements are the same as the lip movements of the target audio, so that the lip movements of the character seen by the user correspond to the target audio after language conversion, thus improving the user's viewing experience.

[0084] S400: Generate a target video based on the target audio and the second person image.

[0085] After obtaining the second person image, the lip-syncing system can replace the audio to be converted in the video with the target audio after language conversion, and replace the first person image in the video with the second person image after lip-syncing, so that after language conversion, the lip-syncing of the target person is synchronously converted in the video, and the lip-syncing of the person in the video corresponds to the language pronunciation.

[0086] In some embodiments, since only the lip-sync area of ​​the target person is transformed in the first person image, the lip-sync system only replaces the lip-sync area of ​​the target person when replacing the first person image with the second person image, so that the overall facial area of ​​the target person remains unchanged.

[0087] In order to output a second image based on a first image and target audio, the lip-syncing system needs to perform a specific training process on the lip-syncing model based on training data containing the target image to obtain the lip-syncing model.

[0088] In some embodiments, the lip-syncing system can acquire training data for training the lip-syncing model, and the training data is video data. The training data may include a first training image, which includes at least one training person performing lip-syncing actions. The training person can be the target person performing the lip-syncing during the application. For example, the training data may include images of person A. After training the lip-syncing model's lip-syncing capability using the training data, the lip-syncing system can perform lip-syncing actions on person A during the application.

[0089] The training subjects can also be people other than the target person. For example, the training data can be images of people including person A. After training the lip-syncing model's lip-syncing ability using the training data, the lip-syncing system can perform lip-syncing on person B, who is different from person A, during the application process.

[0090] like Figure 2 As shown, the lip-syncing system can input training data into a model to be trained. This model is an untrained lip-syncing model that has not yet converged, and it shares the same model structure as the original lip-syncing model. After receiving the training data, the model to be trained can extract the training audio features to determine the pronunciation lip shape based on these features. The model to be trained can also extract a first training image from the training data. Since the training data consists of video data containing the target person, the first training image contains a facial image of the target person.

[0091] The model to be trained can perform feature channel transformation on the first training image based on the training audio features to generate a second training image. In the second training image, the target person makes the same mouth movements as the pronunciation of the training audio features. Since the parameters of the model to be trained have not been trained to converge, the second training image generated by the model should have a certain training loss. Therefore, the lip-syncing system can calculate the training loss between the second training image and the lip-sync image labels using a loss function. The lip-sync image labels are used to represent the pronunciation mouth movements of the training audio features, and the training loss is used to determine whether the second training image meets the output standard.

[0092] To determine whether the second training image meets the output standard, the lip-syncing system can set a loss threshold. If the training loss is greater than the loss threshold, it indicates that the training loss between the second training image and the lip-sync image label is too large, and the second training image cannot be output. Therefore, the lip-syncing system can iteratively train the training model to update the model parameters of the model to be trained and regenerate the second training image. When the training loss is less than or equal to the loss threshold, it indicates that the second training image meets the output standard, and the lip-syncing system can obtain the lip-syncing model based on the model parameters of the model to be trained.

[0093] In some embodiments, considering the diversity of languages ​​in the training data, the model to be trained can perform speech extraction on the audio portion of the training data using the wav2vec model to obtain training audio features. The wav2vec model can perform sequence layering on the audio portion of the training data and extract audio features from the middle layer of the audio sequence to obtain training audio features, making the obtained audio features have good generalization for different languages, different receiving devices, and different speakers.

[0094] The model to be trained may include an encoder, a transform module, and a decoder. Figure 3 This is a flowchart illustrating the training process of the model to be trained in an embodiment of this application. See also... Figure 3 After the training data is input into the model to be trained, the model to be trained extracts the training audio features and the first training image from the training data, respectively, and encodes the training audio features through an encoder to obtain the training audio code, and encodes the first training image through an encoder to obtain the first training encoded image.

[0095] After the encoder completes encoding, the training audio code and the first training image code can be input into the transformation module. The transformation module then performs lip-syncing processing on the first training image code based on the training audio code. The transformation module consists of fully connected layers. Through these fully connected layers, the transformation module learns the transformation relationships between different images, transforming the first training image code to the desired lip-syncing state based on the training audio code.

[0096] After inputting the training audio encoding and the first training image encoding into the transformation module, the first training image encoding becomes the image encoding to be transformed. The transformation module can determine the feature channel calculation relationship corresponding to the first training image encoding based on the training audio encoding, thereby realizing the transformation of the training character's lip movements in the first training image at the feature channel level. The feature channel calculation relationship corresponds to different transformation relationships of the feature channels. For example, the feature channel calculation relationship can include rotation, translation, and scaling relationships.

[0097] To determine the feature channel calculation relationships, the lip-syncing system can extract a reference image from the training data based on the encoding of the first training image. The reference image can be an image of the same person as the training subject in the first training image, used to represent the image of the training subject in the training data and serving as a reference image for obtaining the feature channel calculation relationships. For example, the reference image can be a frontal image of the training subject. By using the reference image as a reference, the transformation relationships of different feature channels, such as the position, facial angle, or facial size of the first training image relative to the reference image, can be determined. Based on these transformation relationships, the feature channel calculation relationships can be determined to facilitate subsequent lip-syncing of the encoding of the first training image.

[0098] In some embodiments, the reference image may also be multiple images of a person from different angles to determine the feature channel calculation relationship from different angular dimensions. For example, the reference image may include a first reference image, a second reference image, and a third reference image from different angles.

[0099] After obtaining the baseline image, in order to improve the accuracy of the feature channel calculation relationship, such as... Figure 4 As shown, the reference image and the first training image can be concatenated to obtain a training concatenated image. Transformation coefficients are then calculated based on the concatenated training image and the training audio encoding to determine the feature channel calculation relationship based on the training audio features. Specifically, when performing lip-sync conversion on the first training image encoding, the transformation module can perform corresponding transformation calculations on the feature channels of the first training image encoding based on the transformation coefficients to obtain the second training image encoding. The transformation coefficients indicate the transformation relationship of the first training image relative to the reference image.

[0100] Since the reference image and the first training image are different images, feature stitching errors may occur in the training stitched image after image stitching is performed.

[0101] To eliminate feature splicing errors, in some embodiments, such as Figure 5 As shown, the encoder may include an alignment encoder configured to perform image registration on the training stitched images. Image registration is the process of finding matching points between different images and aligning those points spatially during the stitching of multiple images from different sources to minimize the desired error.

[0102] Based on this, the alignment encoder can perform feature alignment on the stitched training image according to the matching points between the first training image and the reference image during the stitching process, and extract the alignment features of the stitched training image. Based on the alignment features, the transformation module can calculate transformation coefficients according to the alignment features and training audio features.

[0103] In some embodiments, the transform coefficients may include multiple sub-transform coefficients corresponding to the transformation operations to be performed on different feature channels. For example, such as Figure 6As shown, corresponding to the calculation of feature channel rotation relationships, feature channel translation relationships, and feature channel scaling relationships, the transformation coefficients can include a first transformation coefficient, a second transformation coefficient, and a third transformation coefficient. The first transformation coefficient is a sub-transformation coefficient for calculating the rotation angle of the feature channels; the transformation module can calculate the rotation angle of each feature channel in the first training image encoding based on the first transformation coefficient. The second transformation coefficient is a sub-transformation coefficient for calculating the translation distance of the feature channels; the transformation module can calculate the translation distance of each feature channel in the first training image encoding based on the second transformation coefficient. The third transformation coefficient is a sub-transformation coefficient for calculating the scaling factor of the feature channels; the transformation module can calculate the scaling factor of each feature channel in the first training image encoding based on the second transformation coefficient. After performing the corresponding rotation, translation, and scaling calculations on each feature channel of the first training image encoding, the first training image encoding can be converted into the second training image encoding based on the calculated rotation angle, translation distance, and scaling factor.

[0104] It should be noted that the embodiments of this application only illustrate three transformation calculation methods for feature channels: rotation, translation, and scaling. In actual training, the transformation calculation methods for feature channels can be increased or decreased according to the desired lip-sync effect of the lip-sync model. When increasing transformation calculation methods, the transformation coefficients may include three or more sub-transformation coefficients, such as a fourth transformation coefficient, a fifth transformation coefficient, etc. The embodiments of this application do not specifically limit the number of transformation calculation methods for feature channels.

[0105] The second training image encoding is the image encoding after the training person performs lip-syncing based on the training audio features. The transformation module can output the second training image encoding to the decoder, so that the decoder can decode the second training image encoding to obtain the second training image, thus completing the lip-syncing in the training process.

[0106] As can be seen from the above technical solution, this embodiment performs transformation on each feature channel during the lip-sync conversion process, resulting in low computational load and improved efficiency. Furthermore, since it does not regenerate the first image of the person in the application process, it can better preserve the details of the person and their corresponding lip movements in videos with lip occlusion, thereby completing the lip-sync conversion and synthesis of the occluded video. In the above process, the transformation module learns the transformation process with a training person, rather than training on a specific person. Therefore, it has good generalization ability during lip-sync conversion and is not limited to the person corresponding to the lip-sync conversion, thus eliminating the need for targeted training on a specific person.

[0107] In some embodiments, the encoder is further configured to perform attention computation on the second training image encoding to perform attention encoding on the second training image. For this purpose, the encoder also includes a hierarchical structure formed by a predetermined number of encoding units, wherein the input of an encoding unit is the output of the previous encoding unit, and the output of an encoding unit is the input of the next encoding unit. The input of the encoder can also be connected to the output of the transformation module, i.e., the input of the encoder is the second training image encoding output by the transformation module.

[0108] To improve model fitting ability and reduce overall computational cost, before the encoder performs attention calculations on the second training image encoding, the lip-syncing system can divide the second training image encoding into multiple first image segments according to the calculation window. Each first image segment includes a certain number of pixels. The lip-syncing system can input the first image segments into the encoder, so that attention calculations can be performed independently on each first image segment through a certain encoding unit to obtain the first attention encoding result.

[0109] However, in order to fully learn the features at different locations within the first feature encoding, after completing one attention calculation, the lip-sync system needs to adjust the first image segment in a preset manner, thereby re-dividing the second training image encoding into a preset number of second image segments. The first image segment may partially overlap with its corresponding second image segment, and the number of first and second image segments may vary depending on the size of the image segmentation.

[0110] After the segmentation is complete, the encoding unit can input the first attention encoding result into the unit of the next level corresponding to the current level. Based on the first attention encoding result, the encoding unit performs attention calculation on the second image segment to obtain the second feature encoding result. The encoder iteratively performs the above attention calculation operation. Based on the second attention encoding result and the number of layers of the remaining encoding units, iterative attention calculation is performed to output the second feature encoding. It should be understood that in each iteration of attention calculation, the image segment needs to be re-segmented based on the first feature encoding. The segmentation method can be based on preset rules, such as segmentation angle, image shape, image area, etc., or the image segmentation can be randomized. It should be ensured that every pixel in the first feature encoding participates in the attention calculation. For example, in the first layer encoding unit, the window segmentation is positioned as the initial window. In the next layer encoding unit, the window segmentation is moved one pixel to the right and down from the initial window. Through this iteration, as the layers deepen, each pixel can be considered in the calculation of different windows. This allows the encoder to simultaneously combine local and global information encoded in the second training image during the attention mechanism calculation process, thereby preserving detailed local information and global context during the generation of the second training image.

[0111] In some embodiments, a hierarchical design is employed between coding units, wherein each level gradually reduces the resolution of the first training image while increasing the dimension of the features.

[0112] In some embodiments, the decoder may include a preset number of upsampling layers, wherein the image space dimension of the upsampling layers gradually increases in the order of arrangement, while the number of feature channels gradually decreases. After obtaining the second feature encoding, the second feature encoding can be input into the decoder, and a decoding operation is performed on the second feature encoding according to the upsampling layers to obtain the second training image.

[0113] In some embodiments, the feature dimensions of the upsampling layer and the coding unit increase with the increase of the layer, while the number of feature channels of the upsampling layer and the coding unit decreases with the increase of the layer.

[0114] Since the coding units and upsampling layers are hierarchical structures, and the number of feature channels gradually decreases while the feature dimensionality increases with the increase of layers, the coding units in the coding module can be distributed with the upsampling layers in the decoder according to a corresponding structure. That is, the upsampling layers and coding units are connected at corresponding levels. For example, the coding unit at the first level is connected to the upsampling layer at the first level, the coding unit at the second level is connected to the upsampling layer at the second level, and so on.

[0115] It should be understood that the above embodiments are described using the hierarchical structure formed by the coding units as a whole. In the actual encoding process, the actual output of each coding unit is the attention coding result obtained by performing attention calculation on the image segment. As the coding unit hierarchy deepens, the encoder performs encoding calculation on each pixel in the first training image, thereby summarizing the attention coding results into the first training image encoding. Correspondingly, the input of the upsampling layer is the attention coding result after performing feature channel conversion. After completing the decoding operation of all attention coding results, the decoder outputs the second training image after performing feature channel conversion to complete the lip-syncing in the training process.

[0116] To facilitate the execution of the person processing method for video conversion described in the above embodiments, some embodiments of this application also provide a person processing system for video conversion. The system includes a controller and a lip-syncing model. The lip-syncing model includes a transformation module, configured to perform feature transformation on the image feature channels of the training data input to the lip-syncing model during training. The training data includes at least one training person image with lip-syncing actions. The controller is configured to:

[0117] S100: Obtain the video to be converted, which includes the audio to be converted and the image of the first person.

[0118] The language information of the audio to be converted is a first language, and the first person image includes the first lip-syncing action of the target person speaking in the first language.

[0119] S200: Perform language conversion on the audio to be converted according to the second language to obtain the target audio, wherein the second language is a different language from the first language;

[0120] S300: Input the target audio and the first person image into the lip-sync model to obtain the second person image output by the lip-sync model.

[0121] The second character image includes a second lip-syncing action of the target character speaking in the second language.

[0122] S400: Generate a target video based on the target audio and the second person image.

[0123] As can be seen from the above technical solutions, this application provides a method and system for character processing in video conversion. The method involves acquiring a video to be converted, performing language conversion on the audio of the video to be converted in a second language, inputting the target audio and a first character image into a lip-syncing model, obtaining a second character image output by the lip-syncing model, and then generating a target video based on the target audio and the second character image. This application adjusts the lip movements in the first character image using translated audio data, ensuring that the lip movements of the target character in the target video correspond to the pronunciation of the translated target audio, thereby improving the user experience.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0125] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the foregoing exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be made based on the foregoing teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better utilize the described embodiments.

Claims

1. A character processing method applied to video conversion, characterized by, include: The video to be converted is obtained, which includes audio to be converted and a first person image. The language information of the audio to be converted is a first language, and the first person image includes at least the first lip-syncing action of the target person speaking in the first language. The audio to be converted is converted into a different language than the first language to obtain the target audio. The target audio and the first person image are input into a lip-sync model to obtain a second person image output by the lip-sync model. The second person image includes at least a second lip-sync action of the target person speaking in the second language. The lip-sync model includes a transformation module configured to calculate, at least based on the target audio, a transformation coefficient corresponding to each feature channel in the image features of the first person image, and perform feature transformation on the image feature channels of the first person image based on the transformation coefficients to obtain the image features of the first person image. During the process of replacing the first person image with the second person image, only the lip-sync action area of ​​the target person is replaced so that the overall facial area of ​​the target person remains unchanged. Generate a target video based on the target audio and the second person image; Before the step of inputting the target audio and the first person image into the lip-sync model, the method further includes: Acquire training data; the training data includes a first training image, the first training image including at least one training figure with lip movements; The training data is input into the model to be trained so as to extract the training audio features and the first training image of the training data through the model to be trained. The model to be trained is a lip-sync model that has not been trained to convergence. Based on the training audio features, the first training image is transformed using the model to be trained to generate the second training image; The training loss between the second training image and the lip shape image label is calculated based on the loss function; If the training loss is greater than the loss threshold, then the model to be trained is trained iteratively. If the training loss is less than or equal to the loss threshold, then the lip-sync model is input according to the model parameters of the model to be trained. The model to be trained includes an encoder, a transform module, and a decoder. Based on the training audio features, the step of performing feature channel transformation on the first training image using the model to be trained includes: The encoder encodes the training audio features to obtain the training audio code, and the encoder encodes the first training image to obtain the first training image code; The transformation module determines the feature channel calculation relationship based at least on the trained audio encoding and the first trained image encoding. Based on the feature channel calculation relationship, feature channel transformation is performed on the first training image encoding to obtain the second training image encoding; The second training image is obtained by decoding the second training image encoding through the decoder; wherein, the transformation module is composed of fully connected layers, and the transformation module learns the transformation relationship between different images through the fully connected layers, and transforms the first training image encoding to the desired lip-sync state according to the training audio encoding; The encoder includes a hierarchical structure formed by a preset number of coding units. The input of each coding unit is the output of the previous coding unit, and the output of each coding unit is the input of the next coding unit. The step of decoding the second training image encoded by the decoder includes: The second training image is encoded and divided into a preset number of first image segments; The first image segment is input into the encoder so that attention calculation is performed independently on each of the first image segments through a certain encoding unit to obtain the first attention encoding result; The first image segment is adjusted in a preset manner to divide the second training image encoding into a preset number of second image segments; wherein, there is partial overlap between the first image segment and the corresponding second image segment; The first attention encoding result is input into the next level encoding unit to perform attention calculation on the second image segment through the encoding unit, so as to obtain the second attention encoding result; Iterate the above operations to obtain the second feature encoding based on the second attention encoding result; Decoding the second feature encoding yields the second training image.

2. The character processing method for video conversion according to claim 1, wherein, The step of determining the feature channel calculation relationship through the transformation module, at least based on the trained audio encoding and the first trained image encoding, includes: A reference image is obtained based on the encoding of the first training image, wherein the reference image is a person image that is the same as the training person in the first training image; The reference image is stitched together with the first training image to obtain a training stitched image; The transformation coefficients are calculated based on the training audio encoding and the training stitched image to determine the feature channel calculation relationship, wherein the transformation coefficients are used to indicate the transformation relationship of the first training image relative to the reference image.

3. The character processing method for video conversion according to claim 2, wherein, The step of calculating the transform coefficients based on the training stitched image includes: The encoder extracts the alignment features of the trained stitched image; Transformation coefficients are calculated based on the alignment features and the training audio features.

4. The character processing method for video conversion according to claim 3, wherein, The transform coefficients include a first transform coefficient, a second transform coefficient, and a third transform coefficient. The step of performing feature channel transformation on the encoding of the first training image based on the feature channel calculation relationship includes: The rotation angle of the feature channel encoded in the first training image is calculated based on the first transformation coefficient; The translation distance of the feature channels encoded in the first training image is calculated based on the second transformation coefficients; The scaling factor of the feature channels encoded in the first training image is calculated based on the third transformation coefficient; The first training image code is converted into a second training image code based on the rotation angle, the translation distance, and the scaling factor.

5. The character processing method for video conversion according to claim 1, wherein The decoder includes a preset number of upsampling layers; The step of decoding the second feature encoding to obtain the second training image includes: The second feature encoding is input into the decoder, and the second feature encoding is decoded according to the upsampling layer to obtain the second training image; In this module, a preset number of encoding units are distributed with the upsampling layer according to a corresponding structure. In the decoder, the second feature encoding of the upsampling layer performing the decoding operation is output by a non-corresponding encoding unit.

6. The character processing method for video conversion according to claim 5, wherein, The feature dimensions of the upsampling layer and the coding unit increase with the increase of the layer, while the number of feature channels of the upsampling layer and the coding unit decreases with the increase of the layer. The encoding unit and the upsampling layer are connected at the corresponding level.

7. A character processing system applied to video conversion, characterized by, The system is configured with the person processing method for video conversion as described in claim 1. The system includes a controller and a lip-sync model. The lip-sync model includes a transformation module, which is configured to calculate, at least based on the target audio, transformation coefficients corresponding to each feature channel in the image features of the first person image, and perform feature transformation on the image feature channels of the first person image based on the transformation coefficients to obtain the image features of the first person image. The controller is configured to: The video to be converted is obtained, which includes audio to be converted and a first person image. The language information of the audio to be converted is a first language, and the first person image includes at least the first lip-syncing action of the target person speaking in the first language. The audio to be converted is converted into a different language than the first language to obtain the target audio. The target audio and the first person image are input into the lip-sync model to obtain a second person image output by the lip-sync model. The second person image includes at least the second lip-sync action of the target person speaking in the second language. A target video is generated based on the target audio and the image of the second person.

Images