Processing method and device and electronic equipment

Abstract

The invention discloses a processing method and device and electronic equipment. The method comprises the following steps: extracting initial text content and action information from an initial video; the action information represents a target action associated with the initial text content in the initial video and the occurrence time of the target action in the initial video; performing language conversion on the initial text content to obtain target text content; obtaining a target video based on the target text content and the action information; the target text content in the target video is matched with the action corresponding to the target text content.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a processing method, device and electronic device. Background Art

[0002] In video translation scenarios, there is often a problem that the translated speech does not match the actions of the characters in the original video. Summary of the invention

[0003] The technical solutions provided by this application are as follows:

[0004] The first aspect of the present application provides a processing method, comprising:

[0005] Extracting initial text content and action information from an initial video; wherein the action information represents a target action associated with the initial text content in the initial video and a time when the target action appears in the initial video;

[0006] Performing language conversion on the initial text content to obtain target text content;

[0007] A target video is obtained based on the target text content and the action information; the target text content in the target video and the action corresponding to the target text content are matched.

[0008] In a possible implementation, obtaining a target video based on the target text content and the action information includes:

[0009] Based on the target text content, generate target audio;

[0010] Determining a target time in the target audio corresponding to the target action;

[0011] Determine the video frame to be adjusted corresponding to the target time from the initial video;

[0012] Adjust the to-be-adjusted video frame according to the action information to obtain a target video frame;

[0013] The target video is obtained according to the target video frame, the initial video and the target audio.

[0014] The adjusting the to-be-adjusted video frame according to the action information includes:

[0015] The action of the target image in the video frame to be adjusted is changed according to the action information.

[0016] In a possible implementation, obtaining the target video according to the target video frame, the initial video, and the target audio includes:

[0017] generating the supplementary video frame according to the target video frame and the initial video;

[0018] The target video frame is used to replace the video frame to be adjusted, and the previous and next video frames of the video frame to be adjusted are replaced by the supplementary video frame to obtain an updated video, wherein the supplementary video frame can make the target video frame and other frames in the updated video meet a coherent state;

[0019] The target video is obtained according to the updated video and the target audio.

[0020] In a possible implementation, extracting motion information from the initial video includes at least one of the following:

[0021] If the audio of the initial video contains an emphatic tone, extracting action information corresponding to the emphatic tone from the initial video;

[0022] According to the emotion expressed by the initial text content, extracting action information corresponding to the emotion from the initial video;

[0023] Keywords in the initial text content are identified, and action information corresponding to the keywords is extracted from the initial video.

[0024] In a possible implementation, obtaining a target video based on the target text content and the action information includes:

[0025] Based on the target text content and the action information, a target audio is generated; the expression attribute of the target audio matches the target action represented by the action information;

[0026] The target audio and the video frames in the initial video are synthesized to obtain the target video.

[0027] In a possible implementation, the action information includes: an action feature sequence; the action feature sequence represents the action appearing in each video frame in the initial video;

[0028] Generating a target audio based on the target text content and the action information, including:

[0029] The target text content and the action feature sequence are processed based on an audio generation model to generate target audio.

[0030] In a possible implementation, performing language conversion on the initial text content to obtain target text content includes:

[0031] The initial text content is inferred based on the target model to obtain target text content, where the target text content and the initial text content are in different languages.

[0032] Another aspect of the present application provides a processing device, comprising:

[0033] An extraction module, used to extract initial text content and action information from an initial video; the action information represents a target action associated with the initial text content in the initial video and a time when the target action appears in the initial video;

[0034] A conversion module, used for performing language conversion on the initial text content to obtain target text content;

[0035] A processing module is used to obtain a target video based on the target text content and the action information; the target text content in the target video is matched with the action corresponding to the target text content.

[0036] A third aspect of the present application provides an electronic device, including:

[0037] The memory is used to store computer programs;

[0038] The processor is used to execute the computer program so that the electronic device can implement any of the processing methods described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0040] Figure 1 A schematic diagram of a process for a treatment method provided in Example 1 of the present application;

[0041] Figure 2 A schematic diagram of a process flow of a processing method provided in Example 2 of the present application;

[0042] Figure 3 A schematic diagram of an implementation scenario of a processing method provided in this application;

[0043] Figure 4 A schematic diagram of another implementation scenario of a processing method provided in this application;

[0044] Figure 5 A schematic diagram of another implementation scenario of a processing method provided in this application;

[0045] Figure 6 A schematic diagram of a process for a treatment method provided in Example 6 of the present application;

[0046] Figure 7 A schematic diagram of a process for a treatment method provided in Example 7 of the present application;

[0047] Figure 8 A training diagram of an audio generation model provided in this application;

[0048] Fig. 9 Another training schematic diagram of an audio generation model provided in this application;

[0049] Fig.10 A schematic diagram of another implementation scenario of a processing method provided in this application;

[0050] Fig.11 A schematic diagram of the structure of a processing device provided in this application. DETAILED DESCRIPTION

[0051] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0052] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0053] The terms "first", "second" etc. in the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0054] In the field of video translation, virtual human video translation technology has demonstrated its unique advantages. It can accurately translate the voice in the original video into another language, or accurately modify the lip shape of the character in the video according to the input new voice, ensuring perfect synchronization between the picture and the voice. This technology has made significant progress in language conversion and the consistency between lips and language, bringing unprecedented convenience and efficiency to video translation.

[0055] However, despite the remarkable achievements of virtual human video translation technology in these aspects, it still faces some challenges in practical applications. In particular, when translating the speech in the original video, due to the significant differences in language, word order, grammatical structure and language expression between different languages, the translated content is often difficult to keep completely consistent with the tone, expression and action of the characters in the original video. This inconsistency is particularly prominent in video translation and may cause confusion or discomfort to the audience.

[0056] Specifically, in the original video, characters may emphasize their spoken content through specific actions (such as nodding, smiling, nodding, etc.). However, after being translated into other languages, these actions that were originally closely related to the spoken content may conflict with the new spoken content due to language differences and the choice of translation strategy. For example, an action that was originally emphasized in the original video may become an inappropriate action corresponding to a disgusting word after translation because it does not match the new spoken content, thereby destroying the coherence of the video and the audience's viewing experience.

[0057] In addition, in the scenario where the lip shape of a person in a video is modified based on the input new voice, another issue that needs attention is the matching degree between the gestures of the person in the video and the rhythm of the new voice. If the gestures do not match the rhythm of the new voice, the generated video will appear unnatural and may even affect the audience's understanding and acceptance of the video content.

[0058] Therefore, the present application provides a processing method to make the voice content and character movements in the translated video consistent.

[0059] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0060] Reference Figure 1 , is a flow chart of a processing method provided in Example 1 of the present application, such as Figure 1 As shown, the method may include but is not limited to the following steps:

[0061] Step S101, extracting initial text content and action information from an initial video; the action information represents a target action associated with the initial text content in the initial video and a time when the target action appears in the initial video.

[0062] In this embodiment, the speech content in the initial video may be converted into text content as the initial text content through language recognition technology.

[0063] In this embodiment, the video frames in the initial video can be analyzed by computer vision technology (such as human posture estimation and action recognition), the target action (such as gestures (such as pointing, waving), facial expressions (such as smiling, frowning) or body postures (such as leaning forward, leaning back), etc.) associated with the initial text content can be identified, and the time when the target action appears in the initial video can be recorded. According to the relationship between the synchronous playback of video and audio and the time when the target action appears in the initial video, the time when the target action appears in the initial audio in the initial video can be determined.

[0064] For example, if in the initial video, a person is saying in Chinese: "Welcome to our company. ..." The initial audio of the sentence "Welcome to our company" in the initial video starts from the 5th second and lasts to the 7th second. At the same time, when he said "Welcome" (about 5.5 seconds in the initial audio), he nodded, and this action lasted until the 6th second in the initial audio.

[0065] Through speech recognition technology, the initial text content of this sentence can be extracted: "Welcome to our company." The timestamp of this initial text content is from the 5th second to the 7th second. Through computer vision technology, the nodding action can be identified and the time when the nodding action appears in the initial video can be recorded. According to the time when the nodding action appears in the initial video, it can be determined that the time when the nodding action appears in the initial audio of the initial video is from the 5.5th second to the 6th second.

[0066] Step S102: performing language conversion on the initial text content to obtain target text content.

[0067] The target text content and the original text content are in different languages. For example, if the original text content is "Welcome to our company." in Chinese, the original text content can be converted into English to obtain the target text content: "Welcome to our company."

[0068] Step S103: obtaining a target video based on the target text content and the action information; matching the target text content in the target video with the action corresponding to the target text content.

[0069] The target text content in the target video and the action corresponding to the target text content match each other, which can be understood as follows: in the target video, the target text content (voice) and the action corresponding to the target text content are synchronized in time, and the action corresponding to the target text content is consistent with the semantics of the target text content.

[0070] The action corresponding to the target text content is obtained according to the action information.

[0071] For example, in step S101, the initial text content "Welcome to our company." and the corresponding action information (the nodding action and its time in the video) have been extracted from the initial video. In step S102, the initial text content has been converted into the English target text content "Welcome to our company.".

[0072] In the target video, the character's voice has been changed to the English "Welcome to our company." If the pronunciation of the English "Welcome" is longer than the Chinese "Welcome", and the pronunciation of the English "Welcome" corresponding to the Chinese "Welcome" in the target video starts from the 5.6th second and lasts to the 6.2th second, then the nodding action can start from the 5.6th second or a later time point and last until the time point that matches the end of the English pronunciation "Welcome".

[0073] In this embodiment, by extracting the initial text content and action information from the initial video, the action information represents the target action associated with the initial text content in the initial video and the time when the target action appears in the initial video, and performing language conversion on the initial text content to obtain the target text content, a target video is obtained based on the target text content and the action information. The target text content in the target video matches the action corresponding to the target text content, that is, the target text content in the target video and the action corresponding to the target text content are synchronized in time and semantically consistent, which can improve the accuracy and naturalness of video translation, and enhance the viewing experience and understanding of the audience.

[0074] As another optional embodiment of the present application, refer to Figure 2 , is a flow chart of a processing method provided in Example 2 of the present application. This example is mainly an implementation method of step S103 in Example 1. Figure 2 As shown, step S103 may include but is not limited to the following steps:

[0075] Step S1031: Generate target audio based on the target text content.

[0076] The process of generating the target audio usually involves text-to-speech (TTS) technology, which can convert the target text content (such as "Welcome to our company." in English) into corresponding speech audio.

[0077] Step S1032: Determine a target time in the target audio corresponding to the target action.

[0078] This step may include but is not limited to:

[0079] Step S11, based on the time when the target action appears in the initial video and the relationship between the synchronous playback of the video and the audio, the time of the target action in the initial audio in the initial video can be determined, and the initial text range corresponding to the time of the target action in the initial audio in the initial video can be obtained from the initial text content.

[0080] For example, as shown in the example of step S101 in Example 1, the time of the nodding action in the initial audio of the initial video is from 5.5 seconds to 6 seconds, and the initial text range corresponding to 5.5 seconds to 6 seconds can be determined as "Welcome" from the initial text content "Welcome to our company. ..."

[0081] Step S12: Determine a target text range corresponding to the initial text range from the target text content.

[0082] For example, after language conversion of the initial text content "Welcome to our company. ...", the target text content is "Welcome to our company. ...", and the target text range corresponding to "Welcome" is determined to be "Welcome" from the target text content "Welcome to ourcompany. ...".

[0083] Step S13: The time of the target text range in the target audio is taken as the target time corresponding to the target action.

[0084] For example, if the target text range is "Welcome ", and the time of "Welcome " in the target audio is from 5.6 seconds to 6.2 seconds, then the target time corresponding to the target action is from 5.6 seconds to 6.2 seconds.

[0085] Step S1033: Determine the video frame to be adjusted corresponding to the target time from the initial video.

[0086] In video processing, a video is composed of a series of continuous video frames, each of which has a specific time point (or timestamp). When playing a video, these frames will be played continuously in chronological order to form dynamic video content. At the same time, the audio is also played in chronological order and kept in sync with the video frames. Therefore, in a scenario where the target video and the video frames in the initial video can be played synchronously, the video frame to be adjusted corresponding to the target time can be determined from the initial video based on the synchronization relationship. For example, based on the example in Example 1, it is assumed that the pronunciation time of "Welcome" in the English target text content "Welcome to our company." in the target audio starts from the 5.6th second and lasts to the 6.2nd second. This period of time is the target time corresponding to the target action (the action of nodding).

[0087] Since it is known that the English pronunciation in the target audio is slightly delayed, starting from the 5.6th second, we can select the video frame starting from the 5.6th second (or a time point slightly earlier to ensure the continuity of the action) of the target audio from the initial video as the starting point of the video frame to be adjusted, and the video frame until the time point matching the end of the English pronunciation "Welcome" (i.e., around the 6.2th second, but may need to be slightly adjusted to ensure the integrity of the action) as the end point of the video frame to be adjusted.

[0088] Step S1034: adjust the to-be-adjusted video frame according to the action information to obtain a target video frame.

[0089] Since the video frame to be adjusted is directly extracted from the initial video, the video frame to be adjusted may not completely match the target text content. For example, the action in the video frame to be adjusted may not match the target audio. Therefore, in this step, the video frame to be adjusted can be adjusted according to the action information to obtain a target video frame whose action matches the target text content.

[0090] For example, in the initial video, when a person says "Welcome to our company." in Chinese (around 5.5 seconds), he nods, and this action lasts until the 6th second, that is, the nodding action appears in the video frame corresponding to the pronunciation of Chinese in the initial video (starting from 5.5 seconds and lasting to 6 seconds).

[0091] After converting it into the English target text content "Welcome to our company.", if the pronunciation of the English "Welcome" starts from the 5.6th second and lasts to the 6.2th second, then the video frame to be adjusted is the video frame corresponding to the pronunciation of the English "Welcome" (starting from the 5.6th second and lasting to the 6.2th second) in the initial video. If the video frame to be adjusted is used directly, the nodding action will be slightly ahead of the start of the English pronunciation. Therefore, the video frame corresponding to the English pronunciation (starting from the 5.6th second and lasting to the 6.2th second) in the initial video (i.e., the video frame to be adjusted) can be adjusted according to the action information to obtain the target video frame so that the nodding action matches the target text content "Welcome to our company."

[0092] Step S1035: Obtain the target video according to the target video frame, the initial video and the target audio.

[0093] In this embodiment, the target video frame can be inserted into the corresponding position in the initial video according to the starting point and the ending point of the video frame to be adjusted in the initial video to replace the video frame to be adjusted in the initial video. While replacing the video frame to be adjusted, the video frame in the replaced initial video can be adjusted according to the actual situation of the target audio to obtain the target video, so as to ensure the synchronization between the target audio and the video frame in the target video.

[0094] In this embodiment, based on the target text content, a target audio is generated, a target time corresponding to the target action in the target audio is determined, and a time correspondence relationship between the target audio and the target action is established. On this basis, the video frame to be adjusted corresponding to the target time is determined from the initial video to locate the latest position of the target action, and the video frame to be adjusted is adjusted according to the action information to obtain a target video frame, so that the target video frame and the target action are semantically consistent. The target video is obtained based on the target video frame, the initial video and the target audio, so that the target text content and the target action in the target video are synchronized in time and semantically consistent, which can improve the accuracy and naturalness of video translation, and enhance the viewing experience and understanding of the audience.

[0095] For example, Figure 3 As shown in Figure 1, if in the initial video, a person is saying "Welcome to our company..." in Chinese, the sentence "Welcome to our company" starts at the 5th second of the audio and lasts until the 7th second. At the same time, when he says "Welcome" (about the 5.5th second), he nods (represented as target action A), and this action lasts until the 6th second.

[0096] Convert "Welcome to our company..." to English, and get the target text content Welcome to ourcompany...." If the pronunciation of English is longer than that of Chinese, "Welcome to our company." lasts from 5.1 seconds to 7.5 seconds, and the pronunciation time of "Welcome" in the target audio starts from 5.6 seconds and lasts to 6.2 seconds, the target time starts from 5.6 seconds and lasts to 6.2 seconds. The time correspondence between the target audio and the target action can be established as the audio time corresponding to the target action A starts from 5.6 seconds and lasts to 6.2 seconds.

[0097] According to the audio time corresponding to the target action A starting from 5.6 seconds and lasting to 6.2 seconds, the video frame corresponding to the audio time from 5.6 seconds to 6.2 seconds can be determined from the initial video as the latest position of the target action A, that is, the video frame A to be adjusted.

[0098] The video frame A to be adjusted is adjusted according to the action information corresponding to the target action A to obtain the target video frame A, and the target video frame A is semantically consistent with the target action A.

[0099] It should be noted that Figure 3 As a schematic diagram of the latest position of the target action A, it does not serve as a limitation for the target video frame and the video frame to be adjusted.

[0100] For example, Figure 4 As shown, if in the initial video, a person is saying in Chinese "Welcome to our company, today we will show the latest product. ...", the sentence "Welcome to our company" starts from the 5th second of the audio and lasts to the 7th second. At the same time, when he said "Welcome" (about 5.5 seconds), he nodded (represented as target action A), and this action lasted to the 6th second; the sentence "Today we will show the latest product" starts from the 7.5th second of the initial video and lasts to the 9.5th second, and when he said "show" (about 8th second), he raised his hand to point to the company's products (represented as target action B), and this action lasted to the 8.5th second.

[0101] Convert "Welcome to our company, today we will showcase our latest products. ..." to English, and get the target text content as "Today we will showcase our latest products. Welcome to ourcompany. ...". If the pronunciation of English is longer than that of Chinese, "Today we will showcase ourlatest products." starts from 5.1 seconds and lasts to 7.5 seconds, and the pronunciation time of "showcase" in the target audio starts from 6.0 seconds and lasts to 6.6 seconds. The target time starts from 6.0 seconds and lasts to 6.6 seconds. The time correspondence between the target audio and the target action can be established as the audio time corresponding to the target action B starts from 6.0 seconds and lasts to 6.6 seconds.

[0102] Since the audio time corresponding to the target action B starts from 6.0 seconds and lasts to 6.6 seconds, the video frame corresponding to the audio time from 6.0 seconds to 6.6 seconds can be determined from the initial video as the latest position of the target action B, that is, the video frame A to be adjusted.

[0103] The video frame A to be adjusted is adjusted according to the action information corresponding to the target action B to obtain the target video frame A, and the target video frame A is semantically consistent with the target action B.

[0104] In addition, "Welcome to our company." lasts from the 8th second to the 10.5th second. The pronunciation time of "Welcome" in the target audio starts from the 8th second and lasts to the 9.1th second. The target time starts from the 8th second and lasts to the 9.1th second. The time correspondence between the target audio and the target action can be established: the audio time corresponding to the target action A starts from the 8th second and lasts to the 9.1th second.

[0105] According to the audio time corresponding to the target action A starting from 8.5 seconds and lasting to 9.1 seconds, the video frame corresponding to the audio time from 8.5 seconds to 9.1 seconds can be determined from the initial video as the latest position of the target action A, that is, the video frame B to be adjusted.

[0106] The video frame B to be adjusted is adjusted according to the action information corresponding to the target action A to obtain the target video frame B, and the target video frame B is semantically consistent with the target action A.

[0107] It should be noted that Figure 4 As a schematic diagram of the latest positions of the target actions A and B, it does not serve as a limitation for the target video frame and the video frame to be adjusted.

[0108] As another optional embodiment of the present application, a processing method is provided for embodiment 3 of the present application. This embodiment is mainly an implementation method for adjusting the video frame to be adjusted according to the action information in step S1034 of embodiment 2. Adjusting the video frame to be adjusted according to the action information may include but is not limited to the following steps:

[0109] Step S10341: changing the action of the target image in the video frame to be adjusted according to the action information.

[0110] In this embodiment, the action information may include but is not limited to: the type, start time, duration, and action parameters (eg, amplitude and speed of the action).

[0111] In this embodiment, the action sequence of the target image in the video frame to be adjusted may be generated according to the type, start time, and duration of the action to obtain a new action of the target image.

[0112] In this embodiment, the action of the target image may be replaced by the new action of the target image.

[0113] Alternatively, the action of the target image in the video frame to be adjusted may be adjusted according to the action parameters.

[0114] In this embodiment, by adjusting the action of the target image in the video frame to be adjusted according to the action information, it can be ensured that the action in the target video is semantically consistent with the target text content (i.e., the target audio). This adjustment can correct the mismatch between the action and the audio caused by language conversion, thereby improving the accuracy and naturalness of the video translation.

[0115] As another optional embodiment of the present application, a processing method is provided for embodiment 4 of the present application. This embodiment is mainly an implementation method of step S1035 in embodiment 2. Step S1035 may include but is not limited to the following steps:

[0116] Step S10351: Generate the supplementary video frame according to the target video frame and the initial video.

[0117] Step S10351 may include but is not limited to the following steps:

[0118] Step S21, using the target video frame and information between adjacent video frames in the initial video to generate a new video frame through an interpolation algorithm (such as linear interpolation, bilinear interpolation or a more advanced image interpolation algorithm).

[0119] The generated new video frame can maintain a smooth transition with the target video frame at the pixel level.

[0120] Step S22: Perform motion smoothing processing on the new video frame so that the motion of the processed video frame remains consistent in time.

[0121] Step S23: adjust the color and brightness of the processed video frame to obtain a supplementary video frame.

[0122] The color and brightness of the supplementary video frame can be consistent with the target video frame, so as to avoid visual abruptness.

[0123] Step S10352: Replace the to-be-adjusted video frame with the target video frame, and replace the preceding and following video frames of the to-be-adjusted video frame with the supplementary video frame to obtain an updated video.

[0124] In this embodiment, the video frames before and after the video frame to be adjusted can be determined as needed, and are not limited in this application. For example, the video frames before and after the video frame to be adjusted may include: one or more video frames before the video frame to be adjusted, and one or more video frames after the video frame to be adjusted.

[0125] The supplementary video frame can make the target video frame and other frames in the updated video meet a coherent state.

[0126] Step S10353: Obtain the target video according to the updated video and the target audio.

[0127] In this embodiment, the update video and the target audio may be synchronously processed to obtain the target video.

[0128] In this embodiment, by generating a supplementary video frame and replacing the preceding and following video frames of the video frame to be adjusted, the visual abruptness caused by directly replacing the video frame to be adjusted can be reduced, so that the target video is synchronized in time with the target text content and the action corresponding to the target text content, and is more visually coherent on the basis of semantic consistency.

[0129] For example, Figure 5 As shown, if in the initial video, a person is saying in Chinese "Welcome to our company, today we will show the latest product. ...", the sentence "Welcome to our company" starts from the 5th second of the audio and lasts to the 7th second. At the same time, when he said "Welcome" (about 5.5 seconds), he nodded (represented as target action A), and this action lasted to the 6th second; the sentence "Today we will show the latest product" starts from the 7.5th second of the initial video and lasts to the 9.5th second, and when he said "show" (about 8th second), he raised his hand to point to the company's products (represented as target action B), and this action lasted to the 8.5th second.

[0130] Convert "Welcome to our company, today we will showcase our latest products. ..." to English, and get the target text content as "Today we will showcase our latest products. Welcome to ourcompany. ...". If the pronunciation of English is longer than that of Chinese, "Today we will showcase ourlatest products." starts from 5.1 seconds and lasts to 7.5 seconds, and the pronunciation time of "showcase" in the target audio starts from 6.0 seconds and lasts to 6.6 seconds. The target time starts from 6.0 seconds and lasts to 6.6 seconds. The time correspondence between the target audio and the target action can be established as the audio time corresponding to the target action B starts from 6.0 seconds and lasts to 6.6 seconds.

[0131] Since the audio time corresponding to the target action B starts from 6.0 seconds and lasts to 6.6 seconds, the video frame corresponding to the audio time from 6.0 seconds to 6.6 seconds can be determined from the initial video as the latest position of the target action B, that is, the video frame A to be adjusted.

[0132] The video frame A to be adjusted is adjusted according to the action information corresponding to the target action B to obtain the target video frame A, and the target video frame A is semantically consistent with the target action B.

[0133] The supplementary video frames A1 and A2 are generated according to the target video frame A and the initial video.

[0134] The target video frame A replaces the video frame A to be adjusted, and the supplementary video frames A1 and A2 replace the video frames before and after the video frame A to be adjusted to obtain an updated video. The supplementary video frames A1 and A2 can make the target video frame A and other frames in the updated video meet a coherent state.

[0135] In addition, "Welcome to our company." lasts from the 8th second to the 10.5th second. The pronunciation time of "Welcome" in the target audio starts from the 8th second and lasts to the 9.1th second. The target time starts from the 8th second and lasts to the 9.1th second. The time correspondence between the target audio and the target action can be established: the audio time corresponding to the target action A starts from the 8th second and lasts to the 9.1th second.

[0136] According to the audio time corresponding to the target action A starting from 8.5 seconds and lasting to 9.1 seconds, the video frame corresponding to the audio time from 8.5 seconds to 9.1 seconds can be determined from the initial video as the latest position of the target action A, that is, the video frame B to be adjusted.

[0137] The video frame B to be adjusted is adjusted according to the action information corresponding to the target action A to obtain the target video frame B, and the target video frame B is semantically consistent with the target action A.

[0138] The supplementary video frames B1 and B2 are generated according to the target video frame B and the initial video.

[0139] The target video frame B replaces the video frame B to be adjusted, and the supplementary video frames B1 and B2 replace the previous and next video frames of the video frame B to be adjusted to obtain an updated video. The supplementary video frames B1 and B2 can make the target video frame B and other frames in the updated video meet a coherent state.

[0140] As another optional embodiment of the present application, a processing method is provided in Embodiment 5 of the present application. This embodiment is mainly an implementation method for extracting action information from the initial video in Embodiment 1. Extracting action information from the initial video may include but is not limited to at least one of the following:

[0141] Step S31: If the audio of the initial video contains an emphatic tone, extract action information corresponding to the emphatic tone from the initial video.

[0142] In this embodiment, the audio of the initial video can be analyzed using audio processing technology to identify the start and end time of the emphatic tone (such as the increase in voice volume, the decrease in speech speed, the rise or fall in voice tone, etc.), and the action information corresponding to the emphatic tone can be identified based on the video frames within the start and end time of the emphatic tone.

[0143] In this embodiment, by associating the emphatic tone with the corresponding action information, it can be ensured that the emphatic tone can still be reflected through the action in the target video, thereby enhancing the audience's understanding and feelings.

[0144] Step S32: extracting action information corresponding to the emotion from the initial video according to the emotion expressed by the initial text content.

[0145] In this embodiment, natural language processing technology can be used to analyze the initial text content to determine the emotions expressed by the initial text content, such as happiness, sadness, anger, surprise, etc.

[0146] Based on the identified emotions, computer vision technology is used to extract action information corresponding to the emotions from the initial video.

[0147] In this embodiment, by associating emotions with corresponding action information, it can be ensured that the target video can accurately convey the emotions in the original video, thereby improving the viewing experience of the audience.

[0148] Step S33: identifying keywords in the initial text content, and extracting action information corresponding to the keywords from the initial video.

[0149] In this embodiment, natural language processing technology can be used to analyze the initial text content and identify keywords therein. Keywords can be nouns, verbs, or adjectives with specific semantics and contexts.

[0150] In this embodiment, the action information corresponding to the keyword can be extracted from the initial video using computer vision technology according to the keyword. For example, if the keyword is "welcome", the pronunciation time period of the keyword "welcome" in the initial video can be determined, and the action information in the pronunciation time period can be extracted.

[0151] In this embodiment, by associating keywords with corresponding action information, it can be ensured that the target video can accurately convey the key information in the initial video, thereby improving the accuracy and comprehensibility of the target video.

[0152] As another optional embodiment of the present application, refer to Figure 6 , is a flow chart of a processing method provided in Example 6 of the present application. This example is mainly an implementation method of step S103 in Example 1. Figure 6 As shown, step S103 may include but is not limited to the following steps:

[0153] Step S1036: Generate target audio based on the target text content and the action information.

[0154] In this embodiment, the target audio is not only the voiced version of the target text content, but more importantly, the expression attribute of the target audio can match the target action represented by the action information.

[0155] The expression attribute of the target audio may include but is not limited to: at least one of the speech rate, rhythm and intonation of the target audio. Accordingly, the expression attribute of the target audio may match the target action represented by the action information, which can be understood as: the speech rhythm, intonation change, speech rate, etc. of the target text content in the target audio are consistent with the target action in the initial video in time.

[0156] For example, if in the initial video, a virtual person is saying in Chinese: "Welcome to our company. ...". When he says "Welcome", he nods. After converting the initial text content "Welcome to our company. ..." into the target text content "Welcome to our company. ...", the target audio can be generated based on the target text content and action information. The "Welcome" part in the target audio can be slightly accelerated and the tone slightly raised to match the nodding action.

[0157] Step S1037: synthesize the target audio and the video frames in the initial video to obtain a target video.

[0158] In this embodiment, the synthesis processing of the target audio and the video frame in the initial video may include: time aligning the target audio with the video frame of the initial video to ensure that the playback progress of the audio is consistent with the playback progress of the video frame.

[0159] On the basis of time alignment, the target audio and the video frames in the initial video are synchronously processed to obtain the target video.

[0160] In this embodiment, the target audio and the video frames in the initial video are synthesized based on the consistency of the speech rhythm, intonation, speech speed, etc. in the target audio with the target action in the initial video in time, so that the playback time of the target text content in the target audio in the target video and the time of the target action in the target video are synchronized. This synchronization can enhance the coherence of the target video and improve the viewing experience of the user.

[0161] As another optional embodiment of the present application, refer to Figure 7 , is a flow chart of a processing method provided in Example 7 of the present application. This example is mainly an implementation method of step S101 and step S1036 in Example 6. Figure 7 As shown, step S101 may include but is not limited to the following steps:

[0162] Step S1011, extracting initial text content and action feature sequences from an initial video; the action feature sequences represent actions appearing in each video frame in the initial video.

[0163] Step S1036 may include but is not limited to:

[0164] Step S10361: Process the target text content and the action feature sequence based on the audio generation model to generate target audio.

[0165] In this embodiment, the audio generation model can adjust the expression properties (such as rhythm, speaking speed, etc.) of the speech corresponding to the target text content based on the action feature sequence to generate the target audio.

[0166] In this embodiment, the timbre information of the target person can also be used as the input of the audio generation model. The timbre information of the target person, the target text content and the action feature sequence are processed based on the audio generation model to generate target audio, so that the generated target audio will be closer to the original voice of the target person.

[0167] Reference Figure 8 , the audio generation model can be trained in the following ways:

[0168] Step S41: Obtain video samples, and extract audio samples from the video samples.

[0169] Step S42: using a text conversion module to extract initial text sample content from the audio sample, and performing language conversion on the initial text sample content to obtain target text sample content.

[0170] Of course, after the initial text sample content is extracted from the audio sample, the initial text sample content may be input into the text conversion module. Accordingly, the text conversion module only needs to perform language conversion on the initial text sample content.

[0171] Step S43: extracting a sequence of motion feature samples from the video samples using a motion feature extraction module.

[0172] The action feature sample sequence can represent the actions appearing in each video frame in the video sample.

[0173] Step S44: input the target text sample content and the action feature sample sequence into the audio generation model to obtain the target sample audio generated by the audio generation model.

[0174] Step S45, fixing the parameters of the text conversion module, and adjusting the parameters of the motion feature extraction module and the audio generation model based on the loss.

[0175] The loss can characterize the difference between the target sample audio and the real audio. The real audio can be extracted from a real video, the real audio matches the action represented by the action information extracted from the real video, and is in the same language as the target text sample content (but it does not need to have the same semantics, as long as it is in the same language).

[0176] Based on the loss, adjusting the parameters of the action feature extraction module and the audio generation model can enable the audio generation model to learn the association between the action features and the expressive properties of the audio.

[0177] In this embodiment, the target text content and the action feature sequence are processed based on the audio generation model to generate the target audio, ensuring that the speech rhythm, intonation change, speech speed, etc. in the target audio are consistent with the target action in the initial video in time. On this basis, the target audio and the video frame in the initial video are synthesized to ensure that the playback time of the target text content in the target audio in the target video is synchronized with the time of the target action in the target video. This synchronization can enhance the coherence of the target video and improve the user's viewing experience.

[0178] In this embodiment, during the training process, reference timbre information can also be obtained, such as Fig. 9 As shown, step S44 may include but is not limited to: inputting reference timbre information, target text sample content and action feature sample sequence into the audio generation model to obtain target sample audio generated by the audio generation model.

[0179] Accordingly, based on the loss, adjusting the parameters of the action feature extraction module and the audio generation model can enable the audio generation model to learn the association between the action features and the expressive properties of the audio, and generate audio according to the reference timbre information.

[0180] In this embodiment, if Fig.10 As shown, the synthesis process of the target audio and the video frame in the initial video may include but is not limited to:

[0181] The lip shape editing module is used to perform lip shape editing on the video frames in the initial video based on the target audio to obtain the target video.

[0182] In this embodiment, the lip shape editing module is used to perform lip shape editing on the video frames in the initial video based on the target audio to obtain the target video. This can ensure the multi-faceted matching of voice, lip shape and action in the target video, enhance the coherence of the target video, and improve the user's viewing experience.

[0183] As another optional embodiment of the present application, a processing method is provided for Embodiment 8 of the present application. This embodiment is mainly an implementation method of performing language conversion on the initial text content in the aforementioned embodiments to obtain the target text content. Performing language conversion on the initial text content to obtain the target text content may include but is not limited to the following steps:

[0184] Step S1021 : inferring the initial text content based on the target model to obtain target text content, wherein the target text content and the initial text content are in different languages.

[0185] In this embodiment, the target model may be constructed based on machine learning, deep learning or other artificial intelligence algorithms. For example, the target model may include a language model with natural language processing capabilities and deep semantic understanding capabilities.

[0186] In this embodiment, automatic translation between different languages ​​can be achieved by reasoning with the target model. In addition, since the target model usually has deep semantic understanding capabilities, it can better understand the deep meaning of the original text content. Therefore, the target text content is closer to the original intention, which helps to reduce information loss and misunderstanding during the translation process.

[0187] Next, the processing device provided in the present application is introduced. The processing device introduced below and the processing method introduced above can be referenced to each other.

[0188] Reference Fig.11 The processing device includes: an extraction module 100, a conversion module 200 and a processing module 300.

[0189] The extraction module 100 is used to extract initial text content and action information from the initial video; the action information represents the target action associated with the initial text content in the initial video and the time when the target action appears in the initial video.

[0190] The conversion module 200 is used to perform language conversion on the initial text content to obtain target text content.

[0191] The processing module 300 is used to obtain a target video based on the target text content and the action information; the target text content in the target video is matched with the action corresponding to the target text content.

[0192] In this embodiment, the processing module 300 may be specifically used for:

[0193] Based on the target text content, generate target audio;

[0194] Determining a target time in the target audio corresponding to the target action;

[0195] Determine the video frame to be adjusted corresponding to the target time from the initial video;

[0196] Adjust the to-be-adjusted video frame according to the action information to obtain a target video frame;

[0197] The target video is obtained according to the target video frame, the initial video and the target audio.

[0198] The processing module 300 adjusts the to-be-adjusted video frame according to the action information, which may specifically include:

[0199] The action of the target image in the video frame to be adjusted is changed according to the action information.

[0200] The processing module 300 obtains the target video according to the target video frame, the initial video and the target audio, which may specifically include:

[0201] generating the supplementary video frame according to the target video frame and the initial video;

[0202] The target video frame is used to replace the video frame to be adjusted, and the previous and next video frames of the video frame to be adjusted are replaced by the supplementary video frame to obtain an updated video, wherein the supplementary video frame can make the target video frame and other frames in the updated video meet a coherent state;

[0203] The target video is obtained according to the updated video and the target audio.

[0204] The extraction module 100 extracts the action information from the initial video, which may include at least one of the following:

[0205] If the audio of the initial video contains an emphatic tone, extracting action information corresponding to the emphatic tone from the initial video;

[0206] According to the emotion expressed by the initial text content, extracting action information corresponding to the emotion from the initial video;

[0207] Keywords in the initial text content are identified, and action information corresponding to the keywords is extracted from the initial video.

[0208] The processing module 300 may be specifically used for:

[0209] Based on the target text content and the action information, a target audio is generated; the expression attribute of the target audio matches the target action represented by the action information;

[0210] The target audio and the video frames in the initial video are synthesized to obtain the target video.

[0211] The action information may include: an action feature sequence; the action feature sequence represents the action appearing in each video frame in the initial video;

[0212] The processing module 300 generates a target audio based on the target text content and the action information, which may specifically include:

[0213] The target text content and the action feature sequence are processed based on an audio generation model to generate target audio.

[0214] The conversion module 200 may be specifically used for:

[0215] The initial text content is inferred based on the target model to obtain target text content, where the target text content and the initial text content are in different languages.

[0216] In another embodiment of the present application, an electronic device is provided, including:

[0217] The memory is used to store computer programs;

[0218] The processor is used to execute the computer program so that the electronic device can implement the processing method introduced in any one of embodiments 1-8.

[0219] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0220] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0221] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0222] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A processing method comprising: Extracting initial text content and action information from the initial video; The action information represents a target action associated with the initial text content in the initial video and a time when the target action appears in the initial video; Performing language conversion on the initial text content to obtain target text content; Based on the target text content and the action information, obtaining a target video; The target text content in the target video matches an action corresponding to the target text content.

2. The processing method according to claim 1, obtaining a target video based on the target text content and the action information, comprising: Based on the target text content, generate target audio; Determining a target time in the target audio corresponding to the target action; Determine the video frame to be adjusted corresponding to the target time from the initial video; Adjust the to-be-adjusted video frame according to the action information to obtain a target video frame; The target video is obtained according to the target video frame, the initial video and the target audio.

3. The method according to claim 2, wherein adjusting the video frame to be adjusted according to the action information comprises: The action of the target image in the video frame to be adjusted is changed according to the action information.

4. The method according to claim 2, wherein obtaining the target video according to the target video frame, the initial video and the target audio comprises: generating the supplementary video frame according to the target video frame and the initial video; The target video frame is used to replace the video frame to be adjusted, and the previous and next video frames of the video frame to be adjusted are replaced by the supplementary video frame to obtain an updated video, wherein the supplementary video frame can make the target video frame and other frames in the updated video meet a coherent state; The target video is obtained according to the updated video and the target audio.

5. The processing method according to claim 1, extracting motion information from the initial video, comprising at least one of the following: If the audio of the initial video contains an emphatic tone, extracting action information corresponding to the emphatic tone from the initial video; Extracting action information corresponding to the emotion from the initial video according to the emotion expressed by the initial text content; Keywords in the initial text content are identified, and action information corresponding to the keywords is extracted from the initial video.

6. The processing method according to claim 1, obtaining a target video based on the target text content and the action information, comprising: Generate target audio based on the target text content and the action information; The expression attribute of the target audio matches the target action represented by the action information; The target audio and the video frames in the initial video are synthesized to obtain the target video.

7. The processing method according to claim 6, wherein the action information comprises: Action feature sequence; Said The action feature sequence represents the action appearing in each video frame in the initial video; Generating a target audio based on the target text content and the action information, including: The target text content and the action feature sequence are processed based on an audio generation model to generate target audio.

8. According to the processing method according to any one of claims 1 to 7, performing language conversion on the initial text content to obtain the target text content comprises: The initial text content is inferred based on the target model to obtain target text content, where the target text content and the initial text content are in different languages.

9. A processing device comprising: An extraction module, for extracting initial text content and action information from the initial video; The action information represents a target action associated with the initial text content in the initial video and a time when the target action appears in the initial video; A conversion module, used for performing language conversion on the initial text content to obtain target text content; A processing module, used for obtaining a target video based on the target text content and the action information; The target text content in the target video matches an action corresponding to the target text content.

10. An electronic device comprising: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the processing method as described in any one of claims 1 to 8.

Images