Video generation method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention discloses a video generation method and device, electronic equipment and a computer readable medium. The method comprises the following steps: determining a target role in a first video played by a client; when a session message sent by the client is received, generating a reply text for the session message based on the role information of the target role; based on a pre-trained voice synthesis model, converting the reply text into a first voice matched with the tone of the target role, and based on the reply text and a pre-trained video generation model, generating an image video of the target role; and generating a second video based on the first voice and the image video, and returning the second video to the client, so that the client plays the second video. Therefore, the user can realize high-fidelity interaction with the movie and television role in the movie watching process, and the immersion of the interaction process of the user and the movie and television role is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a video generation method, device, electronic device, and computer-readable medium. Background Art

[0002] With the development of Internet technology, users can watch film and television content through various video applications. When users watch film and television content, when they encounter fascinating plots and characters, they often feel that their emotions need to be released and communicated in a timely manner.

[0003] In the prior art, the interaction between the user and the character can be simulated through simple character dialogue. However, this interaction method lacks realism and limits the user's immersion. Summary of the invention

[0004] The embodiments of the present application propose a video generation method, device, electronic device and computer-readable medium to solve the technical problem in the prior art that the interaction between the user and the character in the video lacks realism.

[0005] In a first aspect, an embodiment of the present application provides a video generation method, the method comprising: determining a target character in a first video played by a client; upon receiving a conversation message sent by the client, generating a reply text for the conversation message based on the character information of the target character; based on a pre-trained speech synthesis model, converting the reply text into a first voice matching the timbre of the target character, and generating an image video of the target character based on the reply text and a pre-trained video generation model; generating a second video based on the first voice and the image video, and returning the second video to the client so that the client plays the second video.

[0006] In a second aspect, an embodiment of the present application provides a video generating device, which includes: a determination unit, used to determine a target character in a first video played by a client; a first generating unit, used to generate a reply text for the conversation message based on the character information of the target character when receiving a conversation message sent by the client; a second generating unit, used to convert the reply text into a first voice matching the timbre of the target character based on a pre-trained speech synthesis model, and generate an image video of the target character based on the reply text and a pre-trained video generation model; a third generating unit, used to generate a second video based on the first voice and the image video, and return the second video to the client so that the client plays the second video.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any embodiment of the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the first aspect.

[0009] The video generation method, device, electronic device and computer-readable medium provided by the embodiment of the present application first determine the target character in the first video played by the client, and when receiving the conversation message sent by the client, generate a reply text for the conversation message based on the character information of the target character, and then convert the reply text into a first voice matching the timbre of the target character based on the pre-trained speech synthesis model, and generate an image video of the target character based on the reply text and the pre-trained video generation model, and finally generate and play the second video based on the first voice and the image video. Thus, after the user inputs the conversation message, the reply content in the form of a video can be output with the portrait and timbre of the target character, so that the user can achieve high-fidelity interaction with the film and television characters during the viewing process, and improve the immersion of the user in the interaction process with the film and television characters. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0011] Figure 1 is a flow chart of an embodiment of the video generation method of the present application;

[0012] Figure 2A It is a schematic diagram of an application scenario of the video generation method of the present application;

[0013] Figure 2B It is a schematic diagram of an application scenario of the video generation method of the present application;

[0014] Figure 3 is a structural schematic diagram of an embodiment of a video generating device of the present application;

[0015] Figure 4 It is a schematic diagram of the structure of an electronic device used to implement an embodiment of the present application. DETAILED DESCRIPTION

[0016] All actions to obtain signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0017] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0018] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] Please refer to Figure 1 , which shows a process 100 of an embodiment of a video generation method according to the present application. The video generation method can be applied to various electronic devices with data processing functions, such as servers, etc. The execution subject of the video generation method can be a processor in the above electronic device.

[0020] The video generation method comprises the following steps:

[0021] Step 101, determining a target character in a first video played by a client.

[0022] In this embodiment, the first video may be any video watched by the user. The type of the video is not specifically limited. For example, it may include but is not limited to film and television dramas, variety show videos, news videos, live videos, short videos, etc. The target character may be a character selected by the user in the first video, or a character in a screen selected by the user.

[0023] In some optional implementations of this embodiment, during the process of playing the first video on the client, if the client detects that the first video is paused, the video screen when the first video is paused can be extracted. Then, the target role can be determined from the roles included in the video screen. Among them, if the video screen includes at least two roles, the target role can be selected in a variety of ways. For example, according to the importance of the role, the most important role can be selected as the target role; according to the proportion of the role area in the video screen, the role with the largest proportion can be selected as the target role; the role focused on by the camera in the video screen can also be used as the target role; and the user can also select the target role.

[0024] Step 102: upon receiving the conversation message sent by the client, a reply text for the conversation message is generated based on the role information of the target role.

[0025] In this embodiment, the playback interface of the first video may have an entrance to a conversation interface. The user can click on the entrance to call out the conversation interface, and enter a conversation message in the conversation interface, which is then sent by the client to the server. The conversation message may be a text message, a voice message, a video message, or other message, which is not specifically limited in this embodiment of the application.

[0026] In this embodiment, when a conversation message sent by a client is received, a reply text for the conversation message can be generated based on the role information of the target role. The role information of the target role may include but is not limited to the name, age, personality characteristics, hobbies, occupation, common language, speaking style characteristics, etc. of the target role. In practice, a large language model (LLM) can be used to generate a reply text for the above conversation message.

[0027] In practice, the large language model is a deep learning model trained based on massive text data. It can not only generate natural language text, but also deeply understand the meaning of the text and handle various natural language tasks, such as text summarization, question answering, translation, etc. In practical applications, the above-mentioned large language model can be trained based on an existing language model, or it can be trained using a self-developed language model, which is not specifically limited here. As an example, the large language model can adopt models such as ChatGPT (ChatGenerative Pre-trained Transformer).

[0028] In some optional implementations of this embodiment, the conversation message may be input by voice. In this case, the conversation message may include voice (which may be recorded as the second voice). The above-mentioned step of generating a reply text for the conversation message based on the role information of the target role may include the following steps:

[0029] The first step is to obtain the role information of the target role, the plot information of the first video, and the historical conversation content between the user and the target role. The historical conversation content here may refer to the historical conversation content after being converted into text. The plot information of the first video may include but is not limited to the plot introduction, character relationship introduction and other information.

[0030] The second step is to convert the second speech into text to obtain the target text. Here, a speech-to-text tool or program can be used to convert the second speech into text.

[0031] The third step is to input the character information, plot information, target text and historical conversation content into the pre-trained large language model to obtain a reply text for the second voice. Here, a model input content template may be pre-stored, and the character information, plot information, target text and historical conversation content may be substituted into the model input content template to obtain model input information for the large language model. Then, the model input information is input into the large language model, and the reply text for the second voice is calculated by the large language model.

[0032] Therefore, the embodiments of the present application can be applied to scenarios where users input conversation messages through voice. In this scenario, after the user inputs the conversation content through voice, the reply content to the conversation content is output in the form of video with the portrait and voice of the target character, which can further enable the user to achieve highly realistic interaction with the film and television characters during the movie watching process, further improving the immersion of the user in the interaction process with the film and television characters.

[0033] Step 103, based on the pre-trained speech synthesis model, convert the reply text into a first voice that matches the timbre of the target character, and generate an image video of the target character based on the reply text.

[0034] In this embodiment, the reply text can be converted into a first voice by means of TTS (Text To Speech) technology. TTS technology is a technology that intelligently converts text into a natural voice stream through the design of a neural network. In view of this, the embodiment of the present application can convert the reply text into a first voice based on a pre-trained speech synthesis model with a text-to-speech function.

[0035] In practice, the speech synthesis model can use the ChatTTS (Chat Text To Speech) model, the GPT-SoVITS (Generative Pre-Training for Speaker Voice Imitation and Text-to-Speech Synthesis) model, and the like.

[0036] In this embodiment, the image video may include an image area of ​​the target character, which may include but is not limited to a facial area. The lip shape and facial muscle movements of the target character in the image video may be matched with the pronunciation of each word in the reply text in turn to improve the authenticity of the image video.

[0037] In some optional implementations of the present embodiment, the above-mentioned conversion of the reply text into the first voice matching the timbre of the target character based on the pre-trained speech synthesis model may include: extracting a third voice of the target character from the first video. Inputting the third voice and the reply text into the pre-trained speech synthesis model to obtain a second voice matching the timbre of the target character. The above-mentioned third voice may be one or more lines of the target character. The above-mentioned speech synthesis model may first learn the timbre features of the target character in the third voice, and then convert the reply text into the second voice matching the timbre features.

[0038] In some optional implementations of this embodiment, the speech synthesis model includes a speech feature generation module and a speech synthesis module. The speech feature generation module can be used to convert text into speech features, and the speech synthesis module can be used to convert speech features into speech. Taking the GPT-SoVITS model as an example, the speech feature generation module can be the SoVITS module in the GPT-SoVITS model, which can learn the relationship between text and speech features. The speech synthesis module can be the GPT module in the GPT-SoVITS model, which can learn the relationship between speech features and speech.

[0039] On this basis, after the third speech and the reply text are input into the speech synthesis model, the speech synthesis model can generate the second speech through the following steps:

[0040] In the first step, the speech synthesis module is trained based on the third speech to obtain an updated speech synthesis module. Here, the third speech can be used as a sample. By training the speech synthesis module with the third speech, the speech synthesis module can learn the timbre of the target character, so that the speech of the timbre can be output.

[0041] The second step is to pre-process the reply text to obtain a processed text. The pre-processing may include but is not limited to adding necessary tag information and other processing methods.

[0042] The third step is to convert the processed text into speech features through a speech feature generation module. Here, the processed text can be input into a speech feature generation module, such as the SoVITS module in the GPT-SoVITS model, and the speech feature generation module performs text-to-speech conversion on the processed text to obtain speech features corresponding to the processed text. The speech features can be features in the form of sequences.

[0043] Step 4: Convert the speech features into the second speech through the updated speech synthesis module. Here, the speech features output by the speech feature generation module can be input into a speech synthesis module, such as the GPT module in the GPT-SoVITS model, and the speech features can be converted into the second speech through the speech synthesis module.

[0044] It should be noted that after the second speech is generated, the second speech may be post-processed, for example, noise is eliminated, details are added, etc., to improve the quality of the second speech.

[0045] In this way, the reply text can be converted into a voice that matches the timbre of the target character. Since the speech synthesis model can learn the timbre of the target character online through the input third voice, it can quickly and flexibly generate the second voice that matches the timbre of the target character, thereby improving the generation efficiency of the second voice.

[0046] In some optional implementations of this embodiment, the generation of the target character's image video based on the reply text and the pre-trained video generation model may include: first, extracting a key frame of the first video, the key frame includes the target character, the target character in the key frame includes a facial region, and the facial region includes facial features. Then, inputting the reply text and the key frame into the pre-trained video generation model to obtain the target character's image video.

[0047] The key frame may be a frame in which a key action is located during the movement of the target character in the first video. In addition, the key frame may also be selected by the user, for example, it may be a video screen when the user pauses the first video. The video generation model may be any Vincent video model, for example, it may include but is not limited to the Pika model, the Sora model, etc. The Vincent video model may generate a video that matches the text.

[0048] In some optional implementations of the present embodiment, the video generation model includes a speech conversion module, a speech encoder, an image encoder and a video generation module. The speech conversion module can convert text into speech through natural language processing technology. The speech encoder can be used to encode the speech to obtain facial expression features and lip shape features corresponding to each syllable in the speech. The image encoder can be used to extract portrait features in the image. The video generation module can be used to synthesize a video based on the facial expression features, lip shape features and portrait features.

[0049] On this basis, after inputting the reply text and key frames into the pre-trained video generation model, the video generation model can generate the image video through the following steps:

[0050] The first step is to convert the reply text into the fourth voice through the voice conversion module.

[0051] In the second step, the fourth speech is input into a speech encoder, and the speech encoder converts the fourth speech into facial expression features and lip shape features. Here, syllables can be used as units, and each syllable corresponds to a set of facial expression features and lip shape features, thereby obtaining a facial expression feature sequence and a lip shape feature sequence.

[0052] In the third step, the key frame is input into the image encoder, and the image encoder is used to extract the portrait features of the target character. The portrait features may include but are not limited to facial features. The facial features may include but are not limited to the position coordinates of multiple key points on the face.

[0053] In the fourth step, the facial expression features, mouth shape features and portrait features are input into a video generation module to obtain an image video. The image video may include but is not limited to a facial expression video.

[0054] Through the above method, an image video of the target character matching the reply text can be generated, and the lip shape and facial muscle movements of the target character in the image video can be matched with the pronunciation of each word in the reply text in sequence, thereby improving the authenticity of the image video.

[0055] Step 104, generating a second video based on the first voice and the image video, and returning the second video to the client so that the client plays the second video.

[0056] In this embodiment, the first voice and the image video can be synthesized by means of a video editing tool to generate a second video. After the second video is generated, the second video can be returned to the client so that the client plays the second video. Therefore, when the user enters a conversation message during the movie watching process, the image video of the target character replying to the conversation message during the movie watching process can be obtained, so that the user can have a dialogue with the movie character, realize a highly realistic interaction with the movie character during the movie watching process, and improve the immersion of the user in the interaction process with the movie character.

[0057] As an example, see Figure 2A and Figure 2B The user paused the TV show and entered a voice message in the input box. Figure 2A As shown in the reference numeral 201 in FIG. 1 . After receiving the voice input by the user, the system can recognize the character (eg Figure 2A ), and take the character as the target character, generate a reply text for the conversation message based on the character information of the target character, and then convert the reply text into a voice that matches the timbre of the target character, and generate an image video of the target character based on the reply text, and finally synthesize the two, return and play a video of the target character replying to the voice input by the user to the user, see Figure 2B As shown in the reference numeral 202.

[0058] The method provided by the above-mentioned embodiment of the present application first determines the target character in the first video played by the client, and when receiving the conversation message sent by the client, generates a reply text for the conversation message based on the character information of the target character, and then converts the reply text into a first voice matching the timbre of the target character based on the pre-trained speech synthesis model, and generates an image video of the target character based on the reply text and the pre-trained video generation model, and finally generates and plays the second video based on the first voice and the image video. Thus, after the user inputs the conversation message, the reply content in the form of a video can be output with the portrait and timbre of the target character, so that the user can achieve high-fidelity interaction with the film and television characters during the viewing process, which improves the immersion of the user in the interaction process with the film and television characters.

[0059] Further references Figure 3 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a video generation device, and the device embodiment is Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0060] like Figure 3 As shown, the video generating device 300 of this embodiment includes: a determining unit 301, used to determine the target character in the first video played by the client; a first generating unit 302, used to generate a reply text for the conversation message based on the character information of the target character when receiving the conversation message sent by the client; a second generating unit 303, used to convert the reply text into a first voice matching the timbre of the target character based on a pre-trained speech synthesis model, and generate an image video of the target character based on the reply text and a pre-trained video generation model; a third generating unit 304, used to generate a second video based on the first voice and the image video, and return the second video to the client so that the client plays the second video.

[0061] In some optional implementations of this embodiment, the determination unit 301 is also used for: during the process of the client playing the first video, if it is detected that the client pauses the first video, extracting the video picture when the first video is paused; and determining the target character from the characters contained in the video picture.

[0062] In some optional implementations of this embodiment, the second generation unit 303 is also used to: extract a third voice of the target character from the first video; input the third voice and the reply text into a pre-trained speech synthesis model to obtain a second voice that matches the timbre of the target character.

[0063] In some optional implementations of the present embodiment, the speech synthesis model includes a speech feature generation module and a speech synthesis module; the speech synthesis model generates the second speech through the following steps: based on the third speech, the speech synthesis module is trained to obtain an updated speech synthesis module; the reply text is preprocessed to obtain a processed text; the processed text is converted into speech features through the speech feature generation module; and the speech features are converted into the second speech through the updated speech synthesis module.

[0064] In some optional implementations of the present embodiment, the second generation unit 303 is further used to: extract key frames of the first video, the key frames including the target character, the target character in the key frames including a facial area, and the facial area including facial features; input the reply text and the key frames into a pre-trained video generation model to obtain an image video of the target character.

[0065] In some optional implementations of this embodiment, the video generation model includes a speech conversion module, a speech encoder, an image encoder and a video generation module; the video generation model generates the image video through the following steps: converting the reply text into a fourth speech through the speech conversion module; inputting the fourth speech into the speech encoder, and converting the fourth speech into facial expression features and lip shape features through the speech encoder; inputting the key frames into the image encoder, and extracting the portrait features of the target character through the image encoder; inputting the facial expression features, the lip shape features and the portrait features into the video generation module to obtain the image video.

[0066] In some optional implementations of this embodiment, the conversation message includes a second voice; the first generation unit 302 is further used to: obtain role information of the target role, plot information of the first video, and historical conversation content between the user and the target role; convert the second voice into text to obtain a target text; input the role information, the plot information, the target text and the historical conversation content into a pre-trained large language model to obtain a reply text for the second voice.

[0067] The device provided by the above-mentioned embodiment of the present application first determines the target character in the first video played by the client, and when receiving the conversation message sent by the client, generates a reply text for the conversation message based on the character information of the target character, and then converts the reply text into a first voice matching the timbre of the target character based on the pre-trained speech synthesis model, and generates an image video of the target character based on the reply text and the pre-trained video generation model, and finally generates and plays the second video based on the first voice and the image video. Thus, after the user inputs the conversation message, the reply content in the form of a video can be output with the portrait and timbre of the target character, so that the user can achieve high-fidelity interaction with the film and television characters during the viewing process, which improves the immersion of the user in the interaction process with the film and television characters.

[0068] Reference below Figure 4 , which shows a schematic structural diagram of an electronic device for implementing some embodiments of the present application. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0069] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0070] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a disk, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0071] In particular, according to some embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present application are executed.

[0072] It should be noted that the computer-readable medium described in some embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0073] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0074] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: determines the target character in the first video played by the client; upon receiving the conversation message sent by the client, generates a reply text for the conversation message based on the character information of the target character; based on the pre-trained speech synthesis model, converts the reply text into a first voice matching the timbre of the target character, and generates an image video of the target character based on the reply text and the pre-trained video generation model; generates a second video based on the first voice and the image video, and returns the second video to the client so that the client plays the second video. This enables users to achieve highly realistic interactions with film and television characters during the viewing process, improving the immersion of users in the process of interacting with film and television characters.

[0075] Computer program code for performing the operations of some embodiments of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++; and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, or may be connected to an external computer (e.g., through the Internet using an Internet service provider), including a local area network (LAN) or a wide area network (WAN).

[0076] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0077] The units described in some embodiments of the present application may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a first determination unit, a second determination unit, a selection unit, and a third determination unit. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0078] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0079] The above description is only some preferred embodiments of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present application is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with (but not limited to) the technical features with similar functions disclosed in the embodiments of the present application to form a technical solution.

Claims

1. A video generation method, characterized in that: The method comprises: Determine a target character in a first video played by a client; When receiving a conversation message sent by the client, generating a reply text for the conversation message based on the role information of the target role; Based on a pre-trained speech synthesis model, convert the reply text into a first speech that matches the timbre of the target character, and based on the reply text and a pre-trained video generation model, generate an image video of the target character; Based on the first voice and the image video, a second video is generated, and the second video is returned to the client so that the client plays the second video.

2. The method according to claim 1, characterized in that The step of determining a target character in a first video played by a client includes: During the process of playing the first video by the client, if it is detected that the client performs a pause operation on the first video, extracting the video picture when the first video is paused; A target character is determined from the characters included in the video image.

3. The method according to claim 1, characterized in that The step of converting the reply text into a first voice matching the timbre of the target character based on a pre-trained speech synthesis model comprises: Extracting a third voice of the target character from the first video; The third voice and the reply text are input into a pre-trained speech synthesis model to obtain a second voice that matches the timbre of the target character.

4. The method according to claim 3, characterized in that The speech synthesis model includes a speech feature generation module and a speech synthesis module; the speech synthesis model generates the second speech through the following steps: Based on the third speech, training the speech synthesis module to obtain an updated speech synthesis module; Preprocessing the reply text to obtain a processed text; The processed text is converted into speech features by the speech feature generation module; The updated speech synthesis module converts the speech feature into the second speech.

5. The method according to claim 1, characterized in that The step of generating the image video of the target character based on the reply text and the pre-trained video generation model comprises: Extracting a key frame of the first video, wherein the key frame includes the target character, the target character in the key frame includes a facial area, and the facial area includes facial features; The reply text and the key frames are input into a pre-trained video generation model to obtain an image video of the target character.

6. The method according to claim 5, characterized in that The video generation model includes a speech conversion module, a speech encoder, an image encoder and a video generation module; the video generation model generates the image video through the following steps: The reply text is converted into a fourth voice by the voice conversion module; Inputting the fourth speech into the speech encoder, and converting the fourth speech into facial expression features and lip shape features through the speech encoder; Inputting the key frame into the image encoder, and extracting the portrait features of the target character through the image encoder; The facial expression features, the lip shape features and the portrait features are input into the video generation module to obtain the image video.

7. The method according to claim 1, characterized in that The conversation message includes a second voice; and generating a reply text for the conversation message based on the role information of the target role includes: Acquire the role information of the target role, the plot information of the first video, and the historical conversation content between the user and the target role; Converting the second speech into text to obtain a target text; The character information, the plot information, the target text and the historical conversation content are input into a pre-trained large language model to obtain a response text for the second voice.

8. A video generating device, characterized in that: The device comprises: A determination unit, used to determine a target character in a first video played by a client; A first generating unit, configured to generate a reply text for the conversation message based on the role information of the target role when receiving the conversation message sent by the client; A second generation unit is used to convert the reply text into a first voice matching the timbre of the target character based on a pre-trained speech synthesis model, and generate an image video of the target character based on the reply text and a pre-trained video generation model; The third generating unit is used to generate a second video based on the first voice and the image video, and return the second video to the client so that the client plays the second video.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images