Video generation methods, apparatus, equipment and media
By acquiring visual and configuration data and using machine learning models to generate videos, the inefficiency of existing technologies is solved, and the matching of lip movements with speech is achieved, thus improving the efficiency and accuracy of video generation.
Patent Information
- Application Number
- CN202510630545.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies require users to perform numerous manual operations when generating videos, resulting in low efficiency and high costs, as well as the problem of mismatch between the lip movements of objects in the video and their speech.
By acquiring visual and configuration data, and using machine learning models to generate videos, the shape of an object's mouth is matched with the speech, thus automatically generating videos.
It improves the efficiency and accuracy of video generation, avoids mismatches between the subject's lip movements and speech, and simplifies the operation process.
Smart Images

Figure CN123123652A_ABST
Abstract
Description
Technical Field
[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, electronic devices, computer-readable storage media, and computer program products for generating video from visual data. Background Technology
[0002] With the rapid development of computer technology, more and more applications and platforms are designed to provide users with various services. For example, users can edit, create, and publish videos of various content within applications. Videos can be generated based on a variety of visual content, such as images, image sets, and videos. Existing creation processes may involve multiple applications and a large amount of manual work, thus there is a desire to improve the efficiency of generating video-type media content based on visual data. Summary of the Invention
[0003] In a first aspect of this disclosure, a video generation method is provided. The method includes: acquiring visual data and configuration data, the visual data including an object and the object including a mouth, and the configuration data specifying the speech of the object; and generating a video based on the visual data and the speech, the visual content of the video being used to represent the shape of the object's mouth and the shape of the object's mouth matching the speech, and the auditory content of the video being used to represent the speech.
[0004] In a second aspect of this disclosure, a video generation apparatus is provided. The apparatus includes: an acquisition module configured to acquire visual data and configuration data, the visual data including an object and the object including a mouth, and the configuration data specifying speech of the object; and a generation module configured to generate a video based on the visual data and speech, the visual content of the video being used to represent the shape of the object's mouth and the shape of the object's mouth matching the speech, and the auditory content of the video being used to represent the speech.
[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the electronic device to perform the method of the first aspect.
[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The medium stores computer instructions that, when executed by a processor, implement the method of the first aspect.
[0007] In a fifth aspect of this disclosure, a computer program product is provided. The product includes a computer program, wherein when executed by a processor, the computer program implements the method according to a first aspect of this disclosure.
[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:
[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;
[0011] Figures 2A to 2I An example interface for generating video from visual data according to some embodiments of the present disclosure is shown;
[0012] Figure 3 A flowchart is shown illustrating a method for generating video from visual data according to some embodiments of the present disclosure;
[0013] Figure 4 An exemplary structural block diagram of an apparatus for generating video from visual data according to some embodiments of the present disclosure is shown; and
[0014] Figure 5 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation
[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.
[0017] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.
[0018] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.
[0019] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.
[0020] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.
[0021] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0022] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.
[0023] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In this example environment 100, an application 115 is installed on a terminal device 110. A user 140 can interact with the application 115 via the terminal device 110 and / or an attached device of the terminal device 110.
[0024] In some embodiments, application 115 may be any suitable application capable of generating media content (e.g., text, images, audio, video, or rich media data, etc.). Application 115 may, for example, provide user 140 with various services related to media content, including playback, generation, publishing, and commenting on media content. In some embodiments, application 115 may also be any other suitable application capable of generating media content.
[0025] exist Figure 1In environment 100, if application 115 is active, terminal device 110 can display the interface 150 of application 115. Interface 150 may include various pages provided by application 120, such as editing pages, publishing pages, playback pages, etc.
[0026] In some embodiments, terminal device 110 communicates with server 120 to provide services to application 115. Terminal device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 can also support any type of user-facing interface (such as "wearable" circuitry).
[0027] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server 120 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in cloud environments, etc.
[0028] In some embodiments, terminal device 110 may generate media content using machine learning model 130. Machine learning model 130 may be deployed locally on terminal device 110 or deployed on other devices / systems (e.g., server 120). Terminal device 110 may, for example, directly utilize the locally deployed machine learning model 130 or invoke the machine learning model 130 deployed on other devices / systems via communication connections to generate media content.
[0029] As an example, if the machine learning model 130 is deployed on server 120, the terminal device 110 can provide the server 120 with the information required for the model input via a communication connection with server 120. Server 120 can determine the model input and provide it to the machine learning model 130. Server 120 can obtain the model output of machine learning model 130 and send the obtained model output to terminal device 110.
[0030] Machine learning model 130 can be based on any suitable model architecture, including but not limited to Transformer models, convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep neural networks (DNNs), and so on. In some embodiments, machine learning model 130 can be based on a language model (LM). A language model learns from a large corpus and has question-answering capabilities. Machine learning model 130 can also be based on other suitable models. Although only a single machine learning model 130 is shown in the figure, multiple machine learning models can exist. If machine learning model 130 includes multiple models, these multiple models can include models configured to perform the same or similar tasks, or models configured to perform different tasks.
[0031] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.
[0032] Traditionally, if users expect to generate video-type media content based on visual content, they typically need to perform numerous manual operations (e.g., speech generation, editing, audio addition, video frame adjustment), which consumes significant manpower and time, resulting in inefficient and costly video generation. Specifically, assuming the desired outcome is to generate a video featuring a person speaking, the user needs to specify images and the text of the speech, convert the text to audio in an audio generation tool, modify the person's lip movements and generate an image sequence for the video in an image processing tool, and finally combine the image sequence and audio in a video generation tool to create the video.
[0033] In view of this, according to embodiments of the present disclosure, an improved scheme for generating video from visual data is proposed. According to the scheme of the embodiments of the present disclosure, visual data and configuration data are acquired, the visual data including an object and the object including a mouth, and the configuration data specifying the speech of the object; and a video is generated based on the visual data and the speech, the visual content of the video being used to present the shape of the object's mouth and the shape of the object's mouth matching the speech, and the auditory content of the video being used to present the speech.
[0034] This method generates videos based on visual data and user-specified speech, matching the mouth shapes of objects in the video with the spoken words. This approach allows for convenient and rapid video generation, improving efficiency. Furthermore, it avoids mismatches between lip movements and speech, enhancing video accuracy.
[0035] The following description continues with reference to the accompanying drawings, illustrating some exemplary embodiments of this disclosure. The interactive methods involved in this disclosure can be implemented on terminal device 110. It should be noted that the operations performed by terminal device 110 may specifically be performed by a related application (e.g., application 115) installed on terminal device 110. Some operations described with reference to terminal device 110 may require the assistance of server 120 to complete.
[0036] In this disclosure, "trigger" refers to one or more interactive actions performed by a user on a terminal device. Furthermore, these interactive actions may be triggered within the same user interface / pop-up window or within different user interfaces / pop-up windows. This disclosure is not limited in this respect.
[0037] It should be understood that the interface shown in the accompanying drawings is merely an example, and various interface designs are possible in practice. The various graphic elements in the interface can have different arrangements and different visual representations, one or more elements can be omitted or replaced, and one or more other elements may also be present. The embodiments of this disclosure are not limited in this respect.
[0038] In embodiments of this disclosure, visual data and configuration data can be acquired. The visual data can be images or videos. The visual data includes objects, and objects include mouths. Objects can include at least one of people, animals, or virtual characters. Using some implementations of this disclosure, the mouths of various types of objects can be automatically identified and processed, thereby generating videos featuring various types of objects speaking. It is understood that an object can be one or more objects, meaning the visual data can include one or more objects. One or more objects in the visual data can be automatically identified, and a user can select an object via an application to generate a video featuring that object speaking. Alternatively and / or additionally, a user can select multiple objects via an application and specify corresponding configuration data for each object, generating a video featuring dialogue between multiple objects. In this way, richer processing methods can be provided, and videos with richer content can be generated.
[0039] The configuration data can specify the speech to be used to generate the auditory content in the video. The speech source can be text or audio. The configuration data can be user-provided data unrelated to the visual data; for example, if the visual data is an image or video, the configuration data can be a piece of text or audio input by the user. The configuration data can also be data related to the visual data; for example, if the visual data is a video, the configuration data can be obtained based on the video's own audio (e.g., speech extracted from the video). Using some implementations of this disclosure, it is possible to support specifying speech in multiple ways, thereby generating videos based on speech sources of various formats, avoiding the complex process of users calling different tools to prepare audio data separately.
[0040] Terminal device 110 acquires the audio corresponding to the speech. It can be understood that if the source of the speech is text, terminal device 110 can acquire the text corresponding to the text-type speech and obtain the speech by performing a text-to-speech (TTS) process on that text; the resulting audio is the speech data corresponding to the text. If the source of the speech is audio, terminal device 110 can directly acquire the speech specified for that audio type (e.g., it can be referred to as the raw audio) and extract speech data from the raw audio as the speech. Terminal device 110 can, for example, perform any appropriate processing on the raw audio, such as noise reduction or removal of background music, to extract speech data from the raw audio.
[0041] Terminal device 110 determines the visual content in the video based on visual data and speech, and can also determine the auditory content in the video based on speech. The visual content includes objects whose mouth shapes match the speech. As mentioned earlier, the audio corresponding to the speech only includes speech data. If the audio corresponding to the speech is speech data extracted from original audio, the original audio may include speech data and background sound data. The auditory data of the video may, for example, include audio and background sound data of the original audio (i.e., the auditory data includes the original audio). If the audio corresponding to the speech is audio obtained through TTS (Text-to-Speech), then the auditory data of the video may include that audio. That is, the auditory content of the video at least includes the audio corresponding to the speech. Alternatively and / or additionally, the auditory portion of the video can be further edited, for example, by adding narration, background music, etc.
[0042] Terminal device 110 can determine the length of the video based on the lengths (i.e., durations) of the visual data and the corresponding audio, and then determine the visual content in the video based on the video length, the visual data, and the audio. In some embodiments, if the visual data is an image, i.e., the original image, terminal device 110 can generate the visual content of the video based on the original image and the corresponding audio. The duration of the visual content of the video can be equal to or greater than the duration of the audio. Taking object A as an example, the visual content of the video can be, for example, object A reciting or singing, with object A's lip movements matching the audio. In each video frame, except for object A's lip movements, the other content is the same as the content of the original image.
[0043] In some embodiments, if the visual data is video, i.e., the original video in video format, the terminal device 110 may first determine the difference between the length of the original video and the length of the audio. The difference may indicate which of the corresponding lengths of the original video and audio is smaller, and the terminal device 110 may, for example, directly determine the length of the video based on the length of the smaller corresponding length. For example, if the length of the original video is not less than the length of the audio, then the length of the video is equal to the length of the audio. In this case, the terminal device 110 may, for example, extract at least a portion of the original video based on the length of the video, and determine the visual content of the video based on the extracted at least portion of the video and the audio. Alternatively and / or additionally, the length of the video may be equal to the length of the original video, and in the video, object A speaks only during a certain time period and remains silent during other time periods, and the length of that time period is equal to the length of the audio. Alternatively and / or additionally, the terminal device 110 may also directly determine the visual content of the video based on the original video and audio, which is not limited in this disclosure.
[0044] In some embodiments, if the length of the original video is shorter than the length of the audio, the terminal device 110 may default to determining that the length of the video is equal to the length of the original video. In this case, the terminal device 110 may extract a portion of the audio based on the length of the video and directly determine the video based on the original video and the extracted audio portion. In this case, the auditory content of the video includes the extracted audio portion.
[0045] In some embodiments, to ensure that the video may include complete audio content, the terminal device 110 may also determine a reversed version of the original video. The terminal device 110 can then determine the visual content in the video based on the original video, the reversed version of the original video, and the audio. In this case, the auditory content of the video includes the audio. This approach ensures that the video includes complete audio content and also ensures the smoothness and stability of the video frame. This method allows for the generation of videos that better meet various generation requirements.
[0046] It is important to note that if the audio corresponding to the speech is obtained through TTS or extracted from user-specified audio, the terminal device 110 can directly determine the auditory content of the video based on the audio corresponding to the speech. If the audio corresponding to the speech is extracted from visual data of the video type, the terminal device 110 can, for example, perform translation on the audio (e.g., from a first language to a second language) and determine the auditory content of the video based on the translated audio. In this way, lip movements in the visual content and language in the auditory content can be automatically converted, thereby generating a video with a translation effect.
[0047] Specifically, the user can specify the original video to be processed (e.g., a person's Chinese speech). The terminal device 110 can extract the original visual and auditory content from the original video, translate the Chinese speech related to the original auditory content into an English speech, and modify the person's lip movements based on the English pronunciation to generate the corresponding video. At this point, the video can include the person's English speech, and the person's lip movements will match the English pronunciation.
[0048] In some embodiments, the visual content of the video may be generated using a machine learning model (e.g., machine learning model 130). In some embodiments, the terminal device 110 may also acquire a generation mode for the video. Different generation modes may correspond to different machine learning models, and the size, architecture, etc., of the machine learning models corresponding to different generation modes may be different.
[0049] The size of each machine learning model is related to its parameter size and structural complexity. Generally, the larger the parameter size or the more complex the structure of a machine learning model, the larger its size. Larger model sizes require greater resource overhead, including but not limited to computational resources, memory resources, and processing time. Larger model sizes tend to produce higher quality content.
[0050] Terminal device 110 can utilize machine learning models corresponding to the generation modes to generate visual content for video based on visual data and speech. For example, if two generation modes are included (e.g., a simple generation mode or a complex generation mode), the size of the machine learning model corresponding to the simple generation mode is smaller than the size of the machine learning model corresponding to the complex generation mode. Using some implementation methods of this disclosure, different generation modes can meet different generation requirements, thereby generating rich videos while satisfying user-specified resource overhead and generation effects.
[0051] If the generation mode used to generate the video is a simple generation mode, the terminal device 110 can use a corresponding, smaller machine learning model to generate visual content. In this case, the lip movements of the people in the video are modified, while other parts can remain unchanged. If the generation mode used to generate the video is a complex generation mode, the terminal device 110 can use a corresponding, larger machine learning model to generate visual content. The quality of the visual content corresponding to the complex generation mode is higher than that corresponding to the simple generation mode. For example, the lip movements and / or other facial areas, and even head and / or body movements of the people in the video, can be adjusted based on speech. For example, if the content of the speech is exciting and agitated, the lip movements of the people in the video can be more pronounced, and their heads can move within a certain range, and so on.
[0052] In some embodiments, terminal device 110 may also utilize machine learning models to determine audio corresponding to speech, and to determine auditory content of video. For example, terminal device 110 may also utilize different machine learning models to determine audio or auditory content based on the generation mode for the video. For instance, if the generation mode used to generate the video is a simple generation mode, terminal device 110 may utilize a corresponding, smaller machine learning model to perform speech synthesis to determine audio. If the generation mode used to generate the video is a complex generation mode, terminal device 110 may utilize a corresponding, larger machine learning model to perform speech synthesis to determine audio; the audio quality corresponding to the complex generation mode will be higher than the audio quality corresponding to the simple generation mode.
[0053] For example, if the generation mode used to generate the video is a simple generation mode, the terminal device 110 can use a corresponding, smaller machine learning model to translate the audio extracted from the video to determine the audio content of the video. If the generation mode used to generate the video is a complex generation mode, the terminal device 110 can use a corresponding, larger machine learning model to translate the audio extracted from the video to determine the audio content of the video. The quality of the audio content corresponding to the complex generation mode will be higher than the quality of the audio content corresponding to the simple generation mode.
[0054] The steps in the video generation process have already been described; please refer to the following section. Figures 2A to 2I This describes a sample interface used to generate videos from visual data. Figures 2A to 2I Example interfaces 200A to 200I for generating video from visual data according to some embodiments of the present disclosure are shown. The interfaces shown in example interfaces 200A to 200I can be presented at a terminal device 110. For ease of discussion, reference will be made to... Figure 1The example interfaces 200A to 200I are described using environment 100. Terminal device 120 can be used for interaction by receiving operations from associated users (e.g., user 140) to present corresponding interaction results.
[0055] In some embodiments, the terminal device 110 may provide a visual data editing interface. In response to receiving a video generation request on this editing interface, the terminal device 110 may acquire configuration data and generate a video based on the visual data and the configuration data. (See reference...) Figure 2A Example interface 200A illustrates an example of an editing interface for visual data. Example interface 200A includes region 210, in which terminal device 110 can present visual data. Example interface 200A also includes region 202, which may include multiple interactive controls for the visual data. In some embodiments, region 202 may include a first interactive control, and terminal device 110 may determine, in response to receiving a trigger on the first interactive control, that a video generation request has been received for the visual data shown in region 210.
[0056] In some embodiments, the example interface 200A may further include a control 201, and the terminal device 110 may, in response to receiving a trigger on the control 201, present... Figure 2B The example interface 200B is shown. Example interface 200B may include region 203, which may include at least control 204. Terminal device 110 may, in response to receiving a trigger on control 204, determine that a video generation request for the visual data shown in region 210 has been received.
[0057] Terminal device 110, for example, may present a user interface for generating video in response to receiving a video generation request. Terminal device 110 can receive at least configuration data via such a user interface. Terminal device 110 may, for example, provide a control in such a user interface for specifying a source of audio, including text or audio. Terminal device 110 can then obtain the audio specified by the specified control in response to receiving an interaction request with the specified control. (See reference) Figure 2C Example interface 200C includes interface 220, which shows an example of a user interface for generating video.
[0058] Interface 220 includes a cancel control 205. Terminal device 110 can cancel the presentation of example interface 200C in response to triggering the cancel control 205. Terminal device 110 can, for example, return to the editing interface presenting the visual data (e.g., example interface 200A or example interface 200B). Interface 220 includes a "Custom" control and a "Video Translation" control, wherein the "Video Translation" control can be selected only when the visual data is video. When the visual data is video, terminal device 110 can extract text or audio from the video and perform translation in response to the selection of the "Video Translation" control.
[0059] Speech can be extracted from the video, which can be referred to as raw auditory content. Terminal device 110 can translate this raw auditory content to convert it into auditory content (e.g., translating raw auditory content in a first language into auditory content in a second language). If text is extracted from the video, speech synthesis can be performed on the text based on the language of the video to be generated to determine auditory content. Terminal device 110 can then generate a video based on the auditory content. In this case, the speech data is included in the auditory content.
[0060] Terminal device 110 can also respond to the selection of the "Custom" control by receiving user input from a user (e.g., user 140) and determining configuration data based on the user input. When the "Custom" control is selected, referring to example interface 200C, interface 220 can include controls "Generate with Text" and "Generate with Audio". The "Generate with Text" control can specify the source of the voice as text type, and the "Generate with Audio" control can specify the source of the voice as audio type.
[0061] Terminal device 110 can respond to the "Generate Text" control being selected, such as Figure 2C As shown, an input box 221 is provided on interface 220, and text input corresponding to speech is received via input box 221. Terminal device 110 can generate audio corresponding to the speech based on the text input. In some embodiments, terminal device 110 can also receive a generated configuration for configuration data on interface 220. This generated configuration can indicate one or more of the following: timbre, speech rate, volume, language (i.e., dialect), and tone of the audio. In the case of generating audio based on text input, terminal device 110 can also perform TTS on the text input based on the generated configuration to determine the audio.
[0062] refer to Figure 2CThe interface 220 may also include a region 223, which may display multiple timbres. In some embodiments, the terminal device 110 may also display more timbres in response to receiving a sliding operation (e.g., a leftward or rightward sliding operation) on the region 223. The interface 220 may also include a control 222, and the terminal device 110 may also display more timbres in response to receiving a trigger on the control 222. The terminal device 110 may highlight a timbre in the region 223 in response to receiving a selection of a timbre.
[0063] In some embodiments, the terminal device 110 may also provide a default tone, which is selected by default. This tone can be, for example, a tone previously used by the user, multiple tones with the highest user frequency / highest approval rating, or a tone recommended by the application. For example, such as... Figure 2C As shown, if tone 1 is selected or tone 1 is the default tone, terminal device 110 can highlight tone 1 in area 220. Terminal device 110 can highlight the selected tone in any appropriate way, such as by thickening the border, highlighting, or adding a background; this disclosure does not limit the specific highlighting method. Terminal device 110 can determine the tone specified in the generation configuration through area 220, and then perform TTS on the text input based on the tone to determine the speech.
[0064] Terminal device 110 can respond to the "Generate with audio" control being selected, such as Figure 2D As shown, interface 220 provides at least one control for receiving audio. For example, terminal device 110 may provide area 227 on interface 220, which may include multiple controls corresponding to different methods of receiving audio. For example, area 227 may include controls for "Import from video", "Select favorite songs", "Select from music library", and "Record your own audio". Terminal device 110 may receive video input from the user in response to the "Import from video" control being selected, extract audio input from the video input (the audio input may include background sound data and speech data from the video), and extract speech data from the extracted audio input. Terminal device 110 may identify the extracted speech data as speech (of course, terminal device 110 may also extract speech data from video input).
[0065] Terminal device 110 can respond to the "Select Favorite Song" control being selected, displaying a list of songs the user has historically favorited, and can identify that song as audio input from the user upon receiving a selection of a song from that list. Terminal device 110 can respond to the "Select from Music Library" control being selected, displaying at least one music library. It can respond to receiving a selection of a music library, displaying a list of audio files included in that music library. Terminal device 110 can identify that audio file as audio input from the user upon receiving a selection of an audio file from that list. Terminal device 110 can respond to the "Record Yourself" control being selected, displaying an audio acquisition interface and receiving audio input recorded by the user through this interface. Terminal device 110 can then determine the audio corresponding to the speech based on the audio input. Terminal device 110 can extract speech data from the audio input and identify the extracted speech data as the audio corresponding to the speech.
[0066] In some embodiments, the terminal device 110 can also play a song / audio from the list in response to a received playback operation. This helps the user understand the selected song / audio and avoid errors. In some embodiments, the terminal device 110 can also play an audio input in response to a received playback operation. This helps the user quickly understand the specific content of the audio input and facilitates subsequent operations (such as continuing to generate video or adjusting the audio input). In some embodiments, after determining the audio corresponding to the voice, the terminal device 110 can also play the audio in response to a received playback operation. This allows the user to check the audio and perform subsequent operations. All of these contribute to improving the user experience.
[0067] It is understandable that when the "Generate with Audio" control is selected, the terminal device 110 determines the auditory content of the video based on the audio input. The auditory content may include only the extracted speech data (i.e., only the audio corresponding to the speech), or it may also include background sound data from the audio input.
[0068] In some embodiments, if the visual data is video, the terminal device 110 may also receive configuration information associated with the video via a user interface for generating the video. The configuration information indicates whether the auditory content of the video to be generated includes background sound data for the video's auditory content. (See reference...) Figure 2CTerminal device 110 can provide control 224 on interface 220. In response to receiving a selection of control 224, terminal device 110 can determine that the auditory content of the video to be generated includes background audio data of the video's auditory content. In this case, terminal device 110 can determine the auditory content of the video based on the background audio data and speech (i.e., speech data). Terminal device 110 can also determine that the auditory content of the video to be generated does not include background audio data of the video's auditory content if control 224 is not selected. In this case, terminal device 110 can directly determine the auditory content of the video based on speech.
[0069] In some embodiments, terminal device 110 may also provide a selection control for selecting audio segments from speech on a user interface for generating video. Terminal device 110 may, in response to receiving a selection request for the selection control, determine the auditory content in the video based on the audio segment corresponding to the selection control. For example, refer to... Figure 2E In example interface 200E, terminal device 110 can display audio 228 corresponding to the speech in interface 220. Terminal device 110 can overlay selection control 229 on the audio input 228. The two vertical lines (the vertical lines at both ends of the double-headed arrow) of selection control 229 and the range indicated by the double-headed arrow represent the selected audio segment. Users can adjust the selected audio segment by dragging the two vertical lines of selection control 229. In this way, users can select the desired audio segment, thereby specifying the duration of the sound portion in the video.
[0070] Terminal device 110 may also display a prompt message associated with audio 228, which can be used to indicate the length of the currently selected audio segment to the user (e.g., the text "Selected XX seconds" shown in the figure). In some embodiments, the audio duration may be long, and to save computing resources, the initially default selected audio segment can be a predetermined duration (which can be any appropriate length; for example, it could be 30 seconds). Terminal device 110 may, for example, determine the auditory content in the video based on the audio segment indicated by the selection control, in which case the auditory content of the video includes the audio segment rather than the complete audio corresponding to the audio segment.
[0071] In some embodiments, the terminal device 110 may also provide multiple controls for the generation mode (e.g., Figure 2CAs shown in the diagrams (controls 225 and 226), it can be understood that multiple controls correspond to multiple generation modes, and may include any number of generation modes. For example, in response to the selection of control 225, the terminal device 110 may generate the visual content of the video based on the machine learning model corresponding to the corresponding generation mode (e.g., normal mode). Similarly, in response to the selection of control 226, the terminal device 110 may generate the visual content of the video based on the machine learning model corresponding to the corresponding generation mode (e.g., dynamic mode). The size of the machine learning model corresponding to the dynamic mode may, for example, be larger than the size of the machine learning model corresponding to the normal mode. In some embodiments, to reduce computational load, control 225 may be selected by default.
[0072] Terminal device 110 may also provide a generate control 232 and a clear control 231 on interface 220. In response to the selection of the generate control 232, terminal device 110 may determine visual content in the video based on visual data and speech, determine auditory content in the video based on speech, and combine visual and auditory content to determine the video. In some embodiments, in response to the triggering of the generate control 232, if the visual data is the original video and the length of the original video is shorter than the length of the audio, terminal device 110 may present... Figure 2F The example interface 200F is shown. Terminal device 110 can provide interface 240 within example interface 200F, which includes a prompt message informing the user that the length of the original video is shorter than the length of the audio, and that a reversed version of the original video will be determined. Further, based on the original video, the reversed version of the original video, and the audio, the visual content in the video is determined. In this way, the relevant attributes of the generated video can be communicated, thereby supporting the user in performing subsequent operations.
[0073] Interface 240 may include controls 241 and 242. Terminal device 110 may, in response to a trigger on control 241, determine the reversed video of the original video, and then, based on the original video, the reversed video, and audio, determine the visual content in the video. Terminal device 110 may also, in response to a trigger on control 242, cancel the presentation of interface 240; for example, terminal device 110 may return to the presentation of interface 220. It should be understood that the audio here may be the original audio or an audio segment selected by the user using selection control 229.
[0074] In some embodiments, interface 240 may further include a control (not shown) indicating a reverse video of an uncertain original video, and a control for determining the visual content of the video directly based on the original video and audio. Terminal device 110 may, in response to receiving a trigger on the control, extract a portion of audio from the audio based on the duration of the original video, and determine the visual content of the video directly based on the original video and the extracted audio portion.
[0075] Before generating video, terminal device 110 can, in response to receiving a trigger on clear control 231, clear the acquired audio and various configurations. For example, terminal device 110 can, in response to the trigger on clear control 231, present... Figure 2G The example interface 200G is shown. Terminal device 110 can provide interface 250 within example interface 200G. Interface 250 includes a prompt message that alerts the user to clear voice and configuration, requiring secondary confirmation. Interface 250 includes controls 251 and 252. Terminal device 110 can clear voice and various configurations in response to the triggering of control 251. Terminal device 110 can return to the presented interface 220 in response to the triggering of control 252, while maintaining the voice and various configurations unchanged.
[0076] In some embodiments, after the video is generated, the terminal device 110 can also provide a preview of the video. This allows users to easily view, adjust, apply, and publish the video. Alternatively and / or additionally, if the user is not satisfied with the generated video, they can re-edit the configuration data. For example, they can return to the various interfaces described above to set the corresponding configuration data.
[0077] The above Figures 2A to 2G An interface provided by a mobile terminal device 110 is shown. In some embodiments, if the terminal device 110 is a personal computer (PC) terminal device 110, Figure 2H and Figure 2I An example of the interface provided by a PC-type terminal device 110 is shown. Example interface 200H includes area 260 for presenting visual data. Example interface 200H includes area 270 for receiving various configuration information related to the visual data.
[0078] In some embodiments, in response to receiving a video generation request for visual data, terminal device 110 may provide an identifier “Verbalization” in area 270, indicating that a video generation operation is currently being performed. Similar to the interface provided by mobile terminal device 110, terminal device 110 may provide controls for “Text Generation” and “Audio Generation” in area 270. When the “Text Generation” control is selected, terminal device 110 may provide an input box 271 for receiving text input, multiple timbres, a control for receiving configuration information associated with the visual data (i.e., the “Background Sound Settings” control shown in the figure), and a generation control 272 in area 270 of example interface 200H. When the “Audio Generation” control is selected, terminal device 110 may provide control 273 in area 270 of example interface 200I, and may obtain audio input from the user in response to the selection of control 273.
[0079] In summary, according to the various embodiments of this disclosure, a video can be generated based on video data and audio, and the mouth shape of an object in the generated video matches the audio. This method allows for convenient and quick video generation, improving efficiency. Furthermore, it avoids mismatches between the lip shape of an object in the video and the audio, thus improving the overall video generation quality.
[0080] Figure 3 A flowchart of a method 300 for generating video from visual data according to some embodiments of the present disclosure is shown. Method 300 may be implemented at a terminal device 110. At block 310, visual data and configuration data are acquired, the visual data including an object and the object including a mouth, and the configuration data specifying the speech of the object. At block 320, a video is generated based on the visual data and the speech, the visual content of the video being used to represent the shape of the object's mouth and the shape of the object's mouth matching the speech, and the auditory content of the video being used to represent the speech.
[0081] In some embodiments, obtaining configuration data includes: providing a specified control for specifying the source of the voice, the source of the voice including a text type or an audio type; and in response to receiving an interaction request with the specified control, obtaining the voice specified by the specified control.
[0082] In some embodiments, acquiring speech includes: in response to determining that the source of the speech is text type, acquiring text input corresponding to the speech; and generating speech based on the text input.
[0083] In some embodiments, the configuration data further includes a generation configuration that specifies at least one of the following: the timbre, speed, volume, language, or tone of the speech; and generating speech based on text input includes performing speech synthesis on the text input based on the generation configuration to determine the speech.
[0084] In some embodiments, generating auditory content in a video includes: determining the auditory content in the video based on speech.
[0085] In some embodiments, acquiring speech includes: in response to determining that the source of the speech is an audio type, acquiring audio input corresponding to the speech; and extracting speech data from the audio input as speech.
[0086] In some embodiments, generating auditory content in a video includes determining auditory content in the video based on audio input corresponding to speech.
[0087] In some embodiments, obtaining configuration data includes: in response to determining that the visual data is an original video in video format, converting the original auditory content in the original video into auditory content, wherein the original auditory content is represented in a first language and the auditory content is represented in a second language.
[0088] In some embodiments, generating auditory content in a video includes: providing a selection control for selecting an audio segment from speech; and, in response to receiving a selection request for the selection control, determining the auditory content in the video based on the audio segment corresponding to the selection control.
[0089] In some embodiments, generating visual content in a video further includes: in response to determining that the visual data is an original video in video format, determining a difference between the length of the original video and the length of the speech; in response to determining that the difference indicates that the length of the original video is less than the length of the speech, determining visual content in the video based on the original video and a reverse video of the original video; or in response to determining that the difference indicates that the length of the original video is not less than the length of the speech, determining visual content in the video based on the original video.
[0090] In some embodiments, generating visual content in a video further includes: acquiring a generation pattern for generating the visual content; and using a machine learning model corresponding to the generation pattern to determine the visual content in the video, wherein the faces of objects in the visual content match the audio.
[0091] In some embodiments, visual data includes images or videos, and the object includes at least one of the following: a person, an animal, or a virtual character.
[0092] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4An exemplary structural block diagram of an apparatus 400 for generating a target video from visual data according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in a terminal device 110. The various modules / components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.
[0093] like Figure 4 As shown, the device 400 includes: an acquisition module 410 configured to acquire visual data and configuration data, the visual data including an object and the object including a mouth, and the configuration data specifying the speech of the object; and a generation module 420 configured to generate a video based on the visual data and the speech, the visual content of the video being used to present the shape of the mouth of the object and the shape of the mouth of the object matching the speech, and the auditory content of the video being used to present the speech.
[0094] In some embodiments, the acquisition module 410 is further configured to: provide a specified control for specifying the source of the voice, the source of the voice including text type or audio type; and in response to receiving an interaction request with the specified control, acquire the voice specified by the specified control.
[0095] In some embodiments, the acquisition module 410 is further configured to: acquire text input corresponding to the speech in response to determining that the source of the speech is text; and generate speech based on the text input.
[0096] In some embodiments, the configuration data further includes a generation configuration that specifies at least one of the following: the timbre, speed, volume, language, or tone of the speech; and the acquisition module 410 is further configured to perform speech synthesis on the text input based on the generation configuration to determine the speech.
[0097] In some embodiments, the generation module 420 is further configured to: determine auditory content in the video based on speech.
[0098] In some embodiments, the acquisition module 410 is further configured to: acquire audio input corresponding to the speech in response to determining that the source of the speech is an audio type; and extract speech data from the audio input as speech.
[0099] In some embodiments, the generation module 420 is further configured to: determine auditory content in the video based on audio input corresponding to speech.
[0100] In some embodiments, the acquisition module 410 is further configured to: in response to determining that the visual data is an original video in video format, convert the original auditory content in the original video into auditory content, wherein the original auditory content is represented in a first language and the auditory content is represented in a second language.
[0101] In some embodiments, the generation module 420 is further configured to: provide a selection control for selecting an audio segment from speech; and, in response to receiving a selection request for the selection control, determine auditory content in the video based on the audio segment corresponding to the selection control.
[0102] In some embodiments, the generation module 420 is further configured to: in response to determining that the visual data is an original video in video format, determine the difference between the length of the original video and the length of the speech; in response to determining that the difference indicates that the length of the original video is less than the length of the speech, determine the visual content in the video based on the original video and a reverse video of the original video; or in response to determining that the difference indicates that the length of the original video is not less than the length of the speech, determine the visual content in the video based on the original video.
[0103] In some embodiments, the generation module 420 is further configured to: acquire a generation pattern for generating visual content; and utilize a machine learning model corresponding to the generation pattern to determine visual content in a video, wherein the faces of objects in the visual content match the audio.
[0104] In some embodiments, visual data includes images or videos, and the object includes at least one of the following: a person, an animal, or a virtual character.
[0105] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 400 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.
[0106] It should be understood that one or more steps in the above methods can be performed by suitable electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices may include, for example, […]. Figure 1 Terminal device 110 in the middle.
[0107] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 Terminal equipment 110 or Figure 4 Device 400.
[0108] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processor 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processor 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.
[0109] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.
[0110] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.
[0111] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.
[0112] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).
[0113] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.
[0114] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0115] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0116] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some, as newer, implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0118] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.
Claims
1. A video generation method, comprising: Acquire the visual data and configuration data, wherein the visual data includes an object and the object includes a mouth, and the configuration data specifies the speech of the object; as well as A video is generated based on the visual data and the speech, wherein the visual content of the video is used to present the shape of the object's mouth and the shape of the object's mouth matches the speech, and the auditory content of the video is used to present the speech.
2. The method according to claim 1, wherein obtaining the configuration data includes: Provide a control for specifying the source of the speech, wherein the source of the speech includes text type or audio type; as well as In response to receiving an interaction request with the specified control, the voice specified by the specified control is obtained.
3. The method according to claim 2, wherein acquiring the voice comprises: In response to determining that the source of the speech is text type, text input corresponding to the speech is obtained; as well as The speech is generated based on the text input.
4. The method of claim 3, wherein the configuration data further includes a generation configuration, the generation configuration specifying at least one of the following: the timbre, speed, volume, language, or intonation of the speech; and generating the speech based on the text input includes: Based on the generated configuration, speech synthesis is performed on the text input to determine the speech.
5. The method according to claim 3, wherein generating the auditory content in the video comprises: Based on the speech, the auditory content in the video is determined.
6. The method according to claim 2, wherein acquiring the voice comprises: In response to determining that the source of the speech is an audio type, audio input corresponding to the speech is obtained; as well as Extract the speech data from the audio input to obtain the speech data.
7. The method of claim 6, wherein generating the auditory content in the video comprises: The auditory content in the video is determined based on the audio input corresponding to the speech.
8. The method according to claim 1, wherein obtaining the configuration data includes: In response to determining that the visual data is an original video in video format, the original auditory content in the original video is converted into the auditory content, the original auditory content being represented in a first language and the auditory content being represented in a second language.
9. The method of claim 1, wherein generating the auditory content in the video comprises: Provide a selection control for selecting audio segments from the speech; as well as In response to receiving a selection request for the selection control, the auditory content in the video is determined based on the audio segment corresponding to the selection control.
10. The method of claim 1, wherein generating the visual content in the video further comprises: In response to determining that the visual data is an original video in video format, the difference between the length of the original video and the length of the audio is determined; In response to determining that the difference indicates that the length of the original video is less than the length of the speech, the visual content in the video is determined based on the original video and a reverse video of the original video; or In response to determining that the difference indicates that the length of the original video is not less than the length of the speech, the visual content in the video is determined based on the original video.
11. The method of claim 1, wherein generating the visual content in the video further comprises: Obtain the generation mode used to generate the visual content; as well as Using a machine learning model corresponding to the generation pattern, the visual content in the video is determined, wherein the face of the object in the visual content matches the audio.
12. The method of claim 1, wherein the visual data includes images or videos, and the object includes at least one of the following: a person, an animal, or a virtual character.
13. A video generation apparatus, comprising: An acquisition module is configured to acquire the visual data and configuration data, the visual data including an object and the object including a mouth, and the configuration data specifying the speech of the object; as well as A generation module is configured to generate a video based on the visual data and the speech, wherein the visual content of the video is used to present the shape of the mouth of the object and the shape of the mouth of the object matches the speech, and the auditory content of the video is used to present the speech.
14. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 12 when executed by the at least one processor.
15. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 12.
16. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 12.
