Video generation method, related device, equipment and storage medium

Abstract

The invention discloses a video generation method, a related device, equipment and a storage medium. The method comprises the following steps: acquiring a target audio material; obtaining a target text material corresponding to the target audio material; performing feature extraction on the target audio material to obtain audio description information related to the target audio material; obtaining K candidate videos through a video generation model based on the target text material and the audio description information; and according to the target audio material, editing the K candidate videos to obtain a target synthetic video, the target synthetic video comprising a video clip from at least one candidate video, and the background sound of the target synthetic video being generated based on the target audio material. According to the method provided by the invention, the video editing efficiency can be improved, so that the user experience is enhanced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, related apparatus, device, and storage medium for video generation. Background Technology

[0002] With the rapid advancement of technology and the widespread use of the internet, more and more people are using videos to transmit information and share snippets of their lives. When music is combined with video, the music not only provides the emotional and rhythmic foundation for the video, but the video, through concrete images and actions, also concretizes and visualizes the emotions and artistic conception of the music.

[0003] Currently, video processing software is a commonly used application on terminals, widely used in various scenarios. When creating a video, users first need to manually shoot or select video footage, then edit the video footage according to the rhythm of the music, and finally use video processing software to combine the music and the edited video footage into a single video.

[0004] The inventors have discovered at least the following problems with current solutions: in related technologies, users need to spend a lot of effort and time, as well as possess certain professional knowledge, when editing video footage using video processing software. Therefore, the creative threshold is high, resulting in a poor user experience. An effective method is urgently needed to solve these problems. Summary of the Invention

[0005] This application provides a method, related apparatus, device, and storage medium for video generation, which can improve video editing efficiency and thus enhance the user experience.

[0006] In view of this, this application provides a method for video generation, comprising:

[0007] Acquire target audio material, wherein the target audio material includes at least one of music and speech;

[0008] Obtain the target text material corresponding to the target audio material;

[0009] Feature extraction is performed on the target audio material to obtain audio description information related to the target audio material. The audio description information includes at least one of timbre information, beat information, rhythm information, intensity information, emotion information, spectrum information, and pitch information.

[0010] Based on the target text material and audio description information, K candidate videos are obtained through a video generation model, where each candidate video includes at least one video segment, and K is an integer greater than 1;

[0011] Based on the target audio material, K candidate videos are edited to obtain the target synthesized video. The target synthesized video includes video segments from at least one candidate video, at least one candidate video belongs to the K candidate videos, and the background sound of the target synthesized video is generated based on the target audio material.

[0012] Another aspect of this application provides a video generation apparatus, comprising:

[0013] The acquisition module is used to acquire target audio material, wherein the target audio material includes at least one of music and speech;

[0014] The acquisition module is also used to acquire the target text material corresponding to the target audio material;

[0015] The extraction module is used to extract features from the target audio material to obtain audio description information related to the target audio material. The audio description information includes at least one of timbre information, beat information, rhythm information, intensity information, emotion information, spectrum information, and pitch information.

[0016] The acquisition module is also used to acquire K candidate videos based on the target text material and audio description information through a video generation model, wherein each candidate video includes at least one video segment, and K is an integer greater than 1;

[0017] The editing module is used to edit K candidate videos based on the target audio material to obtain the target synthesized video. The target synthesized video includes video segments from at least one candidate video, at least one candidate video belongs to the K candidate videos, and the background sound of the target synthesized video is generated based on the target audio material.

[0018] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0019] The acquisition module is specifically used to respond to the upload operation of an audio file and obtain the target audio material based on the audio file;

[0020] or,

[0021] In response to an input action for an audio link, retrieve the target audio material based on the audio link.

[0022] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0023] The acquisition module is specifically used to respond to input operations on text content and treat the text content as the target text material corresponding to the target audio material;

[0024] or,

[0025] In response to a speech recognition operation targeting a target audio material, obtain the target text material corresponding to the target audio material.

[0026] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0027] The extraction module is specifically used to preprocess the target audio material to obtain T audio frames, where T is an integer greater than 1;

[0028] Perform a Fourier transform on each of the T audio frames to obtain the spectrum of each audio frame;

[0029] The power spectrum is obtained by taking the square of the modulus of the spectrum of each audio frame;

[0030] The power spectrum is filtered by a Mel filter bank to obtain the filtering result corresponding to each Mel filter. The Mel filter bank includes at least one Mel filter.

[0031] Logarithmic compression and discrete cosine transform are performed on the filtering results corresponding to each Mel filter to obtain the timbre information corresponding to the target audio material, where the timbre information is represented by Mel frequency cepstral coefficients.

[0032] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0033] The extraction module is specifically used to convert the target audio material into a time-domain signal;

[0034] Perform a Fourier transform on the time-domain signal to obtain the frequency-domain signal;

[0035] Generate a spectrum diagram based on the frequency domain signal;

[0036] The peak positions in the spectrum are obtained by analyzing the spectrum using a peak detection algorithm.

[0037] The beat information corresponding to the target audio material is determined based on the peak position in the spectrogram.

[0038] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0039] The extraction module is specifically used to convert the target audio material into a time-domain signal;

[0040] Perform a Fourier transform on the time-domain signal to obtain the frequency-domain signal;

[0041] Generate a spectrum diagram based on the frequency domain signal;

[0042] The intensity information corresponding to the target audio material is obtained from the spectrogram. The intensity information includes the intensity of different frequency components in the frequency domain signal.

[0043] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0044] The extraction module is specifically used to obtain the emotion probability distribution based on the target audio material through an emotion recognition model, wherein the emotion probability distribution includes at least one probability value;

[0045] Based on the emotional probability distribution, determine the emotional information corresponding to the target audio material.

[0046] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0047] The acquisition module is specifically used to process the target text material through a text encoding model to obtain the first text feature vector;

[0048] The audio description information is characterized to obtain an audio feature vector;

[0049] Generate the target feature vector based on the text feature vector and the audio feature vector;

[0050] Based on the target feature vector, K candidate videos are obtained through a video generation model.

[0051] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0052] The acquisition module is specifically used to generate audio description text based on audio description information;

[0053] The target text material and the audio description text are combined to obtain the comprehensive description text;

[0054] The comprehensive descriptive text is processed using a text encoding model to obtain the second text feature vector;

[0055] Based on the second text feature vector, K candidate videos are obtained through a video generation model.

[0056] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0057] The editing module is specifically used to select N video segments from K candidate videos based on the target audio material, where N is an integer greater than or equal to 1;

[0058] By splicing together N video clips, the target composite video is obtained.

[0059] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0060] The editing module is specifically used to select N video segments from K candidate videos based on the target audio material, where N is an integer greater than or equal to 1;

[0061] By splicing together N video clips, a final video can be obtained.

[0062] The video to be synthesized is processed according to the visual effect parameters to obtain the target synthesized video. The visual effect parameters include at least one of the following: video duration, video quality, video aspect ratio, video filter, video style, transition effect, and dubbing style.

[0063] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0064] The editing module is specifically used to perform keyframe recognition on each of the K candidate videos to obtain a set of video segments. The set of video segments includes each video segment included in each of the K candidate videos, and the video segments are determined based on keyframes.

[0065] Based on the audio description information, select N video clips from the video clip set that meet the time dimension of the target audio material.

[0066] In another aspect, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the methods described above.

[0067] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described above.

[0068] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described above.

[0069] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0070] This application provides a method for video generation. First, target audio material and its corresponding target text material are obtained. The target text material guides the generation of video content. Then, features are extracted from the target audio material, and a video generation model is used to analyze the extracted audio description information and target text material, thereby outputting several candidate videos. Finally, these candidate videos are intelligently edited to automatically generate the target composite video. Through this method, based on user-input audio material, artificial intelligence technology is used to automatically analyze the audio material, and the corresponding video content is automatically generated based on the audio analysis results. This improves video editing efficiency and enhances the user experience. Attached Figure Description

[0071] Figure 1 This is a schematic diagram of an interface applied to a music video generation scenario in an embodiment of this application;

[0072] Figure 2 This is a schematic diagram of an interface applied to an animation video generation scenario in an embodiment of this application;

[0073] Figure 3 This is a schematic diagram of an implementation environment for the video generation method in this application embodiment;

[0074] Figure 4 This is a schematic diagram of another implementation environment for the video generation method in this application;

[0075] Figure 5 This is a flowchart illustrating a video generation method in an embodiment of this application;

[0076] Figure 6 This is a schematic diagram of the architecture of the video generation model in the embodiments of this application;

[0077] Figure 7 This is a schematic diagram of an interface providing the target audio material in an embodiment of this application;

[0078] Figure 8 This is a schematic diagram of an interface providing target text material in an embodiment of this application;

[0079] Figure 9 This is a schematic diagram illustrating the extraction of timbre information in an embodiment of this application;

[0080] Figure 10 This is a schematic diagram illustrating the extraction of beat information in an embodiment of this application;

[0081] Figure 11 This is a schematic diagram illustrating the extraction of intensity information in an embodiment of this application;

[0082] Figure 12This is a schematic diagram of an interface for setting visual effect parameters in an embodiment of this application;

[0083] Figure 13 This is a schematic diagram of an interface for recommending the generation of video styles in an embodiment of this application;

[0084] Figure 14 This is a schematic diagram of the architecture of the video generation system in an embodiment of this application;

[0085] Figure 15 This is a schematic diagram of the overall process of the video generation method in the embodiments of this application;

[0086] Figure 16 This is a schematic diagram of an interactive process of the video generation method in an embodiment of this application;

[0087] Figure 17 This is a schematic diagram of a video generation device in an embodiment of this application;

[0088] Figure 18 This is a schematic diagram of the structure of a computer device in an embodiment of this application. Detailed Implementation

[0089] This application provides a method, related apparatus, device, and storage medium for video generation, which can improve video editing efficiency and thus enhance the user experience.

[0090] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0091] It is understood that, in the specific embodiments of this application, data involving audio materials, text materials, etc., requires user permission or consent before being used in specific products or technologies. That is, before collecting user data, users may be prompted through interfaces, pop-ups, or voice prompts to indicate that their data needs to be collected. The process of collecting user data only begins after obtaining user permission or consent. In other words, all user data collected in this application is collected with the user's consent, and the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0092] Artificial intelligence (AI) video editing is a technology that uses AI to edit videos. It learns from large amounts of video data to automatically identify keyframes, shot transitions, speech recognition, and other elements in the video, and then automatically edits and splices them. During the editing process, it can also optimize based on user-defined settings, such as adjusting the video pacing and trimming the video length.

[0093] Currently, AI video editing primarily relies on pre-built video footage libraries. This means that, in conjunction with user-uploaded audio, relevant segments are selected from the video footage library and spliced ​​together. Clearly, these fixed footage sets limit the diversity of the output content, potentially affecting the quality and originality of the generated video.

[0094] Based on this, this application provides a video generation method. On the one hand, it utilizes AI technology to automatically analyze audio and generate matching video content, thereby reducing human intervention. On the other hand, it uses a video generation model to generate high-quality and diverse video content, surpassing the limitations of traditional material libraries.

[0095] Before introducing the specific methods of this application, the application scenarios of this application will be illustrated by example. It should be understood that the following application scenarios are merely illustrative and are not limited to these examples.

[0096] I. Music video (MV) generation scenario;

[0097] For example, for ease of understanding, please refer to Figure 1 , Figure 1 This is a schematic diagram of an interface applied to a music video generation scenario in an embodiment of this application, such as... Figure 1 As shown in Figure (A), 101 indicates an audio upload control; clicking the audio upload control allows you to upload the target audio material (e.g., a song). 102 indicates a text input control, allowing you to directly input the target text material (e.g., a description of the song). Clicking the generate control indicated by 103 displays the following... Figure 1 The interface shown in Figure (B) is 104, which is used to instruct the generation of a music video based on the target text material and the target audio material.

[0098] Understandably, when a user clicks the regenerate control indicated by 105, a different music video can be regenerated based on the target text material and the target audio material.

[0099] II. Animated video generation scenario;

[0100] For example, for ease of understanding, please refer to Figure 2 , Figure 2 This is a schematic diagram of an interface applied to an animated video generation scenario in an embodiment of this application, such as... Figure 2 As shown in Figure (A), 201 indicates an audio upload control; clicking the audio upload control allows you to upload the target audio material (e.g., an audiobook). 202 indicates a text input control, allowing you to directly input the target text material (e.g., a description of the audiobook content). Clicking the generate control indicated by 203 displays the following... Figure 2 The interface shown in Figure (B) is 204, which indicates the generation of an animated video based on the target text material and the target audio material.

[0101] Understandably, when a user clicks the regenerate control indicated by 205, a different animated video can be regenerated based on the target text and audio materials.

[0102] It should be noted that the above application scenarios are merely examples, and the video generation method provided in this embodiment can also be applied to other scenarios, which are not limited here.

[0103] As one implementation method, the method provided in this application can be applied to Figure 3 The implementation environment shown includes a terminal 301, a network 302, and a server 303. The network 302 involved in this application uses standard communication technologies and / or protocols, typically the Internet, but can also be any network, including but not limited to Bluetooth, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), mobile, private networks, or any combination of virtual private networks. In some embodiments, customized or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0104] The terminal 301 involved in this application includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, smart voice interaction devices, virtual reality devices, smart home appliances, vehicle terminals, and aircraft. The client 3011 is deployed on the terminal 301. The client 3011 can run on the terminal 301 via a browser, or via a standalone application (APP) or mini-program.

[0105] The server 303 involved in this application can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence (AI) platforms. It should be noted that this application uses the configuration and deployment of the video generation model on server 303 as an example for illustration. In some embodiments, the configuration of the video generation model can also be deployed on terminal 301. In some embodiments, part of the video generation model's configuration is deployed on terminal 301, and part of its configuration is deployed on server 303.

[0106] In the above implementation environment, in step A1, the user uploads the target audio material and inputs the corresponding target text material through client 3011. In step A2, terminal 301 sends the target audio material and target text material to server 303 through network 302. In step A3, server 303 extracts features from the target audio material to obtain audio description information related to the target audio material. In step A4, server 303 obtains K candidate videos based on the target text material and audio description information using a video generation model. In step A5, server 303 edits the K candidate videos according to the target audio material to obtain the target composite video. In step A6, server 303 sends the target composite video to terminal 301 through network 303. In step A7, terminal 301 displays the target composite video through client 3011.

[0107] As another implementation method, the method provided in this application can be applied to Figure 4The implementation environment shown includes terminal 401. Terminal 401 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, smart voice interaction devices, virtual reality devices, smart home appliances, vehicle terminals, and aircraft. Client 4011 and the video generation model are both deployed on terminal 401. Client 4011 can run on terminal 401 via a browser, or via a standalone app or mini-program.

[0108] In the above implementation environment, in step B1, the user uploads the target audio material and inputs the corresponding target text material through client 4011. In step B2, terminal 401 extracts features from the target audio material to obtain audio description information related to the target audio material. In step B3, based on the target text material and audio description information, terminal 401 obtains K candidate videos through a video generation model. In step B4, terminal 401 edits the K candidate videos according to the target audio material to obtain the target synthesized video. In step B5, terminal 401 displays the target synthesized video through client 4011.

[0109] Based on the above introduction, the method for video generation in this application will be described below. Please refer to [link / reference]. Figure 5 The video generation method in this application embodiment can be completed independently by the server, independently by the terminal, or jointly by the terminal and the server. The method provided in this application includes:

[0110] S501. Obtain target audio material, wherein the target audio material includes at least one of music and speech;

[0111] In one or more embodiments, the target audio material is audio material used in the video generation process. For example, when the target audio material is a song, it includes music; if the song contains lyrics, it also includes speech. For example, when the target audio material is the audio of an audiobook, it includes speech.

[0112] Specifically, the target audio material can be audio material uploaded by the user from their local device, or audio material obtained based on an audio link (i.e., a Uniform Resource Locator (URL) address) entered by the user, or audio material searched based on keywords entered by the user (e.g., song title).

[0113] S502. Obtain the target text material corresponding to the target audio material;

[0114] In one or more embodiments, the target text material is text material used in the video generation process, and the target text material is associated with the target audio material. The target text material can be used to describe what information is needed to generate the video based on the target audio material; for example, the target text material may include lyrics, synopsis, theme, etc.

[0115] Specifically, the target text material can be text content manually entered by the user, or text content obtained after speech recognition of the target audio material. The target text material is a text prompt, such as, "Three people were chatting and laughing, then turned right together, and the two on the right squatted down, while the person on the left pointed at the two on the right."

[0116] S503. Extract features from the target audio material to obtain audio description information related to the target audio material, wherein the audio description information includes at least one of timbre information, beat information, rhythm information, intensity information, emotion information, spectrum information and pitch information;

[0117] In one or more embodiments, a music analysis engine can be invoked to analyze the target audio material to extract audio description information related to the target audio material. The extracted audio description information includes, but is not limited to, at least one of timbre information, beat information, rhythm information, intensity information, emotional information, spectrum information, pitch information, melody information, structural information, and harmony information.

[0118] Specifically, timbre information describes the unique quality of a sound. Meter information is the unit used to measure rhythm. Rhythm information refers to the variations and repetitions of the loudness and softness of sound beats in music. Intensity information describes the strength of a sound. By analyzing the meter, intensity, and rhythm information of an audio signal, the dynamic characteristics and dynamic range of music can be identified; that is, changes in loudness can affect the intensity and expression of emotion. Emotional information represents the emotional state of audio, such as joy or sadness. Spectral information describes the intensity distribution of different frequency components in an audio signal. Pitch information represents the highness or lowness of a sound and can be obtained using autocorrelation algorithms, fundamental frequency estimation (YIN) algorithms, or harmonic analysis-based methods. Melody information is an organic combination of many basic musical elements, including but not limited to mode, rhythm, meter, dynamics, and timbre. Structural information represents musical structure, such as poetic forms, choruses, and bridges. Harmonic information includes chord progressions and tonality, reflecting the harmonic atmosphere of the music.

[0119] S504. Based on the target text material and audio description information, obtain K candidate videos through a video generation model, wherein each candidate video includes at least one video segment, and K is an integer greater than 1.

[0120] In one or more embodiments, the target text material and audio description information are used as input to the video generation model, and the video generation model outputs K candidate videos, each candidate video including several temporally ordered video segments. The video segments can be divided based on a fixed duration, keyframes, or perspective changes.

[0121] Specifically, video generation models can be text-to-video (T2V) models, such as the "Sora" model. They can also be generative adversarial networks (GANs), variational autoencoders (VAEs), or other text-to-graph models. The Sora model, for example, can generate videos and images of varying durations, aspect ratios, and resolutions based on text descriptions. The Sora model is developed based on a diffusion model and a Transformer model architecture. It utilizes the Transformer model to capture high-level abstractions and contextual information from the input data, while the diffusion model progressively removes noise and reconstructs the original image. The combination of these two approaches enables accurate conversion and creation from text to complex, dynamic visual content.

[0122] For easier understanding, please refer to Figure 6 , Figure 6 This is a schematic diagram of the video generation model in an embodiment of this application. As shown in the figure, the original video indicated by 601 is encoded into the latent space by the visual encoder indicated by 602 to form the spacetime latent patches indicated by 603. These latent latent patches, combined with text information, are trained and generated by the Transformer model using diffusion. The generated latent latent patches are then decoded into the pixel space by the visual decoder indicated by 604, thereby obtaining the candidate video indicated by 605. Here, diffusion refers to a technique used in the generation model, such as a denoising diffusion probability model. Diffusion refers to a process that gradually spreads over time, which helps the model gradually generate realistic data samples.

[0123] S505. Based on the target audio material, edit the K candidate videos to obtain the target synthesized video, wherein the target synthesized video includes video segments from at least one candidate video, at least one candidate video belongs to the K candidate videos, and the background sound of the target synthesized video is generated based on the target audio material.

[0124] In one or more embodiments, the video segments in the candidate videos have a temporal sequence that corresponds to the target audio material. Based on this, one possible implementation is to arbitrarily select N video segments from K candidate videos for splicing, and use the audio as background sound to obtain the target synthesized video. Another possible implementation is to selectively select N video segments from K candidate videos based on the audio description information corresponding to the target audio material, splice them, and use the audio as background sound to obtain the target synthesized video.

[0125] This application provides a method for video generation. Using user-input audio material as a basis, artificial intelligence technology is combined to automatically analyze the audio material and automatically generate corresponding video content based on the audio analysis results. This improves video editing efficiency and enhances the user experience.

[0126] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, obtaining the target audio material may specifically include:

[0127] In response to an upload operation for an audio file, retrieve the target audio material based on the audio file;

[0128] or,

[0129] In response to an input action for an audio link, retrieve the target audio material based on the audio link.

[0130] In one or more embodiments, two methods for obtaining target audio materials are described. As can be seen from the foregoing embodiments, users can directly upload audio files as target audio materials, or they can import target audio materials by entering an audio link. These will be described in detail below with reference to the illustrations.

[0131] Specifically, for ease of understanding, please refer to Figure 7 , Figure 7 This is a schematic diagram of an interface providing the target audio material in an embodiment of this application, such as... Figure 7As shown in Figure (A), one implementation is that when the user clicks the audio upload control indicated by 701, they can select an audio file from their local storage to upload. Another implementation is that when the user enters an audio link in the input box indicated by 702, the relevant audio file is imported based on the audio link. After obtaining the audio file, the entire audio file can be used directly, i.e., the audio file can be used as the target audio material. Alternatively, a segment of audio can be selected from the audio file as the target audio material, i.e., as shown in Figure (A). Figure 7 As shown in Figure (B), users can select a segment of audio as the target audio material by dragging the selection box indicated by 703.

[0132] It should be noted that the term "in response to" in this application refers to the conditions or states upon which the execution of an operation depends, and one or more operations that can be executed when certain conditions or states are met. These operations can be real-time or have a certain delay.

[0133] Secondly, this application provides two methods for acquiring target audio materials. These methods enable users to upload target audio materials in different ways, demonstrating that diverse upload methods can meet the needs and preferences of different users, thereby improving the user experience.

[0134] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, obtaining the target text material corresponding to the target audio material may specifically include:

[0135] In response to input operations on text content, the text content is treated as the target text content corresponding to the target audio content;

[0136] or,

[0137] In response to a speech recognition operation targeting a target audio material, obtain the target text material corresponding to the target audio material.

[0138] In one or more embodiments, two methods for obtaining target text materials are described. As can be seen from the foregoing embodiments, users can directly input text as target text materials, or they can obtain target text materials by performing automatic speech recognition (ASR) on target audio materials. These will be described in detail below with reference to the illustrations.

[0139] Specifically, for ease of understanding, please refer to Figure 8 , Figure 8 This is a schematic diagram of an interface providing target text material in an embodiment of this application, such as... Figure 8As shown in Figure (A), one implementation method is that when the user enters text content in the input box indicated by 801, the text content is used as the target text content corresponding to the target audio content. Figure 8 As shown in Figure (B), another implementation method is that after importing the target audio material (e.g., a song), you can also click the audio recognition control indicated by 802, thereby performing a speech recognition operation on the target audio material to obtain the target text material (e.g., lyrics).

[0140] It should be noted that in practical applications, a chatgenerative pretrained transformer (ChatGPT) can also be used to understand the text content input by the user, thereby optimizing the target audio material.

[0141] Secondly, this application provides two methods for obtaining the target text material. Through these methods, users can choose the method of providing the target text material according to their actual needs, thereby meeting the needs and preferences of different users and improving the user experience.

[0142] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, feature extraction is performed on the target audio material to obtain audio description information related to the target audio material, which may specifically include:

[0143] The target audio material is preprocessed to obtain T audio frames, where T is an integer greater than 1;

[0144] Perform a Fourier transform on each of the T audio frames to obtain the spectrum of each audio frame;

[0145] The power spectrum is obtained by taking the square of the modulus of the spectrum of each audio frame;

[0146] The power spectrum is filtered by a Mel filter bank to obtain the filtering result corresponding to each Mel filter. The Mel filter bank includes at least one Mel filter.

[0147] Logarithmic compression and discrete cosine transform are performed on the filtering results corresponding to each Mel filter to obtain the timbre information corresponding to the target audio material, where the timbre information is represented by Mel frequency cepstral coefficients.

[0148] In one or more embodiments, a method for extracting timbre information is described. As can be seen from the foregoing embodiments, timbre information can be extracted by calling a music analysis engine to analyze the target audio material. The process of extracting timbre information based on the target audio material will be described below with reference to the illustrations.

[0149] Specifically, for ease of understanding, please refer to Figure 9 , Figure 9 This is a schematic diagram illustrating the extraction of timbre information in an embodiment of this application. As shown in the figure, the target audio material is first preprocessed to divide the continuous audio signal into discrete parts, resulting in T audio frames. The preprocessing includes pre-emphasis, framing, and windowing. Pre-emphasis refers to boosting the high-frequency components using a high-pass filter to compensate for the attenuation of high-frequency components caused by the channel response. Framing involves dividing the long audio signal into at least one short frame, typically with a frame length of 20 to 30 milliseconds. Windowing involves applying a window to each frame of the signal; commonly used window functions include the Hamming window and the Hanning window.

[0150] Next, a Fast Fourier Transform (FFT) is performed on each audio frame to convert the time-domain signal into a frequency-domain signal. In the frequency domain, the frequency composition of the audio signal can be observed more intuitively. The spectrum shows the energy distribution of the audio signal at different frequencies. Through the Fourier Transform, it is possible to determine which frequency components are present in the audio signal and the intensity of these components, which is crucial for analyzing the timbre, pitch, harmonics, and other characteristics of the audio.

[0151] The power spectrum is obtained by taking the square of the modulo of the audio frame's spectrum. Taking the square modulo means squaring a number and then taking the modulo of the result. The power spectrum reflects the power distribution of the audio signal at different frequencies. The larger the power spectrum value, the stronger the energy at that frequency. The power spectrum provides insight into the energy contribution of the audio signal at various frequencies, which is important for analyzing characteristics such as audio intensity and loudness. The power spectrum is obtained by processing the spectrum and provides input data for subsequent filtering using a Mel-filter bank.

[0152] Mel filter banks simulate nonlinear characteristics and typically consist of at least one bandpass filter, each covering a specific frequency range. By filtering the power spectrum, frequency regions sensitive to the human ear can be highlighted while suppressing insensitive regions. The filtering result corresponding to each Mel filter reflects the energy distribution of the audio signal within a specific frequency range after being weighted by human perception. These filtering results better match how the human ear perceives sound, helping to extract more meaningful audio features.

[0153] Logarithmic compression of the filtering results is performed to compress the large dynamic range into a smaller range, making the values ​​more stable and easier to process, and highlighting important features.

[0154] Performing a discrete cosine transform (DCT) on the filtered result after logarithmic compression converts the data to another space, yielding MFCCs. MFCCs capture the shape of the short-time power spectrum of audio signals, which corresponds to human auditory characteristics; therefore, they can be used to describe timbre. Extracting MFCCs converts the audio signal into a set of digital features, facilitating subsequent processing and analysis.

[0155] Secondly, this application provides a method for extracting timbre information. Through this method, feature extraction is performed on the target audio material to obtain timbre information. This timbre information is then applied to video synthesis, making the synthesized video more consistent with the timbre characteristics, thereby improving the video generation effect.

[0156] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, feature extraction is performed on the target audio material to obtain audio description information related to the target audio material, which may specifically include:

[0157] Convert the target audio material into a time-domain signal;

[0158] Perform a Fourier transform on the time-domain signal to obtain the frequency-domain signal;

[0159] Generate a spectrum diagram based on the frequency domain signal;

[0160] The peak positions in the spectrum are obtained by analyzing the spectrum using a peak detection algorithm.

[0161] The beat information corresponding to the target audio material is determined based on the peak position in the spectrogram.

[0162] In one or more embodiments, a method for extracting beat information is described. As can be seen from the foregoing embodiments, beat information can be extracted by calling a music analysis engine to analyze the target audio material. The process of extracting beat information based on the target audio material will be described below with reference to the illustrations.

[0163] Specifically, for ease of understanding, please refer to Figure 10 , Figure 10 This is a schematic diagram illustrating the extraction of beat information in an embodiment of this application. As shown in the figure, firstly, the target audio material is converted into a time-domain signal, which reflects the changes in audio over time. For example, for a piece of music audio, the time-domain signal can show the fluctuations in the audio, changes in rhythm, etc.

[0164] Next, the time-domain signal is converted into a frequency-domain signal. The frequency-domain signal reveals the distribution of different frequency components in the audio. Through Fourier transform, we can understand which frequencies are present in the audio and the intensity of these frequency components. Features are extracted from the frequency-domain representation, such as spectral centroid, spectral flatness, and spectral roll-off; these features can also describe different properties of the target audio material.

[0165] A spectrogram visualizes a frequency-domain signal, typically plotting frequency on the x-axis and signal intensity on the y-axis. Based on this, peak detection algorithms can determine the locations of peaks in the spectrogram. Determining these peak locations provides crucial information for subsequent beat determination. In audio, beats are usually associated with the periodic occurrence of specific frequencies. By analyzing the distribution of peak locations (e.g., using autocorrelation analysis, peak clustering, etc.), the beat information of the audio can be inferred.

[0166] It should be noted that peak detection algorithms are methods used to find local maxima in data. In a spectrogram, peak locations typically correspond to frequency components with higher energy in the audio.

[0167] Secondly, this application provides a method for extracting beat information. Using this method, during video generation, the beat information based on the audio can achieve synchronization between audio and video, enhancing the video's expressiveness. Corresponding visual effects, such as flashing lights or dynamic graphics, can be added based on the audio beat information to make the video and music more coordinated, thereby improving the video generation effect.

[0168] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, feature extraction is performed on the target audio material to obtain audio description information related to the target audio material, which may specifically include:

[0169] Convert the target audio material into a time-domain signal;

[0170] Perform a Fourier transform on the time-domain signal to obtain the frequency-domain signal;

[0171] Generate a spectrum diagram based on the frequency domain signal;

[0172] The intensity information corresponding to the target audio material is obtained from the spectrogram. The intensity information includes the intensity of different frequency components in the frequency domain signal.

[0173] In one or more embodiments, a method for extracting intensity information is described. As can be seen from the foregoing embodiments, beat information can be extracted by calling a music analysis engine to analyze the target audio material. The process of extracting intensity information based on the target audio material will be described below with reference to the figures.

[0174] Specifically, for ease of understanding, please refer to Figure 11 , Figure 11 This is a schematic diagram illustrating the extraction of intensity information in an embodiment of this application. As shown in the figure, firstly, the target audio material is converted into a time-domain signal, which reflects the changes in audio over time. Next, a Fourier transform is used to convert the time-domain signal into a frequency-domain signal. Based on the frequency-domain signal, a corresponding spectrogram is obtained. The spectrogram can help analyze the energy distribution (i.e., intensity) of the audio signal at different frequencies.

[0175] Secondly, this application provides a method for extracting intensity information. Using this method, feature extraction is performed on the target audio material to obtain intensity information. This intensity information is then applied to video synthesis, making the synthesized video more consistent with intensity characteristics, thereby improving the video generation effect.

[0176] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, feature extraction is performed on the target audio material to obtain audio description information related to the target audio material, which may specifically include:

[0177] Based on the target audio material, an emotion probability distribution is obtained through an emotion recognition model, wherein the emotion probability distribution includes at least one probability value;

[0178] Based on the emotional probability distribution, determine the emotional information corresponding to the target audio material.

[0179] In one or more embodiments, a method for extracting emotional information is described. As can be seen from the foregoing embodiments, since specific acoustic features are related to emotional states, an emotion recognition model can be invoked to perform emotion analysis on the target audio material, thereby obtaining the emotional information corresponding to the target audio material.

[0180] Specifically, the emotion probability distribution output by the emotion recognition model can be represented as a discrete result, that is, the discrete result classifies emotions into a set of discrete categories, such as happiness, sadness, and anger. The emotion probability distribution includes at least one probability value, each representing the likelihood that the target audio material belongs to a specific emotion category. Assuming the emotion probability distribution is (0.3, 0.1, 0.6), where 0.3 represents the probability value of the target audio material belonging to the "happiness" category, 0.1 represents the probability value of the target audio material belonging to the "anger" category, and 0.6 represents the probability value of the target audio material belonging to the "sadness" category, it can be seen that the probability value belonging to the "sadness" category is the highest. Therefore, the emotion information corresponding to the target audio material is determined to be of the "sadness" category. Furthermore, the emotion probability distribution output by the emotion recognition model can also be represented as a continuous result, that is, the continuous result maps emotion classification to a two-dimensional space, such as pleasant and unpleasant. Another example is activation and inactivation.

[0181] Emotion recognition models can be trained using transfer learning, which involves using a model pre-trained on a large dataset to extract high-level features from audio, and then applying these features to emotion classification. Human-annotated emotion datasets containing audio and corresponding emotion labels are used to train and validate machine learning models.

[0182] It's important to note that emotion recognition models can employ Support Vector Machines (SVMs), which find optimal boundaries in a multi-dimensional space to distinguish different emotion categories. Emotion recognition models can also utilize Deep Neural Networks (DNNs), which can learn complex patterns from audio and perform emotion classification. DNNs include, but are not limited to, Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).

[0183] Secondly, this application provides a method for extracting emotional information. Through this method, rules established based on music theory and psychological research can directly correlate specific acoustic features with emotional states. Therefore, by extracting features from the target audio material and obtaining emotional information, and then applying this emotional information to video synthesis, the synthesized video can better match the emotional tone of the audio, thereby improving the video generation effect.

[0184] Optionally, in the above Figure 5Based on one or more corresponding embodiments, in another optional embodiment provided by this application, K candidate videos are obtained through a video generation model based on the target text material and audio description information, which may specifically include:

[0185] The target text material is processed using a text encoding model to obtain the first text feature vector;

[0186] The audio description information is characterized to obtain an audio feature vector;

[0187] Generate the target feature vector based on the text feature vector and the audio feature vector;

[0188] Based on the target feature vector, K candidate videos are obtained through a video generation model.

[0189] In one or more embodiments, a method for predicting candidate videos based on a video generation model is described. As can be seen from the foregoing embodiments, audio description information includes different types of information, and based on this, feature engineering is used to characterize this information. For example, one-hot encoding can be used to encode emotional information.

[0190] Specifically, a text encoding model is used to feature the target text material, thereby obtaining the first text feature vector. In addition, various types of information in the audio description information need to be encoded to obtain the audio feature vector. Based on this, a target feature vector is generated from the text and audio feature vectors. One implementation method is to use a simple vector concatenation method to sequentially connect the text and audio feature vectors to form the target feature vector. Another implementation method is to fuse the text and audio feature vectors together using weighted fusion or attention mechanisms to form the target feature vector. The target feature vector is then used as input to a video generation model, which outputs K candidate videos.

[0191] It should be noted that the text encoding model involved in this application may be a word embeddings model, or a pre-trained language model based on a transformer architecture (bidirectional encoder representations from transformers, BERT), or other models may be used, which are not limited here.

[0192] Secondly, this application provides a method for predicting candidate videos based on a video generation model. By merging multiple feature vectors into a single feature vector, the model structure can be simplified. This reduces the number of model parameters, lowers model complexity, and improves training efficiency.

[0193] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, K candidate videos are obtained through a video generation model based on the target text material and audio description information, which may specifically include:

[0194] Generate audio description text based on the audio description information;

[0195] The target text material and the audio description text are combined to obtain the comprehensive description text;

[0196] The comprehensive descriptive text is processed using a text encoding model to obtain the second text feature vector;

[0197] Based on the second text feature vector, K candidate videos are obtained through a video generation model.

[0198] In one or more embodiments, another method for predicting candidate videos based on a video generation model is introduced. As can be seen from the foregoing embodiments, audio description information includes different types of information, and based on this, an audio description text can be generated according to the audio description information. For example, the audio description text is "This is a song in 4 / 2 time, the tempo speeds up between 29 and 35 seconds, and the song has a positive and uplifting emotion."

[0199] Specifically, the target text material is concatenated with the generated audio description text to obtain a composite description text. Transition words can be added during the concatenation process to make the resulting composite description text smoother and more natural. This composite description text is then used as input to a text encoding model, which outputs a second text feature vector. Finally, this second text feature vector is used as input to a video generation model, which outputs K candidate videos.

[0200] Secondly, this application provides another method for predicting candidate videos based on a video generation model. By merging the target text material with the audio description text, richer information can be obtained, making feature extraction more accurate. Furthermore, the merged text reduces the impact of differences between texts from different sources during feature extraction, resulting in more consistent and stable extracted features.

[0201] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, K candidate videos are edited according to the target audio material to obtain the target synthesized video, which may specifically include:

[0202] Based on the target audio material, select N video segments from K candidate videos, where N is an integer greater than or equal to 1;

[0203] By splicing together N video clips, the target composite video is obtained.

[0204] In one or more embodiments, a method for editing to obtain a target synthetic video is described. As can be seen from the foregoing embodiments, the candidate video generated based on the target audio material is temporally sequential, that is, each video segment included in the candidate video corresponds to a segment of audio in the target audio material.

[0205] Specifically, at least one video segment can be extracted from each of the K candidate videos, resulting in N video segments. Since each video segment is generated based on audio information from a specific time period in the target audio material, the N video segments can be ordered chronologically. The ordered N video segments are then spliced ​​together to combine different scenes and plots into a coherent target composite video. Different candidate videos can display content from different perspectives; therefore, selecting video segments from different candidate videos for splicing allows for richer camera movement effects in the target composite video.

[0206] Secondly, this application provides a method for editing to obtain a target composite video. By selecting video segments based on the target audio material and splicing them together, the video and audio can better match in content, rhythm, and emotion, thereby enhancing the video's expressiveness and improving the overall video generation effect.

[0207] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, K candidate videos are edited according to the target audio material to obtain the target synthesized video, which may specifically include:

[0208] Based on the target audio material, select N video segments from K candidate videos, where N is an integer greater than or equal to 1;

[0209] By splicing together N video clips, a final video can be obtained.

[0210] The video to be synthesized is processed according to the visual effect parameters to obtain the target synthesized video. The visual effect parameters include at least one of the following: video duration, video quality, video aspect ratio, video filter, video style, transition effect, and dubbing style.

[0211] In one or more embodiments, another method for editing to obtain the target composite video is described. As can be seen from the foregoing embodiments, visual effect parameters can also be introduced to generate the target composite video, wherein the visual effect parameters are user-defined. The following will describe how to set the visual effect parameters with examples.

[0212] Specifically, for ease of understanding, please refer to Figure 12 , Figure 12 This is a schematic diagram of an interface for setting visual effect parameters in an embodiment of this application. As shown in the figure, 1201 indicates the video duration of the target composite video. 1202 indicates a video quality selection control, which is used to select the video quality of the target composite video (e.g., high definition, ultra-high definition, standard definition, etc.). 1203 indicates a video aspect ratio selection control, which is used to select the aspect ratio of the target composite video (e.g., 16:9, 4:3, 1:1, etc.). 1204 indicates a video filter selection control, which is used to select a video filter for the target composite video (e.g., warm color, retro, fresh, cinematic, etc.). 1205 indicates a video style selection control, which is used to select the video style of the target composite video (e.g., artistic and fresh, realistic, cartoon, abstract, etc.). 1206 indicates a transition effect selection control, which is used to select the transition effect of the target composite video (e.g., fade in / out, flash, rotation, etc.). 1207 is used to indicate the dubbing style selection control, where the video style selection control is used to select the dubbing style of the target synthesized video (e.g., gentle, passionate, humorous, etc.).

[0213] Regarding the configuration of the voice-over style, firstly, users can record a voice clip as narration using a voice input device. The voice clip's duration must be less than or equal to the duration of the target audio material. If the user uses real-time recording, after uploading the target audio material, a total recordable duration is provided based on the target audio material's duration, and the user must record voice within that duration. If the user uploads voice clips, after uploading the target audio material, a portion of voice content with the corresponding duration of the target audio material is extracted from the uploaded audio clip. Therefore, the target synthesized video includes not only the target audio material as background sound but also the voice-over content.

[0214] Secondly, this application provides another method for editing to obtain the target composite video. By selecting video segments and splicing them based on the target audio material, the video and audio can better match in terms of content, rhythm, and emotion, thereby enhancing the video's expressiveness and improving the overall video generation effect. Simultaneously, combining visual effect parameters can also improve the flexibility of video creation.

[0215] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, selecting N video segments from K candidate videos according to the target audio material may specifically include:

[0216] For each of the K candidate videos, keyframe recognition is performed to obtain a set of video segments. The set of video segments includes each video segment included in each of the K candidate videos, and the video segments are determined based on keyframes.

[0217] Based on the audio description information, select N video clips from the video clip set that meet the time dimension of the target audio material.

[0218] In one or more embodiments, a method for synchronizing audio and video is described. As described in the foregoing embodiments, keyframe identification is performed on candidate videos to divide them into several video segments. These video segments are then added to a video media library for selection.

[0219] Specifically, for ease of understanding, please refer to Table 1, which is a schematic diagram of the relevant information of the video segments corresponding to the K candidate videos. In this example, K equals 3.

[0220] Table 1

[0221] 0 to 10 seconds 10 to 20 seconds 20 to 30 seconds 30 to 40 seconds Candidate Video 1 Video clip 1A Video clip 1B Video clip 1C Video clip 1D Candidate Video 2 Video clip 2A Video clip 2B Video clip 2C 2D video clip Candidate Video 3 Video clip 3A Video clip 2C Video clip 3C 3D video clips

[0222] As shown in Table 1, taking a target audio clip of 40 seconds as an example, assuming that each candidate video yields 4 video segments after keyframe recognition, each video segment corresponds to a time period. Therefore, a video segment set containing 12 video segments is obtained.

[0223] Based on this, N video clips that meet the time dimension of the target audio material can be selected from the video clip set according to the audio description information. Following the time dimension, first, one video clip is selected from the three video clips in the "0 to 10 seconds" range (e.g., video clip 2A); then, one video clip is selected from the three video clips in the "10 to 20 seconds" range (e.g., video clip 1B); next, one video clip is selected from the three video clips in the "20 to 30 seconds" range (e.g., video clip 3C); and finally, one video clip is selected from the three video clips in the "30 to 40 seconds" range (e.g., video clip 3D). Thus, the N video clips are video clip 2A, video clip 1B, video clip 3C, and video clip 3D. These four video clips are then concatenated sequentially to obtain the target composite video.

[0224] It's important to note that different video clips within the same time period offer different perspectives. Therefore, selecting video clips from various candidate videos can enhance the depth and dynamism of the image. Shooting angles can alter the overall framework, spatial scale, and environmental atmosphere of the scene, thereby better constructing the video's atmosphere.

[0225] In practical applications, synchronizing target audio and video content requires ensuring that changes in the video (e.g., scene transitions, color changes, animation effects) match the rhythm, intensity, and emotional changes of the target audio. The synchronization process includes timeline alignment, event matching, and dynamic adjustment. Timeline alignment involves aligning the timelines of the target audio and video to ensure they are processed on the same timeframe. Event matching involves matching audio analysis results (e.g., rhythm points, emotional changes) with video analysis results (e.g., scene transition points, color changes). Dynamic adjustment uses control algorithms (e.g., proportional-integral-derivative (PID) controllers) to adjust video effects in real time, ensuring they match the rhythm and emotional changes of the audio.

[0226] Furthermore, this application provides a method for synchronizing audio and video. Using this method, candidate videos are segmented into meaningful video segments based on keyframe recognition. Then, based on audio analysis techniques and time series modeling, the video content is closely synchronized with the rhythm, intensity, and emotion of the audio.

[0227] Optionally, in the above Figure 5 In addition to one or more corresponding embodiments, another optional embodiment provided in this application may further include:

[0228] This displays the recommended parameters for the visual effects.

[0229] In one or more embodiments, a method for displaying recommended parameters is described. As can be seen from the foregoing embodiments, visual effect parameters include video quality, video aspect ratio, video filters, video style, transition effects, narrative style, etc. For these visual effect parameters, the system can also provide parameter recommendation information. The following will use recommended video filters and video styles as examples to illustrate the basis for providing recommended parameters based on video style.

[0230] I. Based on the type of the target audio material;

[0231] For example, a type analysis is performed on the target audio material to determine its type. For instance, if the target audio material belongs to the "classical music" type, the recommended parameters for the video style could be "artistic and fresh," and the recommended parameters for the video filter could be "retro." It is evident that the suitable display effect for the video can be inferred based on the characteristics of the audio type.

[0232] II. Content based on the target text material;

[0233] For example, the content of the target text material is analyzed to determine its type. For instance, if the target text material includes words conveying negative emotions, the recommended video style parameter could be "dark," and the recommended video filter parameter could be "film." This demonstrates that the appropriate display effect for the video can be inferred based on the emotional content of the text.

[0234] III. Videos generated based on history;

[0235] For example, a user may have generated more than a threshold (e.g., 10 times) of videos within a certain period (e.g., within the last month). Based on this, the system can determine the user's preferred visual effects by considering the visual effect parameters used in each generated video, thereby generating corresponding recommendation parameters. For instance, if a user has repeatedly selected a cinematic video filter and a cartoon video style in the past, the recommended parameter for the video style could be "cartoon," and the recommended parameter for the video filter could be "cinematic." Thus, the system infers the suitable display effect for a video based on its historically generated videos.

[0236] IV. User-defined preference parameters;

[0237] For example, users can set parameters according to their own preferences. For instance, if a user sets their preferred video style to "cartoon," then the recommended parameter for the video style can be "cartoon."

[0238] Specifically, for ease of understanding, please refer to Figure 13 , Figure 13 This is a schematic diagram of an interface for recommending video styles in an embodiment of this application. As shown in the figure, 1301 is used to indicate the recommended parameters corresponding to the video quality (e.g., high definition). 1302 is used to indicate the recommended parameters corresponding to the video aspect ratio (e.g., 16:9). 1303 is used to indicate the recommended parameters corresponding to the video filter (e.g., retro). 1304 is used to indicate the recommended parameters corresponding to the video style (e.g., realistic). 1305 is used to indicate the recommended parameters corresponding to the transition effect (e.g., fade in / fade out).

[0239] Secondly, this application embodiment provides a method for displaying recommended parameters. Through this method, the system can also analyze and determine suitable recommended parameters for each visual effect parameter, allowing the user to choose. This improves video creation efficiency, optimizes the creation process, and ultimately enhances the user experience.

[0240] Based on the above examples, the following will combine... Figure 14 and Figure 15 This describes the process of obtaining a target synthetic video based on a video generation system. Please refer to [link / reference]. Figure 14 , Figure 14 This is a schematic diagram of the architecture of a video generation system in an embodiment of this application. As shown in the figure, the video generation system includes a user interface (UI) and system modules. The UI provides upload, editing, and sharing / export functions. The system modules provide an audio analysis engine, a video generation model, an automated synchronization module, and rendering and output functions.

[0241] For example, if the audio is music, please refer to [link / reference]. Figure 15 , Figure 15 This is a schematic diagram illustrating the overall process of the video generation method in this application embodiment. As shown in the figure, the upload function provided by the UI allows users to upload music, describe video content, and perform basic editing operations. The audio analysis engine is responsible for analyzing the uploaded music and can extract audio description information such as rhythm, melody, harmony, dynamics, and emotion. The audio analysis engine uses signal processing techniques and machine learning algorithms to understand the structure and emotional content of the music.

[0242] Video generation models (e.g., the Sora model) generate several candidate videos based on the audio description information output by the music analysis engine and the target text material.

[0243] The automated synchronization module is responsible for synchronizing music and video content, ensuring that changes in the video (such as scene transitions, color changes, and animation effects) are consistent with the rhythm, intensity, and emotional changes of the music.

[0244] Once the video content is generated and synchronized with the music, the system will render the target composite video. For videos that support high resolution and high frame rate, image processor acceleration and other processing can also be performed. Finally, users can share or export the target composite video.

[0245] Therefore, for ease of understanding, please refer to Figure 16 , Figure 16 This is a schematic diagram of an interactive process of the video generation method in an embodiment of this application, as shown in the figure. Specifically:

[0246] In step S1, the user uploads the target audio material through the UI.

[0247] In step S2, the audio analysis engine is invoked to extract features from the target audio material. The extracted features include, but are not limited to, timbre information, intensity information, etc., which will not be elaborated here.

[0248] In step S3, the user can also provide a piece of text as the target text material.

[0249] In step S4, the user can trigger the video generation operation through the UI. Alternatively, the audio analysis engine can trigger the video generation operation after providing the target text material.

[0250] In step S5, the video generation model generates several candidate videos in batches based on the target audio and text materials provided by the user. These candidate videos share consistent content.

[0251] In step S6, the multiple candidate videos generated in batches are used as a video material library, and an automated synchronization module is provided.

[0252] In step S7, the audio analysis engine is invoked to perform audio analysis on the target audio material.

[0253] In step S8, the beat information, rhythm information, intensity information, and emotional information of the target audio material are obtained through audio analysis engine.

[0254] In step S9, the audio analysis engine synchronizes the beat information, rhythm information, intensity information, and emotional information to the automated synchronization module.

[0255] In step S10, the automated synchronization module matches video clips from the video material library with target audio material. Similar to AI-assisted video editing functions, matching is achieved through keyframe extraction (e.g., scene changes, action climaxes) and keyframe analysis (e.g., edge detection, color change detection, motion detection).

[0256] In step S11, a target composite video is rendered.

[0257] In step S12, the target composite video is output as a draft video.

[0258] In step S13, the user can preview the target composite video and edit and optimize it.

[0259] In step S14, the user can also export or share the final video.

[0260] The video generation apparatus of this application is described in detail below. Please refer to [link / reference]. Figure 17 , Figure 17This is a schematic diagram of one embodiment of the video generation apparatus in this application. The video generation apparatus 170 includes:

[0261] The acquisition module 1701 is used to acquire target audio material, wherein the target audio material includes at least one of music and speech;

[0262] The acquisition module 1701 is also used to acquire the target text material corresponding to the target audio material;

[0263] The extraction module 1702 is used to extract features from the target audio material to obtain audio description information related to the target audio material. The audio description information includes at least one of timbre information, beat information, rhythm information, intensity information, emotion information, spectrum information, and pitch information.

[0264] The acquisition module 1701 is also used to acquire K candidate videos based on the target text material and audio description information through a video generation model, wherein each candidate video includes at least one video segment, and K is an integer greater than 1;

[0265] The editing module 1703 is used to edit K candidate videos based on the target audio material to obtain a target composite video. The target composite video includes video segments from at least one candidate video, at least one candidate video belongs to the K candidate videos, and the background sound of the target composite video is generated based on the target audio material.

[0266] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 170 provided in this application,

[0267] The acquisition module 1701 is specifically used to respond to the upload operation of an audio file and obtain the target audio material based on the audio file;

[0268] or,

[0269] In response to an input action for an audio link, retrieve the target audio material based on the audio link.

[0270] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 170 provided in this application,

[0271] The acquisition module 1701 is specifically used to respond to input operations on text content and treat the text content as the target text material corresponding to the target audio material;

[0272] or,

[0273] In response to a speech recognition operation targeting a target audio material, obtain the target text material corresponding to the target audio material.

[0274] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 170 provided in this application,

[0275] Extraction module 1702 is specifically used to preprocess the target audio material to obtain T audio frames, where T is an integer greater than 1;

[0276] Perform a Fourier transform on each of the T audio frames to obtain the spectrum of each audio frame;

[0277] The power spectrum is obtained by taking the square of the modulus of the spectrum of each audio frame;

[0278] The power spectrum is filtered by a Mel filter bank to obtain the filtering result corresponding to each Mel filter. The Mel filter bank includes at least one Mel filter.

[0279] Logarithmic compression and discrete cosine transform are performed on the filtering results corresponding to each Mel filter to obtain the timbre information corresponding to the target audio material, where the timbre information is represented by Mel frequency cepstral coefficients.

[0280] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 170 provided in this application,

[0281] Extraction module 1702 is specifically used to convert the target audio material into a time-domain signal;

[0282] Perform a Fourier transform on the time-domain signal to obtain the frequency-domain signal;

[0283] Generate a spectrum diagram based on the frequency domain signal;

[0284] The peak positions in the spectrum are obtained by analyzing the spectrum using a peak detection algorithm.

[0285] The beat information corresponding to the target audio material is determined based on the peak position in the spectrogram.

[0286] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 170 provided in this application,

[0287] Extraction module 1702 is specifically used to convert the target audio material into a time-domain signal;

[0288] Perform a Fourier transform on the time-domain signal to obtain the frequency-domain signal;

[0289] Generate a spectrum diagram based on the frequency domain signal;

[0290] The intensity information corresponding to the target audio material is obtained from the spectrogram. The intensity information includes the intensity of different frequency components in the frequency domain signal.

[0291] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 170 provided in this application,

[0292] The extraction module 1702 is specifically used to obtain the emotion probability distribution based on the target audio material through an emotion recognition model, wherein the emotion probability distribution includes at least one probability value;

[0293] Based on the emotional probability distribution, determine the emotional information corresponding to the target audio material.

[0294] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 170 provided in this application,

[0295] The acquisition module 1701 is specifically used to process the target text material through a text encoding model to obtain the first text feature vector;

[0296] The audio description information is characterized to obtain an audio feature vector;

[0297] Generate the target feature vector based on the text feature vector and the audio feature vector;

[0298] Based on the target feature vector, K candidate videos are obtained through a video generation model.

[0299] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 170 provided in this application,

[0300] The acquisition module 1701 is specifically used to generate audio description text based on audio description information;

[0301] The target text material and the audio description text are combined to obtain the comprehensive description text;

[0302] The comprehensive descriptive text is processed using a text encoding model to obtain the second text feature vector;

[0303] Based on the second text feature vector, K candidate videos are obtained through a video generation model.

[0304] Optionally, in the above Figure 17Based on the corresponding embodiments, in another embodiment of the video generation apparatus 170 provided in this application,

[0305] The editing module 1703 is specifically used to select N video segments from K candidate videos based on the target audio material, where N is an integer greater than or equal to 1;

[0306] By splicing together N video clips, the target composite video is obtained.

[0307] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 170 provided in this application,

[0308] The editing module 1703 is specifically used to select N video segments from K candidate videos based on the target audio material, where N is an integer greater than or equal to 1;

[0309] By splicing together N video clips, a final video can be obtained.

[0310] The video to be synthesized is processed according to the visual effect parameters to obtain the target synthesized video. The visual effect parameters include at least one of the following: video duration, video quality, video aspect ratio, video filter, video style, transition effect, and dubbing style.

[0311] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 170 provided in this application,

[0312] The editing module 1703 is specifically used to perform keyframe recognition on each of the K candidate videos to obtain a set of video segments. The set of video segments includes each video segment included in each of the K candidate videos, and the video segments are determined based on keyframes.

[0313] Based on the audio description information, select N video clips from the video clip set that meet the time dimension of the target audio material.

[0314] This application also provides a computer device, such as... Figure 18 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. In the embodiments of this application, a mobile phone is used as an example for illustration:

[0315] Figure 18 This diagram illustrates a partial structural design of a mobile phone related to the computer device provided in an embodiment of this application. (Reference) Figure 18The mobile phone includes components such as a radio frequency (RF) circuit 1810, a memory 1820, an input unit 1830, a display unit 1840, a sensor 1850, an audio circuit 1860, a wireless fidelity (WiFi) module 1870, a processor 1880, and a power supply 1890. Those skilled in the art will understand that... Figure 18 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0316] The following is combined with Figure 18 A detailed introduction to each component of a mobile phone:

[0317] RF circuit 1810 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 1880; additionally, it transmits uplink data to the base station. Typically, RF circuit 1810 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), and a duplexer. Furthermore, RF circuit 1810 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Message Service (SMS).

[0318] The memory 1820 can be used to store software programs and modules. The processor 1880 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1820. The memory 1820 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1820 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0319] The input unit 1830 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1830 may include a touch panel 1831 and other input devices 1832. The touch panel 1831, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1831), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 1831 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 1880, and can also receive and execute commands sent by the processor 1880. In addition, the touch panel 1831 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1831, the input unit 1830 may also include other input devices 1832. Specifically, other input devices 1832 may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a mouse, and a joystick.

[0320] The display unit 1840 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1840 may include a display panel 1841, which may optionally be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, a touch panel 1831 may cover the display panel 1841. When the touch panel 1831 detects a touch operation on or near it, it transmits the information to the processor 1880 to determine the type of touch event. Subsequently, the processor 1880 provides corresponding visual output on the display panel 1841 based on the type of touch event. Although in Figure 18 In this embodiment, the touch panel 1831 and the display panel 1841 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 1831 and the display panel 1841 can be integrated to realize the input and output functions of the mobile phone.

[0321] The mobile phone may also include at least one sensor 1850, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1841 according to the ambient light level, and the proximity sensor can turn off the display panel 1841 and / or the backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0322] Audio circuit 1860, speaker 1861, and microphone 1862 provide an audio interface between the user and the mobile phone. Audio circuit 1860 converts received audio data into electrical signals and transmits them to speaker 1861, where speaker 1861 converts them into sound signals for output. On the other hand, microphone 1862 converts collected sound signals into electrical signals, which are received by audio circuit 1860, converted into audio data, and then processed by processor 1880 before being transmitted via RF circuit 1810 to, for example, another mobile phone, or the audio data can be output to memory 1820 for further processing.

[0323] WiFi is a short-range wireless transmission technology. Mobile phones, through their WiFi modules (1870), can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 18 The WiFi module 1870 is shown, but it is understood that it is not an essential component of a mobile phone and can be omitted as needed without changing the essence of the invention.

[0324] The processor 1880 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes various functions and processes data by running or executing software programs and / or modules stored in the memory 1820, and by calling data stored in the memory 1820. Optionally, the processor 1880 may include one or more processing units; optionally, the processor 1880 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 1880.

[0325] The mobile phone also includes a power supply 1890 (such as a battery) that supplies power to various components. Optionally, the power supply can be logically connected to the processor 1880 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0326] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0327] The steps performed by the computer device in the above embodiments can be based on this Figure 18 The computer device structure shown.

[0328] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0329] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0330] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0331] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0332] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0333] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0334] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0335] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a server or terminal device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0336] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for video generation, characterized in that, include: Acquire target audio material, wherein the target audio material includes at least one of music and speech; Obtain the target text material corresponding to the target audio material; Feature extraction is performed on the target audio material to obtain audio description information related to the target audio material, wherein the audio description information includes at least one of timbre information, beat information, rhythm information, intensity information, emotional information, spectrum information, and pitch information; Based on the target text material and the audio description information, K candidate videos are obtained through a video generation model, wherein each candidate video includes at least one video segment, and K is an integer greater than 1; Based on the target audio material, the K candidate videos are edited to obtain a target synthesized video, wherein the target synthesized video includes video segments from at least one candidate video, the at least one candidate video belongs to the K candidate videos, and the background sound of the target synthesized video is generated based on the target audio material.

2. The method according to claim 1, characterized in that, The acquisition of the target audio material includes: In response to an upload operation for an audio file, the target audio material is obtained based on the audio file; or, In response to an input operation for an audio link, the target audio material is obtained based on the audio link.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the target text material corresponding to the target audio material includes: In response to an input operation on text content, the text content is used as the target text material corresponding to the target audio material; or, In response to a speech recognition operation on the target audio material, the target text material corresponding to the target audio material is obtained.

4. The method according to any one of claims 1 to 3, characterized in that, The step of extracting features from the target audio material to obtain audio description information related to the target audio material includes: The target audio material is preprocessed to obtain T audio frames, where T is an integer greater than 1; Perform a Fourier transform on each of the T audio frames to obtain the spectrum of each audio frame; The power spectrum is obtained by taking the square of the modulus of the spectrum of each audio frame; The power spectrum is filtered by a Mel filter bank to obtain the filtering result corresponding to each Mel filter, wherein the Mel filter bank includes at least one Mel filter. Logarithmic compression and discrete cosine transform are performed on the filtering results corresponding to each Mel filter to obtain the timbre information corresponding to the target audio material, wherein the timbre information is represented by Mel frequency cepstral coefficients.

5. The method according to any one of claims 1 to 3, characterized in that, The step of extracting features from the target audio material to obtain audio description information related to the target audio material includes: Convert the target audio material into a time-domain signal; Perform a Fourier transform on the time-domain signal to obtain the frequency-domain signal; Generate a spectrum diagram based on the frequency domain signal; The peak positions in the spectrum are obtained by analyzing the spectrum using a peak detection algorithm. The beat information corresponding to the target audio material is determined based on the peak position in the spectrogram.

6. The method according to any one of claims 1 to 3, characterized in that, The step of extracting features from the target audio material to obtain audio description information related to the target audio material includes: Convert the target audio material into a time-domain signal; Perform a Fourier transform on the time-domain signal to obtain the frequency-domain signal; Generate a spectrum diagram based on the frequency domain signal; The intensity information corresponding to the target audio material is obtained from the spectrogram, wherein the intensity information includes the intensity of different frequency components in the frequency domain signal.

7. The method according to any one of claims 1 to 3, characterized in that, The step of extracting features from the target audio material to obtain audio description information related to the target audio material includes: Based on the target audio material, an emotion probability distribution is obtained through an emotion recognition model, wherein the emotion probability distribution includes at least one probability value; Based on the emotional probability distribution, the emotional information corresponding to the target audio material is determined.

8. The method according to any one of claims 1 to 7, characterized in that, The step of obtaining K candidate videos based on the target text material and the audio description information through a video generation model includes: The target text material is processed using a text encoding model to obtain a first text feature vector; The audio description information is subjected to feature processing to obtain an audio feature vector; Generate a target feature vector based on the text feature vector and the audio feature vector; Based on the target feature vector, the K candidate videos are obtained through the video generation model.

9. The method according to any one of claims 1 to 7, characterized in that, The step of obtaining K candidate videos based on the target text material and the audio description information through a video generation model includes: Based on the audio description information, generate audio description text; The target text material and the audio description text are concatenated to obtain the comprehensive description text; The comprehensive description text is processed using a text encoding model to obtain a second text feature vector; Based on the second text feature vector, the K candidate videos are obtained through the video generation model.

10. The method according to any one of claims 1 to 9, characterized in that, The step of editing the K candidate videos based on the target audio material to obtain the target synthesized video includes: Based on the target audio material, select N video segments from the K candidate videos, where N is an integer greater than or equal to 1; The N video segments are spliced ​​together to obtain the target composite video.

11. The method according to any one of claims 1 to 9, characterized in that, The step of editing the K candidate videos based on the target audio material to obtain the target synthesized video includes: Based on the target audio material, select N video segments from the K candidate videos, where N is an integer greater than or equal to 1; The N video segments are spliced ​​together to obtain the video to be synthesized; The video to be synthesized is processed according to visual effect parameters to obtain the target synthesized video. The visual effect parameters include at least one of video duration, video quality, video aspect ratio, video filter, video style, transition effect, and dubbing style.

12. The method according to claim 10 or 11, characterized in that, The step of selecting N video segments from the K candidate videos based on the target audio material includes: Keyframe recognition is performed on each of the K candidate videos to obtain a set of video segments, wherein the set of video segments includes each video segment included in each of the K candidate videos, and the video segments are determined based on keyframes; Based on the audio description information, select N video clips from the video clip set that satisfy the time dimension of the target audio material.

13. A video generation apparatus, characterized in that, include: An acquisition module is used to acquire target audio material, wherein the target audio material includes at least one of music and speech; The acquisition module is also used to acquire the target text material corresponding to the target audio material; The extraction module is used to extract features from the target audio material to obtain audio description information related to the target audio material, wherein the audio description information includes at least one of timbre information, beat information, rhythm information, intensity information, emotional information, spectrum information, and pitch information; The acquisition module is further configured to acquire K candidate videos based on the target text material and the audio description information through a video generation model, wherein each candidate video includes at least one video segment, and K is an integer greater than 1; The editing module is used to edit the K candidate videos according to the target audio material to obtain a target synthesized video, wherein the target synthesized video includes video segments from at least one candidate video, the at least one candidate video belongs to the K candidate videos, and the background sound of the target synthesized video is generated based on the target audio material.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images