An AIGC music video generation method, device and equipment and storage medium
By performing semantic analysis and feature extraction on audio data and text prompts, and using multimodal fusion and Transformer architecture to generate music video clips, the problem of the disconnect between visual content and musical mood in existing technologies is solved, and deep alignment and high-quality generation of music videos are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- MALANSHAN AUDIO & VIDEO LABORATORY
- Filing Date
- 2026-02-26
- Publication Date
- 2026-05-29
AI Technical Summary
Existing music video generation technologies cannot understand the emotions and narrative content expressed in lyrics or music, resulting in a disconnect between the visual content and the musical mood. When generating longer videos, they lack a coherent theme, style, and narrative logic, making it difficult to form a complete work.
By acquiring target audio data and text prompts, semantic parsing and feature extraction are performed. Music video clips are generated using a sequence generation model based on multimodal fusion and the Transformer architecture. Finally, video diffusion models are used for splicing and synthesis to achieve deep alignment between audio semantics and visual content.
It achieves deep alignment between musical semantics and visual content, improves the coherence and narrative logic of generated videos, ensures macro-level semantic alignment and micro-level beat alignment of music videos, and solves the controllability problem in the generation process.
Smart Images
Figure CN122120570A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video generation, and in particular to an AIGC music video generation method, apparatus, device, and storage medium. Background Technology
[0002] With the rise of short video platforms, the demand for music-driven video generation is increasing. Existing music video generation technologies mainly rely on low-level audio features (such as beat and volume) for simple "beat-matching" editing. However, simply cutting based on beat or loudness fails to capture the emotions and narrative content expressed in the lyrics or music, resulting in a disconnect between the visual content and the musical mood. When generating longer videos (e.g., over 30 seconds), random segment splicing is often used, lacking a coherent theme, style, and narrative logic, making it difficult to create a complete work. Summary of the Invention
[0003] In view of this, the purpose of this invention is to provide an AIGC music video generation method, apparatus, device, and storage medium, which can achieve deep alignment of audio semantics and visual content during video generation. The specific solution is as follows: In a first aspect, this application discloses an AIGC music video generation method, including: The process involves acquiring target audio data and text prompts, performing semantic parsing on the text prompts to obtain target text semantic vectors, and using a target audio analysis model to extract features from the target audio data to obtain the target audio semantic vectors corresponding to the target audio data. The target text semantic vector and the target audio semantic vector are fused in a multimodal manner based on a preset semantic vector fusion method to obtain the corresponding target semantic description. The target sequence generation model is then used to output a target semantic segment sequence based on the target semantic description, so as to determine the target control signal based on the target semantic segment sequence. The target sequence generation model is an end-to-end sequence generation model based on the Transformer architecture. The target video diffusion model is used to generate music video segments corresponding to each target semantic segment sequence based on the target control signal, and the music video segments are spliced together to obtain a spliced video. The spliced video is then synthesized with the target audio data to obtain the target music video corresponding to the target audio data and the text prompt words.
[0004] Optionally, obtaining the target audio data and text prompts to be processed includes: The system acquires target audio data, text prompts, and target reference data to be processed, and determines the target video style based on the target reference data; the target reference data includes target reference videos and target reference images. Accordingly, the multimodal fusion of the target text semantic vector and the target audio semantic vector based on the preset semantic vector fusion method includes: Multimodal fusion of the target text semantic vector and the target audio semantic vector is performed based on a preset semantic vector fusion method and the target video style.
[0005] Optionally, the step of using a target audio analysis model to extract features from the target audio data to obtain the target audio semantic vector corresponding to the target audio data includes: The original audio features corresponding to the target audio data are determined using a target audio analysis model. The target audio data is semantically feature extracted using the target audio analysis model based on the audio features, so as to obtain the target audio semantic vector and structural segmentation features corresponding to the target audio data. The original audio features include the audio start point, audio beat, audio speed, and short-time energy curve.
[0006] Optionally, the step of using a target sequence generation model to output a target semantic fragment sequence based on the target semantic description, and then determining the target control signal based on the target semantic fragment sequence, includes: Using a target sequence generation model, a target semantic fragment sequence and corresponding visual descriptive prompts are output based on the target semantic description. The target timeline script is determined based on the target semantic segment sequence and the corresponding visual descriptive cues; the target timeline script includes the duration of each scene, the camera movement method, the color tone of each scene, and the transition method between scenes; The target control signal is determined based on the target timeline script; The target control signal includes a continuous curve control signal and a discrete token control signal. The continuous curve control signal includes an energy curve signal for controlling the light intensity or particle activity of the image, a motion intensity curve for controlling the camera movement speed or the motion amplitude of objects in the image, and a cut probability curve for controlling the possibility of scene switching. The discrete token control signal is a semantic tag signal used to adjust the video generation conditions of the target video diffusion model.
[0007] Optionally, before generating the music video segments corresponding to each of the target semantic segment sequences based on the target control signal using the target video diffusion model, the method further includes: A target ControlNet class structure is embedded in the target video diffusion model to receive the continuous curve control signal in the target control signal using the target ControlNet class structure.
[0008] Optionally, the step of generating music video segments corresponding to each of the target semantic segment sequences based on the target control signal using the target video diffusion model includes: The last frame of the music video segment corresponding to the target semantic segment sequence described above is determined as the target reference frame corresponding to the current target semantic segment sequence; Using a target video diffusion model, the music video segment corresponding to the current target semantic segment sequence is generated based on the target control signal and the target reference frame.
[0009] Optionally, splicing the music video segments to obtain the spliced video includes: Determine the aspect ratio of the target video; The music video segments are spliced together based on the timestamps corresponding to each music video segment to obtain the video to be adjusted, and the aspect ratio of the video to be adjusted is adjusted based on the aspect ratio of the target video to obtain the corresponding spliced video.
[0010] Secondly, this application discloses an AIGC music video generation device, comprising: The semantic vector extraction module is used to acquire the target audio data to be processed and the text prompt words, perform semantic parsing on the text prompt words to obtain the target text semantic vector, and use the target audio analysis model to extract features from the target audio data to obtain the target audio semantic vector corresponding to the target audio data. The control signal determination module is used to perform multimodal fusion of the target text semantic vector and the target audio semantic vector based on a preset semantic vector fusion method to obtain a corresponding target semantic description, and to use a target sequence generation model to output a target semantic segment sequence based on the target semantic description, so as to determine the target control signal based on the target semantic segment sequence; the target sequence generation model is an end-to-end sequence generation model based on the Transformer architecture; The music video generation module is used to generate music video segments corresponding to each target semantic segment sequence based on the target control signal using a target video diffusion model, and to splice the music video segments to obtain a spliced video. The spliced video is then combined with the target audio data to obtain the target music video corresponding to the target audio data and the text prompt words.
[0011] Thirdly, this application discloses an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned AIGC music video generation method.
[0012] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned AIGC music video generation method.
[0013] In this application, when generating AIGC music videos, target audio data and text prompts are acquired. Semantic parsing is performed on the text prompts to obtain target text semantic vectors. A target audio analysis model is used to extract features from the target audio data to obtain target audio semantic vectors corresponding to the target audio data. A multimodal fusion method is used to fuse the target text semantic vectors and the target audio semantic vectors to obtain corresponding target semantic descriptions. A target sequence generation model is used to output target semantic fragment sequences based on the target semantic descriptions, and target control signals are determined based on these target semantic fragment sequences. The target sequence generation model is an end-to-end sequence generation model based on the Transformer architecture. A target video diffusion model is used to generate music video fragments corresponding to each target semantic fragment sequence based on the target control signals. The music video fragments are then spliced together to obtain a spliced video. The spliced video is then synthesized with the target audio data to obtain the target music video corresponding to the target audio data and the text prompts. As can be seen, after obtaining the target audio data and text prompts, this application first performs feature extraction and multimodal vector fusion to obtain a target semantic description that can represent the theme, scene, style, emotional keywords, and other content of the target audio data. Then, the target semantic description is input into the target sequence generation model. Based on the semantic segment generation mechanism of the target sequence generation model, the target semantic description is directly mapped into a sequence of target semantic segments with temporal structure. The target control signal is obtained based on the target semantic segment sequence. Finally, the target video diffusion model is used to generate music video segments corresponding to each target semantic segment sequence based on the target control signal, thus obtaining the final target music video. This achieves "semantic alignment" at the macro level (such as the emotional color change of the verse / chorus) and "beat alignment" at the micro level (such as the action timing at the drum beat), realizing unified control of coarse and fine granularity, solving the controllability problem in the generation process, and achieving deep alignment between music semantics and visual content. Attached Figure Description
[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0015] Figure 1 This is a flowchart of an AIGC music video generation method disclosed in this application; Figure 2 This is a schematic diagram of the overall architecture of a specific semantic-driven AIGC music video generation system disclosed in this application; Figure 3 This is a schematic diagram of a specific AIGC music video generation method disclosed in this application; Figure 4 This is a schematic diagram of a specific hybrid control signal timing curve disclosed in this application; Figure 5 This is a schematic diagram of the structure of an AIGC music video generation device disclosed in this application; Figure 6 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation
[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0017] With the rise of short video platforms, the demand for music-driven video generation is increasing. Existing music video generation technologies mainly rely on low-level audio features (such as beat and volume) for simple "beat-matching" editing. However, simply cutting based on beat or loudness fails to capture the emotions and narrative content expressed in the lyrics or music, resulting in a disconnect between the visual content and the musical mood. When generating longer videos (e.g., over 30 seconds), random segment splicing is often used, lacking a coherent theme, style, and narrative logic, making it difficult to form a complete work. To address these technical issues, this application discloses an AIGC music video generation method that achieves deep alignment between audio semantics and visual content during video generation.
[0018] See Figure 1 As shown, this embodiment of the invention discloses an AIGC music video generation method, including: Step S11: Obtain the target audio data and text prompt words to be processed, perform semantic parsing on the text prompt words to obtain the target text semantic vector, and use the target audio analysis model to extract features from the target audio data to obtain the target audio semantic vector corresponding to the target audio data.
[0019] In this embodiment, as Figure 2As shown, the overall architecture of the semantically driven AIGC music video generation system includes an input interface module, a multimodal analysis module, a semantic fusion generation module, a timing control coding module, a video generation module, and a synthesis output module. Figure 3 As shown, when generating AIGC music videos, the input data obtained, in addition to the target audio data A to be processed and the text prompt P, may also include optional references R and F. In one specific implementation, obtaining the target audio data and text prompts includes: obtaining the target audio data, text prompts, and target reference data, and determining the target video style based on the target reference data; the target reference data includes target reference videos and target reference images. The system supports common audio formats such as MP3 and WAV, as well as input data such as natural language descriptive text, and the received reference videos or images can be used as style guidance.
[0020] In this embodiment, semantic parsing is performed on the input text prompts to extract topic, scene, style, and sentiment keywords as corresponding target text semantic vectors. A target audio analysis model is used to extract features from the target audio data to obtain the target audio semantic vectors corresponding to the target audio data. Specifically, this includes: using the target audio analysis model to determine the original audio features corresponding to the target audio data; and using the target audio analysis model to extract semantic features from the target audio data based on the audio features to obtain the target audio semantic vectors and structural segmentation features corresponding to the target audio data. The original audio features include audio start points, audio beats, audio tempo, and short-time energy curves. In a specific implementation, an audio analysis model (such as Essentia) can be used to extract features from audio A. Specifically, this includes: extracting low-level features (i.e., original audio features) such as Onsets, Beats, BPM, and short-time energy curve E(t); and using a pre-trained audio model (such as CLAP) to extract the target audio semantic feature vector V_{audio} and high-level features such as structural segmentation features (Intro, Verse, Chorus, Outro).
[0021] Step S12: Perform multimodal fusion of the target text semantic vector and the target audio semantic vector based on a preset semantic vector fusion method to obtain the corresponding target semantic description, and use a target sequence generation model to output a target semantic fragment sequence based on the target semantic description, so as to determine the target control signal based on the target semantic fragment sequence; the target sequence generation model is an end-to-end sequence generation model based on the Transformer architecture.
[0022] In this embodiment, if the input includes the target video style, etc., the target text semantic vector and the target audio semantic vector can be multimodally fused based on a preset semantic vector fusion method and the target video style to obtain a corresponding comprehensive semantic description as the target semantic description. For example... Figure 2 As shown, after obtaining the target semantic description, a target sequence generation model is used to output a target semantic fragment sequence based on the target semantic description. The target control signal is then determined based on the target semantic fragment sequence. Specifically, this can include: using the target sequence generation model to output a target semantic fragment sequence and corresponding visual descriptive cues based on the target semantic description; determining a target timeline script based on the target semantic fragment sequence and the corresponding visual descriptive cues; the target timeline script includes shot duration, camera movement method, shot color tone, and shot transition method; and determining the target control signal based on the target timeline script. The target control signal includes a continuous curve control signal and a discrete token control signal. The continuous curve control signal includes an energy curve signal for controlling the intensity of light effects or particle activity in the image, a motion intensity curve for controlling the camera movement speed or the amplitude of object movement in the image, and a cut probability curve for controlling the possibility of scene switching. The discrete token control signal is a semantic tag signal used to adjust the video generation conditions of the target video diffusion model.
[0023] In one specific implementation, such as Figure 4The diagram shows a specific hybrid control signal timing curve provided in this embodiment. The target semantic description is input into an end-to-end sequence generation model (i.e., a target sequence generation model, such as a Transformer-based architecture), outputting a semantic fragment sequence S = {S_1, S_2, ..., S_N}. Each fragment S_i contains a start time t_{start}, an end time t_{end}, and a corresponding visual description Prompt (e.g., "cyberpunk street under neon lights, slow motion"). Based on the semantic fragment sequence, a detailed target timeline script is generated. The target timeline script plans the duration, shot type (push / pull / pan / tilt), color tone, and transition method for each shot. For portrait output mode, the target timeline script automatically adjusts the composition description, emphasizing the central subject. Then, the target timeline script is converted into a hybrid control signal (i.e., a target control signal) C(t) that can be recognized by the generation model. The target control signals include continuous curve control signals and discrete token control signals. Continuous curve control signals include an energy curve signal E(t) for controlling the intensity of light effects or particle activity in the image, a motion intensity curve m(t) for controlling the camera movement speed or the amplitude of object movement in the image, and a cut probability curve p_{cut} for controlling the possibility of scene switching. Discrete token control signals are semantic tag signals used to adjust the video generation conditions of the target video diffusion model, corresponding to… Figure 4 The semantic tags shown in the image (such as "night," "neon," and "loneliness") are injected as conditions into the Cross-Attention layer of the target video diffusion model within a specific time window. Figure 4 As can be seen, this embodiment achieves timeline alignment of "structural paragraphs - semantic tokens - control curves," using curves to control motion intensity and cut probability, and tokens to control scene semantics, thus solving the controllability problem in the generation process. Based on the above process, this embodiment takes "what the music wants to express" as the main thread, performing "semantic alignment" at the macro level (such as the emotional color changes of the verse / chorus) and "beat alignment" at the micro level (such as the action timing at drum beats), achieving unified control of both coarse and fine granularity. This allows the camera language to change with the verse / chorus / bridge and emotional progression of the original audio, thereby significantly improving the semantic consistency, narrative coherence, and visual sophistication of the visuals and music.
[0024] Step S13: Using the target video diffusion model, generate music video segments corresponding to each target semantic segment sequence based on the target control signal, and splice the music video segments to obtain a spliced video. Then, synthesize the spliced video with the target audio data to obtain the target music video corresponding to the target audio data and the text prompt words.
[0025] In this embodiment, before generating music video segments corresponding to each target semantic segment sequence based on the target control signal using the target video diffusion model, the method further includes: embedding a target ControlNet class structure into the target video diffusion model to receive continuous curve control signals in the target control signal using the target ControlNet class structure. Using a pre-trained target video diffusion model (such as AnimateDiff or Stable Video Diffusion), video segments V_i are generated in segments under the constraint of the mixed control signal C(t). The introduction of the ControlNet class structure to receive curve signals ensures that the video dynamics are precisely synchronized with the music rhythm (e.g., the screen vibrates at drum beats).
[0026] In one specific implementation, generating music video segments corresponding to each target semantic segment sequence based on the target control signal using a target video diffusion model may include: determining the last frame of the music video segment corresponding to the previous target semantic segment sequence as the target reference frame corresponding to the current target semantic segment sequence; and generating the music video segment corresponding to the current target semantic segment sequence based on the target control signal and the target reference frame using the target video diffusion model. That is, in this embodiment, to solve the style abruptness problem in long video generation, a "consistency bridging" technique is adopted, using the last frame of the previous segment as the Latent Noise initialization reference (or ImagePrompt) for the generation of the next segment. Simultaneously, the RIFE (Real-Time Intermediate Flow Estimation) algorithm can be used for frame rate upsampling to make the video smoother. The music video segments are then spliced to obtain the spliced video, which may specifically include: determining the aspect ratio of the target video; splicing the music video segments based on the timestamps corresponding to each music video segment to obtain the video to be adjusted; and adjusting the aspect ratio of the video to be adjusted based on the aspect ratio of the target video to obtain the corresponding spliced video. All generated video segments are spliced along the timeline and synthesized with the original audio A.
[0027] For example, for short video platforms, the aspect ratio can be adjusted to 9:16, and the corresponding camera strategy can be adjusted to the camera strategy corresponding to the vertical screen mode. This embodiment can introduce attention mask and motion prior for vertical screen composition, and intelligently crop or complete the generated video to obtain the final target music video, optimizing the short video distribution experience and better meeting the viewing habits of mobile users who prefer "strong rhythm, strong stimulation, and strong theme". On this basis, this embodiment can flexibly adjust the camera strategy (including subject composition, information density, cut frequency, and motion amplitude adaptive with semantics / energy) as needed, and derive multiple platform versions (expandable to 16:9) from the same semantic script, reducing the cost of repeated production and making commercialization more direct. In addition, C2PA standard watermark information can be embedded in the target music video to ensure content traceability. This embodiment significantly reduces the risk of long-term drift and style loss of control by breaking down the long video generation process into "manageable, editable, and regenerable" units, which is especially suitable for scenarios with strong rhythm and strong narrative, such as music videos.
[0028] As can be seen, after obtaining the target audio data and text prompts, this application first performs feature extraction and multimodal vector fusion to obtain a target semantic description that can represent the theme, scene, style, emotional keywords, and other content of the target audio data. Then, the target semantic description is input into the target sequence generation model. Based on the semantic segment generation mechanism of the target sequence generation model, the target semantic description is directly mapped into a sequence of target semantic segments with temporal structure. The target control signal is obtained based on the target semantic segment sequence. Finally, the target video diffusion model is used to generate music video segments corresponding to each target semantic segment sequence based on the target control signal, thus obtaining the final target music video. This achieves "semantic alignment" at the macro level (such as the emotional color change of the verse / chorus) and "beat alignment" at the micro level (such as the action timing at the drum beat), realizing unified control of coarse and fine granularity, solving the controllability problem in the generation process, and achieving deep alignment between music semantics and visual content.
[0029] See Figure 5 As shown, this application discloses an AIGC music video generation device, comprising: The semantic vector extraction module 11 is used to acquire the target audio data to be processed and the text prompt words, perform semantic parsing on the text prompt words to obtain the target text semantic vector, and use the target audio analysis model to extract features from the target audio data to obtain the target audio semantic vector corresponding to the target audio data. The control signal determination module 12 is used to perform multimodal fusion of the target text semantic vector and the target audio semantic vector based on a preset semantic vector fusion method to obtain a corresponding target semantic description, and to use a target sequence generation model to output a target semantic segment sequence based on the target semantic description, so as to determine the target control signal based on the target semantic segment sequence; the target sequence generation model is an end-to-end sequence generation model based on the Transformer architecture; The music video generation module 13 is used to generate music video segments corresponding to each of the target semantic segment sequences based on the target control signal using the target video diffusion model, and to splice the music video segments to obtain a spliced video. The spliced video is then combined with the target audio data to obtain the target music video corresponding to the target audio data and the text prompt words.
[0030] As can be seen, after obtaining the target audio data and text prompts, this application first performs feature extraction and multimodal vector fusion to obtain a target semantic description that can represent the theme, scene, style, emotional keywords, and other content of the target audio data. Then, the target semantic description is input into the target sequence generation model. Based on the semantic segment generation mechanism of the target sequence generation model, the target semantic description is directly mapped into a sequence of target semantic segments with temporal structure. The target control signal is obtained based on the target semantic segment sequence. Finally, the target video diffusion model is used to generate music video segments corresponding to each target semantic segment sequence based on the target control signal, thus obtaining the final target music video. This achieves "semantic alignment" at the macro level (such as the emotional color change of the verse / chorus) and "beat alignment" at the micro level (such as the action timing at the drum beat), realizing unified control of coarse and fine granularity, solving the controllability problem in the generation process, and achieving deep alignment between music semantics and visual content.
[0031] In one specific embodiment, the semantic vector extraction module 11 may include: A style determination unit is used to acquire target audio data, text prompts, and target reference data to be processed, and to determine the target video style based on the target reference data; the target reference data includes target reference video and target reference image; Accordingly, the control signal determination module 12 may specifically include: The vector fusion unit is used to perform multimodal fusion of the target text semantic vector and the target audio semantic vector based on a preset semantic vector fusion method and the target video style.
[0032] In one specific embodiment, the semantic vector extraction module 11 may include: The first feature analysis unit is used to determine the original audio features corresponding to the target audio data using the target audio analysis model; The second feature analysis unit is used to extract semantic features from the target audio data based on the audio features using the target audio analysis model, so as to obtain the target audio semantic vector and structural segmentation features corresponding to the target audio data. The original audio features include the audio start point, audio beat, audio speed, and short-time energy curve.
[0033] In one specific embodiment, the control signal determination module 12 may include: The prompt word generation unit is used to utilize the target sequence generation model to output a target semantic fragment sequence and visual description prompt words corresponding to the target semantic fragment sequence based on the target semantic description. A timeline script generation unit is used to determine a target timeline script based on the target semantic segment sequence and the corresponding visual descriptive cues; the target timeline script includes the duration of a scene, the camera movement method, the color tone of a scene, and the transition method between scenes. A control signal determination unit is used to determine a target control signal based on the target time axis script; The target control signal includes a continuous curve control signal and a discrete token control signal. The continuous curve control signal includes an energy curve signal for controlling the light intensity or particle activity of the image, a motion intensity curve for controlling the camera movement speed or the motion amplitude of objects in the image, and a cut probability curve for controlling the possibility of scene switching. The discrete token control signal is a semantic tag signal used to adjust the video generation conditions of the target video diffusion model.
[0034] In one specific embodiment, the device may further include: A control signal receiving module is used to embed a target ControlNet class structure in a target video diffusion model, so as to receive the continuous curve control signal in the target control signal using the target ControlNet class structure.
[0035] In one specific embodiment, the music video generation module 13 may include: The reference frame determination unit is used to determine the last frame of the music video segment corresponding to the previous target semantic segment sequence as the target reference frame corresponding to the current target semantic segment sequence. The video segment generation unit is used to generate the music video segment corresponding to the current target semantic segment sequence based on the target control signal and the target reference frame using a target video diffusion model.
[0036] In one specific embodiment, the music video generation module 13 may include: The aspect ratio determination unit is used to determine the aspect ratio of the target video. The video adjustment unit is used to splice the music video segments based on the timestamps corresponding to each music video segment to obtain a video to be adjusted, and to adjust the aspect ratio of the video to be adjusted based on the aspect ratio of the target video to obtain the corresponding spliced video.
[0037] Furthermore, embodiments of this application also disclose an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0038] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the AIGC music video generation method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.
[0039] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0040] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon can include an operating system 221, computer programs 222, etc., and the storage method can be temporary storage or permanent storage.
[0041] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the AIGC music video generation method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.
[0042] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned AIGC music video generation method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0043] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0044] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0045] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0046] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0047] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for generating AIGC music videos, characterized in that, include: The process involves acquiring target audio data and text prompts, performing semantic parsing on the text prompts to obtain target text semantic vectors, and using a target audio analysis model to extract features from the target audio data to obtain the target audio semantic vectors corresponding to the target audio data. The target text semantic vector and the target audio semantic vector are fused in a multimodal manner based on a preset semantic vector fusion method to obtain the corresponding target semantic description. The target sequence generation model is then used to output a target semantic segment sequence based on the target semantic description, so as to determine the target control signal based on the target semantic segment sequence. The target sequence generation model is an end-to-end sequence generation model based on the Transformer architecture. The target video diffusion model is used to generate music video segments corresponding to each target semantic segment sequence based on the target control signal, and the music video segments are spliced together to obtain a spliced video. The spliced video is then synthesized with the target audio data to obtain the target music video corresponding to the target audio data and the text prompt words.
2. The AIGC music video generation method according to claim 1, characterized in that, The process of acquiring the target audio data and text prompts to be processed includes: The system acquires target audio data, text prompts, and target reference data to be processed, and determines the target video style based on the target reference data; the target reference data includes target reference videos and target reference images. Accordingly, the multimodal fusion of the target text semantic vector and the target audio semantic vector based on the preset semantic vector fusion method includes: Multimodal fusion of the target text semantic vector and the target audio semantic vector is performed based on a preset semantic vector fusion method and the target video style.
3. The AIGC music video generation method according to claim 1, characterized in that, The step of extracting features from the target audio data using a target audio analysis model to obtain the target audio semantic vector corresponding to the target audio data includes: The original audio features corresponding to the target audio data are determined using a target audio analysis model. The target audio data is semantically feature extracted using the target audio analysis model based on the audio features, so as to obtain the target audio semantic vector and structural segmentation features corresponding to the target audio data. The original audio features include the audio start point, audio beat, audio speed, and short-time energy curve.
4. The AIGC music video generation method according to claim 1, characterized in that, The step of using a target sequence generation model to output a target semantic fragment sequence based on the target semantic description, and then determining the target control signal based on the target semantic fragment sequence, includes: Using a target sequence generation model, a target semantic fragment sequence and corresponding visual descriptive prompts are output based on the target semantic description. The target timeline script is determined based on the target semantic segment sequence and the corresponding visual descriptive cues; the target timeline script includes the duration of each scene, the camera movement method, the color tone of each scene, and the transition method between scenes; The target control signal is determined based on the target timeline script; The target control signal includes a continuous curve control signal and a discrete token control signal. The continuous curve control signal includes an energy curve signal for controlling the light intensity or particle activity of the image, a motion intensity curve for controlling the camera movement speed or the motion amplitude of objects in the image, and a cut probability curve for controlling the possibility of scene switching. The discrete token control signal is a semantic tag signal used to adjust the video generation conditions of the target video diffusion model.
5. The AIGC music video generation method according to claim 4, characterized in that, Before generating the music video segments corresponding to each target semantic segment sequence based on the target control signal using the target video diffusion model, the method further includes: A target ControlNet class structure is embedded in the target video diffusion model to receive the continuous curve control signal in the target control signal using the target ControlNet class structure.
6. The AIGC music video generation method according to claim 1, characterized in that, The step of generating music video segments corresponding to each target semantic segment sequence based on the target control signal using the target video diffusion model includes: The last frame of the music video segment corresponding to the target semantic segment sequence described above is determined as the target reference frame corresponding to the current target semantic segment sequence; Using a target video diffusion model, the music video segment corresponding to the current target semantic segment sequence is generated based on the target control signal and the target reference frame.
7. The AIGC music video generation method according to claim 1, characterized in that, The process of splicing the music video segments to obtain the spliced video includes: Determine the aspect ratio of the target video; The music video segments are spliced together based on the timestamps corresponding to each music video segment to obtain the video to be adjusted, and the aspect ratio of the video to be adjusted is adjusted based on the aspect ratio of the target video to obtain the corresponding spliced video.
8. An AIGC music video generation device, characterized in that, include: The semantic vector extraction module is used to acquire the target audio data to be processed and the text prompt words, perform semantic parsing on the text prompt words to obtain the target text semantic vector, and use the target audio analysis model to extract features from the target audio data to obtain the target audio semantic vector corresponding to the target audio data. The control signal determination module is used to perform multimodal fusion of the target text semantic vector and the target audio semantic vector based on a preset semantic vector fusion method to obtain a corresponding target semantic description, and to use a target sequence generation model to output a target semantic segment sequence based on the target semantic description, so as to determine the target control signal based on the target semantic segment sequence; the target sequence generation model is an end-to-end sequence generation model based on the Transformer architecture; The music video generation module is used to generate music video segments corresponding to each target semantic segment sequence based on the target control signal using a target video diffusion model, and to splice the music video segments to obtain a spliced video. The spliced video is then combined with the target audio data to obtain the target music video corresponding to the target audio data and the text prompt words.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the AIGC music video generation method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the AIGC music video generation method as described in any one of claims 1 to 7.