Method, device and computer readable storage medium for generating multimedia video

Abstract

This disclosure relates to a method and apparatus for generating multimedia video, and a computer-readable storage medium, and relates to the field of computer technology. The method for generating multimedia video includes: extracting musical features and lyric features from an audio file (S1); generating a storyboard based on the musical features and lyric features using a first machine learning model (S2); and generating a multimedia video based on the storyboard and the audio file using a second machine learning model (S3).

Description

Methods, apparatus and computer-readable storage media for generating multimedia video Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus and computer-readable storage medium for generating multimedia video. Background Technology

[0002] Artificial intelligence-generated content (AIGC) refers to the use of artificial intelligence technology to automatically generate various types of content, such as text, images, audio, and video.

[0003] The development of AIGC is the result of both technological advancements and market demand. As artificial intelligence technology continues to mature, AIGC will be widely applied in more fields, bringing higher levels of automation, personalization and customization, innovation and diversity.

[0004] Summary of the Invention

[0005] According to a first aspect of some embodiments of this disclosure, a method for generating multimedia video is provided, comprising: extracting music features and lyrics features from an audio file; generating a storyboard based on the music features and lyrics features using a first machine learning model; and generating a multimedia video based on the storyboard and the audio file using a second machine learning model.

[0006] According to a second aspect of some embodiments of the present disclosure, a multimedia video generation apparatus is provided, comprising: an extraction module configured to extract music features and lyrics features from an audio file; a script generation module configured to generate a storyboard script based on the music features and lyrics features using a first machine learning model; and a video generation module configured to generate a multimedia video based on the storyboard script and the audio file using a second machine learning model.

[0007] According to a third aspect of some embodiments of the present disclosure, a multimedia video generation apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute a multimedia video generation method according to some embodiments of the present disclosure based on instructions stored in the memory.

[0008] According to a fourth aspect of some embodiments of the present disclosure, a computer-readable storage medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement a method for generating multimedia video according to some embodiments of the present disclosure.

[0009] According to a fifth aspect of some embodiments of the present disclosure, a computer program product includes computer program instructions that, when executed by a processor, implement a method for generating multimedia video according to some embodiments of the present disclosure.

[0010] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0011] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0012] Preferred embodiments of the present disclosure are described below with reference to the accompanying drawings. The accompanying drawings, which are included to provide a further understanding of the present disclosure, and which, together with the following detailed description, are incorporated in and form a part of this specification and are used to explain the present disclosure. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and are not intended to limit the present disclosure. In the drawings:

[0013] Figure 1 shows a flowchart illustrating a method for generating multimedia video according to some embodiments of the present disclosure;

[0014] Figure 2 illustrates a flowchart of generating a storyboard script according to some embodiments of the present disclosure;

[0015] Figure 3 illustrates a schematic diagram of generating multimedia video according to some embodiments of the present disclosure;

[0016] Figure 4 shows a block diagram of a multimedia video generation apparatus according to some embodiments of the present disclosure;

[0017] Figure 5 shows a block diagram of a multimedia video generation apparatus according to some embodiments of the present disclosure;

[0018] Figure 6 shows a block diagram of an electronic device according to other embodiments of the present disclosure.

[0019] It should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not necessarily drawn to actual scale. The same or similar reference numerals are used in the various drawings to denote the same or similar parts. Therefore, once an item is defined in one drawing, it may not be discussed further in subsequent drawings. Detailed Implementation

[0020] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. However, it is obvious that the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of the embodiments is merely illustrative and is in no way intended to limit this disclosure or its application or use. It should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein.

[0021] It should be understood that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement and numerical values ​​of components and steps set forth in these embodiments should be interpreted as merely exemplary and do not limit the scope of this disclosure.

[0022] As used in this disclosure, the term "comprising" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". Furthermore, as used in this disclosure, the term "including" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". Therefore, "comprising" and "including" are synonymous. The term "based on" means "at least partially based on".

[0023] Throughout this specification, the terms "one embodiment," "some embodiments," or "embodiment" mean that a specific feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. For example, the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments." Furthermore, the appearance of the phrases "in one embodiment," "in some embodiments," or "in an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, but may refer to the same embodiment.

[0024] It should be noted that the concepts of "first," "second," etc., used in this disclosure are used only to distinguish different devices, modules, or units, and are not intended to define the order of functions performed by these devices, modules, or units or their interdependencies. Unless otherwise specified, the concepts of "first," "second," etc., are not intended to imply that the objects described herein must be in a given temporal, spatial, rank, or any other given order.

[0025] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0026] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0027] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings; however, this disclosure is not limited to these specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. Furthermore, in one or more embodiments, specific features, structures, or characteristics can be combined in any suitable manner that will be apparent to those skilled in the art from this disclosure.

[0028] In the context of this disclosure, "image" can refer to any of a variety of images, such as color images, grayscale images, etc. It should be noted that the type of image is not specifically limited in the context of this specification.

[0029] Among related technologies, music videos (MVs) need to be produced based on scripts created by humans, which is relatively inefficient.

[0030] This disclosure provides a method, apparatus, and computer-readable storage medium for generating multimedia video, which can improve the efficiency of multimedia video production.

[0031] Figure 1 shows a schematic flowchart of a method for generating multimedia video according to some embodiments of the present disclosure.

[0032] As shown in Figure 1, the method for generating multimedia video includes: step S1, extracting music features and lyrics features from the audio file; step S2, using a first machine learning model to generate a storyboard based on the music features and lyrics features; and step S3, using a second machine learning model to generate a multimedia video based on the storyboard and the audio file.

[0033] Audio files may include songs. Lyrics for an audio file can be obtained by finding the corresponding lyrics file (such as an LRC file) or by transcribing the audio content into text.

[0034] Musical features include, for example, features of vocals and background music. Both musical and lyrical features are, for example, in text form.

[0035] For lyrics, we can analyze the information they contain to extract details such as time, place, people, relationships, and events. For music, we can analyze the rhythm and emotion of the melody based on information such as the melody and vocals to identify the song's genre, such as sad or inspirational.

[0036] A storyboard, also known as a shot-by-shot script, describes the content to be filmed in multiple shots. A storyboard is typically in text format.

[0037] The first machine learning model includes, for example, a multimedia video generation model; the second machine learning model includes, for example, a video generation model.

[0038] Multimedia videos include, for example, music videos that include visuals and background music.

[0039] The multimedia video generation method disclosed herein can be executed on the client side or partially on the server side.

[0040] According to this disclosure, a storyboard is generated using a first machine learning model based on the musical and lyrical features of a song, and then a multimedia video is generated using a second machine learning model, thus improving the production efficiency of music videos. Furthermore, the synergy between the first and second machine learning models can generate matching visual effects based on the song, improving the consistency between music and visuals.

[0041] The following, with reference to Figure 2, first describes a flowchart illustrating the generation of storyboard scripts according to some embodiments of the present disclosure.

[0042] As shown in Figure 2, step S2 includes: step S21, generating narrative text based on music features and lyric features; step S22, generating storyboard script based on the narrative text using the first machine learning model.

[0043] Narrative text, such as the story of a song, includes elements such as characters, plot, theme, and conflict, which together constitute the structure and content of the story. Step S21 can be achieved, for example, through a text generation model.

[0044] The song, through its lyrics, melody, sound, and cultural elements, constructs a rich and colorful story world. The text generation model, by analyzing musical and lyrical features, can identify the themes and emotions the song intends to express, and then generate a story based on these elements. This story provides a clearer narrative thread for the music video, making the video content more coherent and engaging, and consistent with the song's theme and mood.

[0045] Musical features include at least one of the following: theme, imagery, emotional type, style type, melody features, rhythmic features, harmonic features, arrangement features, and timbre features; lyrical features include at least one of the following: theme, characters, relationships between characters, plot, time, place, and environment.

[0046] The themes of the stories include love, inspiration, nature, travel, and friendship.

[0047] The images include, for example, city night scenes, rural landscapes, beaches, forests, and starry skies.

[0048] Emotional types include happiness, sadness, inspiration, romance, anger, tranquility, nostalgia, tension, etc.

[0049] Style types include pop, rock, folk, electronic, jazz, classical, etc.

[0050] Melodic characteristics include, for example, lively and cheerful, simple and unadorned, melodious and lyrical, fast and intense.

[0051] Rhythmic characteristics include, for example, moderate tempo, allegro, and slow tempo.

[0052] Harmonic features include, for example, triads, seventh chords, and no harmony.

[0053] Arrangement characteristics include electronic arrangement, acoustic arrangement, orchestral arrangement, etc.

[0054] Vocal characteristics include, for example, youthful, mature, and weathered.

[0055] The themes of the lyrics include love, growth, dreams, freedom, hometown, and friendship.

[0056] Role relationships include, for example, romantic relationships and friendships.

[0057] In some embodiments, a first machine learning model is used to generate a storyboard based on narrative text, including: determining multiple audio segments of an audio file based on musical features and lyric features; determining matching relationships between multiple plots of the narrative text and the multiple audio segments; and generating a storyboard based on the narrative text and the matching relationships using the first machine learning model.

[0058] For example, an audio file can be divided into multiple audio segments based on the emotional changes and themes of a song. Some musical segments may include the vocal parts, while others may only include the interlude.

[0059] Based on the development of the plot, the narrative text is also divided into multiple plots, and each plot is analyzed in detail to clarify the time, place, background, characters involved, and main events.

[0060] If the number of plots is greater than the number of musical segments, then the best matching plot is determined for each musical segment from the multiple plots, so that the final selected plots match the musical segments one by one.

[0061] In some embodiments, determining the matching relationship between multiple plots of narrative text and multiple audio segments includes: determining the plot that matches each audio segment based on the lyrics of each audio segment of the multiple audio segments.

[0062] For example, a text similarity algorithm can be used to determine the matching relationship between audio segments and storylines based on the text similarity between the lyrics and the storyline of each audio segment. Before calculating the similarity, the lyrics and storyline text can be preprocessed, including stop word removal, stemming, and word frequency statistics, to improve the accuracy of the similarity calculation.

[0063] Based on the text similarity calculation results, a threshold is set to determine which audio segments have a matching relationship with the storyline text. The threshold can be adjusted according to the actual situation.

[0064] Based on a set threshold, audio clips are matched with storyline text. Storyline texts with a similarity higher than the threshold are considered to have a matching relationship.

[0065] For example, if the lyrics of a musical excerpt are "You are the person I only dare to contact after I'm drunk," then a storyline matched from multiple scenarios would be: "A got drunk at a friend's party. Under the influence of alcohol, he finally mustered up the courage to pick up his phone and dial B's number."

[0066] In some embodiments, determining a plot matching each audio segment based on the lyrics of each of the plurality of audio segments includes: determining a plot matching each audio segment based on the lyrics of each audio segment, the order of the plurality of audio segments, and the order of the plurality of plots.

[0067] For example, in the matching process, not only the lyrics of each audio segment are considered, but also the overall order of the audio segments.

[0068] If audio segment 1 precedes audio segment 2, and plot 1 precedes plot 2, and audio segment 1 matches plot 2, then audio segment 2 is considered not to match plot 1, but only to match plots that follow plot 2, thus ensuring the consistency between the plot development of the music video and the song.

[0069] In some embodiments, the storyboard includes descriptive text for the scene corresponding to each audio segment. A first machine learning model is used to generate the storyboard based on the narrative text and matching relationships. This includes: determining the characters in the storyboard shots corresponding to each audio segment based on the plot matching each audio segment; and using the first machine learning model, generating descriptive text for the scene corresponding to each audio segment based on the characters in the storyboard shots corresponding to each audio segment and the plot matching each audio segment.

[0070] The descriptive text includes details such as the content of the image, colors, lighting effects, and the characters' actions and expressions.

[0071] For example, based on the storyline "A drank too much at a friend's party. Under the influence of alcohol, he finally mustered up the courage to pick up his phone and dial B's number," the characters are identified as A and B.

[0072] Based on the above storyline, the text description of the scene in the storyboard generated by the model for this shot, which revolves around characters A and B, is: "In the dim light, A leans against the gray wall, his eyes glazed over as he takes out his phone from his pocket, his fingers trembling slightly as he finds B's name in the contacts, and then taps to dial."

[0073] In some embodiments, the storyboard includes the shot size of the shot corresponding to each audio segment. The storyboard is generated using a first machine learning model based on the narrative text and matching relationships, including: using the first machine learning model to determine the shot size of the shot corresponding to each audio segment based on the descriptive text of the scene corresponding to each audio segment.

[0074] For example, the first machine learning model performs a detailed analysis of the text describing the scene corresponding to each audio segment. This includes extracting key information from the text, such as scene descriptions, character actions, and emotional expressions. Based on the detailed text descriptions and the definitions of different shot sizes, the model analyzes the visual elements and narrative needs contained in the text to determine the shot size of the storyboard shot corresponding to each audio segment.

[0075] For example, based on the text description of a certain scene, "A, with blurry eyes, takes out his phone from his pocket, his fingers trembling slightly as he finds B's name in the contacts, and then taps to call," the first machine learning model determines that this scene is suitable for a close-up, because a close-up allows viewers to experience A's emotional fluctuations more deeply.

[0076] In some embodiments, the duration of the corresponding storyboard shot is determined based on the duration of each audio segment; using a first machine learning model, a descriptive text for the scene corresponding to each audio segment is generated based on the character and duration of the storyboard shot corresponding to each audio segment and the plot matching each audio segment.

[0077] For example, the duration of an audio clip can be the same as the duration of a storyboard shot. If the storyboard shot is 10 seconds long, then this scene is relatively short, and correspondingly, the descriptive text for the corresponding audio clip should also be shorter, thus aligning the video and audio in time.

[0078] In some embodiments, extracting musical features and lyric features from an audio file includes: determining the speech signal and instrument audio signal of the audio file; extracting speech features from the speech signal; extracting instrument features from the instrument audio signal; and determining musical features based on the speech features and instrument features.

[0079] For example, a pre-trained separation model can be used to separate the speech signal and the instrument audio signal from an audio file. Speech features are determined by capturing the spectral characteristics of the speech signal; instrument features are determined by capturing the spectral characteristics of the instrument audio signal.

[0080] In some embodiments, extracting musical features and lyric features from an audio file includes: determining the relationship between the musical features of the audio file and the lyric text of the audio file; and extracting lyric features based on the lyric text and the relationship.

[0081] For example, if a song is sung by a group, and the timbre of a particular segment indicates the singer is a child, then the lyrics can be inferred to describe the actions and thoughts of child character A. If the timbre of a segment indicates the singer is an adult, then the lyrics describe the actions and thoughts of adult character B. Based on the descriptions of the actions and thoughts of both child character A and adult character B, the parent-child relationship between A and B can be inferred, serving as a characteristic of the lyrics.

[0082] The previous section introduced how to generate storyboards. The following section describes the process of generating video from storyboards.

[0083] The second machine learning model includes a first sub-model and a second sub-model. Using the second machine learning model, multimedia video is generated based on the storyboard and audio files. This includes: using the first sub-model to generate storyboard images corresponding to each of the multiple storyboard shots based on the storyboard; using the second sub-model to generate dynamic images corresponding to each storyboard shot based on the storyboard images and the storyboard; and generating multimedia video based on the dynamic images and audio files.

[0084] The first sub-model is, for example, a text-to-image model, and the second sub-model is, for example, an image-to-video model.

[0085] Storyboard images are static images that represent key scenes in a music video. Dynamic images, on the other hand, are animated video clips formed by a series of consecutive frames.

[0086] The descriptions in the storyboard are input into the image generation model to generate storyboard images. The generated storyboard images are then arranged in sequence to form an image sequence. Using an image-to-video model, intermediate frames are inserted between adjacent images to make the transitions smoother, ultimately resulting in a dynamic video.

[0087] In some embodiments, using a first sub-model, generating a storyboard image corresponding to each of the multiple storyboard shots based on the storyboard script includes: generating an image of a character in the storyboard script; and using the first sub-model, generating a storyboard image corresponding to each of the multiple storyboard shots based on the storyboard script and the image of the character.

[0088] For example, based on the story, use text to generate image models and create character illustrations.

[0089] When generating storyboard images using the first sub-model, a natural language processing model is first used to determine whether there are characters in the storyboard shot. If characters are included, the previously generated character image is called.

[0090] For the same character in different shots, the same character image is used to ensure consistency in the character's appearance.

[0091] In some embodiments, a second sub-model is used to generate dynamic images corresponding to each storyboard shot based on the storyboard images and storyboard script, including: generating images of characters in the storyboard script; and using the second sub-model to generate dynamic images corresponding to each storyboard shot based on the storyboard images, storyboard script, and images of characters.

[0092] For example, the second sub-model analyzes the instructions in the storyboard, predicts and synthesizes continuous motion frames based on the storyboard images and character images, determines how to animate static images, and ensures that the character images are consistent throughout.

[0093] The character image can be used when generating storyboard images using the first sub-model. It can also be used when generating motion graphics using the second sub-model. Alternatively, the character image can be generated only once, but used in both storyboard image generation and motion graphics generation. This disclosure does not impose any limitations on this approach.

[0094] Figure 3 illustrates a schematic diagram of generating multimedia video according to some embodiments of the present disclosure.

[0095] As shown in Figure 3, each step in generating multimedia video can be implemented by an intelligent agent, which may include an agent for analyzing lyrics, an agent for analyzing music, an agent for generating song stories, an agent for generating scripts, and an agent for generating multimedia videos.

[0096] Intelligent agents for analyzing lyrics, generating song stories, and generating scripts are, for example, based on text generation models. Intelligent agents for analyzing music are, for example, based on audio processing models. Intelligent agents for generating multimedia videos are, for example, based on video generation models.

[0097] The following example illustrates the process of generating multimedia videos.

[0098] First, extract lyric and music features, using the audio file and the following prompt text as input to the feature extraction model.

[0099] "You are a highly professional and insightful song analysis expert with a wealth of musical knowledge and a solid foundation in music theory. You are able to deeply analyze all aspects of a song, including but not limited to melody, rhythm, harmony, arrangement, lyrics, and singing style."

[0100] When faced with a song, please analyze it in detail from the following key aspects:

[0101] Describe the melody's progression: is it smooth and melodious or lively and agile? For example, does it employ common musical progression patterns such as ascending, descending, or undulating? Use simple musical terms like "stepwise" or "leap" to describe the connections between key notes. Point out the unique features or memorable elements of the melody, such as whether there are repeated melodic fragments to reinforce the theme, or unexpected note transitions to create surprise. Analyze the melody's emotional coloring, determining whether it conveys joy, sadness, excitement, or tranquility, and explain how this emotion is expressed through the combination of notes.

[0102] Determine the song's rhythm type: is it a lively 2 / 4, a steady 3 / 4, or an energetic 4 / 4? Discuss rhythmic variations, such as whether the rhythm intensifies in the chorus or slows down in the bridge to create a transition. Discuss the impact of rhythm on the overall atmosphere of the song; for example, how a fast tempo evokes the listener's emotions and fills them with energy, while a slow tempo creates a lyrical atmosphere.

[0103] Identify the main harmonic progressions used in the song and explain whether they belong to common harmonic patterns or have unique innovations. Analyze the role of harmony in different sections, such as how it supports the melody in the verses and enhances the tension in the climax. Observe the relationship between harmony and melody; is it harmonious and unified, or does it create tension through dissonant harmonies?

[0104] List the instruments or musical elements used in the song, such as guitar, piano, drums, and strings, and describe their respective performance and role in the song. Analyze the layering of the arrangement, that is, how the various instruments cooperate with each other at different ranges and points in time to jointly construct a rich musical picture. Discuss the role of the arrangement in shaping the style of the song; for example, using electronic instruments may create a fashionable and modern feel, while using traditional folk instruments may give the song a unique regional flavor.

[0105] Interpret the theme and main idea of ​​the lyrics, clarifying the core thought or emotional story they express. Analyze the literary techniques used in the lyrics, such as metaphor, personification, hyperbole, and rhyme, and how these techniques enhance the expressiveness of the lyrics. Consider the relationship between the lyrics and other elements of the song, such as how the content of the lyrics echoes the melody and rhythm to jointly convey a complete musical mood. Extract key information points, such as location, characters, relationships between characters, and events.

[0106] Evaluate the singer's vocal techniques, including pitch, timbre, breath control, and vocal technique. Describe the singer's ability to interpret the song's emotions, and whether they can genuinely convey the emotions to the audience. Analyze the fit between the singer's style and the song's style; for example, how a rock singer can inject unique energy into a rock song.

[0107] When analyzing a song, please adopt a professional and objective approach, combining specific musical examples and details to provide listeners with an in-depth and comprehensive interpretation, helping them better understand and appreciate the song's artistic value. At the same time, strive to use concise and clear language, avoiding overly complex musical jargon, so that even non-specialists can easily understand your analysis.

[0108] The model output is as follows.

[0109] Time: Winter; Location: Somewhere; Characters: The singer and the object of his longing, "you"; Relationship: They had a past emotional entanglement; The singer has a deep longing and affection for "you" and is now in a state of not daring to contact him easily; Event: In the cold winter, the singer suffers the pain of unrequited love alone, wanting to contact "you" but not daring to.

[0110] Theme and main idea: It expresses feelings of longing and affection; literary technique: metaphor; the connection between the lyrics and other elements: the line "I only dare to contact you after I'm drunk" in the lyrics echoes the restrained and repressed emotional atmosphere presented by the overall melody and rhythm of the song.

[0111] Melodic progression: The overall melody is smooth and melodious, complementing the lyrical theme. It employs a wave-like progression, with a relatively stable melody expressing longing, while occasional leaps in intensity appear to emphasize inner pain and increase emotional tension. Unique features: Repetitive melodic fragments of key phrases such as "You're the person I only dare contact when I'm drunk."

[0112] Emotional tone: Through the rise and fall of the melody and the combination of notes, emotions of sadness and longing are conveyed. For example, in the parts describing inner pain, slightly lower notes and a slower rhythm are used to express this.

[0113] Rhythm type: steady beat to create a lyrical atmosphere; Rhythm variation: the rhythm intensity is increased in the chorus to highlight the emotional climax, and the rhythm is appropriately slowed down in the bridge to express the inner struggle and hesitation; The influence of rhythm on atmosphere: the slower rhythm creates a lyrical and sad atmosphere.

[0114] Instruments or musical elements: The piano is the main accompaniment instrument. Drums are used to lay the background and add layers to the music; the drum rhythm is relatively soft. The bass is added during the climax.

[0115] Singing style: Warm and gentle.

[0116] Based on the features obtained from the above analysis, the plot of the generated narrative text is as follows.

[0117] "Once upon a time, A met B, and the two fell in love. However, B had to leave. On the day of their separation, A stood at the station, silently watching B's departing figure..."

[0118] Based on the above story, the following storyboard script was generated using a text generation model.

[0119] Roles: A and B

[0120] Table 1

[0121] Based on the script above, video footage is generated using a video generation model.

[0122] Align the lyrics, song, and generated video in time and combine them into a multimedia video to obtain the finished music video.

[0123] Figure 4 shows a block diagram of a multimedia video generation apparatus according to some embodiments of the present disclosure.

[0124] As shown in Figure 4, the multimedia video generation device 4 includes: an extraction module 41 configured to extract music features and lyrics features from an audio file; a script generation module 42 configured to generate a storyboard script based on the music features and lyrics features using a first machine learning model; and a video generation module 43 configured to generate a multimedia video based on the storyboard script and the audio file using a second machine learning model.

[0125] The extraction module 41 of the multimedia video generation device 4 can be used to execute step S1 of FIG1, the script generation module 42 can be used to execute step S2 of FIG1, and the video generation module 43 can be used to execute step S3 of FIG1.

[0126] Figure 5 shows a block diagram of a multimedia video generation apparatus according to some embodiments of the present disclosure.

[0127] Memory 51 is used to store one or more computer-readable instructions. Memory 51 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Memory 51 may, for example, store operating systems, application programs, boot loaders, databases, and other programs, as well as various application programs and various data.

[0128] The processor 52 is configured to execute computer-readable instructions to implement the song selection method of any of the foregoing embodiments or the method of any of the foregoing embodiments. Specific implementations of each step of the method can be found in the above embodiments, and repeated details will not be elaborated here.

[0129] Processor 52 can be configured to execute the steps shown in Figures 1 and 2. Processor 52 can be various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The central processing unit (CPU) can be based on x86 or ARM architectures, etc.

[0130] The processor 52 and the memory 51 can communicate with each other directly or indirectly. For example, the processor 52 and the memory 51 can communicate via a network. The network can include a wireless network, a wired network, and / or any combination of wireless and wired networks. The processor 52 and the memory 51 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0131] It should be noted that the components of the multimedia video generation apparatus 5 shown in Figure 5 are merely exemplary and not limiting. Depending on the specific application requirements, the multimedia video generation apparatus 5 may also have other components. The processor 52 can control other components in the multimedia video generation apparatus 5 to perform the desired functions.

[0132] The multimedia video generation device 5 can be implemented by software, firmware and / or hardware, and can be integrated into a device with the relevant application installed.

[0133] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement a method for generating multimedia video according to some embodiments of this disclosure.

[0134] This disclosure also provides a computer program product, including computer program instructions that, when executed by a processor, implement a method for generating multimedia video according to some embodiments of this disclosure.

[0135] Figure 6 shows a block diagram of an electronic device according to other embodiments of the present disclosure.

[0136] The electronic device 6 shown in Figure 6 can be a computer system with a dedicated hardware structure, capable of performing corresponding functions when relevant applications are installed.

[0137] Electronic devices include, but are not limited to, mobile terminals such as smartphones, laptops, personal digital assistants (PDAs), tablet computers (PCs), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), wearable devices, and fixed terminals such as digital televisions and desktop computers.

[0138] As shown in Figure 6, the Central Processing Unit (CPU) 61 performs various processes based on a program stored in the Read-Only Memory (ROM) 62 or a program loaded from the Storage Section 68 into the Random Access Memory (RAM) 63. The RAM 63 stores data required as needed when the CPU 61 performs various processes. The CPU is merely exemplary and can also be other types of processors, such as the various processors described above. The ROM 62, RAM 63, and Storage Section 68 can be various forms of computer-readable storage media. It should be noted that although the ROM 62, RAM 63, and Storage Section 68 are shown separately in Figure 6, one or more of them can be combined or located in the same or different memories or storage modules.

[0139] CPU 61, ROM 62 and RAM 63 are interconnected via bus 64. Input / output interface 65 is also connected to bus 64.

[0140] The following components are connected to the input / output interface 65: input section 66, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output section 67, including displays such as cathode ray tube (CRT), liquid crystal display (LCD), speakers, vibrators, etc.; storage section 68, including hard disks, magnetic tapes, etc.; and communication section 69, including network interface cards such as LAN cards, modems, etc. The communication section 69 allows communication processing to be performed via a network such as the Internet. It is readily understood that although parts of the electronic device 6 shown in Figure 6 communicate via bus 64, they can also communicate via a network or other means, wherein the network can include wireless networks, wired networks, and / or any combination of wireless and wired networks.

[0141] As needed, drive 610 is also connected to input / output interface 65. Removable media 611, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 610 as needed, so that computer programs read from them can be installed into storage section 68 as needed.

[0142] When the above series of processes are implemented through software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as removable medium 611.

[0143] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product that, when run on a computer, causes the computer to perform the methods described in any of the foregoing embodiments. The computer program product includes computer instructions carried on a computer-readable medium, containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer instructions can be downloaded and installed from a network via communication section 69, or installed from storage section 68, or installed from ROM 62. When the computer program is executed by CPU 61, the methods of embodiments of this disclosure are performed.

[0144] It should be noted that, in the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0145] A computer-readable medium may be a computer-readable storage medium, a computer-readable signal medium, or any combination thereof.

[0146] Computer-readable storage media include, but are not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Computer instructions are stored on the computer-readable storage medium that, when executed by a processor, implement the methods described in any of the foregoing embodiments.

[0147] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0148] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0149] In some embodiments, a computer program is also provided, comprising: instructions that, when executed by a processor, cause the processor to perform the methods described in any of the foregoing embodiments. For example, the instructions may be embodied in computer program code.

[0150] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0153] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for generating multimedia video, comprising: Extracting musical and lyric features from audio files; Using the first machine learning model, a storyboard is generated based on the music features and lyrics features; Using a second machine learning model, a multimedia video is generated based on the storyboard script and the audio file.

2. The video generation method according to claim 1, wherein, The process of generating a storyboard using a first machine learning model based on the music features and lyrics features includes: Based on the aforementioned musical and lyrical features, a narrative text is generated; Using a first machine learning model, a storyboard is generated based on the narrative text.

3. The video generation method according to claim 2, wherein, The step of generating a storyboard based on the narrative text using a first machine learning model includes: Based on the musical and lyrical features, multiple audio segments of the audio file are determined; Determine the matching relationship between multiple plots of the narrative text and the multiple audio segments; Using the first machine learning model, the storyboard is generated based on the narrative text and the matching relationship.

4. The video generation method according to claim 3, wherein, Determining the matching relationship between multiple plots of the narrative text and the multiple audio segments includes: Based on the lyrics of each of the plurality of audio segments, determine the plot that matches each of the audio segments.

5. The video generation method according to claim 4, wherein, The step of determining the plot matching each audio segment based on the lyrics of each of the plurality of audio segments includes: Based on the lyrics of each audio segment, the order of the multiple audio segments, and the order of the multiple plots, determine the plot that matches each audio segment.

6. The video generation method according to any one of claims 3-5, wherein, The storyboard includes descriptive text for the scene corresponding to each audio segment. Generating the storyboard using the first machine learning model, based on the narrative text and the matching relationship, includes: Based on the plot that matches each audio segment, determine the characters in the storyboard shots corresponding to each audio segment; Using the first machine learning model, based on the storyboard shots corresponding to each audio segment... The character and the plot that matches each audio segment are used to generate descriptive text for the scene corresponding to each audio segment.

7. The video generation method according to claim 6, wherein, The storyboard includes the shot size of the scene corresponding to each audio segment. Generating the storyboard using the first machine learning model based on the narrative text and the matching relationship includes: Using the first machine learning model, the shot size of the scene corresponding to each audio segment is determined based on the descriptive text of the scene corresponding to each audio segment.

8. The video generation method according to claim 6 or 7, wherein, The storyboard script includes the duration of the storyboard shot corresponding to each audio segment. The step of using the first machine learning model to generate descriptive text for the scene corresponding to each audio segment, based on the characters in the storyboard shot and the plot matching each audio segment, includes: The duration of the corresponding storyboard shot is determined based on the duration of each audio segment; Using the first machine learning model, based on the character and duration of the storyboard shot corresponding to each audio segment and the plot matching each audio segment, descriptive text for the scene corresponding to each audio segment is generated.

9. The video generation method according to any one of claims 1-8, wherein, The second machine learning model includes a first sub-model and a second sub-model. The step of generating multimedia video using the second machine learning model based on the storyboard and the audio file includes: Using the first sub-model, a storyboard image corresponding to each of the multiple storyboard shots is generated according to the storyboard script; Using the second sub-model, dynamic scenes corresponding to each storyboard shot are generated based on the storyboard images and the storyboard script; The multimedia video is generated based on the animated images and the audio file.

10. The video generation method according to claim 9, wherein, The step of generating a storyboard image corresponding to each of the multiple storyboard shots based on the storyboard script using the first sub-model includes: Generate images of the characters in the storyboard script; Using the first sub-model, based on the storyboard script and the character's image, a storyboard image corresponding to each of the multiple storyboard shots is generated.

11. The video generation method according to claim 9 or 10, wherein, The step of generating dynamic footage corresponding to each storyboard shot using the second sub-model, based on the storyboard images and the storyboard script, includes: Generate images of the characters in the storyboard script; Using the second sub-model, dynamic scenes corresponding to each storyboard shot are generated based on the storyboard images, the storyboard script, and the character images.

12. The video generation method according to any one of claims 1-11, wherein, The extraction of musical and lyric features from audio files includes: Determine the speech signal and instrument audio signal of the audio file; Extract speech features from the speech signal; Extract instrument features from the instrument's audio signal; The musical features are determined based on the speech features and the instrument features.

13. The video generation method according to any one of claims 1-12, wherein, The extraction of musical and lyric features from audio files includes: Determine the relationship between the musical features of the audio file and the lyrics text of the audio file; Based on the lyrics text and the relationship, extract the lyrics features.

14. The video generation method according to any one of claims 1-13, wherein: The musical features include at least one of the following: musical theme, imagery, emotional type, style type, melody features, rhythmic features, harmonic features, arrangement features, and timbre features; The lyric features include at least one of the following: theme, characters, relationships between characters, plot, time, place, and setting.

15. A multimedia video generation apparatus, comprising: The extraction module is configured to extract musical and lyric features from audio files; The script generation module is configured to use a first machine learning model to generate a storyboard script based on the music features and lyrics features; The video generation module is configured to use a second machine learning model to generate multimedia video based on the storyboard script and the audio file.

16. A multimedia video generation apparatus, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to operate based on instructions stored in the memory. Perform the method for generating multimedia video according to any one of claims 1-14.

17. A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method for generating multimedia video according to any one of claims 1-14.

18. A computer program product comprising computer program instructions that, when executed by a processor, implement the method for generating multimedia video according to any one of claims 1-14.

Images