Music video generation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a music video generation method and device, equipment and a computer readable storage medium. The method comprises the following steps: acquiring a generation request, wherein the generation request is associated with reference music; and providing a music video generated based on the generation request, the music video being generated based on a video template and a set of images, the video template being determined based on audio characteristics of the reference music, the set of images being generated based on description information of the reference music. Therefore, according to the embodiment of the invention, the music video can be efficiently generated based on the characteristics of the music and the video template, so that the generation efficiency of the music video is improved, and the integrating degree of the music video and the music is improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the computer field, and more particularly, to a method, apparatus, device, and computer-readable storage medium for generating a music video. Background Art

[0002] With the development of computer technology, more and more users are sharing music on video platforms. For example, users can make music content into video content and publish the video content to the video platform. Some video platforms also provide users with the function of automatically generating music videos to facilitate users to create video content. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for generating a music video is provided. The method comprises: obtaining a generation request, the generation request being associated with reference music; and providing a music video generated based on the generation request, wherein the music video is generated based on a video template and a set of images, the video template being determined based on audio features of the reference music, and the set of images being generated based on description information of the reference music.

[0004] In a second aspect of the present disclosure, a device for generating a music video is provided. The device includes: an acquisition module configured to acquire a generation request, the generation request being associated with reference music; and a providing module configured to provide a music video generated based on the generation request, wherein the music video is generated based on a video template and a set of images, the video template being determined based on audio features of the reference music, and the set of images being generated based on description information of the reference music.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments according to the present disclosure may be implemented;

[0010] FIG. 2A to FIG. 2B shows an example interface according to some embodiments of the present disclosure;

[0011] Figure 3 An example block diagram showing a method for generating a music video according to some embodiments of the present disclosure;

[0012] Figure 4 A flowchart illustrating an example process for the generation of a music video according to some embodiments of the present disclosure;

[0013] Figure 5 A schematic structural block diagram showing an example apparatus for generating a music video according to some embodiments of the present disclosure; and

[0014] Figure 6 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0017] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0018] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects are subject to the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user knows and confirms. Accordingly, when implementing each embodiment of the present disclosure, the type, scope of use, usage scenario, etc. of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method can vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0019] In this specification and the embodiments, if personal information processing is involved, it will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or it is necessary to perform a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0020] As mentioned above, with the development of computer technology, more and more users are sharing music on video platforms. For example, users can make music content into video content and publish the video content to the video platform. Some video platforms also provide users with the function of automatically generating music videos, which is convenient for users to make video content. The music videos generated by the automatic generation function of traditional music videos have poor fit with the music, poor dynamic effects and monotonous content, which makes it difficult for the automatically generated music videos to meet the needs of users.

[0021] The embodiment of the present disclosure proposes a music video generation scheme. The scheme includes: obtaining a generation request, the generation request is associated with reference music; and providing a music video generated based on the generation request, wherein the music video is generated based on a video template and a set of images, the video template is determined based on the audio features of the reference music, and the set of images is generated based on the description information of the reference music.

[0022] In this way, by generating music videos based on the audio features and video templates of music, the embodiments of the present disclosure can efficiently generate music videos based on the features of music and video templates, thereby improving the efficiency of music video generation, improving the fit between the music video and the music, and increasing the content richness of the music video.

[0023] Various example implementations of the solution are described in detail below in conjunction with the accompanying drawings.

[0024] Example Environment

[0025] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, example environment 100 may include electronic device 110 .

[0026] In this example environment 100, the electronic device 110 may run an application 120 that supports the generation of a music video. The application 120 may be any suitable type of application for the generation of a music video, examples of which may include, but are not limited to, video applications, live broadcast applications, or other suitable applications that can provide music video generation services. The user 140 may interact with the application 120 via the electronic device 110 and / or its attached devices.

[0027] exist Figure 1 In the environment 100 , if the application 120 is active, the electronic device 110 may present an interface 150 for supporting generation of a music video through the application 120 .

[0028] In some embodiments, the electronic device 110 communicates with the server 130 to provide services for the application 120. The electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a handheld computer, a portable game terminal, a VR / AR device, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio receiver, an e-book device, a game device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the user (such as a "wearable" circuit, etc.).

[0029] The server 130 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server 130 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, etc. The server 130 may provide background services for the application 120 that supports the generation of music videos in the electronic device 110.

[0030] A communication connection may be established between the server 130 and the electronic device 110. The communication connection may be established in a wired manner or a wireless manner. The communication connection may include, but is not limited to, a Bluetooth connection, a mobile network connection, a Universal Serial Bus (USB) connection, a Wireless Fidelity (WiFi) connection, etc., and the embodiments of the present disclosure are not limited in this respect. In the embodiments of the present disclosure, the server 130 and the electronic device 110 may implement signaling interaction through the communication connection between the two.

[0031] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0032] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0033] Example Interaction

[0034] FIG. 2A to FIG. 2B Example interfaces 200A to 200B according to some embodiments of the present disclosure are shown. For example, interfaces 200A to 200B may be composed of Figure 1 The electronic device 110 is provided as shown.

[0035] Interfaces 200A to 200B may be interfaces within a video application for consuming video content.

[0036] refer to Figure 2A In some embodiments, the electronic device 110 may present an interface 200A, in which music video content is presented, wherein the music video content includes an image 220, and the image 220 is associated with the music content of the music video being played in the interface 200A.

[0037] like Figure 2AAs shown, interface 200A also presents descriptive information such as the name and author of the music, and interface 200A may also present dynamic effects associated with the music. For example, image 220 or lyrics presented in interface 200A may move up and down with the rhythm or beat of the music to present dynamic effects that conform to the audio characteristics of the music.

[0038] In some embodiments, the interface 200A may further present a control 210. The electronic device 110 may obtain a generation request in response to the control 210 being triggered, wherein the generation request is associated with reference music. For example, in response to the control 210 being triggered, the electronic device 110 may present a reference music selection interface, where the reference music includes music for generating a music video.

[0039] In some embodiments, the reference music includes: uploaded first music and / or second music selected from a group of candidate music and / or third music determined based on query information and / or fourth music generated based on generation parameters.

[0040] In some examples, the first music may include music uploaded by the current user or other users. The second music may include music selected by the current user from a set of music stored in the electronic device 110, or the second music may also include a set of available music obtained from a server. The third music may include music presented based on query information input by the user, for example, the user may specify information such as the type, length, author, etc. of the music to find the third music from the available music. The fourth music may include music generated based on some parameters, for example, the electronic device 110 may generate the fourth music based on some input from the user, such as parameters such as music beat, music style, etc. And the generated fourth music may be further used as reference music to generate a corresponding music video.

[0041] In some embodiments, after determining the reference music, the electronic device 110 may obtain a generation request, and the generation request is associated with the reference music. For example, the generation request may be associated with the reference music, for example, the generation request may include some parameters, and these parameters indicate some attributes and characteristics of the reference music, for example, these parameters may indicate the duration, type, lyrics content, etc. of the reference music.

[0042] In some embodiments, the electronic device 110 may provide a music video generated based on the generation request. For example, the electronic device 110 may generate a corresponding music video in response to receiving the generation request. In other examples, the electronic device 110 may initiate a music video generation process in response to receiving the generation request. The generation process may be performed in the music video. Figure 3 Detailed description is given in , so it will not be repeated here.

[0043] In some embodiments, the music video is generated based on a video template and a set of images. The video template may include, for example, a dynamic effect of the video. The set of images may be generated by a machine learning model, where the machine learning model may include any machine learning model capable of generating image content, and the present disclosure is not limited thereto.

[0044] In some embodiments, the video template is determined based on the audio features of the reference music. For example, the audio features here may include the rhythm of the audio, the length of the audio, the timbre of the musical instrument in the audio, etc. The electronic device 110 can generate a video template that matches the music based on these features.

[0045] In some embodiments, the audio features may be determined based on the following process: determining the rhythm information of the reference music, and determining the audio features of the reference music based on the rhythm information. For example, the electronic device 110 may determine the rhythm information of the reference music (e.g., the number of beats per minute, audio beat changes, etc.). Afterwards, the electronic device 110 may determine the audio features of the reference music based on the rhythm information.

[0046] In some embodiments, the video template is generated based on the following process: generating a set of dynamic effects that match the rhythm information; and generating the video template based on the set of dynamic effects. For example, the electronic device 110 can generate a set of dynamic effects that match the beats per minute of the music, and the set of dynamic effects can include lyrics that move up and down according to the beat of the music, a set of images that rotate according to the rhythm, or interface icons that move according to the rhythm. And the electronic device 110 can also generate the video template based on a set of dynamic effects, and the video template can be applied to the music video.

[0047] In some embodiments, at least one of the set of motion effects is applied to the lyrics content of the reference music. For example, the lyrics content presented in the interface can move along with the music.

[0048] In some embodiments, a set of images is generated based on the description information of the reference music. The description of the set of images and the description information of the reference music will be described below. Figure 3 A detailed description is given and will not be repeated here.

[0049] refer to Figure 2B , Figure 2B Interface 200B is shown, in which another music video may be presented, and the other music video may be a reference Figure 2A Generated based on the properties of the music video shown in .

[0050] In some embodiments, the electronic device 110 may generate a generation request associated with a reference video in response to an operation on the control 210, and generate a corresponding music video based on the generation request.

[0051] In some embodiments, the generation request is also associated with a reference video, and the music video is also generated based on at least one attribute of the reference video. For example, the reference video may include the music video shown in interface 200A, and the music video shown in interface 200B may be associated with the parameters of the reference video.

[0052] In some embodiments, at least one attribute includes at least one of the following: a background style of the reference video, a video template associated with the reference video, and a content layout of the reference video. For example, the background style of the music video shown in interface 200B may be the same as or similar to the background style of the music video shown in interface 200A.

[0053] In some examples, the video template of the music video in interface 200B is the same as or similar to the video template of the music video in interface 200A, for example, both have similar dynamic effects, or a set of image content with a similar style.

[0054] In addition, the content layout of the music video in interface 200B may be the same as or similar to the content layout of the music video in interface 200A. For example, the position of image 230 in interface 200B is close to the position of image 220 in interface 200A.

[0055] refer to Figure 3 , Figure 3 An example block diagram 300 of a method for generating a music video according to some embodiments of the present disclosure is shown.

[0056] like Figure 3 As shown, in block 302, the electronic device 110 may receive an audio file, which may be selected by a user or generated by the electronic device 110. For example, the audio file here may include an audio file uploaded by the current user or other users, and may also include an audio file selected by the current user from a group of audio files stored in the electronic device 110. The audio file may also be an available audio file obtained from a server.

[0057] In addition, the audio file may include audio files presented based on query information input by the user, for example, the user may specify information such as the type, length, and author of the music to filter out the audio files required for video generation from a group of files. The music in the audio file may include music generated based on some parameters, for example, the electronic device 110 may generate the audio file based on some input from the user, such as parameters such as music beat and music style.

[0058] In box 306, the electronic device 110 can extract audio information. The audio information here can include the properties and characteristics of the music stored in the audio file. For example, the electronic device 110 can extract the accompaniment instrument in the music, and determine the properties and characteristics of the music based on the accompaniment instrument. The electronic device 110 can also, for example, extract the beat rate of the audio in the audio file. The above-mentioned audio information can be used as the material of the video motion effect template in the subsequent generation step. For example, the electronic device 110 can select appropriate motion effects based on the accompaniment instruments of the music to display in the music video. For example, if the accompaniment instruments of the music include drum instruments, the electronic device 110 can add ripple-shaped motion effects to the video motion effect template to match the drum beats in the music to increase the fun of the music video.

[0059] In box 310, the electronic device 110 can obtain the beat rate of the music. Here, the beat rate is only used as an example of the subsequent generation process. The electronic device 110 can also generate subsequent motion effect templates based on other music information, such as the accompaniment instrument type mentioned above. In addition, in other embodiments, the electronic device 110 can also use a variety of audio information to assist in the generation of video motion effect templates. For example, the electronic device 110 can automatically select the corresponding motion effect template based on the beat rate and the accompaniment instrument type. The above-mentioned motion effect templates can be pre-configured by the developer or manually adjusted by the user. Secondly, these motion effect templates can also be selected based on information provided by the server. For example, motion effect templates that are used more frequently can be preferentially selected by the electronic device 110.

[0060] In box 314, the electronic device 110 can generate a motion effect template for the video based on the beat rate obtained at box 310. Here, the generation of the motion effect template can be based on a preset configuration of the video application, which can be configured by a developer. In addition, in order to save computing resources of the electronic device 110, the electronic device 110 can also send a template generation request to a server or cloud, and the electronic device 110 can also use some machine learning models to generate the template.

[0061] In box 318, the electronic device 110 can generate a music video motion effect based on the video motion effect template generated at box 314, and the motion effect of the music video matches the rhythm of the music. As some non-limiting examples, the motion effect of the music video can include, for example, a ripple motion effect, and the movement frequency of the ripple motion effect can match the number of beats of the music. For example, if the number of beats per minute of the music is 120 beats per minute, the electronic device 110 can present a ripple motion effect with a frequency of 120 Hz in the interface. Such motion effects can bring users a visual experience similar to a metronome, thereby increasing the fun of the music video.

[0062] In box 304, the electronic device 110 can obtain music information (e.g., song information, i.e., the description information mentioned above) of the music. This information can be input by the user, or this information can also be extracted by the electronic device 110 based on the selected music. Here, the process of extracting the music information can be completed by the electronic device 110 alone (i.e., without the help of online resources), but in other embodiments, the process of extracting the music information can be completed with the assistance of a server, for example, the server can return information about the music, for example, the server can determine the author of the music and the release date of the music based on the file name of the music.

[0063] In block 308, in some embodiments, the electronic device 110 may construct prompt words based on the description information. For example, the electronic device 110 may generate required prompt words based on the description information via prompt word engineering, and the prompt words may be used to generate a set of images, which may be, for example, images 220 in the interface 200A. Here, the set of images includes pictures and videos, which may be presented in the music video to provide a richer visual experience for the user.

[0064] In box 312, the electronic device 110 can obtain standardized prompt words through the above prompt word engineering. In some embodiments, the electronic device 110 can construct prompt words based on the description information and the preset prompt word template. The above preset prompt word template can be preset by the developer. The prompt words here can be constructed in the form of natural language, such as "Song name: ABCD; Author EFG; Genre: Rock; Output image: 3 pictures; Image content: Rock band". These prompt words can be presented in the interface of the electronic device 110, and can be adjusted by the user to make the generated image more in line with the user's requirements.

[0065] In box 316, the electronic device 110 can provide prompt words to the image generation model to generate a group of images. The image generation model can include a pre-trained large model, for example. A group of images can include a group of pictures or a group of videos. The above group of images can have a logical association, for example, a group of pictures can show a complete story or plot, and these stories or plots can correspond to the music content of the reference music. Through the above process, the electronic device 110 can present images that are more consistent with the music content, thereby increasing the attractiveness of the music content.

[0066] In box 320, the electronic device 110 can obtain the generated set of images. The above set of images can be presented in the interface of the electronic device 110 as part of the screen of the music video, for example. For example, a set of images can be presented at the location of the image 220 in the interface 200A. In addition, in other embodiments, the presentation position of a set of images in the video can be changed, for example, the position of a set of images can be moved up and down in the music video to fit the rhythm of the music, or it can also be moved in the interface according to a preset configuration to present a special dynamic effect.

[0067] In block 322, in some embodiments, the electronic device 110 may determine a background style of the music video based on a set of images. For example, the electronic device 110 may extract the core color of the picture or video, or the color that occupies the largest part, or the main color, based on the set of images, as the background color of the music video. In addition to the background color, the background style of the music video may also include a background pattern of the music video, etc.

[0068] In block 324, the electronic device 110 may generate a theme color of the music video based on the colors of the extracted set of images. The theme color may be applied to the music video. For example, if the content of the generated set of images is about the sea, the electronic device 110 may extract blue as the theme color of the video, and the electronic device 110 may add a color saturation change effect to the theme color as one of the dynamic effects of the music video.

[0069] In block 326, the electronic device 110 may splice or combine the above video elements, that is, the electronic device 110 may combine the video motion effects, a group of images in the music video, and the theme color of the music video to generate a complete music video.

[0070] In block 328, the electronic device 110 may provide the generated music video. For example, the electronic device 110 may directly present the generated music video in the interface, and the electronic device 110 may also save the generated music video as a video file in the memory of the electronic device 110. In addition, the electronic device 110 may also directly publish the music video to the video platform for other users to watch.

[0071] In some embodiments, similar to what is shown in the above interface 200A and interface 200B, the electronic device 110 may further generate another music video based on the music video generated in box 328. For example, the music video generated in box 328 may include music video A, and the electronic device 110 may generate music video B similar to the video style or video parameters of music video A in response to the user's operation (e.g., triggering control 210). The music video B may, for example, have a similar theme color to music video A, a set of images of a similar theme, or a similar set of motion effects. The above music video generation process can provide music video generation efficiency and reduce the amount of operations required by the user when generating a music video.

[0072] In this way, by generating music videos based on the audio features and video templates of music, the embodiments of the present disclosure can efficiently generate music videos based on the features of music and video templates, thereby improving the efficiency of music video generation, improving the fit between the music video and the music, and increasing the content richness of the music video.

[0073] In addition, through the above-mentioned automatic generation process of music videos, the embodiments of the present disclosure can efficiently and conveniently generate music videos that match the music.

[0074] Example Process

[0075] Figure 4 4 is a flowchart showing an example process 400 for generating a music video according to some embodiments of the present disclosure. The process 400 may be implemented at the electronic device 110. Figure 1 4. The process 400 is described below.

[0076] like Figure 4 As shown, in box 410, the electronic device 110 obtains a generation request, and the generation request is associated with reference music.

[0077] In box 420, the electronic device 110 provides a music video generated based on the generation request, wherein the music video is generated based on a video template and a set of images, the video template is determined based on audio features of reference music, and the set of images is generated based on description information of the reference music.

[0078] In some embodiments, the audio features are determined based on the following process: determining rhythm information of reference music; and determining the audio features of the reference music based on the rhythm information.

[0079] In some embodiments, the video template is generated based on the following process: generating a set of motion effects matching the rhythm information; and generating the video template based on the set of motion effects.

[0080] In some embodiments, at least one of the set of animations is applied to the lyrical content of the reference music.

[0081] In some embodiments, a set of images is generated based on the following process: constructing prompt words based on the description information; and providing the prompt words to an image generation model to generate a set of images.

[0082] In some embodiments, constructing a prompt word based on the description information includes: constructing a prompt word based on the description information and a preset prompt word template.

[0083] In some embodiments, the background style of the music video is determined based on a set of images.

[0084] In some embodiments, the generation request is further associated with a reference video, and the music video is further generated based on at least one attribute of the reference video.

[0085] In some embodiments, the at least one attribute includes at least one of the following: a background style of the reference video, a video template associated with the reference video, and a content layout of the reference video.

[0086] In some embodiments, the reference music includes at least one of the following: first music that has been uploaded; second music that has been selected from a group of candidate music; third music that has been determined based on query information; and fourth music that has been generated based on generation parameters.

[0087] Example devices and equipment

[0088] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 5 A schematic structural block diagram of an example apparatus 500 for generating a music video according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in the electronic device 110. Each module / component in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0089] like Figure 5 As shown, the device 500 includes: an acquisition module 510, configured to acquire a generation request, the generation request being associated with reference music; and a providing module 520, configured to provide a music video generated based on the generation request, wherein the music video is generated based on a video template and a set of images, the video template is determined based on audio features of the reference music, and the set of images is generated based on description information of the reference music.

[0090] In some embodiments, the apparatus 500 further includes a rhythm information determination module configured to: determine the rhythm information of the reference music; and determine the audio features of the reference music based on the rhythm information.

[0091] In some embodiments, the device 500 further includes a motion effect generation module configured to: generate a set of motion effects matching the rhythm information; and generate a video template based on the set of motion effects.

[0092] In some embodiments, at least one of the set of animations is applied to the lyrical content of the reference music.

[0093] In some embodiments, the apparatus 500 further includes a prompt word construction module configured to: construct prompt words based on the description information; and provide the prompt words to the image generation model to generate a set of images.

[0094] In some embodiments, the prompt word construction module is further configured to: construct prompt words based on the description information and a preset prompt word template.

[0095] In some embodiments, the background style of the music video is determined based on a set of images.

[0096] In some embodiments, the generation request is further associated with a reference video, and the music video is further generated based on at least one attribute of the reference video.

[0097] In some embodiments, the at least one attribute includes at least one of the following: a background style of the reference video, a video template associated with the reference video, and a content layout of the reference video.

[0098] In some embodiments, the reference music includes at least one of the following: first music that has been uploaded; second music that has been selected from a group of candidate music; third music that has been determined based on query information; and fourth music that has been generated based on generation parameters.

[0099] like Figure 6 As shown, the electronic device 600 is in the form of a general electronic device. The components of the electronic device 600 may include, but are not limited to, one or more processors or processing units 610, a memory 620, a storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 600.

[0100] The electronic device 600 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 600.

[0101] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 6 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0102] The communication unit 640 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0103] The input device 650 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 660 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 600, or communicate with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0104] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0105] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0106] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0107] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0108] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0109] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating a music video, comprising: Obtaining a generation request, wherein the generation request is associated with reference music; as well as A music video generated based on the generation request is provided, wherein the music video is generated based on a video template and a set of images, the video template is determined based on the audio features of the reference music, and the set of images is generated based on the description information of the reference music.

2. The method of claim 1, wherein the audio features are determined based on the following process: Determining the rhythm information of the reference music; and Based on the tempo information, the audio features of the reference music are determined.

3. The method according to claim 2, wherein the video template is generated based on the following process: generating a set of motion effects matching the rhythm information; and Based on the set of motion effects, the video template is generated. 4 . The method according to claim 3 , wherein at least one of the set of animations is applied to the lyric content of the reference music.

5. The method of claim 1, wherein the set of images is generated based on the following process: Based on the description information, construct prompt words; and The cue words are provided to an image generation model to generate the set of images.

6. The method according to claim 5, wherein constructing a prompt word based on the description information comprises: The prompt word is constructed based on the description information and a preset prompt word template.

7. The method of claim 1, wherein a background style of the music video is determined based on the set of images.

8. The method of claim 1, wherein the generation request is further associated with a reference video, and the music video is further generated based on at least one attribute of the reference video.

9. The method of claim 8, wherein the at least one attribute comprises at least one of the following: The background style of the reference video, a video template associated with the reference video, The content layout of the reference video.

10. The method according to claim 1, wherein the reference music comprises at least one of the following: The first music uploaded; second music selected from a group of candidate music; A third music determined based on the query information; The fourth music is generated based on the generation parameters.

11. An apparatus for generating a music video, comprising: An acquisition module is configured to acquire a generation request, wherein the generation request is associated with reference music; as well as A module is provided, configured to provide a music video generated based on the generation request, wherein the music video is generated based on a video template and a set of images, the video template is determined based on the audio features of the reference music, and the set of images is generated based on the description information of the reference music.

12. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 10.

Images