Video generation method and apparatus, electronic device, medium, and program product

Abstract

The present disclosure relates to a video generation method and apparatus, an electronic device, a medium, and a program product. The video generation method provided in the present disclosure is applied to a client, and comprises: in response to a video generation operation of a user, displaying a first interface; in response to an interaction operation of the user for an audio acquisition component on the first interface, acquiring a target audio; acquiring a target lyric text and a first attribute of the target audio in response to an interaction operation of the user for a confirmation component on the first interface, and displaying a second interface; and displaying a second attribute of a music video in an attribute selection area of the second interface, and displaying a target music video in a music video display area of the second interface, the target music video being determined on the basis of the target lyric text, the first attribute, and the second attribute. In this way, the video generation process is simpler and more convenient, so that the production difficulty of music videos can be effectively reduced, and the production efficiency of music videos can be improved.

Description

Video generation methods, devices, electronic equipment, media, and software products

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411223249.9, filed on September 2, 2024, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to a method, apparatus, electronic device, medium, and program product for generating video. Background Technology

[0004] In related technologies, when producing music videos, the production staff needs to have certain professional knowledge and skills to complete tasks such as lyric writing, timing calibration, material shooting, and audio and video editing. This requires a lot of time and effort, which leads to low video generation efficiency. Summary of the Invention

[0005] In view of this, the present disclosure provides a method, apparatus, electronic device, medium, and program product for generating video.

[0006] Firstly, this disclosure provides a method for generating video, applied to a client, the method comprising:

[0007] The first screen is displayed in response to the user's video generation action;

[0008] In response to user interaction with the audio acquisition component on the first interface, the target audio is acquired;

[0009] In response to the user's interaction with the confirmation component on the first interface, the target lyrics text and first attribute of the target audio are obtained, and the second interface is displayed, which includes an attribute selection area for the music video and a music video display area.

[0010] The second attribute of the music video is displayed in the attribute selection area, and the target music video is displayed in the music video display area. The target music video is determined based on the target lyrics text, the first attribute, and the second attribute.

[0011] Secondly, this disclosure provides a video generation method applied on a server, the method including:

[0012] Receive the execution request for the music video generation task sent by the client, and determine the target audio for music video generation;

[0013] Analyze the target audio to obtain the target lyrics text and the first attribute of the target audio, and send the target lyrics text and the first attribute to the client;

[0014] Determine the second attribute of the music video;

[0015] Based on the target lyrics text, the first attribute, and the second attribute, generate the target music video and send the target music video to the client in response to the execution request.

[0016] Thirdly, this disclosure provides a video generation apparatus for use on a client, the apparatus comprising:

[0017] The first display module is used to display the first interface in response to the user's video generation operation;

[0018] The first processing module is used to respond to the user's interactive operation on the audio acquisition component on the first interface and acquire the target audio.

[0019] The second display module is used to respond to the user's interactive operation on the confirmation component on the first interface, obtain the target lyrics text and the first attribute of the target audio, and display the second interface, which includes an attribute selection area for the music video and a music video display area.

[0020] The third display module is used to display the second attribute of the music video in the attribute selection area and the target music video in the music video display area. The target music video is determined based on the target lyrics text, the first attribute, and the second attribute.

[0021] Fourthly, this disclosure provides a video generation apparatus for use on a server, the apparatus comprising:

[0022] The second processing module is used to receive the execution request for the music video generation task sent by the client and determine the target audio for music video generation.

[0023] The third processing module is used to analyze the target audio, obtain the target lyrics text and the first attribute of the target audio, and send the target lyrics text and the first attribute to the client;

[0024] The determination module is used to determine the second attribute of the music video;

[0025] The fourth processing module is used to generate a target music video based on the target lyrics text, the first attribute, and the second attribute, and send the target music video to the client in response to the execution request.

[0026] Fifthly, this disclosure provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the video generation method of the first aspect or any corresponding embodiment described above, or the video generation method of the second aspect or any corresponding embodiment described above.

[0027] In a sixth aspect, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to perform a video generation method according to the first aspect or any of the corresponding embodiments described above, or a video generation method according to the second aspect or any of the corresponding embodiments described above.

[0028] In a seventh aspect, the present invention provides a computer program product, including computer instructions for causing a computer to execute the video generation method of the first aspect or any corresponding embodiment thereof, or the video generation method of the second aspect or any corresponding embodiment thereof. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure, the accompanying drawings used in the description of the specific embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 is a flowchart illustrating a video generation method according to an embodiment of the present disclosure;

[0031] Figure 2 is a schematic diagram of the display of a first interface provided according to an embodiment of the present disclosure;

[0032] Figure 3 is a schematic diagram of the display of the second interface provided according to an embodiment of the present disclosure;

[0033] Figure 4 is a flowchart illustrating another video generation method provided according to an embodiment of the present disclosure;

[0034] Figure 5 is a schematic diagram of the display of the third interface provided according to an embodiment of the present disclosure;

[0035] Figure 6 is a schematic diagram showing the attribute selection area according to an embodiment of the present disclosure;

[0036] Figure 7 is a schematic diagram of another second interface provided according to an embodiment of the present disclosure;

[0037] Figure 8 is a flowchart illustrating another video generation method provided according to an embodiment of the present disclosure;

[0038] Figure 9 is a flowchart illustrating a method for generating another video according to an embodiment of the present disclosure;

[0039] Figure 10 is a flowchart illustrating a video generation method according to another embodiment of the present disclosure;

[0040] Figure 11 is a flowchart illustrating a video generation method according to an embodiment of the present disclosure;

[0041] Figure 12 is a structural block diagram of another video generation apparatus provided according to an embodiment of the present disclosure; and

[0042] Figure 13 is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0044] In related technologies, creating music videos requires pre-producing lyrics and aligning the lyrics with the audio timeline to ensure the lyrics correspond to the audio playback. To enrich the video visuals, separate footage needs to be shot, and then editing tools are used to combine the audio with the lyrics to obtain the desired music video. This results in a high barrier to entry and a long production time for music videos.

[0045] In view of this, this disclosure provides a video generation method. After the client receives a user's video generation operation, a first interface is displayed. In response to the user's interaction with the audio acquisition component on the first interface, the target audio is acquired. In response to the user's interaction with the confirmation component on the first interface, the target lyrics text and first attribute of the target audio are acquired, and a second interface is displayed. Then, the second attribute of the music video is displayed in the attribute selection area of ​​the second interface, and the generated target music video is displayed in the music video display area of ​​the second interface. This makes the video generation process more convenient and efficient, effectively reducing the barrier to music video production and improving its production efficiency.

[0046] As an optional application scenario of the video generation method provided in this embodiment of the invention, the target music video is generated based on the interaction between the client and the server. Both the first interface and the second interface are displayed on the electronic device where the client is located. The server is used to determine the target lyrics and first attribute of the target audio, and to generate the target music video based on the target lyrics, the first attribute, and the second attribute.

[0047] According to an embodiment of this disclosure, a method for generating video is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0048] This embodiment provides a video generation method, which can be used in the aforementioned clients, such as tablets, laptops, and computers. Figure 1 is a flowchart of the video generation method according to an embodiment of this disclosure. As shown in Figure 1, the process includes the following steps:

[0049] Step S101: In response to the user's video generation operation, the first interface is displayed.

[0050] A user's video generation request can be understood as an instruction to request the generation of a music video. When a user's video generation request is received, it indicates that the user currently has a need to generate a music video; therefore, the first interface is displayed to respond to the user's video generation request.

[0051] The first interface can be understood as an entry point to guide users in creating music videos. By displaying the first interface, users can clearly understand where to start the video generation process and then perform targeted operations, thereby helping to improve generation efficiency.

[0052] Step S102: In response to the user's interactive operation on the audio acquisition component on the first interface, acquire the target audio.

[0053] As shown in Figure 2, the first interface includes an audio acquisition component that can retrieve audio files. This component can have modules or buttons for acquiring audio files, and can be configured according to needs. Responding to user interaction with the audio acquisition component on the first interface, the source of the audio file can be clearly identified, allowing the acquisition of the audio file used to generate the music video. This satisfies the user's personalized needs and provides a foundation for subsequent music video production.

[0054] In step S103, in response to the user's interaction with the confirmation component on the first interface, the target lyrics text of the target audio and the first attribute are obtained, and the second interface is displayed.

[0055] As shown in Figure 2, a confirmation component is pre-set on the first interface so that the user's final selected audio can be clearly identified based on the interaction with this component. This confirmation component can be a button, icon, or other interactive interface element, depending on the actual layout requirements.

[0056] In response to the user's interaction with the confirmation component on the first interface, and determining that the acquired target audio can be used to create a music video, to improve the accuracy of video generation, the target lyrics text and the first attribute of the target audio are acquired. The target lyrics text clarifies the musical content of the target audio, and the first attribute clarifies the audio style, thus providing a multi-dimensional understanding of the audio information. The first attribute can include information such as the musical style, rhythm, instruments involved, and their performance and combination in the music. Musical style can include genres such as pop, rock, classical, and jazz. Musical rhythm can include the beat and tempo, such as fast or slow tempo.

[0057] In some optional examples, the initial lyrics text of the target audio is obtained by processing it using a pre-defined phono-text conversion model. In other optional examples, the initial lyrics text of the target audio is obtained by searching a pre-defined lyrics library based on audio name matching.

[0058] After responding to the user's interaction with the confirmation component on the first interface, a second interface, as shown in Figure 3, is displayed, allowing the user to quickly preview the generated video. The second interface includes an attribute selection area and a video display area. The attribute selection area is used to determine the second attributes of the music video. These attributes include, but are not limited to, the video's visual style, resolution, and aspect ratio, which can be determined according to requirements. The video display area showcases the generated video.

[0059] Therefore, by displaying a second interface, users can more intuitively select music video attributes and preview the generated effects, thereby helping to improve the efficiency of identifying target music videos.

[0060] Step S104: Display the second attribute of the music video in the attribute selection area, and display the target music video in the music video display area.

[0061] The target music video is determined based on the target lyrics text, the first attribute, and the second attribute, which helps to ensure that the generated target music video is more in line with the content expressed by the music in the target audio and the user's generation needs.

[0062] The video generation method provided in this embodiment displays a first interface after the client receives the user's video generation operation. This helps to clearly define the entry point for music video production. Then, in response to the user's interaction with the audio acquisition component on the first interface, the target audio is acquired. In response to the user's interaction with the confirmation component on the first interface, the target lyrics and first attribute of the target audio are acquired, allowing for a multi-dimensional understanding of the audio information of the target audio. A second interface is then displayed, showing the second attributes of the music video in the attribute selection area and the generated target music video in the music video display area. This allows users to more intuitively select music video attributes and preview the generation effect, making the video generation process more convenient and efficient. This effectively lowers the barrier to music video production and improves production efficiency.

[0063] This embodiment provides a video generation method, which can be used in the aforementioned clients, such as tablets, laptops, and computers. Figure 4 is a flowchart of the video generation method according to an embodiment of this disclosure. As shown in Figure 4, the process includes the following steps:

[0064] Step S401: In response to the user's video generation operation, the first interface is displayed. For details, please refer to step S101 of the embodiment shown in Figure 1, which will not be repeated here.

[0065] Step S402: In response to the user's interaction with the audio acquisition component on the first interface, the target audio is acquired. For details, please refer to step S102 of the embodiment shown in Figure 1, which will not be repeated here.

[0066] In step S403, in response to the user's interaction with the confirmation component on the first interface, the target lyrics text of the target audio and the first attribute are obtained, and the second interface is displayed.

[0067] Specifically, step S403 includes:

[0068] In step S4031, in response to the user's interaction with the confirmation component on the first interface, the initial lyrics text and first attribute of the target audio are obtained, and the second interface is displayed.

[0069] The initial lyrics text can be understood as lyrics text that has not been modified or edited.

[0070] In step S4032, in response to the user's interactive operation on the lyrics control in the second interface, the initial lyrics text is displayed on the third interface.

[0071] A lyrics control is a button or icon used to trigger the display of lyrics. As shown in Figure 2, a lyrics control is pre-set in the second interface to meet the user's need to view the lyrics of the target audio.

[0072] In response to the user's interaction with the lyrics control on the second interface, indicating that the user needs to view the lyrics of the target audio, the third interface is displayed, and the initial lyrics text is displayed through the third display interface so that the user can browse the lyrics content more intuitively and clearly.

[0073] In step S4033, in response to the modification operation on the initial lyrics text, the target lyrics text is displayed on the third interface.

[0074] Since the initial lyrics text may differ from the actual lyrics content of the target audio or fail to meet user needs, the third interface also has a function to modify the initial lyrics text. In response to modification operations on the initial lyrics text, the initial lyrics text is modified, and the modification results are displayed on the third interface, resulting in the final target lyrics text displayed on the third interface. Modification operations can include deleting, adding, and replacing text, adjusting the paragraph structure of the lyrics, and modifying the rhythm of the lyrics, depending on the specific modification operation. In other words, the target lyrics text can be understood as the final lyrics text determined after performing modification operations on the initial lyrics text. The number of modification operations performed on the initial lyrics text can be determined according to user needs.

[0075] By allowing modification of the initial lyrics text, it can be ensured that the resulting target lyrics text meets the user's needs. Consequently, when generating music videos using the target lyrics text, the generated music videos will better meet the user's needs, thus improving the user experience.

[0076] In some optional implementations, the interface diagram of the third interface is shown in Figure 5. The initial lyrics text includes a one-to-one correspondence of lyrics and lyrics playback time. Modification operations include any one or more combinations of the following operations: lyrics modification operation and time modification operation. Specifically, the lyrics modification operation is triggered in response to user interaction with the lyrics editing component in the third interface, and the time modification operation is triggered in response to user interaction with the time editing component in the third interface. The lyrics editing component is used to edit the initial lyrics text of the corresponding lyrics, and the time editing component is used to edit the corresponding lyrics playback time.

[0077] In other words, in the third interface, each lyric has its own editable lyric editing component, and each lyric playback time has its own editable timing component. By setting up several lyric and timing editing components on the third interface, modifications to the initial lyric text become more flexible and easier to meet personalized needs. This allows the target lyric text to better match the music in the target audio, thereby improving the quality of the generated music video.

[0078] Step S404: Display the second attribute of the music video in the attribute selection area, and display the target music video in the music video display area.

[0079] The video generation method provided in this embodiment responds to the interactive operation with the lyrics control in the second interface during the music video generation process and displays the initial lyrics text on the third interface. This can simultaneously meet the user's needs to view and modify lyrics, thereby not only increasing the interactivity with the user, but also making the generated target music video more in line with the user's expectations in terms of lyrics, thus effectively improving the overall generation quality of the music video.

[0080] In some optional implementations, as shown in Figure 6, the attribute selection area includes at least one type of attribute selection control, and the method further includes:

[0081] Step a1: In response to the user's interaction with the attribute selection control, determine the attribute selection result for the corresponding type;

[0082] Step a2: Based on the attribute selection results of the corresponding type, obtain the second attribute of the music video.

[0083] Specifically, to make the generated target music video better meet the user's needs, the generation effect of the video can be specifically set through the attribute selection control in the attribute selection area of ​​the second interface.

[0084] Each attribute selection control includes multiple attribute options corresponding to its type. In response to user interaction with the attribute selection control, the currently selected attribute option is identified and used as the attribute selection result for that type of attribute selection control. For example, if the attribute type of the attribute selection control is video aspect ratio, the corresponding attribute options could be: 16:9, 4:3, 3:2, 1:1, 9:16, 3:4, and 2:3. In response to user interaction with the attribute selection control, when the selected attribute option is determined to be 16:9, then 16:9 is used as the attribute selection result for the video aspect ratio.

[0085] If the attribute selection control corresponds to the attribute type of video resolution, then the corresponding attribute options can be: 240p, 360p, 480p, 720p, and 1080p. In response to user interaction with the attribute selection control, when the selected attribute option is determined to be 720p, then 720p will be used as the video resolution attribute selection result.

[0086] If the attribute selection control corresponds to the subtitle color attribute, the corresponding attribute options can be: gradient color or white. In response to user interaction with the attribute selection control, when the selected attribute option is determined to be gradient color, the gradient color is used as the attribute selection result for the video resolution.

[0087] When the attribute selection area contains only one type of attribute selection control, the attribute selection result corresponding to that type of control is used as the second attribute of the music video. However, when the attribute selection area contains multiple types of attribute selection controls, the second attribute of the music video is a collection of multiple attribute selection results. In some optional examples, the more diverse the types of attribute selection controls, the more closely the final generated music video will match the user's needs.

[0088] By interacting with the attribute selection control, the second attribute of the music video can be determined, making the final target music video more diverse and thus meeting the user's generation needs in different scenarios. This makes the video generation method more flexible and targeted.

[0089] In some optional implementations, in conjunction with the first interface shown in Figure 2, the process of acquiring the target audio in response to user interaction with the audio acquisition component on the first interface includes:

[0090] Step b1: In response to the user's interaction with the audio acquisition component on the first interface, display the audio input interface;

[0091] Step b2: In response to input instructions on the audio input interface, acquire the target audio.

[0092] Specifically, the audio acquisition component triggers the display of an audio input interface, allowing the user to provide relevant audio access information through this interface. In response to the user's interaction with the audio acquisition component on the first interface, and determining that the user needs to specify the target audio to be acquired, the audio input interface is displayed.

[0093] In response to an input command on the audio input interface, the source of the target audio is identified, and the target audio is obtained from that source. The input command includes at least one of an audio name input command and an audio path input command.

[0094] For example: If the input command is an audio name input command, the source can be the audio playback history or search history of relevant music platforms. If the input command is an audio path input command, the source can be the storage path of the target audio, an audio link shared in an instant message, or an audio link of a specified audio resource. The storage path of the target audio can be a folder on the local device (local storage path) or a specific location in a cloud storage service (cloud storage path).

[0095] By obtaining the target audio in the above manner, the interaction process can be clearly defined, allowing users to easily provide the required audio to the client for processing. This makes the user's operation more convenient and effectively reduces the time and effort spent by users in searching for and selecting audio, thereby helping to accelerate the production progress of music videos and improve generation efficiency.

[0096] In some optional implementations, as shown in Figure 7, the second interface includes a name modification control and is also used to display the name of the target music video. The method further includes:

[0097] Step c1: In response to the user's interaction with the name modification control, obtain the modified name of the target music video;

[0098] Step c2: The modified name of the target music video is displayed on the second interface.

[0099] Specifically, to facilitate users' identification and management of the generated target music video, the name of the target music video is displayed on the second display interface.

[0100] If a user is not satisfied with the name of the target music video, they can modify it through interaction with the name editing control. The client can respond to the user's interaction with the name editing control, specify the modification content, obtain the modified name of the target music video, and display it through a second display interface. This not only satisfies the personalized setting of the music video name but also allows the user to intuitively see the modified result.

[0101] Based on the same inventive concept, the present invention also provides a method for generating video applied to the aforementioned server.

[0102] This embodiment provides a video generation method, which can be used in the aforementioned server-side applications, such as servers and cloud platforms. Figure 8 is a flowchart of the video generation method according to an embodiment of this disclosure. As shown in Figure 8, the process includes the following steps:

[0103] Step S801: Receive the execution request for the music video generation task sent by the client, and determine the target audio for music video generation.

[0104] The execution request is used to instruct the server to perform a music video generation task. Upon receiving the execution request from the client, the server can identify task information related to the music video generation task, and then obtain the target audio for music video production from this task information. In some examples, the task information includes the target audio or the source of the target audio, ensuring that the server can successfully obtain the target audio when it initiates the music video generation task, thus guaranteeing the successful execution of the task.

[0105] Step S802: Analyze the target audio, obtain the target lyrics text and the first attribute of the target audio, and send the target lyrics text and the first attribute to the client.

[0106] To reduce the computational burden on the client, after obtaining the target audio, a targeted analysis is performed on the target audio to determine the target lyrics text and the first attribute of the target audio based on the analysis results.

[0107] The target lyrics text can be obtained through speech recognition technology. For example, a pre-built speech-to-text conversion model can be used to identify the lyrics in the target audio and convert the identified lyrics into text, thus obtaining the target lyrics text.

[0108] The primary attribute can be obtained through audio feature analysis of the target audio. For example, features such as rhythm, melody, pitch, timbre, and volume of the target audio can all be used as the primary attribute. Analyzing these audio features yields descriptive information about the target audio, such as a brisk rhythm, beautiful melody, high pitch, mellow timbre, and loud volume. Alternatively, the primary attribute can be defined and selected based on actual needs and application scenarios. For example, in music classification, music style (such as pop, rock, classical, etc.) can be used as the primary attribute.

[0109] After the analysis is completed, the server sends the obtained target lyrics text and first attribute back to the client so that the client can understand the relevant information of the target audio and provide a reference for subsequent operations.

[0110] Step S803: Determine the second attribute of the music video.

[0111] To improve the quality of video generation, a second attribute of the music video is determined. This second attribute clarifies the generation effect of the target video, thereby making the generated target music video more in line with the user's expectations and needs.

[0112] The second attribute can include, but is not limited to, attributes such as the video's generation style (e.g., romantic, dynamic, retro), visual style, color scheme, and editing rhythm, which can be determined according to actual needs.

[0113] Step S804: Based on the target lyrics text, the first attribute, and the second attribute, generate the target music video and send the target music video to the client in response to the execution request.

[0114] After defining the target lyrics, primary attribute, and secondary attribute, video editing techniques can be used to combine and edit images, video clips, and other materials related to the target lyrics to match the primary attribute of the target audio. Simultaneously, based on the visual style, color scheme, and editing rhythm requirements determined by the secondary attribute, the generated music video can be further optimized and adjusted to create a target music video that matches the target audio. Once the target music video is generated, the server sends it back to the client, allowing the client to display the generated target music video to the user through a secondary interface, thus completing the response to the execution request.

[0115] The video generation method provided in this embodiment lays the foundation for subsequent video generation by receiving an execution request from the client and determining the target audio. By analyzing the target audio to obtain the lyrics and a first attribute, and sending them to the client, the client can better understand the audio content, providing a basis for generating a music video that better meets the requirements. Furthermore, based on the determined second attribute of the music video, the generation style and characteristics of the video can be further clarified. Thus, based on the target lyrics, the first attribute, and the second attribute, a target music video is generated and sent to the client. This not only meets the user's music video generation needs but also improves the efficiency and quality of music video generation.

[0116] In some alternative implementations, the process of generating a target music video based on the target lyrics text, a first attribute, and a second attribute includes:

[0117] Step d1: Understand the text content of the target lyrics text, and based on the first attribute and the text content, process it through a preset storyboard generation model to obtain the video storyboard content;

[0118] Step d2: Generate the target music video based on the video storyboard content, the target lyrics text, and the second attribute.

[0119] Specifically, understanding the textual content of the target lyrics requires in-depth analysis of the lyrics' meaning and theme. Simultaneously, combining the primary attributes of the target audio, such as rhythm and style, provides more reference information for the storyboard generation model.

[0120] The pre-built storyboard generation model generates a series of video storyboard content based on the input text content and primary attributes. Leveraging the capabilities of a large language model, it utilizes deep learning algorithms and image processing techniques to produce these storyboards. These storyboards are designed with factors such as the plot development of the lyrics, the rhythm of the music, and the atmosphere in mind, including considerations for camera angles, shot sizes, movement, and composition. For example, if the target lyrics describe a cheerful scene and the target audio has a fast tempo and a lively style, the storyboard generation model might generate dynamic shots, such as rapidly switching close-ups and extreme close-ups, as well as dynamic camera movements.

[0121] After obtaining the video storyboard content, it is combined with the target lyrics text and the second attribute of the music video for further processing and optimization. This allows for the selection of suitable images, video materials, or animation to create scenes that correspond to the storyboard, thereby generating the final target music video.

[0122] In some optional examples, step d1 above includes:

[0123] Step d11: Generate storyboard images based on the video storyboard content and the second attribute;

[0124] Step d12: Based on the lyrics playback time in the target lyrics text, establish the correspondence between the lyrics and the storyboard images;

[0125] Step d13: Generate the target music video based on the correspondence.

[0126] Specifically, since the second attribute may involve requirements regarding the video's style, color, etc., which affect the generation of storyboard images, storyboard images are first created based on the video storyboard content and the second attribute. Then, according to the playback time of the lyrics in the target lyrics text, a correspondence between the lyrics and storyboard images is constructed to ensure that the content of the lyrics and the storyboard images can be accurately matched in time, enhancing the expressiveness and coherence of the music video. For example: the target audio is a pop song with a cheerful rhythm and a vibrant style, and the target lyrics text describes A enjoying life to the fullest in summer. Based on the video storyboard content and the second attribute, the resulting video storyboard content may include scenes such as A running on the beach, surfing, and playing games with friends. The second attribute stipulates that the overall style of the music video should be full of sunshine and vitality, with bright colors. The generated storyboard images can be as follows: one image shows A running happily on a golden beach, with a background of a clear blue sea and bright sunshine, the image is bright and vibrant. Another image shows A standing on a surfboard, speeding across the turbulent sea, with splashing waves, the image is full of dynamism. Another picture shows friends sitting on the beach, laughing and chatting, creating a warm and lively scene.

[0127] By leveraging the established correspondence, the final target music video is generated, enabling the video to express the target audio content richly and accurately through visuals, thereby improving the quality and reliability of the generated video.

[0128] In other optional examples, the desired storyboard content can be obtained by visually transforming key elements extracted from the target lyrics text. For example, by carefully analyzing the target lyrics text, key elements such as characters, scenes, and actions can be extracted, and then combined with a second attribute, a pre-defined storyboard generation model can be used for visual transformation to obtain the desired storyboard images.

[0129] In some optional implementations, the process of determining the second attribute of the music video includes: receiving music attribute information sent by the client, and determining the second attribute of the music video based on the music attribute information. That is, the second attribute is determined by the client, which ensures the reliability of the second attribute and makes the video generation process more targeted and easier to meet user needs.

[0130] This embodiment provides a video generation method, which can be used on the aforementioned server, such as a server or cloud platform. Figure 9 is a flowchart of the video generation method according to an embodiment of this disclosure. As shown in Figure 9, the process includes the following steps:

[0131] Step S901: Receive the execution request for the music video generation task sent by the client, and determine the target audio for music video generation.

[0132] Step S902: Analyze the target audio to obtain the target lyrics text and the first attribute of the target audio, and send the target lyrics text and the first attribute to the client.

[0133] Step S903: Determine the second attribute of the music video.

[0134] Step S904: Based on the target lyrics text, the first attribute, and the second attribute, generate the target music video and send the target music video to the client in response to the execution request.

[0135] Step S905: In response to the received status query instruction, determine the current execution status of the music video generation task.

[0136] When the server receives a status query command from the client, it begins checking the current progress of the music video generation task. This execution status may include the following: 1. Preparing materials, such as collecting music files, lyrics, and any necessary images or video clips. 2. Designing storyboards, determining the various shots and frames of the video based on the music and lyrics. 3. Editing the video, combining materials into a complete video, and adding effects, transitions, etc. 4. Generation completed; the music video is finished and awaits further processing (such as compression and uploading). 5. An error occurred during the generation process, such as missing materials or software malfunction, causing the task to be unable to continue.

[0137] The server will determine the current stage based on the actual task execution status and use this information as the current execution status.

[0138] In some optional examples, the client sends status query commands via the queryTask interface in a polling manner to keep track of the progress of the music video generation task.

[0139] Step S906: Send the current execution status to the client so that the client can control the progress of the server in executing the music video generation task based on the current execution status.

[0140] After determining the current execution status of the music video generation task on the server side, it sends this information back to the client. Upon receiving this status information, the client can take appropriate actions to control the task progress on the server side as needed.

[0141] In some optional examples, the client controls the progress of the server-side music video generation task based on the current execution state. This includes: if the current execution state indicates that the music video generation task has failed or the execution duration exceeds a specified execution duration threshold, it indicates that there is an anomaly in the execution process of the music video generation task. Execution failure indicates that an error occurred during the music video generation process, causing the task to fail to complete normally. Examples include incorrect material processing, software crashes, and data loss. An execution duration exceeding the specified execution duration threshold indicates that the execution time of the music video generation task has exceeded the expected reasonable duration (the specified execution duration threshold). For example, slow task progress may be caused by lag, excessive resource consumption, or other performance issues at certain stages.

[0142] Therefore, in order to reduce the client's waiting time and improve the efficiency of target video generation, the server is controlled to re-execute the music video generation task to avoid potential problems from continuously affecting the completion of the music video generation task.

[0143] In some alternative examples, if the current execution status indicates that the music video generation task has been successfully executed, the generated target music video is directly obtained. If the current execution status indicates that the execution time of the music video generation task is less than a specified execution time threshold, it indicates that there is no abnormality in the execution process of the music video generation task. Therefore, the music video generation task can be controlled to continue execution until the target music video is obtained.

[0144] The video generation method provided in this embodiment can improve video generation efficiency by enabling the client to better understand the progress of the music video generation task performed by the server based on the interaction between the client and the server.

[0145] As one or more specific application embodiments of this disclosure, as shown in Figure 10, after obtaining the target audio, the client sends the target audio to the server for lyrics recognition processing to obtain the target lyrics text, and the server determines the first attribute of the target audio. The user can correct the target lyrics text of the target audio on a third interface of the client. Based on the first attribute and the text content, a preset storyboard generation model is used to process the data to obtain video storyboard content. Based on the video storyboard content, the target lyrics text, and the second attribute, a target music video is generated.

[0146] The video generation method provided by this invention is easy to operate, which not only lowers the threshold for music video production and improves the efficiency of music video production, but also enriches the generation effects of the video.

[0147] This embodiment also provides a video generation apparatus for implementing the embodiments described above; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0148] This embodiment provides a video generation device applied to a client, as shown in Figure 11, including:

[0149] The first display module 1101 is used to display the first interface in response to the user's video generation operation;

[0150] The first processing module 1102 is used to respond to the user's interactive operation on the audio acquisition component on the first interface and acquire the target audio.

[0151] The second display module 1103 is used to respond to the user's interactive operation on the confirmation component on the first interface, obtain the target lyrics text and the first attribute of the target audio, and display the second interface, which includes an attribute selection area for the music video and a music video display area.

[0152] The third display module 1104 is used to display the second attribute of the music video in the attribute selection area and the target music video in the music video display area. The target music video is determined based on the target lyrics text, the first attribute, and the second attribute.

[0153] In some optional implementations, the second display module 1103 includes: an acquisition unit, configured to acquire the initial lyrics text and first attribute of the target audio in response to a user's interactive operation on a confirmation component on a first interface, and display the second interface; a display switching unit, configured to display the initial lyrics text on a third interface in response to a user's interactive operation on a lyrics control in the second interface; and an adjustment unit, configured to display the target lyrics text on the third interface in response to a modification operation on the initial lyrics text.

[0154] In some optional implementations, the initial lyrics text includes a one-to-one correspondence between lyrics and lyrics playback time. The modification operation includes a lyrics modification operation and / or a time modification operation. The lyrics modification operation is triggered in response to the user's interaction with the lyrics editing component in the third interface, and the time modification operation is triggered in response to the user's interaction with the time editing component in the third interface. The lyrics editing component is used to edit the initial lyrics text of the corresponding lyrics, and the time editing component is used to edit the corresponding lyrics playback time.

[0155] In some optional implementations, the attribute selection area includes at least one type of attribute selection control, and the device further includes: a selection module for determining the attribute selection result of the corresponding type in response to the user's interactive operation on the attribute selection control; and an attribute determination module for obtaining a second attribute of the music video based on the attribute selection result of the corresponding type.

[0156] In some optional implementations, the first processing module 1102 includes: a first processing unit, configured to display an audio input interface in response to a user's interactive operation on the audio acquisition component on the first interface; and a second processing unit, configured to acquire target audio in response to an input instruction on the audio input interface, wherein the input instruction includes at least one of an audio name input instruction and an audio path input instruction.

[0157] In some optional implementations, the second interface includes a name modification control and is also used to display the name of the target music video. The device further includes: an adjustment module for obtaining the modified name of the target music video in response to the user's interaction with the name modification control; and a fourth display module for displaying the modified name of the target music video on the second interface.

[0158] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0159] This embodiment provides a video generation device applied to a server, as shown in Figure 12, including:

[0160] The second processing module 1201 is used to receive the execution request for the music video generation task sent by the client and determine the target audio for music video generation.

[0161] The third processing module 1202 is used to analyze the target audio, obtain the target lyrics text and the first attribute of the target audio, and send the target lyrics text and the first attribute to the client;

[0162] Module 1203 is used to determine the second attribute of the music video;

[0163] The fourth processing module 1204 is used to generate a target music video based on the target lyrics text, the first attribute, and the second attribute, and send the target music video to the client in response to the execution request.

[0164] In some optional implementations, the fourth processing module 1204 includes: a third processing unit, used to understand the text content of the target lyrics text, and based on the first attribute and the text content, process it through a preset storyboard generation model to obtain video storyboard content; and a video generation unit, used to generate a target music video based on the video storyboard content, the target lyrics text, and the second attribute.

[0165] In some optional implementations, the video generation unit includes: a first execution unit, configured to generate a storyboard image based on the video storyboard content and a second attribute; a second execution unit, configured to establish a correspondence between lyrics and storyboard images based on the lyrics playback time in the target lyrics text; and a third execution unit, configured to generate a target music video based on the correspondence.

[0166] In some optional implementations, the determining module 1203 includes: an attribute determining unit, configured to receive music attribute information sent by the client and determine a second attribute of the music video based on the music attribute information.

[0167] In some optional implementations, the apparatus further includes: a status detection module, configured to determine the current execution status of the music video generation task in response to a received status query command; and a sending module, configured to send the current execution status to the client, so that the client can control the progress of the music video generation task executed by the server based on the current execution status.

[0168] In some optional implementations, the device for controlling the progress of the music video generation task executed by the server based on the current execution state includes: a control module, used to control the server to re-execute the music video generation task if the current execution state indicates that the music video generation task has failed or the execution duration is greater than a specified execution duration threshold.

[0169] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0170] In this embodiment, the video generation device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0171] This disclosure also provides an electronic device having the video generation apparatus shown in FIG11 or FIG12 above.

[0172] Please refer to Figure 13, which is a schematic diagram of the structure of an electronic device provided in an optional embodiment of this disclosure. As shown in Figure 13, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other using different buses and can be mounted on a common motherboard or otherwise installed as needed. The processor can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 13 uses one processor 10 as an example.

[0173] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0174] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0175] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0176] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0177] The electronic device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30 and output device 40 can be connected via a bus or other means, as shown in Figure 13, which illustrates a bus connection.

[0178] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touch screen.

[0179] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded over a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium may be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0180] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0181] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0182] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0183] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0184] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0185] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A video generation method, applied to a client, comprising: The first screen is displayed in response to the user's video generation action; In response to the user's interaction with the audio acquisition component on the first interface, the target audio is acquired; In response to the user's interaction with the confirmation component on the first interface, the target lyrics text and first attribute of the target audio are obtained, and a second interface is displayed, which includes an attribute selection area for the music video and a music video display area. The second attribute of the music video is displayed in the attribute selection area, and the target music video is displayed in the music video display area. The target music video is determined based on the target lyrics text, the first attribute, and the second attribute.

2. The method according to claim 1, wherein, The step of responding to a user's interaction with the confirmation component on the first interface, obtaining the target lyrics text and first attribute of the target audio, and displaying the second interface includes: In response to the user's interaction with the confirmation component on the first interface, the initial lyrics text and first attribute of the target audio are obtained, and the second interface is displayed; In response to the user's interaction with the lyrics control in the second interface, the initial lyrics text is displayed in the third interface; In response to the modification operation on the initial lyrics text, the target lyrics text is displayed on the third interface.

3. The method according to claim 2, wherein, The initial lyrics text includes a one-to-one correspondence between lyrics and lyrics playback time. The modification operation includes lyrics modification operation and / or time modification operation. The lyrics modification operation is triggered in response to the user's interaction with the lyrics editing component in the third interface. The time modification operation is triggered in response to the user's interaction with the time editing component in the third interface. The lyrics editing component is used to edit the initial lyrics text of the corresponding lyrics, and the time editing component is used to edit the corresponding lyrics playback time.

4. The method according to any one of claims 1-3, wherein, The attribute selection area includes at least one type of attribute selection control, and the method further includes: In response to the user's interactive operation on the attribute selection control, determine the attribute selection result of the corresponding type; Based on the attribute selection result of the corresponding type, the second attribute of the music video is obtained.

5. The method according to any one of claims 1-4, wherein, The step of acquiring the target audio in response to user interaction with the audio acquisition component on the first interface includes: In response to the user's interaction with the audio acquisition component on the first interface, the audio input interface is displayed; In response to an input instruction on the audio input interface, the target audio is acquired, wherein the input instruction includes at least one of an audio name input instruction and an audio path input instruction.

6. The method according to any one of claims 1-5, wherein, The second interface includes a name modification control, and the second interface is also used to display the name of the target music video. The method further includes: In response to the user's interaction with the name modification control, the modified name of the target music video is obtained; The modified name of the target music video is displayed on the second interface.

7. A video generation method, applied on a server, comprising: Receive the execution request for the music video generation task sent by the client, and determine the target audio for music video generation; Analyze the target audio to obtain the target lyrics text and the first attribute of the target audio, and send the target lyrics text and the first attribute to the client; Determine the second attribute of the music video; Based on the target lyrics text, the first attribute, and the second attribute, a target music video is generated and sent to the client in response to the execution request.

8. The method according to claim 7, wherein, The step of generating a target music video based on the target lyrics text, the first attribute, and the second attribute includes: The text content of the target lyrics is understood, and based on the first attribute and the text content, it is processed by a preset storyboard generation model to obtain video storyboard content. The target music video is generated based on the video storyboard content, the target lyrics text, and the second attribute.

9. The method according to claim 8, wherein, The process of generating the target music video based on the video scene content, the target lyrics text, and the second attribute includes: Generate a storyboard image based on the video storyboard content and the second attribute; Based on the lyrics playback time in the target lyrics text, establish a correspondence between the lyrics and the storyboard images; The target music video is generated based on the aforementioned correspondence.

10. The method according to any one of claims 7-9, wherein, Determining the second attribute of the music video includes: The system receives music attribute information sent by the client and determines a second attribute of the music video based on the music attribute information.

11. The method according to any one of claims 7-10, further comprising: In response to the received status query command, determine the current execution status of the music video generation task; The current execution status is sent to the client so that the client can control the progress of the server in executing the music video generation task based on the current execution status.

12. The method according to claim 11, wherein, The process by which the client controls the progress of the server in executing the music video generation task based on the current execution state includes: If the current execution status indicates that the music video generation task has failed or the execution time exceeds a specified execution time threshold, then the server is controlled to re-execute the music video generation task.

13. A video generation apparatus, applied to a client, comprising: The first display module is configured to display a first interface in response to a user's video generation operation; The first processing module is configured to acquire the target audio in response to the user's interactive operation on the audio acquisition component on the first interface; The second display module is configured to respond to the user's interactive operation on the confirmation component on the first interface, obtain the target lyrics text and the first attribute of the target audio, and display the second interface, which includes an attribute selection area for the music video and a music video display area. The third display module is configured to display the second attribute of the music video in the attribute selection area and to display the target music video in the music video display area, wherein the target music video is determined based on the target lyrics text, the first attribute, and the second attribute.

14. A video generation apparatus, applied on a server, comprising: The second processing module is configured to receive the execution request for the music video generation task sent by the client and determine the target audio for music video generation. The third processing module is configured to analyze the target audio, obtain the target lyrics text and the first attribute of the target audio, and send the target lyrics text and the first attribute to the client; The determination module is configured to determine a second attribute of the music video; The fourth processing module is configured to generate a target music video based on the target lyrics text, the first attribute, and the second attribute, and send the target music video to the client in response to the execution request.

15. An electronic device comprising: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the video generation method of any one of claims 1 to 6 or the video generation method of any one of claims 7 to 12.

16. A computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the video generation method according to any one of claims 1 to 6 or the video generation method according to any one of claims 7 to 12.

17. A computer program product comprising computer instructions, wherein, The computer instructions are used to cause the computer to execute the video generation method according to any one of claims 1 to 6 or the video generation method according to any one of claims 7 to 12.

Images