Singing video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a singing video generation method and device, electronic equipment and a storage medium, after target music is generated through a target application program, a release page corresponding to the target music is displayed, and a video generation control used for generating a singing video matched with the target music is configured in the release page; in response to a first trigger operation for the video generation control, displaying a first video generation page in the target application, the first video generation page being configured to display at least one video material; and in response to a second trigger operation for the first video generation page, determining a target video material, and generating a singing video based on the target video material, the mouth shape action of the target object in the singing video being matched with the music content of the target music. According to the method and the device, the'mouth-to-mouth 'video matched with the target music is generated, the process is realized without other three-party software, and the generation efficiency of the singing video is improved.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of Internet technology, and in particular to a method, device, electronic device and storage medium for generating a singing video. Background Art

[0002] Currently, Artificial Intelligence Generated Content (AIGC) technology has been increasingly used in various industries. For example, by training a music generation model through music samples, the model can be given the ability to "create" music, thereby generating "original music" that meets user requirements, that is, intelligently generated music.

[0003] After intelligently generated music with melody and corresponding lyrics is generated through intelligent content generation technology, the intelligently generated music can be disseminated through platform publishing, sharing, etc. However, since there is no matching singing video, the dissemination effect of the intelligently generated music is poor. In related technologies, the intelligently generated music can be processed through additional third-party software to generate the corresponding video, and then returned to the original intelligently generated music generation platform for publishing or sharing.

[0004] However, in the above process of generating corresponding videos for intelligently generated music, there are problems of low interaction efficiency and low matching degree between the generated videos and the intelligently generated music. Summary of the invention

[0005] The embodiments of the present disclosure provide a singing video generation method, device, electronic device and storage medium to overcome the problems of low interaction efficiency in the process of generating corresponding videos for intelligently generated music, and low matching degree between the generated video and the intelligently generated music.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for generating a singing video, comprising:

[0007] After generating the target music through the target application, a publishing page corresponding to the target music is displayed in the target application, wherein the publishing page is configured with a video generation control for generating a singing video matching the target music; in response to a first trigger operation on the video generation control, a first video generation page is displayed in the target application, and the first video generation page is configured to display at least one video material; in response to a second trigger operation on the first video generation page, the target video material is determined, and the singing video is generated based on the target video material, wherein the singing video and the target video material contain the same target object, and the lip movements of the target object in the singing video match the music content of the target music.

[0008] In a second aspect, an embodiment of the present disclosure provides a singing video generating device, comprising:

[0009] A first display module is used to display a publishing page after generating the target music, wherein the publishing page is configured with a video generation control for generating a performance video matching the target music;

[0010] A second display module, configured to display a first video generation page in the target application in response to a first trigger operation on the video generation control, wherein the first video generation page is configured to display at least one video material;

[0011] A generation module is used to determine the target video material in response to a second trigger operation on the first video generation page, and generate the singing video based on the target video material, wherein the singing video and the target video material contain the same target object, and the lip movements of the target object in the singing video match the music content of the target music.

[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0013] The memory stores computer-executable instructions;

[0014] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the singing video generation method described in the first aspect and various possible designs of the first aspect.

[0015] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the singing video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0016] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the singing video generation method as described in the first aspect and various possible designs of the first aspect.

[0017] The singing video generation method, device, electronic device and storage medium provided in the present embodiment generate target music through a target application, and then display a publishing page corresponding to the target music in the target application, wherein the publishing page is configured with a video generation control for generating a singing video matching the target music; in response to a first trigger operation on the video generation control, display a first video generation page in the target application, wherein the first video generation page is configured to display at least one video material; in response to a second trigger operation on the first video generation page, determine the target video material, and generate the singing video based on the target video material, wherein the singing video and the target video material contain the same target object, and the lip movements of the target object in the singing video match the music content of the target music. By configuring a video generation control in the publishing page after the intelligently generated music is generated, the client can directly jump to the first video generation page for generating a singing video matching the intelligently generated music after the intelligently generated music is generated, and the corresponding video material is displayed. Then, based on the selected target video material, a target music containing a target object that is consistent with the target video material and can perform lip movements matching the music content is generated, that is, a "lip-sync" video matching the target music. This process does not require the use of other third-party software, thereby improving the generation efficiency of the singing video. Moreover, since it does not involve cross-platform image transmission and avoids audio and image compression processes, the generated singing video can have a higher matching degree with the intelligently generated music and a higher video clarity, resulting in a better video effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 An application scenario diagram of the singing video generation method provided in the embodiment of the present disclosure;

[0020] Figure 2 A schematic diagram of the process of the singing video generation method provided in the embodiment of the present disclosure Figure 1 ;

[0021] Figure 3 for Figure 2 A flowchart of a specific implementation method of step S102 in the illustrated embodiment;

[0022] Figure 4A schematic diagram of a first video generation page provided in an embodiment of the present disclosure;

[0023] Figure 5 A flowchart of a specific implementation method for generating a singing video based on a target video material;

[0024] Figure 6 A schematic diagram of the process of the singing video generation method provided in the embodiment of the present disclosure Figure 2 ;

[0025] Figure 7 for Figure 6 A flowchart of a specific implementation method of step S207 in the illustrated embodiment;

[0026] Figure 8 A schematic diagram of a process for generating a singing video provided by an embodiment of the present disclosure;

[0027] Fig. 9 A structural block diagram of a singing video generating device provided in an embodiment of the present disclosure;

[0028] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0029] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] The application scenarios of the embodiments of the present disclosure are explained below:

[0033] Figure 1This is an application scenario diagram of the singing video generation method provided by the embodiment of the present disclosure. The singing video generation method provided by the embodiment of the present disclosure can be applied to applications (APP, Application) with music generation function and video generation function, such as music applications, short video applications, etc. More specifically, it can be applied to the application scenario of generating corresponding singing videos for intelligently generated music and making intelligently generated music videos. The execution subject of this embodiment can be a terminal device running the above-mentioned application with music generation function or video generation function, or a server deploying the server corresponding to the above-mentioned application, or other electronic devices with similar functions. Among them, when the execution subject is a terminal device, the terminal device executes the method provided by this embodiment by running the above-mentioned application; when the execution subject is a server, the server of the above-mentioned application with music generation function or video generation function can be partially or completely run on the server, and the method provided by this embodiment is executed on the server side, and the terminal device runs the client of the application, and the communication between the server and the terminal device is based on the server-client, so that the terminal device can obtain the execution result of the method provided by this embodiment and display it as needed.

[0034] Among them, in some embodiments, the terminal device or server can implement the singing video generation method provided by the embodiment of the present disclosure by running various computer executable instructions or computer programs. For example, computer executable instructions can be program-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local application, that is, a program that needs to be installed in the operating system to run, or it can be a small program embedded in any APP, that is, a program that runs based on a browser environment. In summary, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form, and the specific implementation form can be configured as needed. Furthermore, in the process of implementing the singing video generation method provided by the embodiment of the present disclosure, the terminal device can execute the method by running a computer executable instruction or computer program set locally, or it can execute the method by calling a computer executable instruction or computer program set in an external server. In some embodiments, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud storage, cloud communications, cloud databases, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms. Among them, cloud services can be interactive processing services for terminal devices to call.

[0035] refer to Figure 1 As shown in , taking the terminal device as an example, a target application with a music generation function is running in the terminal device. After the target application is started, the user operates the terminal device to input keywords for generating music into the target application, such as: "sadness", "lyrical", "clear river", etc., as shown in the figure. Afterwards, the target application calls the music generation model deployed in the cloud based on the above keywords, generates a corresponding "original song", that is, intelligently generated music (shown as Music_AI_No.001 in the figure), and sends it back to the terminal device for playback and display. Afterwards, the user can further add a cover image to the intelligently generated music and publish it to the Internet platform (music platform) corresponding to the target application to realize the dissemination of the intelligently generated music.

[0036] In the prior art, in the above application scenarios, after the user generates intelligently generated music through intelligent generation technology, the intelligently generated music can be disseminated through platform publishing, sharing, etc., but since there is no matching singing video, the dissemination effect of the intelligently generated music is poor. In the related art, an additional third-party image generation software can be used to generate a singing video that matches the intelligently generated music based on the image intelligent generation technology, which can also be called a "lip-sync video". Afterwards, the user can publish or share the singing video on the original intelligently generated music generation platform, thereby improving the dissemination effect of the created intelligently generated music. However, since the generated intelligently generated music needs to be generated and transmitted across platforms, on the one hand, it will cause additional operations and waiting time for the user, affecting the efficiency of video production. On the other hand, in the process of intelligently generated music transmission, the audio data needs to be transmitted to the third-party software for processing, and the intelligently generated music needs to be compressed in the process, resulting in the loss of music details therein, so that the generated singing video has a low content matching degree with the intelligently generated music. That is, in the above process of generating corresponding videos for intelligently generated music, there are problems of low interaction efficiency and low matching degree between the generated video and the intelligently generated music.

[0037] The disclosed embodiment provides a singing video generation method to solve the above-mentioned problem.

[0038] refer to Figure 2 , Figure 2 A schematic diagram of the process of the singing video generation method provided in the embodiment of the present disclosure Figure 1 The method of this embodiment can be applied in a terminal device, and the method for generating a singing video includes:

[0039] Step S101: After the target music is generated through the target application, a publishing page corresponding to the target music is displayed in the target application, wherein the publishing page is configured with a video generation control for generating a performance video matching the target music.

[0040] Step S102: In response to a first trigger operation on a video generation control, a first video generation page is displayed in a target application, where the first video generation page is configured to display at least one video material.

[0041] Step S103: In response to the second trigger operation for the first video generation page, the target video material is determined, and a singing video is generated based on the target video material, wherein the singing video and the target video material contain the same target object, and the lip movements of the target object in the singing video match the music content of the target music.

[0042] For example, reference Figure 1 As shown in the schematic diagram of the application scenario, the terminal device generates target music based on intelligent generation technology by running a target application with a music generation function, that is, intelligently generated music. Among them, the process of generating such intelligently generated music based on intelligent generation technology usually includes receiving keywords input by users, and calling a pre-trained music generation module in combination with the keywords to generate music matching the keywords, which will not be introduced in detail this time. Afterwards, after the target music is generated, the target application jumps to the publishing page corresponding to the target music. In the publishing page, a video generation control for generating a singing video matching the target music is configured. The user can generate a singing video of the target music in the current target application by targeting the video generation control.

[0043] Specifically, after the terminal device receives the first trigger operation for the video generation control, it controls the target application to display a first video generation page within the target application. The first video generation page is configured to display at least one video material, which is a material file used to generate a singing video, such as a picture containing facial features.

[0044] Thereafter, the terminal device can determine a target video material in the first video generation page by responding to the user's second trigger operation on the first video generation page. In one possible implementation, the second trigger operation is a selection operation for the video material in the first video generation page, and the target video material is determined from the first video generation page by responding to the selection operation; in another possible implementation, the terminal device will automatically select a default video material as the target video material, or determine a recommended video material as the target video material through calculation, and the second trigger operation is a confirmation operation for the above-mentioned default video material or recommended video material, such as clicking the "Confirm" control in the first video generation page, so that the terminal device responds to the second trigger operation to determine the final target video material.

[0045] Further, after determining the target video material, the terminal device generates a singing video based on the target video material, completing the process of generating a singing video matching the target music. Among them, the target video material includes a target object, illustratively, such as an object with facial features such as a person or a small animal. The terminal device processes the target video material by calling a video generation model to generate a video that is equal to (or similar to) the duration of the target music, contains the same target object, and the target object has a lip movement that matches the lyrics and / or melody of the target music, that is, the singing video corresponding to the target music. Specifically, the singing video and the target video material contain the same target object, and the lip movement of the target object in the singing video matches the music content of the target music, that is, the video content of the singing video includes: the lip shape of the target object (pet dog) in the target video material (such as a photo of a pet dog) changes dynamically with the melody and content of the music to show that the target object (pet dog) is singing the current target song.

[0046] The video generation model can be deployed entirely on the terminal device, or entirely on the cloud, or partially on the local device and partially on the cloud. The specific implementation method can be set as needed. The video generation model is a pre-trained model that has the ability to output a video based on a piece of music (corresponding file) and a picture. The specific implementation method and training process will not be described here.

[0047] Furthermore, in a possible implementation, the first video generation page is configured with a first material area and a second material area, wherein the first material area is used to display a video generation template in the cloud or cached locally; and the second material area is used to display a local image stored in the terminal device;

[0048] Exemplarily, the video generation template displayed in the first material area is a file collection or information collection for generating a singing video, which contains at least one template image. In addition, it can also contain information such as video special effects parameters. The video generation template can be a template uploaded by other users, which is used to realize a singing video of a song in a specific style with a specific target object (such as a person or an animal), so as to achieve the purpose and effect of "remaking the same video". Its main body can be stored in the cloud and previewed locally on the terminal device. The local image displayed in the second material area, that is, the image stored in the terminal device currently running the target application, is displayed in the second material area after obtaining user authorization or.

[0049] Further, for the content displayed in the second material area in the first video generation page, in a possible implementation, as Figure 3 As shown, the specific implementation of step S102 includes:

[0050] Step S1021: After obtaining user authorization, obtain the local image stored in the terminal device.

[0051] Step S1022: Filtering local images to be selected that contain a target object from local images stored in the terminal device, wherein the target object is an object with facial features.

[0052] Step S1023: Displaying the local image to be selected in the second material area of ​​the first video generation page.

[0053] In this embodiment, before the terminal device displays the first video generation page in the target application, in order to solve the problem that there are many local images and not all of them are suitable for generating singing videos, first, after obtaining user authorization, the terminal device obtains the local images stored in the terminal device and screens the local images. Specifically, the local images are feature detected, and the images containing facial features are determined as the selected local images containing the target object. After that, the selected local images are displayed in the second material area of ​​the first video generation page, thereby completing the display process of the first video generation page. Since the generated singing video needs to display lip movements, the image generated for the singing video is required to clearly display the face and lip shape in a static state. In the steps of this embodiment, the selected local images are images of the target object with facial features that have been screened by the terminal device, which can avoid the subsequent inability to match the lip shape area, thereby causing the generated singing video to be distorted, thereby improving the authenticity of the generated singing video.

[0054] On the other hand, the content displayed in the first material area in the first video generation page, that is, the video generation template, can be personalized and pushed by the cloud and displayed in the first material area in the first video generation page, which will not be elaborated here.

[0055] Further, after displaying the first video generation page, the specific implementation of step S103 includes two methods, wherein one possible implementation method includes:

[0056] Step S103A: In response to the second trigger operation on the first material area, a target video generation template is determined, and a singing video is generated based on the target video generation template.

[0057] Another possible implementation of step S103 includes:

[0058] Step S103B: In response to the second trigger operation for the second material area, determine the target local image, and generate a singing video based on the target local image.

[0059] Figure 4 A schematic diagram of a first video generation page provided in an embodiment of the present disclosure is shown below in conjunction with Figure 4 The above process is described in detail. Figure 4 As shown, exemplarily, first, after jumping from the publishing page to the first video generation page, the upper half of the first video generation page is a first material area for displaying the video generation template in the cloud, and below each video generation template, a description of how to use the video generation template is displayed, such as the "shoot the same video" shown in the figure, to guide the use of the video generation template; the lower half of the first video generation page is a second material area for displaying the local image stored in the terminal device, and then, when the user clicks on the target video generation template in the first material area, such as the video generation template M1 shown in the figure, the terminal device calls the video generation model deployed in the cloud, processes the video generation template M1 and the target music, and thus generates a singing video based on the style of the video generation template M1; and when the user clicks on the target local image in the second material area, such as the local image P1 shown in the figure, the terminal device uploads the local image P1 to the cloud server, and then calls the video generation model to process the local image P1 and the target music, so as to generate a singing video based on the local image P1.

[0060] Of course, in another possible implementation method, the first video generation page may also only include the above-mentioned first material area or second material area, that is, only display the video generation template or local image. The specific implementation method and the corresponding mutual method are similar to the above-mentioned content and will not be repeated here.

[0061] Furthermore, in a possible implementation manner, after determining the target video material, the method further includes:

[0062] Step S103C: Obtain image content features of the target video material; if the image content features do not meet preset feature requirements, display prompt information.

[0063] Exemplarily, after determining the target video material, such as determining the target local image in the above embodiment, in order to ensure that the target local image meets the requirements for generating a singing video (i.e., the target object is contained in the picture), in this embodiment, the terminal device determines whether the target local image selected by the user can be used as the source data for generating a singing video by detecting the image content characteristics of the target local image. If, according to the image content characteristics of the target local image, the target local image contains preset features, such as facial contour features and mouth features, then the subsequent steps are continued; if the preset feature requirements are not met, a prompt message is displayed to prompt the user to make a replacement, thereby avoiding the problem of failure to generate a singing video due to user misoperation.

[0064] Furthermore, in a possible implementation, Figure 5 As shown, after determining the target video material, the specific implementation method of generating a singing video based on the target video material includes:

[0065] Step S1031: Acquire music type information corresponding to the target music, where the music type information is generated by the target application in the process of generating the target music.

[0066] Step S1032: Generate a second prompt word according to the music type information, where the second prompt word represents the action features of the lip movement.

[0067] Step S1033: Input the target music, the target local image and the second prompt word into the video generation model to generate a singing video, wherein the lip movements of the target object in the singing video have the movement features represented by the second prompt word.

[0068] Exemplarily, for the case where the target video material is a local image, after determining the target local image, the terminal device first obtains the music type information corresponding to the target music generated in the previous step. The music type information is information for classifying the type and style of music. More specifically, the music type information corresponding to the target music includes, for example, pop, rock, opera, etc. (divided according to the song category); or includes lyrical, dynamic, sad, etc. (divided according to the emotional style). In the specific implementation of the music type information, it can be a specific classification identifier, without specific restrictions. After that, the terminal device converts the music category information into the corresponding second prompt word according to the preset configuration information. The second prompt word represents the action characteristics of the lip movement, such as the frequency and amplitude of the lip change, so as to realize the mapping of the music category information to the lip movement. Finally, the second prompt word representing the action characteristics of the lip movement is connected to the target music and the target local image, and is input into the video generation model together to generate a singing video. The control of the singing video generation process is realized by the second prompt word, so that the lip movement of the target object in the generated singing video has the action characteristics represented by the second prompt word, thereby improving the matching degree and authenticity between the singing video and the target music.

[0069] At the same time, it should be noted that the solution provided in this embodiment is completed within the same software or platform (target application program). Compared with other solutions in the prior art, the process of transmitting relevant information of the target music across platforms is avoided. Therefore, in this embodiment, since the music type information is generated by the target application program in the process of generating the target music, the terminal device can efficiently and accurately implement step S1031 to obtain the music type information. Compared with the music type information obtained by parsing the target music, feature recognition and other processing methods in the conventional solution, it has higher accuracy and execution efficiency.

[0070] In an embodiment of the present disclosure, after generating target music through a target application, a publishing page corresponding to the target music is displayed in the target application, wherein the publishing page is configured with a video generation control for generating a singing video matching the target music; in response to a first trigger operation on the video generation control, a first video generation page is displayed in the target application, and the first video generation page is configured to display at least one video material; in response to a second trigger operation on the first video generation page, the target video material is determined, and a singing video is generated based on the target video material, wherein the singing video and the target video material contain the same target object, and the lip movements of the target object in the singing video match the music content of the target music. By configuring a video generation control in the publishing page after the intelligently generated music is generated, the client can directly jump to the first video generation page for generating a singing video matching the intelligently generated music after the intelligently generated music is generated, and the corresponding video material is displayed. Then, based on the selected target video material, a target music containing a target object that is consistent with the target video material and can perform lip movements matching the music content is generated, that is, a "lip-sync" video matching the target music. This process does not require the use of other third-party software, thereby improving the generation efficiency of the singing video. Moreover, since it does not involve cross-platform image transmission and avoids audio and image compression processes, the generated singing video can have a higher matching degree with the intelligently generated music and a higher video clarity, resulting in a better video effect.

[0071] refer to Figure 6 , Figure 6 A schematic diagram of the process of the singing video generation method provided in the embodiment of the present disclosure Figure 2 In this embodiment Figure 2 On the basis of the embodiment shown, the scheme is further refined, and a potential step of generating target music is optionally added. The method for generating a singing video includes:

[0072] Step S201: obtaining a first prompt word input by a user, where the first prompt word is used to characterize a music feature of a target music.

[0073] Step S202: The first prompt word is processed by the target application to generate target music.

[0074] For example, in this embodiment, a specific implementation step of generating target music is added, that is, by obtaining the first prompt word input by the user, and then using the target application to call the music generation model to generate the target music. In this step, the first prompt word input by the user will be cached locally by the terminal device, and in the subsequent steps, the first prompt word will be used to select the default material, thereby improving the operation efficiency in the process of generating the singing video.

[0075] Step S203: After the target music is generated through the target application, a publishing page corresponding to the target music is displayed in the target application, wherein the publishing page is configured with a video generation control for generating a performance video matching the target music.

[0076] Step S204: In response to a first trigger operation on the video generation control, a first video generation page is displayed in the target application, and the first video generation page is configured to display at least one video material.

[0077] Step S205: In the first video generation page, based on the first prompt word, select the corresponding default video material.

[0078] Exemplarily, in this embodiment, in combination with the solution of steps S201-S202, after the first video generation page is displayed in the target application, a possible implementation method is to select one of them as the target video material by responding to the trigger operation input by the user; and in the steps of this embodiment, the terminal device automatically determines a video material, that is, the default video material, in combination with the first prompt word obtained in the process of generating the target music. Among them, the first prompt word represents the music characteristics of the target music. Therefore, the video material generated based on the first prompt word is also a video material that matches the music characteristics of the target music. Using the default video material as the target video material and executing the subsequent steps of generating the singing video can make the generated singing video more matched with the music characteristics of the target music (such as music melody, music lyrics), and provide the authenticity of the singing video.

[0079] Step S206: In response to the second trigger operation on the first video generation page, the default video material is determined as the target video material.

[0080] Step S207: Generate a singing video based on the target video material and the target music, wherein the singing video and the target video material contain the same target object, and the lip movements of the target object in the singing video match the music content of the target music.

[0081] Exemplarily, in combination with the previous steps, after the terminal device selects the corresponding default video material, the second trigger operation is, for example, a confirmation operation, that is, an operation of confirming the default video material as the target video material. After that, the terminal device generates the corresponding singing music based on the obtained target video material by calling the video generation model.

[0082] Furthermore, in a possible implementation, Figure 7 As shown, the specific implementation of step S207 includes:

[0083] Step S2071: Acquire target music, and divide the target music into vocal segments and non-vocal segments;

[0084] Step S2072: using the video generation model to process the target video material and the vocal segment to generate a first video segment corresponding to the vocal segment, wherein the lip movements of the target object in the first video segment match the music content of the target music;

[0085] Step S2073: generating a second video segment corresponding to the non-human voice segment based on the target video material, wherein the second video segment is a static video whose video screen does not change;

[0086] Step S2074: Generate a singing video based on combining the playback timestamps of the first video clip and the second video clip.

[0087] After the terminal device obtains the previously generated target music through the target application, it divides the target music into vocal segments and non-vocal segments, wherein the vocal segments are segments corresponding to the vocal part of the target music, in other words, segments in which the target object in the corresponding singing video needs to perform lip movements. On the contrary, the non-vocal segments are segments corresponding to the non-vocal part of the target music, in other words, segments in which the target object in the corresponding singing video does not need to perform lip movements. Vocal segments and non-vocal segments can be represented by paired timestamps. Further, for the specific implementation method of dividing vocal segments and non-vocal segments, one possible implementation method is to divide vocal segments and non-vocal segments by parsing the audio features of the target music and performing vocal recognition; another possible implementation method is to divide vocal segments and non-vocal segments by reading the music information corresponding to the target music (such as lyrics text and corresponding timestamps), which can be set according to needs. After that, the target video material and the vocal clip are processed using the video generation model to generate the first video clip corresponding to the vocal clip, i.e., the "lip sync video" for the vocal clip; and for the non-vocal clip, the target video material is directly used to generate a static video, i.e., the second video clip. Finally, the first video clip and the second video clip are combined based on the playback timestamps to generate the singing video.

[0088] Figure 8 A schematic diagram of a process for generating a singing video provided by an embodiment of the present disclosure is shown below. Figure 8 For a detailed description of the above steps, refer to Figure 8As shown, the target music is a 10s (second) audio; by parsing the target music, it is divided into a vocal segment (shown as segment A in the figure) and a non-vocal segment (shown as segment B in the figure), for example, the vocal segment is the first 4 seconds of the target music, and the non-vocal segment is the last 6 seconds of the target music. Afterwards, the terminal device uploads the vocal segment and the target video material of the target music to the video generation model in the cloud, and generates the corresponding first video segment, i.e., video segment Video_1, through the video generation model; on the other hand, the terminal device generates the second video segment whose video screen does not change by using the generated target video material locally, i.e., video segment Video_2; or, sends the target video material to the cloud server to generate the second video segment whose video screen does not change (this situation is not shown in the figure). Finally, the terminal device combines the video segment Video_1 and the video segment Video_2 locally to generate a singing video. Alternatively, the terminal device uploads the local video segment Video_1 to the cloud server, and the cloud server combines the video segment Video_1 and the video segment Video_2 to generate a singing video, and sends it back to the terminal device (this situation is not shown in the figure).

[0089] In the steps of this embodiment, the target music is divided into vocal segments and non-vocal segments, and the corresponding video segments are generated by processing them separately. Then, the video segments are combined to generate a singing video. This saves the model resource overhead of generating static images corresponding to the non-vocal segments in the singing video, reduces the network resource overhead of the server and the model pressure of the video generation model, and improves the generation efficiency of the singing video.

[0090] Optionally, in this embodiment, during the process of executing step S207, this embodiment further includes:

[0091] Step S208: Display a second video generation page in the target application, where the second video generation page is configured to display the current generation progress of the singing video.

[0092] Optionally, after step S207 is completed, the present embodiment further includes:

[0093] Step S209: After the performance video is generated, a preview video of the performance video is displayed in the second video generation page.

[0094] Exemplarily, since the singing video needs to call the video generation model for generation, considering the load of the model resources, in high-concurrency tasks, the video generation model usually needs to queue up to process the requests for generating singing videos sent by each terminal device. Therefore, users usually need to wait for the generation of the singing video to be completed. In this embodiment, during the generation of the singing video, a second video generation page is displayed in the target application, and the current generation progress of the singing video is displayed through the second video generation page. After the generation of the singing video is completed, a preview video of the singing video is displayed in the second video generation page, so that the user can know the remaining time, and thus perform other operations in the remaining time, thereby improving the operating efficiency of the target application.

[0095] Optionally, this embodiment also includes:

[0096] Step S209A: In response to the third trigger operation on the second video generation page, switching the second video generation page to background operation;

[0097] Step S209B: In response to the fourth trigger operation on the second video generation page, cancel the generation of the singing video and return to the first video generation page or the publishing page.

[0098] Exemplarily, in combination with the steps of the previous embodiments, during the generation process of the singing video, the terminal device can further respond to the third trigger operation for the second video generation page, switch the second video generation page to background operation, thereby executing other functions within the target application and improving the efficiency of operating the target application; or, in response to the fourth trigger operation for the second video generation page, cancel the generation of the singing video, and return to the first video generation page or the publishing page, thereby interrupting the generation process of the singing video, and the subsequent steps of modifying and regenerating the singing video, which can also improve the efficiency of operating the target application and generating the final singing video.

[0099] Optionally, this embodiment also includes:

[0100] Step S210: In response to the modification instruction for the target music, generate the changed target music, and return to execute step S207.

[0101] Exemplarily, after generating the target music, the terminal device can modify the target music again through the target application by responding to the modification instruction input by the user, for example, modifying the prompt word when generating the target music (such as the first prompt word in the previous embodiment step), or modifying the lyrics of the target music, etc., to fully or partially regenerate the target music, that is, generate the changed target music. Afterwards, by utilizing the characteristics of generating the target music and generating the singing video in the same target application, the terminal device can regenerate the singing video based on the changed target music and target video material (i.e., return to execute step S207), so as to quickly regenerate the singing video without repeating the above steps such as selecting the target video material, thereby achieving the effect of synchronously modifying the corresponding singing video after modifying the intelligently generated music, and improving the operational efficiency of adjusting the singing video.

[0102] In this embodiment, the implementation method of step S203 is the same as that of the present disclosure. Figure 2 The implementation method of step S101 in the illustrated embodiment is the same and will not be described in detail here.

[0103] Corresponding to the singing video generation method of the above embodiment, Fig. 9 This is a structural block diagram of a singing video generation device provided in an embodiment of the present disclosure. The method described in the above embodiment can be executed by the singing video generation device, which can be implemented by software and / or hardware, and can be integrated in an electronic device with certain data processing functions. The electronic device can include but is not limited to mobile terminals with big data processing capabilities, as well as fixed terminals with big data processing capabilities such as desktop computers and supercomputers.

[0104] For the sake of convenience, only the parts related to the embodiments of the present disclosure are shown. Fig. 9 , the singing video generating device 3 comprises:

[0105] A first display module 31 is used to display a publishing page after generating the target music, wherein the publishing page is configured with a video generation control for generating a performance video matching the target music;

[0106] A second display module 32, configured to display a first video generation page in the target application in response to a first trigger operation on the video generation control, wherein the first video generation page is configured to display at least one video material;

[0107] The generation module 33 is used to determine the target video material in response to the second trigger operation for the first video generation page, and generate a singing video based on the target video material, wherein the singing video and the target video material contain the same target object, and the lip movements of the target object in the singing video match the music content of the target music.

[0108] According to one or more embodiments of the present disclosure, a first material area and a second material area are configured in the first video generation page, wherein the first material area is used to display a video generation template; the second material area is used to display a local image stored in the terminal device; the generation module 33 is specifically used to: determine a target video generation template in response to a second trigger operation on the first material area, and generate a singing video based on the target video generation template; or, determine a target local image in response to a second trigger operation on the second material area, and generate a singing video based on the target local image.

[0109] According to one or more embodiments of the present disclosure, when the second display module 32 displays the first video generation page in the target application, it is specifically used to: obtain local images stored in the terminal device after obtaining user authorization; filter out candidate local images containing a target object from the local images stored in the terminal device, wherein the target object is an object with facial features; and display the candidate local images in the second material area of ​​the first video generation page.

[0110] According to one or more embodiments of the present disclosure, when generating a singing video based on a target local image, the generation module 33 is specifically used to: obtain music type information corresponding to the target music; generate a second prompt word based on the music type information, the second prompt word representing the action characteristics of the lip movement; input the target music, the target local image and the second prompt word into a video generation model to generate a singing video, wherein the lip movement of the target object in the singing video has the action characteristics represented by the second prompt word.

[0111] According to one or more embodiments of the present disclosure, before generating the target music, the generation module 33 is further used to: obtain a first prompt word input by the user, the first prompt word being used to characterize the music characteristics of the target music; and generate the target music by processing the first prompt word through a target application; after the second display module 32 displays the first video generation page in the target application in response to a first trigger operation on the video generation control, it is further used to: select the corresponding default video material in the first video generation page based on the first prompt word.

[0112] According to one or more embodiments of the present disclosure, after generating a singing video based on the target video material, the generation module 33 is further used to: generate changed target music in response to a modification instruction for the target music; and regenerate the singing video based on the changed target music and the target video material.

[0113] According to one or more embodiments of the present disclosure, after determining the target video material, the generation module 33 is also used to: display a second video generation page within the target application, where the second video generation page is configured to display the current generation progress of the singing video; after the singing video generation is completed, display a preview video of the singing video within the second video generation page.

[0114] According to one or more embodiments of the present disclosure, the generation module 33 is also used for at least one of the following: in response to a third trigger operation on the second video generation page, switching the second video generation page to background operation; in response to a fourth trigger operation on the second video generation page, canceling the generation of the singing video, and returning to the first video generation page or the publishing page.

[0115] According to one or more embodiments of the present disclosure, after determining the target video material, the generation module 33 is further used to: obtain image content features of the target video material; if the image content features do not meet preset feature requirements, display prompt information.

[0116] According to one or more embodiments of the present disclosure, when generating a singing video based on a target video material, the generation module 33 is specifically used to: obtain the target music, and divide the target music into vocal segments and non-vocal segments; use a video generation model to process the target video material and the vocal segments, and generate a first video segment corresponding to the vocal segment, and the lip movements of the target object in the first video segment match the music content of the target music; based on the target video material, generate a second video segment corresponding to the non-vocal segment, and the second video segment is a static video whose video screen does not change; and generate a singing video based on the combination of the first video segment and the second video segment based on the playback timestamps.

[0117] The first display module 31, the second display module 32 and the generating module 33 are connected in sequence. The singing video generating device 3 provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0118] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown in FIG. Fig.10 As shown, the electronic device 4 includes:

[0119] A processor 41, and a memory 42 communicatively connected to the processor 41;

[0120] The memory 42 stores computer executable instructions;

[0121] The processor 41 executes the computer execution instructions stored in the memory 42 to implement the following Figure 2-Figure 8 The singing video generation method in the illustrated embodiment.

[0122] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0123] For related instructions, please refer to Figure 2-Figure 8 The relevant descriptions and effects corresponding to the steps in the corresponding embodiments can be understood, and no further elaboration is made here.

[0124] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which are used to implement the present invention when executed by a processor. Figure 2-Figure 8 A singing video generation method provided in any one of the corresponding embodiments.

[0125] The present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the present invention is realized. Figure 2-Figure 8 A singing video generation method provided in any one of the corresponding embodiments.

[0126] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0127] refer to Fig.11 , which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.11 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0128] like Fig.11As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0129] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.11 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0130] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0131] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0132] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0133] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0134] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit or module does not, in some cases, constitute a limitation on the unit itself.

[0137] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0138] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0139] In a first aspect, according to one or more embodiments of the present disclosure, a method for generating a singing video is provided, comprising:

[0140] After generating target music through a target application, a publishing page corresponding to the target music is displayed in the target application, wherein the target music is intelligently generated music, and the publishing page is configured with a video generation control for generating a singing video matching the target music; in response to a first trigger operation on the video generation control, a first video generation page is displayed in the target application, and the first video generation page is configured to display at least one video material; in response to a second trigger operation on the first video generation page, the target video material is determined, and the singing video is generated based on the target video material, wherein the singing video and the target video material contain the same target object, and the lip movements of the target object in the singing video match the music content of the target music.

[0141] According to one or more embodiments of the present disclosure, a first material area and a second material area are configured in the first video generation page, wherein the first material area is used to display a video generation template in the cloud; the second material area is used to display a local image stored in a terminal device; in response to a second trigger operation on the first video generation page, the target video material is determined, and the singing video is generated based on the target video material, including: in response to the second trigger operation on the first material area, the target video generation template is determined, and the singing video is generated based on the target video generation template; or, in response to the second trigger operation on the second material area, the target local image is determined, and the singing video is generated based on the target local image.

[0142] According to one or more embodiments of the present disclosure, displaying the first video generation page within the target application includes: obtaining local images stored in the terminal device after obtaining user authorization; screening out candidate local images containing a target object from the local images stored in the terminal device, wherein the target object is an object having facial features; and displaying the candidate local images in a second material area of ​​the first video generation page.

[0143] According to one or more embodiments of the present disclosure, generating the singing video based on the target local image includes: acquiring music type information corresponding to the target music; generating a second prompt word based on the music type information, wherein the second prompt word represents action features of lip movements; inputting the target music, the target local image and the second prompt word into a video generation model to generate the singing video, wherein the lip movements of the target object in the singing video have the action features represented by the second prompt word.

[0144] According to one or more embodiments of the present disclosure, before generating the target music, it also includes: obtaining a first prompt word input by a user, wherein the first prompt word is used to characterize the music characteristics of the target music; processing the first prompt word through the target application to generate the target music; after responding to a first trigger operation for the video generation control, displaying a first video generation page in the target application, it also includes: in the first video generation page, based on the first prompt word, selecting a corresponding default video material.

[0145] According to one or more embodiments of the present disclosure, after generating the singing video based on the target video material, the method further includes: generating changed target music in response to a modification instruction for the target music; and regenerating the singing video based on the changed target music and the target video material.

[0146] According to one or more embodiments of the present disclosure, after the target video material is determined, it also includes: displaying a second video generation page within the target application, wherein the second video generation page is configured to display the current generation progress of the singing video; after the singing video is generated, displaying a preview video of the singing video within the second video generation page.

[0147] According to one or more embodiments of the present disclosure, the method also includes at least one of the following: in response to a third trigger operation on the second video generation page, switching the second video generation page to background operation; in response to a fourth trigger operation on the second video generation page, canceling the generation of the singing video, and returning to the first video generation page or the publishing page.

[0148] According to one or more embodiments of the present disclosure, after determining the target video material, the method further includes: acquiring image content features of the target video material; and displaying prompt information if the image content features do not meet preset feature requirements.

[0149] According to one or more embodiments of the present disclosure, generating the singing video based on the target video material includes: obtaining the target music, and dividing the target music into vocal segments and non-vocal segments; using a video generation model to process the target video material and the vocal segments, and generating a first video segment corresponding to the vocal segment, wherein the lip movements of the target object in the first video segment match the music content of the target music; based on the target video material, generating a second video segment corresponding to the non-vocal segment, wherein the second video segment is a static video whose video screen does not change; and generating the singing video based on combining the first video segment and the second video segment based on their playback timestamps.

[0150] In a second aspect, according to one or more embodiments of the present disclosure, a singing video generating device is provided, comprising:

[0151] A first display module is used to display a publishing page after generating the target music, wherein the publishing page is configured with a video generation control for generating a performance video matching the target music;

[0152] A second display module, configured to display a first video generation page in the target application in response to a first trigger operation on the video generation control, wherein the first video generation page is configured to display at least one video material;

[0153] A generation module is used to determine the target video material in response to a second trigger operation on the first video generation page, and generate the singing video based on the target video material, wherein the singing video and the target video material contain the same target object, and the lip movements of the target object in the singing video match the music content of the target music.

[0154] According to one or more embodiments of the present disclosure, the first video generation page is configured with a first material area and a second material area, wherein the first material area is used to display a video generation template; the second material area is used to display a local image stored in a terminal device; the generation module is specifically used to: determine a target video generation template in response to a second trigger operation on the first material area, and generate the singing video based on the target video generation template; or, determine a target local image in response to a second trigger operation on the second material area, and generate the singing video based on the target local image.

[0155] According to one or more embodiments of the present disclosure, when the second display module displays the first video generation page in the target application, it is specifically used to: obtain the local image stored in the terminal device after obtaining user authorization; filter out the selected local image containing the target object from the local images stored in the terminal device, wherein the target object is an object with facial features; and display the selected local image in the second material area of ​​the first video generation page.

[0156] According to one or more embodiments of the present disclosure, when the generation module generates the singing video based on the target local image, it is specifically used to: obtain the music type information corresponding to the target music; generate a second prompt word based on the music type information, the second prompt word representing the action characteristics of the lip movement; input the target music, the target local image and the second prompt word into the video generation model to generate the singing video, wherein the lip movement of the target object in the singing video has the action characteristics represented by the second prompt word.

[0157] According to one or more embodiments of the present disclosure, before generating the target music, the generation module is further used to: obtain a first prompt word input by a user, wherein the first prompt word is used to characterize a musical feature of the target music; and generate the target music by processing the first prompt word through the target application. After the second display module displays a first video generation page in the target application in response to a first trigger operation for the video generation control, the second display module is further used to: select a corresponding default video material in the first video generation page based on the first prompt word.

[0158] According to one or more embodiments of the present disclosure, after generating the singing video based on the target video material, the generation module is further used to: generate changed target music in response to a modification instruction for the target music; and regenerate the singing video based on the changed target music and the target video material.

[0159] According to one or more embodiments of the present disclosure, after determining the target video material, the generation module is also used to: display a second video generation page within the target application, wherein the second video generation page is configured to display the current generation progress of the singing video; after the generation of the singing video is completed, display a preview video of the singing video within the second video generation page.

[0160] According to one or more embodiments of the present disclosure, the generation module is also used for at least one of the following: in response to a third trigger operation on the second video generation page, switching the second video generation page to background operation; in response to a fourth trigger operation on the second video generation page, canceling the generation of the singing video and returning to the first video generation page or the publishing page.

[0161] According to one or more embodiments of the present disclosure, after determining the target video material, the generation module is further used to: obtain image content features of the target video material; if the image content features do not meet preset feature requirements, display prompt information.

[0162] According to one or more embodiments of the present disclosure, when the generation module generates the singing video based on the target video material, it is specifically used to: obtain the target music, and divide the target music into vocal segments and non-vocal segments; use the video generation model to process the target video material and the vocal segments to generate a first video segment corresponding to the vocal segment, and the lip movements of the target object in the first video segment match the music content of the target music; based on the target video material, generate a second video segment corresponding to the non-vocal segment, and the second video segment is a static video whose video screen does not change; generate the singing video based on the combination of the first video segment and the second video segment based on the playback timestamps.

[0163] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0164] The memory stores computer-executable instructions;

[0165] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the singing video generation method described in the first aspect and various possible designs of the first aspect.

[0166] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the singing video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0167] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the singing video generation method as described in the first aspect and various possible designs of the first aspect.

[0168] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0169] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0170] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for generating a singing video, characterized in that: include: After the target music is generated by the target application, a publishing page corresponding to the target music is displayed in the target application, wherein the publishing page is configured with a video generation control for generating a performance video matching the target music; In response to a first trigger operation on the video generation control, displaying a first video generation page in the target application, wherein the first video generation page is configured to display at least one video material; In response to a second trigger operation for generating a page for the first video, a target video material is determined, and the singing video is generated based on the target video material, wherein the singing video and the target video material contain the same target object, and the lip movements of the target object in the singing video match the music content of the target music.

2. The method according to claim 1, characterized in that The first video generation page is configured with a first material area and a second material area, wherein the first material area is used to display the video generation template; the second material area is used to display the local image stored in the terminal device; In response to a second trigger operation on the first video generation page, a target video material is determined, and the singing video is generated based on the target video material, including: In response to a second trigger operation on the first material area, a target video generation template is determined, and the singing video is generated based on the target video generation template; or, in response to a second trigger operation on the second material area, a target local image is determined, and the singing video is generated based on the target local image.

3. The method according to claim 2, characterized in that The step of displaying the first video generation page in the target application comprises: After obtaining authorization from the user, obtaining the local image stored in the terminal device; Filtering a candidate local image containing a target object from local images stored in the terminal device, wherein the target object is an object having facial features; The local image to be selected is displayed in the second material area of ​​the first video generation page.

4. The method according to claim 2, characterized in that: The step of generating the singing video based on the target local image comprises: Acquire music type information corresponding to the target music; generating a second prompt word according to the music type information, wherein the second prompt word represents the action characteristics of the lip movement; The target music, the target local image and the second prompt word are input into a video generation model to generate the singing video, wherein the lip movements of the target object in the singing video have the movement features represented by the second prompt word.

5. The method according to claim 1, characterized in that The target music is intelligently generated music. Before generating the target music, the method further includes: Acquire a first prompt word input by a user, where the first prompt word is used to characterize a music feature of the target music; Processing the first prompt word through the target application to generate the target music; After displaying a first video generation page in the target application in response to a first trigger operation on the video generation control, the method further includes: In the first video generation page, the corresponding default video material is selected based on the first prompt word.

6. The method according to claim 5, characterized in that After generating the singing video based on the target video material, the method further includes: In response to the modification instruction for the target music, generating the modified target music; Based on the changed target music and the target video material, the performance video is regenerated.

7. The method according to claim 1, characterized in that After the target video material is determined, the method further includes: Displaying a second video generation page in the target application, wherein the second video generation page is configured to display the current generation progress of the singing video; After the generation of the singing video is completed, a preview video of the singing video is displayed in the second video generation page.

8. The method according to claim 7, characterized in that The method further comprises at least one of the following: In response to a third trigger operation on the second video generation page, switching the second video generation page to background operation; In response to the fourth trigger operation on the second video generation page, the generation of the singing video is canceled and the page returns to the first video generation page or the publishing page.

9. The method according to claim 1, characterized in that: After the target video material is determined, the method further includes: Acquire image content features of the target video material; If the image content feature does not meet the preset feature requirements, a prompt message is displayed.

10. The method according to claim 1, characterized in that The step of generating the singing video based on the target video material comprises: Acquire the target music, and divide the target music into vocal segments and non-vocal segments; Processing the target video material and the vocal segment using a video generation model to generate a first video segment corresponding to the vocal segment, wherein the lip movements of the target object in the first video segment match the music content of the target music; Based on the target video material, generating a second video segment corresponding to the non-human voice segment, wherein the second video segment is a static video whose video screen does not change; The singing video is generated by combining the playing timestamps of the first video clip and the second video clip.

11. A singing video generating device, characterized in that: include: A first display module is used to display a publishing page after generating the target music, wherein the publishing page is configured with a video generation control for generating a performance video matching the target music; A second display module, configured to display a first video generation page in the target application in response to a first trigger operation on the video generation control, wherein the first video generation page is configured to display at least one video material; A generation module is used to determine the target video material in response to a second trigger operation on the first video generation page, and generate the singing video based on the target video material, wherein the singing video and the target video material contain the same target object, and the lip movements of the target object in the singing video match the music content of the target music.

12. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the singing video generation method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the singing video generation method as described in any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating a singing video as described in any one of claims 1 to 10 is implemented.

Images