Video generation method and device based on digital human, storage medium and program product

Abstract

The embodiment of the invention provides a video generation method and device based on a digital human, a storage medium and a program product. In the embodiment of the invention, application scene information and role description information provided by a target merchant are acquired, a plurality of global port broadcast manuscripts corresponding to the application scene information are generated by using an AI language model, and at least two video description files are generated based on the plurality of global port broadcast manuscripts and the role description information; and automatically generating at least two template videos in batches based on the at least two video description files, returning the at least two template videos to the terminal equipment, and generating a target video based on the digital human by the target merchant based on the at least two template videos. Wherein the template videos generated in batches provide rich editing templates for users, merchants do not need to operate complex video editing software, and the complexity of video generation is reduced.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a method, device, storage medium and program product for generating a video based on a digital human. Background Art

[0002] With the continuous development of mobile Internet, online live broadcast technology has also been rapidly improved. Among the many live broadcast modes currently available, the live broadcast mode has always been the mainstream live broadcast mode. However, with the development of artificial intelligence technology, virtual anchors have become an important part of Internet activities. Virtual anchors are composed of digital people / virtual people (hereinafter collectively referred to as digital people). Specifically, when the set live broadcast start time arrives, the pre-generated multimedia video is inferred to the terminal device. In the multimedia video, the digital person can simulate the live broadcast based on the preset image, sound, and scene in the live broadcast room to introduce the product. When the set live broadcast end time arrives, the streaming is stopped and the live broadcast ends.

[0003] In the prior art, multimedia videos can be generated by powerful video editing software, but such software has a relatively high threshold and is relatively complicated to operate, resulting in low efficiency in generating multimedia videos, especially when multiple multimedia videos need to be generated. Summary of the invention

[0004] Multiple aspects of the present application provide a method, device, storage medium and program product for generating a video based on a digital human, so as to improve the efficiency of generating a video.

[0005] The embodiment of the present application provides a video generation method based on digital humans, which is applicable to a server-side device, comprising: receiving a batch video generation request sent by a terminal device, the batch video generation request including digital human attribute information of a target merchant, the digital human attribute information including at least role description information and application scenario information; inputting the application scenario information into a first prompt word template corresponding thereto to obtain a first prompt word, the first prompt word template also including first task description information for describing a task to be performed by an artificial intelligence (AI) language model; inputting the first prompt word into the AI ​​language model, and generating a plurality of global oral scripts corresponding to the application scenario information according to the first task description information; and A plurality of global spoken scripts and role description information are used to generate at least two video description files of the target merchant, wherein the video description files include the target role information and the target global spoken script; for any video description file, a video screen of the digital human broadcast script is generated according to the target role information and the target global spoken script, and the target background information is selected from the material library according to the target global spoken script; the video screen and the target background information are synthesized to generate a template video corresponding to any video description file; at least two template videos corresponding to at least two video description files are sent to a terminal device, so that the terminal device can batch generate target videos based on digital humans according to the at least two template videos.

[0006] The embodiment of the present application also provides a method for generating videos based on digital humans, which is applicable to terminal devices. The method includes: when a target merchant opens a batch video generation service, displaying a first page, which includes a batch video generation control; in response to an editing operation on the first page, obtaining the digital human attribute information of the target merchant; in response to a triggering operation on the batch video generation control, generating a batch video generation request, which includes the digital human attribute information of the target merchant, and the digital human attribute information includes role description information and application scenario information; providing the batch video generation request to a server device, so that the server device generates and returns at least two template videos based on the above method; receiving at least two template videos, and batch generating target videos based on digital humans according to the at least two template videos.

[0007] An embodiment of the present application also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement each step in the digital human-based video generation method provided in the embodiment of the present application.

[0008] The embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements each step of the digital human-based video generation method provided in the embodiment of the present application.

[0009] The embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is caused to implement each step of the digital human-based video generation method provided in the embodiment of the present application.

[0010] In the embodiment of the present application, the application scenario information and role description information provided by the target merchant are obtained, and multiple global oral broadcast scripts corresponding to the application scenario information are generated using the AI ​​language model. At least two video description files are generated based on the multiple global oral broadcast scripts and role description information; at least two template videos are automatically generated in batches based on the at least two video description files, and the at least two template videos are returned to the terminal device, and the target merchant generates a target video based on the digital human based on the at least two template videos. Among them, the template videos generated in batches provide users with rich editing templates, without the need for merchants to operate complex video editing software, reducing the complexity of generating videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1a A flowchart of a method for generating a video based on a digital human provided by an exemplary embodiment of the present application;

[0013] Figure 1b A schematic diagram of displaying a template video provided by an exemplary embodiment of the present application;

[0014] Figure 2 A flowchart of another method for generating a video based on a digital human provided by an exemplary embodiment of the present application;

[0015] Figure 3a A schematic diagram of a first page provided for an exemplary embodiment of the present application;

[0016] Figure 3b A schematic diagram of a video editing page provided for an exemplary embodiment of the present application;

[0017] Figure 3c A schematic diagram of a drag effect corresponding to a drag operation provided by an exemplary embodiment of the present application;

[0018] Figure 3d A schematic diagram of another video editing page provided for an exemplary embodiment of the present application;

[0019] Figure 3e A schematic diagram of a video editing process provided for an exemplary embodiment of the present application;

[0020] Figure 4 A schematic structural diagram of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0022] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0023] The various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0024] In view of the above technical problems, in an embodiment of the present application, the application scenario information and role description information provided by the target merchant are obtained, and a plurality of global oral broadcast scripts corresponding to the application scenario information are generated using an AI language model, and at least two video description files are generated based on the plurality of global oral broadcast scripts and role description information; at least two template videos are automatically generated in batches based on the at least two video description files, and the at least two template videos are returned to the terminal device, and the target merchant generates a target video based on a digital human based on the at least two template videos. Among them, the template videos generated in batches provide users with a wealth of editing templates, and merchants do not need to operate complex video editing software, thereby reducing the complexity of generating videos. At the same time, merchants can choose template videos according to their own wishes, and edit them based on the template videos, so as to further improve the efficiency of generating videos, and can efficiently generate videos that meet their wishes.

[0025] A solution provided by an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0026] Figure 1a A flowchart of a method for generating a video based on a digital human provided by an exemplary embodiment of the present application, the GIA method is applicable to a server-side device, such as Figure 1a As shown, the method includes:

[0027] 101. Receive a batch video generation request sent by a terminal device, where the batch video generation request includes digital human attribute information of a target merchant, where the digital human attribute information includes at least role description information and application scenario information;

[0028] 102. Input the application scenario information into the corresponding first prompt word template to obtain a first prompt word, wherein the first prompt word template also includes first task description information, which is used to describe the task that the artificial intelligence AI language model needs to perform;

[0029] 103. Input the first prompt word into the AI ​​language model, and generate a plurality of global oral broadcast scripts corresponding to the application scenario information according to the first task description information;

[0030] 104. Generate at least two video description files of the target merchant according to the multiple global voice-over scripts and role description information, wherein the video description files include target role information and the target global voice-over script;

[0031] 105. For any video description file, generate a video screen of the digital human broadcast script according to the target role information and the target global broadcast script, and select target background information from the material library according to the target global broadcast script; synthesize the video screen and the target background information to generate a template video corresponding to any video description file;

[0032] 106. Send at least two template videos corresponding to the at least two video description files to a terminal device, so that the terminal device can batch generate target videos based on digital humans according to the at least two template videos.

[0033] In this embodiment, the terminal device may be a smart handheld device, such as a smart phone, tablet computer, laptop computer or desktop computer, etc.; for another example, the terminal device may also be a smart wearable device, such as a smart watch, smart bracelet, etc.; for another example, the terminal device may also be various smart home appliances with display screens, such as smart TVs, smart large screens or smart robots, etc. Among them, the server device may be a physical server, a cloud server, a server cluster, etc.

[0034] In this embodiment, the communication connection between the terminal device and the server device can be a wireless communication connection or a wired communication connection. The specific communication connection method used depends on the implementation form of the server device. When the server device is implemented as a physical machine server, both a wireless communication connection method and a wired communication connection method can be used. When the server device is implemented as a cloud server, a wireless communication connection method can be used. For the wireless communication connection method, optionally, the communication connection can be achieved through a mobile network. Accordingly, the network standard of the mobile network can be 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), 5G, WiMax or any of the new network standards that will appear in the future.

[0035] In addition, when the terminal device and the server device are located in the same local area network, they can also achieve wireless communication connection through Bluetooth, WiFi, infrared, zigbee or NFC.

[0036] In this embodiment, various applications (APPs) can be installed on the terminal device, and the application can be an independent APP or a small program that depends on the independent APP to run, and there is no limitation on this. For example, the application can be a shopping application, a taxi application, a video playback application, etc.

[0037] In this embodiment, the terminal device can provide a batch video generation service for merchants. The batch video generation service is a service that provides merchants with batch generated template videos, and the merchants generate videos that meet their needs based on the template videos. Among them, the batch generated template videos provide merchants with a wealth of editing templates, and do not require merchants to operate complex video editing software, thereby reducing the complexity of generating videos. At the same time, merchants can select template videos according to their own wishes and edit them based on the template videos, further improving the efficiency of generating videos, and can efficiently generate videos that meet their wishes.

[0038] The types of merchants are not limited. For example, merchants may include but are not limited to: food merchants, beauty and hairdressing merchants, laundry and shoe merchants, cleaning merchants, and moving merchants. For the sake of distinction and description, the following description takes the terminal device providing a batch video generation service for a target merchant as an example.

[0039] In this embodiment, the terminal device and the server device can cooperate with each other to provide batch video generation services for target merchants. The terminal device mainly involves interface interaction, and the server device is mainly responsible for data calculation. Among them, the introduction of the terminal device can be found in the subsequent embodiments, which will not be repeated here. The implementation process on the server device is described below as an example.

[0040] In this embodiment, the server device may receive a batch video generation request sent by the terminal device, where the batch video generation request includes digital human attribute information of the target merchant.

[0041] The digital human attribute information refers to the attribute information of the digital human included in the generated template video. The digital human attribute information can be set according to the target merchant's own industry needs and preferences. The digital human attribute information at least includes: role description information and application scenario information. The role description information may include characteristics such as the gender and personality of the digital human. For example, personality characteristics may include but are not limited to: friendliness, humor, wisdom, seriousness, and cheerfulness, etc. Figure 3a As shown. Among them, personality characteristics can be reflected in the voice characteristics of the digital human's broadcast voice. Digital humans with different personalities use different voice characteristics when broadcasting manuscripts. Application scenario information refers to the application scenarios corresponding to the target merchant. Application scenario information can include but is not limited to: food, beauty, beauty and hairdressing, moving, nanny, cleaning, confinement nanny, and home appliance repair, etc.

[0042] In this embodiment, an artificial intelligence (AI) language model is run on the server device, and the AI ​​language model is used to generate a global voice-over script for the application scenario information, wherein the global voice-over script is relative to the entire template video. Different application scenarios correspond to different global voice-over scripts. For example, if the application scenario information is food, the global voice-over script can be information that introduces the food sold by the merchant. If the application scenario information is a nanny, the global voice-over script can be introductory information about the nanny industry and the nanny.

[0043] Among them, the AI ​​language model can be a large language model (LLM). For example, the AI ​​language model includes but is not limited to: a generative model (GPT) based on a transformer and a pre-trained model (BERT) based on a transformer.

[0044] In this embodiment, the AI ​​language model corresponds to a first prompt word template, and the first prompt word template may include but is not limited to: role, task, background, and requirements. Among them, the role is the profession or identity played by the AI ​​language model, what personality characteristics the profession has, or what special skills the identity has, for example, the role played by the AI ​​language model is a manuscript generator; the background refers to some contextual information required by the role to complete the task, so that the AI ​​language model can better understand the task background and related details. For example, descriptive information for each application scenario information, historical interaction information of the target merchant, descriptive information of the target merchant, etc. The task refers to the specific task or goal completed by the role, for example, outputting a global oral script for the input application scenario information. Among them, in order to facilitate description and distinction, the task information in the first prompt word template is called the first task description information, and the first task description information is used to describe the task that the AI ​​language model needs to perform. Restrictions refer to the specific requirements or restrictions for the role to complete the task, such as content requirements for output data and time limits for output data. Among them, content requirements may include but are not limited to: format requirements, output data type requirements, removal of sensitive information, etc.

[0045] In this embodiment, the application scenario information is input into the corresponding first prompt word template to obtain the first prompt word, the first prompt word is input into the AI ​​language model, and multiple global oral broadcast scripts corresponding to the application scenario information are generated according to the first task description information. The multiple global oral broadcast scripts can be 2, 5, or 10, etc.

[0046] In this embodiment, at least two video description files of the target merchant are generated based on multiple global voice-over scripts and character description information, and the video description files include target character information and target global voice-over scripts. Among them, the video description file is used to describe the screen content of the video. The format of the video description file can be a file in object notation (JavaScript ObjectNotation, JavaScript, JSON) format. The target character information refers to the description information of the digital person in the video, such as the digital person's gender, clothing style, and appearance characteristics. The target global voice-over script is the script information broadcast by the digital person in the video.

[0047] In this embodiment, for any video description file, a video screen of the digital human broadcast script is generated according to the target role information and the target global oral script, and target background information is selected from the material library according to the target global oral script; the video screen of the digital human broadcast script and the target background information are synthesized to generate a template video corresponding to any video description file. For example, each video description file corresponds to a template video. Alternatively, each video description file corresponds to multiple template videos, for example, 2 or 3, etc., which are not limited to this.

[0048] The target background information refers to the background information of the digital human broadcast script, and the background information can be an image, a video, or an animation. The target background information is adapted to the target global oral broadcast script. For example, if the target global oral broadcast script is implemented as an introduction to winter food, the background information can be an image or video of a snowy scene; if the target global oral broadcast script is implemented as an introduction to cleaning, the background information can be a home scene.

[0049] In this embodiment, at least two template videos corresponding to at least two video description files can be sent to the terminal device, so that the terminal device can batch generate target videos based on digital humans according to the at least two template videos. The terminal device can generate multiple target videos based on one template video, or can generate multiple target videos based on digital humans according to multiple template videos, for example, generating five target videos based on digital humans according to two template videos. For detailed description, please refer to the subsequent embodiments, which will not be repeated here.

[0050] Among them, Figure 1b In the example, the template video is displayed through the page provided by the terminal device, and the template video includes background information, digital human and editable controls. The editable controls can be used to input copy information, image information (such as product images), video information and link information (such as product links). In addition, the template video also corresponds to a playback control. When the playback control is triggered, the terminal device responds to the triggering operation and plays the template video. During the playback process, the digital human broadcasts the script information.

[0051] In an embodiment of the present application, the application scenario information and role description information provided by the target merchant are obtained, and multiple global oral broadcast scripts corresponding to the application scenario information are generated using an AI language model. At least two video description files are generated based on the multiple global oral broadcast scripts and role description information; at least two template videos are automatically generated in batches based on the at least two video description files, and the at least two template videos are returned to the terminal device, and the target merchant generates a target video based on a digital human based on the at least two template videos. Among them, the template videos generated in batches provide users with a wealth of editing templates, without the need for merchants to operate complex video editing software, reducing the complexity of generating videos. At the same time, merchants can choose template videos according to their own wishes, and edit them based on the template videos, further improving the efficiency of generating videos, and can efficiently generate videos that meet their wishes.

[0052] In an optional embodiment, the server-side device generates at least two template videos for a target merchant that has opened a batch video generation service. Specifically, the batch video generation request also includes: identification information of the target merchant. Among them, the identification information of the target merchant may include but is not limited to: nickname, number, ID or description information, etc. Before inputting the application scenario information into the corresponding first prompt word template to obtain the first prompt word, determine whether the target merchant has opened a batch video generation information service based on the identification information of the target merchant; if so, input the application scenario information into the corresponding first prompt word template to obtain the first prompt word; if not, return a reminder message for opening the batch video generation service. The reminder message may include but is not limited to: "You have not opened the batch video generation service, do you want to open it?".

[0053] In an optional embodiment, the implementation method of generating at least two video description files of a target merchant based on multiple spoken scripts and character description information is not limited. A specific implementation method is provided below: obtaining multiple candidate character information that matches the character description information from a character material library; determining at least two groups based on the matching degree between multiple spoken scripts and multiple candidate character information, each group including mutually adapted target spoken scripts and target character information; describing the target spoken scripts and target character information included in at least two groups according to a structured grammatical format to obtain at least two video description files of the target merchant.

[0054] The character material library includes multiple character information, and each character information corresponds to a digital human. Different character information corresponds to different character description information. The character description information included in the batch video generation request is calculated similarity with the character description information corresponding to each character information in the character material library, and the character information whose similarity exceeds the set similarity threshold is used as candidate character information. The similarity threshold can be 80%, 90%, 95%, etc.

[0055] Among them, according to the matching degree between the multiple global oral scripts and the multiple candidate role information, at least two groups are determined, and each group includes mutually adapted target oral scripts and target role information. Among them, all or part of the multiple global oral scripts can be used as target oral scripts, and each group includes a target global oral script; the matching degree of the target oral script and the multiple candidate role information is calculated, and according to the matching degree of the target oral script and the multiple candidate role information, the target role information adapted to the target oral script is determined from the multiple candidate role information.

[0056] The method for calculating the degree of matching between the target spoken script and the information of multiple candidate characters is not limited. For example, the cosine similarity between the target spoken script and each candidate character information is calculated as the degree of matching. For another example, a matching degree determination model is pre-trained, the target spoken script and each candidate character information are input into the matching degree determination model, and the matching degree determination model outputs the degree of matching between the target spoken script and each candidate character information.

[0057] Regardless of which of the above-mentioned methods for calculating the degree of match is used, the implementation method of determining the target character information that matches the target spoken script from multiple candidate character information based on the degree of match between the target spoken script and multiple candidate character information is not limited. The candidate character information with the highest degree of match is used as the target character information that matches the target spoken script. Alternatively, select candidate character information with a degree of match higher than a set matching degree threshold, and randomly select one candidate character information from the multiple candidate character information as the target character information that matches the target spoken script.

[0058] The structured grammatical format can abstract and re-express information such as the target spoken script and target role information in any group. The target spoken script and target role information included in at least two groups are described using a formatted grammatical structure to obtain at least two video description files. One video description file can be generated for one group, or one video description file can be generated for multiple groups, without limitation.

[0059] In an optional embodiment, the implementation method of generating a video screen of a digital human broadcast script according to target role information and target oral broadcast script is not limited. A specific implementation method is provided below: according to a batch video generation request, determine the target script structure for the target merchant; input the target oral broadcast script and the target script structure into the second prompt word template to obtain the second prompt word, the second prompt word template also includes second task description information, which is used to describe the task that the AI ​​language model needs to perform, and the script structure includes multiple storyboards; input the second prompt word into the AI ​​language model, and generate multiple storyboard oral broadcast scripts corresponding to the multiple storyboards according to the second task description information; generate a video screen of the digital human broadcast script according to the target role information and the multiple storyboard oral broadcast scripts.

[0060] The implementation method of determining the target script structure for the target merchant based on the batch video generation request is not limited. For example, in addition to providing the digital human attribute information, the batch video generation request also provides information such as the product and the purpose of the video, and determines the target script structure based on the product, video purpose, etc. provided by the target merchant. For another example, if the batch video generation request also includes the target script structure expected by the target merchant, the target script structure can be directly obtained from the batch video generation request.

[0061] The script structure can be used to guide the elements and links in the video generation process. Different types of script structures can be used to generate different types of videos, including narrative, explanatory, etc. The script structure can include multiple storyboards, each of which can be used to guide the generation of one of the video segments of the target video, which can be a link in the process of generating the target video. Take the three-segment script structure as an example. This structure divides the video content into three main parts or storyboards. First, the first storyboard can focus on emotional marketing, with the goal of establishing a connection with the audience through emotional resonance, stimulating the audience's interest and emotional response; the second storyboard is dedicated to service introduction, where the characteristics and advantages of the service provided can be introduced in detail to ensure that the audience has a clear understanding of the service itself; the third storyboard shows the applicable population of the service and shares the feedback and evaluation of the experiencer, which helps the audience understand the actual application effect of the service and enhances the sense of trust.

[0062] Through this structured approach, the video script can be developed in an orderly manner, with each storyboard designed for specific goals and content, thus forming a coherent and persuasive video narrative. This clear script structure not only helps guide video shooting and production, but also helps ensure that the final video content can effectively convey information, attract target audiences, and achieve established marketing goals.

[0063] The target oral script and the target script structure can be input into the second prompt word template to obtain the second prompt word. The second prompt word template also includes second task description information, which is used to describe the task that the AI ​​language model needs to perform. The script structure includes multiple storyboards. The second prompt word is input into the AI ​​language model, and multiple storyboard oral scripts corresponding to the multiple storyboards are generated according to the second task description information. The storyboard oral script can include the narration or dialogue text in the current storyboard.

[0064] Among them, the AI ​​language model corresponds to a second prompt word template, and the second prompt word template may include but is not limited to: role, task, background and requirements. Among them, the role is the profession or identity played by the AI ​​language model, what personality characteristics the profession has, or what special skills the identity has, for example, the role played by the AI ​​language model is a storyboard script generator; the background refers to some contextual information required by the role to complete the task, so that the AI ​​language model can better understand the task background and related details. For example, description information for various script structures and storyboard description information in the script structure. Task refers to the specific task or goal completed by the role, for example, output storyboard script for the input global oral script and target script structure. Among them, for the convenience of description and distinction, the task information in the second prompt word template is called the second task description information, and the second task description information is used to describe the task that the AI ​​language model needs to perform. Restrictions refer to the specific requirements or restrictions for the role to complete the task, such as content requirements for output data and time limits for output data. Among them, content requirements may include but are not limited to: format requirements, output data type requirements, removal of sensitive information, etc.

[0065] An implementation method for generating a video screen of a digital human broadcasting a script according to target character information and a plurality of storyboard oral broadcast scripts, comprising: converting the plurality of storyboard oral broadcast scripts into a plurality of storyboard oral broadcast audios according to character sound information in the target character information; and generating a plurality of storyboard video screens of a digital human broadcasting the plurality of storyboard oral broadcast audios according to character appearance information in the target character information and the plurality of storyboard oral broadcast audios.

[0066] Among them, through the text-to-speech conversion technology, combined with the character voice information in the target character information, each storyboard oral script is converted into storyboard oral audio, and the sound characteristics of the oral audio are adapted to the character voice information. For example, if the character voice information is lively, the tone or timbre of the oral audio will achieve a "lively" style.

[0067] Among them, the character appearance information includes: the appearance features of the digital human, such as gender, height, weight, hairstyle and clothing. According to the character appearance information in the target character information, the digital human model is determined, and the digital human model and multiple storyboard oral audios are synthesized to obtain multiple storyboard video images. The digital human models in different storyboard video images are the same or different. Among them, the lip shape features of the digital human can be generated based on the storyboard oral audio; the lip shape features are synthesized into the digital human model to generate a storyboard video image. The server can analyze the oral audio to determine the rhythm, intonation and pronunciation of the speech, so as to generate lip shape features synchronized with the audio. The lip shape features may include, for example, the opening and closing of the lips, the position of the tongue, and possible accompanying facial expressions.

[0068] Further optionally, an implementation method of selecting target background information from a material library according to a target spoken word script includes: selecting multiple storyboard background information that are respectively adapted to the multiple storyboard spoken word scripts from the material library according to multiple storyboard spoken word scripts. For example, the similarity between each storyboard spoken word script and the background information in the material library can be calculated, and the background information with the highest similarity can be used as the storyboard background information corresponding to the storyboard spoken word script. Alternatively, the background information with a similarity higher than a set similarity threshold can be used as candidate background information, and candidate background information can be randomly selected from the candidate background information as the storyboard background information corresponding to the storyboard spoken word script.

[0069] An implementation method of synthesizing a video screen and target background information to generate a template video corresponding to any video description information file includes: synthesizing multiple storyboard video screens and multiple storyboard background information respectively to obtain a template video corresponding to any video description file.

[0070] Among them, template videos are automatically generated in batches for target merchants according to the target script structure, and the storyboard script and target character information used in the template video match the target merchant, thereby improving the accuracy of the template video content while improving the generation efficiency of the template video.

[0071] For video publishing services, target merchants can pre-authorize video publishing accounts and information collection components for different video playback platforms, and can authorize one or more video publishing accounts for any video playback platform.

[0072] When publishing a target video, in response to the playback platform selection operation performed by the target merchant, at least one target video playback platform selected by the target merchant can be obtained, and the target information collection component corresponding to each of the at least one target video playback platform can be obtained; the target video can be published to the target video playback platform using the publishing account authorized for the target video playback platform, and the call entry corresponding to the target information collection component can be displayed in the target video. In the case where there are multiple publishing accounts authorized for the target video playback platform, a publishing account can be randomly selected for publishing the target video, or a prompt window can be displayed to the target merchant, so that the target merchant can specify a publishing account for publishing the target video. Of course, other methods can also be used to publish the target video. For example, the target merchant can download the target video generated for itself to the local computer, and then use its own video playback platform account to manually publish the target video to the video playback platform.

[0073] For the video sharing service, the target merchant can share the target video generated for itself by the service platform with other merchants. After successful sharing, they can obtain sharing rewards given by the service platform.

[0074] For live video streaming services, the service platform can create live streaming based on the target video generated for the target merchant and broadcast it live on the video playback platform.

[0075] For user information collection services and information management services, after the target video is published on the video playback platform, or when the target video is used for live broadcast on the video playback platform, the comment area corresponding to the target video or the bullet screen area corresponding to the live broadcast can be monitored. If new comment content is detected in the comment area / bullet screen area, a reply content is constructed for the comment content according to a preset guidance template to guide the first user who posted the comment content to initiate a private chat request for the target merchant. In the case where the first user is detected to initiate a private chat request for the target merchant, the private chat content of the first user is generated according to the preset information collection template, and the private chat content is sent to the video playback platform to display the private chat content in the private chat window corresponding to the first user through a dialogue; if the user information input by the first user collected from the video playback platform contains a specified information field, the first user is determined as a potential user, and the user information corresponding to the first user is added to the potential user information library for the target merchant to view. Among them, the specific implementation details of the scheme can be referred to the relevant description in the aforementioned embodiment, which will not be repeated here.

[0076] For video data management services, the server-side device can provide a data display interface to the target merchant to display the playback data of the target video to the target merchant through the data display interface. The playback data includes but is not limited to: the number of likes and shares today, the number of comments on each video playback platform, the number of private messages on each video playback platform, the amount of user information collected, etc.

[0077] In this embodiment, by providing a one-stop service platform for merchants, target videos for marketing can be produced and published for merchants, and by monitoring the interaction between the target videos published on the video playback platform and the users of the video playback platform, the users are guided to initiate private chats with the target merchants, so as to mine potential users for the target merchants through private chats. In addition, the mined potential users are synchronized to the merchants in real time, so that the merchants can intuitively understand the user information of the potential users and develop potential users based on the obtained user information.

[0078] The embodiment of the present application also provides a method for generating a video based on a digital human, which is applicable to a terminal device such as Figure 2 As shown, the method includes:

[0079] 201. When the target merchant activates the batch video generation service, display a first page, which includes a batch video generation control;

[0080] 202. In response to the editing operation on the first page, obtaining the digital human attribute information of the target merchant;

[0081] 203. In response to a trigger operation on a batch video generation control, a batch video generation request is generated, where the batch video generation request includes digital human attribute information of the target merchant, where the digital human attribute information includes role description information and application scenario information;

[0082] 204. Providing a batch video generation request to a server device, so that the server device generates and returns at least two template videos based on the digital human-based video generation method provided in an embodiment of the present application;

[0083] 205. Receive at least two template videos, and generate target videos based on digital humans in batches according to the at least two template videos.

[0084] In this embodiment, the target application on the terminal device can provide a service purchase page, and the target merchant can open the batch video generation service on the service page. When the target merchant opens the batch video generation service, the first page is displayed, and the first page is used to obtain the digital human attribute information of the target merchant. Figure 3a As shown, an example of a first page is provided, but it is not limited thereto.

[0085] In an optional embodiment, a method for batch generating target videos based on digital humans according to at least two template videos includes: displaying editing entries of at least two template videos on a second page; in response to a triggering operation on the editing entry of any template video, displaying a video editing page of any template video, wherein any template video includes multiple original materials and their hierarchical relationships; in response to an editing operation on any template video on the video editing page, updating the content or hierarchical relationship of at least part of the multiple original materials according to the editing result corresponding to the editing operation; generating a target video based on digital humans corresponding to any template video according to the updated multiple original materials and their hierarchical relationships.

[0086] Among them, one implementation of the second page can be a template video list page, which displays at least two template videos received from the server device. An editing control is set for each template video, and the editing control is an implementation of an editing entry for the template video.

[0087] The editing operation for any template video may include, but is not limited to: modifying the original material in the template video and adding new material, etc. For example, modifying the oral script in the template video, adding the product material corresponding to the target merchant in the template video, etc.

[0088] Among them, the original materials in the template video can be understood as the video elements required for video generation, such as a digital human, a sticker, a text, an audio or a video, etc.

[0089] Optionally, the video editing page is embedded with a timeline manager and a video editor, and the timeline manager and the video editor interact with each other using structured data (Structured Data); wherein structured data refers to data with a fixed format, which can be written based on any protocol such as JSON or XML (eXtensible Markup Language), and this embodiment does not impose any restrictions.

[0090] The video editing page includes a first display area corresponding to the video editor, and a second display area corresponding to the timeline manager; the first display area is used to display the original material to be edited, and the second display area is used to display the timeline corresponding to the original material, and different types of original materials correspond to different timelines; in response to a trigger operation for the editing entrance of any template video, the video editing page of any template video is displayed, including: in response to a trigger operation for the editing entrance of any template video, multiple original materials are loaded, and the multiple original materials are displayed in the first display area; and when there is no timeline corresponding to any original material in the second display area, a timeline is generated according to the type of any original material, and the timeline is displayed in the second display area.

[0091] like Figure 3b As shown, an example of a video editing page is provided. The video editing page may include a first display area corresponding to the video editor and a second display area corresponding to the timeline manager. The first display area is used to display the original material to be edited. The second display area is used to display the timeline corresponding to the original material, wherein the timeline of the original material is used to organize and display the time sequence and duration of the original material. It should be noted that Figure 3b This is only an exemplary description of the video editing page, and does not limit the relative position relationship between the various display areas of the video editing page and the display size of each display area. The original material may correspond to different types, such as audio type, video type, text type, and picture type, etc. Different types of original materials may correspond to different timelines.

[0092] The target merchant can perform the material loading operation according to its own needs, and correspondingly, can respond to the material loading operation, load the original material in any template video, and display the original material in the first display area. This embodiment does not limit the specific loading method of the original material, and multiple original materials can be loaded at one time, or one original material can be loaded at one time, and this embodiment does not limit it.

[0093] In some cases, the second display area may have a timeline corresponding to the original material. For example, if an original material has been loaded before and corresponding material editing has been performed, after this original material that is not loaded for the first time is displayed in the first display area, the timeline corresponding to the original material can be displayed in the second display area. This timeline can be understood as a timeline that has been drawn previously.

[0094] In the case where there is no timeline corresponding to the original material in the second display area, a timeline can be generated according to the type of the original material, and the timeline can be displayed in the second display area. Wherein, the types of original materials are different, and the styles of the generated timelines are also different, so that the user can distinguish different types of timelines during video processing, reducing the risk of incorrect editing. Further optionally, in response to an editing operation on any template video on a video editing page, according to the editing result corresponding to the editing operation, the content or hierarchical relationship of at least part of the original materials in the multiple original materials is updated, including: calling a timeline manager, responding to the editing operation on the first timeline in the second display area, generating structured editing data, and transmitting the structured editing data to the video editor, the structured editing data including identification information of the first timeline, static description information corresponding to the first original material corresponding to the first timeline, and editing actions on the first original material; calling the video editor to perform an editing operation on the first original material according to the static description information and editing actions in the structured editing data, so as to update the content or hierarchical relationship of at least part of the original materials in the multiple original materials; displaying the editing result of the first original material in the first display area.

[0095] The target merchant can edit any one of the multiple timelines in the second display area. For the sake of convenience in description, any one of the multiple timelines is referred to as the first timeline, and the original material corresponding to the first timeline is referred to as the first original material.

[0096] Wherein, correspondingly, the timeline manager can be called, respond to the editing operation, generate structured editing data, and transmit the structured editing data to the video editor. Wherein, the timeline manager can be called by sending a call instruction to the timeline, or by passing parameters, which is not limited in this embodiment. Wherein, the structured editing data may include identification information of the first timeline, static description information corresponding to the first original material corresponding to the first timeline, and editing actions on the first original material. The identification information of the first timeline is used to identify the identity attributes of the first timeline; the static description information corresponding to the first original material corresponding to the first timeline is used to describe the attributes / features of the first timeline and the first original material, and may include: target state information of the first timeline after being edited and identification information and type information of the first original material. Wherein, the target state information is used to describe the state of the first timeline after being edited. The identification information of the first original material is used to identify the identity attributes of the first original material.

[0097] Afterwards, the video editor may be further called. Specifically, the video editor may be called by sending a call instruction to the timeline, or by passing parameters, which is not limited in this embodiment. The video editor may perform an editing operation on the first original material according to the static description information and editing actions in the structured editing data. In other words, the video editor may determine the first original material to be edited according to the structured editing data, and perform corresponding editing operations on the first original material. The video editor may also display the editing result of the first original material in the first display area. Further, the editing status of the first original material may also be displayed in the first display area.

[0098] This embodiment does not limit the specific display mode of the editing state of the first original material, and it can be any mode such as highlight display, flashing display or color change display, and this embodiment does not limit it. After editing multiple original materials, when generating a target video according to the editing results of the multiple original materials, the editing results of the multiple original materials can be synthesized to obtain the target video based on the digital human.

[0099] By displaying the editing status and editing results of the first original material in the first display area, the user can observe the first original material corresponding to the first timeline in the first display area while editing the first timeline. This can be understood as a collaborative display method of multiple display areas. Target merchants can edit the timeline more efficiently based on the timeline manager, thereby more efficiently synchronizing editing operations in the display area corresponding to the video editor, and more intuitively observe the corresponding editing status and editing effects, thereby improving the efficiency of video editing.

[0100] In this embodiment, on the one hand, the target merchant can perform video editing more efficiently through the video editing page provided by the service platform, reducing the performance requirements for the terminal device; on the other hand, through the cooperation between the timeline manager and the video editor, the target merchant can perform timeline editing more efficiently based on the timeline manager, so as to synchronize the editing operation more efficiently in the display area corresponding to the video editor, and more intuitively view the editing status and editing results of the edited original material, thereby improving the efficiency of video editing. In some optional embodiments, such as Figure 3b As shown, the video editing page may also include a third display area, and the third display area may be used for the target merchant to load a variety of original materials to be edited. Figure 3b In the figure, the third display area includes: a digital human, a map, a background, text and a video as an example, but it is not limited to this.

[0101] Based on this, the original material can be loaded in response to the material loading operation through the following implementation:

[0102] Implementation method 1: respond to the material upload operation initiated by the target merchant through the upload portal in the third display area, and use the uploaded material as the original material. The upload portal can be implemented as an upload control or an upload window, etc., which is not limited in this embodiment. The user can select materials such as pictures, texts or videos from a local or remote disk, a picture library, a video library or a text library through the upload portal for upload.

[0103] Implementation method 2: In response to the target merchant's selection operation on the candidate material displayed in the third display area, the selected candidate material is used as the original material.

[0104] That is to say, the first embodiment can support the target merchant to upload materials as original materials on their own, and the second embodiment can support the user to select original materials from existing candidate materials. The first embodiment and the second embodiment can be executed separately or in combination, and this embodiment does not limit them. In this way, the target merchant can upload rich original materials more efficiently through the third display area.

[0105] After the original material is loaded in the above manner, if there is no timeline corresponding to the original material in the second display area, the timeline can be generated according to the type of the original material in the following manner:

[0106] First, track lines are generated according to the default timeline style, wherein Canvas can be called to draw track lines according to the default timeline style. Among them, Canvas is a drawing tool that can execute drawing commands based on JavaScript (a programming language). Afterwards, the background color of the track lines can be determined according to the type of original material. Among them, the background color of the track lines can be determined based on the correspondence between the preset type and the background color of the track lines and the type of original material, and different types of original materials correspond to different background colors. Then, a background color can be added to the track line axis to obtain a timeline. In this way, the type of original material is different, and the background color of the generated timeline is also different, so that the target merchant can distinguish different types of timelines during video processing and reduce the risk of incorrect editing.

[0107] In some optional embodiments, after the timeline is generated according to the type of the original material, the timeline may be displayed in the second display area based on the following steps:

[0108] Step R1: Determine the first position corresponding to the horizontal time base of the second display area according to the start time of the original material. The horizontal time base can be used as the time base of each time axis, such as Figure 3b As shown, the line between point A and point B is the horizontal time baseline. Since the start time of the horizontal time baseline is 00:00, the corresponding first position on the horizontal time baseline of the second display area can be determined according to the start time of the original material and the start time of the horizontal time baseline.

[0109] Step R2: Determine the second position corresponding to the time axis on the vertical position baseline of the second display area according to the hierarchical relationship between the original material and the existing original material in the target video. For example, if the hierarchical relationship between the original material and the existing original material in the target video is: the level of the original material is above the level of the existing original material, then the second position corresponding to the time axis of the original material on the vertical position baseline of the second display area is also above the time axis corresponding to the existing original material.

[0110] In addition, the second position corresponding to the time axis on the vertical position baseline of the second display area can also be determined according to the positional relationship between the original material and the existing original material in the target video. For example, if the positional relationship between the original material and the existing original material in the target video is that the original material is displayed below the existing original material, then the second position corresponding to the time axis on the vertical position baseline of the second display area is also below the time axis corresponding to the existing original material.

[0111] Step R3, taking the intersection of the first position and the second position as the starting position of the time axis, and displaying the time axis in the second display area; wherein the positional relationship of each time axis on the vertical position baseline in the second display area represents the hierarchical relationship or positional relationship of the corresponding original materials in the target video.

[0112] Based on the above steps R1 to R3, the starting position of the timeline can be determined more accurately according to the starting time of the original material and the hierarchical relationship or positional relationship between the original material and the existing original material in the target video. Based on this method, the arrangement logic between the various timelines presented in the second display area can correspond to the arrangement logic of the various original materials in the first display area in the target video, so that the target merchant can more intuitively perceive the relationship between the various original materials in the first display area and the various timelines in the first display area.

[0113] On the basis of the above-mentioned embodiments, when generating structured editing data in response to the editing operation on the first time axis in the second display area, the following can be achieved based on step K1 and step K2:

[0114] Step K1: In response to an editing operation on the first timeline, the first timeline is highlighted.

[0115] The embodiments of the present application do not limit the editing operations on the first timeline. In some exemplary embodiments, the editing operations on the first timeline may include: dragging operations, selecting operations, deleting operations, cropping operations, object adding operations, or zooming operations. Among them, the object adding operation refers to adding a target object to a specified position on the first timeline. Among them, the target object may be text, animation, motion effect, picture or control, etc., which is not limited by this embodiment. The embodiments of the present application do not limit the specific implementation method of highlighting, which may be a color change display, a flashing display, a bold border display, or adding a highlight mark on the first timeline, etc., which is not limited by this embodiment.

[0116] Specifically, the editing effect on the first timeline can be highlighted according to the type of editing operation, and different editing operations correspond to different editing effects. For example, if the type of editing operation is a selected type, the corresponding editing effect can be a bold border display; for another example, if the type of editing operation is a zoom type, the corresponding editing effect can be a flashing display.

[0117] Afterwards, the editing effect can be transmitted to the video editor so that the video editor can synchronously display the editing effect during the editing process of the first original material. Specifically, the structured effect data can be determined according to the set structured data structure, the identification information of the first time axis and the editing effect information of the first time axis can be packaged to obtain the effect data, and the effect data can be sent to the video editor. The video editor can synchronously display the editing effect during the editing process of the first original material according to the effect data.

[0118] In this way, the editing effects can be synchronized in the first display area and the second display area for the same edited original material, so that the target merchant can intuitively perceive the original material corresponding to the timeline in the first display area when editing the timeline in the second display area.

[0119] Step K2, based on the type information of the editing operation, generate an editing action for the first original material. Specifically, in response to the editing operation on the first timeline, the type of the editing operation is determined based on the trajectory information of the editing operation or the editing tool used. Wherein, when determining the type of the editing operation based on the trajectory information of the editing operation, the real-time position corresponding to the editing operation can be monitored to determine the trajectory information of the editing operation based on the monitored real-time position. For example, the trajectory information of the editing operation shows that the editing operation has moved on the horizontal time baseline, that is, the editing operation involves the adjustment of the time point, and the type of the editing operation can be determined as the type of dragging. In other words, the target merchant adjusts the time point of the first timeline through the horizontal dragging operation; for another example, the trajectory information of the editing operation shows that the editing operation has moved on the vertical position baseline, that is, the editing operation involves the adjustment of the hierarchical relationship or positional relationship, and the type of the editing operation can be determined as the type of dragging. In other words, the target merchant adjusts the hierarchical relationship or positional relationship of the first timeline in multiple timelines through the vertical dragging operation.

[0120] The type of the editing operation is determined according to the editing tool used in the editing operation. For example, if the editing operation uses a zoom tool, then the type of the editing operation is a zoom type; for another example, if the editing operation uses a crop tool, then the type of the editing operation is a crop type.

[0121] After determining the type of editing operation, the editing action on the first original material can be determined according to the type of editing operation and the correspondence between the editing operation type and the editing action. Among them, the dragging operation on the horizontal time baseline can correspond to the editing action of adjusting the time point; the dragging on the vertical position baseline can correspond to the editing action of adjusting the hierarchical relationship or the editing action of adjusting the position relationship. The cropping operation can correspond to the cropping action on the first original material. The deleting operation can correspond to the deleting action on the first original material. The scaling operation can correspond to the scaling action on the first original material. The operation of adding a target object can correspond to the action of adding a motion effect or adding a text to the first original material, etc.

[0122] In this way, the editing action on the original material corresponding to the timeline can be determined more accurately based on the editing operation on the timeline.

[0123] Step K3: Generate static description information corresponding to the first original material according to the editing result information of the first time axis. Specifically, the target state information of the first time axis after editing, as well as the identification information and type information of the first original material can be obtained as the static description information.

[0124] Step K4: encapsulate the identification information, static description information and editing actions of the first time axis according to the set structured data structure to obtain structured editing data. The specific implementation of the structured data structure is not limited in this embodiment. In some exemplary embodiments, it can be a JSON data structure or an XML data structure.

[0125] Based on the above steps K1 to K4, structured editing data can be generated more accurately according to the editing operation on the first timeline, so as to achieve synchronous editing of the first original material in the first display area and synchronization of editing effects based on the structured editing data.

[0126] In some optional embodiments, when the editing operation is a drag operation, it is possible to monitor whether the start time or end time of the first time axis coincides with the end time or start time of other time axes according to the track information of the drag operation. Figure 3cAs shown, when overlap occurs, an adsorption effect can be displayed in the overlap position or related area of ​​the two overlapping time axes, and a prompt auxiliary line can be used for prompting. Among them, the prompt auxiliary line is used to show the overlap effect of the two time axes where the end time overlaps or the start time overlaps. Optionally, a drag control is also displayed at the starting position of the first time axis, wherein the drag control can be linked with the first time axis. The drag control can be implemented as a control to assist the dragger, or as other types of controls, which is not limited in this embodiment. When the editing operation is a drag operation, in response to the drag operation of the drag control, a drag operation is initiated on the first time axis. In this way, the first time axis can be dragged more efficiently by dragging the drag control.

[0127] In some optional embodiments, a video editing page is provided through the management backend, and the video editing page includes the following advantages:

[0128] 1. Intuitive operation interface: In the management background, target merchants can clearly see the "Save", "Previous", "Next" and other controls on the video editing page, as well as the visual canvas in the middle, which supports proportional scaling and real-time preview of users' creativity and ideas, making it easier for users to make adjustments. Figure 3d In the figure, a "Save" control is shown, and the current canvas ratio is shown as 44%, wherein controls such as "Previous Step" and "Next Step" are not shown.

[0129] 2. Flexible background and sticker selection: Provide custom and multi-category materials. For example, provide corresponding background materials based on the application scenario information of the target merchant. The target merchant can choose preset backgrounds from the "background library" or customize background settings to meet different scenarios and creative needs.

[0130] 3. Diverse editing elements: Supports editing of text, video, audio and other elements. Target merchants can freely add, adjust and edit these elements on the canvas to achieve rich video effects.

[0131] 4. Dubbing function: Supports target merchants to modify copy, oral script and input copy, and provides different voices, such as boyfriend power, calm, etc. for selection. Target merchants can also audition to find the dubbing effect that best meets the video requirements. Among them, copy refers to the text information displayed in the target video, and oral script is the script read by the digital human.

[0132] 5. Save settings: When saving the target video, you can select the application platform of the target video, such as the video playback platform, to adapt to the compatibility of pushing pictures on different platforms and the association of subsequent streaming; set the video name to facilitate subsequent search and management; set the clarity, including HD, SD and other options; you can also select the video ratio, such as 16:9, 9:16, etc., to adapt to the playback requirements of different platforms.

[0133] In an optional embodiment, a target video based on a digital human may also be generated directly on the video editing page.

[0134] 1. The target merchant logs in to the management backend and creates a new video editing page. The terminal device displays the video editing page.

[0135] 2. The target merchant selects the required background, such as selecting different categories of backgrounds from the background library, or custom uploading a background image.

[0136] 3. Select digital people, text, videos, and other elements from the material library and add them to the canvas.

[0137] 4. Adjust the position, width, height, rotation and other parameters of the added material. For example, drag the text "I am" to the appropriate position and adjust the width, height and angle of the picture. During the operation, you can view the effect of the adjustment in real time through the visual canvas in the middle and intuitively feel the presentation of creativity.

[0138] 5. Enter the text and select different tones (such as boyfriend power, calm, tolerance, etc.) for audition until you find a satisfactory dubbing.

[0139] 6. After editing, click the "Save" button. Select the application scenario, enter the video name, and set the definition and video ratio.

[0140] 7. After confirming the save settings, the video is saved.

[0141] The above overall process can be found in Figure 3e .

[0142] In the embodiment of the present application, the video editing page provided by the video editor provides users with a convenient, efficient, flexible and creative video editing experience through innovative design and convenient operation methods, improves editing efficiency, and fully meets the personalized creative needs of users. It is particularly suitable for target merchants to produce videos of different categories, which can help merchants quickly create high-quality personalized video content and meet the playback needs of different platforms. When saving the target video, different application scenarios are selected to directly associate with users and platforms, and the created video can be directly selected during live streaming.

[0143] Regarding the embodiments of the present application Figure 1a and Figure 2 The detailed implementation and beneficial effects of each step in the method have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0144] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 103 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0145] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0146] Figure 4 A schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application is shown in FIG. Figure 4 As shown, the device includes: a memory 44 and a processor 45.

[0147] The memory 44 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device.

[0148] The processor 45 is coupled to the memory 44 and is used to execute the computer program in the memory 44, so as to: receive a batch video generation request sent by a terminal device, the batch video generation request includes the digital human attribute information of the target merchant, the digital human attribute information includes at least role description information and application scenario information; input the application scenario information into the corresponding first prompt word template to obtain the first prompt word, the first prompt word template also includes first task description information, which is used to describe the task that the artificial intelligence AI language model needs to perform; input the first prompt word into the AI ​​language model, and generate multiple global oral broadcast scripts corresponding to the application scenario information according to the first task description information; according to A plurality of global spoken scripts and role description information are used to generate at least two video description files of the target merchant, wherein the video description files include the target role information and the target global spoken script; for any video description file, a video screen of the digital human broadcast script is generated according to the target role information and the target global spoken script, and the target background information is selected from the material library according to the target global spoken script; the video screen and the target background information are synthesized to generate a template video corresponding to any video description file; at least two template videos corresponding to at least two video description files are sent to a terminal device, so that the terminal device can batch generate target videos based on digital humans according to the at least two template videos.

[0149] In an optional embodiment, the batch video generation request also includes identification information of the target merchant; before inputting the application scenario information into the corresponding first prompt word template to obtain the first prompt word, the processor 45 is also used to: determine whether the target merchant has opened a batch video generation service based on the identification information of the target merchant; if so, input the application scenario information into the corresponding first prompt word template to obtain the first prompt word; if not, return a reminder message for opening the batch video generation service.

[0150] In an optional embodiment, when the processor 45 generates at least two video description files of a target merchant based on multiple spoken scripts and character description information, it is specifically used to: obtain multiple candidate character information that is compatible with the character description information from a character material library; determine at least two groups based on the matching degree between multiple global spoken scripts and multiple candidate character information, each group including mutually compatible target spoken scripts and target character information; describe the target spoken scripts and target character information included in at least two groups according to a structured grammatical format to obtain at least two video description files of the target merchant.

[0151] In an optional embodiment, when the processor 45 generates a video screen of the digital human broadcast script based on the target role information and the target oral broadcast script, it is specifically used to: determine the target script structure for the target merchant based on the batch video generation request; input the target oral broadcast script and the target script structure into the second prompt word template to obtain the second prompt word, the second prompt word template also includes second task description information, which is used to describe the task that the AI ​​language model needs to perform, and the script structure includes multiple storyboards; input the second prompt word into the AI ​​language model, and generate multiple storyboard oral broadcast scripts corresponding to the multiple storyboards according to the second task description information; generate a video screen of the digital human broadcast script based on the target role information and the multiple storyboard oral broadcast scripts.

[0152] Optionally, when the processor 45 generates a video screen of the digital human broadcasting the script based on the target character information and multiple storyboard broadcast scripts, it is specifically used to: convert the multiple storyboard broadcast scripts into multiple storyboard broadcast audios based on the character sound information in the target character information; and generate multiple storyboard video screens of the digital human broadcasting the multiple storyboard broadcast audios based on the character appearance information in the target character information and the multiple storyboard broadcast audios.

[0153] Further optionally, when the processor 45 selects target background information from the material library according to the target spoken script, it is specifically used to: select, according to multiple storyboard spoken scripts, multiple storyboard background information that are respectively adapted to the multiple storyboard spoken scripts from the material library; when the processor 45 synthesizes the video screen and the target background information to generate a template video corresponding to any video description file, it is specifically used to: synthesize the multiple storyboard video screens and the multiple storyboard background information respectively to obtain a template video corresponding to any video description file.

[0154] Regarding the embodiments of the present application Figure 4 The detailed implementation and beneficial effects of each step in the illustrated device have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0155] Further, if Figure 4 As shown, the electronic device also includes: a communication component 46, a display 47, a power component 48, an audio component 49 and other components. Figure 4 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 4 In addition, Figure 4 The components in the dashed box are optional components, not mandatory components, and the specific components may depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a conventional server, cloud server or server array and other server-side devices. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it may include Figure 4If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 4 Components within the dashed box.

[0156] The embodiment of the present application also provides an electronic device, which can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone or an IOT (Internet of Things) device. Figure 4 The implementation structure of the electronic device shown is the same or similar, and can refer to Figure 4 The electronic device provided in this embodiment is implemented as follows: Figure 4 The difference between the electronic devices in the illustrated embodiments mainly lies in that the functions implemented by the processor executing the computer program stored in the memory are different. For the electronic device provided in this embodiment, its processor executes the computer program stored in the memory, which can be used to: display a first page including a batch video generation control when the target merchant opens the batch video generation service; obtain the digital human attribute information of the target merchant in response to the editing operation on the first page; generate a batch video generation request in response to the triggering operation of the batch video generation control, the batch video generation request includes the digital human attribute information of the target merchant, and the digital human attribute information includes role description information and application scenario information; provide the batch video generation request to the server device for the server device to generate the batch video generation request based on the batch video generation service. Figure 1a The method shown generates and returns at least two template videos; receives at least two template videos, and generates target videos based on digital humans in batches according to the at least two template videos.

[0157] In an optional embodiment, when the processor generates target videos based on digital humans in batches according to at least two template videos, the processor is specifically used to: display the editing entries of at least two template videos on the second page; in response to a trigger operation on the editing entry of any template video, display the video editing page of any template video, wherein any template video includes multiple original materials and their hierarchical relationships; in response to an editing operation on any template video on the video editing page, update the content or hierarchical relationship of at least part of the multiple original materials according to the editing result corresponding to the editing operation; and generate a target video based on digital humans corresponding to any template video according to the updated multiple original materials and their hierarchical relationships.

[0158] Optionally, the video editing page is embedded with a timeline manager and a video editor, and structured data is used to interact between the timeline manager and the video editor; the video editing page includes a first display area corresponding to the video editor, and a second display area corresponding to the timeline manager; the first display area is used to display the original material to be edited, and the second display area is used to display the timeline corresponding to the original material, and different types of original materials correspond to different timelines; when the processor responds to a trigger operation for the editing entry of any template video and displays the video editing page of any template video, it is specifically used to: load multiple original materials in response to a trigger operation for the editing entry of any template video, and display the multiple original materials in the first display area; and when there is no timeline corresponding to any original material in the second display area, generate a timeline according to the type of any original material and display the timeline in the second display area.

[0159] Further optionally, when the processor responds to an editing operation on any template video on a video editing page and updates the content or hierarchical relationship of at least part of the multiple original materials according to the editing result corresponding to the editing operation, it is specifically used to: call the timeline manager, respond to the editing operation on the first timeline in the second display area, generate structured editing data, and transmit the structured editing data to the video editor, the structured editing data including identification information of the first timeline, static description information corresponding to the first original material corresponding to the first timeline, and editing actions on the first original material; call the video editor to perform an editing operation on the first original material according to the static description information and editing actions in the structured editing data, so as to update the content or hierarchical relationship of at least part of the multiple original materials; and display the editing result of the first original material in the first display area.

[0160] The detailed implementation and beneficial effects of the electronic device provided in the embodiments of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0161] Accordingly, the embodiment of the present application further provides a computer-readable storage medium storing a computer program, which can achieve the above-mentioned Figure 1a and Figure 2 The steps in the method embodiment shown may be executed by an electronic device.

[0162] Accordingly, the embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, causes the processor to implement the above Figure 1a and Figure 2 The steps in the method embodiment shown may be executed by an electronic device.

[0163] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0164] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0165] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0166] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0167] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0168] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0169] The present application is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0170] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0171] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0172] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0173] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0174] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0175] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0176] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for generating a video based on a digital human, characterized in that: Applicable to server devices, including: Receiving a batch video generation request sent by a terminal device, wherein the batch video generation request includes digital human attribute information of a target merchant, wherein the digital human attribute information includes at least role description information and application scenario information; Input the application scenario information into the first prompt word template corresponding thereto to obtain a first prompt word, wherein the first prompt word template also includes first task description information, which is used to describe the task that the artificial intelligence AI language model needs to perform; Inputting the first prompt word into the AI ​​language model, and generating a plurality of global oral scripts corresponding to the application scenario information according to the first task description information; Generate at least two video description files of the target merchant according to the multiple global spoken scripts and the role description information, wherein the video description files include target role information and target global spoken scripts; For any video description file, according to the target role information and the target global oral script, a video screen of the digital human broadcast script is generated, and according to the target global oral script, target background information is selected from the material library; the video screen and the target background information are synthesized to generate a template video corresponding to any video description file; The at least two template videos corresponding to the at least two video description files are sent to the terminal device, so that the terminal device can batch generate target videos based on digital humans according to the at least two template videos.

2. The method according to claim 1, characterized in that The batch video generation request also includes identification information of the target merchant; before inputting the application scenario information into the corresponding first prompt word template to obtain the first prompt word, the method also includes: According to the identification information of the target merchant, determining whether the target merchant has opened a batch video generation service; If yes, input the application scenario information into the corresponding first prompt word template to obtain the first prompt word; If not, a reminder message for activating the batch video generation service is returned.

3. The method according to claim 1, characterized in that Generating at least two video description files of the target merchant according to the multiple oral scripts and the role description information, including: Acquire multiple candidate role information that matches the role description information from a role material library; Determine at least two groups according to the matching degree between the multiple global oral scripts and the multiple candidate role information, each group including mutually adapted target oral scripts and target role information; According to a structured grammatical format, the target spoken script and target role information included in the at least two groups are described to obtain at least two video description files of the target merchant.

4. The method according to claim 1, characterized in that According to the target character information and the target oral script, a video screen of the digital human broadcasting the script is generated, including: Determining a target script structure for a target merchant based on the batch video generation request; Inputting the target oral script and the target script structure into a second prompt word template to obtain a second prompt word, wherein the second prompt word template also includes second task description information for describing the task to be performed by the AI ​​language model, and the script structure includes multiple storyboards; Inputting the second prompt word into the AI ​​language model, and generating a plurality of storyboard spoken scripts corresponding to the plurality of storyboards according to the second task description information; According to the target character information and multiple storyboard broadcast scripts, a video screen of the digital human broadcasting the script is generated.

5. The method according to claim 4, characterized in that According to the target character information and the multiple storyboard scripts, a video screen of the digital human broadcasting the script is generated, including: According to the character sound information in the target character information, the plurality of storyboard oral broadcast scripts are converted into a plurality of storyboard oral broadcast audios; Based on the character appearance information in the target character information and the multiple storyboard oral audios, multiple storyboard video images of the digital human broadcasting the multiple storyboard oral audios are generated.

6. The method according to claim 5, characterized in that Selecting target background information from the material library according to the target oral script includes: According to the multiple storyboard spoken scripts, multiple storyboard background information respectively adapted to the multiple storyboard spoken scripts from the material library; The video screen and the target background information are synthesized to generate a template video corresponding to any video description file, including: The multiple storyboard video images and the multiple storyboard background information are synthesized respectively to obtain a template video corresponding to any video description file.

7. A method for generating a video based on a digital human, characterized in that: Applicable to a terminal device, the method comprises: When the target merchant activates the batch video generation service, displaying a first page, wherein the first page includes a batch video generation control; In response to an editing operation on the first page, acquiring digital human attribute information of the target merchant; In response to a trigger operation on the batch video generation control, a batch video generation request is generated, wherein the batch video generation request includes digital human attribute information of a target merchant, and the digital human attribute information includes role description information and application scenario information; Providing the batch video generation request to a server device, so that the server device generates and returns at least two template videos based on the method of any one of claims 1 to 6; The at least two template videos are received, and target videos based on digital humans are generated in batches according to the at least two template videos.

8. The method according to claim 7, characterized in that Generating target videos based on digital humans in batches according to the at least two template videos includes: Displaying editing entrances of the at least two template videos on the second page; In response to a trigger operation on an editing entry of any template video, displaying a video editing page of any template video, wherein any template video includes a plurality of original materials and their hierarchical relationships; In response to an editing operation on any of the template videos on the video editing page, updating the content or hierarchical relationship of at least part of the multiple original materials according to an editing result corresponding to the editing operation; According to the updated multiple original materials and their hierarchical relationships, a target video based on a digital human corresponding to any one of the template videos is generated.

9. The method according to claim 8, characterized in that The video editing page is embedded with a timeline manager and a video editor, and the timeline manager and the video editor interact with each other using structured data; the video editing page includes a first display area corresponding to the video editor, and a second display area corresponding to the timeline manager; The first display area is used to display the original material to be edited, and the second display area is used to display the time axis corresponding to the original material, and different types of original materials correspond to different time axes; In response to a trigger operation on an editing entry of any template video, a video editing page of any template video is displayed, including: In response to a triggering operation on an editing entry of any template video, the plurality of original materials are loaded and displayed in the first display area; as well as In the case that there is no time axis corresponding to any original material in the second display area, a time axis is generated according to the type of any original material, and the time axis is displayed in the second display area.

10. The method according to claim 9, characterized in that In response to an editing operation on any of the template videos on the video editing page, updating the content or hierarchical relationship of at least part of the multiple original materials according to an editing result corresponding to the editing operation, including: Calling the timeline manager, responding to the editing operation on the first timeline in the second display area, generating structured editing data, and transmitting the structured editing data to the video editor, wherein the structured editing data includes identification information of the first timeline, static description information corresponding to the first original material corresponding to the first timeline, and editing actions on the first original material; Invoking the video editor to perform an editing operation on the first original material according to the static description information and the editing action in the structured editing data, so as to update the content or hierarchical relationship of at least part of the original materials among the multiple original materials; The editing result of the first original material is displayed in the first display area.

11. An electronic device, characterized in that: include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method of any one of claims 1 to 6 and claims 7 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps in the method according to any one of claims 1 to 6 and claims 7 to 10.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the processor is caused to implement the steps in the method of any one of claims 1 to 6 and claims 7 to 10.

Images