Video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video generation method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining first media content; according to the first media content, first split script data is obtained, the first split script data comprises a first split script fragment, and the first split script fragment comprises second media content and a first split text; the first split text is used for describing at least one picture element in the second media content and a shot feature of the media content generated based on the first split script segment, and the second media content comprises at least one media content in the first media content; generating third media content according to the first split script fragment, and adding the third media content into the first split script fragment to obtain second split script data; and generating a video based on the second split script data. The video generation of the whole process is realized, and the video generation efficiency and the video quality are improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to a video generation method, device, electronic device, and storage medium. Background Art

[0002] Currently, with the development of Artificial Intelligence Generated Content (AIGC) technology, users can use tools provided in video editing software to generate various media materials during the process of video content creation, thereby improving the efficiency of video content production.

[0003] In the prior art, in the process of producing video content, a common implementation method is that a user generates a video that meets the picture requirements by inputting picture description words into an image processing model.

[0004] However, since the picture description words require input skills and experience, the existing technology solutions have difficulty in generating long videos with complex video content, that is, there are problems such as low video generation efficiency and poor video quality. Summary of the invention

[0005] The embodiments of the present disclosure provide a video generation method, device, electronic device and storage medium to overcome the problems of low video generation efficiency, poor video quality and high video similarity.

[0006] In a first aspect, an embodiment of the present disclosure provides a video generation method, including:

[0007] Acquire first media content; acquire first storyboard script data according to the first media content, wherein the first storyboard script data includes a first storyboard script segment, the first storyboard script segment includes second media content and a first storyboard text, the first storyboard text is used to describe at least one screen element in the second media content and a lens feature of the media content generated based on the first storyboard script segment, the second media content includes at least one media content in the first media content; generate third media content according to the first storyboard script segment, and add the third media content to the first storyboard script segment to obtain second storyboard script data; generate a video based on the second storyboard script data.

[0008] In a second aspect, an embodiment of the present disclosure provides a video generating device, including:

[0009] an acquisition module, configured to acquire first media content, and acquire first storyboard data according to the first media content, wherein the first storyboard data includes a first storyboard segment, the first storyboard segment includes second media content and a first storyboard text, the first storyboard text is used to describe at least one screen element in the second media content and a shot feature of the media content generated based on the first storyboard segment, and the second media content includes at least one media content in the first media content;

[0010] A processing module, configured to generate third media content according to the first storyboard script fragment, and add the third media content to the first storyboard script fragment to obtain second storyboard script data;

[0011] A generation module is used to generate a video based on the second storyboard script data.

[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0013] The memory stores computer-executable instructions;

[0014] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video generation method as described in the first aspect and various possible designs of the first aspect.

[0015] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0016] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the video generation method described in the first aspect and various possible designs of the first aspect.

[0017] The video generation method, device, electronic device and storage medium provided in the present embodiment obtain first media content; obtain first storyboard script data according to the first media content, wherein the first storyboard script data includes a first storyboard script segment, the first storyboard script segment includes second media content and a first storyboard text, the first storyboard text is used to describe at least one screen element in the second media content and a shot feature of the media content generated based on the first storyboard script segment, the second media content includes at least one media content in the first media content; generate third media content according to the first storyboard script segment, and add the third media content to the first storyboard script segment to obtain second storyboard script data; and generate a video based on the second storyboard script data. By directly generating first storyboard data based on the first media content provided by the user, and then using the image features of the second media content corresponding to each storyboard segment in the first storyboard data and the lens features represented by the first storyboard text to generate the corresponding third media content, and adding it to the corresponding storyboard segment to generate the second storyboard data, and finally, generating a video based on the storyboard segment in the generated second storyboard data, the whole process of video generation is realized, and there is no need for the user to manually control the media material generation process and the typesetting and editing process, thereby improving the video generation efficiency and video quality. At the same time, due to the differences in the first media content input by the user, the video finally generated based on the first media content is also diverse, avoiding the problem of video similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 An application scenario diagram of the video generation method provided in the embodiment of the present disclosure;

[0020] Figure 2 A schematic diagram of the process of the video generation method provided in the embodiment of the present disclosure Figure 1 ;

[0021] Figure 3 for Figure 2 A flowchart of a specific implementation method of step S102 in the illustrated embodiment;

[0022] Figure 4 for Figure 3 A flowchart of a specific implementation method of step S1021 in the illustrated embodiment;

[0023] Figure 5 for Figure 3 A flowchart of a specific implementation method of step S1022 in the illustrated embodiment;

[0024] Figure 6 A schematic diagram of a process for generating a video provided by an embodiment of the present disclosure;

[0025] Figure 7 A schematic diagram of the process of the video generation method provided in the embodiment of the present disclosure Figure 2 ;

[0026] Figure 8 A schematic diagram of an interactive interface provided by an embodiment of the present disclosure;

[0027] Fig. 9 for Figure 8 A flowchart of a specific implementation method of step S205 in the illustrated embodiment;

[0028] Fig.10 A schematic diagram of a process for generating an optimized storyboard text provided by an embodiment of the present disclosure;

[0029] Fig.11 A schematic diagram of another process for generating optimized storyboard text provided by an embodiment of the present disclosure;

[0030] Fig.12 A structural block diagram of a video generation device provided in an embodiment of the present disclosure;

[0031] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0032] Fig.14 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] The application scenarios of the embodiments of the present disclosure are explained below:

[0036] The video generation method provided by the embodiment of the present disclosure can be applied to an application (APP, Application) with a video editing function, such as a video editing application, a short video application, etc. More specifically, it can be applied to an application scenario in which personalized video content is generated based on AIGC technology. The execution subject of this embodiment can be a terminal device that runs the above-mentioned application with a video editing function, or a server that deploys the server corresponding to the above-mentioned application, or other electronic devices that perform similar functions. Among them, when the execution subject is a terminal device, the terminal device executes the method provided by this embodiment by running the above-mentioned application; when the execution subject is a server, the server of the above-mentioned application with a video editing function can be partially or completely run on the server, and the method provided by this embodiment is executed on the server side, and the terminal device runs the client of the application, and the communication between the server and the terminal device is based on the server-client, so that the terminal device can obtain the execution result of the method provided in this embodiment and display it as needed.

[0037] Among them, in some embodiments, the terminal device or server can implement the video generation method provided by the embodiment of the present disclosure by running various computer executable instructions or computer programs. For example, computer executable instructions can be program-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local application, that is, a program that needs to be installed in the operating system to run, or it can be a small program embedded in any APP, that is, a program that runs based on a browser environment. In summary, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form, and the specific implementation form can be configured as needed. Furthermore, in the process of implementing the video generation method provided by the embodiment of the present disclosure, the terminal device can execute the method by running a computer executable instruction or computer program set locally, or it can execute the method by calling a computer executable instruction or computer program set in an external server. In some embodiments, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud storage, cloud communications, cloud databases, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms. Among them, cloud services can be interactive processing services for terminal devices to call.

[0038] Figure 1 This is an application scenario diagram of the video generation method provided by the embodiment of the present disclosure, referring to Figure 1 As shown in , taking a terminal device as an example, a target application with a video editing function is running in the terminal device. The user can describe the picture content of the video he wants to generate by inputting description words into the target application, such as "generate a sunset video". After that, the terminal device sends the above description words to the image generation model based on AIGC technology deployed on the server side to generate the corresponding video, and returns the video to the terminal device for display, thereby completing the video generation process.

[0039] However, in the prior art, Figure 1In the video generation scenario shown, only video segments with simple picture content and short duration can be generated. For videos with more complex content, such as commercials, promotional videos, video logs (vlogs), and other complex content videos with multiple storyboards and camera movement logic, very complex description words are required to construct the video outline and shot script. Only after repeated debugging can the expected video content be generated. As a result, most ordinary users cannot directly generate the above-mentioned complex content videos through description words, but still use traditional manual editing and typesetting methods to create video content, resulting in problems such as low video generation efficiency and poor video quality.

[0040] The embodiment of the present disclosure provides a video generation method to solve the above-mentioned problem.

[0041] refer to Figure 2 , Figure 2 A schematic diagram of the process of the video generation method provided in the embodiment of the present disclosure Figure 1 The method of this embodiment can be applied in a terminal device or a server. The video generation method includes:

[0042] Step S101: Acquire first media content;

[0043] Step S102: According to the first media content, first storyboard script data is obtained, wherein the first storyboard script data includes a first storyboard script fragment, the first storyboard script fragment includes second media content and a first storyboard text, the first storyboard text is used to describe at least one screen element in the second media content and shot features of the media content generated based on the first storyboard script fragment, and the second media content includes at least one media content in the first media content.

[0044] For example, reference Figure 1 As shown in the schematic diagram of the application scenario, in this embodiment, the provided video generation method is introduced with the terminal device as the execution subject. A target application with image editing function is running in the terminal device. Through the interactive interface of the target application, the user can manually select the media content stored locally on the terminal device or in the cloud, that is, the first media content, so that the terminal device loads the first media content. Among them, the media content is media data, such as videos and pictures shot or downloaded by the user. Specifically, the first media content includes one or more media content. For example, the first media content is 4 pictures selected by the user and stored locally on the terminal device. For another example, the first media content is 2 pictures and 1 video selected by the user and stored locally on the terminal device.

[0045] After obtaining the first media content, the terminal device processes the first media content to generate first storyboard data, wherein the storyboard data may be a data structure in the target application for generating a final output video, illustratively, the first storyboard data is draft data in the target application, more specifically, the first storyboard data is, for example, the draft video_1. The specific implementation process of step S102 is the process of creating a corresponding draft data based on the first media content selected by the user.

[0046] Further, the first storyboard script data includes one or more storyboard script segments, each storyboard script segment corresponds to a storyboard script, and a storyboard segment of the finally generated video. The first storyboard script segment is a storyboard script segment generated and created based on the first media content. Among them, the first storyboard script segment includes the second media content and the first storyboard text, and the second media content includes at least one media content in the first media content. The second media content is, for example, a frame of picture or a video, and the second media content is one or more media content in the first media content, that is, the second media content is a subset of the first media content. More specifically, the first media content includes picture p1, picture p2 and picture p3, and the second media content is, for example, picture p1. The first storyboard text is used to describe at least one screen element in the second media content and the lens features of the media content generated based on the first storyboard script segment. Specifically, the first storyboard text is a description text, and its content describes at least one screen element in the second media content and the lens features of the media content generated based on the first storyboard script segment. Exemplarily, the content of the first storyboard text is: "Wide-angle panoramic lens, the background is a clear sky, and the lens slowly advances a corner." The first storyboard text describes the shot features of a video storyboard. In subsequent steps, a corresponding storyboard video clip will be generated based on the first storyboard text, so that the generated storyboard video clip has the above-mentioned shot features.

[0047] Furthermore, in addition to the first storyboard script segment, the first storyboard script data may also include other storyboard script segments, such as a second storyboard script segment, a third storyboard script segment, etc., which are respectively used to generate other storyboard video segments. Similar to the first storyboard script segment, the second storyboard script segment and the third storyboard script segment also have the characteristics of the first storyboard script segment, that is, they include corresponding second media content and first storyboard text, and the second media content is one or more media contents in the first media content that are different from the second media content corresponding to the first storyboard script segment. The creation process of the first storyboard script segment (i.e., the second media content and the first storyboard text) can be implemented by a pre-trained model.

[0048] Furthermore, if Figure 3As shown, in a possible implementation, the specific implementation of step S102 includes:

[0049] Step S1021: obtaining at least one second media content according to content features of each media content in the first media content, wherein the content features of the second media content correspond to a preset content scene, and the preset content scene represents an image picture composed of picture elements with specific associations.

[0050] Exemplarily, first, the terminal device parses each media content in the first media content through the image processing model to obtain the content features corresponding to each media content, such as an image feature matrix and a vector; then, based on the above content features, the terminal device identifies the media content with the preset content scene as the second media content, wherein the preset content scene represents the image screen composed of screen elements with specific associations. The preset content scene is, for example, "coffee shop exploration", "sunset", "humanistic attractions", etc. For example, the first media content includes pictures p1, p2 and p3. The image processing model is used to extract and classify the content features of the above pictures p1, p2 and p3, and it is determined that the image screen composed of the screen elements (such as coffee tables, coffee tables, billboards) in the picture p1 corresponds to the preset content scene of "coffee shop exploration", and the picture p1 is determined as the second media content (one of them). Similarly, through the image processing model, it is determined that the preset content scene corresponding to the picture p2 is "humanistic attractions", and the picture p2 can also be determined as the second media content. If the preset content scene corresponding to the image p3 is not detected by the image processing model, the image p3 will not be determined as the second media content. The case where the media content is a video is similar and will not be described in detail.

[0051] Furthermore, in the above situation, when the first media content only contains a unique media content, such as picture p1, if the picture p1 corresponds to a preset content scene, it is determined as the second media content; if the picture p1 does not correspond to any preset content scene, it returns and displays a prompt message to prompt the user to reselect, otherwise the final video cannot be generated. When the first media content contains more than one media content, and more than one media content corresponds to a preset content scene, in one possible implementation, one of them can be determined as the second media content by further responding to the user's selection operation. In another possible implementation, the second media content can also be determined by system recommendation.

[0052] Specifically, in a possible implementation, Figure 4 As shown, the specific implementation of step S1021 includes:

[0053] Step S1021 - 1 : identifying at least one preset content scene according to content features of each media content in the first media content.

[0054] Step S1021 - 2 : Determine at least one target content scene from at least one preset content scene, where the target content scene is a hot content scene in the video content platform.

[0055] Step S1021 - 3: determine the media content corresponding to the target content scene as the second media content.

[0056] For example, first, based on the content features of each media content in the first media content, at least one preset content scene is identified, and then, by obtaining the hot content scenes in the video content platform, for example, videos of the "coffee shop exploration" category are "hot videos" of the current video content platform, the "coffee shop exploration" scene (hot content scene) in the at least one preset content scene is determined as the target content scene. Then, the media content corresponding to the target content scene is determined as the second media content.

[0057] In the steps of this embodiment, by combining the popular information of the video content platform, video content scenes with higher dissemination rates and larger viewing volumes, namely hot content scenes, are identified from multiple media contents, and then the second media content is determined based on the hot content scene, and the final video is generated based on the second media content, thereby improving the video dissemination properties of the final generated video.

[0058] Step S1022: Process the second media content by calling the script generation model to generate a first storyboard text corresponding to the second media content.

[0059] Step S1023: Generate first storyboard script data according to the second media content and the corresponding first storyboard text.

[0060] Exemplarily, after determining the second media content, the second media content is processed by calling the script generation model to generate a corresponding first storyboard text, which is a description word (prompt word, prompt) text. The script generation model can be a pre-trained neural network model, which can generate corresponding description words, that is, the first storyboard text, according to the input media content (picture, video). The ability of the script generation model is obtained through training, and its implementation principle will not be elaborated here.

[0061] In one possible implementation, Figure 5 As shown, the specific implementation of step S1022 includes:

[0062] Step S1022-1: Generate a corresponding first prompt word according to a preset content scene corresponding to the second media content.

[0063] Step S1022-2: Input the first prompt word and the second media content into the script generation model to generate a first storyboard text.

[0064] Exemplarily, before the second media content is input into the script generation model, first, the scene name of the preset content scene corresponding to the second media content is obtained. The preset content scene corresponding to the second media content is determined manually or automatically in the previous step, such as a target content scene determined based on a hot content scene. Afterwards, the scene name of the preset content scene corresponding to the second media content is converted into the corresponding first prompt word. The first prompt word is used to control the process of the script generation model to generate the first storyboard text, so that the first storyboard text generated by the script generation model matches the scene name. For example, the content of the first prompt word is "This is a video clip with the theme of exploring coffee shops." Afterwards, the first prompt word and the second media content are input into the script generation model. The script generation model will combine the image content features of the second media content and the content of the above-mentioned first prompt word to generate the corresponding first storyboard text. In this embodiment, the process of generating the first storyboard text by the script generation model is controlled by converting the scene name of the preset content scene corresponding to the second media content into an additional prompt word (first prompt word), and the prior knowledge obtained in the previous steps is fully utilized to match the content of the generated first storyboard text with the content scene corresponding to the second media content (such as the hot content scene), thereby improving the accuracy and content richness of the generated first storyboard text.

[0065] Further, optionally, in this embodiment, after step S101, the following steps are further included:

[0066] Step S100: Acquire a feature tag of a target user, where the feature tag is used to characterize the interest points of the target user.

[0067] Accordingly, the specific implementation of step S102 includes:

[0068] First storyboard script data is obtained according to the first media content and the characteristic tag of the target user.

[0069] Specifically, in a possible implementation, after the terminal device obtains the first media content selected by the user, in addition to selecting the second media content by using the image content features of the first media content as described in the above embodiment, the second media data combined with the personal interest points can also be determined by combining the feature tags of the target user, and then the first storyboard script data is generated based on the second media data combined with the personal interest points. That is, the specific implementation of step S1021 is: obtaining at least one second media content according to the content features of each media content in the first media content and the feature tags of the target user.

[0070] In another possible implementation, after the terminal device determines the second media data, the script generation model is called to process the second media content and the above-mentioned feature tag to generate a first storyboard text combining the personal interest point and the second media content. That is, correspondingly, the specific implementation of step S1022 is: the script generation model is called to process the second media content and the feature tag of the target user to generate a first storyboard text corresponding to the second media content.

[0071] The above-mentioned feature tag can be obtained by the terminal device from the video content platform after authorization by the target user. The specific implementation method of the first storyboard script data obtained based on the feature tag can be configured as needed and is not specifically limited at this time.

[0072] Step S103: Generate third media content according to the first storyboard script fragment, and add the third media content to the first storyboard script fragment to obtain second storyboard script data.

[0073] Step S104: Generate a video based on the second storyboard script data.

[0074] Furthermore, when generating the first storyboard script segment, the video generation model constructed based on the neural network model can be further called through the second media content corresponding to the first storyboard script segment and the first storyboard text to generate the corresponding storyboard video segment, wherein the first storyboard text is a descriptive word that matches the video generation model. In one possible implementation, the first storyboard text can be input into the video generation model, and the video generation capability of the video generation model is used to generate a video that matches the picture features and lens features described in the first storyboard text, i.e., the third media content. In another possible implementation, the first storyboard text and the second media content can be input into the video generation model together, and the video generation capability of the video generation model is used to generate a video that matches the picture features of the second media content and the picture features and lens features described in the first storyboard text. In this case, the first storyboard text contains processing operation instructions for the second media content, such as: continuing to write the second media content (video), generating a video based on the second media content (picture), etc., so as to generate the third media content.

[0075] Afterwards, the third media content is added to the first storyboard script fragment. The third media content can be a picture or a video. The first storyboard script fragment corresponds to a storyboard fragment in the draft for generating a video. After one or more of the above steps, the first storyboard script fragment contains more than one third media content. At the same time, after a storyboard script fragment (the first storyboard script fragment) in the first storyboard script data changes, it is equivalent to forming an updated storyboard script data, that is, the second storyboard script data. Finally, the updated first storyboard script data, that is, the second storyboard script data, is used to perform video fusion and combination to generate the final video.

[0076] Figure 6 A schematic diagram of a process for generating a video provided by an embodiment of the present disclosure is shown below. Figure 6 The above-mentioned embodiment steps are described in more detail. Figure 6 As shown, based on the user's selection operation, the terminal device obtains the first media content and creates the first storyboard script data, which includes picture P1, picture P2, picture P3 and picture P4. Afterwards, the first media content is screened by obtaining and using user tags and hot content scenes, and the second media content is determined to be picture P1 and picture P2. Afterwards, the pre-trained model is used to process the pictures P1 and P2 to generate the corresponding first storyboard texts, namely storyboard text T1 and storyboard text T2. In the first storyboard script data, corresponding storyboard script segments are created for picture P1 and picture P2, namely storyboard script segment A and storyboard script segment B, wherein storyboard script segment A or storyboard script segment B is the first storyboard script segment. In the data structure corresponding to storyboard script segment A, picture P1 and storyboard text T1 are included; in the data structure corresponding to storyboard script segment B, picture P2 and storyboard text T2 are included. Afterwards, the video generation model is used to process the image P1 and the storyboard text T1 corresponding to the storyboard script segment A, and the image P2 and the storyboard text T2 corresponding to the storyboard script segment B, respectively, to obtain the video V1 corresponding to the storyboard script segment A and the video V2 corresponding to the storyboard script segment B. The video V1 and the video V2 are the third media content referred to in the above embodiment. Afterwards, the video V1 and the video V2 are respectively added to the corresponding first storyboard script segment, that is, the video V1 is added to the storyboard script segment A, and the video V2 is added to the storyboard script segment B, to form two storyboard video segments, and finally, the above storyboard script segment A and the storyboard script segment B are subjected to content fusion to generate the final generated video.

[0077] By acquiring first media content; acquiring first storyboard script data according to the first media content, wherein the first storyboard script data includes a first storyboard script segment, the first storyboard script segment includes second media content and a first storyboard text, the first storyboard text is used to describe at least one screen element in the second media content and a shot feature of the media content generated based on the first storyboard script segment, and the second media content includes at least one media content in the first media content; generating third media content according to the first storyboard script segment, and adding the third media content to the first storyboard script segment to obtain second storyboard script data; and generating a video based on the second storyboard script data. By directly generating first storyboard data based on the first media content provided by the user, and then using the image features of the second media content corresponding to each storyboard segment in the first storyboard data and the lens features represented by the first storyboard text to generate the corresponding third media content, and adding it to the corresponding storyboard segment to generate the second storyboard data, and finally, generating a video based on the storyboard segment in the generated second storyboard data, the whole process of video generation is realized, and there is no need for the user to manually control the media material generation process and the typesetting and editing process, thereby improving the video generation efficiency and video quality. At the same time, due to the differences in the first media content input by the user, the video finally generated based on the first media content is also diverse, avoiding the problem of video similarity.

[0078] refer to Figure 7 , Figure 7 A schematic diagram of the process of the video generation method provided in the embodiment of the present disclosure Figure 2 In this embodiment Figure 2 On the basis of the embodiment shown, an interactive step is further added, and the video generation method includes:

[0079] Step S201: Acquire first media content.

[0080] Step S202: acquiring first storyboard script data according to the first media content, wherein the first storyboard script data at least includes a first storyboard script segment and a second storyboard script segment.

[0081] Step S203: determining the sequence of the first storyboard script segment and the second storyboard script segment according to the second media content corresponding to the first storyboard script segment and the fourth media content corresponding to the second storyboard script segment.

[0082] Exemplarily, in this embodiment, the first media content selected by the user includes at least two media contents. While determining the second media content based on the first media content, the fourth media content can also be determined, wherein the fourth media content is the media content in the first media content used to generate the second storyboard script fragment. The scheme for determining the fourth media content is the same as the scheme for determining the second media content, which has been described in detail in the previous embodiment and will not be repeated here. Correspondingly, the first storyboard script data created based on the second media content includes at least two storyboard script fragments, that is, the first storyboard script data includes at least the first storyboard script fragment and the second storyboard script fragment, that is, since the first storyboard script data includes at least two storyboard script fragments, the video generated based on the first storyboard script data is composed of at least two video storyboard fragments (corresponding to the storyboard script fragment). In the related art, when there are multiple storyboard scripts or multiple video storyboard fragments need to be generated, the playback order corresponding to the storyboard video fragments needs to be manually designed, so there are problems of low efficiency and affecting video quality. In this embodiment, after determining the second media content according to the first media content, the model is used to determine the sequence of the first storyboard script segment and the second storyboard script segment according to the screen features of the second media content corresponding to the first storyboard script segment and the screen features of the fourth media content corresponding to the second storyboard script segment. Specifically, for example, the screen features of the second media content and the fourth media content are converted into description texts, and then the order of the description texts is determined based on the semantics of the description texts by using the reasoning ability of the model, and then the sequence of the segments is determined. In this way, the playback order of the storyboard video segments is adaptively matched, and the video generation efficiency and quality are improved.

[0083] Step S204: displaying the first storyboard text and the script score corresponding to the first storyboard text according to the sequence of the segments, where the script score is used to characterize the content richness of the media content described by the first storyboard text.

[0084] Further, after determining the first storyboard script fragment, the second storyboard script fragment and other storyboard script fragments, the contents of the above-mentioned storyboard script fragments are displayed according to the fragment order. Specifically, for example, the first storyboard script fragment corresponds to the first video storyboard fragment, that is, it is located at the first position in the fragment order. The first storyboard script fragment is displayed by default on the terminal device, specifically, the first storyboard text and the script score corresponding to the first storyboard text are displayed. Optionally, the corresponding second media content can also be displayed. Among them, the script score is used to characterize the content richness of the media content described by the first storyboard text. The higher the script score, the higher the content richness of the media content described by the first storyboard text, and vice versa. In a possible implementation, the script score can be determined by the number of words in the first storyboard text. The more words in the text, the higher the script score. Of course, the first storyboard text can also be evaluated by other pre-configured models to obtain a script score, which can be set as needed. Through the script score, it can be prompted and guided to optimize the content of the current first storyboard text to generate a higher quality video.

[0085] Figure 8 A schematic diagram of an interactive interface provided by an embodiment of the present disclosure, such as Figure 8 As shown, in the interactive interface of the target application, the second media content corresponding to the first storyboard script segment, the first storyboard text and the score corresponding to the first storyboard text are displayed, wherein the second media content is the picture P1 shown in the figure, and the content of the first storyboard text is "First act storyboard script: wide-angle panoramic lens, the background is a clear sky, and the lens slowly advances a corner". At the same time, the first storyboard text corresponding to the second storyboard script segment is also displayed in the interactive interface, and the content of the first storyboard text corresponding to the second storyboard script segment is "Second act storyboard script: natural light mid-ground static, slowly pull in from the window to the protagonist, the protagonist is holding a cup of coffee and looking out the window." The content of the first storyboard text corresponding to the second storyboard script segment is a description of the fourth media content corresponding to the second storyboard script segment, which will not be repeated. When the user clicks on the first storyboard text corresponding to the second storyboard script segment, the interactive interface will display the fourth media content corresponding to the second storyboard script segment. At the same time, exemplarily, when a certain storyboard script fragment (for example, the currently selected first storyboard script fragment) is selected, by clicking the "Score" button, the script score of the first storyboard text corresponding to the storyboard script fragment will be displayed. For example, as shown in the figure, the script score corresponding to the first storyboard script fragment is 50 points. For example, the highest score of the script score is 100 points, which means that the content richness of the media content described by the current first storyboard text is relatively low and can be further optimized.

[0086] Step S205: In response to the first user instruction, an optimized storyboard text corresponding to the first storyboard text is generated, wherein the optimized storyboard text includes the first picture element, and the optimized storyboard text is used to characterize the camera movement characteristics based on the first picture element.

[0087] Exemplarily, further, by responding to a first user instruction input by a user, the first storyboard text can be optimized to generate an optimized storyboard text. In one possible implementation, a trigger control is provided in the interactive interface. When the trigger control is triggered, a first user instruction is generated, thereby realizing automatic optimization of the first storyboard text, that is, a "one-key optimization" function. This function can be realized by a pre-configured prompt word model, which will not be described in detail. In another possible implementation, the content of the first storyboard text can be modified by a manually input first user instruction, thereby generating an optimized storyboard text. Among them, the optimized storyboard text obtained after optimization is used to characterize the camera movement characteristics based on the first screen element. For example, the content of an exemplary optimized storyboard text is: "A wide-angle panoramic lens with a clear sky as the lens slowly advances to a corner. A fast handheld camera lens quickly crosses the lake in a dynamic motion and enters the mountains." Among them, "lake" and "mountain" are both picture elements in the second media content, that is, the first picture elements referred to above. In the lens features described in the optimized storyboard text, specific first picture elements in the second media content are combined to describe the camera movement features, thereby achieving a richer and more dynamic description of the generated third media content, improving the richness of the content description, and thereby making the picture content of the generated third media content richer and more accurate.

[0088] Furthermore, if Fig. 9 As shown, illustratively, the specific implementation of step S205 includes:

[0089] Step S2051: In response to a first user instruction, an element identifier of a screen element in the second media content is displayed.

[0090] Step S2052: In response to a click operation on an element identifier, a first screen element is determined, and based on the first screen element, an optimized storyboard text is constructed.

[0091] For example, in combination Figure 8As shown in the schematic diagram of the interactive interface, the terminal device displays the corresponding second media content while displaying the first storyboard text. In this embodiment, the terminal device responds to the first user instruction to display the element identification of the screen element in the second media content. For example, the element identification of the screen elements such as "mountain", "lake", and "cruise boat" (for example, the abbreviation of the screen element) is displayed on the second media content (picture). Afterwards, in response to the user's point selection operation (i.e., trigger operation) on the element identification, the first screen element is determined to construct an optimized storyboard text.

[0092] Furthermore, in a possible implementation, the specific implementation of step S2052 includes:

[0093] Step S2052A-1: acquiring and displaying a corresponding prompt word template according to the content feature of the second media content, wherein the prompt word template includes a space for writing an identifier indicating a target object, and the prompt word template is used to describe a target camera movement feature realized based on at least one target object;

[0094] Step S2052A-2: In response to the click operation on the element identifier of the first screen element, the element identifier of the first screen element is written into the empty space in the prompt word template to generate an optimized storyboard text.

[0095] Exemplarily, in the steps of this embodiment, the terminal device determines a corresponding prompt word template according to the content characteristics of the second media content. The prompt word template includes a portion of fixed description text, which is determined by the content characteristics of the second media content and is used to implement the picture description of the second media content. On the other hand, the prompt word template also includes a space for writing an identifier indicating a target object. The user can determine the first picture element through a trigger operation, and then the terminal device writes the element identifier of the first picture element into the space in the prompt word template, thereby generating an optimized storyboard text.

[0096] Fig.10 A schematic diagram of a process for generating an optimized storyboard text provided by an embodiment of the present disclosure is shown in FIG. Fig.10As shown, in the interactive interface of the target application, the second media content and the first storyboard text corresponding to the first storyboard script fragment are displayed, wherein the second media content is the picture P1 shown in the figure, and the corresponding element identification, such as "mountain", "lake", etc., is also displayed at the position where each image element is located on the picture P1. The first storyboard text is generated based on the prompt word template, and its content is: "Wide-angle panoramic lens, the background is a clear sky, and the lens slowly advances a corner. A fast handheld camera lens quickly passes through ___ in a dynamic motion and enters ___". In the above-mentioned first storyboard text, "___" is a blank, and other text content is provided by the prompt word template. Afterwards, in response to the user's trigger operation on the element identification, "mountain" and "lake" are filled in the above-mentioned "___" respectively, thereby forming an optimized storyboard text, that is, "wide-angle panoramic lens, the background is a clear sky, and the lens slowly advances a corner. A fast handheld camera lens quickly passes through the lake in a dynamic motion and enters the mountains".

[0097] In this embodiment, the prompt word template that matches the image content features of the second media content is determined, and then combined with the user's selection operation, the starting point of the camera movement is selected from the image elements existing in the second media content itself, thereby realizing the design of the video camera movement. This process only requires the user to perform a simple trigger operation, which greatly improves the interaction efficiency and video generation efficiency while achieving a high degree of freedom in the creation of video content.

[0098] In another possible implementation, the specific implementation of step S2052 includes:

[0099] Step S2052B-1: In response to the second user instruction, select target platform media content in the video content platform.

[0100] Step S2052B-2: Obtain and display the storyboard text template corresponding to the target platform media content. The storyboard text template includes a first space, and the first space is used to write an identifier indicating the target object. The storyboard text template is used to describe the picture features of the platform media content and the target camera movement features achieved based on the target object.

[0101] Step S2052B-3: In response to a trigger operation for the element identifier of the first screen element, write the element identifier of the first screen element into the first empty space in the storyboard text template to generate an optimized storyboard text.

[0102] Exemplarily, in another possible implementation, the user can also generate a video by imitating the lens features of the target platform media content by acquiring and selecting the target platform media content in the video content platform. Specifically, first, the terminal device selects the target platform media content in the video content platform by responding to the second user instruction input by the user, such as a popular video in the video content platform. Afterwards, the storyboard text template corresponding to the target platform media content is acquired and displayed. The storyboard text template is similar to the first storyboard text in that both are used to describe the screen elements and lens features (such as camera movement features) of the corresponding media content. The storyboard text template contains a fixed description text and a first space, wherein the fixed description text describes the image features (such as screen elements, positions between screen elements, visual effects, etc.) of the target platform media content, and the first space is used to write an identifier indicating a target object, and the storyboard text template is used to describe the screen features of the platform media content, and the target camera movement features realized based on the target object. The storyboard text template is pre-generated, such as being generated in the process of generating the target platform media content, and its generation process is not repeated here. Afterwards, in response to the trigger operation for the element identifier of the first screen element, the element identifier of the first screen element is written into the first empty position in the storyboard text template to generate the optimized storyboard text. This process is similar to Fig.10 In the illustrated embodiment, the element identifier of the first screen element is written into the empty space in the prompt word template, so that the process of generating the optimized storyboard text is similar and will not be described in detail.

[0103] In this embodiment, the target platform media content in the video content platform is obtained, and an optimized storyboard text is generated based on the storyboard text template of the target platform media content, thereby imitating the camera movement characteristics of the target platform media content. At the same time, in this process, the user can also flexibly change the camera movement characteristics by selecting different picture elements, thereby further improving the efficiency of user-generated videos and the diversity of video content.

[0104] Further, optionally, the storyboard text template further includes a second space, and the second space is used to write a visual effect identifier representing the visual effect of the screen element. The steps of this embodiment also include:

[0105] Step S2052B-4: In response to the first trigger instruction for the second slot, display at least two candidate visual effect identifiers corresponding to the second slot.

[0106] Accordingly, the specific implementation of step S2052B-3 includes:

[0107] In response to a trigger operation for an element identifier of a first screen element, the element identifier of the first screen element is written into a first empty space in a storyboard text template, and in response to a trigger operation for a target visual effect identifier among at least two alternative visual effect identifiers, the target visual effect identifier is written into a second empty space in the storyboard text template to generate an optimized storyboard text.

[0108] Exemplarily, in another possible implementation, the storyboard text template also includes a second space, and the second space is used to write a visual effect identifier that represents the visual effect of the screen element. In short, the second space is used to write relevant descriptive words of the visual effect, such as "layered" and "glass texture". After acquiring the storyboard text template, the terminal device can obtain at least two alternative visual effect identifiers corresponding to each second space according to the storyboard text template. For the first trigger instruction of the second space, the alternative visual effect identifier corresponding to the second space can be displayed. After that, the user can select an alternative visual effect identifier as the target visual effect identifier as needed, and then write it into the second space. After that, the fixed description text of the storyboard text template, the content written in the first space, and the content written in the second space are combined to jointly generate an optimized storyboard text. Thereby, the content richness of the optimized storyboard text is further improved under the premise of ensuring the interaction efficiency.

[0109] Fig.11 A schematic diagram of another process for generating optimized storyboard text provided by an embodiment of the present disclosure, such as Fig.11As shown, in the interactive interface of the target application, the second media content (shown as P1 in the figure) corresponding to the first storyboard script fragment and the first storyboard text are displayed. The content of the first storyboard text is, for example: "First act storyboard: wide-angle panoramic lens, the background is a clear sky, and the lens slowly advances a corner". Afterwards, in response to the second user instruction, the platform media content page of the video content platform is displayed, and based on the user's selection operation, the target platform media content (shown as P2 in the figure) is obtained. Afterwards, based on the storyboard text template of the target platform media content, the text content including the first space and the second space is inserted in the first storyboard text. For example, as shown in the figure, the content inserted in the first storyboard text is: "The sun slants down from the ___ of ___, forming a natural light column like a visual focus. A fast handheld camera lens quickly passes through the ___ of ___ in a dynamic motion and enters ___". After that, the user clicks on each empty space in turn (first trigger instruction), and determines the element identifier or target visual effect identifier of the first screen element from the element identifiers or alternative visual effect identifiers of the screen elements displayed in the second media content, and writes it into the corresponding empty space, thereby generating an optimized storyboard text. For example, as shown in the figure, the content of the optimized storyboard text corresponding to the first storyboard script fragment is "First act storyboard script: wide-angle panoramic lens, with a clear sky as the background, and the lens slowly advances a corner. The sun slants down from the dense clouds, forming a natural light column like a visual focus. A fast handheld camera lens quickly passes through the layers of mountains in a dynamic motion and enters the woods."

[0110] Step S206: Display the second storyboard text corresponding to the second storyboard script fragment, and generate an optimized storyboard text corresponding to the second storyboard text in response to a second user instruction.

[0111] Exemplarily, the second storyboard text of the second storyboard script fragment can be optimized in the same manner based on the above steps to obtain an optimized storyboard text corresponding to the second storyboard text. The specific implementation method refers to the above-mentioned process of generating the optimized storyboard text corresponding to the first storyboard text, which will not be repeated here.

[0112] Step S207: Generate third media content corresponding to the first storyboard script fragment and third media content corresponding to the second storyboard script fragment according to the optimized storyboard text corresponding to the first storyboard script fragment, the optimized storyboard text corresponding to the second storyboard script fragment, and the fragment order of the first storyboard script fragment and the second storyboard script fragment.

[0113] Step S208: adding the third media content to the corresponding first storyboard script fragment and the second storyboard script fragment respectively to obtain second storyboard script data, and generating a video based on the second storyboard script data.

[0114] Furthermore, after generating an optimized storyboard text corresponding to the first storyboard script fragment and an optimized storyboard text corresponding to the second storyboard script fragment, the corresponding third media content is generated in combination with the fragment order of the first storyboard script fragment and the second storyboard script fragment. This process can be performed by inputting the optimized storyboard text corresponding to each storyboard script fragment (the first storyboard script fragment, the second storyboard script fragment, etc.) and the fragment order corresponding to each storyboard script fragment into a video generation model. The video generation model uses the above information as context information, and after comprehensive reasoning and fusion, generates the third media content corresponding to each storyboard script fragment. Since the context information such as the fragment order is combined, the third media content corresponding to each storyboard script fragment can have content continuity, thereby making the content of the finally generated video more coherent and realistic, and having better video quality.

[0115] In this embodiment, the implementation method of step S201 is the same as that of the present disclosure. Figure 2 The implementation method of step S101 in the illustrated embodiment is the same and will not be described in detail here.

[0116] Corresponding to the video generation method of the above embodiment, Fig.12 This is a structural block diagram of a video generation device provided in an embodiment of the present disclosure. The method described in the above embodiment can be executed by the video generation device, which can be implemented by software and / or hardware, and can be integrated in an electronic device with certain data processing functions. The electronic device can include but is not limited to a mobile terminal with big data processing capabilities, and a fixed terminal with big data processing capabilities such as a desktop computer and a supercomputer.

[0117] For the sake of convenience, only the parts related to the embodiments of the present disclosure are shown. Fig.12 , the video generating device 3 comprises:

[0118] An acquisition module 31 is used to acquire first media content, and acquire first storyboard data according to the first media content, wherein the first storyboard data includes a first storyboard segment, the first storyboard segment includes second media content and a first storyboard text, the first storyboard text is used to describe at least one screen element in the second media content and a shot feature of the media content generated based on the first storyboard segment, and the second media content includes at least one media content in the first media content;

[0119] A processing module 32, for generating third media content according to the first storyboard script fragment, and adding the third media content to the first storyboard script fragment to obtain second storyboard script data;

[0120] The generating module 33 is used to generate a video based on the second storyboard script data.

[0121] According to one or more embodiments of the present disclosure, when the acquisition module 31 acquires the first storyboard script data according to the first media content, it is specifically used to: obtain at least one second media content according to the content characteristics of each media content in the first media content, the content characteristics of the second media content correspond to a preset content scene, and the preset content scene represents an image picture composed of picture elements with specific correlations; process the second media content by calling the script generation model to generate a first storyboard text corresponding to the second media content; generate the first storyboard script data according to the second media content and the corresponding first storyboard text.

[0122] According to one or more embodiments of the present disclosure, when the acquisition module 31 obtains at least one second media content based on the content characteristics of each media content in the first media content, it is specifically used to: identify at least one preset content scene based on the content characteristics of each media content in the first media content; determine at least one target content scene from the at least one preset content scene, the target content scene being a hot content scene in the video content platform; and determine the media content corresponding to the target content scene as the second media content.

[0123] According to one or more embodiments of the present disclosure, when the acquisition module 31 processes the second media content by calling the script generation model to generate the first storyboard text corresponding to the second media content, it is specifically used to: generate the corresponding first prompt word according to the preset content scene corresponding to the second media content; input the first prompt word and the second media content into the script generation model to generate the first storyboard text.

[0124] According to one or more embodiments of the present disclosure, the acquisition module 31 is further used to: acquire a feature tag of a target user, where the feature tag is used to characterize the interest points of the target user; when the acquisition module 31 acquires the first storyboard script data according to the first media content, the acquisition module 31 is specifically used to: acquire the first storyboard script data according to the first media content and the feature tag of the target user.

[0125] According to one or more embodiments of the present disclosure, the first storyboard script data also includes at least a second storyboard script fragment, and the processing module 32 is further used to: determine the fragment order of the first storyboard script fragment and the second storyboard script fragment according to the second media content corresponding to the first storyboard script fragment and the fourth media content corresponding to the second storyboard script fragment; based on the fragment order, generate third media content according to the first storyboard script fragment, the second storyboard script fragment, and the fragment order of the first storyboard script fragment and the second storyboard script fragment.

[0126] According to one or more embodiments of the present disclosure, the processing module 32 is further used to: display a first storyboard text and a script score corresponding to the first storyboard text, the script score being used to characterize the content richness of the media content described by the first storyboard text; generate an optimized storyboard text corresponding to the first storyboard text in response to a first user instruction, the optimized storyboard text including a first picture element, the optimized storyboard text being used to characterize the camera movement features based on the first picture element; when the processing module 32 generates the third media content according to the first storyboard script fragment, the processing module 32 is specifically used to: generate the third media content according to the optimized storyboard text.

[0127] According to one or more embodiments of the present disclosure, when the processing module 32 generates an optimized storyboard text corresponding to the first storyboard text in response to a first user instruction, it is specifically used to: display the element identification of the screen element in the second media content in response to the first user instruction; determine the first screen element in response to a trigger operation on the element identification, and construct the optimized storyboard text based on the first screen element.

[0128] According to one or more embodiments of the present disclosure, when the processing module 32 determines the first screen element in response to a trigger operation for an element identifier and constructs an optimized storyboard text based on the first screen element, it is specifically used to: obtain and display a corresponding prompt word template according to content features of the second media content, the prompt word template comprising a space for writing an identifier indicating a target object, the prompt word template being used to describe target camera movement features achieved based on at least one target object; in response to a trigger operation for the element identifier of the first screen element, write the element identifier of the first screen element into the space in the prompt word template to generate an optimized storyboard text.

[0129] According to one or more embodiments of the present disclosure, when the processing module 32 determines the first screen element in response to a trigger operation for an element identifier, and constructs an optimized storyboard text based on the first screen element, it is specifically used to: select the target platform media content in the video content platform in response to a second user instruction; obtain and display the storyboard text template corresponding to the target platform media content, the storyboard text template including a first empty space, the first empty space is used to write an identifier indicating the target object, the storyboard text template is used to describe the screen features of the platform media content, and the target camera movement features achieved based on the target object; in response to a trigger operation for the element identifier of the first screen element, write the element identifier of the first screen element into the first empty space in the storyboard text template to generate the optimized storyboard text.

[0130] According to one or more embodiments of the present disclosure, the storyboard text template further includes a second empty space, and the second empty space is used to write a visual effect identifier representing the visual effect of the screen element; the processing module 32 is also used to: in response to a first trigger instruction for the second empty space, display at least two alternative visual effect identifiers corresponding to the second empty space; when the processing module 32 writes the element identifier of the first screen element into the first empty space in the storyboard text template in response to a trigger operation for the element identifier of the first screen element to generate an optimized storyboard text, it is specifically used to: in response to a trigger operation for the element identifier of the first screen element, write the element identifier of the first screen element into the first empty space in the storyboard text template, and in response to a trigger operation for a target visual effect identifier among the at least two alternative visual effect identifiers, write the target visual effect identifier into the second empty space in the storyboard text template to generate an optimized storyboard text.

[0131] The acquisition module 31, the processing module 32 and the generation module 33 are connected in sequence. The video generation device 3 provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0132] Fig.13 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown in FIG. Fig.13 As shown, the electronic device 4 includes:

[0133] A processor 41, and a memory 42 communicatively connected to the processor 41;

[0134] The memory 42 stores computer executable instructions;

[0135] The processor 41 executes the computer execution instructions stored in the memory 42 to implement the following Figure 2-Figure 11 The video generation method in the illustrated embodiment.

[0136] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0137] For related instructions, please refer to Figure 2-Figure 11 The relevant descriptions and effects corresponding to the steps in the corresponding embodiments can be understood, and no further elaboration is made here.

[0138] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which are used to implement the present invention when executed by a processor. Figure 2-Figure 11 The video generation method provided in any one of the corresponding embodiments.

[0139] The present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the present invention is realized. Figure 2-Figure 11 The video generation method provided in any one of the corresponding embodiments.

[0140] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0141] refer to Fig.14 , which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.14 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0142] like Fig.14 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0143] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.14 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0144] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0145] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0146] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0147] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0148] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit or module does not, in some cases, constitute a limitation on the unit itself.

[0151] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0152] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] In a first aspect, according to one or more embodiments of the present disclosure, a video generation method is provided, including:

[0154] Acquire first media content; acquire first storyboard script data according to the first media content, wherein the first storyboard script data includes a first storyboard script segment, the first storyboard script segment includes second media content and a first storyboard text, the first storyboard text is used to describe at least one screen element in the second media content and a lens feature of the media content generated based on the first storyboard script segment, the second media content includes at least one media content in the first media content; generate third media content according to the first storyboard script segment, and add the third media content to the first storyboard script segment to obtain second storyboard script data; generate a video based on the second storyboard script data.

[0155] According to one or more embodiments of the present disclosure, obtaining first storyboard script data based on the first media content includes: obtaining at least one second media content based on content features of each media content in the first media content, the content features of the second media content corresponding to a preset content scene, the preset content scene representing an image picture composed of picture elements with specific correlations; processing the second media content by calling a script generation model to generate a first storyboard text corresponding to the second media content; and generating the first storyboard script data based on the second media content and the corresponding first storyboard text.

[0156] According to one or more embodiments of the present disclosure, obtaining at least one second media content based on content features of each media content in the first media content includes: identifying at least one preset content scene based on content features of each media content in the first media content; determining at least one target content scene from the at least one preset content scene, the target content scene being a hot content scene in a video content platform; and determining the media content corresponding to the target content scene as the second media content.

[0157] According to one or more embodiments of the present disclosure, the step of processing the second media content by calling a script generation model to generate a first storyboard text corresponding to the second media content includes: generating a corresponding first prompt word according to a preset content scene corresponding to the second media content; and inputting the first prompt word and the second media content into the script generation model to generate the first storyboard text.

[0158] According to one or more embodiments of the present disclosure, it also includes: obtaining a feature tag of a target user, wherein the feature tag is used to characterize the interest points of the target user; obtaining first storyboard script data based on the first media content includes: obtaining first storyboard script data based on the first media content and the feature tag of the target user.

[0159] According to one or more embodiments of the present disclosure, the first storyboard script data also includes at least a second storyboard script fragment, and the method further includes: determining the fragment sequence of the first storyboard script fragment and the second storyboard script fragment according to the second media content corresponding to the first storyboard script fragment and the fourth media content corresponding to the second storyboard script fragment; generating the third media content according to the first storyboard script fragment includes: generating the third media content according to the first storyboard script fragment, the second storyboard script fragment, and the fragment sequence. According to one or more embodiments of the present disclosure, the method further includes: displaying the first storyboard text, and a script score corresponding to the first storyboard text, the script score is used to characterize the content richness of the media content described by the first storyboard text; generating an optimized storyboard text corresponding to the first storyboard text in response to a first user instruction, the optimized storyboard text includes a first screen element, and the optimized storyboard text is used to characterize the camera movement feature based on the first screen element; generating the third media content according to the first storyboard script fragment includes: generating the third media content according to the optimized storyboard text.

[0160] According to one or more embodiments of the present disclosure, generating an optimized storyboard text corresponding to the first storyboard text in response to a first user instruction includes: displaying an element identifier of a screen element in the second media content in response to the first user instruction; determining the first screen element in response to a trigger operation for the element identifier, and constructing the optimized storyboard text based on the first screen element.

[0161] According to one or more embodiments of the present disclosure, in response to a trigger operation for the element identifier, determining the first screen element, and constructing the optimized storyboard text based on the first screen element, includes: acquiring and displaying a corresponding prompt word template according to content features of the second media content, the prompt word template comprising a space, the space being used to write an identifier indicating a target object, the prompt word template being used to describe target camera movement features achieved based on at least one of the target objects; in response to a trigger operation for the element identifier of the first screen element, writing the element identifier of the first screen element into the space in the prompt word template to generate the optimized storyboard text.

[0162] According to one or more embodiments of the present disclosure, in response to a trigger operation for the element identifier, determining the first screen element, and constructing the optimized storyboard text based on the first screen element, includes: in response to a second user instruction, selecting target platform media content within a video content platform; acquiring and displaying a storyboard text template corresponding to the target platform media content, the storyboard text template comprising a first empty space, the first empty space being used to write an identifier indicating a target object, the storyboard text template being used to describe the screen features of the platform media content, and the target camera movement features achieved based on the target object; in response to a trigger operation for the element identifier of the first screen element, writing the element identifier of the first screen element into the first empty space in the storyboard text template to generate the optimized storyboard text.

[0163] According to one or more embodiments of the present disclosure, the storyboard text template further includes a second empty space, and the second empty space is used to write a visual effect identifier representing the visual effect of the screen element; the method also includes: in response to a first trigger instruction for the second empty space, displaying at least two alternative visual effect identifiers corresponding to the second empty space; in response to the trigger operation for the element identifier of the first screen element, writing the element identifier of the first screen element into the first empty space in the storyboard text template to generate the optimized storyboard text, including: in response to the trigger operation for the element identifier of the first screen element, writing the element identifier of the first screen element into the first empty space in the storyboard text template, and in response to the trigger operation for the target visual effect identifier among the at least two alternative visual effect identifiers, writing the target visual effect identifier into the second empty space in the storyboard text template to generate the optimized storyboard text.

[0164] In a second aspect, according to one or more embodiments of the present disclosure, a video generating device is provided, including:

[0165] an acquisition module, configured to acquire first media content, and acquire first storyboard data according to the first media content, wherein the first storyboard data includes a first storyboard segment, the first storyboard segment includes second media content and a first storyboard text, the first storyboard text is used to describe at least one screen element in the second media content and a shot feature of the media content generated based on the first storyboard segment, and the second media content includes at least one media content in the first media content;

[0166] A processing module, configured to generate third media content according to the first storyboard script fragment, and add the third media content to the first storyboard script fragment to obtain second storyboard script data;

[0167] A generation module is used to generate a video based on the second storyboard script data.

[0168] According to one or more embodiments of the present disclosure, when the acquisition module acquires the first storyboard script data according to the first media content, it is specifically used to: obtain at least one second media content according to content features of each media content in the first media content, the content features of the second media content correspond to a preset content scene, and the preset content scene represents an image picture composed of picture elements with specific correlations; process the second media content by calling a script generation model to generate a first storyboard text corresponding to the second media content; generate the first storyboard script data according to the second media content and the corresponding first storyboard text.

[0169] According to one or more embodiments of the present disclosure, when the acquisition module obtains at least one second media content based on content features of each media content in the first media content, it is specifically used to: identify at least one preset content scene based on content features of each media content in the first media content; determine at least one target content scene from the at least one preset content scene, the target content scene being a hot content scene in the video content platform; and determine the media content corresponding to the target content scene as the second media content.

[0170] According to one or more embodiments of the present disclosure, when the acquisition module processes the second media content by calling the script generation model to generate the first storyboard text corresponding to the second media content, it is specifically used to: generate a corresponding first prompt word according to a preset content scene corresponding to the second media content; and input the first prompt word and the second media content into the script generation model to generate the first storyboard text.

[0171] According to one or more embodiments of the present disclosure, the acquisition module is further used to: acquire a feature tag of a target user, wherein the feature tag is used to characterize the interest points of the target user; when the acquisition module acquires the first storyboard script data according to the first media content, the acquisition module is specifically used to: acquire the first storyboard script data according to the first media content and the feature tag of the target user.

[0172] According to one or more embodiments of the present disclosure, the first storyboard script data also includes at least a second storyboard script fragment, and the processing module is further used to: determine the fragment order of the first storyboard script fragment and the second storyboard script fragment according to the second media content corresponding to the first storyboard script fragment and the fourth media content corresponding to the second storyboard script fragment; generating the third media content according to the first storyboard script fragment includes: generating the third media content according to the first storyboard script fragment, the second storyboard script fragment, and the fragment order.

[0173] According to one or more embodiments of the present disclosure, the processing module is further used to: display the first storyboard text and a script score corresponding to the first storyboard text, wherein the script score is used to characterize the content richness of the media content described by the first storyboard text; generate an optimized storyboard text corresponding to the first storyboard text in response to a first user instruction, wherein the optimized storyboard text includes a first screen element, and the optimized storyboard text is used to characterize the camera movement characteristics based on the first screen element; when the processing module generates the third media content according to the first storyboard script fragment, the processing module is specifically used to: generate the third media content according to the optimized storyboard text.

[0174] According to one or more embodiments of the present disclosure, when the processing module generates an optimized storyboard text corresponding to the first storyboard text in response to a first user instruction, it is specifically used to: display an element identifier of a screen element in the second media content in response to the first user instruction; determine the first screen element in response to a trigger operation on the element identifier, and construct the optimized storyboard text based on the first screen element.

[0175] According to one or more embodiments of the present disclosure, when the processing module determines the first screen element in response to a trigger operation for the element identifier and constructs the optimized storyboard text based on the first screen element, it is specifically used to: acquire and display a corresponding prompt word template according to content features of the second media content, the prompt word template comprising a space, the space being used to write an identifier indicating a target object, the prompt word template being used to describe target camera movement features achieved based on at least one of the target objects; and in response to a trigger operation for the element identifier of the first screen element, write the element identifier of the first screen element into the space in the prompt word template to generate the optimized storyboard text.

[0176] According to one or more embodiments of the present disclosure, when the processing module determines the first screen element in response to a trigger operation for the element identifier, and constructs the optimized storyboard text based on the first screen element, it is specifically used to: select the target platform media content in the video content platform in response to a second user instruction; obtain and display the storyboard text template corresponding to the target platform media content, the storyboard text template containing a first empty space, the first empty space is used to write an identifier indicating a target object, the storyboard text template is used to describe the screen features of the platform media content, and the target camera movement features achieved based on the target object; in response to a trigger operation for the element identifier of the first screen element, write the element identifier of the first screen element into the first empty space in the storyboard text template to generate the optimized storyboard text.

[0177] According to one or more embodiments of the present disclosure, the storyboard text template further includes a second empty space, and the second empty space is used to write a visual effect identifier representing the visual effect of the screen element; the processing module is also used to: in response to a first trigger instruction for the second empty space, display at least two alternative visual effect identifiers corresponding to the second empty space; when the processing module writes the element identifier of the first screen element into the first empty space in the storyboard text template in response to a trigger operation for the element identifier of the first screen element to generate the optimized storyboard text, it is specifically used to: in response to a trigger operation for the element identifier of the first screen element, write the element identifier of the first screen element into the first empty space in the storyboard text template, and in response to a trigger operation for a target visual effect identifier among the at least two alternative visual effect identifiers, write the target visual effect identifier into the second empty space in the storyboard text template to generate the optimized storyboard text.

[0178] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0179] The memory stores computer-executable instructions;

[0180] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video generation method described in the first aspect and various possible designs of the first aspect.

[0181] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0182] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the video generation method described in the first aspect and various possible designs of the first aspect.

[0183] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0184] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0185] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video generation method, characterized in that: include: Obtaining first media content; Acquire first storyboard data according to the first media content, wherein the first storyboard data includes a first storyboard segment, the first storyboard segment includes second media content and a first storyboard text, the first storyboard text is used to describe at least one screen element in the second media content and a shot feature of the media content generated based on the first storyboard segment, and the second media content includes at least one media content in the first media content; generating third media content according to the first storyboard script fragment, and adding the third media content to the first storyboard script fragment to obtain second storyboard script data; Based on the second storyboard data, a video is generated.

2. The method according to claim 1, characterized in that The step of obtaining first storyboard script data according to the first media content includes: At least one second media content is obtained according to content features of each media content in the first media content, wherein the content features of the second media content correspond to a preset content scene, and the preset content scene represents an image picture composed of picture elements with specific associations; Processing the second media content by calling a script generation model to generate a first storyboard text corresponding to the second media content; The first storyboard script data is generated according to the second media content and the corresponding first storyboard text.

3. The method according to claim 2, characterized in that The obtaining at least one second media content according to the content features of each media content in the first media content includes: identifying at least one preset content scene according to content features of each media content in the first media content; Determine at least one target content scene from the at least one preset content scene, the target content scene being a hot content scene in the video content platform; The media content corresponding to the target content scene is determined as the second media content.

4. The method according to claim 2, characterized in that: The step of processing the second media content by calling the script generation model to generate a first storyboard text corresponding to the second media content includes: Generate a corresponding first prompt word according to a preset content scene corresponding to the second media content; The first prompt word and the second media content are input into the script generation model to generate the first storyboard text.

5. The method according to claim 1, characterized in that Also includes: Acquire a feature tag of a target user, where the feature tag is used to characterize an interest point of the target user; The step of obtaining first storyboard script data according to the first media content includes: First storyboard script data is acquired according to the first media content and the characteristic tag of the target user.

6. The method according to claim 1, characterized in that The first storyboard script data also includes at least a second storyboard script fragment, and the method further includes: Determining a fragment order of the first storyboard script fragment and the second storyboard script fragment according to the second media content corresponding to the first storyboard script fragment and the fourth media content corresponding to the second storyboard script fragment; Generating third media content according to the first storyboard script fragment includes: The third media content is generated according to the first storyboard script fragment, the second storyboard script fragment, and the fragment sequence.

7. The method according to claim 1, characterized in that The method further comprises: Displaying the first storyboard text and a script score corresponding to the first storyboard text, wherein the script score is used to represent the content richness of the media content described by the first storyboard text; In response to a first user instruction, generating an optimized storyboard text corresponding to the first storyboard text, wherein the optimized storyboard text includes a first screen element, and the optimized storyboard text is used to characterize a camera movement feature based on the first screen element; Generating third media content according to the first storyboard script fragment includes: The third media content is generated according to the optimized storyboard text.

8. The method according to claim 7, characterized in that The step of generating, in response to the first user instruction, an optimized storyboard text corresponding to the first storyboard text comprises: In response to the first user instruction, displaying an element identifier of a screen element in the second media content; In response to a trigger operation for the element identifier, the first screen element is determined, and based on the first screen element, the optimized storyboard text is constructed.

9. The method according to claim 8, characterized in that The step of determining the first screen element in response to the triggering operation for the element identifier, and constructing the optimized storyboard text based on the first screen element, comprises: According to the content feature of the second media content, a corresponding prompt word template is acquired and displayed, wherein the prompt word template includes a space for writing an identifier indicating a target object, and the prompt word template is used to describe a target camera movement feature realized based on at least one of the target objects; In response to a triggering operation on the element identifier of the first screen element, the element identifier of the first screen element is written into a blank space in the prompt word template to generate the optimized storyboard text.

10. The method according to claim 8, characterized in that The step of determining the first screen element in response to the triggering operation for the element identifier, and constructing the optimized storyboard text based on the first screen element, comprises: In response to a second user instruction, selecting target platform media content within the video content platform; Acquire and display a storyboard text template corresponding to the target platform media content, wherein the storyboard text template includes a first space, the first space is used to write an identifier indicating a target object, and the storyboard text template is used to describe the screen features of the platform media content and the target camera movement features achieved based on the target object; In response to a triggering operation on the element identification of the first screen element, the element identification of the first screen element is written into a first empty space in the storyboard text template to generate the optimized storyboard text.

11. The method according to claim 10, characterized in that The storyboard text template further includes a second space, and the second space is used to write a visual effect identifier representing the visual effect of the screen element; the method further includes: In response to a first trigger instruction for the second slot, displaying at least two candidate visual effect identifiers corresponding to the second slot; The step of writing the element identification of the first screen element into a first empty space in the storyboard text template in response to a triggering operation on the element identification of the first screen element to generate the optimized storyboard text comprises: In response to a trigger operation for an element identifier of the first screen element, the element identifier of the first screen element is written into a first empty space in the storyboard text template, and in response to a trigger operation for a target visual effect identifier among the at least two alternative visual effect identifiers, the target visual effect identifier is written into a second empty space in the storyboard text template to generate the optimized storyboard text.

12. A video generating device, characterized in that: include: an acquisition module, configured to acquire first media content, and acquire first storyboard data according to the first media content, wherein the first storyboard data includes a first storyboard segment, the first storyboard segment includes second media content and a first storyboard text, the first storyboard text is used to describe at least one screen element in the second media content and a shot feature of the media content generated based on the first storyboard segment, and the second media content includes at least one media content in the first media content; A processing module, configured to generate third media content according to the first storyboard script fragment, and add the third media content to the first storyboard script fragment to obtain second storyboard script data; A generation module is used to generate a video based on the second storyboard script data.

13. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the video generation method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the video generation method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the video generating method according to any one of claims 1 to 11 is implemented.

Images