Method and device for generating video based on text

Abstract

The embodiment of the invention provides a method and a device for generating a video based on a text, the method is applied to a server, and the method comprises the following steps: acquiring a plot description text uploaded by a client; splitting the plot description text to obtain a plurality of split texts, and determining a split prompt word corresponding to each split text; extracting at least one key content label and a feature text corresponding to each key content label from the plot description text, and determining a label prompt word corresponding to each feature text; and according to the split cue word and the label cue word, carrying out text generation processing to obtain a plurality of split illustrations, and according to the plurality of split illustrations, carrying out editing to generate a target video. According to the technical scheme, the process from splitting the plot description text and extracting the label to generating the illustration can be automatically completed, a user does not need to manually edit and process, the efficiency of converting the copywriting into the video is improved, and the user requirement of producing a large number of videos can be met.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and in particular, to a method, apparatus, computer device, computer-readable storage medium, and computer program product for generating a video based on text. Background Art

[0002] With the rise of short videos, videos have gradually become one of the most popular and effective forms of content promotion. In terms of marketing, videos can attract users' attention, convey information, and achieve the purpose of publicity.

[0003] However, creating high-quality videos requires professional filming and a lot of time for editing. In order to attract user traffic, many creators choose to produce videos by converting text into videos. However, the traditional video generation method has low efficiency in the process of converting text into videos and cannot meet the needs of users who need to produce a large number of videos.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for generating a video based on text to solve or alleviate one or more of the technical problems raised above.

[0006] One aspect of an embodiment of the present application provides a method for generating a video based on text, which is applied to a server, and the method includes: Get the plot description text uploaded by the client; Splitting the plot description text to obtain a plurality of storyboard texts, and determining a storyboard prompt word corresponding to each storyboard text; Extracting at least one key content tag and a feature text corresponding to each key content tag from the plot description text, and determining a tag prompt word corresponding to each feature text; According to the storyboard prompt words and the label prompt words, text image processing is performed to obtain multiple storyboard matching pictures, and according to the multiple storyboard matching pictures, editing is performed to generate a target video.

[0007] Optionally, the key content tags include character tags and scene tags; and extracting at least one key content tag and feature text corresponding to each key content tag from the plot description text includes: When it is identified that the plot description text contains a character name, the character name is extracted as a character tag, and relevant description information corresponding to the character name is extracted to generate a feature text corresponding to the character tag; and / or, When it is identified that there are multiple repeated place names in the plot description text, the place names are extracted as scene labels, and relevant description information corresponding to the place names is extracted to generate feature text corresponding to the scene labels.

[0008] Optionally, the step of obtaining the plot description text uploaded by the client includes: Controlling the display of the video creation page on the graphical user interface provided by the client; A plot setting operation on the video creation page is received, and the plot description text is extracted from the plot setting operation.

[0009] Optionally, before the step of performing text image processing according to the storyboard prompt words and the label prompt words to obtain a plurality of images, the method further includes: Control the client to jump from the video creation page to a first page; wherein the content displayed on the first page includes the multiple storyboard texts and a storyboard update control; In response to an editing operation on a target text among the plurality of storyboard texts, determining first update information of the target text; In response to a triggering operation acting on the update storyboard control, the target text is adjusted according to the first update information.

[0010] Optionally, after the step of controlling the client to jump from the displayed video creation page to the first page, the method further includes: Generate a basic character image according to a character tag in the key content tag and a feature text corresponding to the character tag; And control the display of the basic character image on the first page.

[0011] Optionally, the content displayed on the first page further includes the at least one key content tag and a save tag control, and the method further includes: In response to a triggering operation on a target tag in the at least one key content tag, controlling display of a feature text corresponding to the target tag on the first page; In response to an editing operation on the characteristic text corresponding to the target tag, determining second update information of the characteristic text corresponding to the target tag; In response to a triggering operation acting on the save tag control, a characteristic text corresponding to the target tag is adjusted according to the second update information.

[0012] Optionally, the first page further includes a new label control, and the method further includes: In response to a triggering operation on the new label control, controlling display of a label setting component in the first page; In response to an input operation acting on the label setting component, extracting label configuration information from the input operation; A new tag is generated according to the tag configuration information, and the new tag is controlled to be displayed on the first page.

[0013] Optionally, the first page further includes a second page switching control, and the process of performing text-based image processing according to the storyboard prompt words and the label prompt words to obtain a plurality of storyboard illustrations includes: In response to a trigger operation on the second page switching control, the client is controlled to jump from the first page to the second page; wherein the content displayed on the second page includes a generate illustration control corresponding to each storyboard file; In response to a trigger operation acting on the generate illustration control, a plurality of storyboard illustrations are obtained by performing text-generated image processing according to the storyboard prompt words and the label prompt words.

[0014] Optionally, the method further comprises: Controlling display of the action skeleton setting component in the second page; In response to a setting operation acting on the action skeleton setting control, determining image action parameters; The response acts on the trigger operation of the generate picture control, and performs text-generated picture processing according to the storyboard prompt word and the label prompt word to obtain multiple storyboards, including: In response to a trigger operation on the generate picture control, a plurality of storyboard pictures are obtained by performing text-generated picture processing according to the image action parameters, the storyboard prompt words and the label prompt words.

[0015] Optionally, the method further comprises: Controlling display of the plurality of storyboard texts in the second page; In response to a third editing operation on a target text among the plurality of storyboard texts, the target text is adjusted according to the third editing operation.

[0016] Optionally, the method further comprises: Controlling display of the storyboard prompt words and label prompt words on the second page; The storyboard prompt words and label prompt words are converted into standard prompt words according to a preset format; and the standard prompt words are displayed on the second page.

[0017] Optionally, the method further comprises: Controlling display of a dubbing control corresponding to each storyboard text on the second page; In response to a triggering operation of a dubbing control corresponding to a first text in the plurality of storyboard texts, controlling the display of a dubbing sound effect component corresponding to the first text in the second page; wherein the dubbing sound effect component includes a plurality of timbres; In response to a selection operation on a first timbre among the plurality of timbres, a target dubbing corresponding to the first text is generated according to the first timbre.

[0018] Optionally, the method further comprises: Controlling display of audition controls and setting dubbing controls corresponding to the first dubbing on the second page; In response to a triggering operation on the audition control, controlling the client to play the first dubbing; In response to a setting operation on the setting dubbing control, a second timbre is determined according to the setting operation, and a target dubbing corresponding to the first text is generated according to the second timbre.

[0019] Optionally, the method further comprises: Controlling display of the plurality of storyboards on the second page; In response to a selection operation on the plurality of storyboards, determining a plurality of target images from the plurality of storyboards; The plurality of target images and a target dubbing corresponding to each target image are exported.

[0020] Another aspect of an embodiment of the present application provides a device for generating a video based on text, which is applied to a server, and the device includes: The plot description text acquisition module is used to obtain the plot description text uploaded by the client; A text splitting module is used to split the plot description text to obtain multiple storyboard texts, and determine the storyboard prompt words corresponding to each storyboard text; A tag extraction module, used to extract at least one key content tag and a feature text corresponding to each key content tag from the plot description text, and determine a tag prompt word corresponding to each feature text; The text image module is used to process the text image according to the storyboard prompt words and the label prompt words to obtain multiple storyboard pictures, and to edit the multiple storyboard pictures to generate a target video.

[0021] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating video based on text as described above.

[0022] Another aspect of an embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, the method for generating a video based on text as described above is implemented.

[0023] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method for generating a video based on text as described above.

[0024] The above technical solution adopted in the embodiment of the present application may include the following advantages: by using the splitting and label extraction processing of the plot description text as the basis for the subsequent generation of pictures for each storyboard text, wherein the text splitting processing completes the segmentation of the content into storyboard pictures, and the label extraction processing extracts the key content labels such as characters and scenes in the entire content, and the key content labels can be used as the basis for the subsequent generation of content consistency. When the corresponding label characters or scenes and other key content are selected to generate pictures, the consistency of the characters can be maintained and the quality of the text pictures can be improved. Moreover, the process from splitting the plot description text and label extraction processing to generating pictures can be completed automatically, without the need for manual editing and processing by the user, which improves the efficiency of converting text to video and can meet the needs of users who need to produce videos in large quantities. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0026] Figure 1 The operating environment diagram of the method for generating a video based on text according to the first embodiment of the present application is schematically shown; Figure 2 The flowchart of the method for generating a video based on text according to the first embodiment of the present application is schematically shown; Figure 3 A schematic diagram of a video creation page according to the first embodiment of the present application is schematically shown; Figure 4 A block diagram schematically shows a device for generating a video based on text according to the second embodiment of the present application; and Figure 5The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0028] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0029] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0030] First, the following terms are explained: AIGC: The full name is Artificial Intelligence Generated Content. AIGC is a new production paradigm that uses artificial intelligence technology to create and generate content in different forms such as text, pictures, audio, video, and code.

[0031] Large Language Model: Large Language Model, referred to as LLM, refers to an advanced artificial intelligence model with large-scale parameters and training data that can understand and generate natural language.

[0032] Stable Diffusion: is a deep learning-based text-based graph model that supports generating new images by using prompts to describe elements to include or omit.

[0033] Prompt: A prompt message used to guide users to enter information or perform certain operations.

[0034] Negative Prompt: refers to the emphasis on unwanted or avoided elements, tone, or wording in a message, instruction, or input. It is used to guide the recipient (such as an AI model, user, etc.) to avoid producing or recognizing specific content.

[0035] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: Since creating high-quality videos requires professional shooting and a lot of time for editing, in order to attract user traffic, many creators choose to produce videos by converting text into videos. However, the traditional video generation method has low efficiency in the process of converting text into videos and cannot meet the needs of users who need to produce a large number of videos.

[0036] To this end, the embodiment of the present application provides a technical solution for generating videos based on text. In this technical solution, the plot description text uploaded by the client is obtained; the plot description text is split and processed to obtain multiple storyboard texts, and the storyboard prompt words corresponding to each storyboard text are determined; at least one key content label and the feature text corresponding to each key content label are extracted from the plot description text, and the label prompt words corresponding to each feature text are determined; according to the storyboard prompt words and the label prompt words, the text is processed to obtain multiple storyboard pictures, and the target video is edited according to the multiple storyboard pictures. By splitting the plot description text and extracting labels as the basis for the subsequent generation of pictures for each storyboard text, the text splitting process completes the division of the content into storyboard pictures, and the label extraction process extracts the key content labels such as characters and scenes in the entire content. The key content labels can be used as the basis for the subsequent generation of content consistency. When the key content such as corresponding label characters or scenes are selected to generate pictures in the subsequent generation, the consistency of the characters can be maintained and the quality of the text pictures can be improved. Moreover, the process from splitting the plot description text and extracting labels to generating pictures can be completed automatically, without the need for manual editing by users, which improves the efficiency of converting text to video and can meet the needs of users who need to produce a large number of videos. See below for details.

[0037] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0038] like Figure 1 As shown, the environment diagram includes a service platform 2, a network 4, and a client 6, wherein: The service platform 2 may be composed of a single or multiple computing devices. The multiple computing devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as simulations of computer systems, operating systems, servers, etc. The computing device may load a virtual machine based on a virtual image and / or other data defining specific software (e.g., operating system, dedicated application, server) for simulation. As the demand for different types of processing services changes, different virtual machines may be loaded and / or terminated on one or more computing devices. A hypervisor may be implemented to manage the use of different virtual machines on the same computing device.

[0039] The service platform 2 may be configured to communicate with the client 6 or the like via a network 4. The network 4 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 4 may include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof, and the like, or wireless links, such as cellular links, satellite links, Wi-Fi links, and the like.

[0040] The service platform 2 can provide storage, reading, writing, querying, deleting and other services, such as providing the client with a service for generating videos based on text.

[0041] The client 6 may be an electronic device running an operating system such as Windows, Android™ or iOS, such as a smart phone, a tablet device, a laptop computer, a virtual reality device, a gaming device, a set-top box, a vehicle terminal, or a smart TV. Based on the above operating system, various applications may be run, such as generating videos based on text.

[0042] The client 6 may provide / configure a user access page for manipulating the service platform 2 or uploading an object, etc.

[0043] It should be noted that the above devices are exemplary, and the number and type of devices are adjustable in different scenarios or according to different needs.

[0044] The following uses the service platform 2 as the execution subject to introduce the technical solution of the present application through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be interpreted as being limited to the embodiments described here.

[0045] Embodiment 1 Figure 2 The flowchart of the method for generating a video based on text according to the first embodiment of the present application is schematically shown.

[0046] like Figure 2 As shown, the method for generating a video based on text is applied to the server and may include steps S202 to S208, wherein: Step S202, obtaining the plot description text uploaded by the client; Specifically, a graphical user interface may be provided on the client, and the content displayed by the graphical user interface may include a video creation page, through which the user may input plot description text, and the plot description text is the text needed to guide the generation of the video.

[0047] Step S204, splitting the plot description text to obtain a plurality of storyboard texts, and determining the storyboard prompt words corresponding to each storyboard text; In this embodiment, after receiving the plot description text uploaded by the client, the server can split the plot description text to obtain multiple storyboard texts, wherein each storyboard text can be used to generate a storyboard screen. Specifically, by identifying the line break characters in the plot description text, each line break character can be used as a paragraph segmentation basis to sequentially generate continuous storyboard texts starting from storyboard No. 01. At the same time, the split storyboard text can also be displayed on the graphical user interface on the client, and when the number of content displayed in one line is exceeded, it is displayed in the line (but without line break characters).

[0048] Furthermore, after the plot description text is split to obtain a plurality of storyboard texts, a corresponding storyboard prompt word is determined for each storyboard text. For example, each storyboard text is analyzed to determine that the storyboard prompt word can be used as Prompt prompt information for generating a ligand map.

[0049] Step S206, extracting at least one key content tag and a feature text corresponding to each key content tag from the plot description text, and determining a tag prompt word corresponding to each feature text; In this embodiment, in order to ensure the consistency of the generated key content (such as characters and scenes, etc.), at least one key content tag is extracted from the plot description text. Among them, the key content tags can include character tags and scene tags, etc. The character tags can be obtained by identifying the names of characters mentioned in the text (extracted when they appear), and the scene tags can be obtained by identifying the names of places mentioned in the text (extracted when they appear repeatedly). At the same time, the relevant descriptions of the characters or places are used as the feature text corresponding to the tag, and the tag prompt words (such as characters: gender / country / age / occupation / appearance / dress / hairstyle / hair color, etc.) are extracted from the feature text.

[0050] Step S208, performing text-based image processing according to the storyboard prompt words and the label prompt words to obtain a plurality of storyboard matching pictures, and performing editing according to the plurality of storyboard matching pictures to generate a target video.

[0051] In this embodiment, storyboard prompt words and label prompt words are input as Prompt prompt information into a deep learning model for text graphics, such as a Stable Diffusion model, so that the model can generate storyboard illustrations corresponding to each storyboard text based on the guidance of the Prompt.

[0052] Specifically, the server displays a generated picture control on the client's graphical user interface. Each time the generated picture control is clicked, a storyboard picture is generated and placed in the candidate picture pool. For each storyboard, a maximum of 4 pictures are retained in the candidate pool. If the number exceeds the limit, the earliest generated picture is squeezed out, and the user can select the final required picture for editing to generate the target video.

[0053] In an optional embodiment of the present application, the key content tags include character tags and scene tags; the step of extracting at least one key content tag and feature text corresponding to each key content tag from the plot description text includes: When it is identified that the plot description text contains a character name, the character name is extracted as a character tag, and relevant description information corresponding to the character name is extracted to generate a feature text corresponding to the character tag; and / or, When it is identified that there are multiple repeated place names in the plot description text, the place names are extracted as scene labels, and relevant description information corresponding to the place names is extracted to generate feature text corresponding to the scene labels.

[0054] In this embodiment, the key content tags may include character tags and scene tags, wherein the character tags may be determined by the character name, and the scene tags may be determined by the place name. In addition, the key content tags may also include pet characters or physical objects that appear multiple times in the text, etc., which are not specifically limited in this embodiment of the application.

[0055] Specifically, by identifying the plot description text, if it is identified that the plot description text contains a character name, the character name is extracted as a character tag, and the relevant description information corresponding to the character name is extracted to generate a feature text corresponding to the character tag. The relevant description information corresponding to the character name may include: gender / country / age / occupation / appearance / dress / hairstyle / hair color, etc. In the case of identifying that there are multiple repeated place names in the plot description text, the place name is extracted as a scene tag, and the relevant description information corresponding to the place name is extracted to generate a feature text corresponding to the scene tag. The relevant description information corresponding to the place name may include information such as location, building, color, day and night, and season.

[0056] In an optional embodiment of the present application, the step of obtaining the plot description text uploaded by the client includes: Controlling the display of a video creation page on a graphical user interface provided by a client; receiving a plot setting operation on the video creation page, and extracting the plot description text from the plot setting operation.

[0057] In this embodiment, the server controls the display of the video creation page on the graphical user interface provided by the client, thereby receiving the plot setting operation on the video creation page to extract the plot description text from the plot setting operation.

[0058] Specifically, the video creation page may include a work name input box, a video ratio selection box, a picture style selection box, a content type selection box, and a content full text input box. Among them, the work name input box is used to enter the name of the new project. It is a required field, in text format, with a word count of less than 40 characters, and cannot be repeated with the created project. The video ratio selection box is used to select the ratio of the generated video. It is a required option. The options supported are: 1:1, 16:9, 4:3, 3:4, 9:16, etc., which can be set according to the video specifications. The picture style selection box is used to select the style of the generated content, that is, what model is actually used to generate the picture. It is a required option. For example, the style is: two-dimensional animation style, real-life film and television style, and sketch style, etc. Different styles can also be added as the product iterates. The content type selection box is used to select the type of input text. Different types correspond to different GPT character prompt words for subsequent processing, which is convenient for GPT to better understand the project. It is a required option. For example, the content type is: novel, script, essay, and joke. The full-text input box is used to input the entire story content used to generate the video. It will be used as the raw data for generating the content and enter the subsequent video production process. It is a required field. The input content is plain text and supports line breaks. The number of characters to be input can also be limited, such as 10,000 characters.

[0059] As an example, a video creation page like Figure 3 As shown, users can Figure 3 The plot setting operation is performed on the page. After the user completes the input operation and clicks the Create Work button, the client uploads the collected plot setting operations to the server. The uploaded plot setting operations may include the content input and selected by the user on the page.

[0060] In an optional embodiment of the present application, before the step of performing Wenshengtu processing according to the storyboard prompt words and the label prompt words to obtain a plurality of images, the method further includes: Control the client to jump from the video creation page to the first page; wherein the content displayed on the first page includes the multiple storyboard texts and the update storyboard control; in response to an editing operation on a target text among the multiple storyboard texts, determine first update information of the target text; in response to a trigger operation acting on the update storyboard control, adjust the target text according to the first update information.

[0061] In this embodiment, the server controls the client to jump from the video creation page to the first page, wherein the first page is a page for text and tag management, and the content displayed on the first page includes multiple storyboard texts and update storyboard controls. The user can edit the split storyboard texts in the first page. The server determines the first update information of the target text by responding to the editing operation on the target text in the multiple storyboard texts, and responds to the trigger operation acting on the update storyboard control, and adjusts the target text according to the first update information. Among them, the trigger operation can include single-click, double-click or long-press operations, and the embodiment of the present application does not specifically limit this.

[0062] Specifically, the first page may include the following functions: 1. Storyboard text display / editing function, which is used to identify the created basic text, starting from storyboard No. 01, and display the storyboards corresponding to all text contents; the text part can be modified directly, and after rewriting, you need to manually click the update storyboard button to record and store it, such as Figure 4 As shown. 2. The smart segmentation function is used to automatically re-segment the current storyboard; after clicking the smart segmentation control, the original input text content will be used to segment the text according to its paragraphs, period delimiters, and actual content; after segmentation, each paragraph of text content is within 200 words (normally within 100 words, with an upper limit of 200, and forced segmentation when exceeding); the order of the storyboards remains consistent with the original text and will not be rewritten. The undo function is used to undo the previously edited text, which can be undone before clicking the update storyboard control to save; the number of undo steps is 20, and it is not stored. After re-entering the page, the number of steps is cleared and the recording starts again.

[0063] In an optional embodiment of the present application, after the step of controlling the client to jump from the displayed video creation page to the first page, the method further includes: Generate a basic character image according to the character tag in the key content tag and the feature text corresponding to the character tag; and control the display of the basic character image on the first page.

[0064] In this embodiment, for the extracted character tags, a basic image is directly generated according to the description prompt words of the character for preview, and is also used as a character backing image when subsequently generating illustrations.

[0065] In an optional embodiment of the present application, the content displayed on the first page further includes the at least one key content tag and a save tag control, and the method further includes: In response to a trigger operation on a target tag in the at least one key content tag, control the display of a feature text corresponding to the target tag on the first page; in response to an edit operation on the feature text corresponding to the target tag, determine second update information of the feature text corresponding to the target tag; in response to a trigger operation acting on the save tag control, adjust the feature text corresponding to the target tag according to the second update information.

[0066] In this embodiment, the user can edit and manage the identified key content tags on the first page. In response to a trigger operation on a target tag in at least one key content tag, the feature text corresponding to the target tag is controlled to be displayed on the first page; in response to an edit operation on the feature text corresponding to the target tag, the second update information of the feature text corresponding to the target tag is determined; in response to a trigger operation on the save tag control, the feature text corresponding to the target tag is adjusted according to the second update information.

[0067] Specifically, the first page can include the following functions for tag management: 1. Automatically extract and create tags. When creating a project, it can identify the original text and extract key content tags. For example, character recognition: name / gender / country / age / occupation / appearance / clothing / hairstyle / hair color, etc.; scene recognition: location / building / color scheme / day and night / season, etc.

[0068] 2. Character tag management function. Select a character tag and you will see the specific prompt word content of the tag. At the same time, the tag word will be highlighted in the text on the left. For existing character tags, you can directly modify the character description content; after modification, click Regenerate Image to generate a character preview image (mat); input specifications are: Name: Enter / modify the character name. When the same character has multiple names, use "," to separate the character description, and enter the character's characteristic prompt word (positive) Negative prompt word, mark [Negative] before the prompt word.

[0069] 3. Scene / custom tag management function. Select a tag and you will see the specific prompt word content of the tag. At the same time, the tag word will be highlighted in the text on the left. For existing scene / custom tags, you can directly modify the description content; after modification, after updating the tag, the relevant prompt words when using this scene / custom tag in subsequent generated graphs will be changed. The input specifications are: Tag description, enter the scene / custom feature prompt word (positive) Negative prompt word, mark [Negative] before the prompt word.

[0070] 4. Create a character function. For unrecognized characters, you can also manually create them. Click the [+] button next to the character to open the Create Character window. After entering the character-related information, click the Create button to complete the creation. At this time, a preview of the character will be automatically generated in the character window.

[0071] 5. Create scene / custom tag function. For scenes / custom tags that are not recognized, you can also manually create them. Click the [+] button next to the scene / custom tag to open the Create scene / custom tag window. After entering the relevant description of the scene / custom tag, click the Create button to complete the creation. At this time, the tag content will be displayed in the corresponding tag window. The input specifications are: Name: Enter / modify the character's name. When the same character has multiple names, use "," to separate the character description, and enter the character's characteristic prompt word (positive) negative prompt word, and mark [Negative] before the prompt word.

[0072] In an optional embodiment of the present application, the first page further includes a new label control, and the method further includes: In response to a trigger operation on the new label control, a label setting component is controlled to be displayed on the first page; in response to an input operation on the label setting component, label configuration information is extracted from the input operation; a new label is generated according to the label configuration information, and the new label is controlled to be displayed on the first page.

[0073] In this embodiment, the server controls the display of a label setting component on the first page by responding to a trigger operation on a new label control, and the label setting component is used to manually input a custom label. By responding to an input operation on the label setting component, label configuration information is extracted from the input operation, and a new label is generated according to the label configuration information, and the new label is controlled to be displayed on the first page for the user to view.

[0074] In an optional embodiment of the present application, the first page further includes a second page switching control, and the process of performing text image processing according to the storyboard prompt word and the label prompt word to obtain a plurality of storyboard matching pictures includes: In response to a trigger operation on the second page switching control, the client is controlled to jump from the first page to the second page; wherein the content displayed on the second page includes a generate illustration control corresponding to each storyboard file; in response to a trigger operation on the generate illustration control, a plurality of storyboard illustrations are obtained by performing text-generated image processing according to the storyboard prompt words and the label prompt words.

[0075] In this embodiment, the server controls the client to jump from the first page to the second page by responding to the trigger operation on the second page switching control; wherein the content displayed on the second page includes a generate picture control corresponding to each storyboard file. In response to the trigger operation on the generate picture control, a plurality of storyboard pictures are obtained by performing text-generated image processing according to the storyboard prompt word and the label prompt word. Specifically, each storyboard file will generate a corresponding storyboard picture, and the user can batch select multiple storyboard files and click the generate picture control to realize batch picture generation.

[0076] In an optional embodiment of the present application, the method further includes: Controlling the display of an action skeleton setting component in the second page; and determining image action parameters in response to a setting operation acting on the action skeleton setting control.

[0077] In this embodiment, the server controls the display of an action skeleton setting component in the second page, and the action skeleton setting component is used to control the action of the character in the generated picture. The image action parameters are determined by responding to the setting operation acting on the action skeleton setting control. Specifically, it supports uploading a character action reference picture, the picture size is limited to 5mb, and a clear full-body / half-body action picture of the character.

[0078] In an optional embodiment of the present application, the response acts on the trigger operation of the generate picture control, and performs text image processing according to the storyboard prompt word and the label prompt word to obtain multiple storyboards, including: In response to a trigger operation on the generate picture control, a plurality of storyboard pictures are obtained by performing text-generated picture processing according to the image action parameters, the storyboard prompt words and the label prompt words.

[0079] In this embodiment, by responding to the trigger operation acting on the picture generation control, the text image processing is performed according to the image action parameters, the storyboard prompt words and the label prompt words to obtain multiple storyboard pictures. Specifically, when using the action skeleton function, the controlnet multiple control input capabilities can be used to realize the controlled generation of pictures within the prompt word range, taking into account the characters, actions, and prompt word content.

[0080] In addition, this embodiment also provides functions of enlarging pictures and repairing characters, respectively applying the high-definition enlargement capability of SD basic pictures and the controlnet model repair function to optimize the drawing results; after the picture is enlarged, the maximum width and height (long side) of the picture are set to preset values, such as 1080; when the picture enlargement operation is not performed, the video result is generated using a small-size picture to generate a video.

[0081] The generation of pictures for each storyboard text is a process of production-card drawing, so it is often necessary to generate pictures multiple times. Each time you click the Generate This Shot control, a storyboard picture is generated and placed in the candidate picture pool. For each storyboard, a maximum of 4 pictures are retained in the candidate pool. If the number exceeds, the earliest generated picture will be squeezed out (the pictures in the candidate picture pool are generated and sorted in the order in which the user generates the pictures each time). The user can click on any picture in the candidate picture pool area to switch the picture of this shot to the selected picture.

[0082] In an optional embodiment of the present application, the method further includes: Controlling display of the plurality of storyboard texts in the second page; responding to a third editing operation on a target text in the plurality of storyboard texts, and adjusting the target text according to the third editing operation.

[0083] In this embodiment, the user can also edit and manage the storyboard text in the second page. The server controls the display of multiple storyboard texts in the second page, responds to the third editing operation on the target text in the multiple storyboard texts, and adjusts the target text according to the third editing operation.

[0084] Specifically, the second page may include the following functions: 1. Mirror: used to display the mirror number. Add a check button before the mirror number, which is not selected by default. Support multiple mirror numbers to activate batch operation. Specifications: automatic generation, automatic increment, after inserting / deleting a mirror, automatically adjust the subsequent mirror number.

[0085] 2. Text description: The text is displayed in segments after the above logic processing is completed, which is consistent with the text processing. Supports inserting / deleting storyboards. After the operation, the entire line of storyboards is adjusted (insert a line of empty content storyboards below or the storyboards in this line disappear). The specifications are: storyboard number and text description, which are consistent with the previously disassembled text storyboard number.

[0086] 3. Tags: The tag column displays all tags and highlights the tag content related to the current storyboard. The specifications are: from top to bottom: characters - scenes - custom tags.

[0087] 4. Inference description: Inference description is the Chinese grammar prompt word generated based on the text and label description; label content is prioritized and highlighted. Specifications: Inference keywords can be generated repeatedly, that is, they can be adjusted based on the previous text. Click on the inference keyword to re-execute the previous operation of generating prompt word description based on text description and label description.

[0088] 5. Screen prompt words: Standard prompt words obtained by translating the inference description according to SD grammar. Specifications: Automatically generated, updated along with the inference keyword generation.

[0089] In an optional embodiment of the present application, the method further includes: Controlling the display of the storyboard prompt words and label prompt words on the second page; converting the storyboard prompt words and label prompt words according to a preset format to obtain standard prompt words; and displaying the standard prompt words on the second page.

[0090] The preset format may be a data format recognizable by the text graph model, such as English format. Specifically, the storyboard prompt words and label prompt words displayed on the second page may be in Chinese format, and the server converts the storyboard prompt words and label prompt words according to the preset format to obtain standard prompt words, and displays the standard prompt words on the second page.

[0091] In an optional embodiment of the present application, the method further includes: Control the display of dubbing controls corresponding to each storyboard text in the second page; in response to a trigger operation of the dubbing control corresponding to a first text among the multiple storyboard texts, control the display of a dubbing sound effect component corresponding to the first text in the second page; wherein the dubbing sound effect component includes a plurality of timbres; in response to a selection operation of a first timbre among the plurality of timbres, generate a target dubbing corresponding to the first text according to the first timbre.

[0092] In this embodiment, the plot description text can also be dubbed. The server controls the display of the dubbing control corresponding to each storyboard text on the second page; in response to the triggering operation of the dubbing control corresponding to the first text in the multiple storyboard texts, controls the display of the dubbing sound effect component corresponding to the first text on the second page; wherein the dubbing sound effect component includes multiple timbres. Furthermore, in response to the selection operation of the first timbre in the multiple timbres, the target dubbing corresponding to the first text is generated according to the first timbre.

[0093] In an optional embodiment of the present application, the method further includes: Control the display of the audition control and the dubbing setting control corresponding to the first dubbing in the second page; respond to the trigger operation on the audition control, control the client to play the first dubbing; respond to the setting operation on the dubbing setting control, determine the second timbre according to the setting operation, and generate the target dubbing corresponding to the first text according to the second timbre.

[0094] In this embodiment, after selecting the sound effect to be used, the sound effect of the current dubbing is displayed at the dubbing location. At the same time, the audition control and the setting dubbing control are activated below, and the dubbing that has been selected and created can be modified. Specifically, the server determines the second timbre according to the setting operation by responding to the setting operation acting on the setting dubbing control, and generates the target dubbing corresponding to the first text according to the second timbre.

[0095] It should be noted that in the specific implementation, dubbing and storyboard illustrations can be generated in sequence at will. The product provides a sound library for creators to choose the sound timbre to use. The following basic settings can also be made for the selected dubbing: speed adjustment (0.5-2) times; intonation amplitude setting, volume adjustment, (0.1-10) times db; the playback time of the dubbing corresponding to each storyboard text is set, which is determined by the duration of the dubbing, that is, the longer the text, the longer the dubbing playback time, and the longer the playback time of the storyboard illustration.

[0096] In an optional embodiment of the present application, the method further includes: Controlling the display of the plurality of storyboards in the second page; determining a plurality of target images from the plurality of storyboards in response to a selection operation on the plurality of storyboards; and exporting the plurality of target images and a target dubbing corresponding to each target image.

[0097] In this embodiment, after generating the illustrations and dubbing, the most suitable storyboard illustrations can be further selected to produce the final video. Specifically, the server controls the display of the generated multiple storyboard illustrations on the second page, and then, in response to the selection operation on the multiple storyboard illustrations, determines multiple target images from the multiple storyboard illustrations, and derives the multiple target images and the target dubbing corresponding to each target image.

[0098] After the storyboard pictures and dubbing are completed, in theory, all the conditions required for exporting the video have been met. This embodiment provides the ability to export as a multi-track editing file, that is, to export multiple selected target images to the video track, and export the target dubbing corresponding to each target image to the audio track, and directly render the data in the video track and audio track to export the video.

[0099] In the specific implementation, you can also set the export specifications as follows: 1. Check the status. The storyboard can only be exported when the current status meets the requirements. Minimum export requirements: a. At least one complete storyboard (including inference results, prompt words, generated storyboards, and dubbing); b. The project status is normal; 2. Complete the export settings a. Select special effects: You can select various special effects types currently supported by Jianying (move left and right, move up and down, zoom in and out, zoom in and out, zoom in and out, and random combinations of the above special effects) b. Set the special effects: including amplitude (magnification setting 1.0-2.0) and movement amplitude (0-1), and pause time between shots (0-1000ms).

[0100] The above settings are the parameters when exporting to the clipping track.

[0101] After entering the exporting state, no matter what operation is performed, this export is valid. After the background completes the export processing, it is stored in this project. The user can re-enter the page and download the completed draft of the editing. After the normal export is completed, the pop-up window status shows that the draft has been synthesized, and the user can download the draft locally.

[0102] In order to facilitate understanding of the solution of this embodiment, the solution of this embodiment is further explained as follows. By using the ability of a large language model to process text, SD prompt words are generated for the previously segmented text. The relevant concepts are introduced as follows: 1. Mission Use natural language to tell the big language model the topic of the prompt to be generated. The task of the big language model is to imagine a complete picture based on this topic and then convert it into a detailed, high-quality prompt, so that Stable Diffusion can generate high-quality images.

[0103] 2. Prompt concept 1. A complete prompt consists of two parts: "Prompt: " and "Negative Prompt: ".

[0104] 2. Prompt is used to describe the image. It consists of common words and uses English half-width "," as separators.

[0105] 3. Pegative Prompt is used to describe content that you do not want to appear in the generated image.

[0106] 4. Each word or phrase separated by "," is called a tag. So Prompt and Negative Prompt are composed of a series of tags separated by ",".

[0107] 3. () and [] syntax The equivalent way to adjust keyword strength is to use () and []. (keyword) increases the strength of the tag by 1.1 times, the same as (keyword:1.1), up to a maximum of three layers. [keyword] decreases the strength by 0.9 times, the same as (keyword:0.9).

[0108] IV. Prompt format requirements The following will explain the steps of generating a Prompt. Here, the Prompt can be used to describe a person, a landscape, an object, or an abstract digital art picture. Reasonable but no less than 5 details of the picture can be added as needed.

[0109] A. Prompt requirements 1. The output Stable Diffusion prompt starts with “Prompt:”.

[0110] 2. The prompt content includes the main body of the picture, material, additional details, image quality, artistic style, color tone, lighting, etc., but the output prompt cannot be segmented. For example, segmented descriptions such as "medium:" are not required, and cannot contain ":" and ".".

[0111] 3. Subject: A brief description of the subject, such as A girl in a garden, and a summary of the subject details (the subject can be a person, event, object, or scene) and the core content of the picture. This part is generated based on the theme entered each time, and more reasonable details related to the theme can be added.

[0112] 4. For the character theme, it is very important to describe the character's eyes, nose, and lips, such as 'beautiful detailed eyes, beautiful detailed lips, extremely detailed eyes and face, long eyelashes', to avoid Stable Diffusion randomly generating deformed facial features. You can also describe the character's appearance, emotions, clothes, posture, perspective, action, background, etc. In the character attributes, 1girl means one girl, and 2girls means two girls.

[0113] 5. Material: The material used to create the artwork. Examples include illustrations, paintings, 3D renderings, and photography. Medium has a strong effect because a single keyword can drastically change the style.

[0114] 6. Additional details: Scene details, or character details, describe the details of the picture to make the image look more substantial and reasonable. This part is optional, and attention should be paid to the overall harmony of the picture, and it should not conflict with the theme.

[0115] 7. Image quality: This part of the content always has a specific code at the beginning, such as "(best quality, 4k, 8k, highres, masterpiece: 1.2), ultra-detailed, (realistic, photorealistic, photo-realistic: 1.37)", which is a sign of high quality. Other commonly used tags to improve quality can be added according to the needs of the theme, such as: HDR, UHD, studio lighting, ultra-fine painting, sharp focus, physically-based rendering, extreme detail description, professional, vivid colors, bokeh, etc.

[0116] 8. Artistic style: This part describes the style of the image. Adding the right artistic style can improve the generated image. Commonly used artistic styles include: portraits, landscape, horror, anime, sci-fi, photography, concept artists, etc.

[0117] 9. Color tone: Color, control the overall color of the picture by adding color.

[0118] 10. Lighting: The lighting effect of the overall picture.

[0119] B. Negative prompt requirements 1. The negative prompt part starts with "Negative Prompt:". You can add anything you want to avoid appearing in the image after "NegativePrompt:".

[0120] 2. In any case, the negative prompt must contain the following content: "nsfw, (low quality, normal quality, worst quality, jpeg artifacts), cropped, monochrome, lowres, lowsaturation, ((watermark)), (white letters)".

[0121] 3. If the topic is related to characters, the output needs to add an additional negative prompt related to the characters, which is: "skin spots, acnes, skinblemishes, age spot, mutated hands, mutated fingers, deformed, bad anatomy, disfigured, poorly drawn face, extralimb, ugly, poorly drawn hands, missing limb, floating limbs, disconnected limbs, out of focus, long neck, long body, extrafingers, few fingers, (multi nipples), bad hands, signature, username, bad feet, blurry, bad body".

[0122] C. Restrictions: 1. Tag content is described by English words or phrases, and is not limited to the input words. Note that it can only contain keywords or phrases.

[0123] 2. Be careful not to output sentences or provide any explanations.

[0124] 3. The number of tags is limited to 40, and the number of words is limited to 60.

[0125] 4. Do not put quotation marks ("") around the tag.

[0126] 5. Use English half-width "," as separator.

[0127] 6. Tags are arranged in order of importance from high to low.

[0128] 7. The input topic may be described in Chinese, and the output Prompt and Negative Prompt are only in English.

[0129] Based on the above method, each line of text is processed to generate the corresponding storyboard prompt words for each line of text (and translated into English).

[0130] D. Label prompt words For the previously created labels, generate prompt words for the description content of the labels. The method is the same as the method for generating prompt words for storyboard text. For character labels, after generating prompt words as described above, when using character labels to generate pictures, call the Reference Only capability in the controlnet plug-in to maintain the consistency of generated characters.

[0131] E. Batch Processing Since storyboards are generated in batches, batch processing is supported for multiple storyboards based on the selected storyboards. After selecting the selection box in front of any storyboard, a batch processing toolbar will be displayed below the storyboard, providing the following batch processing functions: 1. Batch reasoning prompt words: After clicking, the prompt word generation process will be re-executed for the selected storyboards, overwriting the prompt words in the current box (but not storing them).

[0132] 2. Batch recognition of tags: Re-recognize the tags in the current text paragraph.

[0133] 3. Batch generate storyboards: Check the prompt words of the selected storyboards. If there are complete prompt words in any storyboard, generate the storyboard pictures of the corresponding storyboards.

[0134] 4. Add dubbing in batches: Apply the dubbing settings of the currently selected storyboard (based on the dubbing settings of the front storyboard) to dub the selected text paragraphs.

[0135] 5. Batch enlarge the pictures: When the selected storyboard has already generated storyboard pictures, batch enlarge the pictures of the selected storyboard.

[0136] 6. Batch delete storyboards: Delete all selected storyboards.

[0137] At the same time, the specification setting of the above batch processing function can be: after clicking, the generation process button is grayed out, and other batch operations can only be continued after the processing is completed.

[0138] In this embodiment, the plot description text is split and the label extraction process is used as the basis for the subsequent generation of pictures for each storyboard text, wherein the text splitting process completes the segmentation of the content into storyboard pictures, and the label extraction process extracts the key content labels such as characters and scenes in the entire content. The key content labels can be used as the basis for the subsequent generation of content consistency. When the corresponding label characters or scenes and other key content are selected to generate pictures, the consistency of the characters can be maintained and the quality of the text pictures can be improved. Moreover, the process from splitting the plot description text and label extraction processing to generating pictures can be completed automatically, without the need for manual editing and processing by the user, which improves the efficiency of converting text to video and can meet the needs of users who need to produce a large number of videos.

[0139] Embodiment 2 Figure 4 The block diagram of the apparatus for generating video based on text according to the second embodiment of the present application is schematically shown. The apparatus can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can perform specific functions. The following description will specifically introduce the functions of each program module in this embodiment. This embodiment is applied to the server, such as Figure 4 As shown, the apparatus 400 for generating a video based on text may include: a plot description text acquisition module 401, a text segmentation module 402, a tag extraction module 403 and a text image module 404, wherein: The plot description text acquisition module 401 is used to acquire the plot description text uploaded by the client; The text splitting module 402 is used to split the plot description text to obtain multiple storyboard texts, and determine the storyboard prompt words corresponding to each storyboard text; The tag extraction module 403 is used to extract at least one key content tag and a feature text corresponding to each key content tag from the plot description text, and determine a tag prompt word corresponding to each feature text; The text image module 404 is used to process the text image according to the storyboard prompt words and the label prompt words to obtain multiple storyboard pictures, and edit the multiple storyboard pictures to generate a target video.

[0140] In an optional embodiment of the present application, the key content tags include character tags and scene tags; the tag extraction module 403 includes: A character label extraction submodule is used to extract the character name as the character label when recognizing that the plot description text contains the character name, and extract the relevant description information corresponding to the character name to generate the feature text corresponding to the character label; and / or, The scene label extraction submodule is used to extract the place names as scene labels when identifying that there are multiple repeated place names in the plot description text, and extract the relevant description information corresponding to the place names to generate feature text corresponding to the scene labels.

[0141] In an optional embodiment of the present application, the plot description text acquisition module 401 is specifically used to: Controlling the display of the video creation page on the graphical user interface provided by the client; A plot setting operation on the video creation page is received, and the plot description text is extracted from the plot setting operation.

[0142] In an optional embodiment of the present application, the device further includes: A first page jump module, used to control the client to jump from the video creation page to the first page; wherein the content displayed on the first page includes the multiple storyboard texts and the update storyboard control; The first storyboard text editing module is used to respond to an editing operation on a target text among the multiple storyboard texts, determine first update information of the target text; and respond to a trigger operation acting on the update storyboard control to adjust the target text according to the first update information.

[0143] In an optional embodiment of the present application, the device further includes: The basic character image display module is used to generate a basic character image according to the character tag in the key content tag and the feature text corresponding to the character tag; and control the display of the basic character image on the first page.

[0144] In an optional embodiment of the present application, the content displayed on the first page further includes the at least one key content tag and a save tag control, and the device further includes: A tag editing module is used to respond to a trigger operation on a target tag in the at least one key content tag, control the display of a feature text corresponding to the target tag on the first page; respond to an edit operation on the feature text corresponding to the target tag, determine second update information of the feature text corresponding to the target tag; respond to a trigger operation acting on the save tag control, and adjust the feature text corresponding to the target tag according to the second update information.

[0145] In an optional embodiment of the present application, the first page further includes a new label control, and the device further includes: A label creation module is used to respond to a trigger operation acting on the new label control, control the display of the label setting component in the first page; respond to an input operation acting on the label setting component, extract label configuration information from the input operation; generate a new label according to the label configuration information, and control the display of the new label on the first page.

[0146] In an optional embodiment of the present application, the first page further includes a second page switching control, and the text image module 404 includes: A second page jump submodule, for responding to a trigger operation on the second page switch control, and controlling the client to jump from the first page to the second page; wherein the content displayed on the second page includes a generate illustration control corresponding to each storyboard file; The text image submodule is used to respond to the trigger operation acting on the generate picture control, and perform text image processing according to the storyboard prompt words and the label prompt words to obtain multiple storyboard pictures.

[0147] In an optional embodiment of the present application, the device further includes: An image action parameter setting module, used for controlling the display of an action skeleton setting component in the second page; and determining image action parameters in response to a setting operation acting on the action skeleton setting control; The Wensheng graph submodule includes: The text-generated image unit is used to respond to the trigger operation acting on the generated picture control, and to process the text-generated image according to the image action parameters, the storyboard prompt words and the label prompt words to obtain multiple storyboards.

[0148] In an optional embodiment of the present application, the device further includes: The second storyboard text editing module is used to control the display of the multiple storyboard texts in the second page; respond to a third editing operation on a target text in the multiple storyboard texts, and adjust the target text according to the third editing operation.

[0149] In an optional embodiment of the present application, the device further includes: The prompt word conversion module is used to control the display of the storyboard prompt words and label prompt words on the second page; convert the storyboard prompt words and label prompt words according to a preset format to obtain standard prompt words; and display the standard prompt words on the second page.

[0150] In an optional embodiment of the present application, the device further includes: A dubbing module is used to control the dubbing controls corresponding to each storyboard text displayed on the second page; in response to a triggering operation of the dubbing controls corresponding to the first text among the multiple storyboard texts, control the display of the dubbing sound effect component corresponding to the first text on the second page; wherein the dubbing sound effect component includes a plurality of timbres; in response to a selection operation of a first timbre among the plurality of timbres, generate a target dubbing corresponding to the first text according to the first timbre.

[0151] In an optional embodiment of the present application, the device further includes: The dubbing adjustment module is used to control the display of the audition control and the dubbing setting control corresponding to the first dubbing in the second page; respond to the trigger operation acting on the audition control, control the client to play the first dubbing; respond to the setting operation acting on the dubbing setting control, determine the second timbre according to the setting operation, and generate the target dubbing corresponding to the first text according to the second timbre.

[0152] In an optional embodiment of the present application, the device further includes: A picture selection module, used for controlling the display of the plurality of storyboard pictures on the second page; and determining a plurality of target images from the plurality of storyboard pictures in response to a selection operation on the plurality of storyboard pictures; The data export module is used to export the multiple target images and the target dubbing corresponding to each target image.

[0153] Embodiment 3 Figure 5 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the method for generating video based on text according to the third embodiment of the present application is shown schematically. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 5 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as the program code of the method for generating video based on text, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0154] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0155] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0156] It should be pointed out that Figure 5 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0157] In this embodiment, the method for generating video based on text stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0158] Embodiment 4 An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the method for generating a video based on text in the embodiment are implemented.

[0159] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the computer-readable storage medium may also include both an internal storage unit of a computer device and an external storage device thereof. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on a computer device, such as the program code of the method for generating a video based on text in the embodiment, etc. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.

[0160] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0161] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0162] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating a video based on text, characterized in that: Applied to the server, the method includes: Get the plot description text uploaded by the client; Splitting the plot description text to obtain a plurality of storyboard texts, and determining a storyboard prompt word corresponding to each storyboard text; Extracting at least one key content tag and a feature text corresponding to each key content tag from the plot description text, and determining a tag prompt word corresponding to each feature text; According to the storyboard prompt words and the label prompt words, text image processing is performed to obtain multiple storyboard matching pictures, and according to the multiple storyboard matching pictures, editing is performed to generate a target video.

2. The method for generating video based on text according to claim 1, characterized in that: The key content tags include character tags and scene tags; the step of extracting at least one key content tag and feature text corresponding to each key content tag from the plot description text includes: When it is identified that the plot description text contains a character name, the character name is extracted as a character tag, and relevant description information corresponding to the character name is extracted to generate a feature text corresponding to the character tag; and / or, When it is identified that there are multiple repeated place names in the plot description text, the place names are extracted as scene labels, and relevant description information corresponding to the place names is extracted to generate feature text corresponding to the scene labels.

3. The method for generating video based on text according to claim 1, characterized in that: The step of obtaining the plot description text uploaded by the client includes: Controlling the display of the video creation page on the graphical user interface provided by the client; A plot setting operation on the video creation page is received, and the plot description text is extracted from the plot setting operation.

4. The method for generating video based on text according to claim 3, characterized in that: Before the step of performing text image processing according to the storyboard prompt words and the label prompt words to obtain a plurality of images, the method further includes: Control the client to jump from the video creation page to a first page; wherein the content displayed on the first page includes the multiple storyboard texts and a storyboard update control; In response to an editing operation on a target text among the plurality of storyboard texts, determining first update information of the target text; In response to a triggering operation acting on the update storyboard control, the target text is adjusted according to the first update information.

5. The method for generating video based on text according to claim 4, characterized in that: After the step of controlling the client to jump from the displayed video creation page to the first page, the method further includes: Generate a basic character image according to a character tag in the key content tag and a feature text corresponding to the character tag; And control the display of the basic character image on the first page.

6. The method for generating video based on text according to claim 4, characterized in that: The content displayed on the first page further includes the at least one key content tag and a save tag control, and the method further includes: In response to a triggering operation on a target tag in the at least one key content tag, controlling display of a feature text corresponding to the target tag on the first page; In response to an editing operation on the characteristic text corresponding to the target tag, determining second update information of the characteristic text corresponding to the target tag; In response to a triggering operation acting on the save tag control, a characteristic text corresponding to the target tag is adjusted according to the second update information.

7. The method for generating video based on text according to claim 4, characterized in that: The first page also includes a new label control, and the method further includes: In response to a triggering operation acting on the new label control, controlling display of a label setting component in the first page; In response to an input operation acting on the label setting component, extracting label configuration information from the input operation; A new tag is generated according to the tag configuration information, and the new tag is controlled to be displayed on the first page.

8. The method for generating video based on text according to claim 4, characterized in that: The first page also includes a second page switching control, and the text image processing is performed according to the storyboard prompt word and the label prompt word to obtain multiple storyboard matching pictures, including: In response to a trigger operation on the second page switching control, the client is controlled to jump from the first page to the second page; wherein the content displayed on the second page includes a generate illustration control corresponding to each storyboard file; In response to a trigger operation acting on the generate illustration control, a plurality of storyboard illustrations are obtained by performing text-generated image processing according to the storyboard prompt words and the label prompt words.

9. The method for generating video based on text according to claim 8, characterized in that: The method further comprises: Controlling display of the action skeleton setting component in the second page; In response to a setting operation acting on the action skeleton setting control, determining image action parameters; The response acts on the trigger operation of the generate picture control, and performs text-generated picture processing according to the storyboard prompt word and the label prompt word to obtain multiple storyboards, including: In response to a trigger operation acting on the generate picture control, a plurality of storyboard pictures are obtained by performing text-generated picture processing according to the image action parameters, the storyboard prompt words and the label prompt words.

10. The method for generating video based on text according to claim 8, characterized in that: The method further comprises: Controlling display of the plurality of storyboard texts in the second page; In response to a third editing operation on a target text among the plurality of storyboard texts, the target text is adjusted according to the third editing operation.

11. The method for generating video based on text according to claim 8, characterized in that: The method further comprises: Controlling display of the storyboard prompt words and label prompt words on the second page; The storyboard prompt words and label prompt words are converted into standard prompt words according to a preset format; and the standard prompt words are displayed on the second page.

12. The method for generating video based on text according to claim 10, characterized in that: The method further comprises: Controlling display of a dubbing control corresponding to each storyboard text on the second page; In response to a triggering operation of a dubbing control corresponding to a first text in the plurality of storyboard texts, controlling the display of a dubbing sound effect component corresponding to the first text in the second page; wherein the dubbing sound effect component includes a plurality of timbres; In response to a selection operation on a first timbre among the plurality of timbres, a target dubbing corresponding to the first text is generated according to the first timbre.

13. The method for generating video based on text according to claim 12, characterized in that: The method further comprises: Controlling display of audition controls and setting dubbing controls corresponding to the first dubbing on the second page; In response to a triggering operation on the audition control, controlling the client to play the first dubbing; In response to a setting operation on the setting dubbing control, a second timbre is determined according to the setting operation, and a target dubbing corresponding to the first text is generated according to the second timbre.

14. The method for generating video based on text according to claim 12 or 13, characterized in that: The method further comprises: Controlling display of the plurality of storyboards on the second page; In response to a selection operation on the plurality of storyboards, determining a plurality of target images from the plurality of storyboards; The plurality of target images and a target dubbing corresponding to each target image are exported.

15. A device for generating video based on text, characterized in that: Applied to the server, the device comprises: The plot description text acquisition module is used to obtain the plot description text uploaded by the client; A text splitting module is used to split the plot description text to obtain multiple storyboard texts, and determine the storyboard prompt words corresponding to each storyboard text; A tag extraction module, used to extract at least one key content tag and a feature text corresponding to each key content tag from the plot description text, and determine a tag prompt word corresponding to each feature text; The text image module is used to process the text image according to the storyboard prompt words and the label prompt words to obtain multiple storyboard pictures, and to edit the multiple storyboard pictures to generate a target video.

16. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for generating a video based on text according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method for generating a video based on text according to any one of claims 1 to 14 is implemented.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for generating a video based on text according to claims 1 to 14 are implemented.

Images