Text-based video generation method and device, electronic equipment and storage medium

Abstract

The invention provides a text-based video generation method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence and the field of financial science and technology. The method comprises the steps of obtaining a target text; performing text analysis on the target text to obtain a text segment, text segment entity information and text segment position information; wherein the text segment position information is used for representing the position of the text segment in the target text; performing material reading on a material database according to the text segment entity information to obtain a text segment basic material; generating a dynamic picture according to the text segment entity information to obtain a text segment dynamic picture material; screening a text segment video template from a video template database according to the text segment and the text segment position information; and performing video synthesis according to the text segment basic material, the text segment dynamic picture material, the text segment video template and the text segment position information to obtain a target video. The method can improve the video generation efficiency, and is suitable for a webpage end.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology and financial technology, and in particular to a text-based video generation method and device, electronic device, and storage medium. Background Art

[0002] Text-based video generation technology refers to the technology of generating videos based on text input by users. For example, in the field of financial technology, for a piece of product-related text input by users (such as car insurance introduction, pet insurance introduction, etc.), relevant marketing videos, product introduction videos or teaching videos can be generated.

[0003] Current technologies usually involve offline rendering or the use of complex video editing software, which is not suitable for web pages. For offline rendering solutions, users need to upload text to the server for processing and then download the generated video file. This method may involve file transfer, server resources and time costs, which is not convenient enough. For solutions using complex video editing software, users need to have relevant skills, and the learning and operation costs are high, which means that the video generation efficiency is low.

[0004] It can be seen that the existing text-based video generation method has the technical problem of low video generation efficiency. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a text-based video generation method and device, electronic device, and storage medium, which can improve the efficiency of video generation and are suitable for web pages.

[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a text-based video generation method, the method comprising:

[0007] Get the target text;

[0008] Performing text parsing on the target text to obtain a text segment, text segment entity information, and text segment position information; wherein the text segment position information is used to represent the position of the text segment in the target text;

[0009] Reading materials from a preset material database according to the text segment entity information to obtain basic materials for the text segment;

[0010] Generate a dynamic picture according to the text segment entity information to obtain a text segment dynamic picture material;

[0011] Filtering a text segment video template from a preset video template database according to the text segment and the text segment position information;

[0012] Video synthesis is performed according to the text segment basic material, the text segment dynamic picture material, the text segment video template and the text segment position information to obtain a target video.

[0013] In some embodiments, the video template database includes candidate video templates of candidate subject categories; the step of selecting a text segment video template from a preset video template database according to the text segment and the text segment position information includes:

[0014] Performing topic identification on the text segment to obtain a reference topic category;

[0015] Screening the candidate video templates according to the reference subject category and the candidate subject category to obtain a preliminarily selected video template;

[0016] The preliminarily selected video template is screened according to the text segment position information to obtain a text segment video template corresponding to the text segment.

[0017] In some embodiments, the screening of the preliminarily selected video template according to the text segment position information to obtain a text segment video template corresponding to the text segment includes:

[0018] Analyzing the position information of the initially selected video template to obtain candidate template position information;

[0019] The candidate template position information is screened according to the text segment position information to obtain the target selected template position information;

[0020] The initially selected video template corresponding to the target selected template position information is determined as the text segment video template corresponding to the text segment.

[0021] In one embodiment, the screening of the candidate template position information according to the text segment position information to obtain the target selected template position information includes:

[0022] Serialize the text segment location information to obtain the text segment character string;

[0023] Serialize the candidate template position information to obtain the template string;

[0024] Compare the text segment string and the template string to determine the number of identical characters.

[0025] The candidate template position information having the same number of characters and meeting a predetermined condition is determined as the target selected template position information.

[0026] In some embodiments, performing video synthesis according to the text segment basic material, the text segment dynamic picture material, the text segment video template, and the text segment position information to obtain the target video includes:

[0027] Synthesize video segments according to the text segment basic material, the text segment dynamic picture material and the text segment video template to obtain a basic video segment;

[0028] The basic video clips are arranged according to the text segment position information to obtain a target video.

[0029] In some embodiments, synthesizing video segments according to the text segment basic material, the text segment dynamic picture material and the text segment video template to obtain a basic video segment includes:

[0030] Dividing the text segment basic material into basic picture materials and obtaining basic non-picture materials;

[0031] Perform material layout according to the text segment dynamic picture material and the basic picture material to obtain picture material layout information;

[0032] Synthesize video segments according to the screen material layout information and the text segment video template to obtain an initial video segment;

[0033] The basic non-screen material is added to the initial video segment to obtain the basic video segment.

[0034] In some embodiments, the material database includes candidate materials of candidate style categories;

[0035] The step of reading material from a preset material database according to the text segment entity information to obtain the text segment basic material includes:

[0036] Performing style recognition on the text segment entity information to obtain a reference style category;

[0037] Screening the candidate style categories according to the reference style categories to obtain a target selected style category;

[0038] The candidate material corresponding to the target selected style category is determined as the text segment basic material.

[0039] In some embodiments, the text parsing of the target text to obtain the text segment, text segment entity information, and text segment location information includes:

[0040] Dividing the target text into paragraphs to obtain the text segments;

[0041] generating the text segment position information according to the position of the text segment in the target text;

[0042] Entity information is extracted from the text segment to obtain entity information of the text segment.

[0043] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a text-based video generation device, the device comprising:

[0044] A text acquisition module is used to acquire the target text;

[0045] A text parsing template, used to perform text parsing on the target text to obtain a text segment, text segment entity information, and text segment position information; wherein the text segment position information is used to characterize the position of the text segment in the target text;

[0046] A material reading module, used to read the material from a preset material database according to the text segment entity information to obtain the basic material of the text segment;

[0047] An animation generation module, used to generate dynamic images according to the text segment entity information to obtain text segment dynamic image material;

[0048] A template screening module, used for screening out a text segment video template from a preset video template database according to the text segment and the text segment position information;

[0049] The video synthesis module is used to perform video synthesis based on the text segment basic material, the text segment dynamic picture material, the text segment video template and the text segment position information to obtain a target video.

[0050] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the text-based video generation method described in the first aspect is implemented.

[0051] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the text-based video generation method described in the first aspect is implemented.

[0052] The text-based video generation method, text-based video generation device, electronic device and computer-readable storage medium proposed in the present application first parse the target text to obtain text segment, text segment entity information and text segment position information. According to the text segment entity information, not only can the text segment basic material be read, but also the text segment dynamic picture material can be generated. According to the text segment and the text segment position information, the text segment video template is screened out, and the text segment video template not only meets the semantic requirements of the text segment, but also meets the position requirements of the text segment in the target text. Finally, video synthesis is performed according to multimodal data (i.e., text segment basic material, text segment dynamic picture material, text segment video template and text segment position information) to obtain the target video. It can be seen that the present application realizes the intelligent conversion from text to video by parsing the target text. Without the need for the user to have any video editing skills, the important information in the text can be automatically identified, suitable dynamic picture material can be generated and suitable basic material and video template can be matched, which greatly simplifies the video production process. In summary, the present application can improve the efficiency of video generation and is very suitable for web page end.

[0053] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flowchart of a text-based video generation method provided in an embodiment of the present application;

[0055] Figure 2 yes Figure 1 Flowchart of step 102 in FIG.

[0056] Figure 3 yes Figure 1 Flow chart of step 103 in;

[0057] Figure 4 yes Figure 1 Flowchart of step 105 in;

[0058] Figure 5 yes Figure 4 Flow chart of step 403 in;

[0059] Figure 6 is a module structure block diagram of a text-based video generation device provided in an embodiment of the present application;

[0060] Figure 7 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0064] First, some nouns involved in this application are analyzed:

[0065] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0066] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0067] In the current industry, there is still a lack of an efficient and user-friendly web-based text-to-video solution. For example, current technologies usually involve offline rendering or the use of complex video editing software, which is not suitable for web pages. For offline rendering solutions, users need to upload text to the server for processing, and then download the generated video file. This method may involve file transfer, server resources and time costs, which is not convenient enough. For solutions using complex video editing software, users need to have relevant skills, and the learning and operation costs are high, which means that the video generation efficiency is low.

[0068] Based on this, the embodiments of the present application propose a text-based video generation method, a text-based video generation device, an electronic device and a computer-readable storage medium, providing a simple, fast and customized text-to-video solution to meet the needs of users to generate videos on web pages.

[0069] The text-based video generation method provided in the embodiment of the present application can also be applied to the terminal and the server side, and can also be software running on the server side. The server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the text-based video generation method, but is not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] The embodiments of the present application provide a text-based video generation method and a text-based video generation device, an electronic device, and a storage medium, which are specifically illustrated by the following embodiments. First, the text-based video generation method in the embodiments of the present application is described.

[0072] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0073] Reference Figure 1 , Figure 1 It is an optional flowchart of the text-based video generation method provided in an embodiment of the present application, which may include but is not limited to steps 101 to 106.

[0074] Step 101, obtaining a target text;

[0075] Step 102, performing text parsing on the target text to obtain text segments, text segment entity information, and text segment position information; wherein the text segment position information is used to represent the position of the text segment in the target text;

[0076] Step 103, reading materials from a preset material database according to the text segment entity information to obtain basic materials for the text segment;

[0077] Step 104, generating a dynamic image according to the text segment entity information to obtain the text segment dynamic image material;

[0078] Step 105, selecting a text segment video template from a preset video template database according to the text segment and the text segment position information;

[0079] Step 106, performing video synthesis based on the text segment basic material, the text segment dynamic picture material, the text segment video template and the text segment position information to obtain the target video.

[0080] Steps 101 to 106 shown in the embodiment of the present application first parse the target text to obtain text segment, text segment entity information and text segment position information. According to the text segment entity information, not only can the text segment basic material be read, but also the text segment dynamic picture material can be generated. According to the text segment and the text segment position information, a text segment video template is screened out, which not only meets the semantic requirements of the text segment, but also meets the position requirements of the text segment in the target text. Finally, video synthesis is performed according to multimodal data (i.e., text segment basic material, text segment dynamic picture material, text segment video template and text segment position information) to obtain the target video. It can be seen that the present application realizes the intelligent conversion from text to video by parsing the target text. Without the need for the user to have any video editing skills, the important information in the text can be automatically identified, suitable dynamic picture material can be generated and suitable basic material and video template can be matched, which greatly simplifies the video production process. In summary, the present application can improve the efficiency of video generation and is very suitable for web page end.

[0081] In step 101 of some embodiments, a target text is obtained. The target text refers to the text used to generate the video. For example, a user enters a car insurance introduction text related to a car insurance product on a web page, and the product introduction text is used as the target text. For another example, a user enters a pet insurance introduction text related to a pet insurance product on a web page, and the pet insurance introduction text is used as the target text.

[0082] In step 102 of some embodiments, the target text is parsed to obtain text segments, text segment entity information, and text segment location information. The target text is parsed by a natural language processing model to obtain text segments, text segment entity information, and text segment location information. A text segment refers to a text line / text block belonging to the same paragraph in the target text. Text segment entity information refers to entities / keywords appearing in a text segment. Entities include people, events, time, place, emotion, etc.

[0083] The text segment position information is used to characterize the position of the text segment in the target text. The text segment position information may include the starting position of the text segment and the ending position of the text segment. For example, the starting position of the text segment is the 55th character, and the ending position of the text segment is the 150th character. For another example, the starting position of the text segment is the 6th line, and the ending position of the text segment is the 10th line. The text segment position information may also include the paragraph number. For example, the paragraph number is 3.

[0084] In one embodiment, the text segment position information may further include a first paragraph relationship and a second paragraph relationship. The first paragraph relationship refers to the relationship between the current paragraph and the previous paragraph. The second paragraph relationship refers to the relationship between the current paragraph and the next paragraph. The first paragraph relationship and the second paragraph relationship may both include:

[0085] (1) Logical relationships: Cause-effect relationship: The content of one paragraph is the result of the content of another paragraph, for example, "Paragraph A explains the cause, and paragraph B gives the result." Contrasting relationship: There is a comparison or contrast between two paragraphs. 3. Supplementary relationship: One paragraph supplements another paragraph or provides additional information.

[0086] (2) Thematic Relationship: Similar Topics: Two paragraphs discuss the same or related topics. Different Aspects: Two paragraphs discuss different aspects or dimensions of the same topic.

[0087] (3) Temporal relations, including: Sequential relations: the content described in one paragraph occurs before or after another paragraph in time. It can be divided into "cause and effect" or "time sequence". Contrasting time: two paragraphs may involve the comparison of different times, such as the comparison of "past and present".

[0088] (4) Hierarchical relationships, including: Primary-secondary relationship: one paragraph may be the main argument, while another paragraph is an auxiliary or explanation of it. Summary-specific relationship: one paragraph summarizes another paragraph, while the other paragraph elaborates on specific examples or details.

[0089] (5) Emotional or tonal relationships, including: Emotional contrast: one paragraph conveys positive emotion while the other paragraph conveys negative emotion or the opposite emotion. Tone change: two paragraphs may have a change in tone, such as from serious to humorous or from formal to informal.

[0090] (6) Structural relationships, including: Introduction and conclusion: one paragraph may be an introduction, and another paragraph may be a conclusion or summary. Transition paragraph: a paragraph used to connect two different topics or ideas, acting as a bridge.

[0091] In one embodiment, referring to Figure 2 , step 102 may include:

[0092] Step 201, dividing the target text into paragraphs to obtain text segments;

[0093] Step 202, generating text segment position information according to the position of the text segment in the target text;

[0094] Step 203: extract entity information from the text segment to obtain text segment entity information.

[0095] In step 201, a paragraph division tool may be called to divide the target text into paragraphs to obtain text segments. For example, in the process of paragraph division, each sentence of the target text is usually first subject-identified to obtain a target central sentence; the target text is segmented according to each target central sentence to obtain a text segment corresponding to each target central sentence.

[0096] In step 202, the text segment location information has been introduced above and will not be repeated here.

[0097] In one embodiment, step 202 may include:

[0098] According to the position sequence of the starting character of the text segment in each character of the target text, the starting position of the text segment is obtained;

[0099] According to the position sequence of the end character of the text segment in each character of the target text, the end position of the text segment is obtained;

[0100] According to the position sequence of the text segment in each text segment of the target text, the paragraph number is obtained;

[0101] According to the paragraph relationship between the text segment and the previous text segment, the first paragraph relationship is obtained;

[0102] According to the paragraph relationship between the text segment and the next text segment, a second paragraph relationship is obtained;

[0103] The text segment position information is generated according to at least one of the text segment start position, the text segment end position, the paragraph number, the first paragraph relationship, and the second paragraph relationship.

[0104] The benefit of the above embodiment is that it can fully utilize the information of each text segment in the target text, so as to generate text segment position information with high information richness.

[0105] It should be noted that whether it is the paragraph relationship between a text segment and the previous text segment, or the paragraph relationship between a text segment and the next text segment, it can be determined based on factors such as the logical relationship, thematic relationship, and temporal relationship between the two paragraphs, and will not be repeated here.

[0106] In step 103 of some embodiments, a preset material database is read according to the text segment entity information to obtain the text segment basic material. The material database refers to a database storing multiple candidate materials. The material database includes candidate materials of candidate style categories. For example, the candidate style categories include animation style, real scene style or mixed style.

[0107] In one embodiment, referring to Figure 3 , step 103 may include:

[0108] Step 301, performing style recognition on text segment entity information to obtain a reference style category;

[0109] Step 302, screening candidate style categories according to reference style categories to obtain target selected style categories;

[0110] Step 303: determine the candidate material corresponding to the target selected style category as the basic material of the text segment.

[0111] In step 301, the text segment entity information can be style identified by a style classification model to obtain a reference style category. The style classification model is a neural network model, such as a BERT model. For example, the reference style category is any one of an animation style, a real-life style, or a mixed style.

[0112] In step 302, the target selected style category refers to the candidate style category that is the same as the reference style category. For example, if the reference style category is an animation style, the animation style in the candidate style category is determined as the target selected style category. If no candidate style category that is the same as the reference style category is screened out, the mixed style is determined as the target selected style category by default.

[0113] In step 303, if the target selected style category is animation style, the candidate material corresponding to the animation style is determined as the basic text segment material. For example, at least one of the picture material, music material, video segment material, etc. corresponding to the animation style is determined as the basic text segment material.

[0114] The benefit of the above-mentioned embodiment of step 301 to step 303 is that material screening is performed based on the style category contained in the text segment entity information, thereby improving the material screening efficiency.

[0115] In one embodiment, after step 103, the text-based video generation method provided in this embodiment may further include: updating the text segment basic material, including: converting the text segment into text speech through text-to-speech technology (TTS), and determining the text speech as the speech material in the text segment basic material. Specifically, the text content input by the user can be automatically converted into natural and fluent speech. Users can choose different styles of speech according to their needs, such as male voice, female voice, different voice intonations and emotional expressions. This embodiment also supports multi-language text input and speech generation, which can meet the video production needs of different users.

[0116] In step 104 of some embodiments, dynamic images are generated based on the text segment entity information to obtain text segment dynamic image materials. By combining AI image generation and animation technology, dynamic images are generated for the text segment entity information to obtain text segment dynamic image materials. The text segment dynamic image materials can display visual effects such as character images and scene switching.

[0117] In one example, the text segment dynamic picture material includes text animation material, picture animation material, chart animation material, etc. By automatically adding dynamic effects, the video presentation is made more vivid. For example, in a product introduction video, the automatically generated chart can appear with the product description and be displayed dynamically.

[0118] In step 105 of some embodiments, a text segment video template is screened out from a preset video template database based on the text segment and the text segment position information. The video template database refers to a database storing video templates. The video template database includes candidate video templates of candidate theme categories. For example, the candidate style categories include emotional themes, character themes, location themes, etc.

[0119] In one embodiment, referring to Figure 4 , step 105 may include:

[0120] Step 401, performing topic identification on the text segment to obtain a reference topic category;

[0121] Step 402, screening candidate video templates according to reference theme categories and candidate theme categories to obtain a preliminarily selected video template;

[0122] Step 403: Screen the preliminarily selected video templates according to the text segment position information to obtain a text segment video template corresponding to the text segment.

[0123] In step 401, topic identification can be performed on the text segment entity information through a topic classification model to obtain a reference topic category. The topic classification model is a neural network model, such as a BERT model. For example, the reference topic category is any one of an emotional topic, a character topic, and a location topic.

[0124] In step 402, the candidate topic categories can be screened according to the reference topic categories to obtain the target selected topic category; the candidate video template corresponding to the target selected topic category is determined as the preliminary selected video template. The target selected topic category refers to the candidate topic category that is the same as the reference topic category. For example, if the reference topic category is an emotional topic, the emotional topic in the candidate topic category is determined as the target selected topic category. If the candidate topic category that is the same as the reference topic category cannot be screened out, the emotional topic is determined as the target selected topic category by default. For example, if the target selected topic category is an emotional topic, the candidate video template corresponding to the emotional topic is determined as the preliminary selected video template.

[0125] In step 403, considering that the video template corresponding to each text segment should conform to the position of the text segment in the target text, the initially selected video templates need to be screened according to the text segment position information to obtain a more accurate text segment video template.

[0126] The benefit of the above-mentioned embodiment of step 401 to step 403 is that the template screening is performed based on the subject category contained in the text segment entity information, thereby improving the efficiency of template screening.

[0127] In one embodiment, referring to Figure 5 , step 403 may include:

[0128] Step 501, analyzing the position information of the initially selected video template to obtain the candidate template position information;

[0129] Step 502, screening candidate template position information according to the text segment position information to obtain target selected template position information;

[0130] Step 503: determine the initially selected video template corresponding to the target selected template position information as the text segment video template corresponding to the text segment.

[0131] In step 501, the preliminarily selected video template generally includes a plurality of key information and elements. Specifically, the preliminarily selected video template may include the following information: video template title, video template theme, video template duration, video template screen layout, etc. The large language model may be called to analyze the information of the preliminarily selected video template to obtain the position of the preliminarily selected video template in a video, that is, to obtain the candidate template position information.

[0132] There is a corresponding relationship between the candidate template position information and the text segment position information. For example, if the text segment position information includes the paragraph number, then the candidate template position information includes the target number. For another example, if the text segment position information includes the first paragraph relationship and the second paragraph relationship, then the candidate template position information includes the first template relationship and the second template relationship. The first template relationship refers to the template relationship between the current template and the previous template. The second template refers to the template relationship between the current template and the previous template. Template relationships are similar to paragraph relationships, and also include logical relationships, time relationships, hierarchical relationships, etc.

[0133] In step 502, the target selected template position information refers to the candidate template position information that is substantially the same as / has the greatest similarity to the text segment position information. For example, if the first paragraph relationship included in the text segment position information is a logical relationship and the second paragraph relationship is a temporal relationship, then the candidate template position information including the first template relationship as a logical relationship and the second template relationship as a temporal relationship is determined as the target selected template position information. If the candidate template position information that is the same as the text segment position information cannot be screened out, a candidate template position information is randomly selected as the target selected template position information.

[0134] In one embodiment, step 502 may include: serializing the text segment position information to obtain the text segment string; serializing the candidate template position information to obtain the template string; performing character comparison based on the text segment string and the template string to determine the number of identical characters; and determining the candidate template position information whose number of identical characters meets a predetermined condition as the target selected template position information. In this way, serialization is used to improve the comparison efficiency between the two position information.

[0135] It should be noted that the predetermined condition may be that the number of identical characters is greater than a number threshold, or that the ratio of the number of identical characters to the total number of characters in the text segment string is greater than a ratio threshold.

[0136] In step 106 of some embodiments, video synthesis is performed based on the text segment basic material, the text segment dynamic picture material, the text segment video template and the text segment position information to obtain the target video.

[0137] In one embodiment, step 106 may include:

[0138] Synthesize video clips according to text segment basic materials, text segment dynamic picture materials and text segment video templates to obtain basic video clips;

[0139] The basic video clips are subjected to video layout transition according to the text segment position information to obtain the target video.

[0140] The benefit of the above embodiment is that a basic video segment is first generated for each text segment, and then the video layout transition is performed according to the text segment position information, thereby improving the video continuity.

[0141] In one embodiment, after the step of performing video layout transition on the basic video segments according to the text segment position information to obtain the target video, the target video is output in the video output format in response to the video output format selected by the user. For example, the video output formats include MP4, WEBM, GIF, etc. Supporting multi-format output can facilitate adaptation to different playback platforms and uses.

[0142] In one embodiment, the step of synthesizing video segments according to the text segment basic material, the text segment dynamic picture material and the text segment video template to obtain the basic video segment may include:

[0143] Dividing the basic material of the text segment into basic picture material and basic non-picture material;

[0144] Perform material layout according to the text segment dynamic picture material and the basic picture material to obtain picture material layout information;

[0145] Synthesize video clips according to the layout information of the picture material and the text segment video template to obtain an initial video clip;

[0146] Basic non-screen materials are added to the initial video clip to obtain a basic video clip.

[0147] Specifically, the basic text segment materials include picture materials, video segment materials, background music materials, and voice materials. Picture materials and video segment materials are visual materials and can be classified as basic picture materials. Background music and voice materials are non-visual materials and can be classified as basic non-picture materials. The material classification model can be called to classify the basic text segment materials. The material classification model is a neural network model, such as the BERT model.

[0148] In addition to the basic picture material being a visual material, the text segment dynamic picture material is also a visual material. In this embodiment, the material layout is first performed, and the obtained picture material layout information can represent the layout information of the text segment dynamic picture element and the basic picture material in the video frame. For example, the text segment dynamic element is in the 5th video frame, and the basic picture material serves as the background of the 5th frame. For another example, the text segment dynamic element is in the first area of ​​the 5th frame, and the basic picture material is in the second area of ​​the 5th frame, and the first area and the second area do not overlap each other.

[0149] Then, the video segments are synthesized according to the layout information of the picture material and the text segment video template to obtain the initial video segment; and then basic non-picture materials are added to the initial video segment to obtain the basic video segment.

[0150] The benefit of the above embodiment is that, on the basis of being able to generate a basic video segment, various influencing factors that affect the generation of the basic video segment are processed in a hierarchical manner, thereby simplifying the generation complexity and improving the generation efficiency.

[0151] In one embodiment, a real-time preview function can be supported: the user can preview the current effect in real time during the process of generating a video, and adjust the text content, template, dubbing, animation and other elements at any time. Editable functions can also be supported: the user can manually adjust the text display position in the video, the duration and order of the animation effect, the volume of the background music, etc. according to their own needs, and increase the personalized options for video generation.

[0152] Based on the above embodiments, this application combines multiple technologies such as NLP, AI image generation, and text-to-speech to achieve the fusion generation of multi-modal images, videos, voices, texts, etc., giving users rich visual and auditory effects and making the generated video content more vivid.

[0153] See also Figure 6 The embodiment of the present application also provides a text-based video generation device, which can implement the above-mentioned text-based video generation method. Figure 6The module structure block diagram of the text-based video generation device provided in the embodiment of the present application includes: a text acquisition module 601, a text parsing template 602, a material reading module 603, an animation generation module 604, a template screening module 605, and a video synthesis module 606. The text acquisition module 601 is used to acquire the target text; the text parsing template 602 is used to perform text parsing on the target text to obtain the text segment, text segment entity information and text segment position information; wherein the text segment position information is used to characterize the position of the text segment in the target text; the material reading module 603 is used to read the material from the preset material database according to the text segment entity information to obtain the text segment basic material; the animation generation module 604 is used to generate dynamic pictures according to the text segment entity information to obtain the text segment dynamic picture material; the template screening module 605 is used to screen out the text segment video template from the preset video template database according to the text segment and the text segment position information; the video synthesis module 606 is used to perform video synthesis according to the text segment basic material, the text segment dynamic picture material, the text segment video template and the text segment position information to obtain the target video.

[0154] In one embodiment, the text parsing template 602 is specifically used to: divide the target text into paragraphs to obtain text segments; generate text segment position information according to the position of the text segment in the target text; and extract entity information from the text segment to obtain text segment entity information.

[0155] In one embodiment, the material reading module 603 is specifically used to: perform style recognition on text segment entity information to obtain a reference style category; screen candidate style categories according to the reference style category to obtain a target selected style category; and determine the candidate material corresponding to the target selected style category as the text segment basic material.

[0156] In one embodiment, the template screening module 605 is specifically used to: perform topic identification on the text segment to obtain a reference topic category; screen the candidate video templates according to the reference topic category and the candidate topic category to obtain a preliminarily selected video template; screen the preliminarily selected video template according to the text segment position information to obtain a text segment video template corresponding to the text segment.

[0157] In another embodiment, the template screening module 605 is specifically used to: perform topic identification on the text segment to obtain a reference topic category; screen the candidate video templates according to the reference topic category and the candidate topic category to obtain a preliminarily selected video template; analyze the position information of the preliminarily selected video template to obtain the candidate template position information; screen the candidate template position information according to the text segment position information to obtain the target selected template position information; determine the preliminarily selected video template corresponding to the target selected template position information as the text segment video template corresponding to the text segment.

[0158] In one embodiment, the video synthesis module 606 is specifically used to: synthesize video clips according to text segment basic materials, text segment dynamic picture materials and text segment video templates to obtain basic video clips; and layout the basic video clips according to text segment position information to obtain target videos.

[0159] In another embodiment, the video synthesis module 606 is specifically used to: divide the basic material of the text segment to obtain basic picture material and basic non-picture material; layout the material according to the dynamic picture material of the text segment and the basic picture material to obtain picture material layout information; synthesize video clips according to the picture material layout information and the text segment video template to obtain an initial video clip; add basic non-picture material to the initial video clip to obtain a basic video clip; layout the basic video clip according to the text segment position information to obtain a target video.

[0160] It should be noted that the specific implementation of the text-based video generation device is basically the same as the specific implementation of the above-mentioned text-based video generation method, and will not be repeated here.

[0161] The embodiment of the present application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned text-based video generation method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0162] See also Figure 7 , Figure 7 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0163] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0164] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the text-based video generation method of the embodiment of this application;

[0165] Input / output interface 703, used to implement information input and output;

[0166] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0167] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0168] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0169] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned text-based video generation method.

[0170] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0171] The present application provides a text-based video generation method, a text-based video generation device, an electronic device and a computer-readable storage medium, by first parsing the target text to obtain a text segment, text segment entity information and text segment position information. According to the text segment entity information, not only can the text segment basic material be read, but also the text segment dynamic picture material can be generated. According to the text segment and the text segment position information, a text segment video template is screened out, and the text segment video template not only meets the semantic requirements of the text segment, but also meets the position requirements of the text segment in the target text. Finally, video synthesis is performed according to multimodal data (i.e., text segment basic material, text segment dynamic picture material, text segment video template and text segment position information) to obtain a target video. It can be seen that the present application realizes the intelligent conversion from text to video by parsing the target text. Without the need for the user to have any video editing skills, the important information in the text can be automatically identified, suitable dynamic picture materials can be generated and suitable basic materials and video templates can be matched, which greatly simplifies the video production process. In summary, the present application can improve the efficiency of video generation and is very suitable for web pages.

[0172] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0173] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0174] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0176] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0177] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0178] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0182] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A text-based video generation method, characterized in that: The method comprises: Get the target text; Performing text parsing on the target text to obtain a text segment, text segment entity information, and text segment position information; wherein the text segment position information is used to represent the position of the text segment in the target text; Reading materials from a preset material database according to the text segment entity information to obtain basic materials for the text segment; Generate a dynamic picture according to the text segment entity information to obtain a text segment dynamic picture material; Filtering a text segment video template from a preset video template database according to the text segment and the text segment position information; Video synthesis is performed according to the text segment basic material, the text segment dynamic picture material, the text segment video template and the text segment position information to obtain a target video.

2. The method according to claim 1, characterized in that The video template database includes candidate video templates of candidate subject categories; The step of selecting a text segment video template from a preset video template database according to the text segment and the text segment position information includes: Performing topic identification on the text segment to obtain a reference topic category; Screening the candidate video templates according to the reference subject category and the candidate subject category to obtain a preliminarily selected video template; The preliminarily selected video template is screened according to the text segment position information to obtain a text segment video template corresponding to the text segment.

3. The method according to claim 2, characterized in that The step of screening the preliminarily selected video template according to the text segment position information to obtain a text segment video template corresponding to the text segment includes: Analyzing the position information of the initially selected video template to obtain candidate template position information; The candidate template position information is screened according to the text segment position information to obtain the target selected template position information; The initially selected video template corresponding to the target selected template position information is determined as the text segment video template corresponding to the text segment.

4. The method according to any one of claims 1 to 3, characterized in that: The step of performing video synthesis according to the text segment basic material, the text segment dynamic picture material, the text segment video template and the text segment position information to obtain a target video includes: Synthesize video segments according to the text segment basic material, the text segment dynamic picture material and the text segment video template to obtain a basic video segment; The basic video clips are arranged according to the text segment position information to obtain a target video.

5. The method according to claim 4, characterized in that The step of synthesizing video segments according to the text segment basic material, the text segment dynamic picture material and the text segment video template to obtain a basic video segment includes: Dividing the text segment basic material into basic picture materials and obtaining basic non-picture materials; Perform material layout according to the text segment dynamic picture material and the basic picture material to obtain picture material layout information; Synthesize video segments according to the screen material layout information and the text segment video template to obtain an initial video segment; The basic non-screen material is added to the initial video segment to obtain the basic video segment.

6. The method according to any one of claims 1 to 3, characterized in that: The material database includes candidate materials of candidate style categories; The step of reading material from a preset material database according to the text segment entity information to obtain the text segment basic material includes: Performing style recognition on the text segment entity information to obtain a reference style category; Screening the candidate style categories according to the reference style categories to obtain a target selected style category; The candidate material corresponding to the target selected style category is determined as the text segment basic material.

7. The method according to any one of claims 1 to 3, characterized in that: The text parsing of the target text to obtain text segments, text segment entity information and text segment location information includes: Dividing the target text into paragraphs to obtain the text segments; generating the text segment position information according to the position of the text segment in the target text; Entity information is extracted from the text segment to obtain entity information of the text segment.

8. A text-based video generation device, characterized in that: The device comprises: A text acquisition module is used to acquire the target text; A text parsing template, used to perform text parsing on the target text to obtain a text segment, text segment entity information, and text segment position information; wherein the text segment position information is used to characterize the position of the text segment in the target text; A material reading module, used to read the material from a preset material database according to the text segment entity information to obtain the basic material of the text segment; An animation generation module, used to generate dynamic images according to the text segment entity information to obtain text segment dynamic image material; A template screening module, used for screening out a text segment video template from a preset video template database according to the text segment and the text segment position information; The video synthesis module is used to perform video synthesis based on the text segment basic material, the text segment dynamic picture material, the text segment video template and the text segment position information to obtain a target video.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images