Text-to-video conversion method and device based on deep semantic analysis, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a text-to-video method based on deep semantic analysis, which comprises the following steps of: extracting keywords and named entities from a to-be-converted text, analyzing a core theme and a text structure of the text, and performing semantic analysis to generate a semantic analysis result; retrieving matched images, video clips, sound effects and background music according to a semantic analysis result, and performing spatial layout to generate a preliminary visual layout framework; and constructing a video timeline, integrating the layout scheme, the background music and the sound effect to generate a complete video content, and finally outputting the video content as a video file. Keywords, named entities, core topics and text structures in texts are accurately extracted through semantic analysis, images, video clips, sound effects and background music conforming to semantics are intelligently matched, and the problem that a large amount of manual intervention is needed is avoided. An adaptive animation effect is generated through a deep learning model, and dynamic presentation of video content and continuity of a visual effect are ensured.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a text-to-video method, device, equipment and storage medium based on deep semantic analysis. Background Art

[0002] In the financial sector, video creation tools have gradually become an important tool for promoting financial services and educating users. Currently, financial institutions use videos to showcase complex financial products, investment strategies, and risk control methods in order to convey information to customers more intuitively. However, existing digital media tools face multiple challenges in the production of financial videos, which severely limits their application effectiveness and efficiency.

[0003] First, existing video production tools lack intelligence and automation. Although some tools support automated operations, in most cases, users still need to manually adjust the video content, especially when the video involves highly professional content such as financial data and investment analysis. This not only increases editing time, but also affects the coherence of the video and the professionalism of the content, making financial institutions face inefficiencies in promotion and education of customers.

[0004] Secondly, existing tools have limitations in semantic understanding and content matching. The financial industry involves a large number of complex terms, data, analysis, and professional terms. However, existing text-to-video technologies have difficulty accurately understanding the financial field's proprietary terms, complex industry terms, and their inherent relevance. Inaccurate semantic understanding can cause the generated video content to deviate from the financial institution's intentions, making it difficult to effectively convey key information and reducing the value of the video.

[0005] In addition, in terms of visual and auditory presentation, the scene matching and sound effect synthesis of existing technologies are also insufficient. Financial videos often require rigor and professionalism in information transmission, but current video creation tools are often unable to intelligently match financial scenes and generate suitable data visualization effects. For example, when showing return on investment or risk prediction, existing tools may not be able to automatically generate appropriate graphics and visual effects, and users still need a lot of manual intervention. In addition, the selection of sound effects and background music also lacks intelligent support, making it difficult to automatically generate suitable background sound effects based on the content and tone of the video, further increasing the complexity of video production.

[0006] Finally, although some platforms support automatic generation of videos from text or PPT, the generated videos often have problems with incoherent content and lack of professionalism, especially when dealing with financial information. The logic of the content, the accuracy of the data, the presentation of the charts, etc. often require financial professionals to repeatedly edit and revise the generated results, which consumes a lot of time and energy. Summary of the invention

[0007] The main purpose of the present invention is to provide a text-to-video method, device, equipment and storage medium based on deep semantic analysis, aiming to solve the technical problem that the prior art cannot intelligently match suitable visual elements, background music and sound effects based on semantic analysis of text, thereby automatically generating coherent and professional video content.

[0008] To achieve the above object, the present invention provides a text-to-video method based on deep semantic analysis, comprising:

[0009] Acquire a text to be converted, perform word segmentation and part-of-speech tagging on the text to be converted, and extract keywords and named entities from the text to be converted;

[0010] Analyzing the core theme and text structure of the text to be converted, wherein the text structure includes titles, subtitles and / or paragraphs;

[0011] Perform semantic analysis based on the keywords, named entities, text structure and core topics to generate semantic analysis results;

[0012] According to the semantic analysis result, a mapping relationship between semantics and visual elements is established, and images, video clips, sound effects and background music matching the semantic analysis result are retrieved from the material library;

[0013] Based on the semantic analysis results, spatially layout the text content corresponding to the keywords and named entities with the images and video clips to generate a preliminary visual layout framework;

[0014] Analyzing layout content features of the preliminary visual layout framework using a deep learning model;

[0015] generating an animation effect based on the layout content characteristics, and applying the animation effect to the text content and visual elements in the preliminary visual layout framework;

[0016] Construct a video timeline based on a preliminary visual layout framework with applied animation effects, background music and sound effects;

[0017] According to the video timeline, the preliminary visual layout framework, background music and sound effects are integrated to generate integrated complete video content.

[0018] Furthermore, to achieve the above-mentioned purpose, the present invention provides a text-to-video device based on deep semantic analysis, comprising:

[0019] A text processing module is used to obtain the text to be converted, perform word segmentation and part-of-speech tagging on the text to be converted, and extract keywords and named entities from the text to be converted;

[0020] A text analysis module, used for analyzing the core theme and text structure of the text to be converted, wherein the text structure includes titles, subtitles and / or paragraphs;

[0021] A semantic analysis module, used to perform semantic analysis based on the keywords, named entities, text structure and core topics, and generate semantic analysis results;

[0022] A material retrieval module is used to establish a mapping relationship between semantics and visual elements according to the semantic analysis results, and retrieve images, video clips, sound effects and background music that match the semantic analysis results from the material library;

[0023] A layout generation module, for spatially arranging the text content corresponding to the keywords and named entities, and the images and video clips based on the semantic analysis results, to generate a preliminary visual layout framework;

[0024] A layout feature analysis module, used to analyze layout content features of the preliminary visual layout framework using a deep learning model;

[0025] An animation effect generation module, used to generate animation effects based on the layout content characteristics, and apply the animation effects to the text content and visual elements in the preliminary visual layout framework;

[0026] A timeline building module for building a video timeline based on a preliminary visual layout framework with applied animation effects, background music and sound effects;

[0027] The video generation module is used to integrate the preliminary visual layout framework, background music and sound effects according to the video timeline to generate integrated complete video content.

[0028] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a text-to-video program based on deep semantic analysis stored in the memory and executable on the processor, wherein the text-to-video program based on deep semantic analysis, when executed by the processor, implements the steps of the text-to-video method based on deep semantic analysis as described above.

[0029] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which is stored a text-to-video program based on deep semantic analysis, and when the text-to-video program based on deep semantic analysis is executed by a processor, the steps of the text-to-video method based on deep semantic analysis as described above are implemented.

[0030] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a text-to-video method based on deep semantic analysis, which extracts keywords and named entities from the text to be converted, analyzes the core theme and text structure of the text, performs semantic analysis to generate semantic analysis results, retrieves matching images, video clips, sound effects and background music according to the semantic analysis results, performs spatial layout to generate a preliminary visual layout framework, generates animation effects using a deep learning model and applies them to the layout scheme, constructs a video timeline, integrates the layout scheme, background music and sound effects to generate complete video content, and finally outputs it as a video file. The present invention accurately extracts keywords, named entities, core themes and text structures in the text through semantic analysis, and intelligently matches images, video clips, sound effects and background music that are consistent with the semantics, avoiding the problem of requiring a large amount of manual intervention in the prior art. At the same time, adaptive animation effects are generated through a deep learning model and applied to the preliminary visual layout framework, ensuring the dynamic presentation of video content and the coherence of visual effects. After constructing the video timeline and integrating all elements, complete video content is automatically generated, which significantly improves the efficiency of video production and ensures professionalism and content consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0032] Figure 1 A schematic diagram of an application environment of a text-to-video method based on deep semantic analysis in one embodiment of the present invention;

[0033] Figure 2 It is a flowchart of an embodiment of a text-to-video method based on deep semantic analysis of the present invention;

[0034] Figure 3 It is a functional module diagram of a preferred embodiment of a text-to-video device based on deep semantic analysis of the present invention;

[0035] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0036] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0038] The text-to-video method based on deep semantic analysis provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In the application environment, the user end communicates with the server end through the network. The server end can extract keywords and named entities from the text to be converted through the user end, analyze the core theme and text structure of the text, perform semantic analysis to generate semantic analysis results, retrieve matching images, video clips, sound effects and background music according to the semantic analysis results, perform spatial layout to generate a preliminary visual layout framework, use the deep learning model to generate animation effects and apply them to the layout plan, build a video timeline, integrate the layout plan, background music and sound effects to generate complete video content, and finally output it as a video file. The present invention accurately extracts keywords, named entities, core themes and text structures in the text through semantic analysis, and intelligently matches images, video clips, sound effects and background music that are consistent with the semantics, avoiding the problem of requiring a large amount of manual intervention in the prior art. At the same time, adaptive animation effects are generated through the deep learning model and applied to the preliminary visual layout framework, ensuring the dynamic presentation of video content and the coherence of visual effects. After constructing the video timeline and integrating all elements, the complete video content is automatically generated, which significantly improves the efficiency of video production and ensures the consistency of professionalism and content. The user end may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The server end may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0039] See also Figure 2 , Figure 2 This is a flowchart of an embodiment of a text-to-video method based on deep semantic analysis provided by the present invention. It should be noted that although a logical sequence is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0040] like Figure 2 As shown, the text-to-video method based on deep semantic analysis proposed by the present invention includes the following steps:

[0041] S10, obtaining a text to be converted, performing word segmentation and part-of-speech tagging on the text to be converted, and extracting keywords and named entities from the text to be converted;

[0042] In this embodiment, the text to be converted can be in a variety of formats, such as ordinary TXT text files, PPT files, or structured or unstructured files such as financial reports. In order to perform subsequent word segmentation and semantic analysis, the system needs to extract and preprocess the input text data. The file input module can read files through manual upload, API interface, or from a specified path. The system supports file types in a variety of formats, and the content of files in different formats will be parsed during preprocessing, such as parsing PPT text content or extracting text parts from PDF files.

[0043] Word segmentation is to break down the text to be converted into the smallest language units (vocabulary) so that the system can recognize specific words in the text. Especially in Chinese, word segmentation is a very important operation. Part-of-speech tagging refers to adding corresponding part-of-speech tags to each word segmentation, such as nouns, verbs, adjectives, etc., to help subsequent semantic understanding. Word segmentation can use existing word segmentation algorithm tools, such as the Chinese jieba word segmenter or other open source natural language processing libraries. For terms in the financial field, a customizable word segmentation dictionary can be used to ensure accurate word segmentation of financial terms. Based on natural language processing technology (for example, using StanfordNLP or SpaCy), the text after word segmentation is tagged with parts of speech. The tagged vocabulary will be accompanied by corresponding part-of-speech tags for subsequent semantic analysis.

[0044] Keyword extraction is to identify the most representative and meaningful words from the text to help understand the core content of the text. Named entity recognition (NER) is used to detect proper nouns in the text, such as names of people, places, institutions, etc., which is particularly important in the financial field. Based on the results of word segmentation and part-of-speech tagging, use technologies such as TF-IDF (term frequency-inverse document frequency) or TextRank to extract keywords with higher weights in the text. Keywords can be sorted according to their frequency of occurrence and importance. Use a named entity recognition model (such as BERT, SpaCy, StanfordNER) to scan the text and identify proper nouns such as names of people, places, and institutions. Generate a list of entities with labels (such as names of people and places) based on the recognition results. Keywords and named entities will be stored in the system and used for semantic analysis and visual mapping in subsequent steps.

[0045] Example description: Taking the annual investment report in the financial field as an example, the system receives the input investment report document, automatically performs word segmentation on the financial terms (such as "price-earnings ratio" and "return on capital") and extracts keywords. At the same time, the named entity recognition module can accurately extract key information such as the company name, financial institution name, and geographic location in the report, providing semantic information for subsequent video generation. For example, the keyword "asset management" will match the relevant icons in the image library, and the institution name "JP Morgan" will automatically match the corresponding company logo and be used for display in the video, so that the generated video can more accurately convey professional content in the financial field.

[0046] Through the above steps, intelligent word segmentation, keyword extraction and named entity recognition of text data are realized, which greatly improves the accuracy of semantic analysis.

[0047] S20, analyzing the core theme and text structure of the text to be converted, wherein the text structure includes titles, subtitles and / or paragraphs;

[0048] In this embodiment, the core theme of the text to be converted is analyzed and the structural hierarchy of the text (such as title, subtitle, paragraph, etc.) is identified to provide logic and orderliness for subsequent video generation. The core theme analysis mainly identifies the main idea of ​​the text through semantic modeling technology, and the text structure analysis is to understand the hierarchical layout of the text, thereby providing a framework for matching visual elements and animation effects.

[0049] Based on the high-frequency keywords, named entities, contextual relationships, and syntactic structures in the text, the core theme of the text is extracted through a preset topic model (such as the LDA topic model or the BERT pre-trained model). The core theme is usually the main idea or purpose expressed in the text.

[0050] Identify structural information such as titles, subtitles, paragraphs, etc. in the text, and automatically distinguish different levels of text blocks based on features such as line spacing, font size, punctuation, etc. Titles usually have greater weight, while paragraphs are used to carry detailed content.

[0051] After analyzing the structure of titles, subtitles, paragraphs, etc., a hierarchical text structure diagram is constructed based on their content and logical relationships. This hierarchical structure will be used for subsequent visual element layout and animation effect generation.

[0052] Use a preset topic model (such as the LDA model) to extract topics. The model generates the main topics of the text based on the distribution of keywords and assigns corresponding topic weights to each paragraph or chapter. In the financial field, especially in annual reports or market analysis reports, common topics include "risk control", "asset allocation", "market outlook", etc. The system trains and matches these topics through the training corpus.

[0053] Based on the formatting information of the text (such as font size, bold style, etc.), the title and subtitle are identified and their content is extracted. The title is usually given a higher weight in the document. Based on the segmentation technology in natural language processing, the paragraphs in the text are identified. The paragraph is usually composed of multiple sentences, each paragraph expounds a small topic, and constitutes the overall framework of the document.

[0054] A hierarchical text structure is generated based on the identified titles, subtitles, and paragraphs. This structure will reflect the logical organization and content arrangement of the text. Different levels of text structures will correspond to different visual effects and display orders in subsequent video generation, and titles often have the highest weight.

[0055] Example description: When processing a financial annual report, the system can automatically identify "market analysis" as the core theme and extract relevant paragraphs and chart data. At the same time, the title "Market Performance in 2023" and subtitles "Stock Market", "Bond Market", etc. in the report will be recognized by the system to help build the logical hierarchy of the report. The generated video will reasonably arrange the visual content according to these hierarchical structures, such as showing relevant charts when playing "Stock Market Performance" and switching the visual background and animation effects when transitioning to "Bond Market Performance" to ensure the coherence and professionalism of the content.

[0056] Through core topic analysis and text structure recognition, we can not only improve the understanding of text content during video generation, but also ensure that the logical structure of the text can be mapped to the visual layout, so that the generated video content has logical consistency. The hierarchical structure of the text helps to display visual elements in an orderly manner, ensuring that the audience can understand the core information of the text more clearly, especially in the financial field, to ensure the orderliness and coherence of reports and analyses.

[0057] S30, performing semantic analysis based on the keywords, named entities, text structure and core topics to generate semantic analysis results;

[0058] In this embodiment, a semantic analysis result is generated by comprehensively analyzing the keywords, named entities, text structure and core themes in the text. This process is a key step in text understanding, which can ensure that the system accurately extracts the semantic relationship of the text so as to achieve the mapping of semantics and visual elements in subsequent steps. Keywords are representative words extracted from the text through natural language processing technology (such as TF-IDF, word frequency analysis). Named entities are specific categories of vocabulary extracted from the text through named entity recognition technology, such as names of people, places, and institutions.

[0059] Through association analysis technology, the semantic association between keywords and named entities is determined. This association can be inferred through context or syntactic structure. For example, keywords and named entities appear in the same paragraph, indicating that they may be semantically related.

[0060] The hierarchical relationship of text structures (such as titles, subtitles, and paragraphs) also plays an important role in semantic analysis. The system needs to analyze the semantic logic between the various parts of the content through the hierarchical information of the text structure. For example, the title may express the theme of the entire text, while the paragraphs provide detailed descriptions, and the subtitles serve as segmented descriptions. Text structure analysis helps the system determine the importance of certain keywords and named entities, further enhancing the accuracy of semantic analysis.

[0061] The core theme is the main idea derived from the topic analysis of the text. Each paragraph and sentence may be associated with one or more core themes. During the semantic analysis process, it is necessary to determine the degree of association between each paragraph and the core theme and generate the semantic association between the paragraph and the core theme. This helps the system understand the semantic emphasis of each paragraph, so as to arrange the visual elements reasonably in the subsequent video generation process.

[0062] The semantics of named entities and keywords depends not only on their own meaning, but also on their context. Context analysis determines the semantic meaning of named entities and keywords in a specific context by combining the context. This analysis ensures that the system can recognize subtle semantic differences.

[0063] The system first extracts keywords and named entities from the text. Then, through context analysis and syntactic analysis, it determines their relationship in the text. For example, if the keyword "market" and the named entity "Shanghai" often appear in the same paragraph, the system can infer that they have a certain semantic relationship.

[0064] The system analyzes the hierarchical structure of the text (such as title, subtitle, paragraph) to determine the semantic function of each part. For example, the title can be considered as the main semantic expression of the entire text, while the subtitle and paragraph play the role of refinement and expansion. The system further optimizes the semantic analysis based on this hierarchical relationship.

[0065] Through models such as LDA or BERT, the system matches paragraph content with core topics and generates semantic relevance between paragraphs and topics. This helps the system prioritize content that is more related to core topics when processing paragraphs. The system uses contextual analysis techniques (such as dependency-based syntactic analysis) to combine the context of the sentences in which keywords and named entities are located to determine their specific semantics.

[0066] Example description: In the application of finance, analyzing the core topics and text structure of the text to be converted can effectively help financial institutions realize intelligent content generation. Taking the financial report of a large commercial bank as an example, the text in the report may include the title, subtitle and multiple paragraphs of the bank's financial status. By performing word segmentation, keyword extraction and named entity recognition on the text, core financial terms such as "loan", "deposit", "net profit" and related named entities such as "such and such bank" and "regional branch" can be accurately identified.

[0067] Through comprehensive semantic analysis of keywords, named entities, text structure and core themes, the system can accurately capture the key information in the text and generate in-depth semantic analysis results. This semantic analysis can help the system generate scenes, backgrounds and visual effects that are highly relevant to the text content in the subsequent visual element matching process, thereby making the video content more coherent and logical, especially in the financial field, ensuring that data, charts and text content can be reasonably matched and displayed.

[0068] S40, establishing a mapping relationship between semantics and visual elements according to the semantic analysis result, and retrieving images, video clips, sound effects and background music that match the semantic analysis result from a material library;

[0069] In this embodiment, based on the results of semantic analysis, the system will associate keywords and named entities in the text with images, video clips, etc. in the visual material library through mapping rules between semantics and visual elements.

[0070] The system associates keywords such as "net profit" with chart materials; and associates named entities such as "such and such bank" with images of bank buildings. The system automatically learns the association between semantics and visual elements using preset mapping rules or through machine learning models. This step relies on the results of semantic analysis to ensure that each keyword and named entity has a corresponding visual material.

[0071] The system will automatically retrieve images and video clips from the material library based on the extracted keywords. For text in the financial field, such as "loan", the system may retrieve financial charts or data displays related to loans.

[0072] For keywords in the text, such as "growth" and "profit", an image of an upward arrow or a video of a financial growth chart will be matched. Based on the semantic information and context of the keyword, matching images and video clips are retrieved so that the visual material can accurately convey the text content.

[0073] Based on the named entity information, the system will retrieve images or video clips that match financial institutions, place names, etc. For example, when the system recognizes the named entity "such and such bank", it will retrieve relevant images or videos such as bank buildings and conference scenes from the material library. For example, the named entity "such and such bank" may correspond to a picture of a bank building; "financial regulatory agency" may match a formal meeting scene video. Combining the category of the named entity (such as a person, an institution) and the contextual semantic meaning, the system retrieves and selects the visual material that matches it.

[0074] The system retrieves visual materials that can express the logic of the text based on information such as titles, subtitles, and paragraph levels in the text structure. For example, the main title can be displayed using a full-screen image, the subtitle uses a simple background material, and the paragraph content is gradually expanded. For example, the title "Annual Financial Report" can be matched with a simple report cover image, and the paragraph content "Net Profit Analysis" may be matched with detailed financial tables or charts. Combined with the hierarchical structure of the text, the system automatically selects appropriate visual materials for each level of content to ensure that the visual expression matches the text structure.

[0075] Based on the emotional tone in the semantic analysis, the system selects background music and sound effects that match the mood of the text. For example, positive emotions (such as financial growth) will choose cheerful music, while neutral or negative emotions (such as losses) use calm or low background music. For example, financial growth may use fast-paced light music, while financial decline may use slow background music. The emotional tone of the text is determined by the sentiment analysis module, and the material that matches the emotion is automatically selected from the sound effect and background music library for matching.

[0076] By combining the results of semantic analysis with the mapping of visual elements, it is possible to intelligently match financial text content with images, video clips, sound effects, and background music in the visual material library, and automatically generate visual expressions that match the text content. This not only improves the efficiency of financial video generation, but also ensures the coherence and professionalism of the video content, reduces the need for human intervention, and effectively improves the quality and expressiveness of video production.

[0077] S50, based on the semantic analysis result, spatially layout the text content corresponding to the keywords and named entities and the images and video clips to generate a preliminary visual layout framework;

[0078] In this embodiment, the system spatially arranges the text information corresponding to the keywords and named entities with the matching images and video clips based on the results of semantic analysis. This layout determines the relative position, arrangement order and display form of the text and visual materials. For example, the keyword "net profit" may be placed in a prominent position on the screen and matched with related chart images to form a contrasting display. For example, the text of the named entity "such and such bank" will be displayed side by side with the image of the bank building or the related video clip, making the information expression intuitive and clear. Based on the principle of visual coordination, the system may align the text information to the left and the image or video clip to the right to ensure that the content is easy to understand and conforms to people's visual browsing habits.

[0079] The system will first parse the results of semantic analysis to determine the visual material corresponding to each keyword and named entity. Then, the system will spatially layout the text and visual materials according to the visual design rules. For example, important financial data may be highlighted in the middle or upper left of the screen, while background images or videos are used as auxiliary content to provide background information or a sense of context. Using the layout algorithm, a reasonable display area is calculated based on the text content and image size to ensure that the text and visual elements do not overlap and the information is clearly transmitted.

[0080] Through the above steps, the system generates a preliminary visual layout framework, which includes the spatial distribution of text information, images, and video clips on the screen. This plan is the basis for subsequent animation effect generation and video production.

[0081] The layout plan will list in detail the specific location, size and distance between each element (such as keywords, named entities, images, and videos). The system will determine the subsequent application of animation effects based on the preliminary layout plan to ensure visual fluency.

[0082] The system generates a complete layout plan based on the aforementioned layout strategy and converts it into specific layout coordinates and arrangement rules. The plan can be adjusted and optimized in subsequent steps to meet the needs of the video timeline, such as dynamically displaying text and images, and the time sequence of video clips.

[0083] By making reasonable spatial layout of keywords, text content of named entities and visual elements, it is possible to automatically generate a layout plan that conforms to visual design principles, reducing manual intervention. Especially in the financial field, it can intuitively present financial data and corresponding charts, simplify the complexity of information display, make the generated video both professional and able to quickly and efficiently convey information, and improve user experience and content quality.

[0084] S60, analyzing layout content features of the preliminary visual layout framework using a deep learning model;

[0085] In this embodiment, a deep learning model is used to analyze the initially generated visual layout framework. The layout content features refer to the visual attributes such as the spatial arrangement, size, color, level, contrast, etc. of each visual element in the layout. Through the analysis of the deep learning model, the layout scheme can be evaluated from the aspects of the visual aesthetics of the layout and the efficiency of information transmission, and provide a basis for the subsequent generation of animation effects.

[0086] Input the previously generated preliminary visual layout framework into the deep learning model. This solution includes the spatial layout information of text content, images, video clips, etc. on the screen. Use a pre-trained deep learning model (such as a CNN convolutional neural network) to extract and analyze the features of the visual layout framework. The model will extract important features in the layout, including but not limited to the contrast of visual elements, color matching, information communication path, etc.

[0087] The model evaluates the visual aesthetics of the layout by analyzing the spatial distribution and contrast relationship of elements, and gives optimization suggestions, such as ensuring that the visual focus is in a key position and avoiding excessive element accumulation that affects the user's viewing experience.

[0088] The model will generate an analysis report, pointing out the advantages and disadvantages in the layout, and provide a basis and guidance for the next step of animation effect generation.

[0089] Example description: In a video that generates an annual financial data analysis, the system uses a deep learning model to analyze layout features to ensure that the line chart and bar chart are arranged properly. With the help of the model, the system automatically optimizes the color matching of the chart to avoid visual conflicts in data visualization, while ensuring that text information (such as "income growth rate") is displayed in the optimal position. The model also recommends adjusting the size of the chart to make the core data clearer, greatly improving the communication effect and visual perception of the video content.

[0090] Through the layout analysis of the deep learning model, the visual aesthetics and information transmission efficiency of the video content can be effectively improved. The model can automatically identify unreasonable layout designs and give improvement suggestions, avoiding tedious manual adjustments.

[0091] S70, generating an animation effect based on the layout content feature, and applying the animation effect to the text content and visual elements in the preliminary visual layout framework;

[0092] In this embodiment, appropriate animation effects are generated based on the analyzed visual layout content features, and these effects are applied to the text content and visual elements in the layout. Animation effects can include fading in and out, scaling, moving, rotating, etc. of text or images, with the purpose of enhancing the expressiveness of the video and the efficiency of information transmission through appropriate dynamic effects, making the video more vivid and attracting the audience's attention.

[0093] Extract key information from the layout content features previously analyzed by the deep learning model, including visual features such as size, position, contrast, etc. of text, images or video clips.

[0094] Based on the analyzed layout features, appropriate animation effects are selected. For example, important visual elements can be highlighted through animation effects such as zooming in and out, while background elements may only require simple fade-in and fade-out effects.

[0095] Configure animation parameters for each visual element, including animation duration, start time, end time, transparency change, scaling, etc. The design of animation effects should match the layout features to make the video content consistent and smooth.

[0096] The generated animation effects are applied to the text and visual elements in the preliminary visual layout framework. The animation effects will be embedded in the timeline to ensure that the visual elements are displayed in the set order and rhythm when playing.

[0097] After the animation is applied, perform a preliminary preview of the video to check the smoothness and coordination of the animation effect. Fine-tune the animation effect as needed to ensure the best viewing experience of the video.

[0098] Example description: In a video that generates an annual financial report, the system designs animation effects for charts and text based on the visual layout characteristics of financial data. For example, the profit curve chart appears with a zoom animation when it enters the screen, while the net profit figure is highlighted through color changes and sliding effects. Through these animation effects, the video not only shows the dynamic changes of financial data, but also helps the audience quickly understand the key points of the data, greatly improving the visual effect of the video and the clarity of information transmission.

[0099] By generating animation effects based on layout content features, the system can automatically design the most appropriate dynamic effects for each visual element. This not only enhances the visual appeal of the video, but also guides the audience's attention to the key content and improves the accuracy of information transmission. Especially in financial data videos, the dynamic display of key data helps the audience quickly understand complex trend changes, thereby improving the professionalism and user experience of the video.

[0100] S80, constructing the video timeline based on the preliminary visual layout framework with applied animation effects, background music and sound effects;

[0101] In this embodiment, the animation effects, background music and sound effects in the generated preliminary visual layout framework are integrated together and reasonably arranged and synchronized based on the timeline. The timeline is the core structure of the entire video playback, which defines the start time, duration and end time of each element, ensuring that all visual and auditory elements are played in a predetermined time sequence to achieve smooth presentation of video content.

[0102] Create a video timeline framework to accommodate the various elements in the video, including text, images, video clips, animation effects, background music and sound effects. The length of the timeline is determined based on the total length of the video.

[0103] According to the preliminary visual layout framework, apply the animation effects of each visual element (such as text, images, video clips) to the timeline, and determine the start time, duration, and transition effect of each element. Text and images may require a shorter display time, while video clips may require a longer duration.

[0104] Embed background music into the timeline, adjust the start time and end time of the background music, and its synchronization with the animation effects to ensure that the background music is smoothly integrated with the video content and avoid conflicts or breaks between the background music and visual elements.

[0105] Place the sound effects at the corresponding positions in the timeline according to the events in the video (such as the appearance of animation effects). The triggering time of the sound effects needs to be consistent with the corresponding visual events to ensure audio-visual synchronization.

[0106] Preview the video's timeline to make sure the timing of all elements makes sense and flows smoothly. Make timing adjustments as needed, such as shortening or lengthening the duration of certain animation effects or rearranging the timing of sound effects triggering.

[0107] Example: When generating a quarterly financial report video, the system first constructs a timeline based on the order in which the financial data is presented, and plays the line chart animation of the company's revenue growth in sync with the background music. When key data such as net profit and gross profit are highlighted, the sound effects are also triggered during this period, allowing the audience to perceive key information both visually and aurally. This timeline construction ensures the smoothness of the video, avoids dislocation or interference between elements, and ultimately achieves a professional and easy-to-understand presentation effect.

[0108] By building a video timeline based on the initial visual layout framework, background music and sound effects with applied animation effects, the synchronization and coordination of all visual and auditory elements in the video are ensured, which improves the coherence and viewing experience of the video. The construction of this timeline can not only ensure the smooth playback of video content, but also adjust the arrangement of background music and sound effects according to the rhythm changes of different content, making the video more rhythmic and attractive, which is particularly suitable for the accurate presentation of financial reports and market analysis videos.

[0109] S90, integrating the preliminary visual layout framework, background music and sound effects according to the video timeline to generate integrated complete video content.

[0110] In this embodiment, based on the constructed video timeline, all elements (text content, images, video clips, etc.), background music and sound effects in the preliminary visual layout framework are finally integrated to ensure that each element is synchronously arranged according to the timeline and perfectly integrated. The goal of this step is to generate a video content with coordinated, unified and smooth visual and auditory effects.

[0111] Based on the constructed video timeline, arrange the text content, images and video clips in the preliminary visual layout framework according to the order and duration in the timeline. Make sure that each visual element is displayed synchronously according to the timeline, for example, the text content appears at a specific time point, and the video clips and images are played in sequence.

[0112] Integrate animation effects related to visual elements into the video timeline. Animation effects are applied to text, images, video clips, etc., and the triggering time and duration of the animation are arranged according to the timeline. For example, text slides in from left to right, and video clips fade in and display, and arrange their start and end times on the timeline.

[0113] Embed the background music into the video timeline and make sure the start time of the background music is synchronized with the beginning of the video. According to the timeline, adjust the duration of the background music to match the rhythm of the video content. If the video timeline is longer than the background music, you can choose to loop the music or adjust the rhythm and length of the music clip.

[0114] Add sound effects to the timeline based on key events in the video (such as the triggering of animation effects and the switching of video clips). Make sure that the sound effects are perfectly synchronized with the corresponding visual events (such as image or video switching) to enhance the presentation of important information in the video.

[0115] After all elements are integrated, preview the entire video to check whether the elements are playing synchronously according to the timeline. If inconsistencies are found (such as delayed animation effects or unsynchronized sound effects), make fine adjustments according to the timeline to ensure that the final integrated video plays smoothly.

[0116] Example description: When generating an annual financial summary video, the system first integrates the company's financial statement animation, background music, and text content of financial analysis according to the timeline. The background music appropriately enhances the atmosphere of the video when the report data changes, and the switching of key performance indicators (such as revenue and net profit) is accompanied by appropriate sound effects, allowing the audience to quickly grasp the key information in a combination of audio and video. This integration method ensures the professionalism and fluency of the final video content, allowing the video to effectively convey complex financial information in a short period of time.

[0117] By integrating the preliminary visual layout framework, background music and sound effects according to the video timeline, the final generated complete video content can achieve a high degree of coordination and unity of all elements in the video, avoiding playback problems caused by the incoordination between different elements. This integration method makes the video display effect smoother and more professional, especially when displaying complex content such as financial data or industry analysis, the video can present a highly consistent visual and auditory experience, greatly improving the user's viewing experience and the efficiency of information understanding.

[0118] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a text-to-video method based on deep semantic analysis, which extracts keywords and named entities from the text to be converted, analyzes the core theme and text structure of the text, performs semantic analysis to generate semantic analysis results, retrieves matching images, video clips, sound effects and background music according to the semantic analysis results, performs spatial layout to generate a preliminary visual layout framework, generates animation effects using a deep learning model and applies them to the layout scheme, constructs a video timeline, integrates the layout scheme, background music and sound effects to generate complete video content, and finally outputs it as a video file. The present invention accurately extracts keywords, named entities, core themes and text structures in the text through semantic analysis, and intelligently matches images, video clips, sound effects and background music that are consistent with the semantics, avoiding the problem of requiring a large amount of manual intervention in the prior art. At the same time, adaptive animation effects are generated through a deep learning model and applied to the preliminary visual layout framework, ensuring the dynamic presentation of video content and the coherence of visual effects. After constructing the video timeline and integrating all elements, complete video content is automatically generated, which significantly improves the efficiency of video production and ensures professionalism and content consistency.

[0119] In one embodiment, the above S10 includes:

[0120] S101, obtaining a text to be converted, performing word segmentation processing on the text to be converted using a natural language processing module, and generating a word segmentation result;

[0121] S102, performing part-of-speech tagging on the word segmentation results to generate a part-of-speech tagging result to identify the part-of-speech category of each word segmentation vocabulary;

[0122] S103, extracting keywords from the word segmentation results based on the part-of-speech tagging results;

[0123] S104, analyzing the relationship network between the segmentation words, determining the importance of the segmentation words based on the position and connectivity of the segmentation words in the relationship network, and sorting the keywords based on the importance;

[0124] S105, using a named entity recognition module, identifying named entities including names of people, places and / or institutions from the text to be converted, and generating a named entity list with entity category labels.

[0125] In this embodiment, the original text data to be processed is obtained. This text can be a text file in the format of a financial report, market analysis, etc. The file input module can read the text to be converted through manual upload, API interface or from a specified database. A variety of file formats are supported, including TXT, PPT, PDF, etc. The text preprocessing module will parse the file format, extract the text therein and store it for subsequent operations.

[0126] Word segmentation is the process of dividing continuous text into the smallest semantic units. Especially in Chinese, word segmentation is crucial to understanding the text. Use natural language processing (NLP) tools such as Jieba word segmentation or word segmentation tools specially customized for the financial field. During the word segmentation process, the system will identify and segment each word in the text and generate a series of word segmentation results. The system supports custom dictionaries to ensure that special words such as financial terms and company names are correctly identified.

[0127] Part-of-speech tagging is to add a part-of-speech tag in the sentence to each word after segmentation, such as noun, verb, adjective, etc. Use natural language processing libraries (such as StanfordNLP, SpaCy) to tag the results after segmentation. The system uses a pre-trained model for tagging and generates a result table containing words and their corresponding part-of-speech tags. Special financial terms, company names, etc. will be accurately tagged according to a special part-of-speech tagging dictionary.

[0128] By analyzing the results of part-of-speech tagging, the most representative and important keywords are extracted from the text. Keywords are extracted from the part-of-speech tagging results using a keyword extraction algorithm (such as TF-IDF or TextRank). Keywords are extracted based on the frequency of occurrence of words in the text and their position in the lexical network. The system sorts the extracted keywords to ensure that the most important keywords are ranked first.

[0129] By building a semantic relationship network between segmented words, the connection strength and relationship between words are determined, and the importance of keywords is ranked. Use graph algorithms (such as PageRank) to build a vocabulary relationship network and analyze the connectivity of words in the segmented results. By calculating the importance of words in the relationship network, its representativeness in the entire text is determined. Sort the keywords by importance and give priority to the most representative words as keywords.

[0130] Named Entity Recognition (NER) is used to identify specific categories of entities from text, such as person names, place names, company names, etc., and assign corresponding category labels to them. Use an NER model (such as BERT, SpaCy, StanfordNER) to scan the text and identify the named entities in it. Determine the category of the entity (such as person name, company name, etc.) according to the context, and add a category label to it. Generate a list containing the named entities and their category labels, and the information in this list will be used in subsequent steps for semantic analysis and material matching.

[0131] Through steps such as word segmentation, part-of-speech tagging, keyword extraction, and named entity recognition in this embodiment, the system can accurately extract key information from financial text. This provides a semantic basis for subsequent semantic analysis and video generation, greatly improving the accuracy of text understanding, especially in the recognition of financial terms, reducing the workload of manual processing, and enhancing the efficiency and professionalism of video generation.

[0132] In one embodiment, the above S20 includes:

[0133] S201, preprocess the text to be converted, and the preprocessing includes removing stop words and punctuation marks;

[0134] S202, perform a theme analysis on the text to be converted based on a preset theme model, and extract key sentences and high-frequency words whose occurrence frequencies exceed a preset threshold in the text to be converted;

[0135] S203, generate one or more core themes based on the key sentences and high-frequency words;

[0136] S204, use a text structure analysis module to identify the title, subtitle, and / or paragraphs in the text to be converted;

[0137] S205, analyze the hierarchical relationship between the title, subtitle, and / or paragraphs, and generate a hierarchical text structure.

[0138] In this embodiment, before text analysis, the system will preprocess the text to be converted, clean up irrelevant content, such as stop words (such as "de", "le") and punctuation marks, which helps to improve the efficiency and accuracy of subsequent analysis. Use a predefined stop word list and regular expressions to clean the stop words and punctuation marks in the text. Save the cleaned text content for subsequent theme analysis and text structure analysis.

[0139] Through topic modeling, the system can identify the main theme of the text and extract representative sentences and high-frequency words, which can help the system understand the core meaning of the text. Use topic model algorithms, such as LDA (Latent Dirichlet Allocation) or LSA (Latent Semantic Analysis), to extract topics from text content. Extract key sentences that reflect the theme in the text. Use word frequency analysis algorithms to count the words in the text and extract high-frequency words whose frequency exceeds the set threshold.

[0140] Based on the key information extracted from the topic analysis, the system is able to generate core topics that describe the main content of the text. These topics provide a basis for subsequent visual content generation and matching. The system classifies and clusters the extracted key sentences and high-frequency words and generates one or more core topics. The core topics will be labeled and saved for subsequent use, such as in material library retrieval and visual element matching.

[0141] By analyzing the text structure, the system can identify titles, subtitles, and paragraphs in the text, helping to understand the hierarchical structure of the text and provide logical organization for subsequent video generation. Using the text analysis module, structural elements in the text are identified, such as titles (H1-H3 levels), subtitles, and paragraph markers. The identified structural elements are hierarchically annotated and saved as a tree structure.

[0142] By analyzing the hierarchical structure of the text, the system can organize the content of the text into different levels to ensure the logic and coherence of the content when the video is generated. Based on the extracted titles, subtitles and paragraphs, a hierarchical text structure is constructed. The system associates these text structures with thematic information to provide reference for the subsequent logical layout of the video content.

[0143] This embodiment can accurately extract the core theme of the text by preprocessing, subject analysis and structured hierarchical analysis of the text, and build a clear text hierarchy by analyzing hierarchical information such as titles, subtitles, paragraphs, etc. This not only improves the accuracy of text semantic analysis, but also provides a logical organization for subsequent video content generation, ensuring that the generated video is more coherent and logically clear when presenting information, avoiding the problems of content fragmentation and chaotic structure in the prior art.

[0144] In one embodiment, the above S30 includes:

[0145] S301, analyzing the association relationship between the keyword and the named entity, and determining the semantic relationship between the keyword and the named entity based on the context and syntactic structure;

[0146] S302, analyzing semantic logic among titles, subtitles and / or paragraphs based on the hierarchical information of the text structure;

[0147] S303, based on the matching analysis between the core theme and the paragraph content, determining the relevance of each paragraph to the core theme;

[0148] S304, performing context analysis on the named entity to determine the occurrence frequency and contextual semantic meaning of the named entity in the text to be converted;

[0149] S305: Generate a comprehensive semantic analysis result based on the semantic relationship between the keywords and the named entities, the relevance between the paragraphs and the core topics, the occurrence frequency of the named entities in the text to be converted, and the contextual semantic meaning.

[0150] In this embodiment, a semantic association is established between keywords and named entities in the text to help the system understand the relationship between key concepts and entities in the text. Context and syntactic structure are used to further enhance the accuracy of this association. The system uses natural language processing technology (such as dependency analysis and syntax tree analysis) to analyze the position and structure of keywords and named entities in sentences. The number of co-occurrences of keywords and named entities and the part-of-speech relationship between them are captured through the context window to determine the semantic association between the two. A semantic graph is generated to display keywords and named entities in the form of a network graph, and the connection between nodes represents the semantic relationship.

[0151] By analyzing the relationship between titles, subtitles, and paragraphs, the system can ensure that it can accurately understand the logical structure of the text based on hierarchical information. Using the hierarchical text analysis module, the titles, subtitles, and paragraphs in the text are annotated and a tree structure is established. The system analyzes the contextual relationship between each level and identifies the relationship between high-level titles and low-level content. Through the continuity analysis between paragraphs, the content of each part is ensured to maintain logical consistency.

[0152] In order to ensure that the paragraph content in the text is closely related to the core topic, the system needs to use a semantic matching algorithm to determine the degree of relevance of each paragraph to the extracted core topic. Using the topic matching algorithm, the extracted core topic and paragraph content are analyzed for similarity matching. The system calculates the matching degree between the paragraph and the core topic based on the distribution and frequency of the topic words, and generates a relevance score between the paragraph and the topic. The generated results are used to prioritize and layout the content of the paragraph during the video generation process.

[0153] By analyzing the frequency of named entities in the text and the context in which they appear, the system can understand the importance of named entities in the text and the specific semantics they represent. The system scans the frequency of occurrence of named entities throughout the text and calculates the specific semantic differences of the entities in different contexts. Combined with contextual information, such as keywords and grammatical structures before and after the named entity, its specific meaning in different contexts is determined. For example, "bank" can refer to "financial institution" or "river bank" in different contexts. Based on the results of the context analysis, the system assigns semantic labels to each named entity and saves the results for subsequent visual element mapping.

[0154] Integrate all the previous semantic analysis results to generate the final comprehensive semantic analysis results to ensure that the semantic information of the text can be completely and accurately extracted and understood. The system summarizes the results of various analysis modules such as keywords, named entities, text structure, core themes, paragraph matching, etc. Through comprehensive semantic analysis, all analysis results are integrated into a detailed semantic map to show the semantic relationship between keywords, named entities, paragraphs, core themes and other elements. The generated comprehensive semantic analysis results provide a basis for subsequent visual element mapping.

[0155] This embodiment improves the accuracy of text semantic understanding through semantic association analysis between keywords and named entities, matching analysis between paragraphs and core topics, and contextual analysis of named entities. Especially when dealing with complex reports in the financial field, the system can accurately identify core topics, key sentences, and important named entities, ensuring that the semantic logic of video generation is clear and coherent, reducing the need for manual intervention. At the same time, the generated semantic analysis results provide high-quality data support for subsequent visual element matching, improving the professionalism and efficiency of video generation.

[0156] In one embodiment, the above S40 includes:

[0157] S401, based on the keyword information in the semantic analysis result and in combination with the mapping rules between the keywords and the visual elements, images or video clips matching the keywords are retrieved from the material library;

[0158] S402, based on the named entity information in the semantic analysis result, combined with the category and contextual semantic meaning of the named entity, retrieve the image or video clip matching the named entity from the material library;

[0159] S403, based on the text structure hierarchy information in the semantic analysis result, searching for images or video clips matching the text structure;

[0160] S404, based on the core theme in the semantic analysis result, selecting a scene background image or scene background video clip that matches the core theme;

[0161] S405, based on the emotional tone in the semantic analysis result, selecting background music and sound effects that match the emotional tone.

[0162] In this embodiment, the system identifies keywords in the text (such as "financial market", "stock price", etc.) through semantic analysis, and matches images or video clips that are semantically related to the keywords according to the pre-set mapping rules between keywords and visual elements. This mapping can be achieved through the relevance of keywords and visual element labels or topics. The system extracts keywords from the semantic analysis results and matches the labeled visual elements in the material library in combination with the mapping rules. For example, the keyword "stock" may be associated with real-time stock price images of the financial market or video clips of exchange scenes. The retrieved images or video clips are sorted according to the matching relevance scores, and the most relevant elements are selected first.

[0163] Named entities (such as "a bank" and "a company") represent different entity categories in different contexts. By combining the categories of named entities (such as financial institutions and enterprises) and contextual semantics, the system can retrieve images or video clips that match the entity from the material library. The system analyzes the category labels of named entities, such as "institutions", "companies", "people", etc., and combines the semantic meaning of the context to filter out visual materials that match the category and context. For example, for the named entity "a bank", the system will retrieve images related to banking business, such as bank buildings, financial product advertisements, etc. The selected images or video clips give priority to matching the specific context of the named entity.

[0164] The structure of the text (such as titles, subtitles, and paragraphs) represents the hierarchy and logical relationship of the content. Based on this hierarchical information, the system retrieves images or video clips that match each text hierarchy from the material library for hierarchical display of the text content. The system retrieves materials based on the hierarchical information of the text structure. For example, for a title like "Market Analysis", the system can retrieve market dynamics charts or news clips related to the topic. The level of the text structure determines the importance and display order of the visual material. For example, the image that matches the title will be displayed at the beginning of the video or when the scene switches, while the paragraph content may be embedded in the middle of the video.

[0165] The core theme is the key content in the text and represents the central idea of ​​the text. The system selects appropriate background images or scene videos to emphasize the theme by semantically matching the core theme. The system matches the background materials in the material library based on the core theme. For example, if the core theme is "financial innovation", the system may choose a modern office environment or a scene with a strong sense of technology as the background. The background image or video clip will be displayed as the main visual element to enhance the theme coherence of the video.

[0166] The emotional tone of the text (such as positive, tense, serious, etc.) can be identified through semantic analysis. Based on this tone, the system selects background music and sound effects that match the emotional tone from the material library to create an audio-visual atmosphere that matches the emotion of the text. The system first identifies the emotional tone of the text and selects music or sound effects with corresponding emotions from the material library. For example, for "rigorous financial reports", low and solemn background music may be selected; while for the positive content in "market analysis", brisk sound effects may be selected. The selected music and sound effects will run through the entire video generation process and will be adjusted according to the emotional needs of different scenes to enhance the audience's emotional experience.

[0167] Example description: Taking an annual report on the financial market as an example, the system first extracts the keywords "stock market fluctuations" and the named entity "a bank" from the text, and maps the keywords and named entities to corresponding visual elements, such as video clips of stock market trading scenes and images of bank buildings. At the same time, based on the core theme of the text "market trends", the system matches relevant market trend charts as the main visual content of the report. In addition, through the analysis of the emotional tone, the system selects background music with a faster rhythm to create a positive market outlook atmosphere. This integrated process ensures that the generated video content not only accurately conveys the key information in the report, but also has a high visual and auditory appeal.

[0168] This embodiment improves the accuracy and professionalism of text-to-video conversion by effectively mapping the semantic analysis results with visual elements, sound effects, and background music. Through multi-level mapping of keywords, named entities, text structures, and core themes, the system can generate video content that is more consistent with the semantic logic of the text, reducing the user's manual intervention and improving the efficiency of video generation. At the same time, by matching the emotional tone with the background music, the generated video can better convey the emotional atmosphere in the text and enhance the viewing experience.

[0169] In one embodiment, the above S90 includes:

[0170] S901, according to the video timeline, synchronizing the images, video clips and animation effects in the preliminary visual layout framework with the video timeline to determine the display order, duration and transition effect of the images, video clips and animation effects;

[0171] S902, adjusting the start time and duration of the background music according to the video timeline;

[0172] S903, associating the sound effect with the corresponding animation effect, and setting the triggering time and duration of the sound effect;

[0173] S904, integrating the visual layout framework, background music and sound effects to generate complete video content that has been synchronized.

[0174] In this embodiment, during the video generation process, the system will coordinate and synchronize the images, video clips and animation effects in the preliminary visual layout framework in time according to the pre-constructed video timeline. This step ensures that each visual element is displayed in the correct order, time and transition effect to avoid conflicts or incoherence between visual elements. The timeline construction module defines the display time and order of each visual element based on the generated semantic analysis results and layout scheme. Images and video clips will be played in the set order in the timeline, and the system will automatically adjust the display order according to the duration and transition effect of the elements. For example, picture 1 is displayed at a certain moment, and then switched to video clip 1 at a subsequent moment, and transition effects such as gradients are applied when switching. Animation effects are synchronized with images and video clips, such as superimposing text animations when images appear, or adding special effects animations during video playback.

[0175] The playback time and duration of background music need to be adjusted according to the timeline of the video to ensure that the music is synchronized with the presentation of images and video clips in the video, while avoiding the situation where the background music does not match the video picture. The background music is set to a time point on the video timeline to start playing, and the system sets the duration of the background music based on the overall length and content of the video. The system automatically detects each scene in the video and adjusts the timing of the music according to the emotional tone to ensure that the key content in the video is consistent with the rhythm of the music. For example, when important images or scenes appear, the rhythm of the background music will gradually increase.

[0176] The system binds specific sound effects to animation effects in the video to ensure that the sound effects are triggered at the right moment and that the duration of the sound effects matches the length of the animation effects. Sound effect triggers set the timing of the sound effects based on the timing of the animation effects. For example, at the beginning of an animation, the sound effects associated with the animation are triggered. The system automatically adjusts the duration of the sound effects to ensure that they are consistent with the animation effects. For example, the animation of an icon flying in is accompanied by a short sound effect, while there is no sound effect interference when the icon stays. The system selects a suitable sound effect library based on the characteristics of each animation effect and applies the relevant sound effects.

[0177] After all visual elements, background music and sound effects have been synchronized through the timeline, the system integrates all elements into a complete video, ensuring that the various parts of the video can be seamlessly connected, and finally output high-quality video content. The integration module integrates all processed images, video clips, animation effects, background music and sound effects into a continuous video stream according to the synchronization settings of the video timeline, background music and sound effects. The system integrates the order and duration of all elements together, and synchronizes the audio and video content to ensure that there are no errors and lags during playback. The integrated complete video content will be output in a standard playable format with complete visual effects, animation effects, sound effects and background music.

[0178] Through the above steps, this embodiment realizes the effective synchronization and integration of the preliminary visual layout framework, background music and sound effects with the video timeline, ensuring the coherence and high-quality display of the video content. It can accurately control the playback order and time of images, video clips, animation effects, background music and sound effects, so that the final generated video content has a strong visual impact and auditory synchronization, reducing the workload of users manually adjusting the timeline and materials, and significantly improving the efficiency and quality of video production.

[0179] In one embodiment, after the above S90, the method further includes:

[0180] S905, rendering the integrated video content, combining all visual elements and audio elements frame by frame;

[0181] S906, using a video encoding module to compress the rendered video content;

[0182] S907, generating a final video file, and outputting the video file in a preset video format.

[0183] In this embodiment, rendering is to combine all the visual and audio elements synchronized with the timeline frame by frame to generate the final video content. During the rendering process, the system will integrate all elements such as images, video clips, animation effects, background music and sound effects into a continuous video stream. The rendering module will process all the visual and audio elements in the video frame by frame starting from the first time point of the timeline to ensure that each frame of video accurately matches its corresponding background music and sound effects. The system will render frame by frame according to the set resolution and frame rate, and combine the images, videos, animation effects, text, background music, sound effects, etc. contained in each frame. After the rendering process is completed, the output is a complete video file containing all visual and audio elements for subsequent compression and encoding.

[0184] After rendering, the video file is usually large in size, so it is necessary to reduce the file size through encoding and compression processing for storage and transmission. The video encoding module is responsible for compressing the rendered video content while maintaining the video quality. The system inputs the rendered original video content into the encoding module and compresses it according to the preset encoding standard (such as H.264, H.265, etc.). During the encoding process, the system analyzes the changes between each frame in the video and reduces the file size by removing redundant data. For example, for video segments with infrequent scene switching, the encoder can reduce the number of redundant frames through inter-frame compression technology. The compressed video will take up less storage space while maintaining a certain picture quality and sound quality, which is suitable for subsequent playback and distribution.

[0185] After the compression process is completed, the system will generate the final video file and output it in the preset video format specified by the user. This step ensures that the generated video file can be played back on various playback devices or platforms. The system encapsulates the compressed video file into a standard playable video format, such as MP4, AVI, MKV, etc. The specific format is selected according to user needs or the requirements of the playback platform. During the encapsulation process, the system will format the video and audio to ensure that the final output file contains complete audio and video streams and can be played normally on common media players. The output video file will be saved in the path specified by the user or transferred to an external storage platform through the API interface for further use or distribution.

[0186] This embodiment effectively optimizes the production and playback performance of the video by rendering, compressing and outputting the integrated video content frame by frame into a preset format. The system can efficiently process a large number of visual and audio elements to ensure that the generated video content is smooth and high-quality. At the same time, through video encoding and compression, the size of the video file is significantly reduced, and the efficiency of storage and transmission is improved. The final output video format is diversified to ensure strong compatibility and meet the needs of various playback devices and platforms.

[0187] In one embodiment, a text-to-video device based on deep semantic analysis is provided, and the text-to-video device based on deep semantic analysis corresponds to the text-to-video method based on deep semantic analysis in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a text-to-video device based on deep semantic analysis of the present invention. Text processing module 10, text analysis module 20, semantic analysis module 30, material retrieval module 40, layout generation module 50, layout feature analysis module 60, animation effect generation module 70, timeline construction module 80 and video generation module 90. The functional modules are described in detail as follows:

[0188] The text processing module 10 is used to obtain the text to be converted, perform word segmentation and part-of-speech tagging on the text to be converted, and extract keywords and named entities from the text to be converted;

[0189] A text analysis module 20, for analyzing the core theme and text structure of the text to be converted, wherein the text structure includes titles, subtitles and / or paragraphs;

[0190] A semantic analysis module 30, configured to perform semantic analysis based on the keywords, named entities, text structure and core topics, and generate semantic analysis results;

[0191] A material retrieval module 40 is used to establish a mapping relationship between semantics and visual elements according to the semantic analysis result, and retrieve images, video clips, sound effects and background music that match the semantic analysis result from the material library;

[0192] A layout generation module 50 is used to spatially layout the text content corresponding to the keywords and named entities and the images and video clips based on the semantic analysis results to generate a preliminary visual layout framework;

[0193] A layout feature analysis module 60, configured to analyze layout content features of the preliminary visual layout framework using a deep learning model;

[0194] Animation effect generation module 70, used to generate animation effects based on the layout content characteristics, and apply the animation effects to the text content and visual elements in the preliminary visual layout framework;

[0195] A timeline construction module 80 for constructing a video timeline based on a preliminary visual layout framework with animation effects applied, background music and sound effects;

[0196] The video generation module 90 is used to integrate the preliminary visual layout framework, background music and sound effects according to the video timeline to generate integrated complete video content.

[0197] In one embodiment, the text processing module 10 is specifically used for:

[0198] Acquire the text to be converted, and use the natural language processing module to perform word segmentation processing on the text to be converted to generate a word segmentation result;

[0199] Performing part-of-speech tagging on the word segmentation results to generate a part-of-speech tagging result to identify the part-of-speech category of each word segmentation vocabulary;

[0200] Based on the part-of-speech tagging result, extract keywords from the word segmentation result;

[0201] Analyze the relationship network between the segmentation words, determine the importance of the segmentation words based on the position and connectivity of the segmentation words in the relationship network, and sort the keywords based on the importance;

[0202] A named entity recognition module is used to identify named entities including names of persons, places and / or institutions from the text to be converted, and a named entity list with entity category labels is generated.

[0203] In one embodiment, the text analysis module 20 is specifically used for:

[0204] Preprocessing the text to be converted, wherein the preprocessing includes removing stop words and punctuation marks;

[0205] Performing topic analysis on the text to be converted based on a preset topic model, extracting key sentences and high-frequency words whose occurrence frequency exceeds a preset threshold in the text to be converted;

[0206] Based on the key sentences and high-frequency words, generate one or more core topics;

[0207] Using a text structure analysis module to identify titles, subtitles and / or paragraphs in the text to be converted;

[0208] The hierarchical relationship between the titles, subtitles and / or paragraphs is analyzed to generate a hierarchical text structure.

[0209] In one embodiment, the semantic analysis module 30 is specifically used for:

[0210] Analyzing the association relationship between the keywords and the named entities, and determining the semantic relationship between the keywords and the named entities based on the context and syntactic structure;

[0211] Analyzing semantic logic among titles, subtitles and / or paragraphs based on hierarchical information of the text structure;

[0212] Based on the matching analysis between the core theme and the paragraph content, determine the degree of relevance of each paragraph to the core theme;

[0213] Conduct context analysis on named entities to determine the frequency of occurrence and contextual semantic meaning of named entities in the text to be converted;

[0214] Based on the semantic relationship between the keywords and the named entities, the relevance of the paragraphs to the core topics, the frequency of occurrence of the named entities in the text to be converted, and the contextual semantic meaning, a comprehensive semantic analysis result is generated.

[0215] In one embodiment, the material retrieval module 40 is specifically used to:

[0216] Based on the keyword information in the semantic analysis results and the mapping rules between keywords and visual elements, images or video clips matching the keywords are retrieved from the material library;

[0217] Based on the named entity information in the semantic analysis results, combined with the category and contextual semantic meaning of the named entity, images or video clips matching the named entity are retrieved from the material library;

[0218] Based on the text structure hierarchical information in the semantic analysis results, images or video clips matching the text structure are retrieved;

[0219] Based on the core theme in the semantic analysis result, a scene background image or scene background video clip matching the core theme is selected;

[0220] Based on the emotional tone in the semantic analysis results, background music and sound effects that match the emotional tone are selected.

[0221] In one embodiment, the video generation module 90 is specifically configured to:

[0222] According to the video timeline, synchronizing the images, video clips and animation effects in the preliminary visual layout framework with the video timeline to determine the display order, duration and transition effect of the images, video clips and animation effects;

[0223] Adjusting the start time and duration of the background music according to the video timeline;

[0224] Associating the sound effect with the corresponding animation effect, and setting the triggering time and duration of the sound effect;

[0225] The visual layout framework, background music and sound effects are integrated to generate complete video content that has been synchronized.

[0226] In one embodiment, the video generation module 90 is specifically configured to:

[0227] Rendering the integrated video content, combining all visual and audio elements frame by frame;

[0228] Use the video encoding module to compress the rendered video content;

[0229] A final video file is generated and outputted in a preset video format.

[0230] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the service side of a text-to-video method based on deep semantic analysis.

[0231] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the user side of a text-to-video method based on deep semantic analysis

[0232] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0233] Acquire a text to be converted, perform word segmentation and part-of-speech tagging on the text to be converted, and extract keywords and named entities from the text to be converted;

[0234] Analyzing the core theme and text structure of the text to be converted, wherein the text structure includes titles, subtitles and / or paragraphs;

[0235] Perform semantic analysis based on the keywords, named entities, text structure and core topics to generate semantic analysis results;

[0236] According to the semantic analysis result, a mapping relationship between semantics and visual elements is established, and images, video clips, sound effects and background music matching the semantic analysis result are retrieved from the material library;

[0237] Based on the semantic analysis results, spatially layout the text content corresponding to the keywords and named entities with the images and video clips to generate a preliminary visual layout framework;

[0238] Analyzing layout content features of the preliminary visual layout framework using a deep learning model;

[0239] generating an animation effect based on the layout content characteristics, and applying the animation effect to the text content and visual elements in the preliminary visual layout framework;

[0240] Construct a video timeline based on a preliminary visual layout framework with applied animation effects, background music and sound effects;

[0241] According to the video timeline, the preliminary visual layout framework, background music and sound effects are integrated to generate integrated complete video content.

[0242] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0243] Acquire a text to be converted, perform word segmentation and part-of-speech tagging on the text to be converted, and extract keywords and named entities from the text to be converted;

[0244] Analyzing the core theme and text structure of the text to be converted, wherein the text structure includes titles, subtitles and / or paragraphs;

[0245] Perform semantic analysis based on the keywords, named entities, text structure and core topics to generate semantic analysis results;

[0246] According to the semantic analysis result, a mapping relationship between semantics and visual elements is established, and images, video clips, sound effects and background music matching the semantic analysis result are retrieved from the material library;

[0247] Based on the semantic analysis results, spatially layout the text content corresponding to the keywords and named entities with the images and video clips to generate a preliminary visual layout framework;

[0248] Analyzing layout content features of the preliminary visual layout framework using a deep learning model;

[0249] generating an animation effect based on the layout content characteristics, and applying the animation effect to the text content and visual elements in the preliminary visual layout framework;

[0250] Construct a video timeline based on a preliminary visual layout framework with applied animation effects, background music and sound effects;

[0251] According to the video timeline, the preliminary visual layout framework, background music and sound effects are integrated to generate integrated complete video content.

[0252] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0253] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0254] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0255] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A text-to-video method based on deep semantic analysis, characterized in that: The following steps are involved: Acquire a text to be converted, perform word segmentation and part-of-speech tagging on the text to be converted, and extract keywords and named entities from the text to be converted; Analyzing the core theme and text structure of the text to be converted, wherein the text structure includes titles, subtitles and / or paragraphs; Perform semantic analysis based on the keywords, named entities, text structure and core topics to generate semantic analysis results; According to the semantic analysis result, a mapping relationship between semantics and visual elements is established, and images, video clips, sound effects and background music matching the semantic analysis result are retrieved from the material library; Based on the semantic analysis results, spatially layout the text content corresponding to the keywords and named entities with the images and video clips to generate a preliminary visual layout framework; Analyzing layout content features of the preliminary visual layout framework using a deep learning model; generating an animation effect based on the layout content characteristics, and applying the animation effect to the text content and visual elements in the preliminary visual layout framework; Construct a video timeline based on a preliminary visual layout framework with applied animation effects, background music and sound effects; According to the video timeline, the preliminary visual layout framework, background music and sound effects are integrated to generate integrated complete video content.

2. The text-to-video method based on deep semantic analysis according to claim 1, characterized in that: Acquiring a text to be converted, performing word segmentation and part-of-speech tagging on the text to be converted, and extracting keywords and named entities from the text to be converted, including: Acquire the text to be converted, and use the natural language processing module to perform word segmentation processing on the text to be converted to generate a word segmentation result; Performing part-of-speech tagging on the word segmentation results to generate a part-of-speech tagging result to identify the part-of-speech category of each word segmentation vocabulary; Based on the part-of-speech tagging result, extract keywords from the word segmentation result; Analyze the relationship network between the segmentation words, determine the importance of the segmentation words based on the position and connectivity of the segmentation words in the relationship network, and sort the keywords based on the importance; A named entity recognition module is used to identify named entities including names of persons, places and / or institutions from the text to be converted, and a named entity list with entity category labels is generated.

3. The text-to-video method based on deep semantic analysis according to claim 1, characterized in that: Analyze the core theme and text structure of the text to be converted, wherein the text structure includes titles, subtitles and / or paragraphs, including: Preprocessing the text to be converted, wherein the preprocessing includes removing stop words and punctuation marks; Performing topic analysis on the text to be converted based on a preset topic model, extracting key sentences and high-frequency words whose occurrence frequency exceeds a preset threshold in the text to be converted; Based on the key sentences and high-frequency words, generate one or more core topics; Using a text structure analysis module to identify titles, subtitles and / or paragraphs in the text to be converted; The hierarchical relationship between the titles, subtitles and / or paragraphs is analyzed to generate a hierarchical text structure.

4. The text-to-video method based on deep semantic analysis according to claim 1, characterized in that: Perform semantic analysis based on the keywords, named entities, text structure and core topics to generate semantic analysis results, including: Analyzing the association relationship between the keywords and the named entities, and determining the semantic relationship between the keywords and the named entities based on the context and syntactic structure; Analyzing semantic logic among titles, subtitles and / or paragraphs based on hierarchical information of the text structure; Based on the matching analysis between the core theme and the paragraph content, determine the degree of relevance of each paragraph to the core theme; Conduct context analysis on named entities to determine the frequency of occurrence and contextual semantic meaning of named entities in the text to be converted; Based on the semantic relationship between the keywords and the named entities, the relevance of the paragraphs to the core topics, the frequency of occurrence of the named entities in the text to be converted, and the contextual semantic meaning, a comprehensive semantic analysis result is generated.

5. The text-to-video method based on deep semantic analysis according to claim 1, characterized in that: According to the semantic analysis results, a mapping relationship between semantics and visual elements is established, and images, video clips, sound effects and background music matching the semantic analysis results are retrieved from the material library, including: Based on the keyword information in the semantic analysis results and the mapping rules between keywords and visual elements, images or video clips matching the keywords are retrieved from the material library; Based on the named entity information in the semantic analysis results, combined with the category and contextual semantic meaning of the named entity, images or video clips matching the named entity are retrieved from the material library; Based on the text structure hierarchical information in the semantic analysis results, images or video clips matching the text structure are retrieved; Based on the core theme in the semantic analysis result, a scene background image or scene background video clip matching the core theme is selected; Based on the emotional tone in the semantic analysis results, background music and sound effects that match the emotional tone are selected.

6. The text-to-video method based on deep semantic analysis as claimed in claim 1, characterized in that: According to the video timeline, the preliminary visual layout framework, background music and sound effects are integrated to generate integrated complete video content, including: According to the video timeline, synchronizing the images, video clips and animation effects in the preliminary visual layout framework with the video timeline to determine the display order, duration and transition effect of the images, video clips and animation effects; Adjusting the start time and duration of the background music according to the video timeline; Associating the sound effect with the corresponding animation effect, and setting the triggering time and duration of the sound effect; The visual layout framework, background music and sound effects are integrated to generate complete video content that has been synchronized.

7. The text-to-video method based on deep semantic analysis according to claim 1, characterized in that: After integrating the preliminary visual layout framework, background music and sound effects according to the video timeline to generate integrated complete video content, the method further includes: Rendering the integrated video content, combining all visual and audio elements frame by frame; Use the video encoding module to compress the rendered video content; A final video file is generated and outputted in a preset video format.

8. A text-to-video device based on deep semantic analysis, characterized in that: The text-to-video device based on deep semantic analysis includes: A text processing module is used to obtain the text to be converted, perform word segmentation and part-of-speech tagging on the text to be converted, and extract keywords and named entities from the text to be converted; A text analysis module, used to analyze the core theme and text structure of the text to be converted, wherein the text structure includes titles, subtitles and / or paragraphs; A semantic analysis module, used to perform semantic analysis based on the keywords, named entities, text structure and core topics, and generate semantic analysis results; A material retrieval module is used to establish a mapping relationship between semantics and visual elements according to the semantic analysis results, and retrieve images, video clips, sound effects and background music that match the semantic analysis results from the material library; A layout generation module, for spatially arranging the text content corresponding to the keywords and named entities, and the images and video clips based on the semantic analysis results, to generate a preliminary visual layout framework; A layout feature analysis module, used to analyze layout content features of the preliminary visual layout framework using a deep learning model; An animation effect generation module, used to generate animation effects based on the layout content characteristics, and apply the animation effects to the text content and visual elements in the preliminary visual layout framework; A timeline building module for building a video timeline based on a preliminary visual layout framework with applied animation effects, background music and sound effects; The video generation module is used to integrate the preliminary visual layout framework, background music and sound effects according to the video timeline to generate integrated complete video content.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a text-to-video program based on deep semantic analysis stored in the memory and executable on the processor. When the text-to-video program based on deep semantic analysis is executed by the processor, the steps of the text-to-video method based on deep semantic analysis as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a text-to-video program based on deep semantic analysis, and when the text-to-video program based on deep semantic analysis is executed by a processor, the steps of the text-to-video method based on deep semantic analysis as described in any one of claims 1-7 are implemented.

Images