Method and device for generating various modal contents based on knowledge base and index and electronic equipment

Abstract

The embodiment of the invention discloses a method and a device for generating various modal contents based on a knowledge base and an index, electronic equipment and a computer readable storage medium, and realizes'knowledge priori constraint 'of a generation process based on dual drive of the knowledge base and a vector index; introducing a cyclic screening and dynamic threshold control mechanism, and constructing a convergent optimization closed loop; multi-modal complementation and consistency verification logic is provided, and integrated generation of cross-modal content is achieved; through a shunting-scheduling-fusion mechanism, the problems of scheduling efficiency and consistency in a multi-model collaboration process are innovatively solved; an end-to-end closed loop is established, knowledge retrieval and content generation are truly connected, and a knowledge base is evolved into a dynamic driving engine from a static information source. According to the technical scheme provided by the invention, through hierarchical design and module decoupling, the expandability of the system is ensured, and the professionality and practicability of the generated content are also improved.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and content generation technology, and in particular to a method, apparatus and electronic device for generating multimodal content based on a knowledge base and index. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-model technologies, represented by deep learning, have achieved significant breakthroughs in their application capabilities in multimodal tasks such as content generation, natural language processing, and image recognition. Intelligent content generation systems have thus emerged and are widely used in various fields such as intelligent customer service, education and training, content creation, and public opinion analysis.

[0003] Currently, existing content generation methods can be broadly categorized into two types: one is content filling methods based on predefined templates, and the other is automatic content generation methods based on large-scale language models (such as generative pre-trained transformation models and bidirectional encoder representation models). While predefined template-based methods can guarantee high accuracy in highly structured domains, they lack flexibility and struggle to adapt to complex and ever-changing user needs. Language model-based methods, while demonstrating powerful natural language generation capabilities, still suffer from significant shortcomings in the professionalism, factual accuracy, and contextual consistency of generated content. Therefore, there is an urgent need for a new content generation solution that deeply integrates knowledge bases, indexing technologies, and multimodal generation capabilities to address the problems of low content accuracy, limited modality, and disconnect between knowledge and generation in existing technologies. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method and apparatus for generating multimodal content based on a knowledge base and index, so as to overcome the above-mentioned defects in the prior art.

[0005] This disclosure combines multiple functional modules such as knowledge material storage and indexing, material feature extraction and filtering, and multimodal content generation to construct an intelligent content generation system that integrates the entire chain of "knowledge-indexing-extraction-generation," and is multimodal and multi-model collaborative.

[0006] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a method for generating multimodal content based on a knowledge base and index, characterized by comprising the following steps: The original multi-modal material is parsed to extract structured features, the structured features are stored in a structured knowledge base, and the semantic vector index of the structured features is generated and stored in a vector database. The user's request text is parsed to form request feature information; The demand feature information is converted into a query vector, and a similarity search is performed in the vector database to obtain a candidate material index; Based on the candidate material index, retrieve the corresponding candidate materials and their features from the structured knowledge base; The candidate materials are evaluated for their matching degree with the required feature information using a large inference model. Multiple rounds of iterative screening are performed based on preset termination conditions to obtain the target materials. The required text is merged with the target material, and a structured prompt word containing task instructions and logical constraints is constructed through a large text generation model; Based on the structured prompts, the target content modality to be generated is determined, and the corresponding multimodal generation model is scheduled to generate the multimodal content result.

[0007] Furthermore, the step of parsing the original multi-modal material, extracting structured features, storing the structured features in a structured knowledge base, and generating semantic vector indexes of the structured features and storing them in a vector database includes the following steps: Import original materials in multiple modalities; Identify the modal type of the original material; Based on the modality type, the corresponding feature extraction model is invoked to extract the structured features of the original material. The structured features include subject matter, type, keywords, and tags. The structured features are stored in the structured knowledge base; The structured features are input into the semantic encoding model to generate corresponding semantic vectors; The semantic vectors are stored in the vector database and an index is created.

[0008] Furthermore, the preset termination condition includes at least one of the following: High matching threshold condition: The matching degree of candidate materials reaches the preset threshold; Modal coverage condition: The selected materials cover all content modalities specified in the requirement feature information; Content value criteria: The accuracy and clarity of the selected materials meet the preset value threshold.

[0009] Furthermore, the step of using a large inference model to evaluate the matching degree between the candidate materials and the demand feature information, and performing multiple rounds of iterative screening based on preset termination conditions to obtain the target materials, also includes: Adaptive convergence mechanism: Set a maximum number of iteration rounds. If any of the termination conditions are not met when the maximum number of iteration rounds is reached, output the material with the highest matching degree in the current round as the target material.

[0010] Furthermore, the structured prompt words adopt a hierarchical structure, including: a basic field layer, a semantic tag layer, a task feature layer, and a logical dependency layer.

[0011] Furthermore, the data of the logical dependency layer is stored in a cross-modal graph database to establish semantic relationships between different modal materials.

[0012] Furthermore, the step of determining the target content modality to be generated based on the structured prompts and scheduling the corresponding multimodal generation model to generate the multimodal content result specifically includes: The splitting sub-step involves determining the target content modality based on the structured prompts and identifying the model type to be invoked. Scheduling sub-step: Select the optimal model from the pool of models of the same type based on task requirements and model performance weights; Generation sub-step: Call the selected model to generate the content of the corresponding modality; Fusion sub-step: Perform consistency verification and fusion on the generated different modal contents.

[0013] Furthermore, the step of determining the target content modality to be generated based on the structured prompts and scheduling the corresponding multimodal generation model to generate the multimodal content result also includes: A modal complementarity mechanism is adopted: if the content generation result of a certain modality is insufficient, other modal generation models are automatically scheduled to supplement the content.

[0014] Furthermore, the method also includes: The generated multimodal content results are fed back to the structured knowledge base and vector database for updating and optimizing knowledge materials and indexes.

[0015] On the other hand, embodiments of this disclosure provide an apparatus for generating multimodal content based on a knowledge base and index, characterized in that it includes: The feature extraction module is used to parse the original multi-modal material to extract structured features, store the structured features in a structured knowledge base, and generate semantic vector indexes of the structured features and store them in a vector database. The requirement parsing module is used to parse the user's requirement text and form requirement feature information; The material retrieval module is used to convert the demand feature information into query vectors, perform similarity retrieval in the vector database, and obtain a candidate material index. The material retrieval module is used to retrieve corresponding candidate materials and their features from the structured knowledge base based on the candidate material index. The material screening module is used to evaluate the matching degree between the candidate materials and the required feature information using a large inference model, and to perform multiple rounds of iterative screening based on preset termination conditions to obtain the target materials. The prompt word construction module is used to merge the requirement text with the target material and construct structured prompt words containing task instructions and logical constraints through a large text generation model; The multimodal generation module is used to determine the target content modality to be generated based on the structured prompts and to schedule the corresponding multimodal generation model to generate the multimodal content result.

[0016] Furthermore, the device also includes a feedback optimization module, which is used to feed back the generated multimodal content results to the feature extraction module, the demand analysis module, the material retrieval module, and the material filtering module to form a closed-loop feedback to dynamically optimize the filtering threshold and the generation strategy.

[0017] On the other hand, embodiments of this disclosure provide an apparatus for generating multimodal content based on a knowledge base and index, characterized in that it includes: A storage module is used to store the structured knowledge base and vector database constructed by any of the methods described above; The extraction module is used to receive the user's input request text, parse the request text, and extract feature information; The processing module is used to retrieve, recall, and filter target materials from the vector database and the structured knowledge base based on the feature information. The generation module is used to merge the requirement text with the target material to generate structured prompt words, and to schedule a multimodal large model to generate the final content; The service module provides a user interface, receives the request text, and outputs the final content.

[0018] On the other hand, embodiments of this disclosure provide an electronic device, including: At least one memory for storing computer-readable instructions; and At least one processor is configured to execute the computer-readable instructions, causing the electronic device to implement any of the methods described in the first aspect above.

[0019] On the other hand, embodiments of this disclosure provide a non-transitory computer-readable storage medium for storing computer-readable instructions that, when executed by a computer, cause the computer to implement the method described in any one of the first aspects above.

[0020] On the other hand, embodiments of this disclosure provide a computer program, including instructions, which, when run on a computer, cause the computer to perform the method for generating multimodal content based on a knowledge base and index according to any one of the first aspects described above.

[0021] This disclosure provides a method, apparatus, electronic device, and computer-readable storage medium for generating multimodal content based on a knowledge base and index. The method includes: parsing original multimodal materials to extract structured features; storing the structured features in a structured knowledge base and generating a semantic vector index of the structured features, which is then stored in a vector database; parsing user request text to form request feature information; converting the request feature information into query vectors and performing similarity retrieval in the vector database to obtain a candidate material index; recalling corresponding candidate materials and their features from the structured knowledge base based on the candidate material index; evaluating the matching degree between the candidate materials and the request feature information using a large-scale inference model, performing multiple rounds of iterative filtering based on preset termination conditions to obtain target materials; fusing the request text with the target materials, constructing structured prompts containing task instructions and logical constraints using a large-scale text generation model; determining the target content modality to be generated based on the structured prompts, and scheduling the corresponding multimodal generation model to generate multimodal content results.

[0022] The technical solution proposed in this invention, through layered design and module decoupling, not only ensures the scalability of the system but also enhances the professionalism and practicality of the generated content.

[0023] Based on the above modules and processes, this invention further proposes an end-to-end "knowledge-guided generation" closed-loop mechanism, the core of which is to deeply integrate knowledge material storage and indexing, feature extraction and filtering, multimodal generation and scheduling to build a full-link, iteratively optimizable intelligent content generation system.

[0024] Knowledge-driven indexing and retrieval: First, by combining a structured knowledge base with vector indexes, a high-precision, low-latency material retrieval mechanism is achieved. Unlike existing generation methods that solely rely on corpus training, the generation starting point of this invention is directly constrained by the knowledge base, ensuring that the content is factual and professional.

[0025] Index-guided multi-round filtering: After material extraction, the system does not immediately begin generation. Instead, it uses a large inference model to perform cyclical filtering and dynamic threshold determination, continuously optimizing the material set until termination conditions such as "high matching degree / multimodal coverage" are met. This mechanism innovatively introduces an "adaptive convergence mechanism" to ensure the high relevance of the output content.

[0026] Feature fusion-based prompt word construction: User demand features and selection materials are jointly transformed into structured prompt words. The prompt words not only include task descriptions, but also introduce logical constraints and style control factors, thereby achieving fine guidance for the large model in the generation process.

[0027] Multimodal collaborative generation and scheduling: The system adopts a three-level logic of diversion, scheduling and fusion: First, the required modality is identified and diverted to the corresponding model pool. Then, performance-weighted scheduling is performed in the same type of model pool. Finally, cross-modal consistency comparison and fusion are performed to ensure that the final result is completely unified in terms of content logic, semantic expression and presentation.

[0028] Closed-loop feedback and adaptive optimization: The generated results will flow back to the feature extraction and filtering module, forming a closed-loop feedback loop. The system dynamically adjusts the optimization threshold according to the task type, achieving an adaptive effect of "pursuing accuracy in professional scenarios and maintaining diversity in creative scenarios".

[0029] Therefore, compared with the prior art, the present invention has the following technical improvements: 1. Based on the dual drive of knowledge base + vector index, the "knowledge prior constraint" of the generation process is realized; 2. Introduce a cyclical screening and dynamic threshold control mechanism to construct a convergent optimization closed loop; 3. Propose multimodal complementarity and consistency verification logic to achieve integrated generation of cross-modal content; 4. Through a diversion-scheduling-fusion mechanism, the scheduling efficiency and consistency issues in the multi-model collaboration process are innovatively solved; 5. Establish an end-to-end closed loop to truly connect knowledge retrieval and content generation, transforming the knowledge base from a static information source into a dynamic driving engine.

[0030] Thus, this invention forms a complete "knowledge-index-extraction-generation" chain, overcoming the shortcomings of existing technologies such as high illusion rate, single modality, and lack of knowledge constraints, and realizing an innovative intelligent content generation system that is scalable, verifiable, and capable of multimodal collaboration.

[0031] The method for generating multi-modal content based on a knowledge base and index disclosed herein has the following beneficial technical effects: 1. Significantly improve content accuracy: By deeply integrating structured knowledge base and vector indexing technology, it provides solid factual basis for the content generation process, effectively avoiding the "illusion" phenomenon and generating more professional and credible content.

[0032] 2. Powerful multimodal collaboration capabilities: A unified "knowledge-index-generation" framework has been built, which can intelligently understand and generate content in multiple modalities such as text, images, audio, and video, and ensure semantic consistency and logical correlation between them, meeting the application needs of complex scenarios.

[0033] 3. Achieved closed-loop generation guided by knowledge: The knowledge base was transformed from a static retrieval source into a dynamic generation engine. An innovative end-to-end closed-loop process was designed from knowledge extraction and material selection to content generation, truly realizing the full-process guidance and optimization of content creation by knowledge.

[0034] 4. Excellent scalability and practicality: Through modular system design and adaptive optimization mechanism, the system can adapt to knowledge content in different fields and can be continuously iterated and optimized based on feedback, thus having broad application prospects.

[0035] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0036] Figure 1 A schematic diagram of the overall system architecture for generating multi-modal content based on a knowledge base and index, provided as an embodiment of this disclosure; Figure 2 This is a schematic diagram of the overall content generation process provided in an embodiment of the present disclosure; Figure 3 A schematic flowchart of a method for generating multi-modal content based on a knowledge base and index provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of a data cleaning and indexing process provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of a feature extraction sub-process provided in an embodiment of the present disclosure; Figure 6 This is a schematic diagram of a material extraction sub-process provided in an embodiment of the present disclosure; Figure 7 This is a schematic diagram of a material screening sub-process provided in an embodiment of the present disclosure; Figure 8 This is a schematic diagram of a prompt word generation sub-process provided in an embodiment of the present disclosure; Figure 9 This is a schematic diagram of a multimodal content generation subprocess provided in an embodiment of the present disclosure; Figure 10 A schematic diagram of an apparatus for generating multimodal content based on a knowledge base and index, provided for another embodiment of this disclosure; Figure 11 A schematic diagram of an apparatus for generating multi-modal content based on a knowledge base and index, provided for another embodiment of this disclosure; Figure 12This is a schematic diagram of the structure of an electronic device corresponding to a method for generating multi-modal content based on a knowledge base and index, provided in another embodiment of this disclosure. Detailed Implementation

[0037] To more clearly describe the technical content of this disclosure, the following description is provided in conjunction with specific embodiments.

[0038] The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0039] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0040] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0041] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0042] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0043] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0044] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. The disclosed embodiments are described in detail below with reference to the accompanying drawings.

[0045] In this disclosure, "large model" refers to a pre-trained artificial intelligence model trained on a large-scale corpus, possessing general language understanding and generation capabilities, such as GPT, BERT, and T5. Large models have powerful knowledge transfer, semantic reasoning, and natural language generation capabilities, and are used in this invention for text generation, reasoning judgment, and prompt word generation.

[0046] This disclosure provides a method and apparatus for generating multimodal content based on knowledge base and indexing technology. This technical solution can automatically complete feature extraction, material retrieval, filtering, and content generation based on user-inputted text. By combining multiple functional modules such as knowledge material storage and indexing, material feature extraction and filtering, and multimodal content generation, it constructs a fully integrated, multimodal, and multi-model collaborative intelligent content generation system encompassing the entire chain of "knowledge-indexing-extraction-generation," ultimately outputting multimodal content results.

[0047] Figure 1 The diagram shows a schematic of the overall system architecture for generating multi-modal content based on a knowledge base and index, according to an embodiment of this disclosure. The system includes a service layer, a large model layer, and a storage layer.

[0048] The service layer serves as the interface between the user and the entire system, responsible for receiving the user's input request text. The service layer then sends these requests to the backend processing module and returns the final generated content to the user. (Note: Input of request text and output of materials can be achieved through application interfaces, APIs, etc., which are not the focus of this patent and will not be elaborated upon here.)

[0049] In the service layer, the extraction module is responsible for understanding user input and finding and extracting relevant content from the existing content library. This module is further divided into three sub-modules: Feature extraction: Analyze user requirement text to extract "feature information" such as keywords, themes, and style requirements.

[0050] Material extraction: These features are used to search for matching material content in the database, which may be in various forms such as images, text, and audio.

[0051] Material filtering: Further filter the search results to remove irrelevant information and retain the most valuable materials.

[0052] The generation module in the service layer is mainly used for prompt word generation and multimodal content generation. Prompt word generation uses a large text generation model to combine materials and user needs into prompt words that the system can understand. Multimodal content generation uses a large multimodal model based on the prompt words to output final content such as images, audio, video, or text.

[0053] The large model layer includes a text generation large model, a reasoning large model, and a multimodal large model. Among them, the text generation large model is responsible for thinking and outputting text-based content; the reasoning large model is used to determine whether the material meets the requirements and analyze the logical structure; the multimodal large model can understand non-textual information such as images and videos and generate multimodal content.

[0054] The storage layer is the system's information infrastructure, consisting of two parts: Knowledge material storage: Save original files, such as PDFs, images, audio, etc. Knowledge base: includes traditional databases (store structured information) and vector databases (used for semantic matching and retrieval), and improves search efficiency and accuracy by building vector indexes.

[0055] Figure 2 This illustration shows a schematic diagram of the overall content generation process provided in an embodiment of the present disclosure. The process demonstrates the entire process from the user initiating a request to the system completing content generation. The overall process includes the following steps: 1) The user inputs the required text; 2) The system calls the "Feature Extraction Process" (see details). Figure 5 (Corresponding process) Identify the content keywords that the user expects; Then the system invokes the "Material Extraction Process" (see details). Figure 6 (Corresponding process) Search the database for materials that match these keywords; 3) Not all extracted content may be suitable, therefore, proceed to the "Material Screening Process" (see details). Figure 7 (corresponding process), using a large model to analyze whether each piece of material highly matches user needs; 4) Then call the "Prompt Generation Process" (see details). Figure 8 (Corresponding process) combines user input with filtered materials to generate targeted prompts; 5) Finally, invoke the "Multimodal Content Generation Process" (see details). Figure 9 (Corresponding process) Generate text, images, videos, and other content based on prompts and return them to the user.

[0056] Figure 3This is a schematic flowchart illustrating a method for generating multi-modal content based on a knowledge base and index, according to an embodiment of this disclosure. The method provided in this embodiment can be executed by a computer device or an intelligent device. This device can be implemented as software or as a combination of software and hardware. The device can be integrated into a mobile device or terminal device, such as a smartphone or tablet computer (PAD). Figure 2 As shown, the method includes the following steps: Step S101: Parse the original multi-modal material to extract structured features, store the structured features in a structured knowledge base, and generate a semantic vector index of the structured features and store it in a vector database.

[0057] In step S101, in this embodiment of the present disclosure, the structured knowledge base refers to a searchable knowledge set organized according to unified fields and rules, such as standards, terminologies, and tag systems in a database. This invention relies on this knowledge base to provide authoritative support for content generation. Vector indexing refers to a technical method that transforms text, image descriptions, and other content into vector form through a semantic embedding model and constructs an index to achieve rapid semantic matching and efficient material retrieval. This step specifically includes: importing original materials in multiple modalities, identifying the modality type of the original materials, calling the corresponding feature extraction model according to the modality type to extract the structured features of the original materials, the structured features including subject matter, type, keywords, and tags, storing the structured features in the structured knowledge base, inputting the structured features into a semantic encoding model to generate corresponding semantic vectors, and storing the semantic vectors in the vector database and establishing an index.

[0058] First, system initialization and material preparation are performed. The purpose of this stage is to build the knowledge infrastructure upon which content generation depends, including core steps such as material import, content parsing, structured data storage, and index construction. The specific steps are as follows: 1) Importing materials Operators import raw materials into the system, including text, images, audio, and video. The system receives the materials through a file upload interface or directory scanning and registers the file metadata.

[0059] 2) Material type identification and analysis The system first uses file header information, file extension, and content recognition algorithms to determine the type of material, distinguishing it into text types (TXT, WORD, etc.) and multimodal types (MOV, MP4, PNG, etc.).

[0060] 3) Content feature extraction and structured storage For text formats (PDF, Word, TXT), a large-scale text generation model is used for feature extraction. For images or videos, a multimodal large-scale model is used for feature extraction. The large-scale model provides suggestions such as: Please describe the characteristics of the above text (image / video), including but not limited to subject matter, type, time, location, people, field, etc., and output a structured result.

[0061] A unified feature structure is constructed for each piece of material, and stored in a structured database.

[0062] Vector index generation and storage in vector database The goal of this stage is to convert the feature information of each piece of material into a semantic embedding vector that can be understood by a computer to support subsequent similarity retrieval. The steps are as follows: 4) Constructing input content: The main text content in the structure (usually title + main_content + keywords + tags) is combined into continuous segments and used as model input. For example: Title: XX Technology Development Trends; Keywords: Artificial Intelligence, Semantic Analysis; Content: ...; Tags: Technology Analysis, AI Model. 5) Select a pre-trained vectorization model: The system internally calls a loaded semantic encoding model, such as a text embedding model under the Transformer architecture (e.g., S-BERT, E5, or a specially trained enterprise private model), and inputs the combined text.

[0063] 6) Vector calculation: The model outputs a dense vector of fixed dimensions (e.g., 384-dimensional or 768-dimensional) representing the location of the material in the semantic space.

[0064] 7) Vector normalization (optional): If a cosine similarity-based retrieval mechanism is subsequently used, the vectors need to be normalized (unitized) to improve retrieval consistency.

[0065] The aforementioned feature structures are input into a pre-trained vectorization model to generate semantic embedding vectors. Simultaneously, a vector index is created for each material record and stored in a high-performance vector database for rapid subsequent retrieval. The knowledge material library and index system built in this stage will provide comprehensive, accurate, and rapid material support for downstream modules.

[0066] For details, please refer to Figure 4 This illustrates the data cleaning and indexing process, which is the foundational construction phase of the system. Its purpose is to process various knowledge materials into a format that the system can recognize and search, laying the groundwork for subsequent use. The specific process includes the following steps: 1) Users import original materials, such as a PDF document, a photo, a Word report, or a video.

[0067] 2) The system determines the type of material.

[0068] 3) If it is a text format (such as Word or PDF), call the multimodal model to parse the text content and extract the knowledge points.

[0069] 4) If it is a non-text format such as an image or video, call the visual or speech recognition model for analysis and extract descriptive features.

[0070] 5) After further processing, all content features are stored in a structured database (content and features).

[0071] 6) Establish an efficient semantic indexing mechanism based on the characteristics of the materials and store it in a vector database to make subsequent content retrieval more accurate and faster.

[0072] Step S102: Parse the user's requirement text to form requirement feature information.

[0073] In the process of system operation, when a user initiates a content generation request (either through the interface or an API call), the system enters a process of understanding and analyzing the input request, which specifically includes the following steps: I) Demand Reception Users submit their request text through the service layer. This text may include task objectives, themes, style preferences, etc. The system records the request and generates a unique identifier for subsequent process tracking.

[0074] II) Language Comprehension and Feature Recognition The language understanding model is invoked to parse user text, including: extracting keywords and main idea information; Analyze tone and style preferences (such as formal, casual, academic, etc.); determine content type preferences (such as needing to generate popular science articles, marketing short videos, or mixed text and image content).

[0075] III) Output Feature Structure All extracted information will be organized into a standardized feature format and submitted to the material extraction module as input conditions for material matching.

[0076] Figure 5 This illustration shows a schematic diagram of a feature extraction sub-process provided in an embodiment of the present disclosure, which specifically includes: 1) The system reads the user's request text; 2) Use language understanding models to identify keywords, main ideas, tone, etc. 3) Output the extraction results as the "feature standard" for the next step of material matching.

[0077] Step S103: Convert the demand feature information into a query vector, perform a similarity search in the vector database, and obtain a candidate material index.

[0078] In step S103, in this embodiment of the present disclosure, based on the aforementioned extracted demand features, the system will search for the most relevant content materials from the knowledge base, perform quality assessment and screening, and retrieve the corresponding multimodal original materials and their descriptive features from the structured database according to the indexing results to form a material candidate set.

[0079] Figure 6 This illustration shows a schematic diagram of a material extraction sub-process provided in an embodiment of the present disclosure. The process includes: The input is the keywords output from the previous process; the system searches for semantically similar material indexes in the vector database; based on the indexes, it retrieves complete multimodal materials from the knowledge base; and outputs all matching results and their feature lists.

[0080] Step S104: Retrieve the corresponding candidate materials and their features from the structured knowledge base according to the candidate material index.

[0081] In step S104, in this embodiment of the present disclosure, the system converts demand keywords, topics, and other information into semantic vectors, compares them with the index in the vector database, and recalls several material items with the closest semantics. By combining a structured knowledge base with vector indexes, a high-precision, low-latency material recall mechanism is achieved. Unlike existing generation methods that solely rely on corpus training, the generation starting point of this invention is directly constrained by the knowledge base, ensuring that the content possesses factual and professional accuracy. Through vector retrieval to recall candidate materials, the system converts demand keywords, topics, and other information into semantic vectors, compares them with the index in the vector database, and recalls several material items with the closest semantics.

[0082] Step S105: Use the large inference model to evaluate the matching degree between the candidate materials and the required feature information, and perform multiple rounds of iterative screening based on preset termination conditions to obtain the target materials.

[0083] In step S105, the inference big model refers to a large model with complex judgment, logical reasoning, and conditional filtering capabilities, used for non-generational tasks such as task relevance analysis and quality scoring of candidate materials. After material extraction, the system does not immediately enter the generation process, but instead uses the inference big model to perform cyclical filtering and dynamic threshold determination, continuously optimizing the material set until termination conditions such as "high matching degree / multimodal coverage" are met. This mechanism innovatively introduces an "adaptive convergence mechanism" to ensure the high relevance of the output content.

[0084] The large inference model is used to evaluate each candidate material to determine whether it meets the feature criteria, such as: whether it conforms to the theme and semantic requirements; whether it meets the modal requirements (such as rich text and images, video and text descriptions, etc.); and whether the content is accurate, clear, and useful.

[0085] The prompt is as follows: "Currently, there are the following requirements features XXX (requirement features). Determine whether the following materials meet the requirements features and give a score from 0 (not meeting the features at all) to 10 (meeting the features at all)."

[0086] Figure 7 A schematic diagram of a material screening process provided in an embodiment of this disclosure is shown. The process includes: The input consists of candidate materials and required keywords; an inference model is used to evaluate each material to see if it matches the requirements; irrelevant content is deleted, and only materials that meet the criteria are retained; the final output is the high-quality materials after filtering.

[0087] In this embodiment, the preset termination condition in step S105 includes at least one of the following: Termination condition 1: High matching degree threshold condition When the matching degree of the candidate material reaches a preset threshold, for example, when the semantic matching score between the candidate material and the requirement reaches ≥9, it is considered that the requirement has been met, and the system automatically stops the loop.

[0088] Termination condition 2: Modal coverage condition The selected materials cover all content modalities specified in the requirement feature information; when the generated results cover all modalities of the user's requirements (such as text, images, and audio), the closed loop is considered complete.

[0089] Termination Condition 3: Content Value Condition The accuracy and clarity of the selected materials have reached the preset value threshold; based on the accuracy and clarity of the current materials, it is determined whether they meet the usability conditions and have sufficient use value.

[0090] This step involves multiple rounds of iterative filtering based on preset termination conditions to obtain the target material, and also includes: Loop Adaptive Convergence Mechanism: To avoid infinite loops, the system sets a maximum number of iterations (e.g., 10 rounds). After exceeding this limit, the system automatically retains and outputs the best result. This mechanism ensures that the generation process does not fall into an infinite loop while dynamically guaranteeing quality. If any of the aforementioned termination conditions are not met when the maximum number of iterations is reached, the material with the highest matching degree in the current round is output as the target material.

[0091] This step ensures high-quality output by discarding substandard materials and retaining high-quality, information-rich, and diverse materials as input for the generation module.

[0092] Step S106: Merge the requirement text with the target material, and construct a structured prompt word containing task instructions and logical constraints through a large text generation model.

[0093] In step S106, in this embodiment of the disclosure, the prompt word refers to the natural language input constructed to guide the large model to output specific content, typically including elements such as background description, task requirements, and output format. In this patent, the prompt word is generated driven by both user needs and material features. User needs features and selected materials are jointly transformed into structured prompt words. The prompt words not only include task descriptions but also introduce logical constraints and style control factors, thereby achieving fine-grained guidance for the large model during the generation process.

[0094] The specific steps for generating prompt keywords are as follows: I) Integrating materials and required content The system combines the user's original input requirements with filtered, high-quality content. It extracts core content blocks based on defined rules and identifies their logical relationships.

[0095] II) Constructing the prompt statement structure Based on different content objectives, the system constructs "task instructions" that are adapted to the input of the text generation model. For example, for image-text generation tasks, prompts should guide the model to conduct "image-text combined explanations"; for video generation tasks, prompts should include shot scripts, narrative tone, and visual cues.

[0096] The structure adopts a "four-layer structure + cross-modal graph database" design: Basic field layer: metadata such as title, time, source, and file type; Semantic tagging layer: Domain tags automatically generated by the large model (such as "mechanical safety" and "structural design"); Task feature layer: includes user requirement features (style preferences, accuracy level, modality combination requirements); Logical Dependency Layer: Defines the semantic and structural relationships between different materials (e.g., "Image A" corresponds to "Text B" description).

[0097] Cross-modal graph database storage: This not only stores data but also establishes "relation edges" between different modalities, forming a knowledge graph-style structured storage. During retrieval, it can simultaneously retrieve "related images + descriptive text," rather than isolated results, ensuring cross-modal semantic connectivity.

[0098] III) Semantic optimization and naturalization The prompts are refined and normalized using a text generation model to conform to natural language expression habits, thereby improving the model's understanding and performance. Prompt words include: The current requirement is XXX (requirement text). Please generate a text (audio / video...) prompt based on the requirement. The existing reference material is XXX (material characteristics).

[0099] Employing "dual-track parallel optimization": Semantic optimization track: Utilize semantic vector similarity and knowledge graph reasoning to ensure a high degree of semantic matching between the recalled materials and the demand.

[0100] Generate optimized tracks: The initial generated results will be further corrected by the style control sub-model to make them conform to the scene requirements in terms of tone, professionalism, and visual style.

[0101] Parallel optimization mechanism: Traditional generation techniques are often serial optimizations, while the embodiments disclosed in this disclosure adopt a dual-track parallel approach of "recall correction + generation correction", which improves the accuracy of content and consistency of style.

[0102] IV) Output the final prompt word The structured prompt text is output to the generation module as a direct driving signal for the generation action.

[0103] Figure 8 A schematic diagram of a prompt word generation process provided in an embodiment of this disclosure is shown. The process includes: The input consists of user requirements and source content; the system combines these two elements and uses a generative model to write a "guided question" or "task instruction"; the output prompts will be directly used to drive content generation.

[0104] The structured prompts employ a hierarchical structure, including: a basic field layer, a semantic tag layer, a task feature layer, and a logical dependency layer. The logical dependency layer stores data in a cross-modal graph database, used to establish semantic relationships between different modal materials.

[0105] Step S107: Determine the target content modality to be generated based on the structured prompt words, and schedule the corresponding multimodal generation model to generate multimodal content results.

[0106] In step S107, in this embodiment of the present disclosure, multimodal content refers to composite content containing multiple information modalities, such as text and images, video dubbing, and voice narration. Multimodal content can enhance the richness and interactivity of information expression and is an important component of the generation goal of this patent. The system adopts a three-level logic of diversion-scheduling-fusion: first, the required modality is identified and diverted to the corresponding model pool; then, performance-weighted scheduling is performed in the same type of model pool; finally, cross-modal consistency comparison and fusion are performed to ensure that the final result is completely unified in terms of content logic, semantic expression, and presentation.

[0107] In this embodiment, step S107 specifically includes: The splitting sub-step involves determining the target content modality based on the structured prompts and identifying the model type to be invoked. The scheduling sub-step involves selecting the optimal model from the pool of models of the same type based on task requirements and model performance weights. The generation sub-step involves invoking the selected model to generate content of the corresponding modality. The fusion sub-step involves performing consistency verification and fusion on the generated content of different modalities.

[0108] This step specifically also includes: A modal complementarity mechanism is adopted: if the content generation result of a certain modality is insufficient, other modal generation models are automatically scheduled to supplement the content.

[0109] This stage is the core of the system's content creation. Based on the prompt words, a suitable large model is dispatched to complete the final content generation, and it is executed according to the following logic: I) Content Type Judgment and Streaming The reasoning module determines whether the final generated target is plain text, mixed text and images, a short video clip, or a voice broadcast based on the content of the prompt words, and selects the appropriate generation engine accordingly.

[0110] A three-tiered mechanism of "distribution-scheduling-fusion" is adopted: Triage Phase: The inference big model determines the demand modality and triages it to the corresponding model (text / multimodal).

[0111] Scheduling phase: Within the same pool, the system makes a weighted selection based on task requirements and model performance (accuracy, generation speed, computational cost).

[0112] Fusion phase: Before outputting results from different modalities, a consistency comparison is performed to ensure logical uniformity before output.

[0113] Unlike the existing "fixed call + spliced ​​output" approach, this invention realizes the entire process of "dynamic diversion - intelligent scheduling - cross-modal fusion", which greatly enhances the system's flexibility and accuracy.

[0114] II) Model Scheduling and Generation Execution If the output is text, the large text generation model is invoked to construct paragraphs, logical structures, and expression styles based on prompts; if the output is an image, the large image-text multimodal model is invoked to draw scenes, illustrations, or visual materials based on prompts; if the output is a video or audio task, the large audio multimodal model is used to generate audio.

[0115] III) Content encapsulation and result output The generated content will be packaged into a uniform output format (such as TXT, MP4, or PNG) according to its type, and will also include metadata such as content tags, reliability scores, and generation time.

[0116] This invention proposes a "three-dimensional multimodal invocation mechanism," which differs from traditional single-modal or simple combination invocation: Dimension 1: Modality Priority Decision Tree The system automatically generates call priorities based on user needs and scenarios. For example, "technical standard documents" prioritize text + engineering drawings; "teaching scenarios" prioritize video + audio.

[0117] Dimension 2: Modal Complementarity Mechanism If the output of a certain modality is insufficient, the system triggers a "compensation call". For example, if the generated structure diagram lacks explanation, the system automatically calls the text generation module to supplement the explanation.

[0118] Dimension 3: Cross-modal consistency verification By using a large inference model to perform consistency comparisons on text, images, videos, and audio, we can ensure that the descriptions are highly consistent with the images, avoid discrepancies between text and images, and guarantee the multimodal integration of the output results.

[0119] Figure 9 The diagram illustrates a multimodal content generation process provided in an embodiment of this disclosure. The process includes: after receiving a prompt word, the system calls an inference model to determine whether the target content is "text-based" or "other modal"; if it is text-based, such as an article or report, the text generation model is called; if it is an image, video, etc., the multimodal model is used for creation; the output content meets the requirements, is rich in content, and has diverse forms.

[0120] In addition to steps S101~S107 above, the method for generating multimodal content based on a knowledge base and index in this embodiment also includes: The generated multimodal content results are fed back to the structured knowledge base and vector database for updating and optimizing knowledge materials and indexes. Specifically, the generated results are fed back to the feature extraction and filtering module, forming a closed-loop feedback loop. The system dynamically adjusts the optimization threshold according to the task type, achieving an adaptive effect of "pursuing accuracy in professional scenarios and maintaining diversity in creative scenarios."

[0121] Based on the above steps and operations, this invention implements "knowledge prior constraints" in the generation process through a dual-drive approach of knowledge base + vector index, and introduces a cyclic filtering and dynamic threshold control mechanism to construct a convergent optimization closed loop. At the same time, it proposes multimodal complementarity and consistency verification logic to achieve integrated generation of cross-modal content. Furthermore, through a diversion-scheduling-fusion mechanism, it innovatively solves the scheduling efficiency and consistency problems in the multi-model collaboration process. By establishing an end-to-end closed loop, it truly connects knowledge retrieval and content generation, enabling the knowledge base to evolve from a static information source into a dynamic driving engine.

[0122] Thus, this invention forms a complete "knowledge-index-extraction-generation" chain, overcoming the shortcomings of existing technologies such as high illusion rate, single modality, and lack of knowledge constraints, and realizing an innovative intelligent content generation system that is scalable, verifiable, and capable of multimodal collaboration.

[0123] In a specific embodiment of the present invention, a closed-loop process is achieved through a comprehensive design of the system's functional structure and workflow, from knowledge material parsing and feature index construction to multimodal content generation based on the required text. The following will detail the implementation of the entire system from the perspective of its actual operation, in conjunction with the functional modules and accompanying drawings, focusing on the technical path and execution logic of each functional step. A specific implementation example is as follows: The following are specific embodiments of the present invention to illustrate the application of generating multimodal content based on knowledge bases and indexes according to demand text.

[0124] Requirement text example: "Please add compliant structural design drawings to Section 1 of Chapter 7 of the 'Specification for Mechanical Safety Protection Devices', including side views, sectional views, and functional breakdown diagrams of typical protective covers, interlocking structures, and emergency braking components. The drawings must conform to the national standard engineering drawing style and be suitable for embedding in the main text of the standard document." 1. Feature Extraction Process After receiving the input text, the system immediately starts the "feature extraction module" and performs the following actions: Keyword identification: Extract keywords such as "mechanical safety", "protective cover", "interlocking structure", "emergency braking", "structural design drawing", and "standard style".

[0125] Semantic classification: The task is identified as generating "structural" content, with modal types including "two-dimensional engineering drawings" and "structural sectional views" as visual content, and the output is "graphic materials" that do not involve animation or video.

[0126] Style feature recognition: Extract semantic features emphasized by the user, such as "conforming to the style of national standard engineering drawings" and "suitable for embedding in standard documents", for subsequent style guidance.

[0127] Output feature structure: The system constructs a standardized feature structure, which includes task category, target modality, design theme, drawing style requirements, etc.

[0128] 2. Material Extraction Process After entering the material extraction module, the system performs the following operations based on the extracted feature structure: semantic vector generation and index retrieval: input the composite semantic vector such as "mechanical safety protection device + structural diagram + engineering drawing style" into the vector database, and retrieve the closest existing material index.

[0129] Material Retrieval: The system retrieves several relevant materials from the knowledge base, including: archived mechanical safety protection structure sketches; standard appendix data from previous years; structural diagrams from textbooks, papers, and patent documents; and neutral format drawings (such as SVG, PNG, etc.) extracted from CAD drawings.

[0130] 3. Material Selection Process To ensure that the output graphic materials accurately match the current standard chapters, the system enters the material screening stage and performs feature correlation judgment: By using a large inference model, we analyze the descriptive text or annotation information of each structural component in the candidate graphic materials one by one to determine whether they belong to the category of "typical protective structures".

[0131] Output graphic list: Finally, retain 5-7 high-quality graphic materials, including the side view of the protective cover, the sectional view of the interlocking structure, and the functional explosion diagram of the emergency braking module.

[0132] 4. Prompt word generation process The system automatically combines the user's original requirements with the filtered graphic content to construct prompts for generating the task: The organization prompts the following logic: guide the output of standardized graphical descriptions for the large model; specify the structural name, application location, and graphical perspective of each figure; emphasize the purpose of "for embedding in standard documents" and maintain neat layout.

[0133] Generate natural language prompts: For example: "Please generate a side view showing the structure of an industrial equipment protective cover. The view should indicate key connection points and fixing methods, follow the style of national standard engineering drawings, and be suitable for embedding in standard document chapters." Prompt words are output to the multimodal generation module.

[0134] 5. Multimodal content generation process Modality determination and model scheduling: The target is "high-quality engineering drawing style image", and the system selects the image generation model branch; based on the prompt words, it calls the multimodal generation model dedicated to structural diagram rendering.

[0135] Content Composition and Graphics Generation: The system composes the corresponding structural sketch draft according to the prompts; it draws each structural part in a modular fashion; and it uses standard engineering drawing elements (such as projected views, center lines, section lines, dimensions, etc.) for composite rendering.

[0136] Image generation example, image processing and metadata addition: The generated image is output in SVG+PNG dual format; metadata such as explanatory text, image number, file path, and generation time are added; and it is pushed to the standard writing system interface for user review and embedding in documents.

[0137] Figure 10 This is a schematic diagram of an apparatus for generating multimodal content based on a knowledge base and index, provided in another embodiment of this disclosure. The apparatus includes: a feature extraction module 1001, a demand analysis module 1002, a material retrieval module 1003, a material recall module 1004, a material filtering module 1005, a prompt word construction module 1006, and a multimodal generation module 1007.

[0138] in: The feature extraction module 1001 is used to parse the original multi-modal material to extract structured features, store the structured features in a structured knowledge base, and generate semantic vector indexes of the structured features to store in a vector database.

[0139] The purpose of this module is to construct the knowledge infrastructure upon which content generation depends. Specifically, it includes a material import module, a content type identification and parsing module, content feature extraction and structured storage, an input content construction module, a pre-trained vectorized model selection module, a vector calculation module, and a vector normalization module. The functional descriptions of these modules are consistent with the method steps in step S101, and will not be repeated here.

[0140] This module includes a data cleaning module and an index building module. The functional descriptions of the data cleaning module and the index building module are consistent with the method steps in step S101, and will not be repeated here.

[0141] The aforementioned feature structure modules are integrated into a pre-trained vectorization model to generate semantic embedding vectors. Simultaneously, a vector index is created for each material record and stored in a high-performance vector database for rapid subsequent retrieval. The knowledge material library and index system built in this stage will provide comprehensive, accurate, and rapid material support for downstream modules.

[0142] The requirement parsing module 1002 is used to parse the user's requirement text and form requirement feature information.

[0143] The requirement parsing module includes: a requirement receiving module, a language understanding and feature recognition module, and an output feature module.

[0144] The requirement acceptance module is used by users to submit requirement text through the service layer. This text may include task objectives, themes, style preferences, etc.

[0145] The language understanding and feature recognition module is used to call the language understanding model to parse user text, including: extracting keywords and main information; analyzing tone and style preferences (such as formal, casual, academic, etc.); and determining content type preferences (such as generating popular science articles, marketing short videos, or mixed text and image content).

[0146] The output feature structure module organizes all extracted information into a standardized feature format and submits it to the material extraction module as input conditions for material matching.

[0147] The module also includes a feature extraction submodule, specifically used for: 1) The system reads the user's request text; 2) Use language understanding models to identify keywords, main ideas, tone, etc. 3) Output the extraction results as the "feature standard" for the next step of material matching.

[0148] The material retrieval module 1003 is used to convert the demand feature information into a query vector, perform a similarity search in the vector database, and obtain a candidate material index.

[0149] The material retrieval module includes a material extraction submodule, specifically used for: The input is the keywords output from the previous process; the system searches for semantically similar material indexes in the vector database; based on the indexes, it retrieves complete multimodal materials from the knowledge base; and outputs all matching results and their feature lists.

[0150] The material retrieval module 1004 is used to retrieve corresponding candidate materials and their features from the structured knowledge base according to the candidate material index.

[0151] Specifically, the module is used to: convert information such as demand keywords and themes into semantic vectors, compare them with the index in the vector database, and recall several material items with the closest semantics.

[0152] The material screening module 1005 is used to evaluate the matching degree between the candidate materials and the required feature information using a large inference model, and to perform multiple rounds of iterative screening based on preset termination conditions to obtain the target materials.

[0153] The module also includes a material filtering module, specifically used for: The input consists of candidate materials and required keywords; an inference model is used to evaluate each material to see if it matches the requirements; irrelevant content is deleted, and only materials that meet the criteria are retained; the final output is the high-quality materials after filtering.

[0154] The preset termination conditions of the module include three types: Termination condition 1: High matching degree threshold condition Termination condition 2: Modal coverage condition Termination Condition 3: Content Value Condition Please refer to the description in step 105 above for details, which will not be repeated here.

[0155] This module performs multiple rounds of iterative filtering based on preset termination conditions to obtain target materials, and also includes: The cyclic adaptive convergence module is specifically designed to: avoid infinite loops by setting a maximum number of iterations (e.g., 10 rounds); after exceeding this limit, the system automatically retains and outputs the best result. This mechanism ensures that the generation process does not fall into an infinite loop while dynamically guaranteeing quality. If any of the aforementioned termination conditions are not met when the maximum number of iterations is reached, the material with the highest matching degree in the current round is output as the target material.

[0156] The prompt word construction module 1006 is used to merge the requirement text with the target material and construct a structured prompt word containing task instructions and logical constraints through a large text generation model.

[0157] The module includes: a module for integrating materials and required content, a module for constructing prompt statement structures, a module for semantic optimization and naturalization, and a module for outputting the final prompt words.

[0158] The integrated materials and requirements module combines the user's original input requirements with filtered, high-quality materials. The system extracts core content blocks by setting rules and identifies their logical relationships.

[0159] A prompt statement structure module is constructed to generate "task instructions" that adapt to the input of the text generation model based on different content objectives.

[0160] The semantic optimization and naturalization module is used to refine and normalize the prompt statements using the text generation model, making them conform to the expression habits of natural language and improving the understanding ability and execution effect of the generation model.

[0161] The final prompt word output module outputs the structured prompt word text to the generation module as a direct driving signal for the generation action.

[0162] This module includes a prompt word generation module, which is specifically used for: The input consists of user requirements and source content; the system combines these two elements and uses a generative model to write a "guided question" or "task instruction"; the output prompts will be directly used to drive content generation.

[0163] The structured prompts employ a hierarchical structure, including: a basic field layer, a semantic tag layer, a task feature layer, and a logical dependency layer. The logical dependency layer stores data in a cross-modal graph database, used to establish semantic relationships between different modal materials.

[0164] The multimodal generation module 1007 is used to determine the target content modality to be generated based on the structured prompt words, and to schedule the corresponding multimodal generation model to generate multimodal content results.

[0165] The module includes a splitting submodule, a scheduling submodule, a generation submodule, and a fusion submodule.

[0166] The splitting submodule is used to determine the target content modality based on the structured prompt words and to determine the model type to be called.

[0167] The scheduling submodule is used to select the optimal model from a pool of models of the same type based on task requirements and model performance weights.

[0168] The generation submodule is used to call the selected model to generate the content of the corresponding modality.

[0169] The fusion submodule is used to perform consistency verification and fusion of the generated content in different modalities.

[0170] This module also includes: The modal complementarity module is used to automatically schedule other modal generation models to supplement the content if the content generation results of a certain modality are insufficient.

[0171] Based on the prompts, a suitable large model is dispatched to complete the final content generation, which specifically includes: I) Content Type Judgment and Traffic Distribution Module The reasoning module determines whether the final generated target is plain text, mixed text and images, a short video clip, or a voice broadcast based on the prompt words, and selects the appropriate generation engine accordingly.

[0172] A three-level mechanism of "diversion-scheduling-fusion" is adopted, as described in step S107 above, and will not be repeated here.

[0173] II) Model Scheduling and Generation Execution Module If the output is text, the large text generation model is invoked to construct paragraphs, logical structures, and expression styles based on prompts; if the output is image, the large image-text multimodal model is invoked to draw scenes, illustrations, or visual materials based on prompts; if the output is video or audio, the large audio multimodal model is used to generate audio.

[0174] III) Content encapsulation and result output module The generated content will be packaged into a uniform output format (such as TXT, MP4, or PNG) according to its type, and will also include metadata such as content tags, reliability scores, and generation time.

[0175] This invention proposes a "three-dimensional multimodal invocation mechanism," which differs from traditional single-modal or simple combination invocation: Dimension 1: Modality Priority Decision Tree Dimension 2: Modal Complementarity Mechanism Dimension 3: Cross-modal consistency verification The details of these three bitmaps are described in step S107 above and will not be repeated here.

[0176] The multimodal content generation module is specifically used for: after receiving prompt words, the system calls the inference model to determine whether the target content is "text-based" or "other modal"; if it is text-based, such as articles or reports, the text generation model is called; if it is images, videos, etc., the multimodal model is used for creation; the output content meets the requirements, is rich in content, and has diverse forms.

[0177] In addition to steps S101~S107 above, the method for generating multimodal content based on a knowledge base and index in this embodiment also includes: Furthermore, the device also includes a feedback optimization module, used to feed the generated multimodal content results back to the feature extraction module, demand analysis module, material retrieval module, and material filtering module. The result data is then fed back to the structured knowledge base and vector database, forming a closed-loop feedback to dynamically optimize the filtering threshold and generation strategy. The system dynamically adjusts the optimization threshold based on the task type, achieving an adaptive effect of "pursuing accuracy in professional scenarios and maintaining diversity in creative scenarios."

[0178] Figure 10 The device shown can perform Figure 3 For the methods shown in the embodiments, the parts not described in detail in this embodiment can be referred to the following: Figure 3 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figure 3 The descriptions in the illustrated embodiments will not be repeated here.

[0179] Figure 11 This is a schematic diagram of an apparatus for generating multimodal content based on a knowledge base and index, according to another embodiment of this disclosure. The apparatus includes: a storage module 1101, an extraction module 1102, a processing module 1103, a generation module 1104, and a service module 1105.

[0180] in: Storage module 1101 is used to store the structured knowledge base and vector database constructed by any of the methods described above.

[0181] The extraction module 1102 is used to receive the user-inputted requirement text, parse the requirement text, and extract feature information.

[0182] The processing module 1103 is used to retrieve, recall, and filter target materials from the vector database and the structured knowledge base based on the feature information.

[0183] The generation module 1104 is used to merge the requirement text with the target material to generate structured prompt words, and to schedule a multimodal large model to generate the final content.

[0184] Reference Appendix Figure 1 The system architecture, with the storage module serving as the system's information infrastructure, comprises two parts: knowledge material storage, which saves original files such as PDFs, images, and audio; and a knowledge base, which includes a traditional database (for storing structured information) and a vector database (for semantic matching and retrieval), using vector indexes to improve search efficiency and accuracy.

[0185] The extraction module is responsible for understanding user input and searching for and extracting relevant content from the existing resource library. This module is further divided into three sub-modules: Feature extraction: Analyze user requirement text to extract "feature information" such as keywords, themes, and style requirements.

[0186] Material extraction: These features are used to search for matching material content in the database, which may be in various forms such as images, text, and audio.

[0187] Material filtering: Further filter the search results to remove irrelevant information and retain the most valuable materials.

[0188] The processing module is used to retrieve, recall, and filter target materials from the vector database and the structured knowledge base based on the feature information.

[0189] Specifically, it includes the above-mentioned modules: requirement analysis, material retrieval, material recall, material filtering, and prompt keyword construction. For detailed functions, please refer to the above. Figure 10 The corresponding device description will not be repeated here.

[0190] The service module serves as the interface between the user and the entire system, responsible for receiving the user's input request text. The service layer then sends these requests to the backend processing module and returns the final generated content to the user (Note: The input of request text and the output of materials can be achieved through application interfaces, APIs, etc., which are not the focus of this patent and will not be elaborated here).

[0191] The generation module includes a prompt word generation module and a multimodal generation module, wherein: Prompt word generation module: Uses text to generate a large model, combining materials and user needs into prompt words that the system can understand; Multimodal content generation module: Based on prompts, it uses a multimodal model to output final content such as images, audio, video, or text.

[0192] The large model module includes a large text generation model, a large inference model, and a multimodal model. Among them: The large-scale text generation model is responsible for thinking about and outputting text-based content; Large-scale reasoning model: used to determine whether the material meets the requirements and to analyze the logical structure; Multimodal large models: capable of understanding non-textual information such as images and videos, and generating multimodal content.

[0193] The service module provides a user interface, receives the request text, and outputs the final content.

[0194] The structure adopts a "four-layer structure + cross-modal graph database" design: Basic field layer: metadata such as title, time, source, and file type; Semantic tagging layer: Domain tags automatically generated by the large model (such as "mechanical safety" and "structural design"); Task feature layer: includes user requirement features (style preferences, accuracy level, modality combination requirements); Logical Dependency Layer: Defines the semantic and structural relationships between different materials (e.g., "Image A" corresponds to "Text B" description).

[0195] Cross-modal graph database storage: This not only stores data but also establishes "relation edges" between different modalities, forming a knowledge graph-style structured storage. During retrieval, it can simultaneously retrieve "related images + descriptive text," rather than isolated results, ensuring cross-modal semantic connectivity.

[0196] The following is for reference. Figure 12 This illustration shows a structural diagram of an electronic device 1200 suitable for implementing another embodiment of the present disclosure. The terminal devices in the embodiments of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 12 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0197] like Figure 12As shown, the electronic device 1200 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage device 1208 into a random access memory (RAM) 1203. The RAM 1203 also stores various programs and data required for the operation of the electronic device 1200. The processing unit 1201, ROM 1202, and RAM 1203 are interconnected via a communication line 1204. An input / output (I / O) interface 1205 is also connected to the communication line 1204.

[0198] Typically, the following devices can be connected to I / O interface 1205: input devices 1206 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1207 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1208 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1209. Communication device 1209 allows electronic device 1200 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 12 An electronic device 1200 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0199] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1209, or installed from storage device 1208, or installed from ROM 1202. When the computer program is executed by processing device 1201, it performs the functions defined in the methods of embodiments of this disclosure.

[0200] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0201] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0202] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0203] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the interaction method described in the above embodiments.

[0204] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0205] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0206] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0207] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0208] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0209] According to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any of the methods described in the first aspect above.

[0210] According to one or more embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, characterized in that the non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform any of the methods described in the first aspect above.

[0211] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0212] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating multimodal content based on a knowledge base and index, characterized in that, Includes the following steps: The original multi-modal material is parsed to extract structured features, the structured features are stored in a structured knowledge base, and the semantic vector index of the structured features is generated and stored in a vector database. The user's request text is parsed to form request feature information; The demand feature information is converted into a query vector, and a similarity search is performed in the vector database to obtain a candidate material index; Based on the candidate material index, retrieve the corresponding candidate materials and their features from the structured knowledge base; The candidate materials are evaluated for their matching degree with the required feature information using a large inference model. Multiple rounds of iterative screening are performed based on preset termination conditions to obtain the target materials. The required text is merged with the target material, and a structured prompt word containing task instructions and logical constraints is constructed through a large text generation model; Based on the structured prompts, the target content modality to be generated is determined, and the corresponding multimodal generation model is scheduled to generate the multimodal content result.

2. The method according to claim 1, characterized in that, The process of parsing the original multi-modal material, extracting structured features, storing the structured features in a structured knowledge base, and generating semantic vector indexes of the structured features and storing them in a vector database includes the following steps: Import original materials in multiple modalities; Identify the modal type of the original material; Based on the modality type, the corresponding feature extraction model is invoked to extract the structured features of the original material. The structured features include subject matter, type, keywords, and tags. The structured features are stored in the structured knowledge base; The structured features are input into the semantic encoding model to generate corresponding semantic vectors; The semantic vectors are stored in the vector database and an index is created.

3. The method according to claim 1, characterized in that, The preset termination condition includes at least one of the following: High matching threshold condition: The matching degree of candidate materials reaches the preset threshold; Modal coverage condition: The selected materials cover all content modalities specified in the requirement feature information; Content value criteria: The accuracy and clarity of the selected materials meet the preset value threshold.

4. The method according to claim 1 or 3, characterized in that, The step of using a large inference model to evaluate the matching degree between the candidate materials and the demand feature information, and performing multiple rounds of iterative screening based on preset termination conditions to obtain the target materials, also includes: Adaptive convergence mechanism: Set a maximum number of iteration rounds. If any of the termination conditions are not met when the maximum number of iteration rounds is reached, output the material with the highest matching degree in the current round as the target material.

5. The method according to claim 1, characterized in that, The structured prompts adopt a hierarchical structure, including: a basic field layer, a semantic tag layer, a task feature layer, and a logical dependency layer.

6. The method according to claim 5, characterized in that, The data of the logical dependency layer is stored in a cross-modal graph database and is used to establish semantic relationships between different modal materials.

7. The method according to claim 1, characterized in that, The step of determining the target content modality to be generated based on the structured prompts and scheduling the corresponding multimodal generation model to generate the multimodal content result specifically includes: The splitting sub-step involves determining the target content modality based on the structured prompts and identifying the model type to be invoked. Scheduling sub-step: Select the optimal model from the pool of models of the same type based on task requirements and model performance weights; Generation sub-step: Call the selected model to generate the content of the corresponding modality; Fusion sub-step: Perform consistency verification and fusion on the generated different modal contents.

8. The method according to claim 7, characterized in that, The step of determining the target content modality to be generated based on the structured prompts and scheduling the corresponding multimodal generation model to generate multimodal content results also includes: A modal complementarity mechanism is adopted: if the content generation result of a certain modality is insufficient, other modal generation models are automatically scheduled to supplement the content.

9. The method according to claim 1, characterized in that, The method further includes: The generated multimodal content results are fed back to the structured knowledge base and vector database for updating and optimizing knowledge materials and indexes.

10. An apparatus for generating multimodal content based on a knowledge base and index, characterized in that, include: The feature extraction module is used to parse the original multi-modal material to extract structured features, store the structured features in a structured knowledge base, and generate semantic vector indexes of the structured features and store them in a vector database. The requirement parsing module is used to parse the user's requirement text and form requirement feature information; The material retrieval module is used to convert the demand feature information into query vectors, perform similarity retrieval in the vector database, and obtain a candidate material index. The material retrieval module is used to retrieve corresponding candidate materials and their features from the structured knowledge base based on the candidate material index. The material screening module is used to evaluate the matching degree between the candidate materials and the required feature information using a large inference model, and to perform multiple rounds of iterative screening based on preset termination conditions to obtain the target materials. The prompt word construction module is used to merge the requirement text with the target material and construct structured prompt words containing task instructions and logical constraints through a large text generation model; The multimodal generation module is used to determine the target content modality to be generated based on the structured prompts and to schedule the corresponding multimodal generation model to generate the multimodal content result.

11. The apparatus according to claim 10, characterized in that, The device also includes a feedback optimization module, which is used to feed back the generated multimodal content results to the feature extraction module, the demand analysis module, the material retrieval module, and the material filtering module to form a closed-loop feedback to dynamically optimize the filtering threshold and the generation strategy.

12. An apparatus for generating multimodal content based on a knowledge base and index, characterized in that, include: A storage module for storing a structured knowledge base and a vector database constructed by the method described in any one of claims 1-9; The extraction module is used to receive the user's input request text, parse the request text, and extract feature information; The processing module is used to retrieve, recall, and filter target materials from the vector database and the structured knowledge base based on the feature information. The generation module is used to merge the requirement text with the target material to generate structured prompt words, and to schedule a multimodal large model to generate the final content; The service module provides a user interface, receives the request text, and outputs the final content.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-9.

Images