An Adaptive Multimodal Rich Text Intelligent Generation Method and System
By constructing a knowledge base of text and image vectors and dynamically fusing multimodal data, the problems of image-text separation and information traceability when generating rich text from large language models are solved, and efficient fusion and accurate generation of multimodal knowledge are achieved.
Patent Information
- Application Number
- CN202511418126.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-30
- Publication Date
- 2026-11-17
AI Technical Summary
Existing large language models suffer from problems such as image-text separation, inaccurate content generation, and untraceable information sources when generating rich text, making it difficult to effectively integrate multimodal data.
By constructing a knowledge base of text and image vectors, using a vectorized model to retrieve the text and image vectors most relevant to the user's question, generating prompt word templates, and dynamically integrating text and image content into a large language model, the system can detect and replace image tags in real time, achieving adaptive generation and source tracing of text and images.
It achieves efficient integration of multimodal domain knowledge, solves the problem of image and text separation, ensures the traceability of data sources, and improves the robustness and accuracy of the generated data.
Smart Images

Figure CN123115301A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of text intelligence development technology, and in particular to an adaptive multimodal rich text intelligent generation method and system. Background Technology
[0002] In recent years, large language model (LLM) technology has developed rapidly (such as GPT-4 and DeepSeek), and its application in the field of text generation has become increasingly widespread. However, the core capabilities of existing large models are still limited to plain text output. Although some models have attempted to incorporate descriptions of unstructured data such as images, the generated content is still mainly linear text and lacks deep integration with multimodal data.
[0003] On the user end, rich text (such as text-image hybrid text) has become the mainstream form of information expression. For example, in scenarios such as education, healthcare, and business analytics, users need to combine visual data with structured information to assist in decision-making. However, the pure text output mode of existing large models has significant limitations, including: the text search and image process relies on manual operation, reducing efficiency; the logic between text and images is disconnected, which can easily lead to the risk of content illusion and generate inaccurate or false information; and the source of images cannot be traced, affecting credibility. Summary of the Invention
[0004] The main objective of this invention is to provide an adaptive multimodal rich text intelligent generation method and system, which realizes the automatic generation and source tracing of structured text and image content by dynamically integrating multimodal knowledge in the domain. The technical solution adopted in this invention is: an adaptive multimodal rich text intelligent generation method, comprising: Raw data is collected from relevant areas of user questions, and text fragments and images are extracted and processed separately. A text knowledge base is built using the text fragments and the text vectors obtained from their processing, and an image knowledge base is built using the images and the image vectors obtained from their processing. The user's question is vectorized by a vectorization model. The most relevant text vectors and image vectors and corresponding documents are retrieved from the text knowledge base and image knowledge base. All documents are numbered and combined into prompt word templates. Input the prompt word template into the large language model to receive the streaming answer; concatenate the answer in real time, and then check if there is a predefined image tag; if there is a predefined image tag, trace the original data corresponding to the image tag and find the nearest image. If the nearest image exists, replace the image tag with the actual address of the nearest image and re-output the answer; otherwise, obtain the content block with the closest image tag, perform vectorization processing, search for the closest image from the image knowledge base, replace the image tag with the actual address of the closest image, and re-output the answer; if there is no predefined image tag, immediately output the processed part of the answer.
[0005] According to the above technical solution, the specific methods for processing text fragments include: removing HTML tags and advertisements from the text fragments using XPath or regular expressions, and extracting the plain text of the text fragments; The TextRank algorithm is used to perform semantic analysis on the extracted plain text, dividing the document into multiple semantic units with independent meaning; Semantic units are encoded using a text vectorization model to generate high-dimensional text vectors; the text vectorization model includes the Bge-large model. The text vectors and their corresponding original text fragments are used to construct a text knowledge base.
[0006] According to the above technical solution, the TextRank algorithm is obtained through the formula... Calculate sentence weights in plain text; where, Sentence The weights; It is the damping coefficient, representing the probability of a random jump; It refers to all sentences. A collection of sentences; it is a set of sentences. The set of all sentences it points to; Sentence and sentences The similarity between them.
[0007] According to the above technical solution, the specific methods for processing images include: extracting images from the original data using XPath or regular expressions; The extracted images are encoded using an image vectorization model and converted into numerical vectors; the image vectorization model includes the Bge-m3 model. An image knowledge base is constructed by combining image vectors with their corresponding original images.
[0008] According to the above technical solution, the method for retrieving the most relevant text vectors and image vectors to the user question from the text knowledge base and the image knowledge base includes: comparing the cosine similarity of the user question vector to each text vector in the text knowledge base and each image vector in the image knowledge base; the calculation formula includes:
[0009] Where A represents the user question vector, and B represents the text vector in the text knowledge base or the image vector in the image knowledge base; Represents the cosine similarity between A and B; and Let A and B represent the values of vectors A and B in the i-th dimension, respectively. The text vector and image vector corresponding to the highest cosine similarity are selected as the text vector and image vector most relevant to the user's question.
[0010] According to the above technical solution, the method for tracing the original data corresponding to the image tag specifically includes: extracting the document number from the image tag using regular expressions; locating the document to which the image tag belongs based on the number; and tracing the corresponding original data using the metadata of the document.
[0011] According to the above technical solution, the method for finding the nearest image specifically includes: Based on the position of the image markers, semantic segmentation is performed on the answer to obtain the content block that is closest to the image markers; The content block is vectorized using a vectorization model; Retrieve the image vector with the highest cosine similarity to the content block vector from the image knowledge base, and take the corresponding image as the closest image.
[0012] Another aspect of the present invention provides an adaptive multimodal rich text intelligent generation system, including a domain knowledge base construction module, a prompt word template construction module, and a rich text answer generation module; wherein, The domain knowledge base construction module is used to collect raw data from relevant domains and extract text fragments and images from them, process them separately, and build a text knowledge base using text fragments and the text vectors obtained from their processing, and build an image knowledge base using images and the image vectors obtained from their processing. The prompt word template building module is used to vectorize user questions through a vectorization model, retrieve the most relevant text vectors and image vectors from the text knowledge base and image knowledge base, and combine them into prompt word templates; The rich text answer generation module is used to input prompt word templates into a large language model and receive the streamed answers. The answers are then concatenated in real time, and the module checks for predefined image tags. If a predefined image tag exists, the module traces the original data corresponding to that tag and searches for the nearest image. If a nearest image exists, the image tag is replaced with the actual address of the nearest image, and the answer is re-output. Otherwise, the module retrieves the content block closest to the image tag, vectorizes it, searches the image knowledge base for the closest image, replaces the image tag with the actual address of the closest image, and re-outputs the answer. If no predefined image tag exists, the processed portion of the answer is immediately output.
[0013] Another aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described adaptive multimodal rich text intelligent generation method.
[0014] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described adaptive multimodal rich text intelligent generation method.
[0015] The beneficial effects of this invention are as follows: This invention provides an adaptive multimodal rich text intelligent generation method and system. By separately constructing text and image vector knowledge bases and dynamically retrieving them across modalities, it achieves efficient fusion and accurate retrieval of multimodal domain knowledge. Through real-time splicing of streaming output and a predefined image tagging traceability mechanism, it solves the problem of image-text position separation and ensures the traceability of data sources. Through the complementary mechanism of semantic segmentation vectorization and cross-database image retrieval, it ensures the robustness and resource accuracy of rich text generation in complex scenarios.
[0016] Furthermore, this invention also achieves high-precision structured storage of text knowledge base by using the TextRank algorithm to realize unsupervised semantic segmentation and weight calculation, which significantly improves the semantic matching efficiency of subsequent retrieval. Furthermore, this invention also solves the problem of accurate alignment between text and image resources and user intent through a cross-modal vector comparison mechanism based on cosine similarity, thereby reducing the risk of error propagation in multimodal association. Furthermore, this invention also constructs a closed-loop verification system for image resources through a dual-path tracing mechanism of document number location and semantic segmentation, reducing the risk of unreliable image citation sources.
[0017] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart of the adaptive multimodal rich text intelligent generation method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the construction process of the domain multimodal knowledge base in the adaptive multimodal rich text intelligent generation method according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating the process of generating rich text answers in the adaptive multimodal rich text intelligent generation method of this invention. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0021] It should be noted that the illustrations provided in the embodiments of the present invention are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0022] In this invention, it should also be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first" and "second" are used only for descriptive and distinguishing purposes and should not be construed as indicating or implying relative importance.
[0023] Example 1 This embodiment provides an adaptive multimodal rich text intelligent generation method, the steps of which include: T1. Collect raw data from the relevant domain of user questions and extract text fragments and images from them. Process them separately. Build a text knowledge base using the text fragments and the text vectors obtained from their processing, and build an image knowledge base using the images and the image vectors obtained from their processing.
[0024] Specifically, collecting raw data from the relevant domain of the user's question means collecting initial data that is directly related to the domain of the user's question. The data source must match the domain attributes of the user's question to ensure the domain relevance of the knowledge base content.
[0025] Extracting text fragments and images for separate processing refers to separating textual and image information from the collected raw data and applying different processing logics to the two types of information. Text processing generates text vectors, and image processing generates image vectors, thus achieving the classification and processing of text and image data.
[0026] Building a text knowledge base by using text fragments and the text vectors obtained from processing them refers to associating and storing the processed text fragments with the text vectors generated from those text fragments, forming a collection specifically for storing text-related domain knowledge.
[0027] Building an image knowledge base by using images and the image vectors obtained from processing them refers to associating and storing the processed images with the image vectors generated from those images, forming a collection specifically for storing image-related domain knowledge.
[0028] T2. Vectorize the user's question using a vectorization model. Retrieve the most relevant text vectors and image vectors, along with their corresponding documents, from the text and image knowledge bases. Number all documents and combine them into a prompt word template.
[0029] Specifically, vectorizing user questions through a vectorization model means using a vectorization model to transform the questions raised by users into vector form, so that the questions have a numerical form that can be matched with vectors in the knowledge base.
[0030] Retrieving the most relevant text vectors and image vectors and their corresponding documents from the text knowledge base and image knowledge base means selecting the text vectors and image vectors with the highest matching degree with the user's question vector from the constructed text knowledge base and image knowledge base, and at the same time obtaining the original documents corresponding to these vectors.
[0031] All documents are numbered and combined into prompt word templates. This means that a unique identifier is assigned to the document corresponding to the retrieved text vector and the document corresponding to the image vector, and these are integrated into structured input content for subsequent input into the large language model.
[0032] T3. Input the prompt word template into the large language model and receive the streaming answer. Concatenate the answer in real time, and then check if there is a predefined image tag. If there is a predefined image tag, trace the original data corresponding to the image tag and find the nearest image. If the nearest image exists, replace the image tag with the actual address of the nearest image and re-output the answer. Otherwise, obtain the content block with the closest image tag and perform vectorization processing. Search for the closest image from the image knowledge base, replace the image tag with the actual address of the closest image, and re-output the answer. If there is no predefined image tag, immediately output the processed part of the answer.
[0033] Specifically, inputting the prompt word template into the large language model and receiving the streaming answer means inputting the constructed prompt word template into the large language model and receiving the answer content returned by the model in the form of outputting answer fragments in real time, rather than outputting the complete answer all at once.
[0034] The answers are spliced together in real time, and then the presence of predefined image tags is checked. This means that the answer fragments output by the model are integrated and spliced in real time to ensure the continuity of the answers, and at the same time, the spliced answers are checked to see if they contain predefined image tags.
[0035] If a predefined image tag exists, trace the original data corresponding to that image tag and find the nearest image. If a nearest image exists, replace the image tag with the actual address of the nearest image and re-output the answer. In other words, if an image tag is detected, find its corresponding original data through the tag association information, and filter the original data for the image closest to the current answer content. If the image is found, replace the image tag in the answer with the actual storage address of the image, and then output the answer containing the image address.
[0036] Otherwise, the content block closest to the image tag is obtained and vectorized. The closest image is searched from the image knowledge base, and the image tag is replaced with the actual address of the closest image. The answer is then output again. If the nearest image is not found in the original data, the text fragment closest to the image tag is extracted, the content block is converted into a vector, and the image with the highest matching degree in the image knowledge base is selected. The tag is replaced with the actual address of the image and the answer is output.
[0037] If no predefined image markers are found, the processed partial answer is output immediately. This means that if no image markers are detected, the processed partial answer is output in real time without waiting for the model to output the complete answer, ensuring the timeliness of answer generation.
[0038] Example 2 Based on the adaptive multimodal rich text intelligent generation method described in Example 1, this example provides another adaptive multimodal rich text intelligent generation method, the process of which is as follows: Figure 1 As shown, the steps include: S1. Construct a domain knowledge base, the process is as follows: Figure 2 As shown. First, data is collected and stored in the original database. Then, the data is processed in two ways: one way extracts plain text, performs semantic segmentation using TextRank, vectorizes it using the Bge-large model, and stores it in the text knowledge base; the other way extracts images, vectorizes them using the Bge-m3 model, and stores them in the image knowledge base. The specific details are as described in step S1.
[0039] S101. Collect raw data from the relevant areas of user questions and extract text fragments and images from them, and process them separately.
[0040] Specifically, methods for collecting raw data from relevant fields include obtaining publicly available data from reliable sources through web crawlers, storing the collected raw data in Elasticsearch (an open-source distributed search and analytics engine) for easy management and retrieval.
[0041] Methods for processing text fragments include removing irrelevant content such as HTML tags and advertisements from the text fragments using XPath, regular expressions, etc., to extract the plain text content.
[0042] The extracted plain text is semantically analyzed using the TextRank algorithm, which segments it into several semantic units with independent meaning, in order to improve the efficiency and accuracy of subsequent processing.
[0043] Furthermore, TextRank is an unsupervised learning method that does not require pre-labeled datasets to train the model, allowing for rapid application to new domains or languages. By converting text into a graph structure, where nodes represent sentences and edges represent the similarity between sentences, TextRank can capture complex structural information within a document, more accurately identifying important sentences and paragraphs. In addition, TextRank can be used not only for semantic segmentation but also extended to other natural language processing tasks, such as keyword extraction and summary generation, demonstrating strong application flexibility. Because it is based on a simple mathematical model like PageRank, TextRank is relatively easy to implement, and its performance can be optimized by adjusting parameters such as the damping coefficient d.
[0044] The TextRank algorithm calculates sentence weights in plain text using its core formula:
[0045] in, Sentence The weights; It is the damping coefficient, representing the probability of a random jump; It refers to all sentences. A collection of sentences; it is a set of sentences. The set of all sentences it points to; Sentence and sentences The similarity between them.
[0046] The semantic units mentioned above are encoded using a pre-trained text vectorization model to generate high-dimensional vector representations. These vectors can capture the semantic features of the text, providing a foundation for subsequent knowledge retrieval. In this embodiment, the Bge-large model is selected as the text vectorization model.
[0047] The image processing method includes extracting images from the original data using methods such as XPath and regular expressions, and then encoding the extracted images using a specialized image vectorization model for image processing, converting them into numerical vector forms. This step allows image data to be processed and compared within the same framework as text data. In this embodiment, the Bge-m3 model is selected as the image vectorization model.
[0048] Text fragments and their processed text vectors are stored in Elasticsearch to build a text knowledge base, and images and their processed image vectors are stored in Elasticsearch to build an image knowledge base.
[0049] Next, rich text answers to user questions are generated based on domain knowledge bases (text knowledge base, image knowledge base), the process is as follows: Figure 3 As shown, the process begins with inputting a user question; then, the Bge-large model is used to vectorize the question and retrieve it from a text knowledge base; finally, templates are assembled, and the question is then answered using a large model.
[0050] After generating content, it checks if an image tag is attached. If not, it is directly streamed. Otherwise, regular expressions are used to extract the document number from the tag and correlate it with the original database (image knowledge base, text knowledge base). Combining the text vector with the original data, it attempts to match the nearest image in the original data based on the text vector. If a match is successful, the tag is replaced with the corresponding image address and streamed. If a match fails, the first part of the generated content is extracted based on the tag position, vectorized using the Bge-m3 model, the image knowledge base is searched, the image with the highest similarity is extracted, the tag is replaced with its address, and then streamed. The specific details are as described in steps S2 to S4.
[0051] S2. Construct prompt word templates.
[0052] S201. Vectorize user questions using a vectorization model.
[0053] Specifically, the questions raised by users in the question-and-answer session are also vectorized using the Bge-large model.
[0054] S202. Retrieve the text vectors and image vectors, and their corresponding documents that are most relevant to the user's question from the text knowledge base and the image knowledge base.
[0055] Using the kNN vector retrieval method provided by Elasticsearch, we retrieve the text vectors and image vectors most relevant to the user's question from the text knowledge base and the image knowledge base, and obtain the documents to which these vectors belong.
[0056] k-Nearest Neighbors (kNN) search in Elasticsearch is a technique for efficiently finding the k nearest neighbors to a query vector in a large number of documents. It is suitable for scenarios such as recommendation systems, image search, and similarity search. It can quickly find nearest neighbors in very large datasets without having to compare each vector pairwise, which can significantly improve retrieval speed.
[0057] In this embodiment, the distributed architecture of Elasticsearch, which stores the data, allows kNN search to be executed in parallel across multiple nodes. This enables it to handle large-scale datasets and scale as the data volume grows. It can also be seamlessly integrated with other Elasticsearch features (such as full-text search and filters) to provide a richer search experience. kNN search also supports various distance metrics, such as Euclidean distance, cosine similarity, and inner product, allowing you to choose the most suitable method based on your specific application scenario.
[0058] In this embodiment, cosine similarity is used to retrieve the text and image vectors most relevant to the user's question. Cosine similarity is particularly suitable for vector comparison in high-dimensional spaces, such as comparing text or image feature vectors. In these scenarios, it can effectively measure the angular differences between vectors, thereby determining whether their directions are similar, rather than simply based on distance. The formula is as follows:
[0059] Where A represents the user question vector, and B represents the text vector in the text knowledge base or the image vector in the image knowledge base; Represents the cosine similarity between A and B; and Let A and B represent the values of vectors A and B in the i-th dimension, respectively.
[0060] The text vector and image vector corresponding to the highest cosine similarity are selected as the text vector and image vector most relevant to the user's question.
[0061] S203. Number all documents and combine them into a prompt word template.
[0062] For example, the first document retrieved is numbered 1, and the second document is numbered 2.
[0063] S3. Input the prompt word template into the large language model, detect the image tags in the output answer and replace them with the corresponding image addresses.
[0064] S301. Input the prompt word template into the large language model, receive the streaming output of the answer, and execute step S302 simultaneously.
[0065] S302. Immediately assemble the answers and then check if there are any predefined image markers. If there are predefined image markers, proceed to step S303; otherwise, proceed to step S304.
[0066] S303. Trace the original data corresponding to the image tag and find the nearest image. If a nearest image exists, replace the image tag with the actual address of the nearest image and re-output the answer. Otherwise, obtain the content block of the closest image tag and perform vectorization processing. Search for the closest image from the image knowledge base, replace the image tag with the actual address of the closest image, and re-output the answer.
[0067] Specifically, when an image tag is identified, a document number is extracted from the tag using a regular expression. The document retrieved in step S2 is located based on the document number, and the metadata of the document is used to trace back to the original data collected in step S1.
[0068] Based on the above text vector, the nearest image is found in the corresponding original data. If a matching image is found, the image tag in the answer is replaced with the actual address of that image and output.
[0069] If a suitable image is not successfully scanned, the answer is semantically segmented forward based on the position of the image tag to obtain the content block that is closest to the image tag. The Bge-m3 model is used to vectorize this content block. The KNN search method of Elasticsearch is used to retrieve the most similar image vector and the corresponding image from the image knowledge base. The actual address of the found image is used to replace the original image tag and output the answer.
[0070] S304. Immediately output the processed portion of the answer.
[0071] This embodiment also provides an adaptive multimodal rich text intelligent generation system, including a domain knowledge base construction module, a prompt word template construction module, and a rich text answer generation module; wherein... The domain knowledge base construction module is used to collect raw data from relevant domains and extract text fragments and images from them, process them separately, build a text knowledge base using the text fragments and the text vectors obtained from their processing, and build an image knowledge base using the images and the image vectors obtained from their processing.
[0072] The prompt word template building module is used to vectorize user questions through a vectorization model, and retrieve the most relevant text vectors and image vectors from the text knowledge base and image knowledge base, and combine them into prompt word templates.
[0073] The rich text answer generation module is used to input prompt word templates into a large language model and receive the streamed answers. The answers are then concatenated in real time, and the module checks for predefined image tags. If a predefined image tag exists, the module traces the original data corresponding to that tag and searches for the nearest image. If a nearest image exists, the image tag is replaced with the actual address of the nearest image, and the answer is re-output. Otherwise, the module retrieves the content block closest to the image tag, vectorizes it, searches the image knowledge base for the closest image, replaces the image tag with the actual address of the closest image, and re-outputs the answer. If no predefined image tag exists, the processed portion of the answer is immediately output.
[0074] This embodiment also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, app store, etc., which stores computer programs. When the program is executed by a processor, it implements the corresponding functions. When the computer-readable storage medium of this embodiment is executed by a processor, it implements the above-described adaptive multimodal rich text intelligent generation method.
[0075] This embodiment also provides a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described adaptive multimodal rich text intelligent generation method.
[0076] In summary, this invention provides an adaptive multimodal rich text intelligent generation method and system that can automatically generate and trace structured text and image content by dynamically integrating multimodal knowledge from various domains. It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.
[0077] The sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0078] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. An adaptive multimodal rich text intelligent generation method, characterized in that, include: Raw data is collected from relevant areas of user questions, and text fragments and images are extracted and processed separately. A text knowledge base is built using the text fragments and the text vectors obtained from their processing, and an image knowledge base is built using the images and the image vectors obtained from their processing. The user's question is vectorized by a vectorization model. The most relevant text vectors and image vectors and corresponding documents are retrieved from the text knowledge base and image knowledge base. All documents are numbered and combined into prompt word templates. Input the prompt word template into the large language model to receive the streaming answer; concatenate the answer in real time, and then check if there is a predefined image tag; if there is a predefined image tag, trace the original data corresponding to the image tag and find the nearest image. If the nearest image exists, replace the image tag with the actual address of the nearest image and re-output the answer; otherwise, obtain the content block with the closest image tag, perform vectorization processing, search for the closest image from the image knowledge base, replace the image tag with the actual address of the closest image, and re-output the answer; if there is no predefined image tag, immediately output the processed part of the answer.
2. The adaptive multimodal rich text intelligent generation method according to claim 1, characterized in that, The specific methods for processing text fragments include: removing HTML tags and advertisements from the text fragments using XPath or regular expressions, and extracting the plain text of the text fragments; The TextRank algorithm is used to perform semantic analysis on the extracted plain text, dividing the document into multiple semantic units with independent meaning; Semantic units are encoded using a text vectorization model to generate high-dimensional text vectors; the text vectorization model includes the Bge-large model. The text vectors and their corresponding original text fragments are used to construct a text knowledge base.
3. The adaptive multimodal rich text intelligent generation method according to claim 2, characterized in that, The TextRank algorithm is based on the formula... Calculate sentence weights in plain text; where, Sentence The weights; It is the damping coefficient, representing the probability of a random jump; It refers to all sentences. A collection of sentences; it is a set of sentences. The set of all sentences it points to; Sentence and sentences The similarity between them.
4. The adaptive multimodal rich text intelligent generation method according to claim 1, characterized in that, The specific methods for processing images include: extracting images from the original data using XPath or regular expressions; The extracted images are encoded using an image vectorization model and converted into numerical vectors; the image vectorization model includes the Bge-m3 model. An image knowledge base is constructed by combining image vectors with their corresponding original images.
5. The adaptive multimodal rich text intelligent generation method according to claim 1, characterized in that, Methods for retrieving the most relevant text vectors and image vectors to user questions from text and image knowledge bases include: comparing the cosine similarity of user question vectors among text vectors in the text knowledge base and among image vectors in the image knowledge base; the calculation formula includes: Where A represents the user question vector, and B represents the text vector in the text knowledge base or the image vector in the image knowledge base; Represents the cosine similarity between A and B; and Let A and B represent the values of vectors A and B in the i-th dimension, respectively. The text vector and image vector corresponding to the highest cosine similarity are selected as the text vector and image vector most relevant to the user's question.
6. The adaptive multimodal rich text intelligent generation method according to claim 1, characterized in that, The method for tracing the original data corresponding to the image tag specifically includes: extracting the document number from the image tag using regular expressions; locating the document to which the image tag belongs based on the number; and tracing the corresponding original data using the metadata of the document.
7. The adaptive multimodal rich text intelligent generation method according to claim 1, characterized in that, The method for finding the nearest image specifically includes: Based on the position of the image markers, semantic segmentation is performed on the answer to obtain the content block that is closest to the image markers; The content block is vectorized using a vectorization model; Retrieve the image vector with the highest cosine similarity to the content block vector from the image knowledge base, and take the corresponding image as the closest image.
8. An adaptive multimodal rich text intelligent generation system, characterized in that, It includes a domain knowledge base construction module, a prompt word template construction module, and a rich text answer generation module; among them, The domain knowledge base construction module is used to collect raw data from relevant domains and extract text fragments and images from them, process them separately, and build a text knowledge base using text fragments and the text vectors obtained from their processing, and build an image knowledge base using images and the image vectors obtained from their processing. The prompt word template building module is used to vectorize user questions through a vectorization model, retrieve the most relevant text vectors and image vectors from the text knowledge base and image knowledge base, and combine them into prompt word templates; The rich text answer generation module is used to input prompt word templates into a large language model and receive the streamed answers. The answers are then concatenated in real time, and the module checks for predefined image tags. If a predefined image tag exists, the module traces the original data corresponding to that tag and searches for the nearest image. If a nearest image exists, the image tag is replaced with the actual address of the nearest image, and the answer is re-output. Otherwise, the module retrieves the content block closest to the image tag, vectorizes it, searches the image knowledge base for the closest image, replaces the image tag with the actual address of the closest image, and re-outputs the answer. If no predefined image tag exists, the processed portion of the answer is immediately output.
9. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the adaptive multimodal rich text intelligent generation method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the steps of the adaptive multimodal rich text intelligent generation method according to any one of claims 1-7.
