Multi-modal image-text dynamic generation method and system based on retrieval enhancement

Abstract

The invention provides a multi-modal image-text dynamic generation method and system based on retrieval enhancement. The method comprises the following steps: encoding user query information through a multi-modal encoding model to obtain a query encoding vector; performing similarity matching on the query coding vector and a database coding vector, and retrieving related recall information; determining that the recall information comprises at least one image and at least one text fragment, and generating an initial text response based on the query information and the text fragment in the recall information; inputting at least one image in the initial text response and the recall information into a pre-trained position decision model, and outputting a target position for embedding the at least one image in the initial text response; and according to the target position, inserting an image mark in the initial text response, and generating and outputting an image-text mixed response result. According to the method, the dynamic response with clear logic and illustrative and graphical features can be generated, and the information transmission efficiency and the user interaction experience are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and multi-modal information processing technology, and in particular to a multi-modal image-text dynamic generation method and system based on retrieval enhancement. BACKGROUND

[0002] Currently, the generated replies of mainstream multi-modal large models are often answers to questions or images, lacking deep integration of multi-modal content. The generation capabilities of the latest text-to-image models are mostly limited to pure text generation, independent image generation, or image description text generation. However, in actual application scenarios, users often expect to directly embed related images in the replies to enhance the intuitiveness and interactive experience of information.

[0003] RAG (Retrieval-Augmented Generation) is an artificial intelligence framework that combines information retrieval technology with generative models. RAG technology enhances the generation capabilities of language models by retrieving external knowledge bases. However, existing RAG systems still have obvious multi-modal processing defects. On the one hand, traditional RAG architectures only support text-based input and output, and cannot effectively process unstructured data such as images and videos, resulting in a single dimension of knowledge retrieval. On the other hand, existing solutions have a semantic gap in cross-modal information fusion, and lack the ability to cooperatively express images and text, making it difficult to achieve deep association and dynamic presentation of knowledge elements. SUMMARY

[0004] The present application provides a multi-modal image-text dynamic generation method and system based on retrieval enhancement, which solves the defects of single retrieval method, rigid image presentation, and insufficient multi-modal collaboration in existing technologies by retrieving external knowledge bases to enhance the generation capabilities of language models, and achieves a more intuitive and intelligent interactive experience.

[0005] The present application provides a multi-modal image-text dynamic generation method based on retrieval enhancement, comprising the following steps: Obtain the query information of a user, encode the query information through a multi-modal encoding model to obtain a query encoding vector, and the query information includes at least one of text and image; Match the query encoding vector with a pre-stored database encoding vector for similarity, and retrieve relevant recall information; wherein the database encoding vector is obtained by parsing input image-text documents to obtain text blocks and images, and encoding the text blocks and images using the multi-modal encoding model; Confirm that the recall information contains at least one image and at least one text segment, and generate an initial text response based on the query information and the text segment in the recall information; inputting the initial text response and at least one image contained in the recall information into a pre-trained position decision model to obtain a target position for embedding the at least one image in the initial text response output by the position decision model; According to the target position, an image mark is inserted into the initial text response to generate and output a graphic-text hybrid response result.

[0006] According to the application, when the recall information contains multiple images, the method further comprises: inputting the initial text response and the multiple images into the position decision model; The position decision model outputs a corresponding target position in the initial text response for each of the multiple images.

[0007] According to the application, the inputting the initial text response and at least one image contained in the recall information into a pre-trained position decision model to obtain a target position for embedding the at least one image in the initial text response output by the position decision model specifically comprises: For multiple potential insertion points in the initial text response, the position decision model outputs a matching score of each potential insertion point with the image, and determines the target position based on the matching score of each potential insertion point with the image.

[0008] According to the application, the position decision model is trained by the following steps: Obtain a training data set containing multiple graphic-text samples; For each graphic-text sample, divide the text content therein into multiple text segments according to a preset rule; Use the multi-modal encoding model to encode the image and each text segment in the graphic-text sample respectively to obtain an image vector and a group of text segment vectors; Determine the text segment with the highest semantic similarity with the image vector by calculating the semantic similarity between the image vector and each text segment vector; Determine the position indicated by the text segment with the highest semantic similarity with the image vector as the reference annotation position for training in the graphic-text sample; Take the image and text content in the graphic-text sample as training input and take the reference annotation position as training label to perform supervised learning training on the position decision model.

[0009] The application provides a multi-modal image-text dynamic generation method based on retrieval enhancement.

[0010] The multi-modal image-text dynamic generation method based on retrieval enhancement further comprises constructing a database for storing the database encoding vectors, specifically comprising: receiving an image-text document; parsing the image-text document to complete layout analysis, text segmentation and image extraction to obtain text blocks and images; encoding the text blocks and images through the multi-modal encoding model respectively to obtain corresponding database encoding vectors and store them.

[0011] The multi-modal image-text dynamic generation method based on retrieval enhancement, wherein the position decision model is a small-scale language model with a parameter scale smaller than that of the multi-modal encoding model.

[0012] The application further provides a multi-modal image-text dynamic generation system based on retrieval enhancement, comprising the following modules: a query information encoding module configured to obtain query information of a user, encode the query information through a multi-modal encoding model to obtain a query encoding vector, wherein the query information comprises at least one of text and images; a recall information retrieval module configured to perform similarity matching between the query encoding vector and a pre-stored database encoding vector to retrieve relevant recall information, wherein the database encoding vector is obtained by parsing an input image-text document to obtain text blocks and images, and encoding the text blocks and images through the multi-modal encoding model; an initial text response generation module configured to confirm that the recall information comprises at least one image and at least one text segment, and generate an initial text response based on the query information and the text segment in the recall information; a target position output module configured to input the initial text response and the at least one image contained in the recall information into a pre-trained position decision model to obtain a target position output by the position decision model for embedding the at least one image in the initial text response; an image-text response generation module configured to insert an image mark in the initial text response according to the target position to generate and output an image-text hybrid response result.

[0013] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the multi-modal image-text dynamic generation method based on retrieval enhancement according to any one of the above when executing the computer program.

[0014] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the multi-modal image-text dynamic generation method based on retrieval enhancement according to any one of the above.

[0015] The application further provides a computer program product, including a computer program, and the computer program is executed by a processor to implement the multi-modal image-text dynamic generation method based on retrieval enhancement according to any one of the above.

[0016] The multi-modal image-text dynamic generation method and system based on retrieval enhancement provided by the application can deeply understand the semantic association between newly generated text content and recalled images through a position decision model, and intelligently embed images into the most appropriate position in the text, thereby realizing high unification of form and connotation; through intelligent image-text layout, the generated result is clear in logic, prominent in focus, and consistent with human reading habits, so that users can more efficiently and intuitively obtain and understand information without switching between text and pictures, thereby improving information transmission efficiency and user experience; the two-stage architecture separating content generation and layout decision is adopted, a large language model is responsible for content creation, and a position decision model is responsible for layout optimization, through the design of clear division of labor, the response speed and running efficiency of the system are considered while ensuring the generation quality, and the method has high practical value. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0018] Figure 1 is one of the flowcharts of the multi-modal image-text dynamic generation method based on retrieval enhancement provided by the application.

[0019] Figure 2 is the second flowchart of the multi-modal image-text dynamic generation method based on retrieval enhancement provided by the application.

[0020] Figure 3 is the flowchart of the multi-granularity database creation provided by the application.

[0021] Figure 4 is a flowchart of dynamic judgment output provided by the present application.

[0022] Figure 5 is a training flow of a position decision model provided by the present application.

[0023] Figure 6 is a structural schematic diagram of a multi-modal image-text dynamic generation system based on retrieval enhancement provided by the present application.

[0024] Figure 7 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0025] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0026] The present application will be described in detail below in combination with the drawings in the specification. The specific operation methods in the method embodiments can also be applied to the device embodiments or the system embodiments. In the description of the present application, unless otherwise specified, “at least one” includes one or more. “Multiple” refers to two or more. For example, at least one of A, B and C includes: A alone, B alone, A and B together, A and C together, B and C together, and A, B and C together. In the present application, “ / ” means or, for example, A / B can mean A or B; “and / or” in this document only describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone.

[0027] Current multimodal large models mainly improve in three directions: (1) Data-level fusion: represented by the FLAVA framework, cross-modal pre-training strategy is adopted to realize shallow feature association through the construction of text-image pair datasets, but there is a risk of information confusion between modalities. (2) Feature-level fusion: such as the unified embedding space scheme proposed by ImageBind, which maps six modalities to a 1280-dimensional shared vector space. However, the feature alignment accuracy is limited by the fixed projection matrix, and its adaptability is insufficient in professional fields such as medical imaging. (3) Decision-level fusion: typical such as the BLIP-2 framework, which connects the visual encoder and the language model through Q-former to realize efficient parameter transfer. However, it is limited by the serial processing architecture, and the response delay in multi-round dialogue is up to 2.3 seconds. However, the current mainstream technical solutions have significant limitations: the InternVL framework supports early fusion (directly splicing pixels and word vectors), late fusion (independent encoding and splicing), and intermediate fusion (cross-modal attention), but its modal interaction weight is fixed and cannot dynamically adapt to the dialogue scene; QwenVL realizes multi-image understanding through three-stage training (visual encoder fine-tuning, cross-modal alignment, and instruction fine-tuning), but the generation of visual concepts still relies on text description translation.

[0028] RAG technology enhances the generation ability of language models by retrieving external knowledge bases. However, existing RAG systems mainly support text retrieval and have limitations in multi-modal content integration. For example, RAG systems usually only support text retrieval, and even if multi-modal retrieval capabilities are introduced, they only retrieve pictures first and then convert them into text descriptions through semantic understanding. In RAG systems, the retrieval module is responsible for retrieving relevant information from the knowledge base, and the generation module generates text replies based on the retrieval results. For example, RAGFlow introduces a step of Semantic Chunking for unstructured data to ensure the quality of data entry.

[0029] Although existing multi-modal fusion technologies can handle multiple modal data, they have limitations in interactive dialogue applications, especially in terms of image-text alignment and dynamic fusion. While mainstream RAG technologies can retrieve external knowledge, they have deficiencies in multi-modal content retrieval and embedding.

[0030] The current mainstream multi-modal large model cannot directly realize multi-modal content reply. The reply is often an answer to the question and the image, and lacks deep integration of multi-modal content. The generation capability of the latest text-to-image model is mostly limited to pure text generation, independent image generation, and image description text generation. Among them, pure text generation is a text answer to an image. For example, a user uploads a scenic picture, and the AI can only generate a text description such as "this is a beautiful forest"; independent image generation is to generate an image according to the text. For example, generate a corresponding picture according to "a white puppy running on the grass"; and image description text generation is to modify the picture according to the picture and generate a text description. For example, a user uploads a blurred portrait photo, and the AI generates a clear portrait photo and describes "this is a clear portrait photo after repair".

[0031] However, in actual application scenarios, users often expect AI such as LLM or MLM to directly embed related pictures in the reply to enhance the intuitiveness of information and the interactive experience. However, the current retrieval augmented generation (RAG) technology still has obvious limitations. First, the RAG retrieval method is single: most RAG systems only support text retrieval. Even if multi-modal retrieval capability is introduced, it is only to retrieve pictures first, then convert them into text descriptions, and finally integrate them into the reply in the form of text. For example, a user asks "the architectural style of Notre-Dame de Paris", and the system only retrieves related pictures and converts them into "Notre-Dame de Paris is a Gothic architectural style" and other text descriptions. Second, the RAG picture presentation method is rigid: even if some systems support inserting pictures in the reply, they mostly use engineering methods to simply attach the pictures at the end of the reply or the end of the paragraph, lacking flexible layout and semantic association capabilities. For example, there is no clear association between the picture and the text, and the user needs to understand the relationship between the picture and the text by himself. Third, the RAG multi-modal coordination is insufficient: existing systems cannot achieve true alignment of pictures and text, that is, deep integration of pictures and text in terms of semantics and logic, resulting in limited user experience. For example, there is a deviation between the picture and the text description, and the user cannot intuitively understand the relationship between the picture and the text.

[0032] Therefore, the present application provides a multi-modal picture-text dynamic generation method and system based on retrieval enhancement to solve the above problems.

[0033] The present application will be described in detail below in conjunction with specific embodiments.

[0034] Embodiment one: a multi-modal picture-text dynamic generation method based on retrieval enhancement.

[0035] The present embodiment provides a multi-modal picture-text dynamic generation method based on retrieval enhancement. Referring to Figure 1 The flow of the method includes the following steps: Step 100, obtaining query information of a user, encoding the query information through a multi-modal encoding model to obtain a query encoding vector, the query information including at least one of text and an image; In the embodiment, the user can input the query information in various ways, such as pure text (‘When was the Eiffel Tower built?’), pure image (uploading a picture of the Eiffel Tower), or a mixed form of text and image.

[0036] In some possible embodiments of the application, the multi-modal encoding model is used to map data of different modalities to the same semantic vector space and is trained through contrast learning on text-image pair data.

[0037] A unified multi-modal encoding model is used to encode the received query information. The model is trained through contrast learning and the like and can map data (text, image) of different modalities to the same high-dimensional semantic vector space. Regardless of the modality of the input, it will finally be converted into a standardized query encoding vector.

[0038] Step 200, performing similarity matching between the query encoding vector and a pre-stored database encoding vector to retrieve relevant recall information; wherein the database encoding vector is obtained by parsing an input text-image document to obtain text blocks and images, and encoding the text blocks and images using the multi-modal encoding model; In some possible embodiments of the application, a database is used to store the database encoding vector, and the construction process can be referred to Figure 2 , and specifically includes: Step one, receiving and parsing a text-image document: Receiving a text-image document input by a user, the text-image document including pure text, pure image, and mixed data including both image and text.

[0039] Further, the text-image document is parsed to complete layout analysis, text segmentation, and image extraction to obtain text blocks and images.

[0040] For example, a document in PDF, Word, or the like format uploaded by a user is deeply parsed through layout analysis, text segmentation, and image extraction, so that information of two modalities is accurately extracted to obtain structured information units, i.e., independent text blocks and images.

[0041] Step two, uniform encoding: Further, the text blocks and images are respectively encoded through the multi-modal encoding model to obtain corresponding database encoding vectors.

[0042] Specifically, before encoding, a dynamic judgment step is further included, that is, the parsed result is dynamically judged in real time, if the parsed result contains an image, the image is base64 encoded and sent to a multi-modal model trained by professionals to extract semantic information thereof; meanwhile, the original image and the text block are sent to a multi-modal encoding model to obtain a database encoding vector.

[0043] Specifically, the same multi-modal encoding model used to build the database is used to independently encode each text block and each image into a high-dimensional vector (for example, 768-dimensional or 1024-dimensional). Since all vectors are located in the same semantic space, the distance between the vector of a cat picture and the vector of a text describing "a cute white kitten" will be very close in the space.

[0044] Step three, store in the database: The generated vector and its corresponding original content (for example, the storage path or ID of the original text or image) are stored in a vector database, that is, the feature vector library in the figure (such as Milvus, Pinecone, etc.) for retrieval.

[0045] When querying, the system performs similarity calculation (for example, cosine similarity) on the obtained query encoding vector and all pre-stored vectors in the database, recalls the top-K results with the highest similarity, and these results are the recalled information. Then the recalled information is judged to generate an initial answer and a final answer through subsequent steps.

[0046] Step 300, confirming that the recalled information contains at least one image and at least one text segment, generating an initial text response based on the query information and the text segment in the recalled information; This step is a pre-judgment of the recalled information. After confirming that the recalled information contains at least one image and at least one text segment, the subsequent image-text fusion generation process is started.

[0047] For example, Figure 3 A multi-granularity database creation process provided by an embodiment of the application is shown, which includes Figure 3The extraction process of image semantic information can also be intuitively understood. The images in the file uploaded by the user are extracted separately and stored in a minio system (a high-performance object storage system). After base64 encoding, the images are output to a multi-modal large model. The model extracts information from different points of interest (such as the shape, color of the object, the atmosphere of the background, the order of the elements, etc.), generates multiple description information, and performs cross-modal similarity calculation with the image. Finally, the description with the highest similarity is selected as the semantic information of the image. Since it is often difficult to accurately capture image features and convert them into semantic information by directly describing the image, the present application introduces a multi-modal model with a thinking chain to efficiently capture image features.

[0048] Since the input can be a file with an image or a pure text file, there may be no image in the recall information, so the output needs to be dynamically judged whether to display the image. Figure 4 is a flowchart of dynamic output form judgment. When the user inputs a question, the question is converted into a vector of the same dimension as the database and similarity recall is performed. The text block with the highest similarity is selected as the candidate auxiliary information. Then, it is judged whether there is an image. If there is no image, only the text is numbered using special symbols. After output, the special symbols are converted into text marks for display through post-processing. If there is an image, the text information corresponding to the image in the database is converted into a list based on this step, and a large language model that has been retrained is used to insert image marks into the initial response. After post-processing, the image marks are converted back into images.

[0049] That is, the original query information of the user and all text segments in the recall information can be used as context and input into a large language model (LLM). The LLM generates an initial text response that is logically coherent and rich in content. For example, for the query "Eiffel Tower", the following response can be generated: "The Eiffel Tower is a famous landmark located in Paris, France, built to commemorate the 100th anniversary of the French Revolution. It was designed by the famous architect Gustave Eiffel and is a masterpiece in the history of world architecture.

[0050] Step 400: inputting the initial text response and at least one image contained in the recall information into a pre-trained position decision model to obtain a target position output by the position decision model for embedding the at least one image in the initial text response; This is the core innovative step of the present application. The initial text response generated in the previous step and the image (such as a picture of the Eiffel Tower) in the recall information are input into a pre-trained position decision model.

[0051] In some possible embodiments of the present application, the position decision model is a small-scale language model with a parameter scale smaller than the multi-modal encoding model.

[0052] The position decision model is a model specially designed for the task of image-text layout. In order to ensure the efficiency and low latency of decision-making, the model can be a small-scale language model with a small parameter scale (e.g., a parameter scale of several billion levels), which is smaller than the large language model used to generate the initial text response.

[0053] For the plurality of potential insertion points in the initial text response, the position decision model outputs a matching score of each potential insertion point with the image, and determines the target position based on the matching score of each potential insertion point with the image.

[0054] That is, the working mode of the position decision model can be: first, divide the initial text response into a plurality of potential insertion points according to a preset rule (such as by sentence or punctuation). Then, the model calculates a matching score for each potential insertion point with the input image, which represents the semantic fit degree of inserting the image at this point. Finally, the model determines a best target position based on the matching scores of all potential insertion points. For example, the model can select the position with the highest score, or consider factors such as score and context length to determine the final position.

[0055] Step 500: According to the target position, insert an image marker in the initial text response to generate and output a hybrid image-text response result.

[0056] After determining the target position, the system inserts a special text marker that can be parsed by the front end in the initial text response at the position, for example, [IMAGE: image_id_001].

[0057] At this time, the generated text stream becomes: "The Eiffel Tower is a famous landmark in Paris, France, built to commemorate the 100th anniversary of the French Revolution. [IMAGE: image_id_001] It was designed by the famous architect Gustave Eiffel and is a masterpiece of world architectural history." This marked text stream is sent to the front-end application (such as a web page or an App). After receiving it, the front-end application parses the marker and pulls the corresponding real picture from the image storage service (such as MinIO) according to image_id_001, and renders and displays it at the position of the marker, finally presenting a perfect image-text content to the user.

[0058] In addition, in some possible embodiments of the present application, when the recalled information contains a plurality of images, the method further comprises: inputting the initial text response and the plurality of images into the position decision model together; outputting, by the position decision model, for each of the plurality of images, a target position corresponding to the initial text response.

[0059] Specifically, when the recall information contains multiple images, the position decision model can simultaneously process the initial text response and the multiple images, and independently determine the best target position of each image in the text, thereby realizing a more rich multi-image layout effect.

[0060] The embodiment of the application integrates the feature extraction method of text and image through a multi-modal encoder, supports multiple input methods such as pure text, pure image and mixed text and image, the multi-granularity knowledge base construction method is more flexible, and the recall information is more accurate. The multi-modal large model retrained realizes the conversion of image features to text features, and reduces the loss of semantic information in the conversion process. Through dynamic verification and double-channel layout, the image and text information are processed in parallel, the image-text output with visual focus is realized, and the user can intuitively obtain various information.

[0061] Embodiment two: training method of position decision model.

[0062] This embodiment details the training method of the position decision model mentioned in embodiment one. Referring to Figure 5 , the training process of the position decision model is as follows: Step 501: Obtain and prepare a training data set.

[0063] A training data set containing a large number of high-quality image-text samples is obtained. These samples can be pictures with captions, articles with images, etc.

[0064] Step 502: Automatically generate training labels (ground truth annotation positions) for each sample.

[0065] This is the key to training. For each image-text sample, the system automatically generates a ground truth annotation position (Ground Truth) of "where the picture should be inserted", and the specific method is as follows: a) For each image-text sample, the text content in it is divided into multiple text segments according to a predetermined rule; Specifically, the text content in the sample is divided into multiple text segments according to a predetermined rule (such as by paragraph or sentence).

[0066] b) Using the multi-modal encoding model, the image and each text segment in the image-text sample are encoded respectively to obtain an image vector and a group of text segment vectors; Specifically, the image in the sample is encoded into an image vector and each text segment is encoded into a corresponding text segment vector by using the same multi-modal encoding model used in the RAG retrieval stage.

[0067] c) determining the text segment with the highest semantic similarity to the image vector by calculating the semantic similarity between the image vector and each text segment vector; Specifically, the text segment with the highest semantic similarity to the image vector is determined by calculating the semantic similarity between the image vector and all text segment vectors. d) determining the position indicated by the text segment with the highest semantic similarity to the image vector as the reference annotation position for training in the image-text sample; Specifically, the position indicated by the most relevant text segment (e.g., the position after the segment) is determined as the reference annotation position for training in the sample. This process ensures that the training label is generated based on deep semantic association rather than simple format rules.

[0068] Step 503: Supervised learning training.

[0069] The position decision model is trained by supervised learning using the image and text content in the image-text sample as training input and the reference annotation position as training label.

[0070] Specifically, the image and complete text content in the dataset are used as the training input (Input) of the model, and the reference annotation position generated in step 502 is used as the training label (Label). Through the standard supervised learning paradigm, an initial language model is fine-tuned. The goal of model learning is to make the predicted insertion position infinitely close to the reference annotation position given any text and image. Through the above training, an efficient, accurate, and specialized position decision model for image-text intelligent layout is obtained.

[0071] Embodiment Three: A multi-modal image-text dynamic generation system based on retrieval enhancement.

[0072] The embodiment provides a multi-modal image-text dynamic generation system based on retrieval enhancement. Referring to Figure 6 The system comprises: A query information encoding module 10 is configured to obtain query information of a user, encode the query information by a multi-modal encoding model to obtain a query encoding vector, and the query information comprises at least one of text and image. The recall information retrieval module 20 is used for similarity matching of the query coding vector and a pre-stored database coding vector to retrieve relevant recall information, wherein the database coding vector is obtained by parsing an input graphic-text document to obtain a text block and an image, and the text block and the image are coded by using the multi-modal coding model to obtain; The initial text response generation module 30 is used for confirming that the recall information contains at least one image and at least one text segment, and generating an initial text response based on the query information and the text segment in the recall information; The target position output module 40 is used for inputting the initial text response and the at least one image contained in the recall information into a pre-trained position decision model to obtain a target position output by the position decision model for embedding the at least one image in the initial text response; The graphic-text response generation module 50 is used for inserting an image mark in the initial text response according to the target position to generate and output a graphic-text hybrid response result.

[0073] The modules work cooperatively to realize all steps in the method embodiment.

[0074] The multi-modal graphic-text dynamic generation method and system based on retrieval enhancement provided by the application realize a major breakthrough in the following dimensions: Firstly, the cross-modal semantic understanding model breaks the single-modal limitation of the traditional RAG, so that the system can simultaneously process image queries and generate graphic-text hybrid responses; Secondly, the dynamic content arrangement algorithm innovatively realizes intelligent combination of response elements, and can automatically adjust the graphic-text proportion and presentation order according to a dialogue scene; Thirdly, the layout optimizer based on the visual cognitive model ensures the readability and information transmission efficiency of the generated content.

[0075] Compared with the prior art, the application also has significant advantages in the following application fields: In an education scene, knowledge explanation with graphics and texts can be dynamically generated; In an e-commerce application, intelligent matching of product images and parameter descriptions can be realized.

[0076] With the evolution of multi-modal large model technology, the framework can seamlessly integrate emerging visual language models (VLM) and diffusion generation models, laying a foundation for building a truly intelligent multi-modal dialogue system, and having broad application prospects in the fields of industrial knowledge management, intelligent education assistance and cross-modal information retrieval.

[0077] Figure 7 An example of an entity structure schematic diagram of an electronic device is shown in Figure 7As shown, the electronic device can include a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 complete mutual communication through the communications bus 740. The processor 710 can invoke the logic instructions in the memory 730 to execute the multi-modal image-text dynamic generation method based on retrieval enhancement, which includes: obtaining query information of a user, encoding the query information through a multi-modal encoding model to obtain a query encoding vector, the query information including at least one of text and image; performing similarity matching on the query encoding vector and a pre-stored database encoding vector to retrieve relevant recall information; wherein the database encoding vector is obtained by encoding a text block and an image obtained by parsing an input image-text document using the multi-modal encoding model; confirming that the recall information includes at least one image and at least one text segment, generating an initial text response based on the query information and the text segment in the recall information; inputting the initial text response and the at least one image included in the recall information into a pre-trained position decision model to obtain a target position output by the position decision model for embedding the at least one image in the initial text response; inserting an image mark in the initial text response according to the target position to generate and output an image-text mixed response result.

[0078] In addition, the logic instructions in the memory 730 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0079] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program being stored on a non-transitory computer-readable storage medium and executable by a processor to enable a computer to perform the multi-modal image-text dynamic generation method based on retrieval enhancement provided by the above method, the method comprising: obtaining query information of a user, encoding the query information by a multi-modal encoding model to obtain a query encoding vector, the query information comprising at least one of text and image; performing similarity matching between the query encoding vector and a pre-stored database encoding vector to retrieve relevant recall information; wherein the database encoding vector is obtained by parsing an input image-text document to obtain a text block and an image, and encoding the text block and the image by the multi-modal encoding model; confirming that the recall information comprises at least one image and at least one text segment, generating an initial text response based on the query information and the text segment in the recall information; inputting the initial text response and the at least one image contained in the recall information into a pre-trained position decision model to obtain a target position output by the position decision model for embedding the at least one image in the initial text response; inserting an image mark in the initial text response according to the target position to generate and output an image-text hybrid response result.

[0080] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the multi-modal image-text dynamic generation method based on retrieval enhancement provided by the above method, the method comprising: obtaining query information of a user, encoding the query information by a multi-modal encoding model to obtain a query encoding vector, the query information comprising at least one of text and image; performing similarity matching between the query encoding vector and a pre-stored database encoding vector to retrieve relevant recall information; wherein the database encoding vector is obtained by parsing an input image-text document to obtain a text block and an image, and encoding the text block and the image by the multi-modal encoding model; confirming that the recall information comprises at least one image and at least one text segment, generating an initial text response based on the query information and the text segment in the recall information; inputting the initial text response and the at least one image contained in the recall information into a pre-trained position decision model to obtain a target position output by the position decision model for embedding the at least one image in the initial text response; inserting an image mark in the initial text response according to the target position to generate and output an image-text hybrid response result.

[0081] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0082] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0083] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for dynamically generating a multi-modal text-image based on retrieval enhancement, characterized in that, The method comprises the following steps: obtain query information of a user, encode the query information through a multi-modal encoding model to obtain a query encoding vector, the query information including at least one of text and image; perform similarity matching between the query encoding vector and a pre-stored database encoding vector to retrieve relevant recall information; wherein the database encoding vector is obtained by parsing input image-text documents to obtain text blocks and images, and the text blocks and images are encoded by using the multi-modal encoding model to obtain database encoding vectors; confirm that the recall information contains at least one image and at least one text segment, and generate an initial text response based on the query information and the text segment in the recall information; input the initial text response and at least one image contained in the recall information into a pre-trained position decision model to obtain a target position output by the position decision model for embedding the at least one image in the initial text response; insert an image mark in the initial text response according to the target position, and generate and output a mixed image-text response result. 2.The method of claim 1, wherein, When the recall information contains multiple images, the method further comprises: input the initial text response and the multiple images into the position decision model; output, by the position decision model, a corresponding target position in the initial text response for each of the multiple images. 3.The method of claim 1, wherein, The step of inputting the initial text response and at least one image contained in the recall information into a pre-trained position decision model to obtain a target position output by the position decision model for embedding the at least one image in the initial text response specifically comprises: for multiple potential insertion points in the initial text response, the position decision model outputs a matching score of each potential insertion point with the image, and determines the target position based on the matching score of each potential insertion point with the image.

4. The multi-modal dynamic generation method based on retrieval enhancement according to any one of claims 1-3, characterized in that, The position decision model is trained by the following steps: obtain a training data set containing multiple image-text samples; for each image-text sample, divide the text content therein into multiple text segments according to a preset rule; encode the image and each text segment in the image-text sample by using the multi-modal encoding model to obtain an image vector and a group of text segment vectors; determine a text segment with the highest semantic similarity with the image vector by calculating the semantic similarity between the image vector and each text segment vector; determine a position indicated by the text segment with the highest semantic similarity with the image vector as a reference annotation position for training in the image-text sample; perform supervised learning training on the position decision model by taking the image and the text content in the image-text sample as training input and taking the reference annotation position as a training label. 5.The method of claim 1, wherein, The multi-modal encoding model is used to map data of different modalities to the same semantic vector space, and is trained by using contrastive learning on image-text data. 6.The method of claim 1, wherein, Further comprising constructing a database for storing the database encoding vector, specifically comprising: receiving image-text documents; The document is parsed to complete layout analysis, text segmentation and image extraction, and text blocks and images are obtained. The text blocks and images are respectively encoded by the multi-modal encoding model to obtain corresponding database encoding vectors and store them. 7.The method of claim 1, wherein, The position decision model is a small-scale language model with a parameter scale smaller than the multi-modal encoding model.

8. A multi-modal dynamic generation system based on search enhancement, characterized in that, It comprises: The query information encoding module is configured to obtain query information of a user, encode the query information by a multi-modal encoding model to obtain a query encoding vector, and the query information comprises at least one of text and image. The recall information retrieval module is configured to perform similarity matching between the query encoding vector and a pre-stored database encoding vector to retrieve relevant recall information, wherein the database encoding vector is obtained by parsing input document to obtain text blocks and images, and the text blocks and images are encoded by the multi-modal encoding model to obtain the database encoding vector. The initial text response generation module is configured to confirm that the recall information contains at least one image and at least one text segment, and generate an initial text response based on the query information and the text segment in the recall information. The target position output module is configured to input the initial text response and at least one image contained in the recall information into a pre-trained position decision model to obtain a target position output by the position decision model for embedding the at least one image in the initial text response. The document-text response generation module is configured to insert an image mark in the initial text response according to the target position to generate and output a document-text hybrid response result.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the multi-modal document-text dynamic generation method based on retrieval enhancement according to any one of claims 1 to 7. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the multi-modal document-text dynamic generation method based on retrieval enhancement according to any one of claims 1 to 7.

Images