Multimodal retrieval methods, devices, equipment, media, and program products

Abstract

This application discloses a multimodal retrieval method, apparatus, device, medium, and program product, relating to the field of artificial intelligence technology. The multimodal retrieval method includes: obtaining a prompt to be responded to, the prompt including: prompt text and a target image; performing data modality rewriting on the prompt to be responded to obtain target text, the target text describing the content represented by the prompt text and the target image; determining a target database from multiple candidate databases based on the prompt text; and performing a search from the target database based on the target text to obtain target retrieval results. This method is used to implement multimodal retrieval.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a multimodal retrieval method, apparatus, device, medium, and program product. Background Technology

[0002] In the field of emergency safety, such as gas, petrochemical, and transportation, it is necessary to retrieve historical event fragments or related knowledge from videos, images, and text documents to assist in emergency decision-making.

[0003] Existing retrieval methods mainly include video structured retrieval based on computer vision (CV) models and text retrieval based on retrieval-augmented generation (RAG) systems. However, existing CV models and RAG systems cannot effectively integrate multimodal data such as text, images, and videos, making it impossible to achieve complex retrieval scenarios, such as "jointly querying emergency cases through images and text".

[0004] Therefore, how to achieve multimodal retrieval is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a multimodal retrieval method, apparatus, device, medium, and program product for implementing multimodal retrieval.

[0006] In a first aspect, embodiments of this application provide a multimodal retrieval method, including:

[0007] Obtain the pending response prompt, which includes: prompt text and target image;

[0008] The prompt to be responded to is rewritten in data modality to obtain target text, which is used to describe the content represented by the prompt text and the target image;

[0009] Based on the prompt text, a target database is determined from multiple candidate databases; the multiple candidate databases include: a multimodal database of the target domain and a knowledge graph database of the target domain, wherein the knowledge graph database is constructed based on the attribute information of the layout elements of rich text documents in the target domain, and the rich text documents;

[0010] Based on the target text, a search is performed on the target database to obtain the target search results.

[0011] In one possible implementation, the candidate database includes a multimodal database of the target domain, and before determining the target database from the multiple candidate databases based on the prompt text, the method further includes:

[0012] Obtain a first sample cue from the target domain, and a first sample image;

[0013] The first sample prompt and the first sample image are input into a multimodal feature generation model to obtain a first sample description text and semantic features corresponding to the first sample image; the first sample description text is used to describe the first sample prompt and the first sample image.

[0014] Based on the first sample description text and the semantic features, the multimodal database is constructed.

[0015] In one possible implementation, the candidate database further includes: a knowledge graph database of the target domain; and before determining the target database from multiple candidate databases based on the prompt text, the method further includes:

[0016] Obtain the domain knowledge document for the target domain;

[0017] In response to the fact that the domain knowledge document is the rich text document, the rich text document is converted into an image format to obtain the image corresponding to the rich text document;

[0018] Using a layout element extraction model, layout elements are extracted from the image corresponding to the rich text document to determine the region block image of the layout elements of the image corresponding to the rich text document, as well as the attribute information of the layout elements.

[0019] Based on the region-blocked image and the multimodal feature generation model, the multimodal feature vectors of each page element in the rich text document are obtained;

[0020] The knowledge graph database is constructed based on the multimodal feature vectors, the rich text document, and the attribute information of the layout elements.

[0021] In one possible implementation, obtaining the multimodal feature vector of the rich text document based on the region-blocked image and the multimodal feature generation model includes:

[0022] The region segmented image and the preset prompt words are input into the multimodal feature generation model to obtain the descriptive text of the region segmented image;

[0023] A text document is obtained based on the descriptive text of the region-blocked image;

[0024] The text document is sliced ​​to obtain multiple document slices;

[0025] The multiple document slices are input into the multimodal feature generation model to obtain the multimodal feature vectors of each document slice of the rich text document.

[0026] In one possible implementation, before inputting the first sample cue and the first sample image into a multimodal feature generation model to obtain the first sample description text and the semantic features corresponding to the first sample image, the method further includes:

[0027] Obtain a second sample cue from the target domain, and a second sample image;

[0028] The preset model is trained using the second sample prompt and the second sample image to obtain the multimodal feature generation model;

[0029] The preset model includes: a hidden layer, a text output module, and a multimodal feature vector output module; the hidden layer is used to extract semantic features based on the second sample prompt and the second sample image to obtain semantic features representing the second sample prompt and the second sample image; the text output module is used to output predicted text describing the second sample prompt and the second sample image based on the semantic features; the multimodal feature vector output module is used to output predicted multimodal features representing the second sample prompt and the second sample image based on the semantic features; the predicted text and the predicted multimodal features are used to calculate the loss function during the training process; the process loss function is used to adjust the model parameters of the preset model.

[0030] In one possible implementation, training the preset model using the second sample cue and the second sample image to obtain the multimodal feature generation model includes:

[0031] The trainable parameters of the multimodal feature vector output module are frozen, and the part of the preset model other than the multimodal feature vector output module is trained by the second sample prompt and the second sample image to obtain the first initial multimodal feature generation model.

[0032] The trainable parameters of the text output module are frozen, and the second sample prompt and the second sample image are used to train the part of the first initial multimodal feature generation model other than the text output module to obtain the second initial multimodal feature generation model.

[0033] Randomly freeze the trainable parameters of the multimodal feature vector output module or the trainable parameters of the text output module, and train the unfrozen modules in the second initial multimodal feature generation model using the second sample prompt, the second sample image, and the optical character recognition OCR dataset to obtain the third initial multimodal feature generation model.

[0034] The third initial multimodal feature generation model is trained using the second sample prompt, the second sample image, and the OCR dataset to obtain the multimodal feature generation model.

[0035] Secondly, embodiments of this application provide a multimodal retrieval device, comprising:

[0036] The acquisition module is used to acquire the prompt to be responded to, the prompt to be responded to includes: prompt text, and target image;

[0037] The rewriting module is used to rewrite the data modality of the prompt to be responded to, so as to obtain the target text, which is used to describe the content represented by the prompt text and the target image;

[0038] A determination module is used to determine a target database from multiple candidate databases based on the prompt text; the multiple candidate databases include: a multimodal database of the target domain and a knowledge graph database of the target domain, wherein the knowledge graph database is constructed based on the attribute information of the layout elements of a rich text document of the target domain, and the rich text document;

[0039] The retrieval module is used to retrieve the target results from the target database based on the target text.

[0040] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0041] The memory stores computer-executed instructions;

[0042] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method described in any of the first aspects above.

[0043] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any of the first aspects above.

[0044] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0045] This application provides a multimodal retrieval method, apparatus, device, medium, and program product. By rewriting data modality, the semantics of prompt text and target image are unified into target text, which can solve the heterogeneity problem of multimodal input and eliminate the need to directly process complex image data during the retrieval process. This application's embodiment matches the target database with the prompt text, avoiding full scanning and improving retrieval efficiency. Then, cross-modal retrieval (such as text-to-image search, image-to-text search, or hybrid retrieval) is achieved through the target text, covering complex query scenarios and realizing multimodal retrieval. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating a multimodal retrieval method provided in an embodiment of this application;

[0049] Figure 2 A flowchart illustrating a method for constructing a multimodal database, provided in an embodiment of this application;

[0050] Figure 3 A flowchart illustrating a method for obtaining image-text pairs provided in an embodiment of this application;

[0051] Figure 4 A schematic diagram of the model architecture of a preset model provided in an embodiment of this application;

[0052] Figure 5 A flowchart illustrating a method for training a preset model provided in an embodiment of this application;

[0053] Figure 6 A flowchart illustrating a specific method for training a preset model, as provided in an embodiment of this application;

[0054] Figure 7 A flowchart illustrating a method for constructing a knowledge graph database, provided in an embodiment of this application;

[0055] Figure 8 A schematic diagram illustrating a specific method for constructing a knowledge graph database, as provided in this application embodiment;

[0056] Figure 9 A flowchart illustrating a method for obtaining multimodal feature vectors of rich text documents, provided in an embodiment of this application;

[0057] Figure 10 A schematic diagram illustrating the construction of a candidate database as provided in an embodiment of this application;

[0058] Figure 11 This application provides a schematic diagram of the structure of a multimodal retrieval device.

[0059] Figure 12 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0060] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0061] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0062] In emergency safety fields, such as gas, petrochemical, and transportation, it is necessary to retrieve historical event fragments or related knowledge from videos, images, and text documents to assist in emergency decision-making. For example, by analyzing abnormal behavior or safety hazards in surveillance videos or images, historical event fragments or related knowledge can be quickly retrieved to aid emergency decision-making. Abnormal behavior or safety hazards can be, for example, any one or more of the following: personnel misconduct, equipment malfunction, etc.

[0063] Assisting emergency decision-making by retrieving text documents can be done by searching through a vast amount of emergency safety documents, such as technical specifications, accident reports, etc., to extract charts, tables, and text information. It can support complex text document searches, such as searching for "the handling process of a gas pipeline leak accident".

[0064] Existing retrieval methods mainly include video structured retrieval based on Computer Vision (CV) models and text retrieval based on Retrieval-Augmented Generation (RAG) systems. CV-based video structured retrieval uses methods such as object detection and attribute recognition to structurally process videos or images, extracting attribute tags, such as any one or more of "people," "vehicles," or "flames," and then performing retrieval based on the query. For example, when a user inputs "red car," this method can complete the retrieval by matching candidate data using attribute tags. However, this method cannot support complex queries; it only supports simple queries based on predefined attributes and cannot handle user-provided text descriptions or multimodal inputs.

[0065] Text retrieval based on the RAG system can construct a text knowledge base and use Best Matching 25 (BM25) or vector retrieval techniques to match user queries. For example, when a user enters "search for emergency measures for gas leaks," this method can retrieve relevant paragraphs from the text knowledge base. However, this method suffers from weak cross-modal retrieval capabilities; it only supports text-to-text retrieval and cannot handle multimodal data such as images and videos, nor can it achieve mixed "text + image" retrieval.

[0066] Therefore, this application embodiment unifies the semantics of the prompt text and the target image into target text through data modality rewriting, which can solve the heterogeneity problem of multimodal input and eliminate the need for direct processing of complex image data during the retrieval process. This application embodiment matches the target database based on the prompt text, avoiding full scanning and improving retrieval efficiency. Ultimately, cross-modal retrieval (such as text-to-image search, image-to-text search, or hybrid retrieval) is achieved through the target text, covering complex query scenarios and realizing multimodal retrieval.

[0067] It should be noted that the executing entity of this application can be any electronic device with processing capabilities, such as a user terminal or a server, for example, a computer.

[0068] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0069] Figure 1 This is a flowchart illustrating a multimodal retrieval method provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0070] S101. Obtain the prompt to be responded to, which includes: prompt text and target image.

[0071] Optionally, the prompt to be responded to can be a question to be retrieved, the prompt text can be the information content expressed in text form in the prompt to be responded to, and the target image can be a visual element part of the prompt to be responded to, such as any one or more of the following: a picture, a video, a chart, etc.

[0072] For example, the prompt text in the pending response prompt could be "Retrieve safety regulations related to gas production", and the target image in the pending response prompt could be "Retrieve whether the current monitoring system has captured any abnormal personnel behavior".

[0073] Optionally, the electronic device can respond to the user's actions on the input interface and thus obtain the prompts to be responded to. For example, the electronic device can obtain the aforementioned prompts to be responded to through an Application Programming Interface (API) or a Graphical User Interface (GUI).

[0074] S102. Perform data modal rewriting on the prompt to be responded to to obtain the target text, which is used to describe the content represented by the prompt text and the target image.

[0075] Optionally, data modality rewriting can rewrite the pending response prompts in different modalities into a form suitable for processing by the backend database or information retrieval system. These different modalities can include any one or more, such as text modalities and image modalities.

[0076] For example, taking a prompt to be responded to, which includes a text modality prompt text and a target image, the prompt text could be, for example, "Analyze whether the behavior of the personnel in the given image complies with gas safety regulations," and the target image could be, for example, at least one frame of an image showing the personnel's actions. The electronic device can input this prompt to be responded to into a data modality rewriting model to perform data modality rewriting and obtain the target text. For example, the target text could include multiple queries, such as: query1: gas; query2: safety regulations; query3: descriptive text describing the behavior of the personnel in the given image (e.g., two people lighting a fire near a gas appliance). These multiple queries constitute the target text.

[0077] Optionally, the electronic device can input the prompt to be responded to into a multimodal data rewriting model to obtain the target text. This multimodal data rewriting model can, for example, refer to any existing large model that has the function of rewriting data of different modalities into text.

[0078] S103. Based on the prompt text, determine the target database from multiple candidate databases; the multiple candidate databases include: a multimodal database of the target domain and a knowledge graph database of the target domain, wherein the knowledge graph database is constructed based on the attribute information of the layout elements of the rich text document of the target domain, and the rich text document.

[0079] Optionally, the candidate database can be a collection of multiple databases that store different types or ranges of data. The candidate database may include multimodal databases and knowledge graph databases of the target domain. The target database can be the one that best matches the requirements of the prompt text.

[0080] Optionally, a rich text document can be a document type that, in addition to text content, can also contain rich formatting and elements such as font styles, images, tables, charts, audio and video, and hyperlinks. Layout elements can include any one or more of the following: figures, tables, chapter titles, text paragraphs, chart titles, and chart annotations. The attribute information of layout elements can be information describing the characteristics and properties of the layout elements; for example, the attribute information of layout elements can include the layout element number.

[0081] Optionally, the electronic device can perform intent recognition on the prompt text and determine the required target database based on the intent recognition. For example, the electronic device can input the prompt text into an intent recognition model (such as any existing large language model) to perform intent recognition and obtain an intent recognition result. Then, based on the intent recognition result and the mapping relationship between the user's intent and the database, the electronic device can determine the target database from multiple candidate databases.

[0082] S104. Based on the target text, retrieve the target database to obtain the target search results.

[0083] Optionally, the target retrieval results may include data of at least one modality. For example, the target retrieval results may include data of at least one modality such as text, images, video, and audio.

[0084] Optionally, the target retrieval result can be a set of relevant data obtained from the target database after retrieving the target text. Optionally, the electronic device can select an appropriate retrieval algorithm and interface to perform the retrieval based on the type and structure of the target database. For example, if the target database is a relational database, a query can be constructed using Structured Query Language (SQL) statements; if the target database is a non-relational database, a corresponding query language or application programming interface can be used for retrieval.

[0085] Optionally, the retrieval of electronic devices may include any one or more of the following: keyword retrieval, basic semantic retrieval, cross-modal retrieval, and hybrid retrieval. Keyword retrieval may employ the BM25 algorithm to match keywords in the target text with text information in the target database to obtain the target retrieval results. Basic semantic retrieval may match the multi-feature vector corresponding to the query in the target text with the multi-feature vector corresponding to all text in the target database to obtain the target retrieval results.

[0086] Cross-modal retrieval involves matching the multi-feature vector corresponding to the query in the target text with the multi-feature vectors of all images or videos in a multi-modal database to obtain the target retrieval results. Hybrid retrieval involves matching the keywords in the target text with the multi-feature vector corresponding to the query in the target text, as well as the multi-feature vectors of all texts and all images or videos, and then fusing them according to certain weights to obtain the target retrieval results.

[0087] Alternatively, the above methods can all calculate the similarity between multiple feature vectors to obtain matching results and thus complete the matching.

[0088] For example, still using the aforementioned prompt text "Analyze whether the behavior of people in the given image complies with the safety regulations for gas production" and the given target image as an example, the electronic device can rewrite the prompt into a data modality to obtain target text including "query1, query2, and query3" as in the previous example. Taking the electronic device determining the candidate database as a multimodal database of gas safety knowledge (i.e., the multimodal database of the aforementioned target domain) based on this prompt text as an example, the electronic device can perform a search on this multimodal database of gas safety knowledge based on the target text to obtain target search results. These target search results may include, for example, the text "The behavior of people in the image does not comply with the safety regulations for gas production; the safety regulations stipulate that ignition near gas appliances is not allowed, for example, refer to Article 3," and an example image of safe gas use.

[0089] The multimodal retrieval method provided in this application unifies the semantics of prompt text and target image into target text through data modality rewriting, which can solve the heterogeneity problem of multimodal input and eliminate the need to directly process complex image data during the retrieval process. This application embodiment matches the target database based on prompt text, avoiding full scanning and improving retrieval efficiency. Then, cross-modal retrieval (such as text-to-image search, image-to-text search, or hybrid search) is achieved through the target text, covering complex query scenarios and realizing multimodal retrieval.

[0090] Taking a candidate database that can include a multimodal database of the target domain as an example, before an electronic device determines the target database from multiple candidate databases based on prompt text, it can construct the multimodal database. Figure 2 This application provides a flowchart illustrating a method for constructing a multimodal database, as shown in the embodiments below. Figure 2 As shown, methods for constructing a multimodal database may include:

[0091] S201. Obtain the first sample cue in the target domain, and the first sample image.

[0092] Optionally, the target domain can be the specific domain that the multimodal database focuses on; for example, the target domain could be the emergency safety domain. The first sample prompt can be a relevant textual description of the target domain. The first sample image can be visual data from the target domain; for example, a 10-second clip of an anomaly or emergency event can be extracted at 1-3 frames per second, and multiple images of the resulting anomaly or emergency event can be used to construct the first sample image.

[0093] Optionally, the electronic device can obtain first sample cues and first sample images of the target domain from a public dataset by calling an interface.

[0094] S202. Input the first sample prompt and the first sample image into the multimodal feature generation model to obtain the first sample description text and the semantic features corresponding to the first sample image; the first sample description text is used to describe the first sample prompt and the first sample image.

[0095] Optionally, the multimodal feature generation model can be a model capable of processing multiple different modalities of data and extracting and generating relevant features from them. The first sample description text can be a text description generated based on the first sample prompt and the first sample image, used to describe the first sample prompt and the first sample image.

[0096] The semantic features corresponding to the first sample image can be a multi-feature vector that reflects the first sample image.

[0097] S203. Based on the first sample description text and semantic features, construct a multimodal database.

[0098] Optionally, a multimodal database can be a database system capable of storing and managing multiple different modalities of data. It can simultaneously accommodate data of multiple modalities such as text and images, and can perform efficient querying, retrieval, and analysis of this data.

[0099] Optionally, the electronic device can input the first sample description text, as well as semantic features, into a multimodal database construction model to obtain a multimodal database. This multimodal database construction model can refer to any existing large-scale model that constructs multimodal databases from data of different modalities.

[0100] The multimodal database constructed in this application embodiment can obtain the descriptive text of the first sample and the semantic features corresponding to the first sample image through a multimodal feature generation model based on the first sample prompt and the first sample image, and then construct the multimodal database. This multimodal database can contain cross-modal association information between text and image, enabling the multimodal database to support accurate semantic-based retrieval and improving the accuracy of multimodal retrieval.

[0101] The following section details how to construct a multimodal feature generation model. Before inputting the first sample cue and the first sample image into the multimodal feature generation model to obtain the first sample description text and the semantic features corresponding to the first sample image, the electronic device can also acquire a second sample cue and a second sample image from the target domain.

[0102] Optionally, the second sample prompt and the second sample image can be an image-text pair, wherein the image in the image-text pair can correspond to the second sample image, and the text in the image-text pair can correspond to the second sample prompt. Electronic devices can obtain the image-text pair directly from an open-source database by calling an interface, or they can obtain the image-text pair by processing the image, table, or video to be processed.

[0103] Taking the processing of images, tables, and videos by electronic devices to obtain image-text pairs as an example, Figure 3 This is a flowchart illustrating a method for obtaining image-text pairs provided in an embodiment of this application, as shown below. Figure 3 As shown, the method for obtaining image-text pairs is as follows:

[0104] 1. Use the existing multimodal large model to process all the images, tables and videos to be processed, and obtain the corresponding detailed text descriptions, i.e. image-text pairs.

[0105] 2. Use another existing multimodal large model to score the text description matching degree of the image-text generated in step 1, and retain image-text pairs with a text description matching degree higher than a threshold. For example, retain image-text pairs with a text description matching degree higher than 8.

[0106] 3. Use existing large language models to analyze the text descriptions in all image-text pairs and filter out image-text pairs that do not match the target domain.

[0107] 4. Use the visual encoder in the existing multimodal large model as the feature extractor for image data, extract the representation vectors only for the image data in all image-text pairs, and store the representation vectors of these images in the vector database.

[0108] 5. Filter vectors in the vector database based on similarity. For example, only one image is kept for each representation vector with a similarity greater than 0.95, and 50% of the images with a similarity between 0.85 and 0.95 are randomly removed, resulting in high-quality, non-repeating image-text pairs for the target domain.

[0109] After acquiring the second sample cue and the second sample image, the electronic device can train a preset model using the second sample cue and the second sample image to obtain a multimodal feature generation model. This preset model may include a hidden layer, a text output module, and a multimodal feature vector output module. Figure 4 This is a schematic diagram of a preset model provided in an embodiment of this application, such as... Figure 4As shown, the preset model may include image preprocessing, VisionTransformer (a model architecture for image processing), pixel unshuffle, projector, prompt, text tokenizer, large language model (LLM), language model head (LM Head), embedding projector, output text module, and multi-vectors module.

[0110] The preset model may also include a hidden layer. Figure 4 The hidden layer is not shown in the diagram. It is used to extract semantic features based on the second sample cue and the second sample image, resulting in semantic features that characterize the second sample cue and the second sample image.

[0111] The text output module can be used to output predicted text based on semantic features, which describes the second sample cue and the second sample image.

[0112] Optionally, the text output module can gain a deeper understanding of the relationship between the second sample prompt and the second sample image based on semantic features, and then generate and output predicted text describing the second sample prompt and the second sample image. This predicted text can be used to calculate the loss function during the training process, and the process loss function is used to adjust the model parameters of the preset model.

[0113] The multimodal feature vector output module is used to output predicted multimodal features based on semantic features to characterize the second sample cue and the second sample image.

[0114] Optionally, the multimodal feature vector output module can fuse the semantic features of the second sample cue and the second sample image to extract key semantic elements. Based on the mapping relationship between the key semantic elements and the multimodal feature vector, predicted multimodal features are obtained. These predicted multimodal features can be used to calculate the loss function during the training process, and the process loss function is used to adjust the model parameters of the preset model.

[0115] This application embodiment acquires second sample prompts and second sample images from the target domain, extracts cross-modal semantic features using hidden layers in a preset model, and then generates predicted descriptive text through a text output module and predicted feature vectors through a multimodal feature vector output module. This achieves integrated training of text-image joint feature learning and generation. This modular design enables the model to simultaneously optimize both semantic understanding (through text generation) and feature representation (through vector output). The trained feature generation model can output accurate text descriptions to support semantic retrieval and provide highly discriminative feature vectors for precise matching, significantly improving the generalization ability of multimodal data processing.

[0116] Figure 5 This is a flowchart illustrating a method for training a preset model provided in an embodiment of this application, as shown below. Figure 5 As shown, the training process of the above-mentioned preset model can be as follows:

[0117] S301. Freeze the trainable parameters of the multimodal feature vector output module, and train the part of the preset model other than the multimodal feature vector output module through the second sample prompt and the second sample image to obtain the first initial multimodal feature generation model.

[0118] Optionally, the trainable parameters can be parameters in the preset model that can be adjusted through optimization algorithms during training. The first initial multimodal feature generation model can be a model obtained by training the parts of the preset model other than the multimodal feature vector output module after freezing the trainable parameters of the multimodal feature vector output module. It can be an initial version of the multimodal feature generation model, which has a certain ability to extract features from the input data (second sample prompts and second sample images) and generate partial outputs.

[0119] S302. Freeze the trainable parameters of the text output module, and train the part of the first initial multimodal feature generation model other than the text output module using the second sample prompt and the second sample image to obtain the second initial multimodal feature generation model.

[0120] Optionally, the second initial multimodal feature generation model can be a model obtained by training the parts of the first initial multimodal feature generation model other than the text output module after freezing the trainable parameters of the text output module. It can be an intermediate stage in the development of multimodal feature generation models, and compared with the first initial model, it can have more optimized parameters and better performance in the parts other than the text output.

[0121] S303. Randomly freeze the trainable parameters of the multimodal feature vector output module or the trainable parameters of the text output module. Using the second sample prompt, the second sample image, and the Optical Character Recognition (OCR) dataset, train the unfrozen modules in the second initial multimodal feature generation model to obtain the third initial multimodal feature generation model.

[0122] Optionally, the electronic device can randomly select to freeze the trainable parameters of the multimodal feature vector output module or the text output module. The Optical Character Recognition (OCR) dataset can be a collection containing a large number of images with text and corresponding text annotations. In multimodal model training, the OCR dataset can provide additional text-image association information, helping the model better learn the semantic relationships between text and images, and improving the model's ability to understand images containing text.

[0123] S304. Using the second sample prompt, the second sample image, and the OCR dataset, train the third initial multimodal feature generation model to obtain the multimodal feature generation model.

[0124] Optionally, this multimodal feature generation model can fuse information from different modalities to uncover the intrinsic connections and semantic relationships between them.

[0125] For example, Figure 6 A flowchart illustrating a specific method for training a preset model, as provided in this application embodiment, is shown below. Figure 6 As shown, the training process of this preset model is as follows:

[0126] 1. Freeze the Embedding task and train the LM Head only on the vertical domain multimodal dataset. The vertical domain refers to the target domain mentioned above.

[0127] 2. Freeze the text generation task and train the Embedding task only on the vertical multimodal dataset.

[0128] 3. For the same batch of training data, there is a 50% chance of randomly starting either the generation or embedding task, and a 50% chance of performing both tasks simultaneously. This training data may include a vertical-domain multimodal dataset and an OCR dataset. The vertical-domain multimodal dataset can account for 70%, and the OCR dataset can account for 30%.

[0129] 4. All batches of training data are used simultaneously for two tasks, but only one batch (epoch) is executed in this stage. This training data is consistent with the training data in step 3.

[0130] This application employs a phased freeze training strategy. First, the feature vector module is frozen to train other parts, enabling the model to initially possess text generation capabilities. Then, the text module is frozen to optimize feature extraction and strengthen semantic feature representation. Subsequently, modules are randomly frozen, and an OCR dataset is introduced. Multi-task hybrid training enhances the model's joint understanding of text and images, addressing the overfitting problem of single-task training. Finally, fine-tuning with the full dataset achieves an optimal balance in text generation, feature extraction, and cross-modal alignment. This application effectively coordinates parameter update conflicts between multiple modules, significantly improving the model's generalization performance in complex scenarios. It ensures that the generated multimodal feature generation model can output accurate text descriptions and extract highly discriminative feature vectors, laying the foundation for building an efficient multimodal database.

[0131] Taking the candidate database as an example, which may include a knowledge graph database of the target domain, the electronic device can construct a knowledge graph database of the target domain before determining the target database from multiple candidate databases based on prompt text. Figure 7 This is a flowchart illustrating a method for constructing a knowledge graph database, as provided in an embodiment of this application. Figure 7 As shown, methods for constructing a knowledge graph database may include:

[0132] S401. Obtain domain knowledge documents for the target domain.

[0133] Optionally, domain knowledge documents can be a collection of files that record and present relevant knowledge within the target domain in various formats such as text, charts, and formulas. For example, domain knowledge documents can include plain text documents and rich text documents.

[0134] Alternatively, electronic devices can obtain domain knowledge documents for the target domain by calling interfaces based on open-source databases.

[0135] S402. In response to the domain knowledge document being a rich text document, convert the rich text document into an image format to obtain the image corresponding to the rich text document.

[0136] Alternatively, electronic devices can use existing image conversion tools or programming libraries to render and output the entire content of a rich text document containing formatting and elements as an image, thereby obtaining the corresponding image.

[0137] S403. Using the layout element extraction model, extract layout elements from the image corresponding to the rich text document, determine the region block image of the layout elements of the image corresponding to the rich text document, and the attribute information of the layout elements.

[0138] Optionally, the layout element extraction model can automatically identify and extract various layout elements from the image converted from the rich text document. The layout element extraction model can adopt an existing deep learning model.

[0139] Optionally, the layout element extraction model can first perform multiple batch processing on all the images converted from rich text documents until all the images have been processed. The following example uses one image per page. Secondly, based on the recognition results of each page, the model can segment out the regions of all elements on all pages, obtaining images corresponding to all elements in all rich text documents and all pages. Thirdly, the model can divide the images corresponding to all elements into two main categories, corresponding to different system prompts: text elements and chart elements. Finally, the model can group elements by image size within each category, reducing size differences within the same batch and avoiding unnecessary computational overhead caused by alignment issues.

[0140] S404. Based on region-segmented images and a multimodal feature generation model, multimodal feature vectors of elements in each page of a rich text document are obtained.

[0141] Optionally, the electronic device can use the multimodal feature vector generated by the multimodal feature generation model and the corresponding relationship of the region-blocked image to obtain the multimodal feature vector of the rich text document. The electronic device can also process the region-blocked image and the multimodal feature generation model to obtain the multimodal feature vector of the rich text document.

[0142] S405. Construct a knowledge graph database based on multimodal feature vectors, rich text documents, and attribute information of page elements.

[0143] Optionally, the rich text document can be the metadata of the rich text document.

[0144] For example, Figure 8 This application provides a specific schematic diagram of constructing a knowledge graph database, as shown in the embodiments. Figure 8 As shown, domain knowledge documents in the target domain can include Portable Document Format (PDF), Office (a document format), and plain text documents.

[0145] Electronic devices can employ conversion tools to convert documents to PDF format. These tools can transform all the aforementioned domain knowledge documents into PDF format, and then into images in Joint Photographic Experts Group (JPG) format. A layout element extraction model can then identify layout elements in the images, extract element numbers, and segment the images into blocks based on areas such as images, tables, body text, and titles.

[0146] Multimodal large models can extract text and knowledge from each element by batch processing text elements, as well as graph and table elements. They can also extract embedding vectors from sliced ​​text, including recombining the recognition results of each element into a Markdown (a markup language) plain text document, and extracting vectors based on document slices.

[0147] The layout elements are numbered, document slices are used to extract vectors, and document metadata is used to construct a domain knowledge graph.

[0148] In one implementation, the domain knowledge graph can include six entity types and six relation types, as shown below:

[0149] Domain (Emergency Safety Sub-sectors): Gas, Water, Petrochemical, Transportation, etc.

[0150] Doc Category: Document Type 1, Document Type 2, Document Type 3, etc.

[0151] Document: papers, regulations, management rules, etc.

[0152] Page: page1, page2, page3, etc.

[0153] Content Element: Image elements, table elements, heading elements, etc.

[0154] Knowledge Chunk: Document Chunk 1, Document Chunk 2, Document Chunk 3, etc.

[0155] The page entity contains four attributes: original document identifier (ID), number of images, number of tables, and page overview. The element entity contains five attributes: original document ID, original page ID, save path, text information, and representation vector. The knowledge fragment entity corresponds to each document slice, and each document slice corresponds to one or more element entities. The knowledge fragment entity contains three attributes: representation vector, text information, and number of elements.

[0156] The relation types are as follows:

[0157] BELONGS_TO (belongs to): describes which category a document belongs to. For example, "Gas Management Regulations" BELONGS_TO "Management Regulations" indicates that "Gas Management Regulations" belongs to the Management Regulations category.

[0158] IN_DOMAIN (Domain): Describes which sub-domain of emergency safety a document type belongs to. For example, "Management Regulations" IN_DOMAIN "Gas" indicates that the Management Regulations belong to the gas field of emergency safety.

[0159] HAS_PAGE (Containing Page): Describes which pages a document includes. For example, "Report" HAS_PAGE Page 1 indicates that the "Report" is located on page 1.

[0160] LOCATED_IN (located on): Describes which page an element is located on. For example, Table 3 LOCATED_IN Page 5 indicates that Table 3 is located on page 5.

[0161] COMPRISED_OF (Composition): Describes the elements contained in a knowledge fragment. For example, knowledge fragment ACOMPRISED_OF Table 3 + Text Paragraph 5 indicates that the knowledge fragment contains Table 3 and Text Paragraph 5.

[0162] CITES (Citation): Describes which documents the current document references. For example, an academic paper CITES another document indicates that the academic paper references another document.

[0163] This knowledge graph can trace the source of each knowledge fragment retrieved to the specific document, page, and location of that fragment, facilitating easy querying and confirmation.

[0164] This application's embodiments convert rich text documents in the target domain into images and use a layout element extraction model for structured parsing. This accurately locates the regional blocks and attribute information of elements such as text and tables, effectively solving the parsing problem of multimodal data under complex layouts and improving the accuracy of subsequent retrieval.

[0165] Taking the example of an electronic device processing region-segmented images and a multimodal feature generation model to obtain multimodal feature vectors for rich text documents, in one implementation... Figure 9 This application provides a flowchart illustrating a method for obtaining multimodal feature vectors of rich text documents, as illustrated in the embodiments of this application. Figure 9 As shown, methods for obtaining multimodal feature vectors of rich text documents may include:

[0166] S501. Input the region segmented image and the preset prompt words into the multimodal feature generation model to obtain the descriptive text of the region segmented image.

[0167] For example, an electronic device can batch input region-segmented images into a multimodal feature generation model, perform inference analysis using preset prompts, and obtain the text in the region-segmented images. For example, the preset prompts can be based on the context of the element, providing detailed information about the message the element conveys, saving the element as an image to a server, and then adding the server address of the image as additional content to the element's detailed description text.

[0168] For text-based elements, electronic devices can set preset prompts to improve the ability of multimodal large-scale text recognition while trying to preserve the original text.

[0169] S502. Based on the description text of the region-block image, obtain a text document.

[0170] Optionally, the electronic device can use a document generation and writing tool to write the text information corresponding to all elements into the document in a top-to-bottom and left-to-right order, thereby obtaining a text document corresponding to the original rich text document.

[0171] S503. Slice the text document to obtain multiple document slices.

[0172] Alternatively, electronic devices can adopt a combination of dynamic adaptation and a limit on the number of characters, instead of using a fixed number of characters for segmentation. When segmenting, both the number of characters and the integrity of page element information must be considered.

[0173] For example, an electronic device can set an upper limit for the number of characters, a moderate limit for the number of characters, and a text overlap limit to handle extreme cases where the number of characters in a single element exceeds the upper limit. When segmenting a text document, as long as the number of characters does not exceed the moderate limit, the text information corresponding to the element is continuously added. If, after adding text information for an element, the total number of characters does not exceed the upper limit but exceeds the moderate limit, then the text document is segmented, and the next text document is segmented. If, after adding text information for an element, the total number of characters exceeds the upper limit, then the text information corresponding to that element is not added to the text document, and the text document is also segmented, and the next text document is segmented. If the text information corresponding to a single element exceeds the upper limit (in extreme cases, such as a very long paragraph or a very large table), then the element is segmented at the upper limit, into two parts. The upper part is the segmented text document, and the lower part plus the last overlap of the upper part is segmented into the next text document.

[0174] S504. Input multiple document slices into the multimodal feature generation model to obtain the multimodal feature vectors of each document slice of the rich text document.

[0175] This application embodiment accurately extracts text information from images by inputting region-block images of rich text documents and preset prompts into a multimodal feature generation model, effectively solving the accuracy problem of text recognition under complex layouts. This application embodiment can generate text documents based on recognized text and perform slicing, transforming long documents into structured short text units, reducing the complexity of semantic understanding. Finally, the multimodal feature generation model extracts features from document slices, fusing text semantics and layout spatial information to generate multimodal feature vectors that comprehensively represent the content of rich text documents. This process achieves efficient conversion from unstructured documents to structured features, preserving both the textual semantic details of the document and capturing the visual features of the layout, significantly improving the accuracy of multimodal retrieval.

[0176] In conclusion, Figure 10 This is a schematic diagram illustrating the construction of a candidate database as provided in an embodiment of this application, such as... Figure 10 As shown, images and videos can be used to obtain a multimodal database based on a multimodal large-scale model and system prompts. This multimodal database can include multi-vectors (multimodal feature vectors) and text descriptions. Image-text documents and text documents can be used to construct a domain knowledge graph based on a rich text document parsing module and a multimodal large-scale model. Candidate databases can include both multimodal databases and domain knowledge graphs.

[0177] The above are the method embodiments provided in this application. The apparatus provided in this application will be described below.

[0178] Figure 11 This is a schematic diagram of the structure of a multimodal retrieval device provided in this application, as shown below. Figure 11 As shown, the multimodal retrieval device 600 provided in this embodiment includes: an acquisition module 601, a rewriting module 602, a determination module 603, and a retrieval module 604. Optionally, the multimodal retrieval device 600 may further include a processing module 605.

[0179] The acquisition module 601 is used to acquire the prompt to be responded to, which includes: prompt text and target image.

[0180] The rewriting module 602 is used to rewrite the data modality of the prompt to be responded to, so as to obtain the target text. The target text is used to describe the content represented by the prompt text and the target image.

[0181] The determination module 603 is used to determine the target database from multiple candidate databases based on prompt text; the multiple candidate databases include: a multimodal database of the target domain and a knowledge graph database of the target domain, the knowledge graph database being constructed based on the attribute information of the layout elements of rich text documents in the target domain, and the rich text documents.

[0182] The retrieval module 604 is used to retrieve the target results from the target database based on the target text.

[0183] Optionally, the candidate database includes a multimodal database of the target domain. Before the determination module 603 determines the target database from multiple candidate databases based on the prompt text, the processing module 605 acquires a first sample prompt and a first sample image from the target domain. The first sample prompt and the first sample image are input into a multimodal feature generation model to obtain a first sample description text and semantic features corresponding to the first sample image; the first sample description text describes the first sample prompt and the first sample image. Based on the first sample description text and the semantic features, a multimodal database is constructed.

[0184] Optionally, the candidate database also includes a knowledge graph database of the target domain. Before the determination module 603 determines the target database from multiple candidate databases based on the prompt text, the processing module 605 is further configured to obtain domain knowledge documents of the target domain. In response to the domain knowledge document being a rich text document, the rich text document is converted into an image format to obtain the image corresponding to the rich text document. Using a layout element extraction model, layout elements are extracted from the image corresponding to the rich text document to determine the region-block images of the layout elements of the image corresponding to the rich text document, as well as the attribute information of the layout elements. Based on the region-block images and a multimodal feature generation model, multimodal feature vectors of each layout element of the rich text document are obtained. Based on the multimodal feature vectors, the rich text document, and the attribute information of the layout elements, a knowledge graph database is constructed.

[0185] For example, processing module 605 is specifically used to input the region-blocked image and preset prompt words into a multimodal feature generation model to obtain descriptive text for the region-blocked image. Based on the descriptive text of the region-blocked image, a text document is obtained. The text document is sliced ​​to obtain multiple document slices. The multiple document slices are input into the multimodal feature generation model to obtain multimodal feature vectors for each document slice of the rich text document.

[0186] In one implementation, before the processing module 605 inputs the first sample prompt and the first sample image into the multimodal feature generation model to obtain the first sample description text and the semantic features corresponding to the first sample image, the processing module 605 is specifically used to acquire the second sample prompt and the second sample image in the target domain. Using the second sample prompt and the second sample image, a preset model is trained to obtain the multimodal feature generation model. The preset model includes: a hidden layer, a text output module, and a multimodal feature vector output module; the hidden layer is used to extract semantic features based on the second sample prompt and the second sample image to obtain semantic features representing the second sample prompt and the second sample image; the text output module is used to output predicted text describing the second sample prompt and the second sample image based on the semantic features; the multimodal feature vector output module is used to output predicted multimodal features representing the second sample prompt and the second sample image based on the semantic features; the predicted text and the predicted multimodal features are used to calculate the training process loss function, which is used to adjust the model parameters of the preset model.

[0187] For example, processing module 605 is specifically used to: freeze the trainable parameters of the multimodal feature vector output module, and train the part of the preset model other than the multimodal feature vector output module using second sample prompts and second sample images to obtain a first initial multimodal feature generation model. Freeze the trainable parameters of the text output module, and train the part of the first initial multimodal feature generation model other than the text output module using second sample prompts and second sample images to obtain a second initial multimodal feature generation model. Randomly freeze the trainable parameters of the multimodal feature vector output module, or the trainable parameters of the text output module, and train the unfrozen modules in the second initial multimodal feature generation model using second sample prompts, second sample images, and an optical character recognition (OCR) dataset to obtain a third initial multimodal feature generation model. Train the third initial multimodal feature generation model using second sample prompts, second sample images, and an OCR dataset to obtain a multimodal feature generation model.

[0188] The multimodal retrieval device provided in this embodiment can execute the methods provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0189] Figure 12 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 12As shown, the electronic device 700 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the device 700 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0190] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0191] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0192] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0193] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0194] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0195] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0196] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0197] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0198] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0199] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0200] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0201] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0202] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0203] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0204] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A multi-modal retrieval method, characterized by, The method comprises: acquiring a to-be-responded prompt, the to-be-responded prompt comprising: prompt text and a target image; performing data modality rewriting on the to-be-responded prompt to obtain target text, the target text being used to describe content represented by the prompt text and the target image; determining a target database from a plurality of candidate databases based on the prompt text, the plurality of candidate databases comprising: a multi-modal database of a target field and a knowledge graph database of the target field, the knowledge graph database being based on attribute information of layout elements of a rich text document of the target field, and the rich text document being constructed; performing retrieval from the target database based on the target text to obtain a target retrieval result; before the determining of the target database from the plurality of candidate databases based on the prompt text, the method further comprises: acquiring a first sample prompt of the target field and a first sample image; inputting the first sample prompt and the first sample image into a multi-modal feature generation model to obtain a first sample description text and semantic features corresponding to the first sample image, the first sample description text being used to describe the first sample prompt and the first sample image; constructing the multi-modal database based on the first sample description text and the semantic features; before the inputting of the first sample prompt and the first sample image into the multi-modal feature generation model to obtain the first sample description text and the semantic features corresponding to the first sample image, the method further comprises: acquiring a second sample prompt of the target field and a second sample image; training a preset model through the second sample prompt and the second sample image to obtain the multi-modal feature generation model; the preset model comprising: a hidden layer, a text output module, and a multi-modal feature vector output module; the hidden layer being used to perform semantic feature extraction based on the second sample prompt and the second sample image to obtain semantic features used to represent the second sample prompt and the second sample image; the text output module being used to output predicted text used to describe the second sample prompt and the second sample image based on the semantic features; the multi-modal feature vector output module being used to output predicted multi-modal features used to represent the second sample prompt and the second sample image based on the semantic features, the predicted text and the predicted multi-modal features being used for process loss function calculation in a training process, and the process loss function being used to adjust model parameters of the preset model.

2. The method of claim 1, wherein, before the determining of the target database from the plurality of candidate databases based on the prompt text, the method further comprises: acquiring a field knowledge document of the target field; in response to the field knowledge document being the rich text document, converting the rich text document into an image format to obtain an image corresponding to the rich text document; The layout element extraction model is used to perform layout element extraction on the image corresponding to the rich text document, to determine a region block image of a layout element of the image corresponding to the rich text document, and to determine attribute information of the layout element; Based on the region block image and the multi-modal feature generation model, a multi-modal feature vector of each layout element of the rich text document is obtained. Based on the multi-modal feature vector, the rich text document, and the attribute information of the layout element, the knowledge graph database is constructed.

3. The method of claim 2, wherein, The method for obtaining the multi-modal feature vector of the rich text document based on the region block image and the multi-modal feature generation model includes: The region block image and a preset prompt word are input into the multi-modal feature generation model to obtain a description text of the region block image; Based on the description text of the region block image, a text document is obtained; The text document is sliced to obtain a plurality of document slices; The plurality of document slices are input into the multi-modal feature generation model to obtain the multi-modal feature vector of each document slice of the rich text document.

4. The method of claim 1, wherein, The method for training the preset model based on the second sample prompt and the second sample image to obtain the multi-modal feature generation model includes: The trainable parameters of the multi-modal feature vector output module are frozen, and the part of the preset model other than the multi-modal feature vector output module is trained based on the second sample prompt and the second sample image to obtain a first initial multi-modal feature generation model; The trainable parameters of the text output module are frozen, and the part of the first initial multi-modal feature generation model other than the text output module is trained based on the second sample prompt and the second sample image to obtain a second initial multi-modal feature generation model; The trainable parameters of the multi-modal feature vector output module or the trainable parameters of the text output module are randomly frozen, and the unfrozen modules of the second initial multi-modal feature generation model are trained based on the second sample prompt, the second sample image, and an optical character recognition (OCR) data set to obtain a third initial multi-modal feature generation model; The third initial multi-modal feature generation model is trained based on the second sample prompt, the second sample image, and the OCR data set to obtain the multi-modal feature generation model.

5. A multi-modal retrieval apparatus characterized by comprising: The method includes: An acquisition module is configured to acquire a to-be-responded prompt, the to-be-responded prompt including a prompt text and a target image; A rewriting module is configured to perform data modality rewriting on the to-be-responded prompt to obtain a target text, the target text being used to describe the content represented by the prompt text and the target image; determining module, configured to determine a target database from a plurality of candidate databases based on the prompt text; the plurality of candidate databases include a multi-modal database of a target field and a knowledge graph database of the target field, the knowledge graph database is based on attribute information of layout elements of a rich text document of the target field, and the rich text document is constructed; retrieving module, configured to retrieve from the target database based on the target text to obtain a target retrieval result; before the determining module is configured to determine the target database from the plurality of candidate databases based on the prompt text, the processing module is configured to: obtain a first sample prompt and a first sample image of the target field; input the first sample prompt and the first sample image into a multi-modal feature generation model to obtain a first sample description text and semantic features corresponding to the first sample image; the first sample description text is used to describe the first sample prompt and the first sample image; construct the multi-modal database based on the first sample description text and the semantic features; before the processing module is configured to input the first sample prompt and the first sample image into a multi-modal feature generation model to obtain a first sample description text and semantic features corresponding to the first sample image, the processing module is specifically configured to: obtain a second sample prompt and a second sample image of the target field; train a preset model through the second sample prompt and the second sample image to obtain the multi-modal feature generation model; the preset model includes a hidden layer, a text output module, and a multi-modal feature vector output module; the hidden layer is configured to perform semantic feature extraction based on the second sample prompt and the second sample image to obtain semantic features for representing the second sample prompt and the second sample image; the text output module is configured to output a predicted text for describing the second sample prompt and the second sample image based on the semantic features; the multi-modal feature vector output module is configured to output a predicted multi-modal feature for representing the second sample prompt and the second sample image based on the semantic features; the predicted text and the predicted multi-modal feature are used for process loss function calculation in a training process, and the process loss function is used for adjusting model parameters of the preset model.

6. An electronic device, comprising: comprise: a memory, a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the method of any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method of any one of claims 1-4.

8. A computer program product, characterised in that, The computer program is executed by the processor to implement the method of any one of claims 1-4.

Images