A multimodal text retrieval method, system, and medium

Abstract

This invention discloses a multimodal text retrieval method, system, and medium. The method involves acquiring the query text, a first text database, and a second text database; constructing an optimal inverted index model based on the first text database and an optimal vector similarity model using the second text database; matching candidate documents between the query text and the first text database using the optimal inverted index model, and calculating a first similarity score between the candidate documents and the first text database; matching candidate vector documents between the query text and the second text database using the optimal vector similarity model, and calculating a second similarity score between the candidate vector documents and the second text database; and using a branch logic monitoring method to determine whether the first and second similarity scores satisfy a threshold termination condition. If so, the retrieval result is output. The beneficial effects of this invention are improved generalization of retrieval content and increased retrieval efficiency.

Description

Technical Field

[0001] This invention relates to the field of text retrieval technology, and more specifically, to a multimodal text retrieval method, system, and medium. Background Technology

[0002] Financial scenarios involve highly complex text applications, encompassing a large volume of business, technical, and contractual documents. These documents need to be circulated and retrieved between various departments, making rapid and accurate document retrieval crucial for improving employee efficiency. Currently, the industry typically uses a dedicated system to collect these documents and provide search functionality. However, due to the expanding scale of business and the unstructured complexity of the documents, accurately identifying target documents with low latency is challenging.

[0003] In existing technologies, keyword retrieval or text vector retrieval methods are usually used to retrieve the text to be retrieved. However, when using this method to retrieve relevant text, the content to be retrieved is limited, the retrieval efficiency is low, and the retrieval time is long.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that in the prior art, the use of vector retrieval or keyword retrieval methods to retrieve relevant documents has limitations in the content to be retrieved, and the retrieval efficiency is low and the retrieval time is long. The purpose is to provide a multimodal text retrieval method, system and medium that improves the generalization of the retrieval content and improves the retrieval efficiency.

[0006] This invention is achieved through the following technical solution:

[0007] A multimodal text retrieval method, comprising the following steps:

[0008] Obtain the text to be queried, a first text library, and a second text library. The first text library is a document text library, and the second text library is a semantic similarity text library.

[0009] An optimal inverted index model is constructed based on the first text library, and an optimal vector similarity model is constructed using the second text library;

[0010] The optimal inverted index model is used to match the candidate documents for word segmentation between the text to be queried and the first text library, and the first similarity score between the candidate documents for word segmentation and the first text library is calculated.

[0011] The optimal vector similarity model is used to match the candidate vector documents between the text to be queried and the second text library, and a second similarity score is calculated between the candidate vector documents and the second text library.

[0012] The branch logic monitoring method is used to determine whether there are scores that meet the threshold termination condition between the first similarity score and the second similarity score. If so, the search results are output.

[0013] Traditional document retrieval methods typically employ keyword or vector search. However, these methods are limited in their scope, inefficient, and time-consuming. This invention provides a multimodal text retrieval method that matches the segmented and vector text of the query text separately. By integrating the relationships between the segmented and vector text with threshold conditions, the method achieves document retrieval, improving the generalization of the search results and increasing retrieval efficiency.

[0014] Preferably, the optimal inverted index model construction method is as follows:

[0015] Build an inverted index model;

[0016] The first text database is segmented using a word segmentation tool to obtain a text database.

[0017] The text database is imported into the inverted index model to obtain the optimal inverted index model.

[0018] Preferably, the method for constructing the optimal vector similarity model is as follows:

[0019] Construct a pre-trained model, and fine-tune the model in the second text library using the pre-trained model to obtain an offline document vector model;

[0020] The offline document model encodes all documents in the second text library into a text vector library, which is then imported into the vector similarity model to obtain the optimal vector similarity model.

[0021] Preferably, the branch logic monitoring method is used to determine whether there are any scores that satisfy the threshold termination condition between the first similarity score and the second similarity score. If so, the search results are output including:

[0022] The branch logic monitoring method divides the score into an unreliable interval, a pending interval, and a reliable interval. The threshold termination condition is that the similarity score is in the unreliable interval or the reliable interval. When both the first similarity score and the second similarity score are in the unreliable interval, a search result of no matching document is output. When both the first similarity score and the second similarity score are in the reliable interval, the candidate document corresponding to the higher similarity score is output.

[0023] Preferably, when neither the first similarity score nor the second similarity score meets the threshold termination condition, a refined search is performed, specifically including:

[0024] The candidate document library is obtained by taking the union of the word segmentation candidate documents and the vector candidate documents;

[0025] The similarity between the query text and each sample in the candidate document library is calculated using the edit distance algorithm, and the number of calculations is equal to the number of samples in the candidate document library to obtain a third similarity score;

[0026] The branch logic monitoring method is used to determine whether the third similarity score meets the threshold termination condition. If it does, the search result is output.

[0027] Preferably, when the third similarity does not meet the threshold termination condition, the retrieval method further includes:

[0028] Calculate the average weight of the first similarity score, the second similarity score, and the third similarity score, and then reorder them to obtain a comprehensive score;

[0029] Based on the comprehensive score, the candidate document library is reordered according to the score, and the top N candidate documents after sorting are selected.

[0030] The fourth similarity score is obtained by calculating the speech similarity between the query text and N candidate documents using a semantic model.

[0031] The branch logic monitoring method is used to determine whether the fourth similarity score meets the threshold termination condition. If it does, the search result is output.

[0032] The present invention also provides a multimodal text retrieval system, including a data acquisition module, a model building module, a first calculation module, a second calculation module, and a judgment module;

[0033] The data acquisition module is used to acquire the text to be queried, a first text library, and a second text library. The first text library is a document text library, and the second text library is a semantic similarity text library.

[0034] The model building module is used to build an optimal inverted index model based on the first text library and to build an optimal vector similarity model through the second text library.

[0035] The first calculation module is used to match the candidate documents for word segmentation between the text to be queried and the first text library through the optimal inverted index model, and to calculate the first similarity score between the candidate documents for word segmentation and the first text library;

[0036] The second calculation module is used to match the candidate vector documents between the text to be queried and the second text library through the optimal vector similarity model, and to calculate the second similarity score between the candidate vector documents and the second text library;

[0037] The judgment module is used to determine whether there are scores that satisfy the threshold termination condition between the first similarity score and the second similarity score through a branch logic monitoring method. If there are, the search result is output.

[0038] Preferably, the model building module includes an optimal inverted index model building module and an optimal vector similarity building module. The optimal inverted index building module is used to build an optimal inverted index model based on the first text library, and the optimal vector similarity building module is used to build an optimal vector similarity model through the second text library.

[0039] Preferably, the optimal inverted index construction module includes a sub-model construction module, a word segmentation module, and a data import module;

[0040] The sub-model building module is used to build the inverted index model;

[0041] The word segmentation module is used to perform word segmentation processing on the first text library using a word segmentation tool to obtain a text database;

[0042] The data import module is used to import the text database into the inverted index model to obtain the optimal inverted index model.

[0043] The present invention also provides a computer storage medium having a computing program stored thereon, which, when executed by a processor, implements the method described above.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] The present invention provides a multimodal text retrieval method, system, and medium that matches the segmented text and vector text of the query text separately, and achieves the retrieval of the query file by comprehensively considering the relationship between the segmented text and vector text and threshold conditions. This improves the generalization of the retrieval content and enhances the retrieval efficiency. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the retrieval method;

[0048] Figure 2 This is a flowchart of the retrieval method. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0050] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that these specific details are not necessary to practice the invention. In other embodiments, well-known structures, circuits, materials, or methods have not been specifically described in order to avoid obscuring the invention.

[0051] Throughout this specification, references to "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with that embodiment or example is included in at least one embodiment of the present invention. Therefore, the phrases "an embodiment," "an example," "an example," or "an example" appearing in various places throughout the specification do not necessarily refer to the same embodiment or example. Furthermore, specific features, structures, or characteristics can be combined in one or more embodiments or examples in any suitable combination and / or sub-combination. Moreover, those skilled in the art will understand that the illustrations provided herein are for illustrative purposes and are not necessarily drawn to scale. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0052] In the description of this invention, the terms "front", "rear", "left", "right", "up", "down", "vertical", "horizontal", "high", "low", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the scope of protection of this invention.

[0053] Example 1

[0054] Traditional document retrieval methods typically employ keyword or vector search. However, these methods have limitations in terms of the content that can be retrieved, resulting in low efficiency and long search times.

[0055] This embodiment discloses a multimodal text retrieval method. By matching the segmented text and vector text of the query text separately, and combining the relationship between the segmented text and vector text with threshold conditions, the method achieves retrieval of the query file, improving the generalization of the search content and increasing retrieval efficiency. The retrieval method steps in this embodiment are as follows: Figure 1 , Figure 2 As shown, the method steps include:

[0056] S1: Obtain the text to be queried, the first text library, and the second text library. The first text library is a document text library, and the second text library is a semantic similarity text library.

[0057] In step S1, the first and second text libraries are used to build corresponding models, which can be used to process the query text. The first and second text libraries build different models. Therefore, the first text library is a document text library, which is mainly used to build a word segmentation document model, and the second text library is a semantic similarity text library, which is used to build a vector document model.

[0058] S2: Construct the optimal inverted index model based on the first text library, and construct the optimal vector similarity model using the second text library;

[0059] The optimal inverted index model construction method is as follows: construct an inverted index model; use a word segmentation tool to perform word segmentation on the first text library to obtain a text database; import the text database into the inverted index model to obtain the optimal inverted index model.

[0060] Specifically, all documents are segmented using a word segmentation tool and stored in a document repository (a regular database is sufficient); the segmented document repository is then loaded into the Elasticsearch inverted index search tool to obtain the optimal inverted index model.

[0061] The optimal vector similarity model is constructed as follows: a pre-trained model is constructed, and the model is fine-tuned in the second text library using the pre-trained model to obtain an offline document vector model; all documents in the second text library are encoded and converted into a text vector library using the offline document model, and the text vector library is imported into the vector similarity model to obtain the optimal vector similarity model.

[0062] The specific implementation method is as follows: A semantic similarity dataset is manually created, for example, approximately 3000 entries for ("Employee Management", "Employee Management Manual", synonyms) and ("Employee Management", "Stamping Process Documents", different from synonyms); the pre-trained model BERT is used to fine-tune the model on the prepared semantic similarity dataset to obtain the offline document vector model Sentence-BERT; all documents are encoded into vectors using the Sentence-BERT model, a time-consuming step (approximately one text per second), so pre-processing is necessary; the vectors are loaded into the FASSI vector search tool to obtain the optimal vector similarity model.

[0063] S3: Match the candidate documents for word segmentation between the text to be queried and the first text library using the optimal inverted index model, and calculate the first similarity score between the candidate documents for word segmentation and the first text library;

[0064] S4: Match the candidate vector documents between the text to be queried and the second text library using the optimal vector similarity model, and calculate the second similarity score between the candidate vector documents and the second text library;

[0065] In steps S3 and S4, the query text is processed by two models to obtain segmented candidate documents and vector candidate documents based on word segmentation. In this embodiment, the union of segmented documents and vector documents is used for simultaneous retrieval, which can increase the generalization of document retrieval. The inverted index-elastic search tool in this embodiment is the optimal inverted index model, and the vector similarity-fasis tool is the optimal vector similarity model. Specifically, the user inputs the query text, such as "Employee Behavior Management Measures". The inverted index-elastic search tool and the vector similarity-fasis tool are used to search and obtain the top N1 most matching candidate options and their corresponding inverted index similarity scores and semantic vector similarity scores S1 and S2 from the candidate document library. The union U is taken according to the retrieval results of the two tools. The number of texts in U is in the range of N1<=U<=2N1. The branch logic monitors and performs the first detection of S1 and S2. If there are candidate options in S1 and S2 that meet the threshold termination condition, the retrieval ends and the matching answer and score are returned. Otherwise, the refined retrieval step is entered.

[0066] S5: Determine whether there are any scores that satisfy the threshold termination condition between the first similarity score and the second similarity score by using the branch logic monitoring method. If they exist, output the search results.

[0067] The branch logic monitoring method determines whether there are any scores that satisfy the threshold termination condition between the first similarity score and the second similarity score. If so, the search results are output, including:

[0068] The branch logic monitoring method divides the score into an unreliable interval, a pending interval, and a reliable interval. The threshold termination condition is that the similarity score is in the unreliable interval or the reliable interval. When both the first similarity score and the second similarity score are in the unreliable interval, a search result of no matching document is output. When both the first similarity score and the second similarity score are in the reliable interval, the candidate document corresponding to the higher similarity score is output.

[0069] Specifically: The scores stored in the score sequence are usually decimals between 0 and 1. The logic monitoring module typically divides the score sequence into three threshold stages, such as [0, 0.4), [0.4, 0.7), and [0.7, 0.1]. When the score falls into the [0, 0.4) interval, it is considered an unreliable interval, and the system returns a message indicating that the condition is not met, prompting the user that no matching document exists. When the score falls into the [0.4, 0.7) interval, it is considered a pending interval, and the system proceeds to the next step for a more detailed search. When the score falls into the [0.7, 0.1] interval, it is considered a reliable interval, and the document ranked first is directly returned as the result. If the similarity score falls into the pending interval, there is no further search step; the top 5 documents are returned as the final result for user confirmation. The threshold interval can be freely adjusted according to the number and content of the documents retrieved.

[0070] When neither the first similarity score nor the second similarity score meets the threshold termination condition, a refined search is performed, specifically including:

[0071] The candidate documents for word segmentation and the candidate documents for vectors are combined to obtain a candidate document library. The similarity between the text to be queried and each sample in the candidate document library is calculated using the edit distance algorithm, and the number of calculations is equal to the number of samples in the candidate document library to obtain a third similarity score. The branch logic monitoring method is used to determine whether the third similarity score meets the threshold termination condition. If it does, the search result is output.

[0072] Edit distance text similarity method description: Input: Undetermined texts A and B; Output: Similarity score between A and B.

[0073] Edit distance is a quantitative measure of the difference between two strings (such as English words). It measures the minimum number of operations required to transform one string into another. Edit distance can be used in natural language processing; for example, spell checking can determine which (or several) of the misspelled words is more likely based on their edit distance to other correct words. Transforming text A into text B involves three operations: inserting a character, deleting a character, and replacing a character. Each operation increments the distance by 1, and the final result is normalized to a value between 0 and 1.

[0074] The edit distance text similarity method is used to calculate the similarity between the query text and each sample in the candidate document library. The number of calculations is equal to the number of candidate documents in the candidate document library, and all edit distance similarity scores S3 are obtained. The branch logic monitors and performs a second detection S3. If there are candidate options that meet the threshold termination condition, the search ends and the matching answer and score are returned. If no matching option is detected, the average weight of S1 and S2 S3 is calculated and sorted to obtain the comprehensive score S1. 23 And proceed to the next step.

[0075] When the third similarity score does not meet the threshold termination condition, the retrieval method further includes: calculating the average weight of the first similarity score, the second similarity score, and the third similarity score, and reordering them to obtain a comprehensive score; based on the comprehensive score, reordering the candidate document library according to the score size, and taking the top N candidate documents after sorting; calculating the speech similarity between the query text and the N candidate documents through a semantic model to obtain a fourth similarity score; and determining whether the fourth similarity score meets the threshold termination condition through a branch logic monitoring method. If it does, the retrieval result is output.

[0076] The specific implementation process is as follows: based on the obtained comprehensive score S... 123 The candidate set is reordered to narrow it down further, and the top N2 candidate documents are selected (N2 is recommended to be less than or equal to 1000). The semantic model M trained in the preparation steps is used to calculate the semantic similarity between the query text and the N2 candidate documents (the model outputs a score from 0 to 1, where closer to 0 indicates the two sentences are not synonymous, and vice versa), resulting in a similarity score S4. Finally, the scores are combined... 123 The final score S is obtained by combining S4 and S4. final According to S final Sort the documents; perform a third check on the branch logic monitoring. When the score is in the unreliable range, push the document with the highest score as the final result; when the score is in the pending range, push the top 5 documents with the highest scores as the final result for user confirmation; when the score is in the reliable range, inform the user that there are no matching search results in the library.

[0077] This embodiment discloses a multimodal text retrieval method that incorporates multimodal content, retrieving information from both literal and semantic perspectives. The introduction of a semantic model increases the generalization of the search; for example, a user input of "must-read for new employees" can semantically match "employee management manual," a feat traditional keyword detection cannot achieve. Furthermore, it eliminates the need to maintain synonyms. Branch logic detection effectively reduces retrieval time, meaning not all searches require a complete process. Under certain precise user input conditions, the final result can be returned in the first step (milliseconds). Even with ambiguous user input, the entire process takes only seconds (for documents under 100,000).

[0078] Example 2

[0079] This embodiment discloses a multimodal text retrieval system. This embodiment is designed to implement the retrieval method as described in Embodiment 1, and includes a data acquisition module, a model construction module, a first calculation module, a second calculation module, and a judgment module.

[0080] The data acquisition module is used to acquire the text to be queried, a first text library, and a second text library. The first text library is a document text library, and the second text library is a semantic similarity text library.

[0081] The model building module is used to build an optimal inverted index model based on the first text library and to build an optimal vector similarity model through the second text library.

[0082] The first calculation module is used to match the candidate documents for word segmentation between the text to be queried and the first text library through the optimal inverted index model, and to calculate the first similarity score between the candidate documents for word segmentation and the first text library;

[0083] The second calculation module is used to match the candidate vector documents between the text to be queried and the second text library through the optimal vector similarity model, and to calculate the second similarity score between the candidate vector documents and the second text library;

[0084] The judgment module is used to determine whether there are scores that satisfy the threshold termination condition between the first similarity score and the second similarity score through a branch logic monitoring method. If there are, the search result is output.

[0085] The model building module includes an optimal inverted index model building module and an optimal vector similarity building module. The optimal inverted index building module is used to build an optimal inverted index model based on the first text library, and the optimal vector similarity building module is used to build an optimal vector similarity model through the second text library.

[0086] The optimal inverted index construction module includes a sub-model construction module, a word segmentation module, and a data import module;

[0087] The sub-model building module is used to build the inverted index model;

[0088] The word segmentation module is used to perform word segmentation processing on the first text library using a word segmentation tool to obtain a text database;

[0089] The data import module is used to import the text database into the inverted index model to obtain the optimal inverted index model.

[0090] Example 3

[0091] This embodiment discloses a computer storage medium storing a computing program, which, when executed by a processor, implements the method described in Embodiment 1.

[0092] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program issuing instructions. These computer program issuing instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the issuing instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program issuing instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing device to function in a particular manner, such that the issuing instructions stored in the computer-readable storage medium produce an article of manufacture including an issuing instruction means, the issuing instruction means being implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing the instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal text retrieval method, characterized in that, The method steps include: Obtain the text to be queried, a first text library, and a second text library. The first text library is a document text library, and the second text library is a semantic similarity text library. An optimal inverted index model is constructed based on the first text library, and an optimal vector similarity model is constructed using the second text library; The optimal inverted index model is used to match the candidate documents for word segmentation between the text to be queried and the first text library, and the first similarity score between the candidate documents for word segmentation and the first text library is calculated. The optimal vector similarity model is used to match the candidate vector documents between the query text and the second text library, and a second similarity score is calculated between the candidate vector documents and the second text library. The branch logic monitoring method is used to determine whether there are any scores that meet the threshold termination condition between the first similarity score and the second similarity score. If so, the search results are output. When neither the first similarity score nor the second similarity score meets the threshold termination condition, a refined search is performed, specifically including: The candidate document library is obtained by taking the union of the word segmentation candidate documents and the vector candidate documents; The similarity between the query text and each sample in the candidate document library is calculated using the edit distance algorithm, and the number of calculations is equal to the number of samples in the candidate document library to obtain a third similarity score; The branch logic monitoring method is used to determine whether the third similarity score meets the threshold termination condition. If it does, the search result is output. When the third similarity does not meet the threshold termination condition, the retrieval method further includes: Calculate the average weight of the first similarity score, the second similarity score, and the third similarity score, and then reorder them to obtain a comprehensive score; Based on the comprehensive score, the candidate document library is reordered according to the score, and the top N candidate documents after sorting are selected. The fourth similarity score is obtained by calculating the speech similarity between the query text and N candidate documents using a semantic model. The branch logic monitoring method is used to determine whether the fourth similarity score meets the threshold termination condition. If it does, the search result is output.

2. The multimodal text retrieval method according to claim 1, characterized in that, The optimal inverted index model construction method is as follows: Build an inverted index model; The first text database is segmented using a word segmentation tool to obtain a text database. The text database is imported into the inverted index model to obtain the optimal inverted index model.

3. The multimodal text retrieval method according to claim 1, characterized in that, The method for constructing the optimal vector similarity model is as follows: Construct a pre-trained model, and fine-tune the model in the second text library using the pre-trained model to obtain an offline document vector model; The offline document model converts all documents in the second text library into a text vector library, which is then imported into the vector similarity model to obtain the optimal vector similarity model.

4. A multimodal text retrieval method according to claim 2 or 3, characterized in that, The branch logic monitoring method determines whether there are any scores that satisfy the threshold termination condition between the first similarity score and the second similarity score. If so, the search results are output, including: The branch logic monitoring method divides the score into an unreliable interval, a pending interval, and a reliable interval. The threshold termination condition is that the similarity score is in the unreliable interval or the reliable interval. When both the first similarity score and the second similarity score are in the unreliable interval, a search result of no matching document is output. When both the first similarity score and the second similarity score are in the reliable interval, the candidate document corresponding to the higher similarity score is output.

5. A multimodal text retrieval system, characterized in that, It includes a data acquisition module, a model building module, a first calculation module, a second calculation module, and a judgment module; The data acquisition module is used to acquire the text to be queried, a first text library, and a second text library. The first text library is a document text library, and the second text library is a semantic similarity text library. The model building module is used to build an optimal inverted index model based on the first text library and to build an optimal vector similarity model through the second text library. The first calculation module is used to match the candidate documents for word segmentation between the text to be queried and the first text library through the optimal inverted index model, and to calculate the first similarity score between the candidate documents for word segmentation and the first text library; The second calculation module is used to match the candidate vector documents between the text to be queried and the second text library through the optimal vector similarity model, and to calculate the second similarity score between the candidate vector documents and the second text library; The judgment module is used to determine whether there are any scores that satisfy the threshold termination condition between the first similarity score and the second similarity score through a branch logic monitoring method. If there are, the search result is output. When neither the first similarity score nor the second similarity score meets the threshold termination condition, a refined search is performed, specifically including: The candidate document library is obtained by taking the union of the word segmentation candidate documents and the vector candidate documents; The similarity between the query text and each sample in the candidate document library is calculated using the edit distance algorithm, and the number of calculations is equal to the number of samples in the candidate document library to obtain a third similarity score; The branch logic monitoring method is used to determine whether the third similarity score meets the threshold termination condition. If it does, the search result is output. When the third similarity does not meet the threshold termination condition, the retrieval method further includes: Calculate the average weight of the first similarity score, the second similarity score, and the third similarity score, and then reorder them to obtain a comprehensive score; Based on the comprehensive score, the candidate document library is reordered according to the score, and the top N candidate documents after sorting are selected. The fourth similarity score is obtained by calculating the speech similarity between the query text and N candidate documents using a semantic model. The branch logic monitoring method is used to determine whether the fourth similarity score meets the threshold termination condition. If it does, the search result is output.

6. A multimodal text retrieval system according to claim 5, characterized in that, The model building module includes an optimal inverted index model building module and an optimal vector similarity building module. The optimal inverted index building module is used to build an optimal inverted index model based on the first text library, and the optimal vector similarity building module is used to build an optimal vector similarity model through the second text library.

7. A multimodal text retrieval system according to claim 6, characterized in that, The optimal inverted index construction module includes a sub-model construction module, a word segmentation module, and a data import module; The sub-model building module is used to build the inverted index model; The word segmentation module is used to perform word segmentation processing on the first text library using a word segmentation tool to obtain a text database; The data import module is used to import the text database into the inverted index model to obtain the optimal inverted index model.

8. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images