Retrieval method, device and equipment based on multi-modal data and medium

Abstract

The invention discloses a retrieval method based on multi-modal data. The method comprises the steps that a retrieval request of a user is received; executing a mixed retrieval operation in a constructed knowledge base based on the retrieval request to obtain a first matching result matched with the retrieval request and a second matching result matched with the retrieval request; fusing the first matching result and the second matching result to obtain a retrieval result matched with the retrieval request; inputting the retrieval result, a user query text corresponding to the retrieval request and a modal label of the user query text into a pre-trained multi-modal large model, and obtaining a text matching score based on a topic matching degree, a detail matching degree and a user intention integrating degree through the multi-model large model, and taking the retrieval result with the highest text matching score as a target retrieval result. The method has the advantages that by executing the mixed retrieval operation and fusing the multi-modal matching result, the multi-modal data can be effectively integrated, cross-modal accurate retrieval is realized, and the accuracy and comprehensiveness of the retrieval result are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a retrieval method, apparatus, device, and medium based on multimodal data. Background Technology

[0002] With the rapid development of information technology, data is showing a trend towards multimodalization, with various file types such as images, audio, video, text, and tables becoming important carriers of information transmission. Traditional search-enhanced generation (RAG) systems mainly process text modalities, and can only manage and retrieve plain text files, failing to effectively process files of other modalities.

[0003] In practical applications, users often need to acquire knowledge based on multimodal information. For example, they might search for product images and manuals using text descriptions, search for product instruction videos, or combine tabular data with text reports to obtain analytical conclusions. However, traditional RAG (Retrieval-augmented Generation) systems, due to limitations in modal processing, cannot effectively integrate multimodal files into their databases, nor can they achieve accurate cross-modal retrieval. This results in a large amount of valuable multimodal information being left idle, failing to meet users' needs for multimodal knowledge retrieval and limiting the application effectiveness of large AI models in multimodal scenarios. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, this application provides a retrieval method, apparatus, device and medium based on multimodal data to solve the above technical problems.

[0005] This application provides a retrieval method based on multimodal data, the retrieval method comprising: Receive user search requests; Based on the search request, a hybrid search operation is performed in the constructed knowledge base to obtain a first matching result and a second matching result that match the search request. The first matching result and the second matching result are merged to obtain the search result that matches the search request; The search results, the user query text corresponding to the search request, and the modal labels of the user query text are input into a pre-trained multimodal large model. The multimodal large model obtains a text matching score based on topic matching degree, detail matching degree, and user intent matching degree, and the search result with the highest text matching score is taken as the target search result.

[0006] In one embodiment of this application, performing a hybrid search operation in the constructed knowledge base based on the search request includes: Keyword extraction is performed on the user's search request to obtain the requested keywords; The request keywords are matched with multiple text segments corresponding to the original files in the knowledge base to obtain the first matching result; The user's search request is vectorized to obtain a vectorized representation of the search request; The vectorized representation of the retrieval request is matched with the vectorized representation of the original file in the knowledge base to obtain a second matching result.

[0007] In one embodiment of this application, obtaining a search result matching the search request based on the first matching result and the second matching result includes: The first matching result and the second matching result are weighted and fused to calculate the comprehensive score corresponding to each original file; wherein, the weight of the first matching result and the weight of the second matching result are determined by the intent recognition result of the intent recognition of the search request; The aggregated data is sorted to determine the search results.

[0008] In one embodiment of this application, the construction of the knowledge base includes: Obtain the original files for multiple modalities; Information extraction and modality conversion are performed on the original files of the various modalities, and the original files of the various modalities are converted into multiple text segment representations; wherein, each original file of a modality corresponds to at least one text segment representation; The multiple text segments are mapped to the same vector space to generate a vectorized representation; A knowledge base is constructed based on the multiple text segment representations and the vectorized representations.

[0009] In one embodiment of this application, the construction of the knowledge base further includes: The original files of the various modalities are verified, including determining whether the file format is within the preset supported range, whether the file is complete, and whether the file is corrupted. If the original file fails verification, an error message will be returned to the user. If the original file passes the verification, a distributed storage architecture is used to classify and store the original file, and an index is built according to the file modality type and format.

[0010] In one embodiment of this application, mapping the plurality of text segments to the same vector space includes: By using a pre-trained language model based on Transformer, the multiple text segment representations are converted into vectors of fixed dimensions to obtain an initial vector set; Modal alignment is performed on the initial vector set to ensure that text segment vectors from different modalities have semantic similarity in the vector space.

[0011] In one embodiment of this application, the original multimodal files include: visual files, audio files, video files, text files, and table files.

[0012] In one embodiment of this application, the step of matching the request keyword with multiple text segments corresponding to the original document in the knowledge base to obtain a first matching result includes: Based on the requested keywords, construct a first set; Construct a second set based on the text segments in the knowledge base; Calculate the intersection of the first set and the second set; Based on the intersection, determine whether the text segment contains at least one of the requested keywords; If included, the text segments in the intersection are determined as candidate text segments; Calculate the keyword matching degree of the candidate text segments; The candidate text segments are sorted based on the keyword matching degree to obtain the first matching result after sorting.

[0013] In one embodiment of this application, a method for calculating the keyword matching degree of the candidate text segment is as follows:

[0014] in, Indicates keyword matching degree, Q represents the first set, and D represents the second set. Indicates the number of elements in the set;

[0015] Where t represents a specific keyword, d represents a candidate text segment, TF(t,d) represents the frequency of word t in text segment d, and N represents the total number of candidate text segments in the knowledge base. This indicates the number of candidate text segments in the knowledge base that contain the keyword t.

[0016] This application provides a retrieval device based on multimodal data, the retrieval device comprising: The request receiving module is used to receive users' search requests; The retrieval and matching module is used to perform a hybrid retrieval operation in the constructed knowledge base based on the retrieval request, and obtain a first matching result and a second matching result that match the retrieval request. The result fusion module is used to fuse the first matching result and the second matching result to obtain the retrieval result that matches the retrieval request; The text matching module is used to input the search results, the user query text corresponding to the search request, and the modal labels of the user query text into a pre-trained multimodal large model. The multimodal large model obtains a text matching score based on topic matching degree, detail matching degree, and user intent matching degree, and takes the search result with the highest text matching score as the target search result.

[0017] This application provides a retrieval device based on multimodal data, the retrieval device comprising: One or more processors; A memory for storing one or more programs, which, when executed by one or more processors, enable the data processing to implement the steps of the multimodal data retrieval method described above.

[0018] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the steps of the above-described retrieval method based on multimodal data.

[0019] The beneficial effects of this application are as follows: This application provides a retrieval method based on multimodal data, comprising: receiving a user's retrieval request; performing a hybrid retrieval operation in a pre-constructed knowledge base based on the retrieval request to obtain a first matching result and a second matching result matching the retrieval request; fusing the first matching result and the second matching result to obtain a retrieval result matching the retrieval request; inputting the retrieval result, the user query text corresponding to the retrieval request, and the modal tags of the user query text into a pre-trained multimodal large model, obtaining a text matching score based on topic matching degree, detail matching degree, and user intent matching degree through the multimodal large model, and taking the retrieval result with the highest text matching score as the target retrieval result. This application achieves accurate cross-modal retrieval by performing a hybrid retrieval operation and fusing multimodal matching results, improving the accuracy and comprehensiveness of retrieval results, providing rich multimodal knowledge base support for AI large model applications, enabling large models to conveniently reference multimodal knowledge, and improving the response capability and application effect of large models in complex scenarios.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 This is a flowchart illustrating an exemplary retrieval method based on multimodal data according to this application; Figure 2 This is an exemplary flowchart of performing a hybrid search operation in a constructed knowledge base based on a search request, as described in this application. Figure 3 A flowchart illustrating the construction of an exemplary knowledge base for this application; Figure 4 A flowchart illustrating the construction of another exemplary knowledge base in this application; Figure 5 A block diagram illustrating a multimodal data retrieval device as an exemplary embodiment of this application; Figure 6 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0022] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0023] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0024] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0025] Please see Figure 1 , Figure 1 The flowchart below illustrates an exemplary retrieval method based on multimodal data, which includes at least steps S110 to S140, detailed below: Step S110: Receive the user's search request; Step S120: Perform a hybrid search operation in the constructed knowledge base based on the search request to obtain a first matching result and a second matching result that match the search request. Step S130: Merge the first matching result and the second matching result to obtain the search result that matches the search request; Step S140: Input the search results, the user query text corresponding to the search request, and the modal label of the user query text into the pre-trained multimodal big model. The multimodal big model obtains the text matching score based on topic matching degree, detail matching degree, and user intent matching degree, and takes the search result with the highest text matching score as the target search result.

[0026] Multimodal large models are deep learning models capable of handling various types of data (such as text, images, audio, video, etc.). Multimodal large models can employ multimodal pre-trained models based on the Transformer architecture (such as CLIP, BLIP, ViLT, etc.), which fuse information from different modalities and generate a unified semantic representation through cross-modal attention mechanisms.

[0027] Topic matching degree refers to the degree of consistency between the search results and the user query on the core topic, which can be achieved by calculating the cosine similarity of semantic vectors or the topic focus score based on the model's self-attention weights.

[0028] Detail relevance refers to the degree to which the specific facts, data, and descriptions in the search results match the detailed requirements mentioned in the user's query.

[0029] User intent relevance refers to the degree to which search results satisfy a user's potential query purpose (such as seeking definitions, comparisons, operation guides, causal analysis, etc.).

[0030] Modal tags are used to indicate the type of raw modal data (such as images, audio, video, structured tables, plain text documents, etc.) from which the query text originates. These tags can be obtained through query text analysis (such as detecting modal keywords), user interaction history, or predefined rule mapping, guiding the model to adjust modal awareness weights. For example, for the query "Tesla Model 3 front design," a text segment derived from an image description is more valuable in terms of detail relevance than a text segment derived from financial news, even if both contain the keyword "Tesla Model 3."

[0031] Specifically, the user's search request is received as initial input, converted into a text field, and then the original intent of the query is directly captured based on the text field, serving as the driving force for subsequent search operations. Based on the search request, a hybrid search operation is performed in a pre-built knowledge base: the search request is precisely matched with the text segment representation in the knowledge base through a keyword matching mechanism to obtain the first matching result; the semantic features of the search request are associated and matched with the vectorized representation in the knowledge base through similarity calculation of the vectorized representation to obtain the second matching result. After the hybrid search is completed, the first and second matching results are fused to generate the final search result. This fusion processing can comprehensively cover the complex semantic relationships of multimodal data. Finally, the multimodal big model jointly processes the input search results, user query text, and modality tags to obtain the topic matching degree. Detail matching User intent matching A quantitative scoring system based on three dimensions is used to obtain the final text matching score. Then, the search result with the highest score is selected as the target search result output.

[0032] Please see Figure 2 , Figure 2 This is an exemplary flowchart of performing a hybrid search operation in a constructed knowledge base based on a search request, as described in this application. Performing a hybrid search operation in a constructed knowledge base based on a search request includes at least steps S210 to S240, which are detailed below: Step S210: Extract keywords from the user's search request to obtain the requested keywords; In the keyword extraction step, a Chinese word segmentation tool is used to segment the query text. The weight value of each segmented word is calculated using the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm, and its formula is defined as follows:

[0033] in, Indicator In the document word frequency in In this context, N represents the total number of documents. This represents the number of documents containing the word t.

[0034] Step S220: Match the requested keywords with multiple text segments corresponding to the original files in the knowledge base to obtain the first matching result; Keyword matching can enhance the ability to accurately retrieve data based on user query intent, providing a high-confidence foundation for subsequent integration.

[0035] In one embodiment, the top 5 high-value words by weight are selected as core keywords. At the same time, it filters out meaningless stop words to reduce the interference of redundant information on search results.

[0036] Step S230: Vectorize the user's search request to obtain a vectorized representation of the search request; Vectorization can be achieved using models such as Word2Vec, FastText, or pre-trained language models based on Transformer. By vectorizing search requests, deep semantic information can be captured, enabling the search system to understand synonyms, near-synonyms, and contextual relationships.

[0037] In one embodiment, a pre-trained language model can be used to convert the query text after removing stop words into a d-dimensional semantic vector. This ensures that the query vector and the multimodal derived text vectors in the knowledge base are in the same semantic space, providing a consistent basis for the similarity calculation of subsequent vector retrieval.

[0038] Step S240: Perform similarity matching between the vectorized representation of the retrieval request and the vectorized representation of the original file in the knowledge base to obtain the second matching result.

[0039] Vector similarity matching measures the similarity between a retrieval request and its vectorized representation in a knowledge base. Specifically, cosine similarity, Euclidean distance, or Manhattan distance can be used to calculate the similarity between two vectors, thus obtaining the second matching result.

[0040] Calculate the query vector using cosine similarity. Text segment vector The semantic similarity is expressed by the formula:

[0041] in, For vector dot product, For vectors Norm. The similarity value ranges from [0,1]. The top 100 text segments with the highest similarity are returned as candidate results to ensure coverage of multimodal knowledge units with high semantic relevance.

[0042] Specifically, the user's search request is processed by keyword extraction and vectorization, generating the search request keywords and the vectorized representation of the search request, respectively; the search request keywords are matched with the text segment representation in the knowledge base to obtain the first matching result, and the vectorized representation of the search request is matched with the vectorized representation in the knowledge base to obtain the second matching result.

[0043] In one embodiment, obtaining search results that match the search request based on a first matching result and a second matching result includes: The first and second matching results are weighted and fused to calculate the overall score for each original file. The weights of the first and second matching results are determined by the intent recognition results of the search request. The overall scores are then sorted to determine the search results.

[0044] By weighted fusion, a comprehensive score can be calculated for each original file, providing a unified quantitative basis for subsequent ranking.

[0045] Specifically, this method generates intent recognition results by analyzing the semantic features of search requests in real time using a machine learning intent recognition algorithm. The weight ratio of the first and second matching results is dynamically adjusted based on the intent recognition results. The two matching results are then weighted and fused based on the adjusted weights to calculate a comprehensive score for each original file. All original files are then sorted according to the comprehensive scores to output highly relevant search results. This application optimizes search accuracy by increasing the weight of the first matching result when the user query is biased towards keyword matching and increasing the weight of the second matching result when the query is biased towards semantic matching.

[0046] This application dynamically adjusts the matching weights based on the intent of the user's search request, effectively solving the problem of inaccurate retrieval caused by fixed weights, and better meeting the user's search needs in different scenarios.

[0047] Please see Figure 3 , Figure 3 The flowchart below illustrates the construction of an exemplary knowledge base according to this application. The construction of the knowledge base includes at least steps S310 to S340, which are described in detail below: Step S310: Obtain the original files for multiple modalities; Among them, original files of various modalities can be obtained from heterogeneous data sources through file upload interfaces and web crawlers.

[0048] In one embodiment, the original multimodal files include: visual files, audio files, video files, text files, and table files.

[0049] Among them, visual files include, but are not limited to, image files in JPEG, JPG, PNG, and BMP formats; audio files refer to file types containing sound data, including but not limited to MP3, WMA, and WAV; video files include, but are not limited to, video files in MP4, MOV, AVI, and WKV formats; text files include, but are not limited to, text files in WORD, PDF, TXT, and MARKDOWN formats; and table files refer to, but are not limited to, table files in XLSX, XLS, and CSV formats.

[0050] For visual documents, OCR image recognition technology is used to extract text content from the images, including printed and handwritten text. The extracted text content is used as the text segment of the image file. If the image contains multiple independent text regions, it can be divided into multiple text segments based on the region location and content relevance.

[0051] For audio files, deep learning-based speech-to-text technology is used to convert the audio content into text information, while simultaneously recording the timestamp information of each text segment to the millisecond level. Based on the semantic features of the text content, such as pauses, topic shifts, and semantic completeness, the text is segmented into multiple text segments, each corresponding to a continuous time segment in the audio file. The time information of the audio file corresponding to each segment is also recorded.

[0052] For video files, multiple extraction methods are used to obtain content: speech-to-text technology is used to convert the audio stream in the video into audio text; a multimodal large model is used to analyze and summarize the overall video content to generate a video summary; keyframes are extracted from the video, and image recognition technology is used to describe the content of the keyframes (including objects, scenes, people, etc.) to form a keyframe summary. The content extracted by different methods is segmented into text segments: audio text segments correspond to the audio part of the video, video summary segments correspond to the overall summary of the video, and keyframe summary segments correspond to the key scenes of the video. Each segment clearly corresponds to a specific paragraph in the video.

[0053] For text files, different processing methods are used depending on the format: For plain text files such as WORD, TXT, and MARKDOWN, the text content is extracted directly based on the paragraph division of the file itself to form corresponding text segments; For PDF files, layout recognition technology is first used to distinguish different elements in the file such as text blocks, image areas, and table areas. The text content is extracted directly from the text blocks, and the text information in the image areas is extracted using OCR image recognition technology. Then, the files are segmented according to the layout structure and content relevance, and the page number information and coordinate range of each segment in the file are retained for easy tracking later.

[0054] For spreadsheet files, the file is split into worksheet units, breaking down a single spreadsheet file containing multiple worksheets into multiple independent worksheet cells. Each worksheet is further divided into multiple text segments with a fixed number of rows (adjustable between 5-50 rows depending on the content density). Each segment carries the corresponding worksheet header information, including column names, data types, and descriptions of column relationships. This ensures that each segment reflects a portion of the table data while also clearly defining the meaning of the data based on the header information.

[0055] Step S320: Extract information and perform modality conversion on the original files of multiple modalities, converting the original files of multiple modalities into multiple text segment representations; wherein, each original file of a modality corresponds to at least one text segment representation; Information extraction and modality conversion can use algorithms to parse the semantic content of non-textual data into a generated textual description, transforming the core information of the original file into a general textual form, preserving semantic integrity while adapting to subsequent text processing procedures.

[0056] Step S330: Map multiple text segments to the same vector space to generate a vectorized representation; After modality conversion is completed, word embedding models, recurrent neural network encoders, or autoencoder architectures can be used to map text segments to the same vector space to convert text segments into numerical vector representations, thereby eliminating modality differences and making semantically similar content closely related in the vector space.

[0057] Step S340: Construct a knowledge base based on multiple text segment representations and vectorized representations.

[0058] Specifically, the process involves retrieving original files from the user's end or external storage; performing information extraction and modality transformation operations on each type of original file, deconstructing non-text data into independent text segment representations, ensuring that each file unit generates at least one semantically coherent text segment; after converting all non-text data into text segments, mapping all text segments to the same vector space, generating fixed-dimensional vector representations through vectorization, aligning the semantics of different modal sources in terms of vector distance; and integrating text segment representations and vectorized representations to construct a knowledge base.

[0059] Please see Figure 4 , Figure 4 This is a flowchart illustrating the construction of an exemplary knowledge base according to this application. The construction of the knowledge base includes at least steps S410 to S440. Step S410: Verify the original files of multiple modalities. The verification includes determining whether the file format is within the preset supported range, whether the file is complete, and whether the file is corrupted. Among them, verification is a systematic verification of the quality of the original file. Verification can filter out invalid data and avoid interruption of subsequent information extraction due to abnormal input files. The preset supported range can be understood as the set of compatible file formats pre-configured by the system. Step S420: If the original file fails the verification, an error message is returned to the user; In step S430, if the original file passes the verification, the original file is classified and stored using a distributed storage architecture, and an index is created based on the file modality type and format.

[0060] Creating an index can be understood as building a retrieval structure for quick data location, which can be achieved using inverted indexes or category tag indexes.

[0061] Specifically, the original file undergoes triple verification for format compatibility, integrity, and corruption status. If any verification fails, subsequent processing is terminated and an error message is returned, thus preventing invalid files from consuming computing resources. For valid files that pass verification, they are automatically assigned to the corresponding partition in the distributed storage cluster based on the original file's modality and format characteristics. Simultaneously, index entries associated with the modality attributes are generated, creating a mapping between the storage structure and retrieval requirements. This application effectively isolates abnormal files through its verification mechanism, allowing the distributed storage to focus on storing high-quality data, thus forming a processing flow from data admission to storage optimization.

[0062] In one embodiment, mapping multiple text segments to the same vector space includes: By using a Transformer-based pre-trained language model, multiple text segment representations are converted into fixed-dimensional vectors to obtain an initial vector set. Modality alignment is then performed on the initial vector set to ensure that text segment vectors from different modalities have semantic similarity in the vector space.

[0063] Among them, the Transformer-based pre-trained language model is a deep neural network model based on the self-attention mechanism. Specifically, it can be a pre-trained model such as BERT, RoBERTa, or ALBERT. The Transformer-based pre-trained language model extracts context-related high-order semantic features from the multimodal transformed text fragments, ensuring that different modal texts retain a consistent semantic structure during the vectorization process. Modality alignment eliminates the semantic gap introduced during modality transformation by adjusting the relative positions in vector space, forcing semantically similar content to converge closely in terms of vector distance.

[0064] Specifically, a pre-trained language model based on Transformer converts text segments into fixed-dimensional vectors, leveraging the Transformer architecture's capabilities in deep semantic modeling to extract high-order features. Modality alignment is then used to dynamically optimize the relative positional relationships in the vector space, ensuring that semantically similar content from different modalities converges tightly in terms of vector distance.

[0065] In one embodiment, matching the request keyword with multiple text segments corresponding to the original file in the knowledge base to obtain a first matching result includes: constructing a first set based on the request keyword; constructing a second set based on the text segments in the knowledge base; calculating the intersection of the first set and the second set; determining whether a text segment contains at least one request keyword based on the intersection; if it does, determining the text segments in the intersection as candidate text segments; calculating the keyword matching degree of the candidate text segments; and sorting the candidate text segments based on the keyword matching degree to obtain a sorted first matching result.

[0066] In one embodiment, the keyword matching degree of the candidate text segment is calculated using the following formula:

[0067] in, Indicates keyword matching degree, Q represents the first set, and D represents the second set. Indicates the number of elements in the set;

[0068] Where t represents a specific keyword, d represents a candidate text segment, TF(t,d) represents the frequency of word t in text segment d, and N represents the total number of candidate text segments in the knowledge base. This indicates the number of candidate text segments in the knowledge base that contain the keyword t.

[0069] In summary, this application performs a hybrid retrieval operation on a pre-constructed knowledge base based on the received user's retrieval request, obtaining a first matching result and a second matching result that match the retrieval request; the first and second matching results are then fused to obtain the retrieval result that matches the retrieval request. By performing a hybrid retrieval operation and fusing multimodal matching results, this application effectively solves the problem of inaccurate retrieval caused by the heterogeneity of multimodal data, and has the advantages of effectively integrating multimodal data, achieving accurate cross-modal retrieval, and improving the accuracy and comprehensiveness of retrieval results.

[0070] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0071] Figure 5 This is a block diagram illustrating a multimodal data retrieval device according to an exemplary embodiment of this application. Figure 5 As shown, this application provides a retrieval device based on multimodal data. The retrieval device includes: The request receiving module 510 is used to receive user search requests; The retrieval matching module 520 is used to perform a hybrid retrieval operation in the constructed knowledge base based on the retrieval request, and obtain a first matching result and a second matching result that match the retrieval request. The result fusion module 530 is used to fuse the first matching result and the second matching result to obtain the retrieval result that matches the retrieval request.

[0072] It should be noted that the retrieval device based on multimodal data provided in the above embodiments and the retrieval method based on multimodal data provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the retrieval device based on multimodal data provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0073] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, enable the electronic device to implement the multimodal data retrieval method provided in the above embodiments.

[0074] Figure 6 A schematic diagram of a computer system suitable for implementing the memory of embodiments of this application is shown. It should be noted that... Figure 6 The computer system with the memory shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0075] like Figure 6 As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on a program stored in Read-Only Memory (ROM) 602 or a program loaded from storage into Random Access Memory (RAM) 603. The RAM also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0076] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0077] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the aforementioned multimodal data-based retrieval method. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, it performs various functions defined in the system of this application.

[0078] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM) 603, read-only memory (ROM) 602, erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block or combination of blocks in a block diagram or flowchart may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0080] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0081] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the aforementioned retrieval method based on multimodal data. This computer-readable storage medium may be included in the memory described in the above embodiments, or it may exist independently and not incorporated into that memory.

[0082] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the multimodal data-based retrieval method provided in the various embodiments described above.

[0083] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A retrieval method based on multimodal data, characterized in that, The retrieval method includes: Receive user search requests; Based on the search request, a hybrid search operation is performed in the constructed knowledge base to obtain a first matching result and a second matching result that match the search request. The first matching result and the second matching result are merged to obtain the search result that matches the search request; The search results, the user query text corresponding to the search request, and the modal labels of the user query text are input into a pre-trained multimodal large model. The multimodal large model obtains a text matching score based on topic matching degree, detail matching degree, and user intent matching degree, and the search result with the highest text matching score is taken as the target search result.

2. The multimodal data retrieval method according to claim 1, characterized in that, The step of performing a hybrid search operation in the constructed knowledge base based on the search request includes: Keyword extraction is performed on the user's search request to obtain the requested keywords; The request keywords are matched with multiple text segments corresponding to the original files in the knowledge base to obtain the first matching result; The user's search request is vectorized to obtain a vectorized representation of the search request; The vectorized representation of the retrieval request is matched with the vectorized representation of the original file in the knowledge base to obtain a second matching result.

3. The multimodal data retrieval method according to claim 1 or 2, characterized in that, The step of obtaining search results matching the search request based on the first matching result and the second matching result includes: The first matching result and the second matching result are weighted and fused to calculate the comprehensive score corresponding to each original file; wherein, the weight of the first matching result and the weight of the second matching result are determined by the intent recognition result of the intent recognition of the search request; The aggregated data is sorted to determine the search results.

4. The multimodal data retrieval method according to claim 1, characterized in that, The construction of the knowledge base includes: Obtain the original files for multiple modalities; Information extraction and modality conversion are performed on the original files of the various modalities, and the original files of the various modalities are converted into multiple text segment representations; wherein, each original file of a modality corresponds to at least one text segment representation; The multiple text segments are mapped to the same vector space to generate a vectorized representation; A knowledge base is constructed based on the multiple text segment representations and the vectorized representations.

5. The multimodal data retrieval method according to claim 4, characterized in that, The construction of the knowledge base also includes: The original files of the various modalities are verified, including determining whether the file format is within the preset supported range, whether the file is complete, and whether the file is corrupted. If the original file fails verification, an error message will be returned to the user. If the original file passes the verification, a distributed storage architecture is used to classify and store the original file, and an index is built according to the file modality type and format.

6. The retrieval method based on multimodal data according to claim 4, characterized in that, Mapping the multiple text segments to the same vector space includes: By using a pre-trained language model based on Transformer, the multiple text segment representations are converted into vectors of fixed dimensions to obtain an initial vector set; Modal alignment is performed on the initial vector set to ensure that text segment vectors from different modalities have semantic similarity in the vector space.

7. The retrieval method based on multimodal data according to claim 4, characterized in that, The original files of the multimodal system include: visual files, audio files, video files, text files, and table files.

8. The retrieval method based on multimodal data according to claim 2, characterized in that, The step of matching the request keywords with multiple text segments corresponding to the original files in the knowledge base to obtain a first matching result includes: Based on the requested keywords, construct a first set; Construct a second set based on the text segments in the knowledge base; Calculate the intersection of the first set and the second set; Based on the intersection, determine whether the text segment contains at least one of the requested keywords; If included, the text segments in the intersection are determined as candidate text segments; Calculate the keyword matching degree of the candidate text segments; The candidate text segments are sorted based on the keyword matching degree to obtain the first matching result after sorting.

9. The retrieval method based on multimodal data according to claim 8, characterized in that, The method for calculating the keyword matching degree of the candidate text segment is as follows: in, Indicates keyword matching degree, Q represents the first set, and D represents the second set. Indicates the number of elements in the set; Where t represents a specific keyword, d represents a candidate text segment, TF(t,d) represents the frequency of word t in text segment d, and N represents the total number of candidate text segments in the knowledge base. This indicates the number of candidate text segments in the knowledge base that contain the keyword t.

10. A retrieval device based on multimodal data, characterized in that, The retrieval device includes: The request receiving module is used to receive users' search requests; The retrieval and matching module is used to perform a hybrid retrieval operation in the constructed knowledge base based on the retrieval request, and obtain a first matching result and a second matching result that match the retrieval request. The result fusion module is used to fuse the first matching result and the second matching result to obtain the retrieval result that matches the retrieval request; The text matching module is used to input the search results, the user query text corresponding to the search request, and the modal labels of the user query text into a pre-trained multimodal large model. The multimodal large model obtains a text matching score based on topic matching degree, detail matching degree, and user intent matching degree, and takes the search result with the highest text matching score as the target search result.

11. A retrieval device based on multimodal data, characterized in that, include: One or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, cause the multimodal data-based retrieval device to implement the steps of the multimodal data-based retrieval method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the steps of the multimodal data retrieval method according to any one of claims 1 to 9.

Images