Multi-modal enhanced retrieval method and system

Abstract

The invention discloses a multi-modal enhanced retrieval method and system, and the method comprises the steps: obtaining heterogeneous data, and executing data processing to obtain a meta-information set; encoding the meta-information set into a high-dimensional meta-information vector, and constructing a vector index; receiving a query statement of a user, calculating the similarity between the query statement and the meta-information vector, and obtaining a meta-information query result; and calling a large language model based on the obtained meta-information to generate a summary result, and merging and outputting the summary result and heterogeneous data corresponding to the meta-information. According to the method, multi-modal data is processed into unified semantic vector representation, efficient vectorization indexing and semantic generation and expansion capacity of a large-scale language model are combined, high-value information can be rapidly extracted from multi-source heterogeneous data, and real-time, cross-modal and high-precision retrieval requirements are supported. A reordering and traceability mechanism introduced in the scheme further enhances the correlation and reliability of the information, and provides comprehensive and accurate knowledge support for the generation of query results.

Description

Technical Field

[0001] The present invention relates to the technical field of data retrieval and generation, and in particular to a multimodal enhanced retrieval method and system. Background Art

[0002] Traditional retrieval methods are often limited to a single information resource or retrieval paradigm, such as relying solely on text data for querying. This single-dimensional dependence leads to the one-sidedness of the retrieval results, and fails to fully integrate multimodal information such as images, audio, and video, resulting in obvious limitations in the accuracy and efficiency of retrieval. In a specific application scenario, this traditional retrieval method does not perform well in terms of data integration, security protection and confidentiality, and adaptability to complex application environments. In terms of multimodal retrieval, some studies have proposed to use the CLIP model to conduct experiments through large-scale image data sets in related scenarios, and design and develop multimodal search tools with the functions of searching images by text and searching images by images. Multimodal retrieval based on knowledge graphs is also an important application direction of information retrieval. Compared with open domain knowledge graph question-and-answer systems, knowledge question-and-answer systems in certain specific application scenarios have the characteristics of limited question forms, lack of open knowledge sources, and the need to deeply utilize special knowledge reasoning. Some scholars have applied CBIR technology to image knowledge data processing, which can quickly pick out relevant images from large-scale image knowledge data sources based on existing specific images to obtain special knowledge information. However, how to quickly retrieve the required knowledge data from large-scale image knowledge data is a challenge currently facing CBIR. Summary of the invention

[0003] In view of this, the present invention proposes a multimodal enhanced retrieval method and system, which can realize cross-modal, high-precision, and fast-response information retrieval and analysis through semantic expression and efficient vectorized retrieval technology, targeting the processing capabilities of complex information in specific fields. The present invention provides the following technical solutions:

[0004] A multimodal enhanced retrieval method, the method comprising: acquiring heterogeneous data and performing data processing to obtain a meta information set; encoding the meta information set into a high-dimensional meta information vector and constructing a vector index; receiving a user's query statement, calculating the similarity between the query statement and the meta information vector, and obtaining a meta information query result; calling a large language model based on the obtained meta information to generate a summary result, and merging and outputting the summary result and the heterogeneous data corresponding to the meta information.

[0005] Optionally, the method of acquiring heterogeneous data and performing data processing to obtain a set of meta-information includes: acquiring a visual training data set and training a visual feature extraction model based on a ResNet-50 network; identifying data types of heterogeneous data, the data types including pictures, videos, audio, and text; calling an optical flow method to calculate motion information in picture data and video data, and extracting key frames; calling a visual feature extraction model to identify target tags in key frames; and storing the key frames and target tags as key frame meta-information.

[0006] Optionally, the method also includes: separating audio data from the video data and storing it as separated audio data; performing noise reduction optimization on the separated audio data and audio data, and calling a speech recognition component to convert them into recognized text data; performing preprocessing on the recognized text data and text data; inputting the preprocessed text into a large language model to generate video summary text and video keywords corresponding to the video data, audio summary text and audio keywords corresponding to the audio data, and summary text and text keywords corresponding to the text data; constructing video meta information based on the video summary text and video keywords, constructing audio meta information based on the audio summary text and audio keywords, and constructing text meta information based on the summary text and text keywords.

[0007] Optionally, the method also includes: identifying the timestamp of the key frame and inserting the timestamp into the key frame meta-information; identifying the transcribed text of the separated audio based on the speech recognition component, identifying the name of the video data corresponding to the separated audio, and inserting the video name and the transcribed text into the video meta-information; identifying the transcribed text of the audio data based on the speech recognition component, identifying the name of the audio data, and inserting the audio name and the transcribed text into the audio meta-information; identifying the name of the text data, and inserting the text name and the original text of the text data into the text meta-information; constructing a meta-information set based on the video meta-information, audio meta-information and text meta-information.

[0008] Optionally, the method for receiving a user's query statement, calculating its similarity with a meta-information vector, and obtaining a meta-information query result includes: receiving a user's query statement and performing vectorization processing to obtain a query vector; calculating the similarity between the query vector and the meta-information vector to obtain multiple meta-information vectors similar to the query vector; sorting the queried meta-information vectors according to their relevance to the query vector based on a rerank model; and obtaining the most similar meta-information vector and the corresponding meta-information based on the relevance sorting.

[0009] Optionally, the method of calling a large language model based on the obtained meta-information to generate a summary result, and merging and outputting the summary result and heterogeneous data corresponding to the meta-information includes: calling a large language model and a prompt template of a corresponding scene based on the most similar meta-information to generate a summary result; inputting the summary result into a quality assessment model of the corresponding scene to output a final query result; mapping the meta-information back to the heterogeneous data through a meta-information tracing component based on the final query result; and outputting the final query result and the corresponding heterogeneous data.

[0010] Optionally, the method of obtaining a visual training data set and training a visual feature extraction model based on a ResNet-50 network includes: constructing a visual training data set for a target application scenario, wherein the visual training data set includes image data of the target application scenario; calling an ImageNet data set and pre-training a ResNet-50 backbone network; calling a COCO data set in conjunction with a visual training data set to train a YOLOv5 detection branch, and calling a Places365 data set in conjunction with a visual training data set to train a scene classification branch; wherein the training process uses a multi-task loss function:

[0011] L total =α*L detection +β*L classification， Among them, L total is the total loss function, L detection is the loss of the target detection task, L classification is the loss of the scene classification task, α and β are weight coefficients used to balance the training tasks; the Adam optimizer is used, and the initial learning rate is set to 1e -4 , and the cosine annealing strategy is used to adjust the learning rate.

[0012] The present invention further discloses a multi-modal enhanced retrieval system, comprising:

[0013] A meta-information acquisition module is used to acquire heterogeneous data and perform data processing to obtain a meta-information set;

[0014] An index building module is used to encode the meta-information set into a high-dimensional meta-information vector and build a vector index;

[0015] The meta-information query module is used to receive the user's query statement, calculate the similarity between the query statement and the meta-information vector, and obtain the meta-information query result;

[0016] The output module is used to call the large language model to generate summary results based on the obtained meta-information, and merge the output summary results and the heterogeneous data corresponding to the meta-information.

[0017] The present invention further discloses a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the multimodal enhanced retrieval method described above is implemented.

[0018] The present invention further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multimodal enhanced retrieval method when executing the program.

[0019] The technical solution according to the present invention realizes accurate information retrieval and intelligent knowledge analysis by processing multimodal data into a unified semantic vector representation, combining efficient vectorized indexing technology with the semantic generation and expansion capabilities of large-scale language models. In specific application fields, the integration and processing capabilities of complex data are significantly improved, enabling the system to quickly extract high-value information from multi-source heterogeneous data and support real-time, cross-modal, and high-precision retrieval requirements. At the same time, the reordering and traceability mechanisms introduced in the solution further enhance the relevance and reliability of information, providing comprehensive and accurate knowledge support for the generation of query results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] For the purpose of illustration and not limitation, the present invention is now described in conjunction with the embodiments of the present invention and the accompanying drawings, in which:

[0021] Figure 1 is a flowchart of a multimodal enhanced retrieval method in an embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of the composition of a multimodal enhanced retrieval system in an embodiment of the present invention;

[0023] Figure 3 is a schematic structural diagram of an electronic device in an embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of a search process in an embodiment of the present invention;

[0025] Figure 5 It is a schematic diagram of the index construction process in an implementation manner of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the implementation mode of the present application. Obviously, the described implementation mode is only a part of the implementation mode of the present application, not all the implementation modes. Based on the implementation mode in the present application, all other implementation modes obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0028] refer to Figure 1 , Figure 4 and Figure 5 This embodiment discloses a multimodal enhanced retrieval method, which includes:

[0029] S100: Obtain heterogeneous data and perform data processing to obtain a set of meta-information. The heterogeneous data in this embodiment includes but is not limited to text, video, audio and picture.

[0030] This implementation method takes the scene application in a special field as an example, and the method for obtaining the meta-information set is as follows: collect image data containing a variety of scenes and target objects from the special field, and define it as a visual training data set, mainly including weapons and equipment, vehicles, special personnel, terrain environment (such as deserts, jungles), etc., to build a special visual training data set (MiltiData). The pictures in the data set are labeled, including target location, target category, and scene label (such as city, wilderness, etc.). Call the ImageNet data set to pre-train the ResNet-50 backbone network to extract common visual features, call the COCO data set and the MiltiData data set to train the YOLOv5 target detection branch for extracting target features in key frames, and call the Places365 data set and the above-constructed visual training data set to train the scene classification branch for identifying scene features. Among them, the training process adopts a multi-task loss function:

[0031] L total =α*L detection +β*L classification , where L total is the total loss function, L detection is the loss of the target detection task, L classification is the loss of the scene classification task, α and β are weight coefficients used to balance the training tasks; the Adam optimizer is used, and the initial learning rate is set to 1e -4 , and the cosine annealing strategy is used to adjust the learning rate. This ensures that the model converges stably during the optimization process, and the final output MiltiVision-X model can efficiently identify special scenes and target features.

[0032] Furthermore, for text, video, audio and image data, we first identify the specific classification of the input heterogeneous data. When inputting a data set, it may contain multiple data types, and different types of data are processed separately. For video data and image data, we assign them to the visual stream processing module, audio data to the audio stream processing module, and text data to the text stream processing module.

[0033] For video data, the optical flow method is used to calculate the pixel motion information between adjacent frames. The optical flow method can effectively detect the motion change area between frames, thereby locating the significant frames in the video. In the picture sequence, the same method is used to perform motion analysis on the continuous pictures in the picture group, and identify the pictures with significant motion as key frame candidates. A motion intensity threshold is set, and frames exceeding the threshold are marked as key frames to ensure that the extracted key frames contain the most informative scenes in the video. For static scenes, secondary screening is performed in combination with inter-frame semantic features to ensure that they are included in the key frame set.

[0034] Call the trained MiltiVision-X model to perform target detection on the keyframes. Use the YOLOv5 branch to identify the target object in the keyframe and output the target category, location and confidence. For example, typical target labels include vehicles, personnel, weapons and equipment, etc. Call the scene classification branch of the MiltiVision-X model to identify the scene features of the keyframe (such as "desert", "jungle", "city", etc.) to provide scene semantic information for subsequent retrieval. For each keyframe, output a set of target labels and scene labels as its structured semantic features.

[0035] Furthermore, the image file path, timestamp, target detection label, scene classification label and other information of the key frame are combined into the meta information of the key frame. The meta information is stored in JSON format for easy subsequent indexing and retrieval.

[0036] For audio data, since video data also contains audio, it is necessary to separate the audio from the video data and name the audio file according to the video file name and timestamp for subsequent associated processing. Use audio signal processing algorithms (such as spectral subtraction or adaptive filters) to reduce the noise of the audio and remove unnecessary audio signals such as background noise and environmental interference. For audio in special scenarios (such as conversations and command communications), apply specific noise reduction models to retain speech clarity, call the speech recognition component, and convert the optimized audio data into text. The recognition process supports multi-language mode to ensure that audio data in different languages ​​can be accurately processed. The generated text includes recognized text data and its corresponding timestamp information. Delete meaningless symbols (such as redundant punctuation and blank characters) from the generated recognized text data. Unify the time format and naming specifications for subsequent analysis and indexing. Segment long texts according to semantics to ensure that each text segment fits the input length limit of the language model. Input the recognized text data and the original text data into a large language model (such as the GPT series model) to extract summaries and keywords using its semantic generation capabilities. Call a specific Prompt template to guide the model to generate a summary, ensuring that the output content is highly readable and accurate. For the audio transcription text, the summary content needs to summarize the core semantics and annotate the key events or instructions. Based on the model's few-shot learning ability, a list of keywords related to the summary semantics is generated through the Prompt template. Keywords include situational words (such as "meeting", "exercise") and entity words (such as "tank", "commander"). Mark and annotate the proper nouns in the text (such as special terms, place names) to facilitate subsequent model analysis. Finally, the preprocessed text is input into the large language model to generate video summary text and video keywords corresponding to the video data, audio summary text and audio keywords corresponding to the audio data, and summary text and text keywords corresponding to the text data. Video meta information is generated using the video summary text and video keywords, with the video file path and key frame timestamps attached. The structured representation of video meta information includes the core events of the video, the targets involved, and scene labels. Audio meta information is generated based on the audio summary text and audio keywords, with the audio file path and the corresponding time range attached. The audio meta information focuses on describing the directive information in the audio. Generate text meta information using summary text and text keywords, and attach text file path and original text fragment. Video meta information, audio meta information and text meta information are stored in a unified format (such as JSON format).

[0037] For video files, in combination with video decoding tools (such as OpenCV) or dedicated video processing libraries, extract the timestamps of key frames on the video timeline, format the timestamps into standard time formats (such as HH:MM:SS) to ensure readability and consistency, add a new field "timestamp" in the key frame meta information, store it together with the image path, target detection label and scene classification label of the key frame, call the speech recognition component (such as DeepSpeech or Whisper model), convert the separated audio data into text, and generate segmented transcribed text (such as dialogue fragments, command records, etc.) according to the timeline. The transcription result includes the text content and the corresponding start and end time. According to the naming rules of the separated audio files, parse the corresponding video name, add the fields "audio transcribed text" and "video name" in the video meta information, and associate the transcribed text with the timestamp. Call the speech recognition component to transcribe the individual audio files globally, generate a complete transcribed text, and annotate important keywords (such as special terms, action instructions), extract the name of the audio file, and record it as the identification field of the audio meta information, and store the audio file name and transcribed text in the audio meta information. Parse the path or file name of the text file, extract the file name and store it as the identification field of the text meta information, load the original text file content, and perform necessary preprocessing (such as removing redundant symbols and unifying the format), and store the text file name and original text content in the text meta information. Integrate the video meta information, audio meta information and text meta information into a meta information set in a unified meta information format.

[0038] S200: Encode the meta information set into a high-dimensional meta information vector and construct a vector index.

[0039] Use encoders based on large-scale language models (such as BERT, CLIP or GPT) and other specialized models to convert multimodal meta information into high-dimensional vectors. Specifically, select an appropriate model based on the type of meta information, for example:

[0040] Text meta information: Use the pre-trained BERT model or a large language model to semantically encode text summaries and keywords.

[0041] Audio meta information: Use Wav2Vec or other audio feature extraction models to vectorize audio transcription text and keywords.

[0042] Video meta-information: Using the CLIP model, the image features of video key frames and the video summary text are encoded into a joint vector representation.

[0043] A high-dimensional vector is generated for each record in the metadata set, and vectors of different modes are combined into a unified high-dimensional representation through vector concatenation or dimensionality reduction technology.

[0044] After obtaining the meta information vector, the meta information vector and its associated data are stored in the vector database. This implementation uses the Milvus database and establishes a basic index table. The K-means or HNSW method is used to cluster the vectors and generate a partition index to improve the retrieval efficiency. Finally, the meta information vector is mapped one-to-one with the original record of the meta information to construct a vector index.

[0045] S300: Receive the user's query statement, calculate its similarity with the meta information vector, and obtain the meta information query result. The system receives the query statement input by the user, parses the query intent, extracts key phrases and context information, and uses a large-scale language model (such as BERT, GPT or CLIP text branch) to vectorize and encode the query statement to obtain a query vector.

[0046] Retrieval is performed by calculating the similarity between the query vector and the meta-information vector. This embodiment exemplarily uses cosine similarity to calculate the similarity between the query vector and the meta-information vector, and obtains multiple meta-information vectors similar to the query vector through the calculated similarity. These meta-information vectors may contain meta-information vectors with low relevance to the query intent. The rerank model performs secondary sorting of the preliminary results through more refined semantic analysis. Specifically, a Transformer-based sorting model is used to input the query statement and each meta-information combination into the rerank model, and the results are re-sorted from high to low according to the relevance score output by the model to obtain the most similar meta-information vector and the corresponding meta-information.

[0047] S400: Based on the obtained meta-information, the large language model is called to generate a summary result, and the summary result and the heterogeneous data corresponding to the meta-information are merged and output. Specifically, according to the specific special scenario to which the meta-information belongs (such as reconnaissance knowledge scenario, combat command scenario, special equipment management scenario, etc.), a pre-designed prompt template that matches it is selected. These prompt templates are optimized with a large amount of special data and actual application scenarios, and can guide the large language model to generate summary content that meets the needs of the scenario. For example, in the reconnaissance knowledge scenario, the prompt template may require the large language model to focus on information such as the location, activity patterns, and potential threats of the target; in the combat command scenario, the template will focus on summarizing key information such as battlefield situation, force deployment, and execution of combat plans. The most similar meta-information (including video meta-information, audio meta-information, and text meta-information) obtained in step S300 is formatted according to the requirements of the prompt template to form a format suitable for the input of the large language model. Then, the processed meta-information is input into the locally deployed large language model, and the model generates a preliminary summary result based on its powerful semantic understanding and text generation capabilities. For example, for a video containing battlefield keyframe meta-information (such as identified special targets, scene environment, etc.), audio transcription text meta-information (such as combat instructions, communication content, etc.), and related text report meta-information (such as combat mission description, force deployment, etc.), the large language model may generate a preliminary summary such as "On the battlefield of [specific location], [a certain team] is performing [combat mission name], and the enemy's [special target type and number] have been discovered. The current battlefield situation is [situation description], and based on the audio communication, [important instructions or knowledge] can be known."

[0048] Furthermore, the summary result is input into the quality assessment model of the corresponding scenario, wherein the quality assessment model of the corresponding scenario is obtained by training based on a large number of manually annotated high-quality summary samples and low-quality summary samples. During the training process, the model learns the correlation between the key features (such as information completeness, accuracy, logic, language expression standardization, etc.) in the summary result and the quality label (high quality or low quality). For example, using a deep learning model architecture such as a convolutional neural network (CNN) or a recurrent neural network (RNN), the summary result text is vectorized, and then feature extraction and classification are performed through a multi-layer neural network to judge the quality of the summary result. The preliminary summary result generated by the large language model is input into the quality assessment model, and the model evaluates the summary result according to the judgment criteria obtained by training. If the summary result is judged to be of high quality (for example, the information is complete, accurate and the logic is clear), it is directly output as the final query result; if the quality of the summary result is not high, the large language model is re-prompted according to the feedback information provided by the evaluation model (such as pointing out the missing key information, the existing logical errors, etc.), and it is required to generate an improved summary until the final query result that meets the quality requirements is obtained.

[0049] After obtaining the final query result, the meta information traceability component is called to map the meta information back to the heterogeneous data. Among them, the meta information traceability component maintains the mapping relationship between the meta information and the original heterogeneous data. In the data processing stage, when constructing the meta information, the associated information with the original data is also recorded, such as the correspondence between the timestamp in the key frame meta information and the original video, the association between the video name in the audio meta information and the original video, and the correspondence between the document name in the text meta information and the original text. With these associated information, when receiving the final query result (including meta information), the meta information traceability component can quickly locate the corresponding original heterogeneous data. According to the meta information in the final query result, the meta information traceability component searches for the storage location or identifier of the corresponding original heterogeneous data in the mapping table it maintains. For example, if the meta information in the final query result contains the meta information of a certain video key frame, the traceability component finds the storage path of the video in the video meta information mapping table through the timestamp information therein, thereby obtaining the original video data; for text meta information, the corresponding original text file is found through the document name, etc. In this way, accurate mapping from meta information to original heterogeneous data is achieved.

[0050] After identifying the mapped meta-information, the final query results obtained after quality assessment are integrated with the corresponding heterogeneous data obtained through the meta-information traceability component. For example, the summary report is combined with the relevant original reconnaissance video, audio recording, combat report text, etc. to form a complete retrieval output package, and the integrated results are presented to the user in an intuitive and easy-to-understand way. Different output formats can be used according to user needs and usage scenarios, such as displaying a comprehensive interface containing a video playback window, audio playback button, and text summary display area on the large screen of the special command center, or generating detailed electronic documents for special analysts to further study and analyze, ensuring that users can fully and deeply understand the retrieved multimodal special information and provide strong support for special decision-making.

[0051] In summary, this multimodal enhanced retrieval technology solution exemplifies the difficulties of multimodal information retrieval in special fields, and achieves efficient and accurate retrieval through a series of innovative means. Similarly, in other application scenarios, in terms of technical solutions, after obtaining heterogeneous data, the optical flow method is used to extract key frames for the visual stream, and the MiltiVision-X model is trained based on a custom data set for target detection and scene classification, and keywords are expanded through a large language model; for the audio stream, it is converted into text after noise reduction and other processing; the text stream is pre-processed by cleaning and other pre-processing, and then the large language model is used to extract keywords and generate summaries, and finally a meta-information set is constructed and encoded as a vector index. During retrieval, the query statement is first vectorized and calculated to measure the similarity with the meta-information vector, and the large language model is called to generate a summary after sorting by the rerank model. After quality assessment, it is mapped back to the heterogeneous data output through the traceability component. Its technical effect is remarkable. It uses the powerful semantic understanding ability of the large language model to achieve efficient integration and deep understanding of multimodal special data, overcome the limitations of traditional technology, and improve the efficiency of cross-modal correlation analysis and key knowledge extraction. It combines vectorized retrieval technology to achieve high-precision and fast retrieval, and uses multi-language processing capabilities to enhance knowledge processing in different languages. It ensures the professionalism of reports through file tracing, and ensures data security through offline deployment, providing strong support for special decision-making.

[0052] refer to Figure 2 This embodiment further discloses a multimodal enhanced retrieval system, including:

[0053] The meta information acquisition module 21 is used to acquire heterogeneous data and perform data processing to obtain a meta information set; it is also used to acquire a visual training data set and train a visual feature extraction model based on a ResNet-50 network, including: constructing a visual training data set for a target application scenario, wherein the visual training data set includes image data of the target application scenario; calling an ImageNet data set and pre-training a ResNet-50 backbone network; calling a COCO data set in conjunction with a visual training data set to train a YOLOv5 detection branch, and calling a Places365 data set in conjunction with a visual training data set to train a scene classification branch; wherein the training process uses a multi-task loss function: L total =α*L detection +β*L classification， Among them, L total is the total loss function, L detection is the loss of the target detection task, L classification is the loss of the scene classification task, α and β are weight coefficients used to balance the training tasks; the Adam optimizer is used, and the initial learning rate is set to 1e -4 , and use the cosine annealing strategy to adjust the learning rate; identify the data type of heterogeneous data, the data type includes pictures, videos, audio and text; call the optical flow method to calculate the motion information in the picture data and video data, and extract key frames; call the visual feature extraction model to identify the target label in the key frame; store the key frame and the target label as key frame meta information. It is further used to separate the audio data in the video data and store it as separated audio data; perform noise reduction optimization on the separated audio data and audio data, and call the speech recognition component to convert it into recognized text data; perform preprocessing on the recognized text data and text data; input the preprocessed text into the large language model to generate video summary text and video keywords corresponding to the video data, audio summary text and audio keywords corresponding to the audio data, and summary text and text keywords corresponding to the text data; construct video meta information based on the video summary text and video keywords, construct audio meta information based on the audio summary text and audio keywords, and construct text meta information based on the summary text and text keywords. It is also used to identify the timestamp of the key frame and insert the timestamp into the key frame meta-information; identify the transcribed text of the separated audio based on the speech recognition component, identify the name of the video data corresponding to the separated audio, and insert the video name and the transcribed text into the video meta-information; identify the transcribed text of the audio data based on the speech recognition component, identify the name of the audio data, and insert the audio name and the transcribed text into the audio meta-information; identify the name of the text data, and insert the text name and the original text of the text data into the text meta-information; and construct a meta-information set based on the video meta-information, the audio meta-information and the text meta-information.

[0054] An index building module 22, used to encode the meta information set into a high-dimensional meta information vector and build a vector index;

[0055] The meta-information query module 23 is used to receive the user's query statement, calculate its similarity with the meta-information vector, and obtain the meta-information query result; it is also used to receive the user's query statement and perform vectorization processing to obtain a query vector; calculate the similarity between the query vector and the meta-information vector to obtain multiple meta-information vectors similar to the query vector; perform relevance sorting on the queried meta-information vector based on the rerank model and the query vector; obtain the most similar meta-information vector and the corresponding meta-information based on the relevance sorting.

[0056] The output module 24 is used to call the large language model based on the obtained meta-information to generate a summary result, and merge the output summary result and the heterogeneous data corresponding to the meta-information; it is also used to call the large language model and the prompt template of the corresponding scene based on the most similar meta-information to generate a summary result; input the summary result into the quality assessment model of the corresponding scene to output the final query result; based on the final query result, map the meta-information back to the heterogeneous data through the meta-information tracing component; output the final query result and the corresponding heterogeneous data.

[0057] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 3 As shown, the electronic device 50 includes: a processor 501 (processor), a memory 502 (memory) and a bus 503;

[0058] The processor 501 and the memory 502 communicate with each other via the bus 503 ; the processor 501 is used to call program instructions in the memory 502 to execute the methods provided by the above-mentioned method implementation methods.

[0059] This embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the methods provided by the above-mentioned method embodiments.

[0060] A person skilled in the art can understand that all or part of the steps for implementing the above-mentioned method implementation method can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium, which, when executed, executes the steps of the above-mentioned method implementation method; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0061] The device implementation described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present implementation scheme. Those of ordinary skill in the art may understand and implement it without creative effort.

[0062] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal enhanced retrieval method, characterized in that: The method comprises: Acquire heterogeneous data and perform data processing to obtain a set of meta-information; Encode the meta-information set into a high-dimensional meta-information vector and construct a vector index; Receive the user's query statement, calculate the similarity between the query statement and the meta-information vector, and obtain the meta-information query result; Based on the obtained meta-information, a large language model is called to generate a summary result, and the output summary result and the heterogeneous data corresponding to the meta-information are merged.

2. The multimodal enhanced retrieval method according to claim 1, characterized in that: The method of acquiring heterogeneous data and performing data processing to obtain a set of meta information includes: Obtain a visual training dataset and train a visual feature extraction model based on the ResNet-50 network; Identify data types of heterogeneous data, including pictures, videos, audio, and text; Use the optical flow method to calculate the motion information in the image data and video data, and extract key frames; Call the visual feature extraction model to identify the target label in the key frame; The keyframes and target labels are stored as keyframe meta information.

3. The multimodal enhanced retrieval method according to claim 2, characterized in that: The method further comprises: Separating audio data from the video data and storing the separated audio data; Perform noise reduction optimization on the separated audio data and the audio data, and call the speech recognition component to convert them into recognized text data; performing preprocessing on the recognition text data and the text data; Inputting the preprocessed text into a large language model to generate video summary text and video keywords corresponding to the video data, audio summary text and audio keywords corresponding to the audio data, and summary text and text keywords corresponding to the text data; Video meta information is constructed based on video summary text and video keywords, audio meta information is constructed based on audio summary text and audio keywords, and text meta information is constructed based on summary text and text keywords.

4. The multimodal enhanced retrieval method according to claim 3, characterized in that: The method further comprises: Identifying a timestamp of a key frame and inserting the timestamp into the key frame meta information; Recognize the transcribed text of the separated audio based on the speech recognition component, recognize the name of the video data corresponding to the separated audio, and insert the video name and the transcribed text into the video meta information; Recognize the transcribed text of the audio data based on the speech recognition component, recognize the name of the audio data, and insert the audio name and the transcribed text into the audio meta information; recognizing a name of text data, and inserting the text name and the original text of the text data into the text meta information; A meta information set is constructed based on video meta information, audio meta information and text meta information.

5. The multimodal enhanced retrieval method according to claim 1, characterized in that: The method of receiving a user's query statement, calculating the similarity between the query statement and the meta information vector, and obtaining the meta information query result includes: Receive the user's query statement and perform vectorization processing to obtain a query vector; Calculating the similarity between the query vector and the meta information vector to obtain a plurality of meta information vectors similar to the query vector; Based on the rerank model, the queried meta-information vector is sorted according to its relevance to the query vector; The most similar meta-information vectors and corresponding meta-information are obtained based on relevance sorting.

6. The multimodal enhanced retrieval method according to claim 1, characterized in that: The method of calling a large language model based on the obtained meta-information to generate a summary result, and merging and outputting the summary result and heterogeneous data corresponding to the meta-information includes: Based on the most similar meta information, the large language model and the prompt template of the corresponding scene are called to generate a summary result; Input the summary results into the quality assessment model of the corresponding scenario to output the final query results; Based on the final query results, the meta information is mapped back to heterogeneous data through the meta information traceability component; Output the final query results and corresponding heterogeneous data.

7. The multimodal enhanced retrieval method according to claim 2, characterized in that: The method of obtaining a visual training data set and training a visual feature extraction model based on a ResNet-50 network includes: Constructing a visual training data set for a target application scenario, wherein the visual training data set includes image data of the target application scenario; Call the ImageNet dataset and pre-train the ResNet-50 backbone network; The COCO dataset is used in conjunction with the visual training dataset to train the YOLOv5 detection branch, and the Places365 dataset is used in conjunction with the visual training dataset to train the scene classification branch. The training process uses a multi-task loss function: L total =α*L detection +β*L classification , where L total is the total loss function, L detection is the loss of the target detection task, L classification is the loss of the scene classification task, α and β are the weight coefficients used to balance the training tasks; The Adam optimizer is used and the initial learning rate is set to 1e -4 , and the cosine annealing strategy is used to adjust the learning rate.

8. A multimodal enhanced retrieval system, characterized in that: include: A meta-information acquisition module is used to acquire heterogeneous data and perform data processing to obtain a meta-information set; An index building module is used to encode the meta-information set into a high-dimensional meta-information vector and build a vector index; The meta-information query module is used to receive the user's query statement, calculate the similarity between the query statement and the meta-information vector, and obtain the meta-information query result; The output module is used to call the large language model to generate summary results based on the obtained meta-information, and merge the output summary results and the heterogeneous data corresponding to the meta-information.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 5 is implemented.

Images