Media file querying and calling method and system based on AI dialogue scene

Abstract

The invention discloses a media file query and retrieval method and system based on an AI dialogue scene, the method combines a mixed mode of vector retrieval and keyword retrieval, realizes accurate retrieval of cross-modal data, and can process various media files including texts, pictures, videos, PPT and the like. According to the method, a double-round scoring mechanism of a large language model (LLM) is combined, firstly, the relevancy of file content is scored, then secondary scoring is carried out according to the matching degree of a file title and user query, and it is ensured that the file most relevant to query is called. In addition, the system can further utilize a user feedback mechanism to continuously optimize the retrieval precision, and the user experience is improved. Compared with the prior art, the method has the advantages that the accuracy and the efficiency of multi-modal data retrieval are remarkably improved, the limitation of a traditional AI dialogue system during multimedia file processing is solved, and the method is suitable for intelligent dialogue application scenes needing to quickly and accurately obtain various types of media files.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a media file query and retrieval method and system based on an AI dialogue scenario. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, especially in the fields of natural language processing (NLP) and machine learning, AI has become an indispensable tool in many industries. In multimodal AI systems, in addition to traditional text data, the processing and utilization of unstructured data such as images, audio, and video has become a hot topic of research. One of the core challenges of multimodal systems is how to effectively process and integrate information from different data sources, especially when information retrieval is required through natural language dialogue, how to accurately retrieve and retrieve relevant files from a database containing multiple types of media files. In real applications, traditional text retrieval technology has been able to process structured text information, but when it comes to unstructured data (such as images, videos, audio, PPT, etc.), traditional retrieval technology is often unable to cope with it. Although LLM (Large Language Model) allows users to conduct interactive questions and answers in natural language and retrieve content within a specified range (knowledge base or database) through RAG (Retrieval Augmented Generation) technology, the data types it targets are mainly text or tables. This requires the system to not only be able to process the user's natural language input, but also to be able to efficiently retrieve and return relevant content across different types of data such as text, images, and videos.

[0003] Existing multimodal retrieval technologies face many challenges when processing databases containing multi-type media files. First, traditional text retrieval methods are usually unable to effectively process and understand non-text information such as images and videos, which makes the accuracy and precision of retrieval results fail to meet user expectations when facing complex queries. Secondly, most current retrieval systems rely on a single data type for retrieval and lack effective fusion of cross-modal information. In multimedia databases, information such as text, images, and videos are usually interrelated, but the existing technology fails to fully realize the intelligent association and understanding between different data types, resulting in low retrieval accuracy and inaccurate results. Furthermore, existing multimodal retrieval systems often rely on keywords, tags, or simple vector retrieval methods, which are inefficient when processing large-scale data, have a long response time, and cannot adapt well to the retrieval needs of different types of media files. Overall, the existing technology has failed to provide an efficient and accurate solution to handle complex retrieval requirements involving multimodal data, especially in AI dialogue scenarios, where users often want to quickly and accurately obtain various types of media files related to their queries through natural language, but the limitations of existing systems lead to problems such as low retrieval accuracy, slow response, and poor user experience. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a media file query and retrieval method and system based on AI dialogue scenarios, which aims to retrieve specified files from a database containing multiple types of media files through natural language dialogue.

[0005] The technical solution provided by the present invention is specifically as follows: A media file query and retrieval method based on an AI dialogue scenario, comprising the steps of: S1. Media file preprocessing and data annotation: Read and classify each media file and extract text data from each media file; Extract keywords from the extracted text data to serve as content tags for each media file; Store the annotated media files and extracted text data in the cloud or local server; S2. Media file cleaning and embedding processing: Convert the extracted text data into a structured text format; Cut the text data in a unified format into several structured text blocks; The text blocks are converted into text embedding vectors respectively through the text embedding model; Store the text embedding vector of each text block, and build an index of the corresponding text data based on the text embedding vector; S3. Retrieve and recall based on user query: Understand and optimize the user's query intent and obtain the user's query text; Retrieve text blocks related to the user query text from the index; Re-ranking the retrieved text blocks by relevance to obtain re-ranked text blocks; S4. Large model scoring and result optimization: LLM is used to obtain the similarity between the user query text and the reordered text blocks and the media file titles corresponding to the reordered text blocks, and the preferred text blocks are screened; S5. Full database matching and query retrieval: Obtaining a media file corresponding to the selected text block, which is recorded as the circled target media file; According to the names, categories and tags of the media files, other media files similar to the identified target media files in the cloud or local server are obtained, thereby forming a target media file list; The media files in the target media file list are sorted by similarity, and the sorted target media file list is fed back to the user, and the media file with the highest relevance to the user's query is retrieved at the same time.

[0006] Furthermore, in step S1, for image files, OCR technology and NLP tools are used to automatically identify and generate text data; for video files, speech-to-text services are used to extract the explanation text of the video content; for PPT files, document parsing is performed and summarized through LLM; for files from which valid text cannot be extracted, brief descriptions are filled in manually.

[0007] Preferably, in step S2, a recursive character text segmentation technique is used to cut the text data into a plurality of structured text blocks according to specific segmentation symbols.

[0008] Preferably, in step S3, if there is ambiguity or subject omission in the user input, the context information will be used to complete or disambiguate it to obtain an optimized and adjusted user query text.

[0009] Furthermore, in step S3, vector retrieval and keyword retrieval are combined to retrieve several text block data related to the user query text from the index; and the retrieval results are re-ranked using a re-ranking model or a "semantic / keyword" weight distribution algorithm.

[0010] Furthermore, in step S4, LLM is used to perform similarity scores between the reordered text blocks and the user query text to obtain text block relevance scores; LLM is used to perform similarity scores between the titles of the media files corresponding to the reordered text blocks and the user query text to obtain media file relevance scores; only text blocks whose text block relevance scores reach a set threshold participate in the media file relevance score, and the text blocks are sorted from high to low according to the total score of the text block relevance score and the media file relevance score to obtain the preferred text blocks.

[0011] Preferably, in step S5, a relevance comparison algorithm is used to obtain other media files in the cloud or local server that are similar to the circled target media file, and the relevance comparison algorithm includes exact matching, Levenshtein edit distance algorithm, Simhash fuzzy algorithm and cosine similarity algorithm.

[0012] Preferably, when the media files in the target media file list are sorted by similarity, the sorting priority is: text block relevance score>media file relevance score>exact match score>other scores.

[0013] A media file query and retrieval system based on the above method includes the following modules: Media file preprocessing module: used to classify media files and extract text data, and extract keywords from the extracted text data to serve as content tags for each media file; it is also used to cut text data into several structured text blocks, and convert the text blocks into text embedding vectors through a text embedding model; Media file database: used to store media files and their corresponding text blocks, text embedding vectors, as well as access paths, tags, and classification information of media files; User interface module: used to receive user input and pass the query content to the query processing module. The user interacts with the system through natural language queries, inputs query questions or specifies tasks; Query processing module: used to perform semantic analysis on the query input by the user, understand the query intent, and output the optimized and adjusted user query text; Multimodal retrieval engine: used to retrieve text blocks related to the user's query text from the media file database as recalled text blocks; LLM scoring and ranking module: LLM is used to obtain the similarity between the user query text and the recalled text block and the media file title corresponding to the recalled text block, and the selected text block is screened; Full-library matching and retrieval module: obtains the media files corresponding to the selected text blocks and other similar media files from the media file database to form a target media file list, and feeds back the sorted target media file list to the user. At the same time, the media files with the highest relevance to the user's query are retrieved for the user to view, download or perform other operations.

[0014] Preferably, the media file query and retrieval system further includes a user feedback module, which is responsible for collecting user feedback during the file retrieval process.

[0015] Compared with the prior art, the present invention significantly improves the accuracy and efficiency of multimodal data retrieval. By combining the hybrid mode of vector retrieval and keyword retrieval, the limitations of single data type retrieval in traditional methods are solved, so that the system can achieve efficient and accurate content retrieval in different types of media files such as text, pictures, and videos. In addition, the present invention effectively integrates multimodal information through the combination of a large language model and a semantic representation model, and can deeply understand and process complex data such as image text and video explanations, making up for the problem of insufficient cross-modal information understanding in the prior art. The double-round scoring mechanism of the large model further improves the relevance of the retrieval results, avoids the problem of false detection and missed detection caused by a single score, and at the same time, the flexible matching algorithm and a variety of similarity calculation methods ensure that personalized and accurate retrieval results can be provided in various business scenarios. The system's efficient index construction and sparse retrieval technology optimize the retrieval efficiency, so that the present invention can respond quickly in large-scale databases to meet the needs of real-time dialogue. In addition, through the continuous optimization mechanism driven by user feedback, the system can continuously improve the retrieval accuracy and user experience, solving the problem of insufficient feedback and difficulty in improvement in the prior art. In summary, the present invention not only achieves a breakthrough in the processing and retrieval accuracy of multimodal data, but also significantly improves the overall performance of the system and user experience through optimization algorithms and continuous feedback mechanisms. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0017] Figure 1 It is a flow chart of a method for querying and retrieving media files provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a media file preprocessing and data annotation process provided by an embodiment of the present invention; Figure 3 It is a schematic diagram of a media file cleaning and embedding process provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of a query recall process provided by an embodiment of the present invention; Figure 5 It is a schematic diagram of a large model scoring and result optimization process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the embodiment of the present invention clearer, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. It should be pointed out that the following detailed descriptions are illustrative and are intended to provide further explanation of the present invention. Unless otherwise specified, each technical and scientific term used in the embodiment has the same meaning as that commonly understood by a person of ordinary skill in the art to which the present invention belongs. Obviously, the embodiments described below are some embodiments of the present invention, not all embodiments. Based on the embodiments in the present invention, other embodiments obtained by a person of ordinary skill in the art without making creative work belong to the scope of protection of the present invention.

[0019] It should be noted that there are three main types of AI models involved in the present invention. The first type is the embedding model (also known as the semantic vector model), which is used to convert natural language into vectors that can be numerically calculated by computers. The implementation example uses bce-embedding-base_v1, which is an open source model. The second type is the LLM model, which is used to interact with users in natural language. The Qwen1.514BChat model and Qwen-turbo-chat are selected in the implementation example. The former is an open source model and is used for local deployment; the latter is a closed source model and is used for online API access. The third type is the reordering model, which is used to reorder the results based on relevance after analyzing the semantics of the recalled content. The bce-reranker-base_v1 is selected in the implementation example, which is an open source model. The three types of models mentioned above have a large number of other models that can be used instead. Therefore, in actual use, a suitable model can be selected according to needs.

[0020] Embodiment 1 This embodiment provides a media file query and retrieval method based on an AI dialogue scenario, which preprocesses and annotates multimedia files, embeds them into a vector database, combines a hybrid retrieval mode of vector retrieval and keyword retrieval, performs a recall test on the embedded blocks in a specific media file library, obtains content blocks with higher recall scores, and uses an LLM model to perform two scorings of content relevance and file title relevance. Combined with the classification information of the target media file in the index, several media files with the highest relevance to the user's question are finally selected, thereby achieving the goal of retrieving media files with high accuracy in AI dialogues so that downstream businesses can play, display and operate media files.

[0021] like Figure 1 As shown, the method mainly includes the following steps: 1. Media file preprocessing and data annotation First, various types of media files (such as pictures, videos, PPT, etc.) are preprocessed and data annotated to ensure the efficiency and accuracy of the subsequent retrieval process. Figure 2 As shown, the specific steps include: Metadata and extractable information processing: Read the metadata information of media files (such as titles, etc.), and extract text from extractable information according to different file types. For image files, use OCR (Optical Character Recognition) technology and NLP tools to automatically identify and generate text annotation content; for videos, use speech-to-text services to extract the explanation text of the video content; for PPT files, use PyMuPDF and other technical solutions to parse documents, extract text content and summarize it through the LLM model.

[0022] Classification and labeling: During the data annotation process, users manually classify media files and use the Jieba word segmentation tool combined with the industry hot word library and machine learning algorithms to extract keywords from the extracted text data as content labels for the files. For files that cannot extract valid text (such as pictures without text or explanation videos without subtitles, etc.), the system allows manual filling of brief descriptions.

[0023] Storage and indexing: The annotated media files and extracted text data are stored in a cloud server or a local server, and their accessible link addresses or server paths are recorded.

[0024] 2. Media file cleaning and embedding After data preprocessing, the next step is to clean, embed and index the media files to ensure that the data can be used efficiently and accurately in the subsequent retrieval process. Figure 3 As shown, the specific steps include: Convert data to text: Convert all text data extracted in step 1 (such as text information extracted from videos and pictures) into structured text formats (such as docx, txt, markdown, etc.), and remove unnecessary symbols and privacy information (mobile phone numbers, private email addresses, etc.).

[0025] Recursive text segmentation: Recursive character text segmentation technology is used to cut text data into several structured text blocks according to specific segmentation symbols (such as single line breaks, double line breaks, specified punctuation marks, etc.). In this process, the size of the text block will be dynamically adjusted according to the contextual restrictions of LLM to ensure that both the accuracy requirements are met and the contextual information of the text block is retained.

[0026] Text vector representation: These text blocks are converted into numerical vectors through text embedding models (such as Embedding) to capture the text semantic information for subsequent calculation and analysis.

[0027] Build index: Build index based on text embedding vector, store the numerical vector of each text block and its corresponding text data, and complete the entire index storage process.

[0028] 3. Retrieve and recall based on user queries Using a hybrid search mode that combines vector query and keyword query, retrieve the text block data most relevant to the user query from the index; use the re-ranking model or "semantic / keyword" weight distribution algorithm to re-rank the text block data to ensure that the results are sorted from high to low according to the relevance to the user query. Figure 4 As shown, the process includes the following steps: User query processing: The AI ​​system understands the user's query intent and ensures the accuracy of the question. If there is ambiguity or subject omission in the user input, it will be completed or disambiguated through context information.

[0029] Hybrid search mode: Combines vector search and keyword search to retrieve the most relevant text blocks from the index. Each search method returns a separate set of search results, usually a list of search results sorted by relevance. Vector search is mainly based on semantic matching, while keyword search is based on matching keywords mentioned in the query.

[0030] Re-ranking of recall results: In order to improve the accuracy of retrieval results, a re-ranking model or a "semantic / keyword" weight distribution algorithm (weighted sum of the relevance of vector retrieval and keyword retrieval, and sorting of retrieval results by the relevance after weighted summation) is used to re-rank the retrieval results to ensure that the most relevant text blocks are taken out first.

[0031] 4. Large model scoring and result optimization After obtaining the recall results, the content needs to be further screened through the LLM scoring mechanism to ensure that the most relevant media files can be accurately retrieved. Figure 5 As shown, this step includes: First scoring: The reordered content block text is scored for similarity with the user query text, and LLM performs single or batch scoring based on a score of 0-10. This step obtains the text block relevance score.

[0032] Second scoring: The media file titles corresponding to the reordered text blocks are scored similarly to the user query text, and the LLM performs single or batch scoring based on a score of 0-10. This step obtains the media file relevance score.

[0033] Results summary and screening: There is a threshold for the first scoring. Content blocks that reach the threshold will participate in the second scoring. Content blocks that do not reach the threshold will be directly discarded and will not participate in the subsequent result selection. The total scores of the two scoring results are sorted from high to low to obtain the final text block score ranking.

[0034] 5. Full database matching and final result sorting The corresponding media files are obtained according to the final text block score ranking result obtained in step 4. The one or more media files are used as the circled target media files.

[0035] Full library matching: Using correlation comparison algorithms such as exact matching, Levenshtein edit distance algorithm, Simhash fuzzy algorithm and cosine similarity, the full media library file comparison is performed according to the three types of information of file name, category and tag, and other media files with the highest similarity to the circled target media files are retrieved to form the final media file list.

[0036] Result sorting: re-sort the final media file results. The sorting rule is based on the priority: first scoring > second scoring > exact match results > other matching results.

[0037] 6. Final result presentation and user feedback After obtaining the final list of media files, the AI ​​system responds with text content and retrieves media files that are highly relevant to the user's query, allowing the user to view, download, or perform other operations. At the same time, users can also rate the accuracy of AI's answers and retrieved media files through the feedback module, and use this as feedback data for analysis, identification of problems and improvement points, and continuous optimization of accuracy.

[0038] In summary, through technical means such as multimodal data fusion, hybrid retrieval, and large model scoring, this method provides an efficient and accurate query and retrieval solution for media files in AI dialogue scenarios. This method can not only improve the accuracy of retrieval, but also realize intelligent association and rapid retrieval between different types of media files, providing users with a more intelligent and personalized service experience.

[0039] Embodiment 2 Based on the above-mentioned media file query and retrieval method, this embodiment provides a media file query and retrieval system based on an AI dialogue scenario. The system has functional modules corresponding to the above-mentioned method and can achieve the technical effects of the above-mentioned method.

[0040] 1. System Architecture The overall architecture of the system can be divided into the following modules: Media file preprocessing module: used to classify media files and extract text data, and extract keywords from the extracted text data as content tags for each media file; it is also used to cut text data into several structured text blocks, and convert the text blocks into text embedding vectors through the text embedding model.

[0041] Media file database: stores all media files and their related metadata, including pictures, videos, PPT, etc. Media files are pre-processed, annotated, and classified and stored through technologies such as OCR, voice recognition, and text extraction. The database also stores index data such as file access paths, tags, classification information, and pre-processed text content.

[0042] User interface module: Users interact with the system through natural language queries, entering query questions or specifying tasks (such as finding a video, picture, PPT, etc.). This module is responsible for receiving user input and passing the query content to the query processing module.

[0043] Query processing module: This module is responsible for semantically parsing the query input by the user, understanding the intent of the query, and converting it into a format that the system can process. It uses natural language processing (NLP) technology to perform text analysis to check whether there are problems such as semantic ambiguity and subject omission. If there are problems, it automatically completes or disambiguates.

[0044] Multimodal retrieval engine: The core function of this module is to perform efficient media file retrieval for user queries. Through a hybrid retrieval mode that combines vector retrieval with keyword retrieval, cross-modal multimedia file retrieval is performed, including data types such as text, images, videos, and PPT. When searching, the system first performs keyword retrieval based on query terms and file metadata (such as tags and classification information), and then performs vector retrieval by calculating the embedded vectors of text, images, and videos to match the semantic relevance of the query and file content.

[0045] LLM scoring and ranking module: LLM is used to obtain the similarity between the user query text and the recalled text block and the media file title corresponding to the recalled text block, and the selected text block is screened. LLM Scoring and Ranking Module: This module re-ranks the recalled text blocks based on the results of the hybrid search. The search results are scored twice using LLM. The first scoring evaluates the relevance of the file content blocks, while the second scoring is based on the similarity between the file title and the user query. The scoring results are combined with the classification information of the file, and the text blocks that are most relevant to the user query are finally output.

[0046] Full library matching and retrieval module: obtains the media files corresponding to the selected text blocks and other similar media files from the media file database to form a target media file list, and feeds back the sorted target media file list to the user, supporting file playback, viewing, downloading and other operations.

[0047] User feedback module: This module is responsible for collecting user feedback during the file retrieval process, such as whether the file is relevant to the query, whether the response is timely, and the user's rating of the file's relevance. User feedback data is used to train and optimize the system to improve the accuracy and response speed of future retrieval.

[0048] 2. Functional Flow The system's workflow is as follows: Step 1: User query and input The user makes a query request to the system through natural language, such as "find PPT about machine learning", and the user's input is transmitted to the query processing module through the UI module.

[0049] Step 2: Query Processing The query processing module parses user input through natural language processing techniques (such as intent recognition, named entity recognition, etc.), understands the meaning of the query, and detects whether there is ambiguity or contextual information that needs to be supplemented. For example, if the query does not specify the specific content or scope of "machine learning", the system may prompt the user to provide more details. The parsed query will be converted into a standard format and passed to the search engine.

[0050] Step 3: Multimodal Retrieval In the search engine, user queries will trigger two types of searches at the same time: keyword search and vector search. Keyword search will search for relevant files based on user input and file metadata (such as title, tag, and category). Vector search performs semantic vectorization on each file text and multimedia data (such as descriptions in pictures and videos), converts files and queries into numerical vectors, and finds the files that are closest to the user's query semantics through similarity calculation.

[0051] Step 4: Reranking and Scoring The retrieved relevant documents are passed to the scoring module. First, the system uses the Large Language Model (LLM) to perform two rounds of scoring on the search results. The first scoring evaluates the relevance of each file content block, and the second scoring focuses on the match between the file title and the query content. Based on the score, the system will sort the search results and select the most relevant documents to return to the user.

[0052] Step 5: Display and Operation The scored and sorted files will be presented on the user interface, and users can view, play or download related media files. During the presentation, users can see the basic information, tags and a brief description of the content related to the query. Users can further interact with the files, such as playing videos, viewing pictures or PPTs.

[0053] Step 6: User feedback and optimization The displayed file results will be fed back to the user, who can evaluate the relevance of the displayed media files. The system records the user's feedback data for subsequent optimization of the retrieval algorithm and scoring mechanism. This feedback will help the system gradually improve and improve the accuracy of future query responses.

[0054] This media file query and retrieval system provides an efficient and accurate solution for media file retrieval in AI dialogue scenarios through innovative multimodal retrieval, dual-round scoring mechanism and user feedback optimization technology, and has broad application prospects.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A media file query and retrieval method based on AI dialogue scenario, characterized in that: Includes steps: S1. Media file preprocessing and data annotation: Read and classify each media file and extract text data from each media file; Extract keywords from the extracted text data to serve as content tags for each media file; Store the annotated media files and extracted text data in the cloud or local server; S2. Media file cleaning and embedding processing: Convert the extracted text data into a structured text format; Cut the text data in a unified format into several structured text blocks; The text blocks are converted into text embedding vectors respectively through the text embedding model; Store the text embedding vector of each text block, and build an index of the corresponding text data based on the text embedding vector; S3. Retrieve and recall based on user query: Understand and optimize the user's query intent and obtain the user's query text; Retrieve text blocks related to the user query text from the index; Re-ranking the retrieved text blocks by relevance to obtain re-ranked text blocks; S4. Large model scoring and result optimization: LLM is used to obtain the similarity between the user query text and the reordered text blocks and the media file titles corresponding to the reordered text blocks, and the preferred text blocks are screened; S5. Full database matching and query retrieval: Obtaining a media file corresponding to the selected text block, which is recorded as the circled target media file; According to the names, categories and tags of the media files, other media files similar to the identified target media files in the cloud or local server are obtained, thereby forming a target media file list; The media files in the target media file list are sorted by similarity, and the sorted target media file list is fed back to the user, and the media file with the highest relevance to the user's query is retrieved at the same time.

2. The media file query and retrieval method according to claim 1, characterized in that: In step S1, for image files, OCR technology and NLP tools are used to automatically identify and generate text data; for video files, speech-to-text services are used to extract the explanation text of the video content; for PPT files, document parsing is performed and summarized through LLM; for files where valid text cannot be extracted, brief descriptions are filled in manually.

3. The media file query and retrieval method according to claim 1, characterized in that: In step S2, a recursive character text segmentation technique is used to cut the text data into a number of structured text blocks according to specific segmentation symbols.

4. The media file query and retrieval method according to claim 1, characterized in that: In step S3, if there is ambiguity or subject omission in the user input, the context information will be used to complete or disambiguate it to obtain an optimized and adjusted user query text.

5. The media file query and retrieval method according to claim 1, characterized in that: In step S3, vector retrieval and keyword retrieval are combined to retrieve several text block data related to the user query text from the index; and the retrieval results are re-ranked using a re-ranking model or a "semantic / keyword" weight distribution algorithm.

6. The media file query and retrieval method according to claim 1, characterized in that: In step S4, the reordered text blocks are scored with the user query text by using LLM to obtain the text block relevance score; the titles of the media files corresponding to the reordered text blocks are scored with the user query text by using LLM to obtain the media file relevance score; only the text blocks whose text block relevance scores reach the set threshold participate in the media file relevance score, and the text blocks are sorted from high to low according to the total score of the text block relevance score and the media file relevance score to obtain the preferred text blocks.

7. The media file query and retrieval method according to claim 1, characterized in that: In step S5, other media files similar to the circled target media file in the cloud or local server are obtained by using a relevance comparison algorithm, wherein the relevance comparison algorithm includes an exact match, a Levenshtein edit distance algorithm, a Simhash fuzzy algorithm, and a cosine similarity algorithm.

8. The media file query and retrieval method according to claim 7, characterized in that: When media files in the target media file list are sorted by similarity, the sorting priority is: text block relevance score > media file relevance score > exact match score > other scores.

9. A media file query and retrieval system based on the method according to any one of claims 1 to 7, characterized in that: Includes the following modules: Media file preprocessing module: used to classify media files and extract text data, and extract keywords from the extracted text data to serve as content tags for each media file; It is also used to cut text data into several structured text blocks and convert the text blocks into text embedding vectors through the text embedding model; Media file database: used to store media files and their corresponding text blocks, text embedding vectors, as well as access paths, tags, and classification information of media files; User interface module: used to receive user input and pass the query content to the query processing module. The user interacts with the system through natural language queries, inputs query questions or specifies tasks; Query processing module: used to perform semantic analysis on the query input by the user, understand the query intent, and output the optimized and adjusted user query text; Multimodal retrieval engine: used to retrieve text blocks related to the user's query text from the media file database as recalled text blocks; LLM scoring and ranking module: LLM is used to obtain the similarity between the user query text and the recalled text block and the media file title corresponding to the recalled text block, and the selected text block is screened; Full-library matching and retrieval module: obtains the media files corresponding to the selected text blocks and other similar media files from the media file database to form a target media file list, and feeds back the sorted target media file list to the user. At the same time, the media files with the highest relevance to the user's query are retrieved for the user to view, download or perform other operations.

10. The media file query and retrieval system according to claim 9, characterized in that: It also includes a user feedback module, which is responsible for collecting user feedback during the file retrieval process.

Images