Method for realizing object storage fuzzy retrieval based on large model

Abstract

The invention particularly relates to a method for realizing object storage fuzzy retrieval based on a large model. According to the method for realizing the fuzzy retrieval of the object storage based on the large model, each object in the object storage is summarized through the large model to form an object summarizing result; the object summary result is converted into a feature vector through an embedded vector technology, and the object name, the object summary result and the feature vector are cached; the query statement is converted into a feature vector, neighbor indexing is carried out in a vector database, if the difference between a neighbor indexing result and the feature vector is smaller than a self-defined threshold value, it is considered that the search is hit, and a corresponding object name and an object summary result are returned. According to the method for realizing fuzzy retrieval of object storage based on the large model, the storage and maintenance cost is reduced, the application scene of object storage is expanded, the usability of the system and the user experience are improved, the universality is high, and the method can be suitable for all object storage systems.

Description

Technical Field

[0001] The present invention relates to the technical field of large models, and particularly relates to a method for realizing fuzzy retrieval of object storage based on a large model. Background Art

[0002] Object storage is a distributed storage product that can natively support the HTTP / HTTPS protocol. As long as it is connected to the Internet, the object storage service can be directly used. With the development and maturity of object storage, more and more Internet websites and APPs have begun to store some static resources in the object storage service, which not only reduces the pressure on the server but also can reduce costs. Object storage has characteristics such as infinite expansion and flattened data structure, which enables it to store a huge amount of data.

[0003] Due to its unique data organizational structure, products in the industry generally can only perform file name prefix queries. Even when combined with products such as "data processing", it can only perform exact matching searches on objects of specific document types. And this method will import all documents into document databases such as Elastic Search. The storage limit of this type of database is limited, and it is difficult to meet the search requirements for TB-level objects. Moreover, this method is very costly, so it is difficult to promote in most scenarios. For other multimedia type documents such as pictures and videos, they can only be simply marked and searched by manually tagging, which is time-consuming and laborious and cannot perform fuzzy searches. These problems have become a pain point in the use of object storage.

[0004] In order to solve the above problems, the present invention proposes a method for realizing fuzzy retrieval of object storage based on a large model. Summary of the Invention

[0005] The present invention provides a simple and efficient method for realizing fuzzy retrieval of object storage based on a large model to make up for the defects of the prior art.

[0006] The present invention is realized by the following technical solutions:

[0007] A method for realizing fuzzy retrieval of object storage based on a large model includes the following steps:

[0008] Step S1: Summarize each object in the object storage through a large model to form an object summary result with no less than the custom number of words;

[0009] Step S2: Through the embedding vector technology, abstract the features of the object summary result, convert the object summary result into a feature vector, and cache the object name, object summary result, and feature vector.

[0010] Step S3: When the user inputs a query statement, first convert the query statement into a feature vector and perform a nearest neighbor search in the vector database. If the difference between the nearest neighbor search result and the feature vector is less than the custom threshold, it is considered a search hit, and the corresponding object name and object summary result are returned.

[0011] In the said Step S1, the implementation process is as follows:

[0012] Step S1.1: File type identification:

[0013] Step S1.1.1: Read the metadata of the object, including the file name, file type, size, and creation time;

[0014] Step S1.1.2: Identify the file type of the object according to the file suffix name of the object (such as txt, pdf, jpg, mp4, etc.);

[0015] Step S1.1.3: Select the corresponding large model according to the file type for content summarization;

[0016] For text files, use text models such as GPT-3, for image files, use image models such as LLaVA, and for video files, use video understanding models;

[0017] Step S1.2: Content extraction:

[0018] For text files: Read all the content of the text file, and then extract the key paragraphs and sentences of the file according to the length and importance of the file content;

[0019] For image files: Use image processing algorithms to preprocess the images, customize the adjustment of the size, and denoise, and use a pre-trained image understanding model (such as ResNet) to customize the extraction of the main visual features of the images;

[0020] For video files: Extract the key frames of the video, perform image processing and feature extraction on each key frame, and use a video understanding model to analyze the video content and generate a video summary;

[0021] Step S1.3: Content summarization:

[0022] Customize the word count threshold, and use the large model to process the extracted content to generate an object summary result not less than the word count threshold; ensure that the object summary result can summarize the main information and characteristics of the object through the prompt;

[0023] At the same time, in order to be compatible with the original file information of the object storage, add the file name and metadata to the object summary result; Step S1.4: Handling of special cases:

[0024] If the object cannot be processed by the large model (such as unknown file type or unrecognizable by the large model), the metadata of the object (such as file name, label) is used to generate the summary result.

[0025] In the step S2, the feature vector generation process is as follows:

[0026] Step S2.1: First, use the language large model to translate the prompt to unify its language (such as Chinese);

[0027] Step S2.2: Use the text vectorization method to vectorize the summary result output by the object summarization module;

[0028] It should be noted that the vectorization can be performed locally or remotely through other means such as API interfaces. In addition, different vectorization methods and different technical implementations based on the methods will affect the vectorization effect, and thus affect the search hit effect. Therefore, choosing a good prompt vectorization method will greatly improve the entire system.

[0029] The text vectorization method includes but is not limited to the text vectorization method based on statistics and the text vectorization method based on neural networks.

[0030] Step S2.3: Build a vector database for storing and managing the feature vectors of the object summary results output by the object summarization module and supporting efficient nearest neighbor search.

[0031] In the step S2.3, when building the vector database, first customarily select a suitable vector database technology, such as FAISS, Annoy, HNSW, etc., and configure the index parameters, including the number of trees and the number of shards;

[0032] Then, select a table building method according to the business requirements to establish a vector data table; the table building methods are as follows:

[0033] Establish a vector data table according to the business scenario;

[0034] Establish a vector data table according to the granularity of the object storage bucket;

[0035] Establish a vector data table according to the granularity of the object storage user;

[0036] When storing, store the summary result and its feature vector as key values in the vector database;

[0037] According to the business requirements, store the object name, label, object creation time, and storage type as additional metadata of the record in the vector database together for other function implementations.

[0038] In the step S3, the process of processing the user query request is as follows:

[0039] Step S3.1: Clean and standardize the query statement input by the user, remove special characters, and unify the case; if the query statement is not the system default language (such as Chinese), use a language model to translate it into the system default language;

[0040] Step S3.2: Use the same text vectorization method as in step S2 (feature vector generation module) to convert the processed query statement into a query vector;

[0041] Step S3.3: Perform a nearest neighbor search in the vector database, calculate the similarity between each feature vector and the query vector, sort the feature vectors in descending order of similarity, and filter out the feature vector results whose similarity exceeds a custom threshold;

[0042] Step S3.4: According to business requirements, customize and specify additional filtering conditions, such as file type, creation time, etc., for further result filtering;

[0043] Step S3.5: According to the filtered results, return the object name, object summary result, and similarity score information. According to business requirements, other metadata of the object, such as file size, last modification time, etc., can be selectively returned.

[0044] In the method for implementing fuzzy retrieval of object storage based on a large model, when a new object is uploaded to the object storage or an old object is updated when creating or updating an object, the induction summary and vectorization processes will be automatically triggered;

[0045] In addition, by customizing and setting up a scheduled task, the induction summary and vectorization processes can be triggered periodically;

[0046] The administrator manually triggers the immediate processing of a specific object through the management interface.

[0047] An apparatus for implementing fuzzy retrieval of object storage based on a large model, comprising:

[0048] An object induction summary module, responsible for using a large model to summarize each object in the object storage to form an object summary result not less than a custom number of words;

[0049] A feature vector generation module, responsible for using the embedding vector technology to perform feature abstraction on the object summary result, converting the object summary result into a feature vector, and caching the object name, object summary result, and feature vector;

[0050] The fuzzy retrieval module is responsible for processing the user's query request, converting the query statement into a feature vector, and performing a nearest neighbor search in the vector database. If the difference between the nearest neighbor search result and the feature vector is less than a custom threshold, it is considered a search hit, and the corresponding object name and object summary result are returned;

[0051] The summary trigger module is responsible for monitoring the object creation and update events stored in the object storage, triggering the object induction summary module and the feature vector generation module using an asynchronous processing mechanism, and is equipped with a failure compensation mechanism to avoid affecting the upload speed.

[0052] When the summary trigger module discovers that a new object is uploaded to the object storage or an old object is updated, it will automatically trigger the induction summary and vectorization processes;

[0053] In addition, the summary trigger module supports custom setting of scheduled tasks to trigger the induction summary and vectorization processes at regular intervals;

[0054] It supports the administrator to manually trigger the immediate processing of specific objects through the management interface.

[0055] A device for implementing fuzzy retrieval of object storage based on a large model, characterized in that it includes a memory and a processor; the memory is used to store computer programs, and the processor is used to implement the above method steps when executing the computer programs.

[0056] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0057] The beneficial effects of the method for implementing fuzzy retrieval of object storage based on a large model are:

[0058] (1) Through the large model and embedding vector technology, efficient fuzzy search of a large number of objects is realized, and users can search for files in the bucket through natural language, filling the gap in this technical field;

[0059] (2) There is no need to import all objects into an expensive document-type database, reducing storage and maintenance costs;

[0060] (3) It is applicable to different types of objects, including text, pictures, videos, etc., expanding the application scenarios of object storage;

[0061] (4) Users can query through natural language, improving the usability and user experience of the system;

[0062] (5) It does not depend on the underlying object storage technology, so it has strong versatility and can be applied to all object storage systems. Description of the Drawings

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0064] Appendix Figure 1 It is a schematic diagram of the method for realizing fuzzy retrieval of object storage based on a large model of the present invention. Specific implementation manners

[0065] In order to enable those skilled in the art of this technology to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in combination with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0066] Large model: It refers to a deep learning model with a large number of parameters, usually with the number of parameters reaching billions or even more. These models can handle complex tasks such as natural language processing, image recognition, recommendation systems, etc., and can provide high-quality results. Multimodal large models can implement scenarios such as reading documents, watching pictures and videos, listening to audio, etc., and have the ability to understand these materials.

[0067] Embedding vector: Embedding vector. In the field of natural language processing (NLP), it refers to mapping words, phrases or other text units into a fixed-length real number vector space. The embedding vector has a lower dimension, usually between dozens and hundreds of dimensions, and each dimension contains a certain degree of semantic information. This means that in the embedding vector space, the relevance between two texts can be judged by determining the distance between their mapped vectors. Therefore, text search based on semantics can be realized by means of this technology.

[0068] Object: The smallest storage unit in object storage, which can be any data file. There can be up to hundreds of millions of objects in a bucket. When an object needs to be used, it can be directly downloaded by simply inputting the object name, and the speed is not affected by the increase in the number of objects in the bucket.

[0069] The method for realizing fuzzy retrieval of object storage based on a large model includes the following steps:

[0070] Step S1: Summarize each object in the object storage through a large model to form an object summary result with no less than the custom number of words;

[0071] Step S2: Through the embedding vector technology, perform feature abstraction on the object summary result, convert the object summary result into a feature vector, and cache the object name, object summary result, and feature vector.

[0072] Step S3: When the user inputs a query statement, first convert the query statement into a feature vector and perform a nearest neighbor search in the vector database. If the difference between the nearest neighbor search result and the feature vector is less than the custom threshold, it is considered a search hit, and the corresponding object name and object summary result are returned.

[0073] The objects in the object storage are diverse and may include different file types. For each different file type, the applicable large model is also different. To ensure correct identification and summarization in the subsequent induction and summarization process, this module will first match the applicable large model according to the file extension. In this step, it is also possible that some file types do not have an applicable large model. In this case, the file name, object label, and other metadata can be used as its summary.

[0074] In the above-mentioned step S1, the implementation process is as follows:

[0075] Step S1.1: File type identification:

[0076] Step S1.1.1: Read the metadata of the object, including the file name, file type, size, and creation time.

[0077] Step S1.1.2: Identify the file type of the object according to the file extension of the object (such as txt, pdf, jpg, mp4, etc.).

[0078] Step S1.1.3: Select the corresponding large model according to the file type for content summarization.

[0079] For text files, use text models such as GPT-3; for image files, use image models such as LLaVA; for video files, use video understanding models.

[0080] Step S1.2: Content extraction:

[0081] Text files: Read all the content of the text file, and then extract the key paragraphs and sentences of the file according to the length and importance of the file content.

[0082] Image files: Use image processing algorithms to preprocess the images, customize the adjustment of the size, and remove noise. Use a pre-trained image understanding model (such as ResNet) to customize the extraction of the main visual features of the images.

[0083] Video file: Extract key frames of the video, perform image processing and feature extraction on each key frame, analyze the video content using a video understanding model, and generate a video summary;

[0084] Step S1.3, Content summary:

[0085] Customize the word count threshold, process the extracted content through a large model, and generate an object summary result not less than the word count threshold; Ensure that the object summary result can summarize the main information and characteristics of the object through the prompt prompt;

[0086] For example, the text summary can be a 200-word paragraph, the picture summary can be a short text describing the main visual elements, and the video summary can be a brief description of the main plot.

[0087] At the same time, in order to be compatible with the original file information of the object, add the file name and metadata to the object summary result; For example, the generated result can be "Text file: file name + tag + summary".

[0088] Step S1.4, Handling special cases:

[0089] If the object cannot be processed by the large model (such as unknown file type or unrecognizable by the large model), then use the metadata of the object (such as file name, tag) to generate the summary result.

[0090] For example, for a file without clear content, the generated summary result can be "Unknown type file: file name + tag".

[0091] In the said step S2, the feature vector generation process is as follows:

[0092] Step S2.1, First, use a language large model to translate the prompt so that its language is unified (such as Chinese), because the vectorization methods for different languages may be different, and for sentences with the same meaning in different languages, the vectorization results often vary greatly.

[0093] Step S2.2, Use a text vectorization method to vectorize the summary result output by the object summarization module;

[0094] It should be noted that vectorization can be performed locally or remotely through other means such as API interfaces. In addition, different vectorization methods and different technical implementations based on the methods will affect the vectorization effect, and thus affect the search hit effect. Therefore, choosing a good prompt vectorization method will greatly improve the entire system.

[0095] The said text vectorization method includes but is not limited to the text vectorization method based on statistics and the text vectorization method based on neural network.

[0096] Step S2.3: Construct a vector database to store and manage the feature vectors of the object summary results output by the object summarization module and support efficient nearest neighbor search.

[0097] In the above step S2.3, when constructing the vector database, first custom-select a suitable vector database technology, such as FAISS, Annoy, HNSW, etc., and configure the index parameters, including the number of trees and the number of shards.

[0098] Then, select a table creation method according to business requirements to create a vector data table. The table creation methods are as follows:

[0099] Create a vector data table according to the business scenario.

[0100] Create a vector data table according to the granularity of the object storage bucket.

[0101] Create a vector data table according to the granularity of the object storage user.

[0102] When storing, store the summary result and its feature vector in the vector database as key-value pairs.

[0103] According to business requirements, store the object name, label, object creation time, and storage type in the vector database together with the additional metadata of the record for other function implementations.

[0104] In the above step S3, the process of processing the user query request is as follows:

[0105] Step S3.1: Clean and standardize the query statement input by the user, remove special characters, and unify the case; if the query statement is not the system default language (such as Chinese), use a language model to translate it into the system default language.

[0106] Step S3.2: Use the same text vectorization method as in step S2 (feature vector generation module) to convert the processed query statement into a query vector.

[0107] Step S3.3: Perform a nearest neighbor search in the vector database, calculate the similarity between each feature vector and the query vector, sort the feature vectors in descending order of similarity, and filter out the feature vector results whose similarity exceeds the custom threshold.

[0108] Step S3.4: According to business requirements, custom-specify additional filtering conditions, such as file type, creation time, etc., for further result filtering.

[0109] Step S3.5: Based on the filtered results, return the object name, object summary result, and similarity score information. According to business requirements, other metadata of the object, such as file size, last modification time, etc., can be selectively returned.

[0110] In the method for implementing fuzzy retrieval of object storage based on a large model, when a new object is uploaded to the object storage or an existing object is updated during object creation or update, the induction summary and vectorization processes will be automatically triggered.

[0111] In addition, by customizing the setting of a scheduled task, the induction summary and vectorization processes can be triggered periodically.

[0112] Administrators can manually trigger the immediate processing of specific objects through the management interface.

[0113] The device for implementing fuzzy retrieval of object storage based on a large model includes:

[0114] An object induction summary module, which is responsible for using a large model to summarize each object in the object storage to form an object summary result with no less than the custom number of words.

[0115] A feature vector generation module, which is responsible for using the embedding vector technology to abstract the features of the object summary result, convert the object summary result into a feature vector, and cache the object name, object summary result, and feature vector.

[0116] A fuzzy retrieval module, which is responsible for processing the user's query request, converting the query statement into a feature vector, and performing a nearest neighbor index in the vector database. If the difference between the nearest neighbor index result and this feature vector is less than the custom threshold, it is considered that the search hits, and the corresponding object name and object summary result are returned.

[0117] A summary trigger module, which is responsible for monitoring object creation and update events in the object storage, triggering the object induction summary module and the feature vector generation module using an asynchronous processing mechanism, and having a failure compensation mechanism to avoid affecting the upload speed. The design of this module has an important impact on the performance and real-time nature of the system.

[0118] When the summary trigger module discovers that a new object is uploaded to the object storage or an existing object is updated, the induction summary and vectorization processes will be automatically triggered.

[0119] In addition, the summary trigger module supports customizing the setting of a scheduled task to trigger the induction summary and vectorization processes periodically.

[0120] It supports administrators to manually trigger the immediate processing of specific objects through the management interface.

[0121] The device for realizing fuzzy retrieval of object storage based on a large model includes a memory and a processor; the memory is used for storing a computer program, and the processor is used for implementing the above method steps when executing the computer program.

[0122] A computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the above method steps are implemented.

[0123] The above embodiments are only one of the specific implementation manners of the present invention, and the common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for implementing fuzzy retrieval of object storage based on a large model, characterized by: The following steps are involved: Step S1: Summarize each object in the object storage through the big model to form an object summary result with no less than a custom word count; Step S2: abstract the object summary result by embedding vector technology, convert the object summary result into a feature vector, and cache the object name, the object summary result and the feature vector; Step S3: When the user enters a query statement, the query statement is first converted into a feature vector and a nearest neighbor index is performed in the vector database. If the difference between the nearest neighbor index result and the feature vector is less than a custom threshold value, the search is considered a hit and the corresponding object name and object summary result are returned.

2. The method for implementing fuzzy retrieval of object storage based on a large model according to claim 1 is characterized in that: In step S1, the implementation process is as follows: Step S1.1, file type identification: Step S1.1.1: Read the metadata of the object, including the file name, file type, size and creation time; Step S1.1.2: Identify the file type of the object according to the file extension of the object; Step S1.1.3: Select the corresponding large model according to the file type to summarize the content; The GPT-3 text model is used for text files, the LLaVA large image model is used for image files, and the video understanding model is used for video files; Step S1.2, content extraction: Text file: read the entire content of the text file, and then extract the key paragraphs and sentences of the file based on the length and importance of the file content; Image files: Use image processing algorithms to preprocess images, resize them, and remove noise. Use pre-trained image understanding models to extract the main visual features of the images. Video files: Extract key frames of the video, perform image processing and feature extraction on each key frame, use the video understanding model to analyze the video content, and generate a video summary; Step S1.3, content summary: Customize the word count threshold, process the extracted content through the big model, and generate an object summary result that is not less than the word count threshold; use the prompt word to ensure that the object summary result can summarize the information and characteristics of the object; At the same time, in order to be compatible with the original file information stored in the object, the file name and metadata are added to the object summary result; Step S1.4, special case processing: If the object cannot be processed by the large model, the object's metadata is used to generate summary results.

3. The method for implementing object storage fuzzy retrieval based on a large model according to claim 1 is characterized in that: In step S2, the feature vector generation process is as follows: Step S2.1, first use the language model to translate the prompts to make them in a unified language; Step S2.2, using a text vectorization method to vectorize the summary result output by the object summarization module; The text vectorization method includes but is not limited to a statistics-based text vectorization method and a neural network-based text vectorization method. Step S2.3: construct a vector database for storing and managing the feature vectors of the object summary results output by the object summarization module and supporting nearest neighbor search.

4. The method for implementing fuzzy retrieval of object storage based on a large model according to claim 3 is characterized in that: In step S2.3, when constructing the vector database, firstly, the vector database technology is customized and selected, and the index parameters are configured, including the number of trees and the number of shards; Then, select a table creation method based on business requirements to create a vector data table; the table creation method is as follows: Create vector data tables according to business scenarios; Create a vector data table based on the granularity of the object storage bucket; Create a vector data table based on the granularity of object storage users; When storing, the summary result and its feature vector are stored as key values ​​in the vector database; According to business needs, the object name, label, object creation time and storage type are stored in the vector database as additional metadata of the record.

5. The method for implementing object storage fuzzy retrieval based on a large model according to claim 1 is characterized in that: In step S3, the process of processing the user query request is as follows: Step S3.1: Clean and standardize the query statement input by the user, remove special characters, and unify the upper and lower case letters; if the query statement is not in the system default language, use the language model to translate it into the system default language; Step S3.2, using the same text vectorization method as in step S2, converting the processed query statement into a query vector; Step S3.3, perform a nearest neighbor search in the vector database, calculate the similarity between each feature vector and the query vector, and sort the feature vectors from high to low according to the similarity, and filter out the feature vector results whose similarity exceeds the custom threshold; Step S3.4: Customize and specify filter conditions according to business needs to further filter results; Step S3.5: Based on the filtered results, the object name, object summary result, similarity score and customized metadata information are returned.

6. The method for implementing object storage fuzzy retrieval based on a large model according to claim 1 is characterized in that: When an object is created or updated, a new object is uploaded to the object storage or an old object is updated, which automatically triggers the summarization and vectorization process; In addition, by customizing the scheduled tasks, the summarization and vectorization processes can be triggered regularly. Administrators manually trigger immediate processing of specific objects through the management interface.

7. A device for implementing fuzzy retrieval of object storage based on a large model, characterized in that: include: The object summarization module is responsible for summarizing each object in the object storage using the large model to form an object summary result with a word count no less than the customized number. The feature vector generation module is responsible for using the embedded vector technology to perform feature abstraction on the object summary results, converting the object summary results into feature vectors, and caching the object name, object summary results and feature vectors; The fuzzy retrieval module is responsible for processing the user's query request, converting the query statement into a feature vector, and performing a nearest neighbor index in the vector database. If the difference between the nearest neighbor index result and the feature vector is less than a custom threshold, the search is considered a hit, and the corresponding object name and object summary result are returned; The summary trigger module is responsible for monitoring the object creation and update events of the object storage, using an asynchronous processing mechanism to trigger the object summary module and the feature vector generation module, and has a failure compensation mechanism to avoid affecting the upload speed.

8. The device for implementing fuzzy retrieval of object storage based on a large model according to claim 7, characterized in that: When the summary trigger module finds that a new object is uploaded to the object storage or an old object is updated, it will automatically trigger the summarization and vectorization process; In addition, the summary trigger module supports custom setting of scheduled tasks, which can trigger the summary and vectorization process on a regular basis; Supports administrators to manually trigger immediate processing of specific objects through the management interface.

9. A device for implementing fuzzy retrieval of object storage based on a large model, characterized in that: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 1 to 6 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images