Image data storage and retrieval method and apparatus based on lightweight multi-modal model

Abstract

The application provides an image data storage and retrieval method and device based on a lightweight multi-modal model, a computer storage medium and a processor. The application reduces the computational complexity of the CLIP model and improves the feature storage and retrieval efficiency by introducing a lightweight CLIP model and a feature clustering storage strategy. By dividing the feature clustering cluster before storing the feature, the retrieval matching is reduced according to the clustering cluster in the retrieval stage, and more accurate retrieval is realized. The introduction of the LLM agent judgment and the LLM agent model re-retrieval improves the relevance and consistency of the retrieval results. Therefore, the application solves the problem of low efficiency caused by low delay in the retrieval of multi-modal data in the vector database in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data communication, in particular to an image data storage and retrieval method based on a lightweight multi-modal model, and an image data storage and retrieval device based on a lightweight multi-modal model, a computer storage medium and a processor. BACKGROUND

[0002] With the explosive growth of Internet information, users have increasingly high requirements for the accuracy and relevance of search results. Traditional search methods often rely on single text or image information, which is difficult to meet the diversified needs of users for search results. To solve this problem, multi-modal search technology has emerged. Multi-modal search combines various information sources such as text and image, and through the efficient cooperation of multi-modal models, it realizes a more accurate and diversified search experience. Among them, the CLIP model as the representative of the current multi-modal technology, its architecture is composed of image encoder (Visual Encoder), text encoder (TextEncoder) and multi-modal contrast learning part trained together.

[0003] In terms of data storage and retrieval, vector databases serve as supporting technology, bearing the task of efficiently storing, indexing and retrieving multi-modal features. Vector databases take vectors as the basic data structure, which are suitable for representing entities with multiple attributes (such as text, image and audio). By constructing an efficient vector index structure, vector databases can achieve fast similarity search in massive data. However, there is currently a problem of low efficiency caused by high low-latency multi-modal data retrieval in vector databases. SUMMARY

[0004] The main purpose of the present application is to provide an image data storage and retrieval method based on a lightweight multi-modal model, and an image data storage and retrieval device based on a lightweight multi-modal model, a computer storage medium and a processor, to at least solve the problem of low efficiency caused by high low-latency multi-modal data retrieval in vector databases in the prior art.

[0005] To achieve the above object, according to one aspect of the present application, a method for image data storage and retrieval based on a lightweight multi-modal model is provided, comprising: classifying based on characteristics and application scenarios of first image data to obtain a plurality of first feature categories, the first image data being image data to be stored; performing feature extraction on the first image data based on at least a lightweight CLIP model to obtain first target features, clustering the first target features based on the first feature categories to obtain a plurality of feature clustering clusters, and storing the feature clustering clusters in a vector database, the lightweight CLIP model being a CLIP model combining 1D convolution and attention layers; obtaining second image data and / or first text data, obtaining second feature categories, the second image data being query images input by a user, the first text data being query text input by a user, and the second feature categories being query feature categories input by a user and including one or more of the first feature categories; performing feature extraction on the second image data and / or the first text data based on at least the lightweight CLIP model to obtain second target features, determining corresponding feature clustering clusters based on the second feature categories to obtain a target clustering cluster, and matching the second target features in the target clustering cluster to obtain a first retrieval result; inputting the first retrieval result and the second image data or the first retrieval result and the first text data into an LLM agent model for evaluation, and in the case of passing the evaluation, determining the first retrieval result as a target retrieval result, the evaluation including one or more of relevance evaluation, confidence evaluation, and consistency evaluation.

[0006] Optionally, the feature extraction on the first image data based on at least the lightweight CLIP model to obtain first target features comprises: in the case that the feature clustering clusters are divided according to the characteristics of the first image data, segmenting the first image data by a SAM model to obtain a plurality of third image data; and performing feature extraction on the first image data and the third image data by the lightweight CLIP model respectively to obtain corresponding first target features.

[0007] Optionally, the feature extraction on the first image data and the third image data by the lightweight CLIP model respectively to obtain corresponding first target features comprises: performing feature extraction on the first image data by the lightweight CLIP model to obtain third target features; performing feature extraction on each of the third image data by the lightweight CLIP model respectively to obtain fourth target features; and adding target identifiers to the third target features and the fourth target features to obtain the first target features, the target identifiers being used to cluster the third target features and the fourth target features to different feature clustering clusters.

[0008] Optionally, after the first search result and the second image data or the first search result and the first text data are input into the LLM agent model for evaluation, if the input data of the LLM agent model is the first search result and the second image data, if the evaluation fails, second text data is obtained, the second text data being query text supplemented by the user; the LLM agent model processes the second text data to generate target description information, the target description information including color, shape, relationship and context information; the LLM agent model processes each of the third image data based on the target description information to generate association description information, the association description information being used to represent the relationship between the target description information of different third image data; the LLM agent model initializes a feature search graph based on the association description information, and generates a first sub-problem for each node in the feature search graph; the light CLIP model matches the vector database based on each first sub-problem, determines data corresponding to the maximum relevance as a second search result, and performs the evaluation in the LLM agent model based on the second search result and the second image data, if the evaluation passes, the second search result is determined as a target search result, and one first sub-problem corresponds to one second search result.

[0009] Optionally, performing the evaluation in the LLM agent model based on the second search result and the second image data includes: processing the second search result by the LLM agent model to obtain search response information corresponding to each node; adding each search response information as a new node to update the feature search graph; and performing the evaluation in the LLM agent model based on the feature search graph and the second image data.

[0010] Optionally, after the first search result is evaluated in the LLM agent model with the second image data or the first search result is evaluated in the LLM agent model with the first text data, the method further comprises: if the input data of the LLM agent model is the first search result and the first text data, if the evaluation fails, processing the first text data by the LLM agent model to obtain a second sub-problem; based on each second sub-problem, matching the vector database by the light CLIP model to determine the data corresponding to the maximum relevance as a third search result, and performing the evaluation in the LLM agent model based on the third search result and the first text data, if the evaluation passes, determining the third search result as the target search result, and one second sub-problem corresponds to one third search result.

[0011] Optionally, based on the matching of the second target feature in the target clustering cluster, a first search result is obtained, comprising: if the number of target clustering clusters is greater than a preset value, the matching process of different target clustering clusters is allocated to different threads for operation.

[0012] According to another aspect of the present application, there is provided an image data storage and retrieval device based on a lightweight multi-modal model, the device comprising: a first processing unit configured to classify based on characteristics of first image data and application scenarios to obtain a plurality of first feature categories, the first image data being image data to be stored; a second processing unit configured to perform feature extraction on the first image data based on at least a lightweight CLIP model to obtain first target features, cluster the first target features based on the first feature categories to obtain a plurality of feature clustering clusters, and store the feature clustering clusters to a vector database, the lightweight CLIP model being a CLIP model combining 1D convolution and attention layers; a first acquisition unit configured to acquire second image data and / or first text data, and acquire second feature categories, the second image data being query image input by a user, the first text data being query text input by a user, the second feature categories being query feature categories input by a user and comprising one or more of the first feature categories; a third processing unit configured to perform feature extraction on the second image data and / or the first text data based on at least the lightweight CLIP model to obtain second target features, determine corresponding feature clustering clusters based on the second feature categories to obtain a target clustering cluster, and perform matching in the target clustering cluster based on the second target features to obtain a first retrieval result; and a fourth processing unit configured to input the first retrieval result and the second image data or the first retrieval result and the first text data into an LLM agent model for evaluation, and determine the first retrieval result as a target retrieval result if the evaluation is passed, the evaluation comprising one or more of relevance evaluation, confidence evaluation, and consistency evaluation.

[0013] According to still another aspect of the present application, there is provided a computer-readable storage medium comprising a stored program, wherein the computer-readable storage medium is caused to perform any one of the image data storage and retrieval methods based on a lightweight multi-modal model when the program is run.

[0014] According to still another aspect of the present application, there is provided a multi-modal data storage system comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise instructions for performing any one of the image data storage and retrieval methods based on a lightweight multi-modal model.

[0015] The technical scheme of the application introduces a lightweight CLIP model and a feature clustering storage strategy, reduces the calculation complexity of the CLIP model, improves the feature storage and retrieval efficiency, divides the feature clustering cluster before the feature is stored, stores according to the clustering cluster, reduces the retrieval matching in the retrieval stage, realizes more accurate retrieval, introduces the LLMagent judgment and the LLM agent model re-retrieval, improves the relevance and consistency of the retrieval result, and therefore, the application solves the problem of low efficiency caused by low delay in the retrieval of multi-modal data in the vector database in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A hardware structure block diagram of a mobile terminal is shown, which provides an image data storage and retrieval method based on a lightweight multi-modal model according to an embodiment of the application;

[0017] Figure 2 A flowchart of an image data storage and retrieval method based on a lightweight multi-modal model is shown, which is provided according to an embodiment of the application;

[0018] Figure 3 An application scenario diagram of image data storage and retrieval based on a lightweight multi-modal model is shown, which is provided according to an embodiment of the application;

[0019] Figure 4 An application scenario diagram of segmentation of the first image data by a SAM model is shown, which is provided according to an embodiment of the application;

[0020] Figure 5 An application scenario diagram of feature clustering cluster clustering is shown, which is provided according to an embodiment of the application;

[0021] Figure 6 A structure block diagram of an image data storage and retrieval device based on a lightweight multi-modal model is shown, which is provided according to an embodiment of the application.

[0022] Among the above drawings, the following reference signs are included:

[0023] 102, processor; 104, memory; 106, transmission device; 108, input and output device. DETAILED DESCRIPTION

[0024] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict. The application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0025] In order to enable personnel in the technical field to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should be within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0027] For ease of description, the following describes some nouns or terms related to the embodiments of the present application:

[0028] Multimodal Model: refers to a machine learning model that can process and fuse multiple types of data, such as text, images, audio, video, etc.

[0029] Multimodal Data: involves different types of data sources, such as images (visual modality), text (language modality), audio (auditory modality), and video (spatiotemporal modality), etc.

[0030] CLIP (Contrastive Language-Image Pretraining): a multimodal model developed by OpenAI, which can understand and associate natural language with images. It is trained on large-scale image-text paired data through contrastive learning, so that the model can map images and text into the same feature space, thereby realizing mutual retrieval and understanding between images and text.

[0031] SAM model (SegmentAnything Model), a powerful multi-purpose model for image segmentation introduced by Meta (the parent company of Facebook) in 2023. The main goal of SAM is to achieve general segmentation, that is, it can handle almost any type of image object segmentation task without the need for specialized training for specific objects or fields.

[0032] A Vision-Language Model (VLM) is a multimodal generative artificial intelligence model that combines visual and language processing capabilities to analyze input data such as text and images. The core of this type of model lies in combining a large-scale language model (LLM) with a visual encoder, enabling the model to both understand text and "see" images, thus completing vision-related reasoning and generation tasks.

[0033] Vector Database: A database system specifically designed for storing, managing, and retrieving high-dimensional vector data.

[0034] Vector search algorithms are a class of methods for efficiently retrieving high-dimensional vector data, widely used in image retrieval, text search, and recommender systems. The goal of these algorithms is to find the vector most similar to the query vector, i.e., to perform an approximate nearest neighbor (ANN) search.

[0035] An LLM (Large Language Model) agent is an intelligent agent based on a large language model (LLM). It combines the powerful natural language understanding and generation capabilities of LLM to handle complex tasks through task planning, tool invocation, and knowledge expansion. An LLM agent can function as a standalone system or be embedded within a larger framework to handle diverse tasks such as information retrieval, logical reasoning, and execution of operational instructions. As described in the background section, existing technologies suffer from low efficiency due to high latency in retrieving multimodal data from vector databases. To address these technical problems, embodiments of this application provide an image data storage and retrieval method based on a lightweight multimodal model, an image data storage and retrieval device based on a lightweight multimodal model, a computer storage medium, and a processor.

[0036] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0037] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal based on a lightweight multimodal model for image data storage and retrieval, according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1The mobile terminal can further include a transmission device 106 for communication function and an input / output device 108. Those skilled in the art can understand that, Figure 1 The structure shown is only schematic and does not limit the structure of the mobile terminal. For example, the mobile terminal can include more or less components than those shown, or have a different configuration of components than those shown. Figure 1 The structure shown is only schematic and does not limit the structure of the mobile terminal. For example, the mobile terminal can include more or less components than those shown, or have a different configuration of components than those shown. Figure 1 The structure shown is only schematic and does not limit the structure of the mobile terminal. For example, the mobile terminal can include more or less components than those shown, or have a different configuration of components than those shown.

[0038] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as the computer program corresponding to the device information display method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, implements the above-mentioned method. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, and these remote memories can be connected to the mobile terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The specific examples of the above-mentioned network can include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC for short), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF for short) module, which is used to communicate with the Internet in a wireless manner.

[0039] In the present embodiment, a method for storing and retrieving image data based on a lightweight multi-modal model running on a mobile terminal, a computer terminal or the like computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0040] Figure 2 is a flowchart of a method for storing and retrieving image data based on a lightweight multi-modal model according to an embodiment of the present application. As Figure 2As shown, the method comprises the following steps:

[0041] In step S201, classification is performed based on the characteristics of the first image data and the application scenarios, to obtain a plurality of first feature categories.

[0042] The first image data is image data to be stored.

[0043] Specifically, according to the characteristics (such as content, style, purpose, etc.) of the first image data and the application scenario requirements (such as medical diagnosis, intelligent security, recommendation system, etc.), the image data is divided into a plurality of feature categories. Each feature category corresponds to a category of data with common features or application requirements, facilitating subsequent feature extraction and storage optimization.

[0044] In step S202, at least a lightweight CLIP model is used to extract features from the first image data to obtain first target features, the first target features are clustered based on the first feature categories, a plurality of feature clustering clusters are obtained, and the feature clustering clusters are stored in a vector database.

[0045] The lightweight CLIP model is a CLIP model combining 1D convolution and attention layers.

[0046] Specifically, the lightweight CLIP model is used to efficiently extract features from the first image data, and the extracted features are represented in the form of high-dimensional vectors. According to the first feature categories, the extracted target features are clustered to form a plurality of feature clustering clusters, which are stored in the vector database. The lightweight CLIP model combines 1D convolution and attention layers, balancing computational efficiency and feature representation ability. The computational complexity of feature extraction and storage is reduced, and the storage structure of the vector database is optimized through clustering, reducing the matching calculation amount in the retrieval stage, improving the retrieval efficiency and response speed.

[0047] In step S203, second image data and / or first text data are obtained, and second feature categories are obtained.

[0048] The second image data is query image input by a user, the first text data is query text input by a user, and the second feature category is a query feature category input by a user and includes one or more of the first feature categories.

[0049] Specifically, the user inputs query data (image or text) and corresponding feature categories. The query feature category can be specified as one or more stored first feature categories to narrow the search range and improve the search relevance. The user can quickly locate the target feature clustering cluster by specifying the feature category, reducing the matching calculation amount of irrelevant features, and further improving the search efficiency and accuracy.

[0050] Step S204, at least based on the lightweight CLIP model, the second image data and / or the first text data are extracted for feature extraction, to obtain the second target feature, determine the corresponding feature clustering cluster based on the second feature category, obtain the target clustering cluster, and perform matching in the target clustering cluster based on the second target feature, to obtain the first retrieval result.

[0051] Specifically, the target feature is extracted from the query data using the lightweight CLIP model, the corresponding feature clustering cluster is located through the second feature category, and the target clustering cluster is determined. Only in the target clustering cluster, the feature matching calculation is performed to obtain the first retrieval result. Focusing on the target clustering cluster, the search range and the calculation complexity are reduced, the search speed and the resource utilization efficiency are greatly improved, and the relevance of the search result is ensured.

[0052] Step S205, input the first retrieval result and the second image data or the first retrieval result and the first text data into the LLM agent model for evaluation, and in the case of passing the evaluation, determine the first retrieval result as the target retrieval result. Wherein, the evaluation includes one or more of relevance evaluation, confidence evaluation and consistency evaluation.

[0053] The first retrieval result and the user query data (image or text) are input into the large language model (LLM Agent) for evaluation. The evaluation content includes the relevance, confidence and consistency of the search result. In the case of passing the evaluation, the target retrieval result is output, otherwise it is re-searched or corrected through the LLM Agent model. By introducing the evaluation and adjustment mechanism of the LLM Agent, the accuracy and reliability of the search result are further improved. As shown in Figure 3 Figure 3 is an application scenario diagram of image data storage and retrieval based on a lightweight multi-modal model according to an embodiment of the present application:

[0054] ​It can be seen that the embodiment of the present application provides an image data storage and retrieval method based on a lightweight multi-modal model. First, based on the characteristics and application scenarios of the first image data, a plurality of first feature categories are obtained. The first image data is image data to be stored. Then, at least based on a lightweight CLIP model, the first image data is feature extracted to obtain a first target feature. The first target feature is clustered based on the first feature category to obtain a plurality of feature clustering clusters, and the feature clustering clusters are stored in a vector database. The lightweight CLIP model is a CLIP model combining 1D convolution and attention layer. Then, the second image data and / or the first text data are obtained, and the second feature category is obtained. The second image data is a query image input by a user, the first text data is a query text input by a user, and the second feature category is a query feature category input by a user and includes one or more of the first feature categories. Then, at least based on the lightweight CLIP model, the second image data and / or the first text data are feature extracted to obtain a second target feature. The corresponding feature clustering cluster is determined based on the second feature category to obtain a target clustering cluster, and the second target feature is matched in the target clustering cluster to obtain a first retrieval result. Finally, the first retrieval result and the second image data or the first retrieval result and the first text data are input into an LLM agent model for evaluation. If the evaluation is passed, the first retrieval result is determined as a target retrieval result. The evaluation includes one or more of relevance evaluation, confidence evaluation and consistency evaluation. The present application reduces the computational complexity of the CLIP model by introducing the lightweight CLIP model and the feature clustering storage strategy, and improves the feature storage and retrieval efficiency. By dividing the feature clustering cluster before storing the feature, the retrieval matching is reduced according to the clustering cluster in the retrieval stage, and more accurate retrieval is realized. The introduction of LLM agent judgment and LLM agent model re-retrieval improves the relevance and consistency of the retrieval result, and solves the problem of high delay and insufficient precision in the prior art.

[0055] As a possible implementation manner, at least based on the lightweight CLIP model, the first image data is feature extracted to obtain a first target feature, including the following steps:

[0056] Step S301, in the case that the feature clustering cluster is divided according to the features of the first image data, the first image data is segmented by a SAM model to obtain a plurality of third image data. As shown in Figure 4 Figure 4 is a scene schematic diagram of segmenting the first image data by the SAM model according to the embodiment of the present application:

[0057] ​Specifically, based on the division of the feature clustering cluster, the first image data is segmented using a segmented attention model (SAM, Segment Anything Model) to decompose the image into multiple fine-grained third image data (such as objects, regions, backgrounds, etc.). The SAM model has high general segmentation capability and can automatically identify and extract key regions and features in the image. The image segmentation refines the feature expression of the image data, ensures high-quality extraction of key regions and features, improves the refinement level of feature clustering, and lays a more accurate foundation for subsequent feature extraction and matching.

[0058] Step S302, based on the first image data and the third image data, respectively through the lightweight CLIP model for feature extraction, to obtain the corresponding first target feature.

[0059] Specifically, the original image data (first image data) and the segmented fine-grained image data (third image data) are respectively extracted by the lightweight CLIP model to generate high-dimensional feature vectors. The lightweight CLIP model combines 1D convolution and attention mechanism, which can efficiently capture global and local features of the image, while reducing the consumption of computing resources. The use of global features and local detail features improves the comprehensiveness and discrimination of feature representation. The lightweight design reduces the computational complexity.

[0060] Therefore, the embodiment of the present application introduces the SAM model to segment the first image data based on the division of the feature clustering cluster, generates fine-grained third image data, and combines the lightweight CLIP model to extract features from the original image data and the segmented image data. The method improves the accuracy and richness of feature extraction by refining the feature expression. The use of lightweight design effectively reduces the computational complexity and resource consumption, solves the problem of high dependence on computing resources for feature extraction and insufficient detail in image feature expression in the prior art, and provides technical support for efficient implementation of multi-modal data storage and retrieval.

[0061] As a possible implementation manner, based on the first image data and the third image data, respectively through the lightweight CLIP model for feature extraction, to obtain the corresponding first target feature, including the following steps:

[0062] Step S401, based on the first image data, through the lightweight CLIP model for feature extraction, to obtain the third target feature.

[0063] Specifically, the first image data (original image) is input into the lightweight CLIP model for feature extraction to generate third target features. The lightweight CLIP model effectively extracts global features of the image by using its 1D convolution combined with attention layers, forming a high-dimensional feature representation. The extraction of global features of the image ensures the integrity of the overall image information, providing a comprehensive basis for subsequent feature clustering and retrieval.

[0064] In step S402, based on each third image data, feature extraction is performed by a lightweight CLIP model respectively to obtain fourth target features.

[0065] Specifically, each third image data (fine-grained image part) generated by the SAM model is input into the lightweight CLIP model for feature extraction to generate corresponding fourth target features. These features represent local information and key region characteristics of the image. The extraction of local features for fine-grained image parts makes up for the details that may be ignored by global features, enhancing the delicacy and discrimination of feature expression.

[0066] In step S403, target labels are added to the third target features and the fourth target features to obtain first target features, and the target labels are used to cluster the third target features and the fourth target features into different feature clustering clusters. As shown in Figure 5 Figure 5 is a scene diagram of feature clustering cluster clustering according to an embodiment of the present application:

[0067] Specifically, to distinguish between global features and local features, target labels are added to the third target features (global features) and the fourth target features (local features). Based on the feature distinguishing function of the target labels, the global features and the local features are divided into different feature clustering clusters to form an explicit feature storage structure. Through the introduction of the labels, the clustering attribution of the global and local features is clear, the structure and targeting of feature clustering are improved, the feature confusion problem is reduced, and the accuracy of retrieval matching is improved.

[0068] Therefore, the embodiments of the present application form a hierarchical storage and retrieval mechanism of features by extracting global features and local features of the first image data respectively and adding target labels. The extraction of global features ensures the integrity of the overall image information, the extraction of local features enhances the expression of detailed information, and the target labels are used to distinguish and cluster into different feature clustering clusters, optimizing the organization structure of the features. The overall scheme improves the retrieval efficiency while improving the accuracy and discrimination of feature matching, solves the problem of poor retrieval effect caused by confusion of global and local features in the prior art, and provides an efficient and accurate solution for multi-modal data storage and retrieval.

[0069] ​As a possible implementation, after the first search result and the second image data or the first search result and the first text data are input into the LLM agent model for evaluation, the following steps are included:

[0070] Step S501, if the input data of the LLM agent model is the first search result and the second image data, the second text data is obtained in the case of evaluation failure.

[0071] The second text data is query text supplemented by the user.

[0072] Specifically, when the first search result and the second image data input by the user fail to pass the evaluation of the LLM agent model, the system guides the user to supplement the query text (second text data). The supplemented text can be a further description of the query image, such as the color, shape, and purpose of the target, which enriches the user's query information. By actively supplementing the query text by the user, the flexibility and accuracy of the query are improved, and the risk of retrieval failure due to insufficient initial query information is reduced.

[0073] Step S502, processing the second text data based on the LLM agent model to generate target description information, the target description information including color, shape, relationship, and context information.

[0074] Specifically, the LLM agent model is used to process the second text data supplemented by the user to extract the description information of the target. The target description information covers color, shape, relationship between objects, and related context information, helping to build more comprehensive query conditions. By mining detailed description information in the user-supplemented text, the semantic expression of retrieval features is enriched, and the accuracy of matching is improved.

[0075] Step S503, processing each third image data based on the target description information through the LLM agent model to generate association description information, the association description information being used to represent the relationship between the target description information of different third image data.

[0076] Specifically, the LLM agent model processes the third image data based on the target description information to generate association description information. The association description information represents the semantic relationship (such as adjacency, containment relationship, and contrast relationship) between different regions or objects in the image data. By generating the association description information, the semantic understanding ability of image retrieval is enhanced, providing support for high-precision retrieval in complex scenarios.

[0077] Step S504, initializing a feature search graph based on the association description information through the LLM agent model, and generating a first sub-problem based on each node in the feature search graph.

[0078] Specifically, a feature search graph is constructed based on the associated description information, with nodes representing different features or target regions. Through structured analysis of the search graph, first sub-questions are generated node by node, each corresponding to a specific feature retrieval target. The feature search graph organizes the hierarchical structure of the retrieval task, decomposes complex queries, and improves retrieval efficiency and accuracy.

[0079] In step S505, the vector database is matched by the lightweight CLIP model based on each first sub-question, the data corresponding to the maximum relevance is determined as the second retrieval result, and the second retrieval result and the second image data are evaluated by the LLM agent model. If the evaluation is passed, the second retrieval result is determined as the target retrieval result, and one first sub-question corresponds to one second retrieval result.

[0080] Specifically, for each first sub-question, the lightweight CLIP model performs feature matching in the vector database to find the highest correlation feature and determine the second retrieval result. Then, the second retrieval result and the user's second image data are evaluated again by the LLM agent model to ensure the accuracy and consistency of the matching result. The fine retrieval process gradually optimizes the matching accuracy, and the multi-round evaluation mechanism improves the reliability of the final retrieval result.

[0081] Therefore, the embodiments of the present application guide the user to supplement the query text, extract target description information and generate associated description information using the LLM agent model, further construct a feature search graph and decompose complex query tasks. Through hierarchical retrieval and multi-round evaluation mechanism, the retrieval accuracy and efficiency of image and text data in the vector database are significantly improved. This method fully utilizes the advantages of user participation and multi-modal feature expression, solves the problems of retrieval failure and inaccurate matching in the prior art, and is suitable for high-precision multi-modal data retrieval requirements in complex scenarios.

[0082] As a possible implementation, the evaluation based on the second retrieval result and the second image data by the LLM agent model includes the following steps:

[0083] In step S601, the second retrieval result is processed by the LLM agent model to obtain search response information corresponding to each node.

[0084] Specifically, the LLM agent model is used for semantic analysis and feature extraction of the second retrieval result, and search response information related to the nodes of the feature search graph is generated. The search response information includes semantic features, matching scores or context information related to the retrieval result, further enriching the description content of the nodes.

[0085] Step S602, each search response information is added as a new node to update the feature search graph.

[0086] Specifically, the generated search response information is integrated into the feature search graph, and new nodes are added to expand the structure of the graph. These new nodes not only contain feature information of the second search results, but also maintain contextual association with the original nodes, forming a more complete search graph network. The addition of new nodes dynamically expands the feature search graph, optimizes the representation ability of the graph, enables it to capture more comprehensive feature relationships, and further improves the accuracy and flexibility of the search.

[0087] Step S603, based on the feature search graph and the second image data, evaluation is performed through the LLM agent model.

[0088] Specifically, based on the updated feature search graph, the LLM agent model is combined with the second image data to perform evaluation. This evaluation process considers the feature information of the nodes in the graph and their semantic relationship with the second image data, performs correlation analysis, confidence judgment, etc., to determine whether the final matching result meets the user query intent. By combining the updated search graph and user input data for comprehensive evaluation, the reliability of the final matching result is improved, ensuring that the search result is highly consistent with the user intent.

[0089] Therefore, the embodiments of the present application expand the nodes and their feature expression ability of the search graph by performing semantic processing on the second search results through the LLM agent model, generating search response information and dynamically integrating it into the feature search graph. At the same time, by combining the user input second image data, the LLM agent model performs comprehensive evaluation on the expanded search graph, effectively improving the semantic relevance and confidence of the search results. This method optimizes the dynamic updating and semantic evaluation mechanism of the feature search graph, solves the problem of inaccurate evaluation caused by insufficient features of search results in the prior art, and significantly improves the applicability and robustness of the search system in complex scenarios.

[0090] As a possible implementation manner, after inputting the first search result and the second image data or the first search result and the first text data into the LLM agent model for evaluation, the following steps are included:

[0091] Step S701, if the input data of the LLM agent model is the first search result and the first text data, then in the case of evaluation failure, the first text data is processed through the LLM agent model to obtain a second sub-problem.

[0092] Specifically, when the LLM agent model's evaluation of the relevance of the first search result to the first text data fails, the model will perform in-depth semantic analysis on the first text data and generate one or more second sub-questions. These sub-questions can be a refinement or an extended expression of the user's query intent, aiming to clarify the search target and improve the accuracy of search matching. By generating sub-questions, the search question that fails the evaluation is decomposed into more specific tasks, enabling the model to focus on a more explicit search direction, thereby optimizing the accuracy and relevance of subsequent search results.

[0093] At step S702, based on each second sub-question, the vector database is matched by a lightweight CLIP model to determine the data corresponding to the maximum relevance value as the third search result. The third search result is evaluated by the LLM agent model based on the third search result and the first text data, and in the case of passing the evaluation, the third search result is determined as the target search result. One second sub-question corresponds to one third search result.

[0094] Specifically, for each generated second sub-question, a lightweight CLIP model is used to match from the vector database. By calculating the similarity between each vector and the sub-question feature, the vector corresponding to the maximum relevance value is determined, and the data associated with it is taken as the third search result. Through the targeted matching of the lightweight CLIP model, it is ensured that each sub-question can find the most relevant database content, greatly improving the refinement level of the search and the accuracy of the results. The third search result and the first text data are input into the LLM agent model again for evaluation, and a comprehensive analysis is performed by comparing the semantic relevance, confidence and consistency of the results. When the evaluation passes, the corresponding third search result is output as the final target search result. The evaluation process of each sub-question is independent to ensure that all targets get the optimal search result. Through multiple rounds of evaluation and matching, the accuracy and reliability of the target search result are further guaranteed, avoiding search deviations caused by initial query ambiguity or insufficient feature matching.

[0095] Therefore, in the case where the search evaluation fails, the embodiments of the present application perform semantic analysis on the first text data through the LLM agent model, generate clear second sub-questions, combine the lightweight CLIP model to perform targeted matching on the database content, and further optimize the search results through multiple rounds of evaluation. This method effectively decomposes user intent in ambiguous query scenarios, improves search accuracy and matching efficiency in complex query scenarios, solves the problem of unsatisfactory search results caused by query ambiguity or insufficient initial matching in the prior art, and significantly enhances the adaptability and robustness of the multi-modal search system.

[0096] As a possible implementation, the second target feature is matched in the target clustering cluster to obtain the first search result. Specifically, in the case that the number of target clustering clusters is greater than a preset value, the matching process of different target clustering clusters is allocated to different threads for operation.

[0097] Specifically, when the number of target clustering clusters exceeds the preset value, in order to improve the matching efficiency, the system will process the matching process of each target clustering cluster in parallel. Specifically, different target clustering clusters are allocated to multiple threads for calculation, thereby realizing parallel operation. Each thread is responsible for matching data in one or more target clustering clusters, accelerating the entire search process. By allocating the matching tasks of different clustering clusters to different threads for parallel operation when the number of target clustering clusters exceeds the preset value, the search speed and system efficiency are greatly improved. This method effectively solves the possible calculation bottleneck problem when a large number of clustering clusters are processed, and significantly speeds up the search process.

[0098] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0099] The embodiment of the present application also provides an image data storage and retrieval device based on a lightweight multi-modal model. It should be noted that the image data storage and retrieval device based on a lightweight multi-modal model of the embodiment of the present application can be used to execute the image data storage and retrieval method based on a lightweight multi-modal model provided by the embodiment of the present application. The device is used to realize the above-mentioned embodiments and preferred embodiments, which have been described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware implementation is also possible and contemplated.

[0100] The image data storage and retrieval device based on a lightweight multi-modal model provided by the embodiment of the present application is described below.

[0101] Figure 6 is a structural block diagram of the image data storage and retrieval device based on a lightweight multi-modal model according to the embodiment of the present application. As Figure 6 shown, the device includes a first processing unit 10, a second processing unit 20, a first acquisition unit 30, a third processing unit 40, and a fourth processing unit 50.

[0102] The first processing unit 10 is configured to classify based on the characteristics and application scenarios of the first image data to obtain a plurality of first feature categories, and the first image data is image data to be stored.

[0103] The second processing unit 20 is configured to perform feature extraction on the first image data based on at least a lightweight CLIP model to obtain first target features, perform clustering on the first target features based on the first feature categories to obtain a plurality of feature clustering clusters, and store the feature clustering clusters in a vector database, wherein the lightweight CLIP model is a CLIP model combined with a 1D convolution and an attention layer.

[0104] The first obtaining unit 30 is configured to obtain second image data and / or first text data, and obtain second feature categories, wherein the second image data is a query image input by a user, the first text data is a query text input by the user, and the second feature categories include one or more of the first feature categories.

[0105] The third processing unit 40 is configured to perform feature extraction on the second image data and / or the first text data based on at least the lightweight CLIP model to obtain second target features, determine corresponding feature clustering clusters based on the second feature categories to obtain target clustering clusters, and perform matching in the target clustering clusters based on the second target features to obtain a first retrieval result.

[0106] The fourth processing unit 50 is configured to input the first retrieval result and the second image data or the first retrieval result and the first text data into an LLM agent model for evaluation, and determine the first retrieval result as a target retrieval result when the evaluation is passed, wherein the evaluation includes one or more of relevance evaluation, confidence evaluation, and consistency evaluation.

[0107] Embodiments of the present application provide an image data storage and retrieval device based on a lightweight multi-modal model, which comprises a first processing unit, a second processing unit, a first obtaining unit, a third processing unit, and a fourth processing unit. The device introduces a lightweight CLIP model and a feature clustering storage strategy, divides feature clustering clusters before feature storage, stores according to the clustering clusters, reduces retrieval matching in the retrieval stage, introduces LLM agent judgment and LLM agent model re-retrieval, and solves the problem of low efficiency caused by low delay in the prior art.

[0108] As a possible implementation manner, the second processing unit comprises:

[0109] The segmentation module is configured to segment the first image data by a SAM model to obtain a plurality of third image data in a case where the feature clustering clusters are divided according to features of the first image data.

[0110] The feature extraction module is configured to perform feature extraction on the first image data and the third image data respectively by using the light CLIP model to obtain corresponding first target features.

[0111] As a possible implementation manner, the feature extraction module comprises:

[0112] The first extraction submodule is configured to perform feature extraction on the first image data by using the light CLIP model to obtain the third target features.

[0113] The second extraction submodule is configured to perform feature extraction on each of the third image data by using the light CLIP model to obtain fourth target features.

[0114] The first target feature acquisition submodule is configured to add a target identifier to the third target features and the fourth target features to obtain the first target features, and the target identifier is used to cluster the third target features and the fourth target features to different feature clustering clusters.

[0115] As a possible implementation manner, the device further comprises:

[0116] The second acquisition unit is configured to, if the input data of the LLM agent model is the first retrieval result and the second image data, acquire second text data in the case of failing to pass the evaluation, and the second text data is query text supplemented by the user.

[0117] The target description information generation unit is configured to process the second text data by using the LLM agent model to generate target description information, and the target description information comprises color, shape, relationship and context information.

[0118] The association description information generation unit is configured to process each of the third image data by using the LLM agent model based on the target description information to generate association description information, and the association description information is used to represent the relationship between the target description information of different third image data.

[0119] The first sub-problem generation unit is configured to initialize a feature search graph by using the LLM agent model based on the association description information, and generate a first sub-problem based on each node in the feature search graph.

[0120] The first target retrieval result determination unit is configured to match the vector database by using the light CLIP model based on each of the first sub-problems, determine data corresponding to a maximum correlation value as a second retrieval result, and perform evaluation by using the LLM agent model based on the second retrieval result and the second image data, and in the case of passing the evaluation, determine the second retrieval result as the target retrieval result, and one first sub-problem corresponds to one second retrieval result.

[0121] As a possible implementation manner, the target retrieval result determination unit comprises:

[0122] The search response information acquisition module is configured to process the second retrieval result by using the LLM agent model to obtain search response information corresponding to each node.

[0123] The feature search graph updating module is configured to add each search response information as a new node to update the feature search graph.

[0124] The evaluation module is configured to evaluate by using the LLM agent model based on the feature search graph and the second image data.

[0125] As a possible implementation manner, the device further comprises:

[0126] The second sub-problem generation unit is configured to, if the input data of the LLM agent model is the first retrieval result and the first text data, process the first text data by using the LLM agent model to obtain a second sub-problem in the case of failing to pass the evaluation.

[0127] The second target retrieval result determination unit is configured to match the vector database by using the light CLIP model based on each second sub-problem, determine data corresponding to the maximum relevance as a third retrieval result, and evaluate by using the LLM agent model based on the third retrieval result and the first text data, and determine the third retrieval result as the target retrieval result in the case of passing the evaluation, one second sub-problem corresponding to one third retrieval result.

[0128] As a possible implementation manner, the third processing unit is specifically configured to, in the case that the number of target clustering clusters is greater than a preset value, distribute the matching processes of different target clustering clusters to different threads for operation.

[0129] The above image data storage and retrieval device based on the light multi-modal model comprises a processor and a memory, and the above units and the like are stored in the memory as program units, and the corresponding functions are realized by the processor executing the above program units stored in the memory. The above modules are located in the same processor. Alternatively, the above modules are located in different processors in any combination.

[0130] The processor comprises a core, and the core retrieves the corresponding program unit from the memory. The core can be one or more, and the communication efficiency can be improved by adjusting the core parameters.

[0131] The memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.

[0132] The embodiment of the present application provides a computer readable storage medium, the computer readable storage medium comprises a stored program, wherein the program controls a device where the computer readable storage medium is located to execute the image data storage and retrieval method based on a lightweight multi-modal model when the program runs.

[0133] The embodiment of the present application provides a processor, the processor is used for running a program, wherein the program executes the image data storage and retrieval method based on a lightweight multi-modal model when the program runs.

[0134] The present application also provides a computer program product, when executed on a data processing device, is suitable for executing the program initialized with the image data storage and retrieval method based on a lightweight multi-modal model.

[0135] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be realized by general computing devices, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, which can be realized by program codes executable by computing devices, so that they can be stored in storage devices and executed by computing devices, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps can be manufactured into a single integrated circuit module. Therefore, the present application is not limited to any specific combination of hardware and software.

[0136] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0137] The present application is described with reference to flowcharts and / or block diagrams according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device for implementing the functions described in the flowcharts and / or block diagrams. Figure 1 one flow or multiple flows and / or blocks Figure 1an apparatus to perform each block or blocks of the flow or multiple flows and / or a function specified by each block or blocks of the flow.

[0138] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a Figure 1 an apparatus to perform each block or blocks of the flow or multiple flows and / or a function specified by each block or blocks of the flow. Figure 1 an apparatus to perform each block or blocks of the flow or multiple flows and / or a function specified by each block or blocks of the flow.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 an apparatus to perform each block or blocks of the flow or multiple flows and / or a function specified by each block or blocks of the flow. Figure 1 an apparatus to perform each block or blocks of the flow or multiple flows and / or a function specified by each block or blocks of the flow.

[0140] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0141] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the computer stores information about an operating system, application software, and / or the like. Memory is an example of computer readable media.

[0142] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic disks storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "comprising", "comprises" or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0144] From the above description, it can be seen that the above-mentioned embodiments of the present application achieve the following technical effects:

[0145] 1) The image data storage and retrieval method based on the lightweight multi-modal model of the present application introduces a lightweight CLIP model and a feature clustering storage strategy, divides feature clustering clusters before feature storage, stores according to the clustering clusters, reduces retrieval matching in the retrieval stage, introduces LLM agent judgment and LLM agent model re-retrieval, and solves the problem of low efficiency caused by low delay and high retrieval of multi-modal data in the vector database in the prior art.

[0146] 2) The image data storage and retrieval device based on the lightweight multi-modal model of the present application, the device comprises a first processing unit, a second processing unit, a first acquisition unit, a third processing unit, and a fourth processing unit. The device introduces a lightweight CLIP model and a feature clustering storage strategy, divides feature clustering clusters before feature storage, stores according to the clustering clusters, reduces retrieval matching in the retrieval stage, introduces LLM agent judgment and LLM agent model re-retrieval, and solves the problem of low efficiency caused by low delay and high retrieval of multi-modal data in the vector database in the prior art.

[0147] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for image data storage and retrieval based on lightweight multi-modal model, characterized in that, The method comprises: classifying based on characteristics of first image data and application scenarios to obtain a plurality of first feature categories, the first image data being image data to be stored; performing feature extraction on the first image data based on at least a lightweight CLIP model to obtain first target features, clustering the first target features based on the first feature categories to obtain a plurality of feature clustering clusters, and storing the feature clustering clusters to a vector database, the lightweight CLIP model being a CLIP model combining 1D convolution and attention layers, and the feature clustering clusters corresponding one-to-one to the first feature categories; obtaining second image data and / or first text data, obtaining second feature categories, the second image data being query images input by a user, the first text data being query text input by a user, and the second feature categories being query feature categories input by a user and including one or more of the first feature categories; performing feature extraction on the second image data and / or the first text data based on at least the lightweight CLIP model to obtain second target features, determining corresponding feature clustering clusters based on the second feature categories to obtain target clustering clusters, and matching the second target features in the target clustering clusters to obtain first retrieval results; inputting the first retrieval results and the second image data or the first retrieval results and the first text data into an LLM agent model for evaluation, in a case where the evaluation is passed, determining the first retrieval results as target retrieval results, and the evaluation including one or more of relevance evaluation, confidence evaluation, and consistency evaluation.

2. The method of claim 1, wherein, performing feature extraction on the first image data based on at least a lightweight CLIP model to obtain first target features, comprising: in a case where the feature clustering clusters are divided according to characteristics of the first image data, segmenting the first image data by a SAM model to obtain a plurality of third image data; performing feature extraction on the first image data and the third image data by the lightweight CLIP model respectively to obtain corresponding first target features.

3. The method of claim 2, wherein, performing feature extraction on the first image data and the third image data by the lightweight CLIP model respectively to obtain corresponding first target features, comprising: performing feature extraction on the first image data by the lightweight CLIP model to obtain third target features; performing feature extraction on each of the third image data by the lightweight CLIP model respectively to obtain fourth target features; adding target identifiers to the third target features and the fourth target features to obtain the first target features, the target identifiers being used to cluster the third target features and the fourth target features to different feature clustering clusters.

4. The method of claim 2, wherein, After inputting the first retrieval results and the second image data or the first retrieval results and the first text data into an LLM agent model for evaluation, the method further comprises: If the input data of the LLM agent model is the first search result and the second image data, if the evaluation does not pass, obtain second text data, the second text data is a query text supplemented by a user; Based on the LLM agent model, the second text data is processed to generate target description information, and the target description information includes color, shape, relationship and context information; Based on the target description information, the LLM agent model is used to process each third image data to generate association description information, and the association description information is used to represent the relationship between the target description information of different third image data; Based on the association description information, the LLM agent model initializes a feature search graph, and generates a first sub-problem based on each node in the feature search graph; Based on each first sub-problem, the lightweight CLIP model is used to match the vector database, the data corresponding to the maximum relevance is determined as a second search result, and the LLM agent model is used to evaluate the second search result and the second image data, if the evaluation passes, the second search result is determined as a target search result, and one first sub-problem corresponds to one second search result.

5. The method of claim 4, wherein, Based on the second search result and the second image data, the LLM agent model is used to evaluate, including: The LLM agent model is used to process the second search result to obtain search response information corresponding to each node; Each search response information is added as a new node to update the feature search graph; Based on the feature search graph and the second image data, the LLM agent model is used to evaluate.

6. The method of claim 4, wherein, After inputting the first search result and the second image data or the first search result and the first text data into the LLM agent model for evaluation, the method further includes: If the input data of the LLM agent model is the first search result and the first text data, if the evaluation does not pass, the LLM agent model is used to process the first text data to obtain a second sub-problem; Based on each second sub-problem, the lightweight CLIP model is used to match the vector database, the data corresponding to the maximum relevance is determined as a third search result, and the LLM agent model is used to evaluate the third search result and the first text data, if the evaluation passes, the third search result is determined as a target search result, and one second sub-problem corresponds to one third search result.

7. The method of claim 6, wherein, Based on the second target feature, the target clustering cluster is matched to obtain a first search result, including: If the number of target clustering clusters is greater than a preset value, the matching process of different target clustering clusters is allocated to different threads for operation.

8. An image data storage and retrieval apparatus based on a lightweight multi-modal model, characterized in that, The device comprises: A first processing unit for classifying based on characteristics of first image data and application scenarios to obtain a plurality of first feature categories, the first image data being image data to be stored; A second processing unit for performing feature extraction on the first image data based on at least a lightweight CLIP model to obtain first target features, clustering the first target features based on the first feature categories to obtain a plurality of feature clustering clusters, and storing the feature clustering clusters to a vector database, the lightweight CLIP model being a CLIP model combining 1D convolution and attention layers, and the feature clustering clusters corresponding one-to-one to the first feature categories; A first acquisition unit for acquiring second image data and / or first text data, and acquiring second feature categories, the second image data being query images input by a user, the first text data being query text input by a user, and the second feature categories being query feature categories input by a user and including one or more of the first feature categories; A third processing unit for performing feature extraction on the second image data and / or the first text data based on at least the lightweight CLIP model to obtain second target features, determining corresponding feature clustering clusters based on the second feature categories to obtain target clustering clusters, and matching the second target features in the target clustering clusters to obtain first retrieval results; A fourth processing unit for inputting the first retrieval results and the second image data or the first retrieval results and the first text data into an LLM agent model for evaluation, determining the first retrieval results as target retrieval results in the case that the evaluation is passed, and the evaluation including one or more of relevance evaluation, confidence evaluation, and consistency evaluation.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises a stored program, wherein the program controls a device where the computer-readable storage medium is located to execute the method of any one of claims 1 to 7 when the program is running.

10. A multi-modal data storage system, characterized by, Comprise: One or more processors, memories, and one or more programs, wherein the one or more programs are stored in the memories and configured to be executed by the one or more processors, and the one or more programs comprise programs for executing the method of any one of claims 1 to 7.

Images