Image data storage and retrieval method and device based on lightweight multi-modal model

Abstract

The invention provides an image data storage and retrieval method and device based on a lightweight multi-modal model, a computer storage medium and a processor. According to the method, the lightweight CLIP model and the feature clustering storage strategy are introduced, so that the calculation complexity of the CLIP model is reduced, and the feature storage and retrieval efficiency is improved. According to the method, feature clustering clusters are divided before feature storage, storage is carried out according to the clustering clusters, retrieval matching is reduced in the retrieval stage, and more accurate retrieval is achieved. According to the method, the LLM agent judgment and the LLM agent model are introduced for re-retrieval, so that the correlation and the consistency of retrieval results are improved, and therefore, the problem of relatively low efficiency caused by relatively high retrieval delay of the multi-modal data in the vector database in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of data communication technology, and in particular to an image data storage and retrieval method based on a lightweight multimodal model, and an image data storage and retrieval device based on a lightweight multimodal model, a computer storage medium and a processor. Background Art

[0002] With the explosive growth of Internet information, users have higher and higher requirements for the accuracy and relevance of search results. Traditional search methods often rely on single text or image information, which is difficult to meet the diverse needs of users for search results. To solve this problem, multimodal search technology came into being. Multimodal search combines multiple information sources such as text and images, and achieves a more accurate and diversified search experience through the efficient collaboration of multimodal models. Among them, the CLIP model, as a representative of current multimodal technology, is composed of an image encoder (Visual Encoder), a text encoder (TextEncoder), and a multimodal contrastive learning part that is trained together.

[0003] In terms of data storage and retrieval, vector databases, as supporting technologies, undertake the task of efficiently storing, indexing, and retrieving multimodal features. Vector databases use vectors as basic data structures and are suitable for representing entities with multiple attributes (such as text, images, and audio). By building an efficient vector index structure, vector databases can achieve fast similarity searches in massive data. However, there is currently a problem of low efficiency due to high latency in retrieving multimodal data in vector databases. Summary of the invention

[0004] The main purpose of the present application is to provide an image data storage and retrieval method based on a lightweight multimodal model, and an image data storage and retrieval device, a computer storage medium and a processor based on a lightweight multimodal model, so as to at least solve the problem of low efficiency caused by high latency in the retrieval of multimodal data in a vector database in the prior art.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for storing and retrieving image data based on a lightweight multimodal model is provided, including: classifying based on the characteristics and application scenarios of first image data to obtain multiple first feature categories, and the first image data is the image data to be stored; extracting features of the first image data at least based on a lightweight CLIP model to obtain a first target feature, clustering the first target feature based on the first feature category to obtain multiple feature clustering clusters, and storing the feature clustering clusters in a vector database, and the lightweight CLIP model is a CLIP model that combines 1D convolution with an attention layer; obtaining second image data and / or first text data, obtaining second feature category, the second image data is a query image input by a user, the first text data is a query text input by a user, the second feature category is a query feature category input by a user and the second feature category includes one or more of the first feature categories; feature extraction is performed on the second image data and / or the first text data at least based on the lightweight CLIP model to obtain a second target feature, the corresponding feature clustering cluster is determined based on the second feature category to obtain a target clustering cluster, and matching is performed in the target clustering cluster based on the second target feature to obtain a first retrieval result; the first retrieval result and the second image data or the first retrieval result and the first text data are input into the LLM agent model for evaluation, and if the evaluation passes, the first retrieval result is determined as the target retrieval result, and the evaluation includes one or more of relevance evaluation, confidence evaluation and consistency evaluation.

[0006] Optionally, feature extraction is performed on the first image data at least based on the lightweight CLIP model to obtain a first target feature, including: when the feature clustering clusters are divided according to the features of the first image data, the first image data is segmented through a SAM model to obtain a plurality of third image data; based on the first image data and the third image data, feature extraction is performed through the lightweight CLIP model respectively to obtain the corresponding first target feature.

[0007] Optionally, feature extraction is performed through the lightweight CLIP model based on the first image data and the third image data to obtain the corresponding first target feature, including: based on the first image data, feature extraction is performed through the lightweight CLIP model to obtain a third target feature; based on each of the third image data, feature extraction is performed through the lightweight CLIP model to obtain a fourth target feature; and a target identifier is added to the third target feature and the fourth target feature to obtain the first target feature, wherein the target identifier is used to cluster the third target feature and the fourth target feature into different feature clustering clusters.

[0008] Optionally, after the first retrieval result and the second image data or the first retrieval result and the first text data are input into the LLM agent model for evaluation, the method further includes: if the input data of the LLM agent model is the first retrieval result and the second image data, then if the evaluation fails, obtaining second text data, the second text data being the query text supplemented by the user; processing the second text data based on the LLM agent model to generate target description information, the target description information including color, shape, relationship and context information; processing each of the third image data through the LLM agent model based on the target description information to generate associated description information, the associated description information being used to characterize the relationship between the target description information of different third image data; initializing a feature search graph through the LLM agent model based on the associated description information, and generating first sub-problems based on each node in the feature search graph; matching the vector database through the lightweight CLIP model based on each of the first sub-problems, determining the data corresponding to the maximum correlation value as the second retrieval result, and matching the vector database based on the second retrieval result and the second image data through the LLM The evaluation is performed in the agent model, and if the evaluation passes, the second search result is determined as the target search result, and one first sub-question corresponds to one second search result.

[0009] Optionally, the evaluation is performed through the LLM agent model based on the second retrieval result and the second image data, including: processing the second retrieval result through the LLM agent model to obtain search response information corresponding to each node; adding each search response information as a new node to update the feature search graph; and performing the evaluation through the LLM agent model based on the feature search graph and the second image data.

[0010] Optionally, after inputting the first retrieval result and the second image data or the first retrieval result and the first text data into the LLM agent model for evaluation, the method further includes: if the input data of the LLM agent model is the first retrieval result and the first text data, then if the evaluation fails, processing the first text data through the LLM agent model to obtain a second sub-problem; matching the vector database through the lightweight CLIP model based on each of the second sub-problems, determining the data corresponding to the maximum correlation value as the third retrieval result, and performing the evaluation through the LLMagent model based on the third retrieval result and the first text data; if the evaluation passes, determining the third retrieval result as the target retrieval result, and one second sub-problem corresponds to one third retrieval result.

[0011] Optionally, matching the target clusters based on the second target feature to obtain a first search result includes: when the number of the target clusters is greater than a preset value, allocating matching processes of different target clusters to different threads for calculation.

[0012] According to another aspect of the present application, there is provided an image data storage and retrieval device based on a lightweight multimodal model, the device comprising: a first processing unit, for classifying based on the characteristics and application scenarios of first image data to obtain a plurality of first feature categories, wherein the first image data is image data to be stored; a second processing unit, for extracting features of the first image data at least based on a lightweight CLIP model to obtain a first target feature, clustering the first target feature based on the first feature category to obtain a plurality of feature clustering clusters, and storing the feature clustering clusters in a vector database, wherein the lightweight CLIP model is a CLIP model that combines 1D convolution with an attention layer; a first acquisition unit, for acquiring second image data and / or first text data, and acquiring second feature category, the second image data is a query image input by a user, the first text data is a query text input by a user, the second feature category is a query feature category input by a user and the second feature category includes one or more of the first feature categories; a third processing unit, used to perform feature extraction on the second image data and / or the first text data based at least on the lightweight CLIP model to obtain a second target feature, determine the corresponding feature clustering cluster based on the second feature category to obtain a target clustering cluster, and match the target clustering cluster based on the second target feature to obtain a first retrieval result; a fourth processing unit, used to input the first retrieval result and the second image data or the first retrieval result and the first text data into an LLM agent model for evaluation, and if the evaluation passes, determine the first retrieval result as a target retrieval result, the evaluation including one or more of relevance evaluation, confidence evaluation and consistency evaluation.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the image data storage and retrieval methods based on the lightweight multimodal model.

[0014] According to another aspect of the present application, a multimodal data storage system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the lightweight multimodal model-based image data storage and retrieval methods.

[0015] By applying the technical solution of the present application, the computational complexity of the CLIP model is reduced and the feature storage and retrieval efficiency is improved by introducing a lightweight CLIP model and a feature clustering storage strategy; by dividing the features into cluster clusters before they are stored in the database, the retrieval matching is reduced in the retrieval stage, thereby achieving a more accurate retrieval; by introducing LLMagent judgment and LLM agent model re-retrieval, the relevance and consistency of the retrieval results are improved. Therefore, the present application solves the problem of low efficiency caused by high latency in the retrieval of multimodal data in a vector database in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A hardware structure block diagram of a mobile terminal according to an image data storage and retrieval method based on a lightweight multimodal model provided in an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram of a process of storing and retrieving image data based on a lightweight multimodal model according to an embodiment of the present application is shown;

[0018] Figure 3 A schematic diagram of an application scenario of image data storage and retrieval based on a lightweight multimodal model provided according to an embodiment of the present application is shown;

[0019] Figure 4 A schematic diagram of an application scenario of segmenting the first image data by using a SAM model according to an embodiment of the present application is shown;

[0020] Figure 5 A schematic diagram of an application scenario of feature clustering provided according to an embodiment of the present application is shown;

[0021] Figure 6 A structural block diagram of an image data storage and retrieval device based on a lightweight multimodal model provided according to an embodiment of the present application is shown.

[0022] The above drawings include the following reference numerals:

[0023] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0028] Multimodal Model: refers to a machine learning model that can process and integrate multiple types of data (such as text, images, audio, video, etc.).

[0029] Multimodal Data: involves different types of data sources, such as images (visual modality), text (linguistic modality), audio (auditory modality), and video (spatial-temporal modality).

[0030] CLIP (Contrastive Language-Image Pretraining): A multimodal model developed by OpenAI that can understand and associate natural language with images. It uses contrastive learning methods and is trained on large-scale image-text pairing data, enabling the model to map images and text into the same feature space, thereby enabling mutual retrieval and understanding between images and text.

[0031] The SAM model (SegmentAnything Model) is a powerful multi-purpose model for image segmentation launched by Meta (Facebook's parent company) in 2023. The main goal of SAM is to achieve universal segmentation, that is, it can handle almost any type of image object segmentation task without specialized training for specific objects or fields.

[0032] VLM (Vision-Language Model) is a multimodal generative artificial intelligence model that can combine vision and language processing capabilities to analyze input data such as text and images at the same time. The core of this type of model is to combine the large-scale language model (LLM) with the visual encoder, so that the model has the ability to understand text and "see" images, completing vision-related reasoning and generation tasks.

[0033] Vector Database: A database system specially designed for storing, managing and retrieving high-dimensional vector data.

[0034] Vector Search Algorithms: A class of methods for efficiently retrieving high-dimensional vector data, widely used in image retrieval, text search, recommendation systems, etc. The goal of these algorithms is to find the vector that is most similar to the query vector, that is, to perform an approximate nearest neighbor (ANN) search.

[0035] LLM Agent (Large Language Model Agent) is an intelligent agent based on a large language model (LLM). It combines the powerful natural language understanding and generation capabilities of the large language model to achieve the processing of complex tasks through task planning, tool calling and knowledge expansion. LLMAgent can be used as an independent system or embedded in a larger framework to handle a variety of tasks, such as information retrieval, logical reasoning, and execution of operation instructions. As introduced in the background technology, there is a problem in the prior art that the retrieval of multimodal data in a vector database has a high latency and low efficiency. In order to solve the technical problems existing in the prior art, the embodiments of the present application provide an image data storage and retrieval method based on a lightweight multimodal model, and an image data storage and retrieval device based on a lightweight multimodal model, a computer storage medium and a processor.

[0036] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0037] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of an image data storage and retrieval method based on a lightweight multimodal model according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0038] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] In this embodiment, a method for storing and retrieving image data based on a lightweight multimodal model that runs on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] Figure 2 : is a flowchart of an image data storage and retrieval method based on a lightweight multimodal model according to an embodiment of the present application. Figure 2As shown, the method comprises the following steps:

[0041] Step S201 , classifying the first image data based on its characteristics and application scenarios to obtain a plurality of first feature categories.

[0042] The first image data is image data to be stored.

[0043] Specifically, the image data is divided into multiple feature categories according to the characteristics of the first image data (such as content, style, purpose, etc.) and application scenario requirements (such as medical diagnosis, intelligent security, recommendation system, etc.). Each feature category corresponds to a type of data with common characteristics or application requirements, which facilitates subsequent feature extraction and storage optimization.

[0044] Step S202: extract features from the first image data based on at least a lightweight CLIP model to obtain first target features, cluster the first target features based on first feature categories to obtain multiple feature clustering clusters, and store the feature clustering clusters in a vector database.

[0045] The lightweight CLIP model is a CLIP model that combines 1D convolution with the attention layer.

[0046] Specifically, the lightweight CLIP model is used to efficiently extract features from the first image data, and the extracted features are represented in the form of high-dimensional vectors. According to the first feature category, the extracted target features are clustered to form multiple feature clusters, which are stored in the vector database. The lightweight CLIP model combines 1D convolution with the attention layer to balance computational efficiency and feature representation capabilities. The computational complexity of feature extraction and storage is reduced, and the storage structure of the vector database is optimized through clustering, which reduces the amount of matching calculations in the retrieval stage and improves retrieval efficiency and response speed.

[0047] Step S203, obtaining the second image data and / or the first text data, and obtaining the second feature category.

[0048] The second image data is a query image input by a user, the first text data is a query text input by a user, the second feature category is a query feature category input by a user, and the second feature category includes one or more of the first feature categories.

[0049] Specifically, the user inputs the query data (image or text) and the corresponding feature category. The query feature category can be specified as one or more stored first feature categories to narrow the search scope and improve the search targeting. The user can quickly locate the target feature cluster by specifying the feature category, reduce the matching calculation amount of irrelevant features, and further improve the search efficiency and accuracy.

[0050] Step S204, extract features from the second image data and / or the first text data at least based on the lightweight CLIP model to obtain a second target feature, determine a corresponding feature cluster based on the second feature category to obtain a target cluster, and match the target cluster based on the second target feature to obtain a first retrieval result.

[0051] Specifically, the lightweight CLIP model is used to extract the target features from the query data, and the corresponding feature cluster is located through the second feature category to determine the target cluster. Feature matching calculation is performed only in the target cluster to obtain the first search result. Accurately focusing on the target cluster reduces the search scope and calculation complexity, greatly improves the search speed and resource utilization efficiency, and ensures the relevance of the search results.

[0052] Step S205: input the first search result and the second image data or the first search result and the first text data into the LLM agent model for evaluation, and if the evaluation passes, determine the first search result as the target search result. The evaluation includes one or more of relevance evaluation, confidence evaluation, and consistency evaluation.

[0053] The first search result and the user query data (image or text) are input into the large language model (LLM Agent) for evaluation. The evaluation content includes the relevance, confidence and consistency of the search results. If the evaluation passes, the target search result is output, otherwise it is re-searched or corrected through the LLMAgent model. By introducing the evaluation and adjustment mechanism of LLMAgent, the accuracy and reliability of the search results are further improved. Figure 3 As shown, Figure 3 This is a schematic diagram of an application scenario of image data storage and retrieval based on a lightweight multimodal model according to an embodiment of the present application:

[0054] It can be seen that the embodiment of the present application provides an image data storage and retrieval method based on a lightweight multimodal model. First, the first image data is classified based on the characteristics and application scenarios to obtain multiple first feature categories, and the first image data is the image data to be stored. Then, the first image data is feature extracted at least based on the lightweight CLIP model to obtain the first target feature, the first target feature is clustered based on the first feature category to obtain multiple feature clustering clusters, and the feature clustering clusters are stored in the vector database. The lightweight CLIP model is a CLIP model that combines 1D convolution with an attention layer. Afterwards, the second image data and / or the first text data are obtained, and the second feature category is obtained. The second image data is a query image input by the user, the first text data is a query text input by the user, and the second feature category is a query feature category input by the user and the second feature category includes one or more of the first feature categories. Afterwards, the second image data and / or the first text data are feature extracted at least based on the lightweight CLIP model to obtain the second target feature, the corresponding feature clustering cluster is determined based on the second feature category, the target clustering cluster is obtained, and the second target feature is matched in the target clustering cluster to obtain the first retrieval result. Finally, the first search result and the second image data or the first search result and the first text data are input into the LLM agent model for evaluation. If the evaluation passes, the first search result is determined as the target search result. The evaluation includes one or more of relevance evaluation, confidence evaluation, and consistency evaluation. The present application reduces the computational complexity of the CLIP model and improves the efficiency of feature storage and retrieval by introducing a lightweight CLIP model and a feature clustering storage strategy. By dividing the features into clusters before they are stored in the database, the search matching is reduced during the search phase according to the clustering clusters, thereby achieving a more accurate search. The introduction of LLM agent judgment and LLM agent model re-search improves the relevance and consistency of the search results, and solves the problems of high latency and insufficient precision in multimodal data retrieval in the prior art.

[0055] As a possible implementation manner, performing feature extraction on the first image data based on at least a lightweight CLIP model to obtain a first target feature includes the following steps:

[0056] Step S301, when the feature clusters are divided according to the features of the first image data, the first image data is segmented by the SAM model to obtain a plurality of third image data. Figure 4 As shown, Figure 4 : is a schematic diagram of a scene in which the first image data is segmented by the SAM model according to an embodiment of the present application:

[0057] Specifically, on the basis of dividing the feature clusters, the Segment Anything Model (SAM) is used to segment the first image data and decompose the image into multiple fine-grained third image data (such as objects, regions, backgrounds, etc.). The SAM model has a highly general segmentation capability and can automatically identify and extract key areas and features in an image. Image segmentation refines the feature expression of image data, ensures high-quality extraction of key areas and features, improves the refinement of feature clustering, and lays a more accurate foundation for subsequent feature extraction and matching.

[0058] Step S302: extract features based on the first image data and the third image data using a lightweight CLIP model to obtain corresponding first target features.

[0059] Specifically, the original image data (first image data) and its segmented fine-grained image data (third image data) are respectively subjected to feature extraction through the lightweight CLIP model to generate a high-dimensional feature vector. The lightweight CLIP model combines 1D convolution and attention mechanisms to efficiently capture the global and local features of the image while reducing computing resource consumption. It utilizes global features and local detail features to improve the comprehensiveness and distinguishing ability of feature representation. The lightweight design reduces computational complexity.

[0060] Therefore, the embodiment of the present application introduces the SAM model to segment the first image data based on the feature clustering cluster division, generates fine-grained third image data, and extracts features from the original image data and the segmented image data respectively in combination with the lightweight CLIP model. This method improves the accuracy and richness of feature extraction by refining feature expression. The lightweight design effectively reduces the computational complexity and resource consumption, solves the problem of high dependence of feature extraction on computing resources and insufficient expression of image features in the prior art, and provides technical support for the efficient implementation of multimodal data storage and retrieval.

[0061] As a possible implementation manner, feature extraction is performed based on the first image data and the third image data through a lightweight CLIP model to obtain corresponding first target features, including the following steps:

[0062] Step S401: Based on the first image data, feature extraction is performed through a lightweight CLIP model to obtain a third target feature.

[0063] Specifically, the first image data (original image) is input into the lightweight CLIP model for feature extraction to generate the third target feature. The lightweight CLIP model uses its architecture combining 1D convolution with attention layer to effectively extract the global features of the image and form a high-dimensional feature representation. The global features of the image are extracted to ensure the integrity of the overall information of the image, and provide comprehensive basic features for subsequent feature clustering and retrieval.

[0064] Step S402: Based on each third image data, feature extraction is performed using a lightweight CLIP model to obtain a fourth target feature.

[0065] Specifically, each third image data (fine-grained image part) generated by the SAM model segmentation is input into the lightweight CLIP model for feature extraction to generate the corresponding fourth target features. These features represent the local information and key area characteristics of the image. Extracting local features for the fine-grained image part makes up for the detail information that may be ignored by the global features, and enhances the meticulousness and distinguishing ability of feature expression.

[0066] Step S403, adding a target identifier to the third target feature and the fourth target feature to obtain a first target feature, wherein the target identifier is used to cluster the third target feature and the fourth target feature into different feature clusters. Figure 5 As shown, Figure 5 This is a schematic diagram of a scene of feature clustering according to an embodiment of the present application:

[0067] Specifically, in order to distinguish global features from local features, target identifiers are added to the third target feature (global feature) and the fourth target feature (local feature). Based on the feature differentiation function of the target identifier, global features and local features are divided into different feature clusters to form a clear feature storage structure. By introducing the identifier, the clustering of global and local features is clarified, the structured and targeted nature of feature clustering is improved, and at the same time, the feature confusion problem is reduced, and the accuracy of retrieval matching is improved.

[0068] Therefore, the embodiment of the present application forms a hierarchical storage and retrieval mechanism for features by extracting the global features and local features of the first image data respectively and adding target identifiers. Global feature extraction ensures the integrity of the overall information of the image, and local feature extraction enhances the expression of detail information. The target identifier is used to distinguish and cluster different feature clusters, thereby optimizing the organizational structure of the features. While improving retrieval efficiency, the overall solution improves the accuracy and discrimination of feature matching, solves the problem of poor retrieval results caused by confusion between global and local features in the prior art, and provides an efficient and accurate solution for multimodal data storage and retrieval.

[0069] As a possible implementation, after the first search result and the second image data or the first search result and the first text data are input into the LLM agent model for evaluation, the following steps are included:

[0070] Step S501, if the input data of the LLM agent model is the first search result and the second image data, then if the evaluation fails, the second text data is obtained;

[0071] The second text data is the query text supplemented by the user.

[0072] Specifically, when the first search result and the second image data input by the user fail to pass the LLM agent model evaluation, the system guides the user to supplement the query text (second text data). The supplementary text can be a further description of the query image, such as the color, shape, purpose, etc. of the target, enriching the user's query information. By allowing the user to actively supplement the query text, the flexibility and accuracy of the query are improved, and the risk of retrieval failure caused by insufficient initial query information is reduced.

[0073] Step S502: Process the second text data based on the LLM agent model to generate target description information, where the target description information includes color, shape, relationship and context information.

[0074] Specifically, the LLM agent model is used to perform natural language processing on the second text data supplemented by the user to extract the description information of the target. The target description information covers the color, shape, relationship between objects and related context information, helping to build more comprehensive query conditions. By mining the detailed description information in the user's supplementary text, the semantic expression of the retrieval feature is enriched and the matching accuracy is improved.

[0075] Step S503: Process each third image data through the LLM agent model based on the target description information to generate associated description information, where the associated description information is used to characterize the relationship between target description information of different third image data.

[0076] Specifically, the LLM agent model processes the third image data based on the target description information and generates associated description information. The associated description information represents the semantic relationship (such as adjacency, inclusion, and contrast) between different regions or objects in the image data. By generating associated description information, the semantic understanding ability of image retrieval is enhanced, providing support for high-precision retrieval in complex scenes.

[0077] Step S504: Initialize a feature search graph through the LLM agent model based on the association description information, and generate a first sub-problem based on each node in the feature search graph.

[0078] Specifically, a feature search graph is constructed based on the associated description information, and the nodes of the graph represent different features or target areas. Through the structured analysis of the search graph, the first sub-questions are generated node by node, and each sub-question corresponds to a specific feature retrieval target. The feature search graph organizes the hierarchical structure of the retrieval task, decomposes complex queries, and improves retrieval efficiency and accuracy.

[0079] Step S505: Match the vector database through the lightweight CLIP model based on each first sub-problem, determine the data corresponding to the maximum correlation value as the second retrieval result, and evaluate the second retrieval result and the second image data through the LLM agent model. If the evaluation passes, the second retrieval result is determined as the target retrieval result, and one first sub-problem corresponds to one second retrieval result.

[0080] Specifically, for each first sub-question, the lightweight CLIP model is used to perform feature matching in the vector database to find the most relevant feature and determine the second search result. Subsequently, the second search result and the user's second image data are evaluated again through the LLM agent model to ensure the accuracy and consistency of the matching results. The retrieval process is refined and the matching accuracy is gradually optimized. At the same time, the multi-round evaluation mechanism improves the reliability of the final retrieval results.

[0081] Therefore, the embodiment of the present application guides the user to supplement the query text, uses the LLM agent model to extract the target description information and generate related description information, further constructs a feature search graph and decomposes complex query tasks. Through the hierarchical retrieval and multi-round evaluation mechanism, the retrieval accuracy and efficiency of image and text data in the vector database are significantly improved. This method gives full play to the advantages of user participation and multimodal feature expression, solves the problems of retrieval failure and inaccurate matching in the prior art, and is suitable for high-precision multimodal data retrieval needs in complex scenarios.

[0082] As a possible implementation, the evaluation is performed through the LLM agent model based on the second search result and the second image data, including the following steps:

[0083] Step S601: Process the second search result through the LLM agent model to obtain search response information corresponding to each node.

[0084] Specifically, the LLM agent model is used to perform semantic analysis and feature extraction on the second search result, and search response information related to the node of the feature search graph is generated therefrom. The search response information includes semantic features, matching scores or context information related to the search result, further enriching the description content of the node.

[0085] Step S602: adding each search response information as a new node to update the feature search graph.

[0086] Specifically, the generated search response information is integrated into the feature search graph, and new nodes are added to expand the graph structure. These new nodes not only contain the feature information of the second search result, but also maintain contextual association with the original nodes, forming a more complete search graph network. The addition of new nodes enables the dynamic expansion of the feature search graph, optimizes the representation ability of the graph, enables it to capture more comprehensive feature relationships, and further improves the accuracy and flexibility of retrieval.

[0087] Step S603: evaluating through the LLM agent model based on the feature search graph and the second image data.

[0088] Specifically, based on the updated feature search graph, the LLM agent model is used to perform evaluation in combination with the second image data. The evaluation process comprehensively considers the feature information of the nodes in the graph and their semantic relationship with the second image data, and performs correlation analysis, confidence judgment, etc. to determine whether the final matching result meets the user's query intent. By combining the updated search graph and user input data for comprehensive evaluation, the reliability of the final matching result is improved, ensuring that the retrieval result is highly consistent with the user's intent.

[0089] Therefore, the embodiment of the present application performs semantic processing on the second search result through the LLM agent model, generates search response information and dynamically integrates it into the feature search graph, thereby expanding the nodes of the search graph and its feature expression capabilities. At the same time, combined with the second image data input by the user, the LLM agent model is used to comprehensively evaluate the expanded search graph, effectively improving the semantic relevance and confidence of the search results. This method optimizes the dynamic update and semantic evaluation mechanism of the feature search graph, solves the problem of inaccurate evaluation caused by insufficient features of the search results in the prior art, and significantly improves the applicability and robustness of the retrieval system in complex scenarios.

[0090] As a possible implementation, after the first search result and the second image data or the first search result and the first text data are input into the LLM agent model for evaluation, the following steps are included:

[0091] Step S701: If the input data of the LLM agent model is the first search result and the first text data, then in the case of failure in the evaluation, the first text data is processed by the LLM agent model to obtain the second sub-problem.

[0092] Specifically, when the LLM agent model fails to evaluate the relevance of the first search result to the first text data, the model will perform deep semantic analysis on the first text data and generate one or more second sub-questions. These sub-questions can be a refinement or extension of the user's query intent, aiming to clarify the search target and improve the accuracy of the search matching. By generating sub-questions, the search questions that failed the evaluation are decomposed into more specific tasks, allowing the model to focus on a clearer search direction, thereby optimizing the accuracy and relevance of subsequent search results.

[0093] Step S702: Match the vector database through the lightweight CLIP model based on each second sub-problem, determine the data corresponding to the maximum correlation value as the third search result, and evaluate the third search result and the first text data through the LLM agent model. If the evaluation passes, determine the third search result as the target search result, and one second sub-problem corresponds to one third search result.

[0094] Specifically, for each generated second sub-question, a lightweight CLIP model is used to match from the vector database. By calculating the similarity between each vector and the sub-question feature, the vector corresponding to the maximum correlation value is determined, and the associated data is used as the third search result. Targeted matching is performed through the lightweight CLIP model to ensure that each sub-question can find the most relevant database content, greatly improving the refinement of the search and the accuracy of the results. The third search result and the first text data are input into the LLM agent model again for evaluation, and a comprehensive analysis is performed by comparing the semantic relevance, confidence and consistency of the results. When the evaluation passes, the corresponding third search result is output as the final target search result. The evaluation process of each sub-question is carried out independently to ensure that all targets obtain the best search results. Through multiple rounds of evaluation and matching, the accuracy and reliability of the target search results are further guaranteed, and the search bias caused by the fuzzy initial query or insufficient feature matching is avoided.

[0095] Therefore, in the embodiment of the present application, when the search evaluation fails, the first text data is semantically parsed through the LLM agent model to generate a clear second sub-question, and the database content is matched in a targeted manner in combination with the lightweight CLIP model, and the search results are further optimized through multiple rounds of evaluation. This method effectively decomposes user intent in fuzzy query scenarios, improves search accuracy and matching efficiency in complex query scenarios, solves the problem of unsatisfactory search results due to fuzzy queries or insufficient initial matching in the prior art, and significantly enhances the adaptability and robustness of multimodal retrieval systems.

[0096] As a possible implementation method, matching is performed in the target clusters based on the second target feature to obtain the first search result. Specifically, when the number of target clusters is greater than a preset value, the matching process of different target clusters is assigned to different threads for calculation.

[0097] Specifically, when the number of target clusters exceeds the preset value, in order to improve the matching efficiency, the system will parallelize the matching process of each target cluster. Specifically, different target clusters are assigned to multiple threads for calculation, thereby realizing parallel computing. Each thread is responsible for matching the data in one or more target clusters, accelerating the entire retrieval process. By assigning the matching tasks of different clusters to different threads for parallel computing when the number of target clusters exceeds the preset value, the retrieval speed and system efficiency are greatly improved. This method effectively solves the computational bottleneck problem that may occur when processing a large number of clusters, and significantly accelerates the retrieval process.

[0098] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0099] The embodiment of the present application also provides an image data storage and retrieval device based on a lightweight multimodal model. It should be noted that the image data storage and retrieval device based on a lightweight multimodal model in the embodiment of the present application can be used to execute the image data storage and retrieval method based on a lightweight multimodal model provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0100] The following introduces an image data storage and retrieval device based on a lightweight multimodal model provided in an embodiment of the present application.

[0101] Figure 6 is a structural block diagram of an image data storage and retrieval device based on a lightweight multimodal model according to an embodiment of the present application. Figure 6 As shown, the device includes: a first processing unit 10, a second processing unit 20, a first acquiring unit 30, a third processing unit 40 and a fourth processing unit 50.

[0102] The first processing unit 10 is used to perform classification based on the characteristics and application scenarios of the first image data to obtain a plurality of first feature categories, and the first image data is image data to be stored.

[0103] The second processing unit 20 is used to extract features from the first image data at least based on a lightweight CLIP model to obtain a first target feature, cluster the first target feature based on a first feature category to obtain a plurality of feature clustering clusters, and store the feature clustering clusters in a vector database. The lightweight CLIP model is a CLIP model that combines 1D convolution with an attention layer.

[0104] The first acquisition unit 30 is used to acquire the second image data and / or the first text data, and acquire the second feature category, wherein the second image data is the query image input by the user, the first text data is the query text input by the user, the second feature category is the query feature category input by the user, and the second feature category includes one or more of the first feature categories.

[0105] The third processing unit 40 is used to perform feature extraction on the second image data and / or the first text data based on at least the lightweight CLIP model to obtain a second target feature, determine a corresponding feature clustering cluster based on the second feature category to obtain a target clustering cluster, and perform matching in the target clustering cluster based on the second target feature to obtain a first retrieval result.

[0106] The fourth processing unit 50 is used to input the first retrieval result and the second image data or the first retrieval result and the first text data into the LLM agent model for evaluation. If the evaluation passes, the first retrieval result is determined as the target retrieval result. The evaluation includes one or more of relevance evaluation, confidence evaluation and consistency evaluation.

[0107] The embodiment of the present application provides an image data storage and retrieval device based on a lightweight multimodal model, the device comprising: a first processing unit, a second processing unit, a first acquisition unit, a third processing unit, and a fourth processing unit. The device introduces a lightweight CLIP model and a feature clustering storage strategy, divides the feature clusters before the features are stored, stores them according to the clusters to reduce retrieval matching in the retrieval stage, introduces LLM agent judgment and LLM agent model re-retrieval, and solves the problem of low efficiency caused by high latency in retrieval of multimodal data in vector databases in the prior art.

[0108] As a possible implementation manner, the second processing unit includes:

[0109] The segmentation module is used to segment the first image data through the SAM model to obtain a plurality of third image data when the feature clustering clusters are divided according to the features of the first image data.

[0110] The feature extraction module is used to extract features based on the first image data and the third image data through a lightweight CLIP model to obtain corresponding first target features.

[0111] As a possible implementation method, the feature extraction module includes:

[0112] The first extraction submodule is used to perform feature extraction based on the first image data through a lightweight CLIP model to obtain a third target feature.

[0113] The second extraction submodule is used to perform feature extraction through a lightweight CLIP model based on each third image data to obtain a fourth target feature.

[0114] The first target feature acquisition submodule is used to add a target identifier to the third target feature and the fourth target feature to obtain the first target feature, and the target identifier is used to cluster the third target feature and the fourth target feature into different feature clustering clusters.

[0115] As a possible implementation manner, the device further includes:

[0116] The second acquisition unit is used to acquire second text data if the input data of the LLM agent model is the first search result and the second image data and if the evaluation fails, the second text data is the query text supplemented by the user.

[0117] The target description information generating unit is used to process the second text data based on the LLM agent model to generate target description information, where the target description information includes color, shape, relationship and context information.

[0118] The associated description information generating unit is used to process each third image data through the LLM agent model based on the target description information to generate associated description information, wherein the associated description information is used to characterize the relationship between the target description information of different third image data.

[0119] The first sub-problem generating unit is used to initialize the feature search graph through the LLM agent model based on the associated description information, and to generate the first sub-problem based on each node in the feature search graph.

[0120] The first target retrieval result determination unit is used to match the vector database through the lightweight CLIP model based on each first sub-problem, determine the data corresponding to the maximum correlation value as the second retrieval result, and evaluate the second retrieval result and the second image data through the LLM agent model. If the evaluation passes, the second retrieval result is determined as the target retrieval result, and one first sub-problem corresponds to one second retrieval result.

[0121] As a possible implementation manner, the target search result determination unit includes:

[0122] The search response information acquisition module is used to process the second search result through the LLM agent model to obtain the search response information corresponding to each node.

[0123] The feature search graph updating module is used to add each search response information as a new node to update the feature search graph.

[0124] The evaluation module is used to perform evaluation through the LLM agent model based on the feature search graph and the second image data.

[0125] As a possible implementation, the device further includes:

[0126] The second sub-question generating unit, if the input data of the LLM agent model is the first search result and the first text data, then in the case of failing the evaluation, processes the first text data through the LLM agent model to obtain the second sub-question.

[0127] The second target retrieval result determination unit is used to match the vector database through the lightweight CLIP model based on each second sub-problem, determine the data corresponding to the maximum correlation value as the third retrieval result, and evaluate the third retrieval result and the first text data through the LLM agent model. If the evaluation passes, the third retrieval result is determined as the target retrieval result, and one second sub-problem corresponds to one third retrieval result.

[0128] As a possible implementation manner, the third processing unit is specifically configured to allocate matching processes of different target clusters to different threads for calculation when the number of target clusters is greater than a preset value.

[0129] The above-mentioned image data storage and retrieval device based on lightweight multimodal model includes a processor and a memory, and the above-mentioned units are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The above-mentioned modules are all located in the same processor. Alternatively, the above-mentioned modules are located in different processors in any combination.

[0130] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the communication efficiency can be improved by adjusting the kernel parameters.

[0131] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0132] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute an image data storage and retrieval method based on a lightweight multimodal model.

[0133] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes an image data storage and retrieval method based on a lightweight multimodal model when running.

[0134] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing an image data storage and retrieval method based on a lightweight multimodal model.

[0135] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0140] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0141] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0142] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0144] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0145] 1) The image data storage and retrieval method based on a lightweight multimodal model of the present application introduces a lightweight CLIP model and a feature clustering storage strategy, divides the feature clusters before the features are stored in the database, stores them according to the clusters to reduce retrieval matching in the retrieval stage, introduces LLM agent judgment and LLM agent model re-retrieval, and solves the problem of low efficiency caused by high latency in retrieval of multimodal data in vector databases in the prior art.

[0146] 2) The image data storage and retrieval device based on the lightweight multimodal model of the present application includes: a first processing unit, a second processing unit, a first acquisition unit, a third processing unit, and a fourth processing unit. The device introduces a lightweight CLIP model and a feature clustering storage strategy, divides the feature clusters before the features are stored, stores them according to the clusters to reduce retrieval matching in the retrieval stage, introduces LLM agent judgment and LLM agent model re-retrieval, and solves the problem of low efficiency caused by high latency in retrieval of multimodal data in vector databases in the prior art.

[0147] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for storing and retrieving image data based on a lightweight multimodal model, characterized in that: include: Classify the first image data based on its characteristics and application scenarios to obtain a plurality of first feature categories, wherein the first image data is image data to be stored; At least based on a lightweight CLIP model, feature extraction is performed on the first image data to obtain a first target feature, the first target feature is clustered based on the first feature category to obtain a plurality of feature clustering clusters, and the feature clustering clusters are stored in a vector database, wherein the lightweight CLIP model is a CLIP model combining a 1D convolution with an attention layer, and the feature clustering clusters correspond to the first feature category one by one; Acquire second image data and / or first text data, and acquire a second feature category, wherein the second image data is a query image input by a user, the first text data is a query text input by a user, the second feature category is a query feature category input by the user, and the second feature category includes one or more of the first feature categories; At least based on the lightweight CLIP model, feature extraction is performed on the second image data and / or the first text data to obtain a second target feature, based on the second feature category, the corresponding feature cluster is determined to obtain a target cluster, and based on the second target feature, matching is performed in the target cluster to obtain a first search result; The first retrieval result and the second image data or the first retrieval result and the first text data are input into the LLM agent model for evaluation. If the evaluation passes, the first retrieval result is determined as the target retrieval result. The evaluation includes one or more of relevance evaluation, confidence evaluation and consistency evaluation.

2. The method according to claim 1, characterized in that Extracting features from the first image data based at least on a lightweight CLIP model to obtain a first target feature includes: In the case where the feature clusters are divided according to the features of the first image data, the first image data is segmented by a SAM model to obtain a plurality of third image data; Based on the first image data and the third image data, feature extraction is performed using the lightweight CLIP model to obtain the corresponding first target feature.

3. The method according to claim 2, characterized in that Based on the first image data and the third image data, feature extraction is performed using the lightweight CLIP model to obtain the corresponding first target feature, including: Based on the first image data, extract features through the lightweight CLIP model to obtain a third target feature; Based on each of the third image data, feature extraction is performed using the lightweight CLIP model to obtain a fourth target feature; A target identifier is added to the third target feature and the fourth target feature to obtain the first target feature, and the target identifier is used to cluster the third target feature and the fourth target feature into different feature clustering clusters.

4. The method according to claim 2, characterized in that: After inputting the first search result and the second image data or the first search result and the first text data into the LLM agent model for evaluation, the method further includes: If the input data of the LLM agent model is the first search result and the second image data, then if the evaluation fails, obtaining second text data, where the second text data is the query text supplemented by the user; Processing the second text data based on the LLM agent model to generate target description information, wherein the target description information includes color, shape, relationship and context information; Processing each of the third image data through the LLM agent model based on the target description information to generate associated description information, where the associated description information is used to characterize the relationship between the target description information of different third image data; Initializing a feature search graph through the LLM agent model based on the association description information, and generating a first sub-problem based on each node in the feature search graph; Based on each of the first sub-problems, the vector database is matched through the lightweight CLIP model respectively, and the data corresponding to the maximum correlation value is determined as the second retrieval result, and based on the second retrieval result and the second image data, the evaluation is performed through the LLM agent model. If the evaluation passes, the second retrieval result is determined as the target retrieval result, and one first sub-problem corresponds to one second retrieval result.

5. The method according to claim 4, characterized in that Performing the evaluation in the LLM agent model based on the second search result and the second image data includes: Processing the second search result by the LLM agent model to obtain search response information corresponding to each of the nodes; Add each of the search response information as a new node to update the feature search graph; The evaluation is performed in the LLM agent model based on the feature search graph and the second image data.

6. The method according to claim 4, characterized in that After inputting the first search result and the second image data or the first search result and the first text data into the LLM agent model for evaluation, the method further includes: If the input data of the LLM agent model is the first search result and the first text data, then in the case where the evaluation fails, the first text data is processed by the LLM agent model to obtain a second sub-problem; Based on each of the second sub-problems, the vector database is matched through the lightweight CLIP model, and the data corresponding to the maximum correlation value is determined as the third search result. Based on the third search result and the first text data, the evaluation is performed in the LLM agent model. If the evaluation passes, the third search result is determined as the target search result, and one second sub-problem corresponds to one third search result.

7. The method according to claim 6, characterized in that The first search result is obtained by matching the second target feature in the target cluster, including: When the number of the target clusters is greater than a preset value, the matching processes of different target clusters are assigned to different threads for calculation.

8. An image data storage and retrieval device based on a lightweight multimodal model, characterized in that: The device comprises: A first processing unit, configured to perform classification based on characteristics and application scenarios of first image data to obtain a plurality of first feature categories, wherein the first image data is image data to be stored; a second processing unit, configured to perform feature extraction on the first image data based at least on a lightweight CLIP model to obtain a first target feature, cluster the first target feature based on the first feature category to obtain a plurality of feature clustering clusters, and store the feature clustering clusters in a vector database, wherein the lightweight CLIP model is a CLIP model combining a 1D convolution with an attention layer, and the feature clustering clusters correspond one-to-one to the first feature category; a first acquisition unit, configured to acquire second image data and / or first text data, and acquire a second feature category, wherein the second image data is a query image input by a user, the first text data is a query text input by a user, the second feature category is a query feature category input by the user, and the second feature category includes one or more of the first feature categories; a third processing unit, configured to perform feature extraction on the second image data and / or the first text data based at least on the lightweight CLIP model to obtain a second target feature, determine the corresponding feature cluster based on the second feature category to obtain a target cluster, and perform matching in the target cluster based on the second target feature to obtain a first search result; The fourth processing unit is used to input the first retrieval result and the second image data or the first retrieval result and the first text data into the LLM agent model for evaluation. If the evaluation passes, the first retrieval result is determined as the target retrieval result. The evaluation includes one or more of relevance evaluation, confidence evaluation and consistency evaluation.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A multimodal data storage system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

Images