Intelligent image-text retrieval system and method based on soft prompt and electronic equipment

Abstract

The invention discloses an intelligent image-text retrieval system and method based on soft prompt and electronic equipment. The system comprises a retrieval interface module, a model retrieval module, a retrieval data collection module and a soft prompt efficient updating model module. Wherein the soft prompt efficient updating model module is used for training on a multi-mode pre-training model by using stored data when a collected image-text pair data set reaches a preset threshold value, obtaining a new soft prompt and updating the model retrieval module by using the new soft prompt. According to the method, a soft prompt method suitable for a large-scale pre-training model is utilized, the characteristics of multi-modal modeling are considered, and the soft prompt method and a multi-modal method are combined to realize optimal alignment of vectors on an image-text retrieval task; modal-shared soft prompts and modal-specific soft prompts are constructed to promote alignment of different modal soft prompt information, so that alignment of different modal information is promoted. The method can be widely applied to the technical field of image-text retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of image and text retrieval, and in particular to an intelligent image and text retrieval system, method and electronic device based on soft prompts. Background Art

[0002] Existing intelligent image-text retrieval systems can implement image-text retrieval according to user needs, but they face some challenges in updating retrieval models with efficient parameters. Because these systems often require a lot of computing resources and time to retrain the model to adapt to new data or improve the algorithm. Therefore, intelligent image-text retrieval systems based on parameter-efficient methods have emerged, which can quickly update and respond in application scenarios that require real-time or near real-time updates to better meet user needs. However, existing research on intelligent image-text retrieval systems based on soft prompts often focuses on the use of soft prompts, without considering the alignment and synergy between multimodal soft prompts, which will result in the inability of soft prompts to fully adjust the vector representations of different modal encoders (such as text encoders, image encoders, etc.), thereby failing to achieve the optimal alignment of these vectors in image-text retrieval tasks. Summary of the invention

[0003] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the object of the present invention is to provide an intelligent image and text retrieval system, method and electronic device based on soft prompts.

[0004] The first technical solution adopted by the present invention is:

[0005] An intelligent image and text retrieval system based on soft prompts, comprising:

[0006] A search interface module, used to interact with the user, obtain input query information, and display search results; the query information includes at least one of characters, texts, or images;

[0007] The model retrieval module is used to splice the query information and soft prompts and input them into the multimodal pre-trained model for feature extraction, perform similarity search between the extracted feature vector and the vector database to obtain the image and text data vector; and obtain the corresponding data from the image and text database according to the image and text data vector as the retrieval result;

[0008] A retrieval data collection module is used to combine query information and corresponding retrieval results into a picture-text pair data set, store it in a picture-text database, and use it to soft-prompt an efficient update model module to efficiently update the retrieval model;

[0009] The soft prompt efficient model update module is used to train the multimodal pre-trained model using the stored data when the collected image-text pair data set reaches a preset threshold, obtain new soft prompts, and use them to update the model retrieval module;

[0010] Among them, in the multimodal pre-training model, two types of soft prompts are used: modality-shared soft prompts and modality-specific soft prompts. By constructing modality-shared soft prompts and modality-specific soft prompts, the alignment of soft prompt information of different modalities is promoted, thereby promoting the alignment of information of different modalities.

[0011] Furthermore, the search interface module is specifically used for:

[0012] After obtaining the input information, preprocessing the input information to perform the query parsing stage; the preprocessing includes cleaning processing, word segmentation processing or conversion processing;

[0013] In the query parsing stage, natural language processing technology is used to analyze the user's intention to extract key information as the input of the model retrieval module.

[0014] Furthermore, the model retrieval module is specifically used for:

[0015] The query information and soft prompts are concatenated and input into the multimodal pre-trained model for feature extraction to obtain a feature vector;

[0016] The extracted feature vectors are searched for similarity in the vector database to obtain relevant graphic and text data vectors;

[0017] According to the image-text data vector, the corresponding data is obtained from the image-text database as the retrieval result.

[0018] Furthermore, the soft hints are learnable vectors, and they are continuously updated during the training process to adapt to specific tasks or contexts, ensuring that the model can quickly absorb new knowledge without forgetting old knowledge while keeping the number of training parameters extremely low;

[0019] Among them, the stored data is used to train the multimodal pre-trained model using a parameter-efficient soft prompt method.

[0020] Furthermore, modality-shared soft prompts can not only promote the interaction of information between different modalities, but also promote the alignment of soft prompts specific to different modalities, while further reducing the amount of training parameters to a certain extent;

[0021] Modality-specific soft prompts are built on a common knowledge base and can facilitate the alignment of the two soft prompts. When set at different modality encoders, they can effectively adapt to the modality-specific information of each modality to guide the model to generate outputs that better meet user needs.

[0022] The text encoder, image encoder, and multimodal encoder use their own modality-specific soft prompts to adapt to their own specific information. By using these two types of soft prompts, an effective combination of the multimodal method and the soft prompt method can be achieved.

[0023] Furthermore, the soft prompt efficient update model module also includes a single-modal soft prompt fusion unit;

[0024] The unimodal soft prompt fusion unit is used to promote the coordination between the unimodal soft prompt and the multimodal soft prompt, by fusing the unimodal soft prompt and adding the corresponding position elements with the multimodal soft prompt to form a new multimodal soft prompt, thereby achieving coordination and alignment between the two.

[0025] Furthermore, the soft prompt efficient update model module adopts an automated testing and evaluation mechanism to ensure that the updated multimodal pre-trained model maintains or improves retrieval accuracy and response speed in practical applications;

[0026] The soft prompt efficient update model module supports continuous iteration of the model. After the model training is completed, it is only necessary to replace the soft prompts in the model retrieval module with the newly generated soft prompts to complete the model update, so that the system can adapt to the ever-changing data and user needs.

[0027] Furthermore, the retrieval data collection module is specifically used for:

[0028] By collecting query information and corresponding search results, data processing and cleaning are performed during the data integration process to form a new graphic data set, which is stored in a location in the graphic database specifically used to store new data, for soft prompts to efficiently update the model.

[0029] The second technical solution adopted by the present invention is:

[0030] An intelligent image-text retrieval method based on soft prompts comprises the following steps:

[0031] Acquire query information input by a user; the query information includes at least one of characters, text or images;

[0032] The query information and soft prompts are concatenated and input into the multimodal pre-trained model for feature extraction. The extracted feature vector is searched for similarity with the vector database to obtain the image-text data vector. According to the image-text data vector, the corresponding data is obtained from the image-text database as the retrieval result.

[0033] The query information and the corresponding search results are combined into a picture-text pair data set and stored in a picture-text database;

[0034] When it is detected that the collected image-text pair data set reaches a preset threshold, the stored data is used to train on the multimodal pre-trained model to obtain new soft prompts;

[0035] Among them, in the multimodal pre-training model, two types of soft prompts are used: modality-shared soft prompts and modality-specific soft prompts. By constructing modality-shared soft prompts and modality-specific soft prompts, the alignment of soft prompt information of different modalities is promoted, thereby promoting the alignment of information of different modalities.

[0036] The third technical solution adopted by the present invention is:

[0037] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, the code set or the instruction set is loaded and executed by the processor to implement an intelligent image and text retrieval method based on soft prompts as described above.

[0038] The fourth technical solution adopted by the present invention is:

[0039] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement an intelligent image and text retrieval method based on soft prompts as described above.

[0040] The fifth technical solution adopted by the present invention is:

[0041] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned intelligent image and text retrieval method based on soft prompts.

[0042] The beneficial effects of the present invention are as follows: the present invention utilizes a soft prompt method suitable for large-scale pre-trained models, takes into account the characteristics of multimodal modeling, combines the soft prompt method with the multimodal method to achieve optimal alignment of vectors in image-text retrieval tasks; constructs modality-shared soft prompts and modality-specific soft prompts to promote the alignment of soft prompt information of different modalities, and further promotes the alignment of information of different modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 It is a structural block diagram of an intelligent image-text retrieval system based on parameter efficient alignment soft prompts in an embodiment of the present invention;

[0045] Figure 2 is a structural block diagram of a search interface module in an embodiment of the present invention;

[0046] Figure 3 is a structural block diagram of a model retrieval module in an embodiment of the present invention;

[0047] Figure 4 is a structural block diagram of a retrieval data collection module in an embodiment of the present invention;

[0048] Figure 5 is a structural block diagram of a soft prompt efficient update model module in an embodiment of the present invention;

[0049] Figure 6 It is a flowchart of the steps of an intelligent image and text retrieval method based on soft prompts in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0051] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0052] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0053] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0054] Technical explanation:

[0055] (1) Multimodal Large-Scale Pre-training Model

[0056] Multimodal large-scale pre-trained models are an advanced AI technology that can process and understand multiple types of data, such as text, images, audio, and video. These models are trained through deep learning techniques and are able to extract features and establish connections from data in different modalities, thereby providing a more comprehensive and in-depth understanding. Their core advantage is the ability to integrate information from different sources to enhance the model's prediction and reasoning capabilities. This integration not only improves the processing capabilities of single-modal data, but also enables the model to more accurately simulate human perception and cognitive processes. Multimodal large models have a wide range of applications in natural language processing, computer vision, speech recognition, and other fields, and can support complex tasks such as joint search of images and texts, video content understanding, and sentiment analysis. With the advancement of technology, these models are becoming more and more intelligent, able to handle more complex data and tasks, and promoting the application and development of artificial intelligence in various fields.

[0057] (2) Efficient parameter migration method for large-scale pre-trained models

[0058] Parameter-efficient transfer learning methods for large-scale pre-trained models, such as adapters and soft prompts, mainly achieve knowledge transfer by introducing additional parameters or prompts in key parts of the model. Adapter methods add small parameter modules to the input or output layer of the model, allowing the model to be fine-tuned for a specific task without changing the parameters of the entire pre-trained model. This method allows the model to quickly adapt to new tasks while maintaining the original knowledge. Soft prompt methods embed task-related prompts into the input of the model to guide the model to generate outputs that better meet the needs of specific tasks. These methods improve the efficiency of transfer learning by reducing the number of parameters that need to be trained, allowing pre-trained models to flexibly cope with various new tasks while reducing the demand for computing resources.

[0059] (3) Existing Intelligent Image and Text Retrieval Systems

[0060] Existing intelligent image and text retrieval systems are advanced systems that integrate computer vision and natural language processing technologies. They use multimodal large-scale pre-trained models to identify and understand image content and related text information. These systems can process users' natural language queries, implement semantic search of image and text data, and provide highly relevant retrieval results. They build efficient data indexes and databases to quickly respond to a large number of search requests, and have intuitive user interfaces that allow users to search and browse easily.

[0061] Example 1

[0062] like Figure 1 As shown, this embodiment provides an intelligent image-text retrieval system based on parameter efficient alignment soft prompts, specifically including:

[0063] A search interface module, used to interact with the user, obtain input query information, and display search results; the query information includes at least one of characters, texts, or images;

[0064] The model retrieval module is used to splice the query information and soft prompts and input them into the multimodal pre-trained model for feature extraction, perform similarity search between the extracted feature vector and the vector database to obtain the image and text data vector; and obtain the corresponding data from the image and text database according to the image and text data vector as the retrieval result;

[0065] A retrieval data collection module is used to combine query information and corresponding retrieval results into a picture-text pair data set, store it in a picture-text database, and use it to soft-prompt an efficient update model module to efficiently update the retrieval model;

[0066] The soft prompt efficient model update module is used to train the multimodal pre-trained model using the stored data when the collected image-text pair data set reaches a preset threshold, obtain new soft prompts, and use them to update the model retrieval module;

[0067] Among them, in the multimodal pre-training model, two types of soft prompts are used: modality-shared soft prompts and modality-specific soft prompts. By constructing modality-shared soft prompts and modality-specific soft prompts, the alignment of soft prompt information of different modalities is promoted, thereby promoting the alignment of information of different modalities.

[0068] The intelligent image-text retrieval system for multimodal modeling using parameter efficient alignment soft prompts proposed in this embodiment includes a retrieval interface module, a model retrieval module, a retrieval data collection module, and a soft prompt efficient update model module. Users can use the image-text retrieval function in the retrieval interface module. When using this function, the retrieval model in the model retrieval module will be used to retrieve and return relevant content in the vector database and the image-text database. At the same time, the retrieval data collection module will collect and record the queries submitted by the user and the answers returned by the system, and store them in the vector database and the image-text database after integration, and use the soft prompt efficient update model module to update the retrieval model.

[0069] Exemplarily, the workflow of the system of this embodiment is as follows: 1) First, the user searches in the search interface module by natural language query, keyword input or image upload. 2) The search interface module parses the user's query, extracts key information and sends it to the model search module and the search data collection module. 3) The model search module converts the query information into a vector, performs search comparison in the vector database, and returns the corresponding results in the image and text database to the search data collection module and the search interface module after obtaining the search results. The search interface module then returns the results to the user. 4) The search data collection module records the query submitted by the user and the answer returned by the system to form an image and text pair data set, which is stored in the image and text database for the soft prompt efficient update model module to efficiently update the search model. 5) When the collected search data reaches the data volume threshold, the soft prompt efficient update model module will use the collected data to train on the pre-trained model using a parameter-efficient soft prompt method, obtain new soft prompts and use them to update the model search module.

[0070] It should be noted that in this embodiment, the soft prompt efficient update model module adopts a parameter-efficient soft prompt method to efficiently update the retrieval model, but it is not limited to this. Other methods can also be used, such as the adapter method, the lora method, etc. These methods should all fall within the scope of protection of this application.

[0071] The following is a detailed explanation of each module of the system in conjunction with the accompanying drawings and specific implementation methods.

[0072] (1) Search interface module

[0073] See also Figure 2 ,The retrieval interface module provides an intuitive and easy-to-use interactive ,interface that enables users to easily enter queries, upload images or ,files, and receive retrieval results for users to browse.

[0074] Specifically, the search interface module first receives the user's input, which can be a text query or an image upload. The system then processes the input, including steps such as cleaning, word segmentation, and conversion, in preparation for query parsing. During the query parsing phase, the system uses relevant natural language processing technology to analyze the user's intent and extract key information. Interaction management displays the search results on the user interface and provides interactive functions, allowing users to filter, sort, or provide feedback on the results.

[0075] (2) Model retrieval module

[0076] See also Figure 3 ,The model retrieval module receives the query information of the retrieval interface module, and then ,joins the soft prompt with the corresponding input, and ,inputs it into the pre-trained model for in-depth feature extraction and ,obtains the corresponding feature vector.,Based on the extracted feature vector, the system will perform an efficient similarity ,search in the vector database, quickly match and retrieve the ,image data vector most relevant to the query, the retrieval results are optimized and ,sorted, and the corresponding index in the image and text database is obtained, and the data corresponding to the image and ,text database is returned as the result according to the optimized and ,sorted results.

[0077] (3) Retrieval data collection module

[0078] See also Figure 4 ,The retrieval data collection module collects the user’s query and retrieval results, ,and will process and clean the data during the data integration process to form a new ,graphic data set and store it in the location of the ,graphic database specifically used for storing new data for ,soft prompts to efficiently update the model.

[0079] (4) Soft prompts to efficiently update model modules

[0080] See also Figure 5 The Soft Hint Efficient Model Update module is a component in the system used to continuously optimize model performance. It introduces new image and text data to fine-tune the pre-trained model in a regular or conditional manner. This module uses parameter-efficient soft hint methods, which embed learnable soft hints in the model input. These soft hints are usually learnable vectors and continuously update these soft hints during the training process to adapt to specific tasks or contexts, ensuring that the model can quickly absorb new knowledge without forgetting old knowledge while keeping the number of training parameters extremely low.

[0081] Specifically, in the multimodal pre-training model, two types of soft prompts are used: modality-shared soft prompts (SharedPrompt) and modality-specific soft prompts (Text / Vision / Multimodal Prompt). Modality-specific soft prompts are built based on common source soft prompts, and the text encoder, image encoder, and multimodal encoder will use their own modality-specific soft prompts to adapt to their own specific information. By using these two types of soft prompts, the effective combination of multimodal methods and soft prompt methods can be achieved. Modality-shared soft prompts can not only promote the interaction of information between different modalities, but also promote the alignment of soft prompts specific to different modalities, and at the same time, further reduce the amount of training parameters to a certain extent; modality-specific soft prompts are built based on a common knowledge base, which can promote their alignment. Setting them at different modality encoders can effectively adapt to the information specific to each modality and guide the model to generate outputs that better meet user needs. In addition, a unimodal soft prompt fusion unit is introduced to promote the collaboration between unimodal soft prompts and multimodal soft prompts. The collaboration and alignment between the two are achieved by fusing unimodal soft prompts and adding corresponding position elements of the unimodal soft prompts to form new multimodal soft prompts.

[0082] In addition, the soft prompt efficient update model module will use an automated testing and evaluation mechanism to ensure that the updated model maintains or improves retrieval accuracy and response speed in actual applications. This module supports the continuous iteration of the retrieval model, and after training is completed, the model retrieval model can be updated by simply replacing the soft prompts in the model retrieval module with the newly generated soft prompts, allowing the system to adapt to changing data and user needs.

[0083] (5) Vector database and graphic database

[0084] The vector database storage system pre-extracts and indexes the image and text feature vectors and image and text alignment vectors, which facilitates the system to quickly compare the similarity between the stored vectors and the generated vectors. The image and text database stores the original data of the image and text, and the image and text data corresponding to the image and text feature vector can be found according to the index stored in the vector database.

[0085] In general, the present embodiment proposes an intelligent image-text retrieval system based on parameter-efficient alignment soft prompts, which utilizes a soft prompt method suitable for large-scale pre-trained models, while taking into account the characteristics of multimodal modeling, and combining the soft prompt method with the multimodal method to achieve the best alignment of vectors in image-text retrieval tasks. Specifically, the present embodiment first constructs modality-shared soft prompts and modality-specific soft prompts (wherein the modality-specific soft prompts are constructed based on common source soft prompts) to promote the alignment of soft prompt information of different modalities, thereby promoting the alignment of information of different modalities. In addition, a unimodal soft prompt fusion unit is proposed to promote the alignment and coordination between unimodal soft prompts and multimodal soft prompts, thereby promoting the multimodal vector to better capture the alignment information between images and texts. This scheme not only greatly reduces the amount of parameters that need to be trained when the retrieval model is updated, saves resources and time costs, but also takes into account the alignment and synergy between multimodal soft prompts, thereby guiding the model to generate output that better meets user needs.

[0086] In summary, the present invention has at least the following advantages and beneficial effects compared to the prior art:

[0087] (1) The present invention proposes an intelligent image-text retrieval system based on parameter-efficient alignment soft prompts, which uses a parameter-efficient soft prompt method to efficiently update and deploy image-text retrieval models on collected image-text data sets. The system can efficiently update image-text retrieval models, and because it uses a parameter-efficient soft prompt method, it can save a lot of computing resources and time costs while ensuring accuracy, solving the problems of long update cycles and high resource consumption in existing intelligent image-text retrieval systems.

[0088] (2) The present invention proposes a parameter-efficient soft prompt method suitable for a multimodal large-scale pre-trained model. Under the condition of keeping the parameters of the multimodal large-scale pre-trained model frozen, high accuracy of image and text retrieval can be achieved by training only the soft prompt part.

[0089] A soft hint method suitable for multimodal large-scale pre-trained models is proposed. First, two types of soft hints are proposed: modality sharing and modality-specific soft hints, to achieve the combination of multimodal methods and soft hints. Modality sharing soft hints can promote the interaction of information between different modalities and promote the alignment of modality-specific soft hints. Modality-specific soft hints are built on a common knowledge base (source soft hints), which can promote the alignment of modality-specific soft hints to a certain extent. The other is the collaboration between unimodal and multimodal soft hints. By fusing unimodal soft hints and adding them to multimodal soft hints to form new multimodal soft hints, the collaboration and alignment of the two are achieved. After training is completed, the model retrieval model can be efficiently updated by simply replacing the soft hints in the model retrieval module with the newly generated soft hints.

[0090] (3) The present invention proposes a retrieval data collection module to collect user retrieval data in real time, which is used to update the retrieval model in real time and provide users with better retrieval services. The retrieval data collection module used in the present invention can automatically collect user retrieval data to further improve the graphic database, and use the collected user retrieval data to further train the soft prompts. After the training is completed and the update deployment is completed, the user's retrieval needs can be better met.

[0091] Example 2

[0092] like Figure 6 As shown, this embodiment provides an intelligent image and text retrieval method based on soft prompts, comprising the following steps:

[0093] Acquire query information input by a user; the query information includes at least one of characters, text or images;

[0094] The query information and soft prompts are concatenated and input into the multimodal pre-trained model for feature extraction. The extracted feature vector is searched for similarity with the vector database to obtain the image-text data vector. According to the image-text data vector, the corresponding data is obtained from the image-text database as the retrieval result.

[0095] The query information and the corresponding search results are combined into a picture-text pair data set and stored in a picture-text database;

[0096] When it is detected that the collected image-text pair data set reaches a preset threshold, the stored data is used to train on the multimodal pre-trained model to obtain new soft prompts;

[0097] Among them, in the multimodal pre-training model, two types of soft prompts are used: modality-shared soft prompts and modality-specific soft prompts. By constructing modality-shared soft prompts and modality-specific soft prompts, the alignment of soft prompt information of different modalities is promoted, thereby promoting the alignment of information of different modalities.

[0098] An intelligent image and text retrieval method based on soft prompts in an embodiment of the present invention has a corresponding relationship with the above-mentioned system. The principle of solving the problem by the method is similar to that of the above-mentioned system, so the method has corresponding functions and beneficial effects to the system.

[0099] Example 3

[0100] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 6 An intelligent image and text retrieval method based on soft prompts is shown.

[0101] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0102] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0103] Since the electronic device is an electronic device corresponding to an intelligent image and text retrieval method based on soft prompts in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0104] Example 4

[0105] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 6 An intelligent image and text retrieval method based on soft prompts is shown.

[0106] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0107] Since the storage medium is a storage medium corresponding to an intelligent image and text retrieval method based on soft prompts in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0108] Example 5

[0109] In some possible implementations, various aspects of the method of the embodiment of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the intelligent image and text retrieval method based on soft prompts according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" used to execute each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0110] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0111] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0112] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent image-text retrieval system based on soft prompts, characterized in that: include: The search interface module is used to interact with the user, obtain the input query information, and display the search results; The query information includes at least one of characters, text or images; The model retrieval module is used to splice the query information and soft prompts and input them into the multimodal pre-trained model for feature extraction, perform similarity search between the extracted feature vector and the vector database to obtain the image and text data vector; and obtain the corresponding data from the image and text database according to the image and text data vector as the retrieval result; A search data collection module is used to combine the query information and the corresponding search results into a picture-text pair data set and store it in a picture-text database; The soft prompt efficient model update module is used to train the multimodal pre-trained model using the stored data when the collected image-text pair data set reaches a preset threshold, obtain new soft prompts, and use them to update the model retrieval module; Among them, in the multimodal pre-training model, two types of soft prompts are used: modality-shared soft prompts and modality-specific soft prompts; By constructing modality-shared soft prompts and modality-specific soft prompts, the alignment of soft prompt information of different modalities is promoted, thereby promoting the alignment of information of different modalities.

2. According to claim 1, the intelligent image and text retrieval system based on soft prompts is characterized in that: The retrieval interface module is specifically used for: After obtaining the input information, preprocessing the input information to perform the query parsing stage; the preprocessing includes cleaning processing, word segmentation processing or conversion processing; In the query parsing stage, natural language processing technology is used to analyze the user's intention to extract key information as the input of the model retrieval module.

3. The intelligent image-text retrieval system based on soft prompts according to claim 1, characterized in that: The model retrieval module is specifically used for: The query information and soft prompts are concatenated and input into the multimodal pre-trained model for feature extraction to obtain a feature vector; The extracted feature vectors are searched for similarity in the vector database to obtain relevant graphic and text data vectors; According to the image-text data vector, the corresponding data is obtained from the image-text database as the retrieval result.

4. The intelligent image-text retrieval system based on soft prompts according to claim 1, characterized in that: The soft hints are learnable vectors, and they are continuously updated during the training process to adapt to specific tasks or contexts, ensuring that the model can quickly absorb new knowledge without forgetting old knowledge while keeping the number of training parameters extremely low; Among them, the stored data is used to train the multimodal pre-trained model using a parameter-efficient soft prompt method.

5. The intelligent image-text retrieval system based on soft prompts according to claim 1, characterized in that: Modality-shared soft prompts can not only promote the interaction of information between different modalities, but also promote the alignment of soft prompts specific to different modalities, while further reducing the amount of training parameters to a certain extent; Modality-specific soft prompts are built on a common knowledge base and can facilitate the alignment of the two soft prompts. When set at different modality encoders, they can effectively adapt to the modality-specific information of each modality to guide the model to generate outputs that better meet user needs. The text encoder, image encoder, and multimodal encoder use their own modality-specific soft prompts to adapt to their own specific information. By using these two types of soft prompts, an effective combination of the multimodal method and the soft prompt method can be achieved.

6. The intelligent image-text retrieval system based on soft prompts according to claim 1, characterized in that: The soft prompt efficient update model module also includes a single-modal soft prompt fusion unit; The unimodal soft prompt fusion unit is used to promote the coordination between the unimodal soft prompt and the multimodal soft prompt, by fusing the unimodal soft prompt and adding the corresponding position elements with the multimodal soft prompt to form a new multimodal soft prompt, thereby achieving coordination and alignment between the two.

7. The intelligent image-text retrieval system based on soft prompts according to claim 1, characterized in that: The soft prompt efficient update model module adopts an automated testing and evaluation mechanism to ensure that the updated multimodal pre-trained model maintains or improves retrieval accuracy and response speed in practical applications; The soft prompt efficient update model module supports continuous iteration of the model. After the model training is completed, it is only necessary to replace the soft prompts in the model retrieval module with the newly generated soft prompts to complete the model update, so that the system can adapt to the ever-changing data and user needs.

8. The intelligent image-text retrieval system based on soft prompts according to claim 1, characterized in that: The retrieval data collection module is specifically used for: By collecting query information and corresponding search results, data processing and cleaning are performed during the data integration process to form a new graphic data set, which is stored in a location in the graphic database specifically used to store new data, for soft prompts to efficiently update the model.

9. An intelligent image and text retrieval method based on soft prompts, characterized in that: The following steps are involved: Acquire query information input by a user; the query information includes at least one of characters, text or images; The query information and soft prompts are concatenated and input into the multimodal pre-trained model for feature extraction. The extracted feature vector is searched for similarity with the vector database to obtain the image-text data vector. According to the image-text data vector, the corresponding data is obtained from the image-text database as the retrieval result. The query information and the corresponding search results are combined into a picture-text pair data set and stored in a picture-text database; When it is detected that the collected image-text pair data set reaches a preset threshold, the stored data is used to train on the multimodal pre-trained model to obtain new soft prompts; Among them, in the multimodal pre-training model, two types of soft prompts are used: modality-shared soft prompts and modality-specific soft prompts; By constructing modality-shared soft prompts and modality-specific soft prompts, the alignment of soft prompt information of different modalities is promoted, thereby promoting the alignment of information of different modalities.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method as claimed in claim 9.

Images