A soft prompt-based intelligent image-text retrieval system, method and electronic device

Abstract

The application discloses a kind of intelligent picture-text retrieval system, method and electronic equipment based on soft prompt, wherein system includes: retrieval interface module, model retrieval module, retrieval data collection module, soft prompt efficient update model module;Wherein soft prompt efficient update model module, for in the collected picture-text pair dataset reaches preset threshold, utilize the data stored on the multimodal pre-training model training, obtain new soft prompt, and update model retrieval module with it.The application utilizes the soft prompt method suitable for large-scale pre-training model, considers the characteristics of multimodal modeling, combines the soft prompt method with multimodal method to realize the best alignment of vector on picture-text retrieval task;Construct modal shared soft prompt and modal specific soft prompt to promote the alignment of different modal soft prompt information, and then promote the alignment of different modal information.The application can be widely applied in picture-text retrieval technical field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image-text retrieval, and in particular to an intelligent image-text retrieval system and method based on soft prompts and an electronic device. BACKGROUND

[0002] Existing intelligent image-text retrieval systems can implement image-text retrieval according to user needs, but face challenges in efficiently updating retrieval models with parameters. Because these systems often need a large amount of computing resources and time to retrain models to adapt to new data or improve algorithms. Therefore, intelligent image-text retrieval systems based on parameter-efficient methods have emerged, which can quickly update responses in real-time or near real-time application scenarios, better meeting user needs. However, existing research on intelligent image-text retrieval systems based on soft prompts often focuses on the use of soft prompts without considering the alignment and synergy between multi-modal soft prompts, which can result in soft prompts not being able to fully adjust the vector representations of different modal encoders (such as text encoders, image encoders, etc.), thus failing to achieve the optimal alignment of these vectors in image-text retrieval tasks. SUMMARY

[0003] To at least partially address one of the technical problems existing in the prior art, the present application aims to provide an intelligent image-text retrieval system and method based on soft prompts and an electronic device.

[0004] The first technical solution adopted by the present application is:

[0005] An intelligent image-text retrieval system based on soft prompts, comprising:

[0006] A retrieval interface module for interacting with a user, obtaining input query information, and displaying retrieval results; the query information includes at least one of characters, text, or images;

[0007] A model retrieval module for concatenating the query information and the soft prompts and inputting them into a multi-modal pre-training model for feature extraction, performing similarity search on the extracted feature vectors and a vector database to obtain image-text data vectors, and obtaining corresponding data from the image-text database as retrieval results according to the image-text data vectors;

[0008] A retrieval data collection module for grouping the query information and the corresponding retrieval results into image-text pair datasets and storing them in an image-text database for efficient updating of the retrieval model by the soft prompt efficient update model module;

[0009] A soft prompt efficient update model module for training on the multi-modal pre-training model using the stored data when the collected image-text pair datasets reach a preset threshold to obtain new soft prompts and update the model retrieval module with them;

[0010] Among them, in the multi-modal pre-training model, two kinds of soft prompts are used: modal shared soft prompts and modal specific soft prompts; by constructing modal shared soft prompts and modal specific soft prompts, the alignment of different modal soft prompt information is promoted, and then the alignment of different modal information is promoted.

[0011] Further, the retrieval interface module is specifically used for:

[0012] After obtaining the input information, the input information is preprocessed for query analysis stage; the preprocessing includes cleaning processing, word segmentation processing or conversion processing;

[0013] In the query analysis stage, the intention of the user is analyzed by using natural language processing technology to extract key information as the input of the model retrieval module.

[0014] Further, the model retrieval module is specifically used for:

[0015] The query information is spliced with the soft prompt, and is input into the multi-modal pre-training model for feature extraction to obtain a feature vector;

[0016] The extracted feature vector is searched for similarity in the vector database to obtain related image-text data vectors;

[0017] According to the image-text data vector, corresponding data is obtained from the image-text database as a retrieval result.

[0018] Further, the soft prompt is a learnable vector, and these soft prompts are continuously updated during the training process to adapt to specific tasks or contexts, ensuring that the model can quickly absorb new knowledge without forgetting old knowledge while ensuring that the training parameter amount is extremely low;

[0019] Among them, the stored data is trained on the multi-modal pre-training model using the parameter-efficient soft prompt method.

[0020] Further, the modal shared soft prompt can not only promote the interaction of information between different modalities, but also promote the alignment of different modal specific soft prompts, while further reducing the training parameter amount to a certain extent;

[0021] The modal specific soft prompt is constructed based on the common knowledge base, which can promote the alignment of the two kinds of soft prompts, and can effectively adapt to the information specific to each modal when set at the different modal encoder to guide the model to generate output more in line with user needs;

[0022] The text encoder, the image encoder and the multi-modal encoder use respective modal-specific soft prompts to adapt to respective specific information; by using the two kinds of soft prompts, the effective combination of the multi-modal method and the soft prompt method is realized.

[0023] Further, the soft prompt efficient updating model module further comprises a single-modal soft prompt fusion unit.

[0024] The single-modal soft prompt fusion unit is used to promote the cooperation between the single-modal soft prompt and the multi-modal soft prompt, and by fusing the single-modal soft prompt and adding it with the multi-modal soft prompt at corresponding position elements, a new multi-modal soft prompt is formed, so that the cooperation and alignment of the two are realized.

[0025] Further, the soft prompt efficient updating model module adopts an automatic testing and evaluation mechanism to ensure that the updated multi-modal pre-training model maintains or improves the retrieval accuracy and response speed in actual application.

[0026] The soft prompt efficient updating model module supports continuous iteration of the model, and after the model training is completed, only the newly generated soft prompt is used to replace the soft prompt in the model retrieval module, the updating of the model is completed, and the system can adapt to the changing data and user demand.

[0027] Further, the retrieval data collection module is specifically used for:

[0028] By collecting query information and corresponding retrieval results, data processing and cleaning are performed in the data integration process, a new image-text data set is formed, and is stored in a position of the image-text database specially used for storing new data, for the soft prompt efficient updating model.

[0029] The second technical solution adopted by the present application is:

[0030] An intelligent image-text retrieval method based on a soft prompt, comprising the following steps:

[0031] Obtaining query information input by a user; the query information comprises at least one of characters, texts or images;

[0032] After the query information and the soft prompt are spliced, they are input into a multi-modal pre-training model for feature extraction, a feature vector obtained by the extraction is subjected to similarity search with a vector database, and an image-text data vector is obtained; corresponding data from the image-text database is obtained as a retrieval result according to the image-text data vector;

[0033] The query information and the corresponding retrieval result are combined to form an image-text pair data set and are stored in the image-text database.

[0034] When it is detected that the collected image-text pair data set reaches a preset threshold, the stored data is used to train the multi-modal pre-training model to obtain new soft prompts;

[0035] Among the multi-modal pre-training model, two kinds of soft prompts are used: modal shared soft prompts and modal specific soft prompts; by constructing modal shared soft prompts and modal specific soft prompts, the alignment of different modal soft prompt information is promoted, and then the alignment of different modal information is promoted.

[0036] The third technical solution adopted by the present application is:

[0037] An electronic device, the electronic device comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the intelligent image-text retrieval method based on soft prompts as described above.

[0038] The fourth technical solution adopted by the present application is:

[0039] A computer-readable storage medium, the storage medium having stored therein at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by a processor to implement the intelligent image-text retrieval method based on soft prompts as described above.

[0040] The fifth technical solution adopted by the present application is:

[0041] A computer program product or computer program, the computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to make the computer device execute the intelligent image-text retrieval method based on soft prompts as described above.

[0042] The present application has the beneficial effects that: the present application uses a soft prompt method suitable for large-scale pre-training models, considers the characteristics of multi-modal modeling, combines the soft prompt method with the multi-modal method to achieve the best alignment of vectors on the image-text retrieval task; modal shared soft prompts and modal specific soft prompts are constructed to promote the alignment of different modal soft prompt information, and then the alignment of different modal information is promoted. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for facilitating the clear description of part of the embodiments of the technical solutions of the present application, and for the technical personnel in the art to obtain other drawings without paying creative labor.

[0044] Figure 1 is a structural block diagram of an intelligent image-text retrieval system based on parameter efficient alignment soft prompt in the embodiments of the present application;

[0045] Figure 2 is a structural block diagram of a retrieval interface module in the embodiments of the present application;

[0046] Figure 3 is a structural block diagram of a model retrieval module in the embodiments of the present application;

[0047] Figure 4 is a structural block diagram of a retrieval data collection module in the embodiments of the present application;

[0048] Figure 5 is a structural block diagram of a soft prompt efficient update model module in the embodiments of the present application;

[0049] Figure 6 is a step flow chart of an intelligent image-text retrieval method based on soft prompt in the embodiments of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for facilitating the description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of the technical personnel in the art.

[0051] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0052] In the description of the present application, the meaning of one or more is one or more, the meaning of multiple is two or more, greater than, less than, more than, etc. are understood as not including the number, above, below, within, etc. are understood as including the number. If the first, second is described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the order of indicated technical features.

[0053] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0054] Technical explanation:

[0055] (1) Multi-modal large-scale pre-training model

[0056] Multi-modal large-scale pre-training model is an advanced artificial intelligence technology that can handle and understand multiple types of data such as text, images, audio and video. These models are trained through deep learning techniques and can extract features from different modalities of data and establish connections, providing a more comprehensive and in-depth understanding. Their core advantage lies in the ability to integrate information from different sources, enhancing the predictive and reasoning capabilities of the model. This integration not only improves the processing capabilities of single-modal data, but also enables the model to more accurately simulate human perception and cognitive processes. Multi-modal large models have a wide range of applications in natural language processing, computer vision, speech recognition and other fields, and can support complex tasks such as joint search of images and text, video content understanding, sentiment analysis, etc. With the advancement of technology, these models are becoming increasingly intelligent, capable of handling more complex data and tasks, and driving the application and development of artificial intelligence in various fields.

[0057] (2) Parameter efficient transfer method for large-scale pre-training model

[0058] Parameter efficient transfer learning methods for large-scale pre-training models, such as adapters and soft prompts, mainly achieve knowledge transfer by introducing additional parameters or prompt information in key parts of the model. The adapter method adds a small parameter module to the input or output layer of the model, allowing the model to fine-tune for specific tasks without changing the parameters of the entire pre-training model. This method allows the model to quickly adapt to new tasks while maintaining existing knowledge. The soft prompt method embeds task-related prompt information into the model input, guiding the model to generate output that better meets the requirements of specific tasks. These methods reduce the number of parameters that need to be trained, improving the efficiency of transfer learning, allowing pre-trained models to flexibly handle various new tasks while reducing the demand for computing resources.

[0059] (3) Existing intelligent image-text retrieval system

[0060] The existing intelligent image-text retrieval system is an advanced system that integrates computer vision and natural language processing technologies. It uses multi-modal large-scale pre-training models to recognize and understand image content and related text information. These systems can handle user natural language queries, implement semantic search of image and text data, and provide highly relevant search results. They build efficient data indexes and databases to quickly respond to a large number of search requests, and have intuitive user interfaces that allow users to easily search and browse.

[0061] Embodiment 1

[0062] As shown in Figure 1 , the embodiment provides an intelligent image-text retrieval system based on parameter efficient alignment soft prompt, specifically comprising:

[0063] A retrieval interface module for interacting with users, obtaining input query information, and displaying search results; the query information includes at least one of characters, text or images;

[0064] A model retrieval module for concatenating the query information and the soft prompt and inputting them into the multi-modal pre-training model for feature extraction, performing similarity search on the extracted feature vectors and the vector database to obtain image-text data vectors; and obtaining corresponding data from the image-text database as the search results according to the image-text data vectors;

[0065] A retrieval data collection module for grouping the query information and the corresponding search results into image-text pair datasets and storing them in the image-text database for efficient updating of the search model by the soft prompt efficient updating model module;

[0066] A soft prompt efficient updating model module for training the stored data on the multi-modal pre-training model when the collected image-text pair datasets reach a preset threshold, obtaining new soft prompts, and updating the model retrieval module with them;

[0067] In the multi-modal pre-training model, two types of soft prompts are used: modality-shared soft prompts and modality-specific soft prompts; by constructing modality-shared soft prompts and modality-specific soft prompts, the alignment of different modality soft prompt information is promoted, and the alignment of different modality information is promoted.

[0068] The intelligent image-text retrieval system using parameter-efficient alignment soft prompt for multi-modal modeling provided in this embodiment includes a retrieval interface module, a model retrieval module, a retrieval data collection module, and a soft prompt efficient update model module. Users can use the image-text retrieval function in the retrieval interface module. When using this function, the retrieval model in the model retrieval module is used to retrieve related content in the vector database and the image-text database and return. At the same time, the retrieval data collection module collects and records the user's query and the system's answer, integrates and stores them in the vector database and the image-text database, and uses the soft prompt efficient update model module to update the retrieval model.

[0069] Exemplarily, the workflow of the system of this embodiment is as follows: 1) First, the user performs retrieval through natural language query, keyword input, or image upload in the retrieval interface module. 2) The retrieval interface module analyzes the user's query, extracts the key information, and sends it to the model retrieval module and the retrieval data collection module. 3) The model retrieval module converts the query information into a vector, retrieves and compares it in the vector database, obtains the retrieval result, and returns the corresponding result in the image-text database to the retrieval data collection module and the retrieval interface module, and the retrieval interface module further returns the result to the user. 4) The retrieval data collection module records the user's query and the system's answer to form an image-text pair dataset, which is stored in the image-text database and used by the soft prompt efficient update model module to efficiently update the retrieval model. 5) When the collected retrieval data reaches the data volume threshold, the soft prompt efficient update model module uses the collected data to train the pre-trained model using the parameter-efficient soft prompt method to obtain new soft prompts and update the model retrieval module.

[0070] It should be noted that in this embodiment, the parameter-efficient soft prompt method is used in the soft prompt efficient update model module to efficiently update the retrieval model, but it is not limited to this. Other methods such as the adapter method and the lora method can also be used, which should fall within the protection scope of this application.

[0071] The system and each module are explained in detail below in combination with the drawings and specific embodiments.

[0072] (1) Retrieval interface module

[0073] Referring to Figure 2 , the retrieval interface module provides an intuitive and easy-to-use interactive interface, allowing users to conveniently input queries, upload images or files, and receive retrieval results for browsing.

[0074] Specifically, the retrieval interface module first receives the user's input, which can be a text query or an image upload. Then, the system processes the input, including steps such as cleaning, tokenization, and conversion, to prepare for query parsing. In the query parsing stage, the system analyzes the user's intent and extracts key information using relevant natural language processing techniques. The interaction management displays the retrieval results on the user interface while providing interactive functions, allowing the user to filter, sort, or provide feedback on the results.

[0075] (2) Model retrieval module

[0076] Referring to Figure 3 , the model retrieval module receives the query information from the retrieval interface module, then concatenates the soft prompt with the corresponding input, and inputs it into the pre-trained model for in-depth feature extraction to obtain the corresponding feature vector. Based on the extracted feature vector, the system will perform efficient similarity search in the vector database, quickly match and retrieve the most relevant image-text data vector to the query, optimize and sort the retrieval results, and obtain the corresponding index in the image-text database, and return the corresponding data in the image-text database as the result according to the optimized and sorted results.

[0077] (3) Retrieval data collection module

[0078] Referring to Figure 4 , the retrieval data collection module collects user queries and retrieval results, and in the data integration process, it processes and cleans the data to form a new image-text data set and stores it in a special location in the image-text database for storing new data, which is used for efficient updating of the soft prompt model.

[0079] (4) Soft prompt efficient updating model module

[0080] Referring to Figure 5 , the soft prompt efficient updating model module is a component in the system for continuously optimizing the performance of the model. It introduces new image-text data to fine-tune the pre-trained model through regular or conditionally triggered methods. This module uses a parameter-efficient soft prompt method, which embeds learnable soft prompts in the model input. These soft prompts are usually learnable vectors, and they are updated during the training process to adapt to specific tasks or contexts. This ensures that the model can quickly absorb new knowledge without forgetting old knowledge while keeping the training parameter size extremely low.

[0081] Specifically, in the multi-modal pre-training model, two kinds of soft prompts are used: a shared prompt for modalities (SharedPrompt) and a modal-specific soft prompt (Text / Vision / Multimodal Prompt). The modal-specific soft prompt is constructed based on a common source soft prompt and is used in the text encoder, image encoder, and multi-modal encoder to adapt to the specific information of each modality. By using these two kinds of soft prompts, the multi-modal method and the soft prompt method can be effectively combined. The shared prompt for modalities not only facilitates the interaction of information between different modalities, but also promotes the alignment of different modal-specific soft prompts, while also reducing the number of training parameters to some extent. The modal-specific soft prompt is constructed based on common knowledge and can promote their alignment. When placed in different modality encoders, it can effectively adapt to the specific information of each modality and guide the model to generate outputs that better meet user needs. In addition, a single-modal soft prompt fusion unit is introduced to promote the cooperation between single-modal soft prompts and multi-modal soft prompts. By fusing single-modal soft prompts and adding them to multi-modal soft prompts at corresponding positions, a new multi-modal soft prompt is formed to realize the cooperation and alignment of the two.

[0082] In addition, the soft prompt efficient update model module will adopt an automated testing and evaluation mechanism to ensure that the updated model maintains or improves retrieval accuracy and response speed in actual applications. This module supports the continuous iteration of the retrieval model, and after training, only the newly generated soft prompts need to be replaced in the model retrieval module to complete the update of the model retrieval model, allowing the system to adapt to changing data and user needs.

[0083] (5) Vector database and image-text database

[0084] The vector database stores pre-extracted and indexed image-text feature vectors and image-text alignment vectors, facilitating fast comparison of the similarity between stored vectors and generated vectors. The image-text database stores the original data of the image-text, and the image-text data corresponding to the image-text feature vector can be found according to the index stored in the vector database.

[0085] Overall, the embodiment proposes an intelligent image-text retrieval system based on parameter efficient alignment soft prompt, which uses a soft prompt method suitable for large-scale pre-training model, and considers the characteristics of multi-modal modeling. The soft prompt method is combined with the multi-modal method to achieve the best alignment of vectors in the image-text retrieval task. Specifically, the embodiment first constructs modal-shared soft prompt and modal-specific soft prompt (where the modal-specific soft prompt is constructed based on common source soft prompt) to promote the alignment of soft prompt information of different modalities, and then promote the alignment of information of different modalities. In addition, a single-modal soft prompt fusion unit is proposed to promote the alignment and cooperation between single-modal soft prompt and multi-modal soft prompt, and then promote the multi-modal vector to better capture the alignment information between images and texts. This scheme not only greatly reduces the amount of parameters that need to be trained when updating the retrieval model, saves resource and time cost, but also considers the alignment and cooperation between multi-modal soft prompts, thereby guiding the model to generate output that better meets user needs.

[0086] In summary, compared with the prior art, the present application has at least the following advantages and beneficial effects:

[0087] (1) The present application proposes an intelligent image-text retrieval system based on parameter efficient alignment soft prompt, which uses a parameter efficient soft prompt method to efficiently update and deploy the image-text retrieval model on the collected image-text dataset. The system can efficiently update the image-text retrieval model, and because the parameter efficient soft prompt method is used, it can save a lot of computing resources and time cost while ensuring accuracy, solving the problem of long update cycle and large resource consumption of existing intelligent image-text retrieval systems.

[0088] (2) The present application proposes a parameter efficient soft prompt method suitable for multi-modal large-scale pre-training model, which can achieve high accuracy of image-text retrieval by only training the soft prompt part under the condition of keeping the parameters of the multi-modal large-scale pre-training model frozen.

[0089] The proposed soft prompt method suitable for multi-modal large-scale pre-training model first proposes to use two kinds of soft prompts: modal-shared and modal-specific soft prompts to realize the combination of multi-modal method and soft prompt. The modal-shared soft prompt can promote the interaction of information between different modalities, and at the same time can promote the alignment of different modal-specific soft prompts. The modal-specific soft prompt is constructed based on common knowledge base (source soft prompt), which can promote the alignment of different modal-specific soft prompts to a certain extent. The other is the cooperation between single-modal and multi-modal soft prompts, which forms a new multi-modal soft prompt by adding the single-modal soft prompt to the multi-modal soft prompt to realize the cooperation and alignment of the two. After training, only the newly generated soft prompt is used to replace the soft prompt in the model retrieval module to complete the efficient update of the model retrieval model.

[0090] (3) The application proposes a retrieval data collection module to collect user retrieval data in real time, for real-time updating of the retrieval model, to provide better retrieval services for users. The retrieval data collection module used in the application can automatically collect user retrieval data to further improve the image-text database, and use the collected user retrieval data to further train the soft prompt. After the training is completed and the deployment is updated, the user's retrieval requirements can be better met.

[0091] Embodiment 2

[0092] As Figure 6 shown, the embodiment provides an intelligent image-text retrieval method based on soft prompt, including the following steps:

[0093] Obtaining query information input by a user; the query information includes at least one of characters, texts or images;

[0094] Concatenating the query information and the soft prompt and inputting them into a multi-modal pre-training model for feature extraction, performing similarity search on the obtained feature vector and a vector database to obtain an image-text data vector, and obtaining corresponding data from the image-text database according to the image-text data vector as a retrieval result;

[0095] Grouping the query information and the corresponding retrieval result into an image-text pair dataset and storing the image-text pair dataset in an image-text database;

[0096] When it is detected that the collected image-text pair dataset reaches a preset threshold, training the stored data on the multi-modal pre-training model to obtain a new soft prompt;

[0097] In the multi-modal pre-training model, two kinds of soft prompts are used: modality-shared soft prompts and modality-specific soft prompts; by constructing the modality-shared soft prompts and the modality-specific soft prompts, the alignment of different modality soft prompt information is promoted, and the alignment of different modality information is promoted.

[0098] The intelligent image-text retrieval method based on soft prompt of the embodiment of the application has a corresponding relationship with the above-mentioned system, the problem solving principle of the method is similar to that of the system, and therefore the method has corresponding functions and beneficial effects of the system.

[0099] Embodiment 3

[0100] The embodiment of the application further provides an electronic device, which includes a processor and a memory, the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to realize an intelligent image-text retrieval method based on soft prompt as shown in Figure 6 .

[0101] It can be understood that the memory can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing various method embodiments described above, etc.; and the data storage area can store data created according to the use of the server, etc.

[0102] The processor can include one or more processing cores. The processor connects various parts within the entire server through various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with a combination of one or more of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs, etc.; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be implemented by a separate chip.

[0103] Since the electronic device is an electronic device corresponding to the intelligent image-text retrieval method based on soft prompt according to the embodiments of the application, and the principle of solving problems of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described here.

[0104] Embodiment 4

[0105] The embodiments of the application also provide a computer readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement the intelligent image-text retrieval method based on soft prompt as shown in Figure 6

[0106] ​Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be instructed by programs to relevant hardware, and the programs can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0107] Since the storage medium is a storage medium corresponding to the intelligent image-text retrieval method based on soft prompt of the embodiments of the present application, and the problem solving principle of the storage medium is similar to the method, the implementation of the storage medium can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0108] Embodiment 5

[0109] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the intelligent image-text retrieval method based on soft prompt according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Wherein, the executable computer program codes or "codes" for executing various embodiments can be written in high-level programming languages such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0110] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combination, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and so forth.

[0111] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Furthermore, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0112] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A soft-prompt based intelligent image-text retrieval system, characterized in that, The method comprises the following steps: a retrieval interface module for interacting with the user, obtaining input query information, and displaying retrieval results; The query information includes at least one of characters, text or images; a model retrieval module for concatenating the query information and the soft prompt and inputting them into a multi-modal pre-training model for feature extraction, performing similarity search on the extracted feature vectors in a vector database to obtain a graph-text data vector, and obtaining corresponding data from the graph-text database according to the graph-text data vector as the retrieval result; a retrieval data collection module for grouping the query information and the corresponding retrieval result into a graph-text pair dataset and storing it in the graph-text database; a soft prompt efficient update model module for training the stored data on the multi-modal pre-training model when the collected graph-text pair dataset reaches a preset threshold, obtaining new soft prompts, and updating the model retrieval module with the new soft prompts; In the multi-modal pre-training model, two types of soft prompts are used: modality-shared soft prompts and modality-specific soft prompts; The modality-specific soft prompts are constructed based on a common knowledge base; the text encoder, the image encoder and the multi-modal encoder use their respective modality-specific soft prompts to adapt to their specific information; The soft prompt efficient update model module further comprises a single-modal soft prompt fusion unit; the single-modal soft prompt fusion unit is used to promote the cooperation between the single-modal soft prompt and the multi-modal soft prompt, form a new multi-modal soft prompt by fusing the single-modal soft prompt and adding the corresponding position elements of the multi-modal soft prompt; The soft prompt efficient update model module adopts an automatic testing and evaluation mechanism; the soft prompt efficient update model module supports continuous iteration of the model, and only needs to replace the soft prompts in the model retrieval module with newly generated soft prompts to update the model after the model training is completed.

2. The intelligent text retrieval system based on soft cues according to claim 1, wherein, The retrieval interface module is specifically used for: After obtaining the input information, pre-processing the input information to perform query analysis; the pre-processing includes cleaning, word segmentation or conversion processing; In the query analysis stage, the user's intention is analyzed by using natural language processing technology to extract key information as the input of the model retrieval module.

3. The intelligent text retrieval system based on soft cues according to claim 1, wherein, The model retrieval module is specifically used for: Concatenating the query information and the soft prompt and inputting them into a multi-modal pre-training model for feature extraction to obtain a feature vector; Performing similarity search on the extracted feature vector in a vector database to obtain related graph-text data vectors; According to the graph-text data vector, the corresponding data is obtained from the graph-text database as the retrieval result.

4. The intelligent text retrieval system based on soft cues according to claim 1, wherein, The soft prompt is a learnable vector, and these soft prompts are updated continuously during the training process to adapt to specific tasks or contexts; The stored data is trained on the multi-modal pre-training model using a parameter-efficient soft prompt method.

5. The intelligent soft-prompt based image-text retrieval system of claim 1, wherein, The retrieval data collection module is specifically used for: By collecting query information and corresponding retrieval results, performing data processing and cleaning during data integration to form a new graph-text dataset, and storing it in a position in the graph-text database dedicated for storing new data for the soft prompt efficient update model.

6. A soft-prompt-based intelligent image-text retrieval method applied to the system of any one of claims 1-5, characterized in that, The method comprises the following steps: obtaining query information input by a user; the query information comprises at least one of a character, a text or an image; concatenating the query information and a soft prompt and inputting the same into a multi-modal pre-training model to perform feature extraction, performing similarity search on a feature vector obtained through the extraction and a vector database to obtain a graph-text data vector; and obtaining corresponding data from a graph-text database according to the graph-text data vector as a search result; grouping the query information and the corresponding search result into a graph-text pair dataset and storing the same in the graph-text database; when it is detected that the collected graph-text pair dataset reaches a preset threshold, training the stored data on the multi-modal pre-training model to obtain a new soft prompt; in the multi-modal pre-training model, two kinds of soft prompts are used: a modality-shared soft prompt and a modality-specific soft prompt; by constructing the modality-shared soft prompt and the modality-specific soft prompt, alignment of soft prompt information of different modalities is promoted, and then alignment of information of different modalities is promoted.

7. An electronic device, comprising: The electronic device comprises a processor and a memory, the memory stores at least one program, the at least one program is loaded and executed by the processor to implement the method of claim 6.

Images