Method for training combined image retrieval model and combined image retrieval method and device

Abstract

The invention provides a method for training a combined image retrieval model and a combined image retrieval method and device. The method comprises a training stage and a retrieval stage. The method comprises the following steps: constructing lt; a reference image, a modified text and a target image gt; according to the triple data set, the sample data set is amplified by using the image difference description model, so that compared with manual annotation, the annotation cost is reduced, and the efficiency is improved; the image-text combination features are obtained by decoupling and fusing the reference image and modifying the text features, and the parameters are updated by calculating the similarity with the target image features, so that the image-text combination is realized on finer granularity, the more accurate image-text combination features are obtained, and the features of the ideal target image are better reflected. Compared with an existing method, the method has the advantages that the recall rate of combined image retrieval is increased, the data annotation cost is reduced, the image retrieval requirement in an interactive scene can be better met, and the user experience is optimized.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method for training a combined image retrieval model, a combined image retrieval method and a combined image retrieval device. Background Art

[0002] With the development of the Internet, the size of image data on the Internet is increasing at an unprecedented rate. Image retrieval tasks focus on how to quickly, accurately, and flexibly retrieve corresponding target images from massive image data according to user needs. It has important applications in real-world fields such as security monitoring, product retrieval, face verification, and search engines. Therefore, the research on image retrieval methods is very important.

[0003] The current image retrieval mainly includes image search, text search, combined image retrieval and other scenarios. Among them, image search and text search only support a single search for the target image based on the query image or text; the combined image retrieval method expresses the query as a combination of the reference image and the modified text, and retrieves the target image that is visually similar and meets the text modification requirements. This method combines the bimodal query form of image and text, allowing users to more accurately control the features of the required image, and can more accurately capture user intentions, making image search more flexible and efficient. However, the mainstream combined image retrieval method has the following problems: 1) In the feature combination stage, the complete reference image features and modified text features are simply weighted and fused, without interaction and alignment of image and text features, and without extracting key semantic information and filtering redundant information; 2) The training of the combined image retrieval model relies on a large number of <reference image, modified text, target image> triplets, and the cost of manual annotation is high.

[0004] Therefore, there is an urgent need for an efficient and accurate method that can not only extract the key semantic information of image and text features and filter out redundant information, but also reduce the amount of manual annotation and training costs. Summary of the invention

[0005] In view of the above problems, this application provides a method for training a combined image retrieval model, a combined image retrieval method and a device by means of a deep learning method. The specific scheme is as follows:

[0006] A method for training a combined image retrieval model, the combined image retrieval model comprising an image encoding sub-model, a text encoding sub-model, an image decoupling sub-model and a text decoupling sub-model, the method comprising:

[0007] Acquire sample data, wherein the sample data includes <reference image, modified text, target image> triplet data;

[0008] The image encoding sub-model is used to extract the features of the reference image and the target image respectively, and the text encoding sub-model is used to extract the features of the modified text;

[0009] The features of the modified text are decoupled by using the text decoupling submodel to obtain common text features and unique text features; the features of the reference image and the common text features are processed by using the image decoupling submodel to obtain common features of the reference image; the features of the target image and the common text features are processed by using the image decoupling submodel to obtain common features of the target image;

[0010] Merging the common features of the reference image and the unique text features to obtain image-text combined features;

[0011] An image retrieval loss function is constructed to train the combined image retrieval model.

[0012] Optionally, it also includes augmenting the sample data set using an image difference description model, wherein the image difference description model determines, based on each <reference image, target image> data pair, to convert the target image into a difference text of the reference image to obtain corresponding <target image, difference text, reference image> triplet data as augmented data.

[0013] Optionally, the image encoding sub-model is an image encoder in a pre-trained CLIP model, and the text encoding sub-model is a text encoder in a pre-trained CLIP model.

[0014] Optionally, the using the image decoupling sub-model to process the features of the reference image and the common text features to obtain the common features of the reference image includes:

[0015] The features of the reference image and the common text features are used as query features and key features, the attention weights are calculated using the cross-attention mechanism, and the common features of the reference image are calculated based on the attention weights and the features of the reference image.

[0016] Optionally, constructing an image retrieval loss function to train the combined image retrieval model includes:

[0017] Constructing a first loss function based on the image-text combination feature and the feature of the target image;

[0018] Constructing a second loss function based on the common features of the reference image and the common features of the target image;

[0019] A total loss function is constructed based on the first loss function and the second loss function, and the image encoding sub-model, the text encoding sub-model, the image decoupling sub-model and the text decoupling sub-model are trained based on the total loss function.

[0020] A combined image retrieval method, comprising:

[0021] Inputting the query image and the modified text into the combined image retrieval model to obtain the image-text combined features of the query image;

[0022] Determining the similarity between the image-text combination feature and features of a plurality of candidate images in an image database;

[0023] An image retrieval result corresponding to the query image is determined according to the similarity.

[0024] A device for training a combined image retrieval model, wherein the combined image retrieval model includes an image encoding sub-model, a text encoding sub-model, an image decoupling sub-model and a text decoupling sub-model, and the device includes:

[0025] An acquisition module, used for acquiring sample data, wherein the sample data includes <reference image, modified text, target image> triplet data;

[0026] A feature extraction module, used to respectively extract features of a reference image and a target image using the image encoding sub-model, and to extract features of a modified text using the text encoding sub-model;

[0027] a decoupling module, configured to decouple the features of the modified text using the text decoupling submodel to obtain common text features and unique text features, process the features of the reference image and the common text features using the image decoupling submodel to obtain reference image common features, and process the features of the target image and the common text features using the image decoupling submodel to obtain target image common features;

[0028] A feature fusion module, used to fuse the common features of the reference image and the unique text features to obtain image-text combined features;

[0029] The training module is used to construct an image retrieval loss function and train the combined image retrieval model.

[0030] A combined image retrieval device, comprising:

[0031] An acquisition module, used for inputting the query image and the modified text into the combined image retrieval model to obtain the image-text combined features of the query image;

[0032] A first determination module is used to determine the similarity between the image-text combination feature and a plurality of candidate image features in an image database;

[0033] The second determination module is used to determine the image retrieval result corresponding to the query image according to the similarity.

[0034] An electronic device includes a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method provided by the present disclosure.

[0035] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method provided by the present disclosure is executed.

[0036] The present disclosure first decouples text features when combining queries of image modality and text modality, extracts unique text features to participate in subsequent image-text combination, filters redundant common text features, decouples reference images by using common text features, pays attention to (extracts) key features that should be retained in the reference image (i.e., common features of the reference image) to participate in subsequent image-text combination, filters redundant features that should be deleted in the reference image, and can realize image-text combination at a finer granularity, obtain more accurate image-text combination features, better reflect the features of the ideal target image, and thus have a higher recall rate when performing combined image retrieval; when training the model, the original triplet data is expanded by using the image difference description method, which significantly reduces the annotation cost and improves the efficiency compared to manual annotation. Through the above optimization, the present disclosure can meet the image retrieval requirements of related applications in interactive scenarios and optimize the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flowchart of a method for training a combined image retrieval model according to an embodiment of the present disclosure;

[0038] Figure 2 is a flowchart of a combined image retrieval method according to an embodiment of the present disclosure;

[0039] Figure 3 is a block diagram of an apparatus for training a combined image retrieval model according to an embodiment of the present disclosure;

[0040] Figure 4 is a block diagram of a combined image retrieval device according to an embodiment of the present disclosure;

[0041] Figure 5 It is a block diagram of an electronic device used to implement the method for training a combined image retrieval model and / or the combined image retrieval method according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0042] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0043] Figure 1 The present invention is a flowchart of a method for training a combined image retrieval model according to an embodiment of the present disclosure.

[0044] like Figure 1 As shown, the method 100 for training a combined image retrieval model may include operations S110 to S150. The combined image retrieval model may include an image encoding sub-model, a text encoding sub-model, an image decoupling sub-model, and a text decoupling sub-model.

[0045] It should be understood that although Figure 1 The steps in the flowchart are shown as indicated by arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0046] In operation S110, sample data is acquired, where the sample data includes <reference image, modified text, target image> triple data.

[0047] In the embodiment of the present disclosure, the sample data may be acquired from a sample database.

[0048] Those skilled in the art can understand that a plurality of <reference image, modified text, target image> triplet data are stored in the sample database, wherein each reference image has a one-to-one corresponding modified text and target image.

[0049] In an embodiment of the present disclosure, an image difference description model can be used to augment a sample data set. The image difference description model determines, based on each <reference image, target image> data pair, to convert the target image into a difference text of the reference image to obtain the corresponding <target image, difference text, reference image> triplet data as augmented data.

[0050] For example, the dataset consisting of the above <reference image, modified text, target image> triples can be expressed as ,in is the number of samples in the dataset; for each triplet, the target image and reference image Input image difference description model, output difference text ,in, is an image difference description model that can output a description of how to Modified to Diff text of; build < target image , Difference text , reference image > The triplet is added as a new training sample to the original data set , get a new data set .

[0051] For example, the above-mentioned image difference description model can be a Transformer model.

[0052] Those skilled in the art can understand that, after being processed by the above-mentioned image difference description model, each triplet data in the original data set can generate a new triplet, so the number of samples in the new data set after expansion is twice that of the original data set.

[0053] In operation S120, the features of the reference image and the features of the target image are respectively extracted using the image encoding sub-model, and the features of the modified text are extracted using the text encoding sub-model.

[0054] In the embodiment of the present disclosure, the above-mentioned image encoding sub-model may be an image encoder in a pre-trained CLIP model, and the above-mentioned text encoding sub-model may be a text encoder in a pre-trained CLIP model.

[0055] For example, the LAION-2B dataset can be used as pre-training data for the CLIP model.

[0056] Those skilled in the art will appreciate that using a pre-trained CLIP model to perform feature encoding on images and texts can better align features of the image and text modalities and reduce the semantic gap between the image and text modalities.

[0057] For example, for the dataset Each triple sample in < , , >, you can add the reference image Input the image encoder of the pre-trained CLIP model and output the reference image features ; Similarly, the target image can be Image encoder, output target image features ; Will modify the text Input the text encoder of the pre-trained CLIP model to obtain the modified text features .

[0058] In operation S130, the features of the modified text are decoupled using the text decoupling submodel to obtain common text features and unique text features; the features of the reference image and the common text features are processed using the image decoupling submodel to obtain common features of the reference image; the features of the target image and the common text features are processed using the image decoupling submodel to obtain common features of the target image.

[0059] For example, you can modify the text feature Input text decoupling module, output common text features and unique text features ,in , are two different fully connected layers, which are a basic layer in deep neural networks and are used to map input features into features with different semantics.

[0060] In the disclosed embodiment, the features of the reference image and the common text features can be used as query features and key features, the attention weights can be calculated using a cross-attention mechanism, and the common features of the reference image can be calculated based on the attention weights and the features of the reference image.

[0061] For example, the reference image features and common text features Input image decoupling module, use cross attention mechanism to calculate corresponding attention weights ,in, , are two different fully connected layers, respectively taking the reference image features and common text features Mapped to corresponding query features and key features, and then through The function calculates the attention weight, which represents the reference image feature Each dimension and common text features The relevance of and reference image features Calculate the common features of the reference image , represents the features in the reference image that are similar to the target image.

[0062] For example, the target image features can be and common text features Input image decoupling module, and obtain the common features of the target image in a similar way , to supervise the extraction of common features of the reference image.

[0063] Those skilled in the art can understand that the above-mentioned text decoupling sub-model is used to decouple the modified text into features with different semantics. Traditional combined image retrieval usually directly fuses the reference image features and the modified text features to obtain the image-text combined features, while the text decoupling sub-model designed in the present disclosure decouples the modified text features more finely into common text features and unique text features through a fully connected layer, wherein: 1) the common text features are used to extract the features in the reference image that should be retained and participate in the image-text combination stage (i.e., the common features of the reference image and the target image) in the image decoupling sub-model; 2) the unique text features only contain the descriptive semantics of the target image features, and do not contain the descriptive semantics of the reference image features, and are used to participate in the subsequent image-text feature combination.

[0064] In operation S140, the common features of the reference images and the unique text features are fused to obtain image-text combined features.

[0065] For example, the reference image can share features and unique text features Perform dynamic weighted fusion and output image and text combination features

[0066] ,

[0067] in , For the feature vector , To splice, A fully connected layer that further maps the input features. is an activation function, which is used to map the input feature to a value between 0 and 1. It represents the weight of the image modality feature in the image-text combination feature, so as to dynamically adjust the contribution of the image-text modality to the image-text combination feature in different samples.

[0068] In operation S150, an image retrieval loss function is constructed to train the combined image retrieval model.

[0069] In the disclosed embodiment, a first loss function can be constructed based on the above-mentioned image-text combination features and the features of the above-mentioned target image; a second loss function can be constructed based on the common features of the above-mentioned reference image and the common features of the above-mentioned target image; a total loss function can be constructed based on the above-mentioned first loss function and the above-mentioned second loss function, and the above-mentioned image encoding sub-model, text encoding sub-model, image decoupling sub-model and text decoupling sub-model can be trained based on the total loss function.

[0070] For example, the above image-text combination features can be calculated and target image features The multi-classification loss is used as the first loss function

[0071] ,

[0072] in is the batch size, Calculate the eigenvector , The cosine similarity of .

[0073] For example, common features of the reference image can be calculated Shares features with the target image The mean square error loss is used as the second loss function .

[0074] For example, the total loss function can be , use the stochastic gradient descent algorithm for back propagation to update the parameters of the above-mentioned image encoding sub-model, text encoding sub-model, image decoupling sub-model and text decoupling sub-model, and finally obtain the trained combined image retrieval model.

[0075] Those skilled in the art will understand that by decoupling and fusing the reference image and modified text features to obtain image-text combination features, and updating the parameters by calculating the similarity with the target image features, it is possible to achieve image-text combination at a finer granularity, obtain more accurate image-text combination features, better reflect the features of the ideal target image, and improve the recall rate of combined image retrieval.

[0076] Figure 2 is a flowchart of a combined image retrieval method according to an embodiment of the present disclosure.

[0077] like Figure 2 As shown, the combined image retrieval method includes operations S210 to S230.

[0078] In operation S210, the query image and the modified text are input into a combined image retrieval model to obtain the image-text combined features of the query image.

[0079] For example, a given query image can be and modify the text Input into the trained combined image retrieval model, after feature encoding, decoupling and combination, output the image and text combination features .

[0080] In operation S220, the similarity between the image-text combination feature and features of a plurality of candidate images in the image database is determined.

[0081] For example, you can put an image collection in an image database Input into the trained combined image retrieval model, after feature encoding, output image feature set ,in is the size of the candidate image set; calculate the image-text combination features Cosine similarity with all candidate image features ,in .

[0082] In operation S230, an image retrieval result corresponding to the query image is determined according to the similarity.

[0083] For example, the candidate image with the greatest similarity may be selected and output as the retrieval result.

[0084] Figure 3 It is a block diagram of an apparatus for training a combined image retrieval model according to an embodiment of the present disclosure.

[0085] like Figure 3 As shown, the apparatus 300 for training a combined image retrieval model may include an acquisition module 310, a feature extraction module 320, a decoupling module 330, a feature fusion module 340, and a training module 350. The combined image retrieval model includes an image encoding sub-model, a text encoding sub-model, an image decoupling sub-model, and a text decoupling sub-model.

[0086] The acquisition module 310 is used to acquire sample data, where the sample data includes <reference image, modified text, target image> triplet data.

[0087] A feature extraction module 320, for respectively extracting features of a reference image and a target image using the above-mentioned image encoding sub-model, and extracting features of a modified text using the above-mentioned text encoding sub-model;

[0088] A decoupling module 330 is used to decouple the features of the modified text using the text decoupling submodel to obtain common text features and unique text features, process the features of the reference image and the common text features using the image decoupling submodel to obtain reference image common features, and process the features of the target image and the common text features using the image decoupling submodel to obtain target image common features;

[0089] A feature fusion module 340 is used to fuse the common features of the reference image and the unique text features to obtain a combined image-text feature;

[0090] The training module 350 is used to construct an image retrieval loss function and train the above-mentioned combined image retrieval model.

[0091] Figure 4 is a block diagram of a combined image retrieval device according to an embodiment of the present disclosure.

[0092] like Figure 4 As shown, the combined image retrieval device 400 may include an obtaining module 410 , a first determining module 420 and a second determining module 430 .

[0093] An acquisition module 410 is used to input the query image and the modified text into a combined image retrieval model to obtain the image-text combined features of the query image;

[0094] A first determination module 420 is used to determine the similarity between the above-mentioned image-text combination feature and a plurality of candidate image features in the image database;

[0095] The second determination module 430 is used to determine the image retrieval result corresponding to the query image according to the similarity.

[0096] Regarding the specific limitations of the above-mentioned device for training a combined image retrieval model and the combined image retrieval device, please refer to the limitations of the method for training a combined image retrieval model and / or the combined image retrieval method above, which will not be repeated here. The various modules in the above-mentioned device for training a combined image retrieval model and the combined image retrieval device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0097] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0098] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0099] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0100] like Figure 5As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0101] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0102] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as a method for training a combined image retrieval model and / or a combined image retrieval method. For example, in some embodiments, the method for training a combined image retrieval model and / or a combined image retrieval method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method for training a combined image retrieval model and / or a combined image retrieval method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured in any other appropriate manner (eg, by means of firmware) to execute the method for training a combined image retrieval model and / or the combined image retrieval method.

[0103] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0105] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0108] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0109] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0110] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a combined image retrieval model, the combined image retrieval model comprising an image encoding sub-model, a text encoding sub-model, an image decoupling sub-model and a text decoupling sub-model, characterized in that: The method comprises: Acquire sample data, wherein the sample data includes <reference image, modified text, target image> triplet data; The image encoding sub-model is used to extract the features of the reference image and the target image respectively, and the text encoding sub-model is used to extract the features of the modified text; The features of the modified text are decoupled by using the text decoupling submodel to obtain common text features and unique text features; the features of the reference image and the common text features are processed by using the image decoupling submodel to obtain common features of the reference image; the features of the target image and the common text features are processed by using the image decoupling submodel to obtain common features of the target image; Merging the common features of the reference image and the unique text features to obtain image-text combined features; An image retrieval loss function is constructed to train the combined image retrieval model.

2. The method for training a combined image retrieval model according to claim 1, characterized in that: It also includes augmenting a sample data set using an image difference description model, wherein the image difference description model determines, based on each <reference image, target image> data pair, to convert the target image into a difference text of the reference image to obtain corresponding <target image, difference text, reference image> triplet data as augmented data.

3. The method for training a combined image retrieval model according to claim 1, characterized in that: The image encoding sub-model is the image encoder in the pre-trained CLIP model, and the text encoding sub-model is the text encoder in the pre-trained CLIP model.

4. The method for training a combined image retrieval model according to claim 1, characterized in that: The step of using the image decoupling sub-model to process the features of the reference image and the common text features to obtain the common features of the reference image includes: The features of the reference image and the common text features are used as query features and key features, the attention weights are calculated using the cross-attention mechanism, and the common features of the reference image are calculated based on the attention weights and the features of the reference image.

5. The method for training a combined image retrieval model according to claim 1, characterized in that: The constructing an image retrieval loss function and training the combined image retrieval model comprises: Constructing a first loss function based on the image-text combination feature and the feature of the target image; Constructing a second loss function based on the common features of the reference image and the common features of the target image; A total loss function is constructed based on the first loss function and the second loss function, and the image encoding sub-model, the text encoding sub-model, the image decoupling sub-model and the text decoupling sub-model are trained based on the total loss function.

6. A combined image retrieval method, comprising: Inputting the query image and the modified text into the combined image retrieval model to obtain the image-text combined features of the query image; Determining the similarity between the image-text combination feature and features of a plurality of candidate images in an image database; An image retrieval result corresponding to the query image is determined according to the similarity.

7. A device for training a combined image retrieval model, the combined image retrieval model comprising an image encoding sub-model, a text encoding sub-model, an image decoupling sub-model and a text decoupling sub-model, the device comprising: An acquisition module, used for acquiring sample data, wherein the sample data includes <reference image, modified text, target image> triplet data; A feature extraction module, used to respectively extract features of a reference image and a target image using the image encoding sub-model, and to extract features of a modified text using the text encoding sub-model; a decoupling module, configured to decouple the features of the modified text using the text decoupling submodel to obtain common text features and unique text features, process the features of the reference image and the common text features using the image decoupling submodel to obtain reference image common features, and process the features of the target image and the common text features using the image decoupling submodel to obtain target image common features; A feature fusion module, used to fuse the common features of the reference image and the unique text features to obtain image-text combined features; The training module is used to construct an image retrieval loss function and train the combined image retrieval model.

8. A combined image retrieval device, comprising: An acquisition module, used for inputting the query image and the modified text into the combined image retrieval model to obtain the image-text combined features of the query image; A first determination module is used to determine the similarity between the image-text combination feature and a plurality of candidate image features in an image database; The second determination module is used to determine the image retrieval result corresponding to the query image according to the similarity.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are executed.

Images