Image retrieval method, device and equipment based on multi-modal semantics

Abstract

The invention discloses an image retrieval method, device and equipment based on multi-modal semantics, which are applied to the technical field of multi-modal retrieval. According to the image retrieval method based on the multi-modal semantics, a retrieval request comprising a reference image and a modified text is obtained; based on the reference image and the modified text, a first global image feature including visual information of the reference image, an object feature including information of the modified object, and a description feature including semantic information of the modified text are extracted, respectively. Therefore, relatively complete visual information and semantic information can be extracted. And integrating the first global image feature, the object feature and the description feature to obtain a retrieval feature. And finally, determining a target image by utilizing the retrieval features, and generating a retrieval result. According to the method, retrieval is carried out by utilizing the retrieval characteristics fusing the visual information and the semantic information, so that the retrieval capability of cross-modal information can be enhanced, the accuracy and the effectiveness of the target image obtained through retrieval are improved, and the retrieval requirements of a user in a multi-modal image retrieval scene are met.

Description

Technical Field

[0001] The present application relates to the technical field of multi-modal retrieval, and particularly relates to an image retrieval method, device and equipment based on multi-modal semantics. Background Art

[0002] Multi-Modal Retrieval is an information retrieval method involving multiple media modalities. Among them, multiple media modalities include media modalities such as text, image, audio, video, etc. Compared with traditional single-modal information retrieval, multi-modal retrieval can process and integrate information from different modalities to provide more comprehensive, accurate and rich retrieval results.

[0003] In the field of image retrieval, Composed Image Retrieval (CIR) is an emerging multi-modal image retrieval method. CIR refers to retrieving a target image obtained by modifying a reference image according to a modification text provided by a user. For example, the user provides a reference image of a white commodity and a modification text "change the color of the commodity to red". Through CIR, a target image of a red commodity is retrieved. Among them, the red commodity is the same as or similar to the white commodity in the reference image.

[0004] Currently, the target images obtained by CIR have the problem of not meeting the user's retrieval purpose, and the retrieval accuracy of CIR is insufficient. Summary of the Invention

[0005] In view of this, the present application provides an image retrieval method, device and equipment based on multi-modal semantics, which can improve the accuracy and effectiveness of the retrieved target images and meet the retrieval requirements of users in multi-modal image retrieval scenarios.

[0006] The technical solutions provided by the present application are as follows:

[0007] In a first aspect, the present application provides an image retrieval method based on multi-modal semantics, and the method includes:

[0008] Obtain a retrieval request, where the retrieval request includes a reference image and a modification text. The reference image is used to assist in retrieving an image, and the modification text is used to describe the modification object and the modification process of the image retrieved from the reference image;

[0009] Extract a first global image feature from the reference image, extract an object feature from the reference image and the modification text, and extract a description feature from the modification text. The first global image feature includes the visual information of the reference image, the object feature includes the information of the modification object, and the description feature includes the semantic information of the modification text;

[0010] Integrate the first global image feature, the object feature, and the description feature to obtain a retrieval feature;

[0011] Use the retrieval feature to determine a retrieval result, where the retrieval result includes a target image that matches the retrieval feature.

[0012] Optionally, the extracting the object feature from the reference image and the modified text includes:

[0013] Map the first global image feature to a token vector, where the token vector is used to represent the visual information of the reference image;

[0014] Write the token vector and the modified text into a splicing template to obtain a spliced text, where the splicing template includes fixed text and text to be written, and the spliced text is used to describe the modified object;

[0015] Extract the object feature from the spliced text.

[0016] Optionally, the method further includes:

[0017] Generate a first prompt word according to the first image description text and the modified text, where the first image description text is used to describe the reference image;

[0018] Call a first model to process the first prompt word to obtain an object text, where the object text is used to identify the modified object, and the first model is used to generate the text of the object involved in the first prompt word based on the input first prompt word;

[0019] The writing the token vector and the modified text into a splicing template to obtain a spliced text includes:

[0020] Write the token vector, the modified text, and the object text into a splicing template to obtain a spliced text.

[0021] Optionally, the extracting the description feature from the modified text includes:

[0022] Call a second model to process the reference image to generate a plurality of first image description texts, where the first image description texts are used to describe the content of the reference image, and the second model is used to generate the text describing the image based on the input image;

[0023] For each of the first image description texts, generate a second prompt word according to the first image description text and the modified text;

[0024] Call the third model to process each of the second prompt words to obtain an updated modified text, where the third model is used to modify the input text;

[0025] Extract the initial features of each of the updated modified texts, where the initial features include the semantic information of the reference image and the semantic information of the modified text;

[0026] Perform an averaging process on the obtained multiple initial features to obtain a description feature.

[0027] Optionally, the integrating the first global image feature, the object feature, and the description feature to obtain a retrieval feature includes:

[0028] Use the weighted values of the first global image feature, the object feature, and the description feature as the retrieval feature, where the weights of the first global image feature, the object feature, and the description feature are all preset values.

[0029] Optionally, the using the retrieval feature to determine a retrieval result includes:

[0030] Obtain the candidate feature of a candidate image, where the candidate image belongs to the database targeted by the retrieval request;

[0031] Determine a target candidate feature that satisfies the retrieval condition with the retrieval feature;

[0032] Use the candidate image corresponding to the target candidate feature as the target image to generate a retrieval result.

[0033] Optionally, the determining the candidate feature of a candidate image includes:

[0034] Extract a second global image feature from the candidate image;

[0035] Call a fourth model to process the candidate image to generate multiple second image description texts, where the second image description texts are used to describe the content of the candidate image, and the fourth model is used to output a text describing the image based on the input image;

[0036] Extract the text features of each of the second image description texts, where the text features of the second image description texts include the semantic information of the candidate image;

[0037] Perform an averaging process on the obtained multiple text features of the second image description texts to obtain the text feature of the candidate image;

[0038] Generate a candidate feature according to the second global image feature and the text feature of the candidate image.

[0039] Second aspect, the present application provides an image retrieval device based on multimodal semantics. The device includes:

[0040] An acquisition unit, configured to acquire a retrieval request, where the retrieval request includes a reference image and a modified text. The reference image is used to assist in retrieving an image, and the modified text is used to describe the modification object and the modification process for obtaining the retrieved image from the reference image;

[0041] An extraction unit, configured to extract a first global image feature from the reference image, extract an object feature from the reference image and the modified text, and extract a description feature from the modified text. The first global image feature includes visual information of the reference image, the object feature includes information of the modification object, and the description feature includes semantic information of the modified text;

[0042] An integration unit, configured to integrate the first global image feature, the object feature, and the description feature to obtain a retrieval feature;

[0043] A retrieval unit, configured to use the retrieval feature to determine a retrieval result, where the retrieval result includes a target image that matches the retrieval feature.

[0044] Optionally, the extraction unit, when extracting an object feature from the reference image and the modified text, includes:

[0045] The extraction unit is configured to map the first global image feature to a token vector, where the token vector is used to represent the visual information of the reference image; write the token vector and the modified text into a splicing template to obtain a spliced text, where the splicing template includes fixed text and text to be written, and the spliced text is used to describe the modification object; extract an object feature from the spliced text.

[0046] Optionally, the extraction unit is further configured to:

[0047] Generate a first prompt word according to a first image description text and the modified text, where the first image description text is used to describe the reference image;

[0048] Invoke a first model to process the first prompt word to obtain an object text, where the object text is used to identify the modification object, and the first model is configured to generate text of an object involved in the first prompt word based on the input first prompt word;

[0049] The extraction unit, when writing the token vector and the modified text into a splicing template to obtain a spliced text, includes:

[0050] The extraction unit is configured to write the token vector, the modified text, and the object text into a splicing template to obtain a spliced text.

[0051] Optionally, the extraction unit is configured to extract descriptive features from the modified text, including:

[0052] The extraction unit is configured to call a second model to process the reference image to generate a plurality of first image description texts for describing the content of the reference image, where the second model is configured to output a text describing the image based on the input image; for each of the first image description texts, generate a second prompt word according to the first image description text and the modified text; call a third model to process each of the second prompt words to obtain an updated modified text, where the third model is configured to modify the input text; extract initial features of each of the updated modified texts, where the initial features include semantic information of the reference image and semantic information of the modified text; and perform an averaging process on the obtained plurality of initial features to obtain descriptive features.

[0053] Optionally, the integration unit is specifically configured to:

[0054] Use the weighted values of the first global image feature, the object feature, and the descriptive feature as retrieval features, where the weights of the first global image feature, the object feature, and the descriptive feature are all preset values.

[0055] Optionally, the retrieval unit is configured to use the retrieval features to determine a retrieval result, including:

[0056] The retrieval unit is configured to obtain candidate features of candidate images, where the candidate images belong to the database targeted by the retrieval request; determine target candidate features that meet the retrieval conditions with the retrieval features; and use the candidate images corresponding to the target candidate features as target images to generate a retrieval result.

[0057] Optionally, the retrieval unit is configured to determine candidate features of candidate images, including:

[0058] The retrieval unit is configured to extract second global image features from the candidate images; call a fourth model to process the candidate images to generate a plurality of second image description texts for describing the content of the candidate images, where the fourth model is configured to output texts describing images based on the input images; extract text features of each of the second image description texts, where the text features of the second image description texts include semantic information of the candidate images; average the obtained text features of the plurality of second image description texts to obtain text features of the candidate images; and generate candidate features according to the second global image features and the text features of the candidate images.

[0059] In a third aspect, the present application provides an image retrieval device based on multi-modal semantics, including: a processor, a memory, and a system bus;

[0060] The processor and the memory are connected through the system bus;

[0061] The memory is configured to store one or more programs, where the one or more programs include instructions that, when executed by the processor, cause the processor to execute the method according to any one of the embodiments in the first aspect above.

[0062] In a fourth aspect, the present application provides a computer-readable storage medium storing instructions that, when running on a terminal device, cause the terminal device to execute the method according to any one of the embodiments in the first aspect above.

[0063] Therefore, the present application has the following beneficial effects:

[0064] The present application provides an image retrieval method, apparatus, and device based on multimodal semantics. In the image retrieval method based on multimodal semantics, a retrieval request including a reference image and a modified text is obtained. The modified text is used to indicate the modification object and the modification process for obtaining the image to be retrieved from the reference image. Based on the reference image and the modified text, a first global image feature including the visual information of the reference image, an object feature including the information of the modification object, and a description feature including the semantic information of the modified text are respectively extracted. In this way, relatively complete visual information and semantic information can be extracted. The first global image feature, the object feature, and the description feature are integrated to obtain a retrieval feature. By integrating multiple features, the mutual complementation of visual information and semantic information is realized, which is beneficial to fully integrating multimodal information and obtaining a retrieval feature including relatively comprehensive information. Finally, the retrieval feature is used to determine the target image and generate a retrieval result. Using the retrieval feature that integrates visual information and semantic information for retrieval can enhance the retrieval ability of cross-modal information such as image retrieval realized by images and texts, thereby improving the retrieval performance, enhancing the accuracy and effectiveness of the retrieved target image, and meeting the retrieval requirements of users in multimodal image retrieval scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 FIG. is a schematic structural diagram of a CIR system provided by an embodiment of the present application;

[0066] Figure 2 FIG. is a schematic flowchart of an image retrieval method based on multimodal semantics provided by an embodiment of the present application;

[0067] Figure 3 FIG. is a schematic diagram of an image retrieval based on multimodal semantics provided by an embodiment of the present application;

[0068] Figure 4 FIG. is a schematic structural diagram of an image retrieval apparatus based on multimodal semantics provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] To facilitate the understanding and interpretation of the technical solutions provided by the embodiments of the present application, the background technology of the present application will be described first.

[0070] In the field of image retrieval, CIR realizes multimodal retrieval based on two media modalities of images and texts based on a reference image and a modified text. CIR retrieves at least one target image from a database for the reference image and the modified text provided by the user. The target image needs to visually and semantically conform to the result of modifying the reference image according to the modification method indicated by the modified text. The CIR method needs to understand both the visual information of the reference image and the semantic information of the modified text to obtain a more accurate result that meets the user's retrieval purpose.

[0071] Currently, the CIR model is mainly used to implement composite image retrieval. In order to reduce the dependence on a large number of manually annotated triples (reference image, modified text, target image) in supervised learning, the Zero-Shot Composed Image Retrieval (ZS-CIR) method is mostly adopted. The ZS-CIR method aims to combine the reference image and the modified text without supervised learning to retrieve the target image that meets the user's needs, reducing the data annotation cost on the one hand and improving the generalization ability of the CIR model on the other hand.

[0072] However, in the process of processing the reference image and the modified text by the ZS-CIR method, the ways of extracting visual information and semantic information are different, and it is difficult to effectively extract visual information and semantic information. This results in insufficient accuracy of the target image retrieved by the current ZS-CIR method, and there is a problem that the retrieved target image is difficult to meet the user's retrieval needs.

[0073] Based on this, the embodiments of the present application provide an image retrieval method based on multimodal semantics. This image retrieval method based on multimodal semantics can be applied to multimodal image retrieval scenarios. Specifically, from a retrieval request including a reference image and a modified text, a first global image feature including the visual information of the reference image, an object feature including the information of the modified object, and a description feature including the semantic information of the modified text are respectively extracted. In this way, relatively complete visual information and semantic information can be extracted from the retrieval request. Then, the first global image feature, the object feature, and the description feature are integrated to obtain a retrieval feature. By integrating multiple features, multimodal information can be fully fused, realizing the mutual complement of visual information and semantic information, and obtaining a retrieval feature including relatively comprehensive information. Finally, the target image is determined using the retrieval feature to generate a retrieval result. The image retrieval method based on multimodal semantics provided by the embodiments of the present application can enhance the fusion ability of multimodal information and the retrieval ability for cross-modal information, effectively improve the retrieval performance of zero-shot composite image retrieval, and meet the retrieval needs of users in multimodal image retrieval scenarios.

[0074] To facilitate the understanding of the technical solutions provided by the embodiments of the present application, the following describes the image retrieval method based on multimodal semantics provided by the embodiments of the present application with reference to the accompanying drawings.

[0075] First of all, it should be noted that in a possible implementation scenario, the image retrieval method based on multimodal semantics provided by the embodiments of the present application can be applied to a CIR system. See Figure 1As shown in the figure, the CIR system 100 includes a Visual Information Extraction component 101, a Semantic Information Extraction component 102, and a Complementary Information Retrieval component 103. The CIR system 100 obtains a retrieval request including a reference image and modified text.

[0076] Among them, the Visual Information Extraction component 101 is mainly used to extract a first global image feature based on the reference image included in the retrieval request, and extract object features based on the reference image and the modified text. The Semantic Information Extraction component 102 is used to extract descriptive features from the modified text. The Complementary Information Retrieval component 103 is used to determine retrieval features according to the first global image feature, the object feature, and the descriptive feature, and use the retrieval features to determine a target image, and output a retrieval result including the target image.

[0077] For the specific implementation manners of the Visual Information Extraction component 101, the Semantic Information Extraction component 102, and the Complementary Information Retrieval component 103, please refer to the detailed introduction below.

[0078] See Figure 2 As shown in the figure, this figure is a schematic flowchart of an image retrieval method based on multimodal semantics provided by an embodiment of the present application. As Figure 2 As shown in the figure, an image retrieval method based on multimodal semantics provided by an embodiment of the present application includes S201 - S204.

[0079] S201: Obtain a retrieval request including a reference image and modified text.

[0080] The retrieval request is used to trigger the retrieval of the target image. The retrieval request includes a reference image and modified text. The reference image is used to assist in retrieving the target image. The modified text is used to describe the modification object and the modification process for obtaining the target image from the reference image. The retrieved target image visually and semantically meets the requirements after the reference image is modified according to the modified text.

[0081] For example, the retrieval request includes a reference image I r and modified text T. Based on the retrieval request, from the database retrieve the target image I t . Among them, I k represents the candidate images included in the database, and K is the number of candidate images included in the database.

[0082] See Figure 3As shown in the figure, this is a schematic flowchart of an image retrieval method based on multi-modal semantics provided by an embodiment of the present application. The reference image is an image including a red double-decker bus. Modify the text to: "is green and is viewed from the front (is green and the perspective is from the front)". The modified text is used to indicate that the object to be modified is the red double-decker bus in the reference image. The modification process is to modify the red double-decker bus to a green double-decker bus and provide an image perspective from the front of the green double-decker bus. The target image retrieved according to the reference image and the modified text should be a front view of a green double-decker bus.

[0083] It should be noted that after the CIR system is established, the CIR system needs to be tested. In the test scenario for the CIR system, the retrieval request can be generated by a technician based on the ZS-CIR dataset. As an example, the ZS-CIR dataset is, for example, the CIRCO dataset (an open-source dataset), the CIRR dataset (an open-source dataset), and the FashionIQ dataset (an open-source dataset). Among them, the test set of the CIRCO dataset contains 800 examples, and each example consists of a reference image, a modified text, and multiple target images. The test set of the CIRR dataset contains 4,100 triples (a reference image, a modified text, and a target image). The validation set of the FashionIQ dataset is divided into three subclasses: Dress (dress), Shirt (shirt), and Toptee (T-shirt), with a total of 6,000 triples (a reference image, a modified text, and a target image). Taking the test sets from the CIRCO dataset and the CIRR dataset, based on the examples included in the test set of the CIRCO dataset and the triples included in the test set of the CIRR dataset, retrieval requests including reference data and modified text are generated to implement the test of the CIR system. Each retrieval request of the CIRCO dataset corresponds to multiple target images. Each retrieval request of the CIRR dataset corresponds to one target image.

[0084] In addition, the validation set from the FashionIQ dataset is used to test the CIR system. Each retrieval request of the FashionIQ dataset corresponds to one target image.

[0085] After the CIR system passes the test, an image retrieval service can be provided to the user. In the application scenario of the CIR system, the retrieval request can be generated by the user based on the retrieval requirement. For example, the user inputs a reference image and a modified text in the interaction interface to trigger the generation of a retrieval request.

[0086] S202: Extract a first global image feature from the reference image, extract an object feature from the reference image and the modified text, and extract a description feature from the modified text.

[0087] Based on the reference image and the modified text included in the retrieval request, features are extracted from the reference image and the modified text respectively, so as to perform image retrieval based on the extracted features.

[0088] Based on the reference image and the modified text, it is necessary to extract the first global image feature, object feature, and description feature. The extraction processes of these three features are introduced below respectively.

[0089] First: Extract the first global image feature from the reference image.

[0090] The first global image feature includes the visual information of the reference image.

[0091] The embodiments of the present application do not limit the implementation manner of extracting the first global image feature from the reference image. As an example, using a Visual Encoder to extract the first global image feature of the reference image can be expressed as:

[0092] q v =f v (I r ) ∈R d×1 (1)

[0093] where f v is the visual encoder. The visual encoder is frozen, that is to say, the visual encoder is an open-source and pre-trained model, and no adjustment of model parameters is required.

[0094] I r is the reference image. q v is the first global image feature. d represents the feature dimension. d×1 represents a d×1-dimensional vector. R represents the real number field.

[0095] The visual information extraction component 101 is used to extract visual information. As an example, the visual information extraction component 101 is a CLIP (Contrastive Language–Image Pre-training) model (an open-source visual language pre-training model for feature extraction).

[0096] See Figure 3 As shown, the reference image included in the retrieval request is input into the visual information extraction component 101. The visual information extraction component 101 extracts the first global image feature through the visual encoder, which can also be called the reference image features.

[0097] Second: Extract the object feature from the reference image and the modified text.

[0098] The reference image includes information about the modification object that needs to be modified to obtain the target image. The modification text also describes the modification object. Object features are extracted from the reference image and the modification text. The object features include information about the modification object. The object features include visual information.

[0099] In a possible implementation, the reference image is first mapped to a token vector. The token vector is used to represent the visual information of the reference image.

[0100] As an example, a mapping network is used to map the first global image feature to a token vector. The mapping process is expressed as:

[0101] $ = Φ(q v ) (2)

[0102] where $ is a token, representing the token vector. The token vector is a d×1-dimensional vector, and d represents the text feature dimension. Φ represents the mapping network. The mapping network is frozen, that is, the mapping network is an open-source and pre-trained model, and no model parameter adjustment is required.

[0103] See Figure 3 As shown, the first global image feature is processed by the Mapping Network to obtain the token vector.

[0104] The obtained token vector and the modification text are concatenated to obtain the concatenated text. The concatenated text is used to describe the modification object.

[0105] The embodiments of the present application do not limit the implementation manner of generating the concatenated text.

[0106] In a possible implementation, a language model can be called to process the token vector and the modification text to generate the concatenated text. Among them, the language model is used based on the concatenated text.

[0107] In another possible implementation, a concatenation template is obtained. The concatenation template includes fixed text and text to be written. The text to be written represents the text that needs to be written. As an example, the concatenation template is "a photo of A that B". Among them, A and B are the text to be written. A corresponds to the token vector. B corresponds to the modification text. Using the token vector as the token and the modification text as T, the token vector and the modification text are used to replace the corresponding text to be written, so as to implement writing the token vector and the modification text into the concatenation template, and the obtained concatenated text is: "a photo of token that T".

[0108] Then, features are extracted from the concatenated text to obtain object features.

[0109] As an example, a text encoder is used to process the concatenated text to obtain object features. It can be expressed as:

[0110] q f = f t (F G ) (3)

[0111] where f t represents the text encoder. F G represents the spliced text. q f represents the object feature.

[0112] In addition, considering that when performing image retrieval, more attention needs to be paid to the modified object rather than all the objects included in the reference image. For example, Figure 3 in the corresponding example, more attention needs to be paid to the double-decker bus in the reference image rather than the trees, roads, and single-decker buses that serve as the background. An object text for identifying the modified object can be generated based on the reference image and the modified text. And the object text is spliced together with the token vector and the modified text to obtain the spliced text. Adding the object text to the spliced text enables the extracted object feature to include more information about the modified object, further focusing the retrieval on the modified object, achieving object-level fine-grained retrieval, narrowing the retrieval granularity, and thus improving the retrieval accuracy.

[0113] The following introduces possible implementation methods for generating the object text, including three steps A1 - A3.

[0114] A1: Before generating the spliced text, obtain the first image description text.

[0115] The first image description text is used to describe the reference image.

[0116] As an example, the first image description text is the text generated by processing the reference image using a second model. The second model can output text describing the image based on the input image. The second model is, for example, the BLIP-2 model.

[0117] The nucleus sampling technique can be used in the generation process of the first image description text. The nucleus sampling technique is a technique used for text generation in natural language processing and generative models. It can avoid generating overly chaotic or incoherent text while maintaining the diversity of the generated text, making the generated text both diverse and fluent. By using the nucleus sampling technique to generate the first image description text, rich semantic information about the reference image can be obtained.

[0118] The process of generating the first image description text can be expressed as:

[0119]

[0120] Among them, C B represents a set including multiple first image description texts. is the i-th first image description text. N is the number of texts of the first image description texts. The preferred value of N is 15. I r is a reference image. BLIP-2() represents the BLIP-2 model.

[0121] It should be noted that the number of texts of the generated first image description texts is preferably multiple. Multiple first image description texts can fully describe the content of the reference image from different description perspectives.

[0122] As an example, see Figure 3 shown. The second model is the BLIP-2 model included in the semantic information extraction component 102. The BLIP-2 model processes the reference image to obtain multiple first image description texts. Among them, one first image description text is: "a double-decker bus is parked next to a bus".

[0123] A2: Generate a first prompt word according to the first image description text and the modified text.

[0124] Based on the first image description text and the modified text, the modification object of the target image to be retrieved relative to the reference image can be determined.

[0125] It should be noted that for the convenience of processing, during the process of generating the modified text, any one of the multiple first image description texts generated above can be arbitrarily selected. A first prompt word is generated using the selected first image description text and the modified text.

[0126] In a possible implementation, a language model is used to process the image description text and the modified text. First, a first prompt word for input to the model is generated based on the image description text and the modified text. As an example, a first prompt word template is obtained first. The first prompt word template includes fixed content and content to be filled in. For example, the first prompt word template is:

[0127] I have a picture. According to the instruction to modify the image, carefully predict the objects that may be added to the modified image. I will provide the image content starting with "Image content:". The instruction I provide will start with "Instruction:". For the most likely added object, please provide a noun phrase to describe the new object introduced by the instruction and add any necessary details according to the modification;

[0128] Your answer should be organized in the following structure:

[0129] Most likely objects to be added: [Noun phrase with relevant details in the instruction]

[0130] Please keep your answer short and focus on the modification;

[0131] Example 1:

[0132] Image content: A man holding a big fish on a stick.

[0133] Instruction: Add a group of people watching behind the fence.

[0134] Most likely objects to be added: A group of people, a fence.

[0135] Example 2:

[0136] Image content: A dog wearing a red hat.

[0137] Instruction: There is a dog of a different breed, and a pirate flag is shown on the hat.

[0138] Most likely objects to be added: A dog of a different breed, a pirate flag on the hat.

[0139] Among them, in the first prompt template, Example 1 and Example 2 are used to facilitate the first model to understand the processing requirements.

[0140] Add the first image description text and the modification text to the first prompt template. For example, add the first image description text after the above "Image content". Add the modification text after the "Instruction" to obtain the first prompt.

[0141] A3: Invoke the first model to process the first prompt to obtain the object text.

[0142] The first model can generate the object text based on the input first prompt. For the convenience of brief processing, the number of object texts is one. The object text is used to identify the modification object. As an example, the object text is the name of the modification object. Additionally, the object text can also include the text used to describe the modification object.

[0143] The first model can be a language model, and the model type of the first model is not limited in this application. As an example, the first model is a large language model (LLM).

[0144] As an example, the process of generating the object text using the LLM can be expressed as:

[0145]

[0146] Among them, T is the modification text, is a selected first image description text. p aRepresents the first prompt template. Represents the first prompt. LLM() represents the first model. F add Is the object text.

[0147] See Figure 3 As shown, the first model is the LLM included in the semantic information extraction component 102. In the semantic information extraction component 102, the LLM is called to process multiple first image description texts to obtain the object text, that is, the Objects most likely added. The number of object texts is one. Taking Figure 3 the first image description text in as an example, the obtained object text is: "a green bus viewed from the front".

[0148] After obtaining the object text, the token vector, modified text, and object text can be concatenated to obtain the concatenated text. Specifically, the token vector, modified text, and object text are written into the concatenation template to obtain the concatenated text.

[0149] As an example, the concatenation template is "a photo of A that B. And the photo should have C". Among them, A, B, and C are texts to be written. A corresponds to the token vector. B corresponds to the modified text. C corresponds to the object text. Taking the token vector as token, the modified text as T, and the object text as Objects most likely added. Replacing the corresponding texts to be written with the token vector, modified text, and object text, the token vector, modified text, and object text are written into the concatenation template, and the obtained concatenated text is: "a photo of token that T. And the photo should have [objects most likely added]".

[0150] Taking Figure 3 the reference image and modified text shown as an example, the visual information extraction component 101 generates the concatenated text based on the object text generated by the semantic information extraction component 102, combined with the token vector and modified text: "a photo of $ that is green and is viewed from the front. And the photo should have a green bus viewed from the front".

[0151] The spliced text including the object text includes fine-grained information at the object level. The fine-grained information enables the retrieval to focus on the object level rather than the entire image, effectively excluding some interference information included in the reference image. The information contained in the object features extracted from the spliced text is more complete and rich, which is beneficial to paying more attention to and locating the modified object when performing image retrieval based on the object features, realizing retrieval for the specific modified object, and improving the accuracy of the retrieval.

[0152] See Figure 3 As shown, for the object features extracted from the spliced text including the object text, they can also be called fine-grained features.

[0153] Third: Extract descriptive features from the modified text

[0154] The descriptive features include the semantic information of the modified text.

[0155] In a possible implementation, the descriptive features are directly extracted from the modified text. For example, the modified text is processed by a text encoder to obtain the descriptive features.

[0156] In another possible implementation, multiple first image description texts can also be generated based on the reference image, and the multiple first image description texts and the modified text are processed to obtain the descriptive features. The first image description text includes the semantic information of the reference image. The descriptive features obtained based on the first image description text and the modified text realize the complete extraction and efficient combination of the semantic information included in the reference image and the semantic information included in the modified text, obtain effective semantic information for the target image, and thus help to improve the accuracy of the retrieval for the target image.

[0157] The process of generating descriptive features using multiple first image description texts and the modified text is introduced below, including five steps B1 - B5.

[0158] B1: Call the second model to process the reference image and generate multiple first image description texts.

[0159] The process of generating multiple first image description texts using the second model has been described in the above step A1: Before generating the spliced text, obtain the first image description text. For details, please refer to the above description and will not be elaborated here.

[0160] Generating multiple first image description texts using the second model can deeply mine the semantic information included in the reference image, obtain more complete and sufficient semantic information, and effectively extract the information contained in the reference image.

[0161] B2: For each first image description text, generate a second prompt according to the first image description text and the modified text.

[0162] Generate a second prompt for the modified text and each first image description text respectively. The number of second prompts is the same as the number of first image description texts.

[0163] The embodiments of this application do not limit the manner of generating the second prompt according to the first image description text and the modified text. As an example, a second prompt template can be obtained, and the first image description text and the modified text are combined based on the second prompt template to obtain the second prompt. The second prompt template can be flexibly set according to needs.

[0164] As an example, the prompt template for the second prompt is:

[0165] I have an image. I will give an instruction to edit the image and then generate a description of the edited image. I will describe the content of my image starting with "Image content:". The instruction I provide will start with "Instruction:". The generated description of the edited image should start with "Edited description:". Only one description of the edited image is generated each time.

[0166] Add the first image description text and the modified text to the second prompt template. For example, add the first image description text after the above "Image content". Add the modified text after "Instruction" to obtain the second prompt.

[0167] B3: Call the third model to process each second prompt respectively to obtain the updated modified text.

[0168] The third model is used to modify and polish the input text. The third model can be an LLM. Use the third model to process each second prompt respectively to obtain the updated modified text. The updated modified text includes more complete and sufficient semantic information.

[0169] As an example, the process of obtaining the updated modified text can be expressed as:

[0170]

[0171] where T is the modified text. is the i-th first image description text. p m is the second prompt template. represents generating the second prompt according to the i-th first image description text and the modified text. LLM() represents the LLM model. is the i-th updated modified text. The value range of i is a positive integer from 1 to N. N is the number of texts of the generated first image description texts.

[0172] As an example, refer to Figure 3 As shown, N updated modified texts (Nx Modified captions) generated by the LLM.

[0173] B4: Extract the initial features of each updated modified text.

[0174] As an example, a text encoder can be used to process the updated modified text to obtain the initial features. The initial features include the semantic information of the reference image and the semantic information of the modified text.

[0175] B5: Average the obtained multiple initial features to obtain the descriptive features.

[0176] As an example, the process of generating the descriptive features can be expressed as:

[0177]

[0178] Where is the i-th updated modified text. f t () represents the text encoder. q m is the descriptive feature.

[0179] Refer to Figure 3 As shown, for N updated modified texts (Nx Modified captions), through the text encoder, the descriptive features are obtained, that is, the average updated modified text features (Average modified captions features).

[0180] The above is an explanation of the extraction process of the first global image feature, object feature, and descriptive feature. Extracting the first global image feature and object feature can achieve a complete expression of visual information. Using a large model to describe the reference image and combining it with the modified text can further generate descriptive features that can completely express semantic information.

[0181] Next, the process of image retrieval using the three features will be continued.

[0182] S203: Integrate the first global image feature, object feature, and descriptive feature to obtain the retrieval feature.

[0183] In a possible implementation, calculate the weighted values of the first global image feature, object feature, and descriptive feature to obtain the retrieval feature. The weights of the first global image feature, object feature, and descriptive feature are all preset values.

[0184] As an example, refer to Figure 3 As shown, the retrieval feature can be determined by the complementary information retrieval component 103. The retrieval feature is:

[0185] q = α·q m + β·q f +(1 - α - β)·q v (8)

[0186] Wherein, q m is the description feature, q f is the object feature, q v is the first global image feature. α is the weight of the description feature. β is the weight of the object feature. (1 - α - β) is the weight of the first global image feature. q is the retrieval feature.

[0187] Taking the weighted values of the first global image feature, object feature, and description feature as the retrieval feature can flexibly adjust the weights of various features based on multiple complex scenarios, so as to meet the scene requirements. For example, when the modified text is relatively simple, the weights of the first global image feature and object feature can be increased, thereby increasing the importance of visual information. When the modified text is relatively important, the weight of the description feature can be increased, thereby increasing the importance of semantic information. In this way, the proportion of different types of features can be adjusted based on different retrieval scenarios, so as to make full use of the relatively effective and reliable information extracted, achieve the efficient fusion of multi-modal information, and thus cope with complex and diverse retrieval scenarios and improve the accuracy of retrieval.

[0188] The above method of taking the weighted value as the retrieval feature is only one possible implementation. In addition, other methods can also be used to determine the retrieval feature. For example, an artificial intelligence model can be called to process the first global image feature, object feature, and description feature to obtain the retrieval feature. The artificial intelligence model is used to integrate features.

[0189] S204: Use the retrieval feature to determine the retrieval result including the target image.

[0190] The database includes multiple candidate images. Use the retrieval feature to determine the candidate image that matches the retrieval feature in the database. The candidate image that matches the retrieval feature is used as the target image.

[0191] Specifically, each candidate image included in the database can be matched with the retrieval feature. The candidate feature with the highest matching degree is used as the target image. In addition, candidate images can also be selected from the database to be matched with the retrieval feature. If the matching result meets the matching condition, then the candidate image is used as the target image.

[0192] The embodiments of the present application provide a possible implementation for matching the retrieval feature with the candidate image, including the following three steps C1 - C3:

[0193] C1: Obtain the candidate feature of the candidate image.

[0194] The candidate features of the candidate image include the information of the candidate image.

[0195] The embodiments of the present application do not limit the manner of obtaining candidate features. As an example, the candidate features of the candidate image are determined in advance and stored in a database. During retrieval, the candidate features of the candidate image determined in advance are obtained. As another example, during retrieval, based on the matching candidate image, the candidate features of the candidate image are generated in real time.

[0196] In a possible implementation manner, the candidate features of the candidate image include the visual information and semantic information of the candidate image.

[0197] Among them, the candidate features of the candidate image can be determined based on the second global image feature of the candidate image and the average text feature of the candidate image. The second global image feature of the candidate image includes the visual information of the candidate image. The average text feature of the candidate image includes the semantic information of the candidate image.

[0198] The extraction processes of the second global image feature of the candidate image and the text feature of the candidate image are introduced below respectively.

[0199] First: The second global image feature of the candidate image.

[0200] The embodiments of the present application do not limit the implementation manner of extracting the second global image feature from the candidate image. As an example, a Visual Encoder is used to extract the second global image feature of the candidate image. It should be noted that a CIR system can be used to extract the candidate features of the candidate image. That is to say, the Visual Encoder for extracting the second global image feature and the Visual Encoder for extracting the first global image feature can be the same Visual Encoder.

[0201] The process of extracting the second global image feature can be expressed as:

[0202] t v = f v (I k ) ∈ R d×1 (9)

[0203] Among them, f v is the Visual Encoder. The Visual Encoder is frozen. That is to say, the Visual Encoder is an open-source and pre-trained model, and no adjustment of model parameters is required.

[0204] I k is the candidate image. t v is the second global image feature. d represents the feature dimension. d×1 represents a d×1-dimensional vector. R represents the real number field.

[0205] Second: Text features of candidate images.

[0206] Invoke the fourth model to process the candidate image and generate multiple second image description texts. The second image description texts are used to describe the content of the candidate image. The fourth model is used to output text describing the image based on the input image. The fourth model and the above-mentioned second model can be the same model. The fourth model is, for example, the BLIP-2 model.

[0207] During the generation process of the second image description texts, the nucleus sampling technique can also be used to obtain relatively rich semantic information for the candidate image.

[0208] The process of generating the second image description texts can be expressed as:

[0209]

[0210] where C T represents a set including multiple second image description texts. is the i-th second image description text. N is the number of texts of the second image description texts. The preferred value of N is 15. I k is the candidate image. BLIP-2() represents the BLIP-2 model.

[0211] It should be noted that the number of texts of the generated second image description texts is preferably multiple. Multiple second image description texts can fully reflect the content of the candidate image from different description perspectives.

[0212] As an example, see Figure 3 As shown, the fourth model is the BLIP-2 model included in the semantic information extraction component 102.

[0213] Furthermore, extract the text features of each second image description text. The text features of the second image description text include the semantic information of the candidate image.

[0214] As an example, a text encoder can be used to process the second image description text to obtain the text features of the second image description text.

[0215] Then, average the text features of the obtained multiple second image description texts to obtain the text features of the candidate image.

[0216] As an example, the process of generating the text features of the candidate image can be expressed as:

[0217]

[0218] where is the i-th second image description text. f t() represents the text encoder. t c is the text feature of the candidate image.

[0219] Finally, candidate features are generated based on the second global image feature and the text feature of the candidate image.

[0220] As an example, the weighted value of the second global image feature and the text feature of the candidate image is used as the candidate feature.

[0221] See Figure 3 As shown, the candidate features can be determined by the complementary information retrieval component 103. The calculation method of the candidate features is as follows:

[0222] t = γ · t c +(1 - γ) · t v (12)

[0223] where, t c [[ID=2**]]is the text feature of the candidate image, t v is the second global image feature. γ is the weight of the text feature of the candidate image. (1 - γ) is the weight of the second global image feature. t is the candidate feature.

[0224] The above use of the weighted value as the candidate feature is only one possible implementation. Additionally, other methods can be adopted to determine the retrieval features. For example, an artificial intelligence model is called to process the second global image feature and the text feature of the candidate image to obtain the candidate features. The artificial intelligence model is used to integrate features.

[0225] C2: Determine the target candidate features that meet the retrieval conditions with the retrieval features.

[0226] The retrieval condition is to select the candidate features of the target image, that is, the condition for the target candidate features. As an example, the retrieval condition is related to the similarity between the candidate features and the retrieval features.

[0227] The embodiments of this application do not limit the method for calculating the similarity between the candidate features and the retrieval features. As an example, the similarity is the cosine similarity.

[0228] As an example, the retrieval condition is that the similarity between the candidate features and the retrieval features is greater than or equal to a threshold. The similarity between the candidate features of each candidate image in the database and the retrieval features is calculated respectively. The candidate features with a similarity greater than or equal to the threshold are used as the target candidate features. The target candidate features determined by this method may be one or more.

[0229] As another example, the retrieval condition is that the similarity between the candidate features and the retrieval features is the highest. The similarity between the candidate features of each candidate image in the database and the retrieval features is calculated respectively. The candidate feature with the highest similarity is used as the target candidate feature. The target candidate feature determined by this method is one.

[0230] The method of determining the target image by using the candidate feature with the highest similarity as the target candidate feature can be expressed by the following formula:

[0231]

[0232] where I t is the target image. q is the retrieval feature. t is the candidate feature. I k is the candidate image.

[0233] C3: Use the candidate image corresponding to the target candidate feature as the target image to generate a retrieval result including the target image.

[0234] Based on the above, it can be known that the image retrieval method provided by this application fully extracts the visual information and image information in the retrieval request, integrates the complementary visual and semantic information, and combines fine-grained information, which can enhance the multi-modal information fusion ability of the CIR system and the retrieval ability for cross-modal information, effectively improve the retrieval ability of the CIR system, and the retrieved target image is relatively accurate, meeting the retrieval needs of users.

[0235] Based on the image retrieval method based on multi-modal semantics provided in the above method embodiment, the embodiment of this application also provides an image retrieval device based on multi-modal semantics. The following will describe the image retrieval device based on multi-modal semantics with reference to the accompanying drawings.

[0236] See Figure 4 As shown in the figure, which is a schematic structural diagram of an image retrieval device based on multi-modal semantics provided in the embodiment of this application. As Figure 4 shown, the image retrieval device based on multi-modal semantics includes:

[0237] An acquisition unit 401, configured to acquire a retrieval request, where the retrieval request includes a reference image and a modified text, the reference image is used to assist in retrieving an image, and the modified text is used to describe the modification object and the modification process of the retrieved image obtained from the reference image;

[0238] [[ID=:34]]An extraction unit 402, configured to extract a first global image feature from the reference image, extract an object feature from the reference image and the modified text, and extract a description feature from the modified text, where the first global image feature includes the visual information of the reference image, the object feature includes the information of the modification object, and the description feature includes the semantic information of the modified text;

[0239] An integration unit 403, configured to integrate the first global image feature, the object feature, and the description feature to obtain a retrieval feature;

[0240] A retrieval unit 404 for determining a retrieval result by using the retrieval feature, where the retrieval result includes target images matching the retrieval feature.

[0241] Optionally, the extraction unit 402 is configured to extract object features from the reference image and the modified text, including:

[0242] The extraction unit 402 is configured to map the first global image feature into a token vector, where the token vector is used to represent the visual information of the reference image; write the token vector and the modified text into a splicing template to obtain a spliced text, where the splicing template includes fixed text and text to be written, and the spliced text is used to describe the modified object; extract object features from the spliced text.

[0243] Optionally, the extraction unit 402 is further configured to:

[0244] Generate a first prompt word according to the first image description text and the modified text, where the first image description text is used to describe the reference image;

[0245] Call a first model to process the first prompt word to obtain an object text, where the object text is used to identify the modified object, and the first model is used to generate text of an object involved in the first prompt word based on the input first prompt word;

[0246] The extraction unit 402 is configured to write the token vector and the modified text into a splicing template to obtain a spliced text, including:

[0247] The extraction unit 402 is configured to write the token vector, the modified text, and the object text into a splicing template to obtain a spliced text.

[0248] Optionally, the extraction unit 402 is configured to extract description features from the modified text, including:

[0249] The extraction unit 402 is configured to call a second model to process the reference image to generate a plurality of first image description texts, where the first image description texts are used to describe the content of the reference image, and the second model is used to generate text describing an image based on the input image; for each first image description text, generate a second prompt word according to the first image description text and the modified text; call a third model to process each of the second prompt words to obtain an updated modified text, where the third model is used to modify the input text; extract initial features of each of the updated modified texts, where the initial features include semantic information of the reference image and semantic information of the modified text; and perform an averaging process on the obtained plurality of initial features to obtain description features.

[0250] Optionally, the integration unit 403 is specifically configured to:

[0251] Use the weighted values of the first global image feature, the object feature, and the description feature as the retrieval feature, where the weights of the first global image feature, the object feature, and the description feature are all preset values.

[0252] Optionally, the retrieval unit 404 is configured to determine a retrieval result by using the retrieval feature, including:

[0253] The retrieval unit 404 is configured to obtain a candidate feature of a candidate image, where the candidate image belongs to the database targeted by the retrieval request; determine a target candidate feature that meets the retrieval condition with the retrieval feature; use the candidate image corresponding to the target candidate feature as the target image, and generate a retrieval result.

[0254] Optionally, the retrieval unit 404 is configured to determine a candidate feature of a candidate image, including:

[0255] The retrieval unit 404 is configured to extract a second global image feature from the candidate image; call a fourth model to process the candidate image to generate a plurality of second image description texts for describing the content of the candidate image, where the fourth model is configured to output a text for describing an image based on an input image; extract text features of each of the second image description texts, where the text features of the second image description texts include semantic information of the candidate image; perform an averaging process on the obtained text features of the plurality of second image description texts to obtain the text feature of the candidate image; generate a candidate feature according to the second global image feature and the text feature of the candidate image.

[0256] Based on the image retrieval method based on multimodal semantics provided in the foregoing method embodiments, the present application provides an image retrieval device based on multimodal semantics, including: a processor, a memory, and a system bus;

[0257] The processor and the memory are connected through the system bus;

[0258] The memory is configured to store one or more programs, where the one or more programs include instructions that, when executed by the processor, cause the processor to execute the image retrieval method based on multimodal semantics described in any of the foregoing embodiments.

[0259] Based on the image retrieval method based on multimodal semantics provided in the above method embodiments, the present application provides a computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions run on a terminal device, the terminal device is enabled to execute the image retrieval method based on multimodal semantics described in any of the above embodiments.

[0260] It should be noted that the various embodiments in this specification are described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions in the method section.

[0261] It should be understood that in the present application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (one) of the following" or its similar expression refers to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0262] It should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0263] The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be implemented directly in hardware, in software modules executed by a processor, or in a combination thereof. The software modules may be located in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium well known in the art.

[0264] The foregoing description of the disclosed embodiments enables those skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image retrieval method based on multimodal semantics, characterized in that The method includes: Obtaining a retrieval request, where the retrieval request includes a reference image and modified text. The reference image is used to assist in retrieving an image, and the modified text is used to describe the modification object and the modification process for obtaining the retrieved image from the reference image; Extracting a first global image feature from the reference image, extracting an object feature from the reference image and the modified text, and extracting a description feature from the modified text. The first global image feature includes the visual information of the reference image, the object feature includes the information of the modification object, and the description feature includes the semantic information of the modified text; Integrating the first global image feature, the object feature, and the description feature to obtain a retrieval feature; Using the retrieval feature to determine a retrieval result, where the retrieval result includes a target image that matches the retrieval feature.

2. The method according to claim 1, wherein The extracting the object feature from the reference image and the modified text includes: Mapping the first global image feature to a token vector, where the token vector is used to represent the visual information of the reference image; Writing the token vector and the modified text into a splicing template to obtain a spliced text. The splicing template includes fixed text and text to be written, and the spliced text is used to describe the modification object; Extracting the object feature from the spliced text.

3. The method according to claim 2, wherein The method further includes: Generating a first prompt word according to a first image description text and the modified text, where the first image description text is used to describe the reference image; Invoking a first model to process the first prompt word to obtain an object text, where the object text is used to identify the modification object, and the first model is used to generate text of the object involved in the first prompt word based on the input first prompt word; The writing the token vector and the modified text into a splicing template to obtain a spliced text includes: Writing the token vector, the modified text, and the object text into the splicing template to obtain a spliced text.

4. The method according to claim 1, wherein The extracting the description feature from the modified text includes: Invoking a second model to process the reference image to generate a plurality of first image description texts, where the first image description texts are used to describe the content of the reference image, and the second model is used to output text describing the image based on the input image; For each of the first image description texts, generating a second prompt word according to the first image description text and the modified text; Invoking a third model to process each of the second prompt words to obtain an updated modified text, where the third model is used to modify the input text; Extracting initial features of each of the updated modified texts, where the initial features include the semantic information of the reference image and the semantic information of the modified text; Performing an averaging process on the obtained plurality of initial features to obtain a description feature.

5. The method according to claim 1, characterized in that The integrating the first global image feature, the object feature, and the description feature to obtain a retrieval feature includes: Use the weighted values of the first global image feature, the object feature, and the description feature as the retrieval feature, where the weights of the first global image feature, the object feature, and the description feature are all preset values.

6. The method according to claim 1, characterized in that The determining the retrieval result by using the retrieval feature includes: Obtain the candidate feature of a candidate image, where the candidate image belongs to the database targeted by the retrieval request; Determine the target candidate feature that meets the retrieval condition with the retrieval feature; Use the candidate image corresponding to the target candidate feature as the target image to generate the retrieval result.

7. The method according to claim 6, wherein The determining the candidate feature of the candidate image includes: Extract a second global image feature from the candidate image; Invoke a fourth model to process the candidate image to generate a plurality of second image description texts for describing the content of the candidate image, where the fourth model is used to output a text describing the image based on the input image; Extract the text features of each of the second image description texts, where the text features of the second image description texts include the semantic information of the candidate image; Perform an averaging process on the text features of the obtained plurality of second image description texts to obtain the text feature of the candidate image; Generate a candidate feature according to the second global image feature and the text feature of the candidate image.

8. An image retrieval device based on multimodal semantics, characterized in that, The apparatus includes: An acquisition unit, configured to acquire a retrieval request, where the retrieval request includes a reference image and a modification text, the reference image is used to assist in retrieving an image, and the modification text is used to describe the modification object and the modification process of the image retrieved from the reference image; An extraction unit, configured to extract a first global image feature from the reference image, extract an object feature from the reference image and the modification text, and extract a description feature from the modification text, where the first global image feature includes the visual information of the reference image, the object feature includes the information of the modification object, and the description feature includes the semantic information of the modification text; An integration unit, configured to integrate the first global image feature, the object feature, and the description feature to obtain a retrieval feature; A retrieval unit, configured to use the retrieval feature to determine a retrieval result, where the retrieval result includes a target image that matches the retrieval feature.

9. An image retrieval device based on multimodal semantics, characterized in that Includes: A processor, a memory, and a system bus; The processor and the memory are connected through the system bus; The memory is used to store one or more programs, where the one or more programs include instructions, and when the instructions are executed by the processor, the processor is caused to execute the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Instructions are stored in the computer-readable storage medium, and when the instructions are run on a terminal device, the terminal device is caused to execute the method according to any one of claims 1-7.

Images