Image retrieval method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an image retrieval method and device, electronic equipment and storage medium, belonging to the technical field of computers. The method comprises: inputting a plurality of reference domain vectors, a plurality of reference category vectors and an embedding vector of a target image into a target prompt word generation model for encoding to obtain a target domain vector and a target category vector; inputting the target domain vector, the target category vector and the embedding vector of the target image into an image feature extraction model for feature extraction to obtain an image feature of the target image; and determining a retrieval result from an image database based on the image feature of the target image. When facing a target image of an unknown domain and an unknown category, the target domain vector and the target category vector are obtained according to the existing reference domain vector and the existing reference category vector, so that the obtained image feature includes domain features and category features, which is beneficial for the image retrieval task and effectively improves the generalization performance of the overall model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an image retrieval method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the advent of the digital age, image data is experiencing explosive growth, making it crucial to accurately locate images of interest to users from massive image databases. Image retrieval technology is a key technology for efficiently processing massive amounts of image data, and it has broad application prospects in various scenarios such as e-commerce, video recommendation, and advertising marketing. Therefore, how to achieve image retrieval is a problem that needs to be solved.

[0003] Typically, image retrieval technology extracts image features to classify, index, and retrieve images, enabling users to quickly and accurately identify the images they need. For example, it first extracts image features from the target image using a deep vision model, then calculates the similarity of image features between the target image and multiple images in an image database, thereby finding the image most similar to the target image.

[0004] However, traditional image retrieval techniques struggle to establish relationships between images in known and unknown domains. When deep vision models process images in unknown domains, they find it difficult to extract specific image features, leading to significant biases in image retrieval results. Summary of the Invention

[0005] This disclosure provides an image retrieval method, apparatus, electronic device, and storage medium, which enables the acquisition of image features, including domain features and category features, when faced with target images of unknown domains and categories. This is more beneficial for subsequent image retrieval tasks and effectively improves the generalization performance of the overall model. The technical solution of this disclosure is as follows:

[0006] According to one aspect of the embodiments of this disclosure, an image retrieval method is provided, the method comprising:

[0007] Multiple reference domain vectors, multiple reference category vectors, and the embedding vector of the target image are input into the target prompt word generation model for encoding to obtain target domain vectors and target category vectors. Different reference domain vectors refer to word vectors used to represent different image domains, and different reference category vectors refer to word vectors used to represent different image categories. The target domain vector refers to the word vector used to represent the image domain to which the target image belongs, and the target category vector refers to the word vector used to represent the image category to which the target image belongs.

[0008] The target neighborhood vector, the target category vector, and the embedding vector of the target image are input into an image feature extraction model for feature extraction to obtain the image features of the target image. The image features of the target image include the neighborhood features of the image neighborhood to which the target image belongs and the category features of the image category to which it belongs.

[0009] Based on the image features of the target image, a retrieval result is determined from an image database. The image database stores images from various image domains and image types. The retrieval result includes at least one image, and the similarity between the at least one image and the target image is greater than the similarity between other images in the image database and the target image.

[0010] According to another aspect of the present disclosure, an image retrieval apparatus is provided, the apparatus comprising:

[0011] The first generation unit is configured to input multiple reference neighborhood vectors, multiple reference category vectors, and the embedding vector of the target image into the target prompt word generation model for encoding to obtain target neighborhood vectors and target category vectors. Different reference neighborhood vectors refer to those used to represent different image neighborhoods, and different reference category vectors refer to those used to represent different image categories. The target neighborhood vector refers to those used to represent the image neighborhood to which the target image belongs, and the target category vector refers to those used to represent the image category to which the target image belongs.

[0012] The first extraction unit is configured to input the target neighborhood vector, the target category vector, and the embedding vector of the target image into an image feature extraction model to extract features and obtain the image features of the target image. The image features of the target image include the neighborhood features of the image neighborhood to which the target image belongs and the category features of the image category to which it belongs.

[0013] The first determining unit is configured to determine a retrieval result from an image database based on the image features of the target image. The image database stores images from multiple image domains and multiple image types. The retrieval result includes at least one image, and the similarity between the at least one image and the target image is greater than the similarity between other images in the image database and the target image.

[0014] In some embodiments, the first determining unit is configured to sort the images in the image database according to the order of the similarity between the image features of each image in the image database and the image features of the target image in descending order; and determine the first N images as the retrieval results, where N is a positive integer.

[0015] In some embodiments, the first determining unit is configured to, for any image in the image database, determine the similarity between the image features of the image and the image features of the target image; and if the similarity is greater than a similarity threshold, use the image as an image in the retrieval results.

[0016] In some embodiments, the apparatus further includes:

[0017] The second generation unit is configured to input at least one reference neighborhood vector, at least one reference category vector, and the embedding vector of the first sample image into the prompt word generation model to be trained for encoding, so as to obtain a sample neighborhood vector and a sample category vector. The sample neighborhood vector refers to the word vector used to represent the image neighborhood to which the first sample image belongs, and the sample category vector refers to the word vector used to represent the image category to which the first sample image belongs.

[0018] The second extraction unit is configured to input the sample neighborhood vector, the sample category vector, and the embedding vector of the first sample image into the image feature extraction model to extract features and obtain the image features of the first sample image. The image features of the first sample image include the neighborhood features of the image neighborhood to which the first sample image belongs and the category features of the image category to which it belongs.

[0019] The third extraction unit is configured to input the first sample text into the text feature extraction model to extract features and obtain the text features of the first sample text. The first sample text is used to describe the image domain and image category to which the first sample image belongs.

[0020] The training unit is configured to train the prompt word generation model to be trained using a cross-entropy loss function based on the similarity between the image features of the first sample image and the text features of the first sample text, thereby obtaining the target prompt word generation model.

[0021] In some embodiments, the apparatus further includes:

[0022] The second determining unit is configured to determine the image region and image category to which the first sample image belongs;

[0023] A masking unit is configured to perform a first masking process on the plurality of reference neighborhood vectors and the plurality of reference category vectors based on the image neighborhood and the image category, to obtain at least one reference neighborhood vector and at least one reference category vector, wherein the first masking process is used to mask the reference neighborhood vector corresponding to the image neighborhood and the reference category vector corresponding to the image category.

[0024] In some embodiments, the masking unit is configured to set a first-class neighborhood mask corresponding to the image neighborhood and a first-class category mask corresponding to the image category to 0, and set a first-class neighborhood mask corresponding to other image neighborhoods and a first-class category mask corresponding to other image categories to 1. The first-class neighborhood mask is used to mask a reference neighborhood vector corresponding to the image neighborhood, and the first-class category mask is used to mask a reference category vector corresponding to the image category. Based on the image neighborhood represented by each reference neighborhood vector, the plurality of reference neighborhood vectors are multiplied by their corresponding first-class neighborhood masks to obtain the at least one reference neighborhood vector. Based on the image category represented by each reference category vector, the plurality of reference category vectors are multiplied by their corresponding first-class category masks to obtain the at least one reference category vector.

[0025] In some embodiments, the apparatus further includes:

[0026] The second extraction unit is further configured to input the reference neighborhood vector to be updated, the reference category vector to be updated, and the embedding vector of the second sample image into the image feature extraction model for feature extraction to obtain the image features of the second sample image. The reference neighborhood vector to be updated refers to the initial neighborhood vector corresponding to the second sample image, and the reference category vector to be updated refers to the initial category vector corresponding to the second sample image. The image features of the second sample image include the neighborhood features of the image neighborhood to which the second sample image belongs and the category features of the image category to which it belongs.

[0027] The third extraction unit is further configured to input the second sample text into the text feature extraction model for feature extraction to obtain the text features of the second sample text, wherein the second sample text is used to describe the image domain and image category to which the second sample image belongs;

[0028] The update unit is configured to update the reference neighborhood vector and the reference category vector to be updated using a cross-entropy loss function based on the similarity between the image features of the second sample image and the text features of the second sample text, so as to obtain the updated reference neighborhood vector and the updated reference category vector. The updated reference neighborhood vector represents the image neighborhood to which the second sample image belongs, and the updated reference category vector represents the image category to which the second sample image belongs.

[0029] In some embodiments, the apparatus further includes:

[0030] The second determining unit is further configured to determine the image region and image category to which the second sample image belongs;

[0031] The masking unit is further configured to perform a second masking process on a plurality of initial neighborhood vectors and a plurality of initial category vectors based on the image neighborhood and the image category to obtain the reference neighborhood vector to be updated and the reference category vector to be updated. The second masking process is used to retain the initial neighborhood vector corresponding to the image neighborhood and the initial category vector corresponding to the image category.

[0032] In some embodiments, the masking unit is further configured to set the second type of neighborhood mask corresponding to the image neighborhood and the second type of category mask corresponding to the image category to 1, and set the second type of neighborhood mask corresponding to other image neighborhoods and the second type of category mask corresponding to other image categories to 0. The second type of neighborhood mask is used to retain the initial neighborhood vector corresponding to the image neighborhood, and the second type of category mask is used to retain the initial category vector corresponding to the image category. Based on the image neighborhood represented by each initial neighborhood vector, the plurality of initial neighborhood vectors are multiplied by their corresponding second type of neighborhood masks to obtain the reference neighborhood vector to be updated. Based on the image category represented by each initial category vector, the plurality of initial category vectors are multiplied by their corresponding second type of category masks to obtain the reference category vector to be updated.

[0033] According to another aspect of the embodiments of this disclosure, an electronic device is provided, the electronic device comprising:

[0034] One or more processors;

[0035] Memory used to store the executable program code of the processor;

[0036] The processor is configured to execute the program code to implement the image retrieval method described above.

[0037] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which, when the program code in the computer-readable storage medium is executed by a processor of an electronic device, enables the electronic device to perform the image retrieval method described above.

[0038] According to another aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described image retrieval method.

[0039] This disclosure provides an image retrieval method. It uses a target prompt word generation model to generate target neighborhood vectors and target category vectors based on the embedding vector of the input target image, representing the image neighborhood and category to which the target image belongs. Simultaneously, an image feature extraction model generates image features of the target image based on the input target neighborhood vector, target category vector, and the target image's embedding vector, and then determines the retrieval result based on these image features. When faced with target images of unknown neighborhoods and categories, this method uses the target prompt word generation model to obtain the target neighborhood vector and target category vector based on existing reference neighborhood vectors and reference category vectors, thus determining the neighborhood and category to which the target image belongs, guiding subsequent image feature extraction. Compared to traditional methods that directly extract image features from the target image, this method obtains image features including neighborhood and category features, which is more beneficial for subsequent image retrieval tasks and effectively improves the overall model's generalization performance.

[0040] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0042] Figure 1 This is a schematic diagram illustrating the implementation environment of an image retrieval method according to an exemplary embodiment.

[0043] Figure 2 This is a flowchart illustrating an image retrieval method according to an exemplary embodiment.

[0044] Figure 3 This is a flowchart illustrating another image retrieval method according to an exemplary embodiment.

[0045] Figure 4 This is a schematic diagram illustrating a first training phase according to an exemplary embodiment.

[0046] Figure 5 This is a schematic diagram illustrating a second training phase according to an exemplary embodiment.

[0047] Figure 6 This is a schematic diagram illustrating an image retrieval process according to an exemplary embodiment.

[0048] Figure 7 This is a block diagram illustrating an image retrieval device according to an exemplary embodiment.

[0049] Figure 8 This is a block diagram illustrating another image retrieval device according to an exemplary embodiment.

[0050] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0051] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0052] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0053] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the target images and images in the image database involved in this disclosure were obtained with full authorization.

[0054] Figure 1 This is a schematic diagram illustrating an implementation environment for an image retrieval method according to an exemplary embodiment. See also... Figure 1 The implementation environment specifically includes: terminal 101 and server 102. Terminal 101 can be connected to server 102 via wireless network or wired network.

[0055] Terminal 101 can be at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), and laptop computer. An application can be installed and run on terminal 101. This application can extract image features from the target image, obtaining image features including neighborhood features and category features. This application is associated with server 102, which provides background services to terminal 101.

[0056] Terminal 101 can refer to one of a plurality of terminals, and this embodiment uses terminal 101 as an example. Those skilled in the art will know that the number of terminals can be more or less. For example, there can be several terminals, or dozens or hundreds of terminals, or more. This embodiment does not limit the number of terminals or the type of devices.

[0057] Server 102 can be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of servers may be more or fewer, and this disclosure does not limit this. Of course, server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, server 102 undertakes the main computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the main computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture. Server 102 can be connected to terminal 101 and other terminals via a wireless network or a wired network. Optionally, the number of servers may be more or fewer, and this disclosure does not limit this.

[0058] Figure 2 This is a flowchart illustrating an image retrieval method according to an exemplary embodiment, such as... Figure 2 As shown, the method is performed by an electronic device and includes the following steps:

[0059] In step S201, multiple reference neighborhood vectors, multiple reference category vectors, and the embedding vector of the target image are input into the target prompt word generation model for encoding to obtain target neighborhood vectors and target category vectors. Different reference neighborhood vectors refer to word vectors used to represent different image neighborhoods, different reference category vectors refer to word vectors used to represent different image categories, target neighborhood vectors refer to word vectors used to represent the image neighborhood to which the target image belongs, and target category vectors refer to word vectors used to represent the image category to which the target image belongs.

[0060] In this embodiment, different reference neighborhood vectors are used to represent different image neighborhoods. Optionally, the image neighborhood can refer to the style of the image, such as "painting," "sketch," or "clipart." Different reference category vectors are used to represent different image categories. Optionally, the image category can refer to the subject of the image, such as "apple," "bicycle," or "cat." Through the target prompt word generation model, based on multiple reference neighborhood vectors and multiple reference category vectors, combined with the embedding vector of the target image, target neighborhood vectors and target category vectors are obtained. This achieves the generalization of neighborhood vectors and category vectors from known neighborhoods and categories to unknown neighborhoods and categories, facilitating the generation of image features corresponding to the image neighborhood and category through the image feature extraction model in subsequent steps.

[0061] In step S202, the target neighborhood vector, the target category vector, and the embedding vector of the target image are input into the image feature extraction model for feature extraction to obtain the image features of the target image. The image features of the target image include the neighborhood features of the image neighborhood to which the target image belongs and the category features of the image category to which it belongs.

[0062] In this embodiment, since the target neighborhood vector and the target category vector indicate the image neighborhood and image category to which the target image belongs, respectively, inputting the target neighborhood vector, the target category vector, and the embedding vector of the target image together into the image feature extraction model helps generate image features that include the neighborhood features of the aforementioned image neighborhood and the category features of the aforementioned image category. Compared to image features generated directly from the target image through image feature extraction, the image features obtained in this way can more accurately represent the target image, thereby distinguishing different images and improving the retrieval efficiency and accuracy of image retrieval.

[0063] In step S203, based on the image features of the target image, the retrieval results are determined from the image database. The image database stores images from various image domains and images of various image types. The retrieval results include at least one image, and the similarity between the at least one image and the target image is greater than the similarity between other images in the image database and the target image.

[0064] In this embodiment of the disclosure, image features are extracted from both the target image and the images stored in the image database to obtain the image features of the target image and the image features of the stored images. The similarity between the image features of the target image and the image features of the stored images is determined, and then the multiple similarities are sorted to determine the retrieval results and feed them back to the user.

[0065] This disclosure provides an image retrieval method. It uses a target prompt word generation model to generate target neighborhood vectors and target category vectors based on the embedding vector of the input target image, representing the image neighborhood and category to which the target image belongs. Simultaneously, an image feature extraction model generates image features of the target image based on the input target neighborhood vector, target category vector, and the target image's embedding vector, and then determines the retrieval result based on these image features. When faced with target images of unknown neighborhoods and categories, this method uses the target prompt word generation model to obtain the target neighborhood vector and target category vector based on existing reference neighborhood vectors and reference category vectors, thus determining the neighborhood and category to which the target image belongs, guiding subsequent image feature extraction. Compared to traditional methods that directly extract image features from the target image, this method obtains image features including neighborhood and category features, which is more beneficial for subsequent image retrieval tasks and effectively improves the overall model's generalization performance.

[0066] In some embodiments, determining retrieval results from an image database based on image features of the target image includes:

[0067] The images in the image database are sorted in descending order of similarity between the image features of each image in the image database and the image features of the target image.

[0068] The first N images are selected as the search results, where N is a positive integer.

[0069] In this embodiment of the disclosure, by means of the above method, a fixed number of images similar to the target image can be determined. When the similarity between the image features is low, the corresponding image can also be determined, which makes it convenient for users to compare and view the search results.

[0070] In some embodiments, determining retrieval results from an image database based on image features of the target image includes:

[0071] For any image in the image database, determine the similarity between the image features of the image and the image features of the target image;

[0072] If the similarity is greater than the similarity threshold, the image will be included in the search results.

[0073] In this embodiment of the disclosure, by means of the above method, all images with a similarity greater than the similarity threshold can be identified as search results, so that users can obtain images that are more similar to the target image, thereby improving the user's search experience.

[0074] In some embodiments, the training method for the target prompt word generation model includes:

[0075] At least one reference neighborhood vector, at least one reference category vector, and the embedding vector of the first sample image are input into the prompt word generation model to be trained for encoding to obtain the sample neighborhood vector and the sample category vector. The sample neighborhood vector refers to the word vector used to represent the image neighborhood to which the first sample image belongs, and the sample category vector refers to the word vector used to represent the image category to which the first sample image belongs.

[0076] The sample neighborhood vector, sample category vector, and embedding vector of the first sample image are input into the image feature extraction model for feature extraction to obtain the image features of the first sample image. The image features of the first sample image include the neighborhood features of the image neighborhood to which the first sample image belongs and the category features of the image category to which the first sample image belongs.

[0077] The first sample text is input into the text feature extraction model for feature extraction to obtain the text features of the first sample text. The first sample text is used to describe the image domain and image category to which the first sample image belongs.

[0078] Based on the similarity between the image features of the first sample image and the text features of the first sample text, the prompt word generation model to be trained is trained using the cross-entropy loss function to obtain the target prompt word generation model.

[0079] In this embodiment of the disclosure, since the image features of the first sample image include the domain features of the image domain indicated by the sample domain vector and the category features of the image category indicated by the sample category vector, and the text features of the first sample text can reflect the known image domain and image category of the second sample image, the similarity can indicate the difference between the actual image domain and image category and the simulated image domain and image category, thereby guiding the training of the prompt word generation model.

[0080] In some embodiments, the method further includes:

[0081] Determine the image region and image category to which the first sample image belongs;

[0082] Based on the image domain and image category, a first masking process is performed on multiple reference domain vectors and multiple reference category vectors to obtain at least one reference domain vector and at least one reference category vector. The first masking process is used to mask the reference domain vector corresponding to the image domain and the reference category vector corresponding to the image category.

[0083] In this embodiment of the disclosure, by masking both the reference neighborhood vector corresponding to the image neighborhood of the first sample image and the reference category vector corresponding to the image category, it is possible to better generalize the reference neighborhood vector of the known neighborhood and the reference category vector of the known category to the unknown neighborhood and the unknown category.

[0084] In some embodiments, based on image neighborhood and image category, a first masking process is performed on multiple reference neighborhood vectors and multiple reference category vectors to obtain at least one reference neighborhood vector and at least one reference category vector, including:

[0085] Set the first type of neighborhood mask corresponding to the image neighborhood and the first type of category mask corresponding to the image category to 0, and set the first type of neighborhood mask corresponding to other image neighborhoods and the first type of category mask corresponding to other image categories to 1. The first type of neighborhood mask is used to mask the reference neighborhood vector corresponding to the image neighborhood, and the first type of category mask is used to mask the reference category vector corresponding to the image category.

[0086] Based on the image domain represented by each reference domain vector, the multiple reference domain vectors are multiplied by the corresponding first-type domain mask to obtain at least one reference domain vector;

[0087] Based on the image category represented by each reference category vector, multiple reference category vectors are multiplied by their corresponding first category masks to obtain at least one reference category vector.

[0088] In this embodiment of the disclosure, by setting different values ​​for the masks corresponding to different image domains and different image categories, the masking of the reference domain vector corresponding to the image domain and the reference category vector corresponding to the image category of the first sample image can be achieved efficiently, quickly and accurately.

[0089] In some embodiments, the method further includes:

[0090] The reference neighborhood vector to be updated, the reference category vector to be updated, and the embedding vector of the second sample image are input into the image feature extraction model for feature extraction to obtain the image features of the second sample image. The reference neighborhood vector to be updated refers to the initial neighborhood vector corresponding to the second sample image, and the reference category vector to be updated refers to the initial category vector corresponding to the second sample image. The image features of the second sample image include the neighborhood features of the image neighborhood to which the second sample image belongs and the category features of the image category to which it belongs.

[0091] The second sample text is input into the text feature extraction model for feature extraction to obtain the text features of the second sample text. The second sample text is used to describe the image domain and image category to which the second sample image belongs.

[0092] Based on the similarity between the image features of the second sample image and the text features of the second sample text, the reference neighborhood vector and the reference category vector to be updated are updated using the cross-entropy loss function, resulting in the updated reference neighborhood vector and the updated reference category vector. The updated reference neighborhood vector represents the image neighborhood to which the second sample image belongs, and the updated reference category vector represents the image category to which the second sample image belongs.

[0093] In this embodiment of the disclosure, since the image features of the second sample image include the domain features of the image domain indicated by the reference domain vector and the category features of the image category indicated by the reference category vector, and the text features of the first sample text can reflect the known image domain and image category of the second sample image, the similarity can indicate the difference between the actual image domain and image category and the simulated image domain and image category, thereby guiding the updating of the reference domain vector and the reference category vector.

[0094] In some embodiments, the method further includes:

[0095] Determine the image region and image category to which the second sample image belongs;

[0096] Based on the image neighborhood and image category, a second masking process is performed on multiple initial neighborhood vectors and multiple initial category vectors to obtain reference neighborhood vectors and reference category vectors to be updated. The second masking process is used to preserve the initial neighborhood vectors corresponding to the image neighborhood and the initial category vectors corresponding to the image category.

[0097] In this embodiment of the disclosure, by retaining the initial neighborhood vector corresponding to the image neighborhood to which the second sample image belongs and the initial category vector corresponding to the image category, it can be ensured that each neighborhood vector can only learn knowledge of a specific image neighborhood and each category vector can only learn knowledge of a specific image category. This facilitates image feature extraction in subsequent processes based on the image neighborhood to which the image belongs and the image category, combined with the neighborhood vector corresponding to the image neighborhood and the category vector corresponding to the image category.

[0098] In some embodiments, based on the image neighborhood and image category, a second masking process is performed on multiple initial neighborhood vectors and multiple initial category vectors to obtain reference neighborhood vectors to be updated and reference category vectors to be updated, including:

[0099] Set the second-class neighborhood mask corresponding to the image neighborhood and the second-class category mask corresponding to the image category to 1, and set the second-class neighborhood mask corresponding to other image neighborhoods and the second-class category mask corresponding to other image categories to 0. The second-class neighborhood mask is used to retain the initial neighborhood vector corresponding to the image neighborhood, and the second-class category mask is used to retain the initial category vector corresponding to the image category.

[0100] Based on the image domain represented by each initial neighborhood vector, the multiple initial neighborhood vectors are multiplied by the corresponding second-type neighborhood mask to obtain the reference neighborhood vector to be updated.

[0101] Based on the image category represented by each initial category vector, the multiple initial category vectors are multiplied by the corresponding second category mask to obtain the reference category vector to be updated.

[0102] In this embodiment of the disclosure, by setting different values ​​for the masks corresponding to different image domains and different image categories, it is possible to efficiently, quickly and accurately preserve the reference domain vector corresponding to the image domain and the reference category vector corresponding to the image category of the first sample image.

[0103] The above Figure 2 The diagram shown is a flowchart of an image retrieval method according to this disclosure. The image retrieval scheme provided by this disclosure will be further described below. Figure 3 This is a flowchart illustrating another image retrieval method according to an exemplary embodiment, see [link to flowchart]. Figure 3 This method is performed by an electronic device and includes the following steps:

[0104] It should be noted that this embodiment includes two training phases: an update phase for the reference domain vector and the reference category vector, and a training phase for the prompt word generation model. For ease of description, the image retrieval method in this embodiment is referred to as Adaptive Prompt Learning (APL). The first training phase is referred to as the Source Prompt Learning (SPL) phase, as described in steps S301 to S304 below. In the first training phase, the reference domain vector and the reference category vector are updated to capture knowledge from the known domain and known categories. Optionally, the reference domain vector is referred to as the Domain Prompt (DP), and the reference category vector is referred to as the Semantic Prompt (SP), as described in steps S305 to S309 below. The second training phase is referred to as the Target Prompts Adaptation (TPA) phase. In the second training phase, a prompt word generation model is trained to dynamically transform the reference domain vector and the reference category vector into the target domain vector and the target category vector. It should be noted that images with a known domain or category refer to images whose domain and category have been labeled. When performing feature extraction, image features containing domain and category features can be directly obtained. Images with an unknown domain or category refer to images without a labeled domain and category. When performing feature extraction directly on such images, the obtained image features may not contain domain and category features, resulting in a poor representation of the image.

[0105] In step S301, the image neighborhood and image category to which the second sample image belongs are determined; based on the image neighborhood and image category, a second masking process is performed on multiple initial neighborhood vectors and multiple initial category vectors to obtain reference neighborhood vectors to be updated and reference category vectors to be updated. The second masking process is used to retain the initial neighborhood vectors corresponding to the image neighborhood and the initial category vectors corresponding to the image category.

[0106] In this embodiment, firstly, a second sample image is acquired, and the image domain and image category to which the second sample image belongs are determined based on the labeled information. The second sample image is any sample image labeled with an image domain and image category. Then, through a second masking process, the initial domain vectors corresponding to the aforementioned image domains and the initial domain vectors corresponding to the aforementioned image categories are retained, while the initial domain vectors corresponding to the remaining image domains and the initial domain vectors corresponding to the remaining image categories are masked. This ensures that each reference domain vector can only learn the knowledge of its corresponding image domain, and each reference category vector can only learn the knowledge of its corresponding image category. Optionally, the image domain can indicate the style of the image, and the image category can indicate the subject of the image. For example, the second sample image is an image from the "painting" image domain with an image category of "bicycle"; or, the second sample image is an image from the "sketch" image domain with an image category of "apple".

[0107] It should be noted that image domain and image category are equivalent to two attribute features of an image, and they are parallel. Correspondingly, images within the same image domain may belong to different image categories, and images within the same image category may belong to different image domains.

[0108] Optionally, different neighborhood vectors refer to word vectors used to represent different image neighborhoods, and different category vectors refer to word vectors used to represent different image categories. It should be noted that the initial neighborhood vector, reference neighborhood vector, and target neighborhood vector refer to neighborhood vectors under different processing steps, and the initial category vector, reference category vector, and target category vector refer to category vectors under different processing steps. For example, the set of multiple initial neighborhood vectors is called DP, and the set of multiple initial category vectors is called SP. Correspondingly, for K image neighborhoods, K learnable word vectors are used as K initial neighborhood vectors; for... Image categories, Learnable word vectors are used as... An initial category vector. See the formula below.

[0109]

[0110]

[0111] Here, DP refers to a set of initial neighborhood vectors, and SP refers to a set of initial class vectors. i This refers to the initial neighborhood vector, sp i This refers to the initial class vector. The dimension used to represent the initial neighborhood vector and the initial class vector. K is used to represent the number of image neighborhoods. Used to indicate the number of image categories.

[0112] In some embodiments, the second masking process is implemented by setting different second-type neighborhood masks and second-type category masks. Accordingly, the second-type neighborhood mask corresponding to the image neighborhood and the second-type category mask corresponding to the image category are set to 1, while the second-type neighborhood masks corresponding to other image neighborhoods and the second-type category masks corresponding to other image categories are set to 0. The second-type neighborhood mask is used to retain the initial neighborhood vector corresponding to the image neighborhood, and the second-type category mask is used to retain the initial category vector corresponding to the image category. Based on the image neighborhood represented by each initial neighborhood vector, multiple initial neighborhood vectors are multiplied by their corresponding second-type neighborhood masks to obtain the reference neighborhood vector to be updated. Based on the image category represented by each initial category vector, multiple initial category vectors are multiplied by their corresponding second-type category masks to obtain the reference category vector to be updated. By setting the masks corresponding to the image neighborhood and the image category to 1, and setting the masks corresponding to the remaining image neighborhoods and the remaining image categories to 0, and then multiplying them by their corresponding initial neighborhood vectors and initial category vectors, the second masking process can be implemented, ensuring that each reference neighborhood vector can only learn the knowledge of its corresponding image neighborhood, and each reference category vector can only learn the knowledge of its corresponding image category.

[0113] In other words, to ensure that each neighborhood vector and category vector learns knowledge of only a specific image neighborhood or image category, in each iteration, only the neighborhood vector and category vector corresponding to the input image are updated, while neighborhood vectors and category vectors irrelevant to the input image are masked. The input image here refers to the image input to the image feature extraction model, i.e., the second sample image described below. It should be noted that in each iteration, the input to the image feature extraction model can be one image or multiple images; this embodiment does not impose any limitation on this.

[0114] For example, the second type of domain mask is called m sd The second type of category mask is called m. sc The input image is called... At this point, the image neighborhood of the input image is d, and the image category of the input image is c. To update only the neighborhood and category vectors related to the input image, we will... and Set it to 1, and set the other masks to 0. That is, only retain the neighborhood vector of image d and the class vector of image c, and mask the neighborhood vectors other than those of image d and the class vectors other than those of image c.

[0115] In step S302, the reference neighborhood vector to be updated, the reference category vector to be updated, and the embedding vector of the second sample image are input into the image feature extraction model for feature extraction to obtain the image features of the second sample image. The image features of the second sample image include the neighborhood features of the image neighborhood to which the second sample image belongs and the category features of the image category to which it belongs.

[0116] In this embodiment, the reference neighborhood vector to be updated refers to the initial neighborhood vector corresponding to the second sample image, and the reference category vector to be updated refers to the initial category vector corresponding to the second sample image. The embedding vector of the second sample image is a vector that can reflect information such as the global features, local features, and semantic features of the second sample image. The second sample image is high-dimensional image data, while the embedding vector of the second sample image is low-dimensional data that is easier to understand and process. Therefore, using the embedding vector of the second sample image facilitates tasks such as model training and image retrieval.

[0117] Image feature extraction models are used to generate image features based on the embedding vectors of an input image. For example, an image feature extraction model refers to a CLIP (Contrastive Language-Image Pre-training) image encoder. Optionally, the embedding vector of a second sample image is input into the image feature extraction model; this second sample image can also be referred to as the input image of the image feature extraction model.

[0118] For example, the reference neighborhood vector to be updated, the reference class vector to be updated, the embedding vector of the second sample image, and other input parameters are collectively referred to as x. p Alternatively, they can all be referred to as prompted input. See the formula below.

[0119] x p =[x0,m sd ⊙DP,m sc ⊙SP,E0]

[0120] Where, m sd This refers to the second type of neighborhood mask, m sc This refers to the second type of category mask. DP refers to the set of multiple reference neighborhood vectors, and SP refers to the set of multiple reference category vectors. ⊙ indicates element-wise multiplication, used to retain the relevant reference neighborhood vectors and reference category vectors of the input image while masking the irrelevant ones. E0 refers to the embedding vector of the second sample image, which is also the embedding vector of the input image. x0 refers to the parameters used in the image feature extraction process.

[0121] In step S303, the second sample text is input into the text feature extraction model for feature extraction to obtain the text features of the second sample text. The second sample text is used to describe the image domain and image category to which the second sample image belongs.

[0122] In this embodiment of the disclosure, the second sample text is used to indicate the image domain and image category to which the labeled second sample image belongs. A text feature extraction model is used to generate text features based on the input text. For example, the text feature extraction model refers to the CLIP text encoder. CLIP primarily uses contrastive learning to map images and text, thereby learning the relationship between them. CLIP optimizes the image encoder and text encoder so that matching image features and text features are as close as possible in the semantic space, while mismatched image features and text features are as far apart as possible.

[0123] It should be noted that since the second sample text describes the image domain and category to which the second sample image belongs, the text features obtained after feature extraction can reflect the aforementioned image domain and category. Since the image features obtained after feature extraction contain both domain and category features, these image features can also reflect the aforementioned image domain and category. In subsequent update steps, the reference domain vector and reference category vector are updated based on the similarity between the text features and image features.

[0124] In step S304, based on the similarity between the image features of the second sample image and the text features of the second sample text, the reference neighborhood vector to be updated and the reference category vector to be updated are updated using the cross-entropy loss function, resulting in the updated reference neighborhood vector and the updated reference category vector.

[0125] In this embodiment of the disclosure, during each iteration, after obtaining the text features and image features, their cosine similarity is calculated. Since the text features reflect the known image neighborhood and image category of the second sample image, and the image features reflect the image neighborhood indicated by the reference neighborhood vector and the image category indicated by the reference category neighborhood vector, this similarity can indicate the difference between the actual image neighborhood and image category and the simulated image neighborhood and image category. Therefore, based on this similarity, the reference neighborhood vector and reference category vector can be updated using the cross-entropy loss function.

[0126] Before the update, the reference neighborhood vector to be updated is a random initial neighborhood vector, and the reference class vector to be updated is a random initial class vector. After the update, the updated reference neighborhood vector indicates the image neighborhood to which the second sample image belongs, and the updated reference class vector indicates the image class to which the second sample image belongs.

[0127] It should be noted that the cross-entropy loss function is shown in the following formula.

[0128]

[0129] Where, x p This refers to the input of the image feature extraction model, P. text c This refers to the input of the text feature extraction model. image (x p ) refers to the output of the image feature extraction model, f text (P text c ) refers to the output of the text feature extraction model. δ(·,·) represents the calculation of cosine similarity. The value is 1 when the image category of the input image is in SP, and 0 when the image category of the input image is not in SP. This means that when multiple images are input, the loss value is calculated for each image.

[0130] The above update method not only updates the reference domain vector and reference category vector, but also aligns image features from different image domains to a unified image category. In other words, text features are obtained based on known input text (i.e., the second sample text). Therefore, regardless of the image domain to which the input image (i.e., the second sample image) belongs, for input images within the same image category, their image features can be aligned according to the image category indicated by the text features, thereby mitigating the domain drift problem. Domain drift refers to the inability to accurately retrieve images during the testing phase due to differences in the data distribution between training and testing data. Under the above update method, image features can be aligned based on the image category, ensuring that the image features generated during the testing phase reflect the category characteristics of the image, making image retrieval more accurate. For ease of description, the masking and image feature alignment methods are referred to as masking and alignment objectives.

[0131] For ease of description, the above update process for the reference neighborhood vector and reference class vector can be found in [reference needed]. Figure 4 As shown, Figure 4This is a schematic diagram illustrating a first training phase according to an exemplary embodiment. 401 is the embedding vector of the input image, which is also the embedding vector of the second sample image. 402 is a set of multiple initial neighborhood vectors. A second masking process is applied to 402, retaining the initial neighborhood vector 403 corresponding to the input image and masking the remaining initial neighborhood vectors. 404 is a set of multiple initial class vectors. A second masking process is applied to 404, retaining the initial class vector 405 corresponding to the input image and masking the remaining initial class vectors. 406 is the parameter used in the image extraction process. 407 is the input text, which is also the second sample text. Figure 4 As shown, in the first stage of training, masking and contrasting targets are used, and the initial neighborhood vector and initial class vector corresponding to the input image are updated based on the cross-entropy loss function to obtain the updated reference neighborhood vector and reference class vector corresponding to the input image. During the first stage of training, the parameters of the text feature extraction model, the image feature extraction model, and the image extraction process remain unchanged. At this time, the output image features are aligned based on the class centers indicated by the text features, and the image features corresponding to different class centers are far apart from each other.

[0132] In step S305, the image neighborhood and image category to which the first sample image belongs are determined; based on the image neighborhood and image category, a first masking process is performed on multiple reference neighborhood vectors and multiple reference category vectors to obtain at least one reference neighborhood vector and at least one reference category vector. The first masking process is used to mask the reference neighborhood vector corresponding to the image neighborhood and the reference category vector corresponding to the image category.

[0133] In this embodiment, firstly, a first sample image is acquired, and the image neighborhood and image category to which the first sample image belongs are determined based on the labeled information. The first sample image is any sample image labeled with an image neighborhood and image category. Optionally, the first sample image and the second sample image may be the same or different. Then, through a first masking process, the reference neighborhood vectors corresponding to the aforementioned image neighborhoods and the reference neighborhood vectors corresponding to the aforementioned image categories are masked, while the reference neighborhood vectors corresponding to the remaining image neighborhoods and the remaining image categories are retained. This allows for the generalization of new sample neighborhood vectors and new sample category vectors using the existing reference neighborhood vectors and existing reference category vectors.

[0134] In some embodiments, the first masking process is implemented by setting different first-class neighborhood masks and first-class category masks. Accordingly, the first-class neighborhood mask corresponding to the image neighborhood and the first-class category mask corresponding to the image category are set to 0, while the first-class neighborhood masks corresponding to other image neighborhoods and the first-class category masks corresponding to other image categories are set to 1. The first-class neighborhood mask is used to mask the reference neighborhood vector corresponding to the image neighborhood, and the first-class category mask is used to mask the reference category vector corresponding to the image category. Based on the image neighborhood represented by each reference neighborhood vector, multiple reference neighborhood vectors are multiplied by their corresponding first-class neighborhood masks to obtain at least one reference neighborhood vector. Based on the image category represented by each reference category vector, multiple reference category vectors are multiplied by their corresponding first-class category masks to obtain at least one reference category vector. By setting the masks corresponding to the image neighborhood and the image category to 0, and setting the masks corresponding to the remaining image neighborhoods and the remaining image categories to 1, and then multiplying them by their corresponding reference neighborhood vectors and reference category vectors, the first masking process can be implemented, thereby generalizing new sample neighborhood vectors and new sample category vectors using existing reference neighborhood vectors and existing reference category vectors.

[0135] In other words, to train the prompt word generation model to generate corresponding sample neighborhood vectors and sample category vectors based on the input image, the known reference neighborhood vectors and known category vectors corresponding to the input image are masked, retaining only the reference neighborhood vectors and reference category vectors irrelevant to the input image, to guide the generation of sample neighborhood vectors and sample category vectors corresponding to the input image. Here, the input image refers to the image input to the prompt word generation model, i.e., the first sample image described below. It should be noted that in each iteration, the input to the prompt word generation model can be one image or multiple images; this embodiment does not impose any limitation on this.

[0136] For example, the first type of domain mask is called m td The first type of category mask is called m. tc The input image is called... At this point, the image neighborhood of the input image is d, and the image category of the input image is c. To generalize from the existing reference neighborhood vector and the existing reference category vector to obtain new sample neighborhood vectors and new sample category vectors, the following steps are taken: and Set one mask to 0, and set the others to 1. That is, only the reference neighborhood vector of image d and the reference category vector of image c are masked, while the reference neighborhood vectors outside the area of ​​image d and the reference category vectors outside the area of ​​image c are preserved. This simulates the generation process of sample neighborhood vectors and sample category vectors to train the prompt word generation model.

[0137] In step S306, at least one reference neighborhood vector, at least one reference category vector, and the embedding vector of the first sample image are input into the prompt word generation model to be trained for encoding to obtain the sample neighborhood vector and the sample category vector.

[0138] In this embodiment of the disclosure, at least one reference neighborhood vector refers to the reference neighborhood vector retained after the first masking process. At least one reference category vector refers to the reference category vector retained after the first masking process. The embedding vector of the first sample image refers to a vector that can reflect information such as global features, local features, and semantic features of the second sample image. The prompt word generation model is used to generate prompt word vectors representing the image neighborhood and image category to which the image belongs based on the embedding vector of the input image. The prompt word vector includes a neighborhood vector and a category vector.

[0139] For example, tp d This is called the sample neighborhood vector, and tp s This is called the sample category vector. The sample neighborhood vector and sample category vector are generated using a prompt word generation model, as shown in the formula below.

[0140] [tp d ,tp s ] = f target ([PT d ,PT s ,m td ⊙DP,m tc ⊙SP,E0])

[0141] Where, m td This refers to the first type of neighborhood mask, m tc This refers to the first category mask. DP refers to the set of updated reference neighborhood vectors, and SP refers to the set of updated reference category vectors. ⊙ represents element-wise multiplication, used to mask reference neighborhood and category vectors relevant to the input image, while retaining irrelevant reference neighborhood and category vectors. E0 refers to the embedding vector of the first sample image, which is also the embedding vector of the input image. PT d This refers to the domain vector template, which indicates the format information of the domain vector and generates sample domain vectors; PT s This refers to the category vector template, which indicates the format information of the category vectors and generates sample category vectors. Correspondingly, ft... target This refers to the prompt word generation model. [PT] d ,PT s ,m td ⊙DP,m tc ⊙SP,E0] refers to the input sequence of the prompt word generation model.

[0142] In step S307, the sample neighborhood vector, sample category vector, and embedding vector of the first sample image are input into the image feature extraction model for feature extraction to obtain the image features of the first sample image. The image features of the first sample image include the neighborhood features of the image neighborhood to which the first sample image belongs and the category features of the image category to which it belongs.

[0143] In this embodiment, the sample neighborhood vector refers to the word vector representing the image neighborhood to which the first sample image belongs, and the sample category vector refers to the word vector representing the image category to which the first sample image belongs. The embedding vector of the first sample image refers to a vector that can reflect information such as the global features, local features, and semantic features of the second sample image. Feature extraction is performed using the same image feature extraction model as in step S302, and will not be described in detail here.

[0144] For example, let the image feature of the first sample image be called x, as shown in the following formula.

[0145] x = f image ([x0,tp d ,tp s ,E0])

[0146] Among them, tp d It refers to the sample neighborhood vector, tp s This refers to the sample category vector. x0 refers to the parameters used in the image feature extraction process. E0 refers to the embedding vector of the first sample image, which is also the embedding vector of the input image. Correspondingly, f image This refers to an image feature extraction model. [x0,tp d ,tp s [E0] refers to the input sequence of the image feature extraction model.

[0147] In step S308, the first sample text is input into the text feature extraction model for feature extraction to obtain the text features of the first sample text. The first sample text is used to describe the image domain and image category to which the first sample image belongs.

[0148] In this embodiment, the first sample text is used to indicate the image domain and image category to which the labeled second sample image belongs. This step is similar to feature extraction using a text feature extraction model in step S303, and will not be described in detail here.

[0149] In step S309, based on the similarity between the image features of the first sample image and the text features of the first sample text, the prompt word generation model to be trained is trained using the cross-entropy loss function to obtain the target prompt word generation model.

[0150] In this embodiment of the disclosure, the same cross-entropy loss function as in step S304 is used to train the prompt word generation model, thereby obtaining a target prompt word generation model for image retrieval, which will not be elaborated here.

[0151] For ease of description, the training process of the prompt word generation model described above can be found in [link to documentation]. Figure 5 As shown, Figure 5 This is a schematic diagram illustrating a second training stage according to an exemplary embodiment. 501 is the embedding vector of the input image, i.e., the embedding vector of the first sample image. 502 is a set of multiple updated reference neighborhood vectors. A first masking process is applied to 502, masking the reference neighborhood vector corresponding to the input image mentioned above (503), while retaining the remaining reference neighborhood vectors. 505 is a set of multiple updated reference category vectors. A first masking process is applied to 505, masking the reference category vector corresponding to the input image mentioned above (505), while retaining the remaining reference category vectors. 506 is the neighborhood vector template and category vector template, 507 is the parameter used in the image extraction process, and 508 is the sample neighborhood vector and sample category vector. As shown in the diagram, in the second training stage, a masking and contrasting target approach is used, based on the cross-entropy loss function, to train the prompt word generation model, resulting in the target prompt word generation model. Meanwhile, during the second phase of training, the domain vector template, category vector template, and other parameters of the auxiliary generation of sample domain prompts and sample category prompts can be updated, while keeping the text feature extraction model, image feature extraction model, and the updated reference domain vector and reference category vector unchanged.

[0152] In step S310, multiple reference neighborhood vectors, multiple reference category vectors, and the embedding vector of the target image are input into the target prompt word generation model for encoding to obtain the target neighborhood vector and the target category vector.

[0153] In this embodiment, the target domain vector refers to a word vector representing the image domain to which the target image belongs, and the target category vector refers to a word vector representing the image category to which the target image belongs. The embedding vector of the target image refers to a vector that reflects information such as the global features, local features, and semantic features of the target image. In the image retrieval stage, the target prompt word generation model, based on multiple updated reference domain vectors and multiple updated reference category vectors, combined with the embedding vector of the target image, obtains the target domain vector and the target category vector, thus achieving generalization from known domains and known categories to unknown domains and unknown categories.

[0154] For example, tp d This is called the target domain vector, and tp sThis is called the target category vector. The target domain vector and target category vector are generated using a target cue word generation model, as shown in the formula below.

[0155] [tp d ,tp s ] = f target ([PT d ,PT s ,DP,SP,E0])

[0156] Here, DP refers to the set of updated reference neighborhood vectors, and SP refers to the set of updated reference class vectors. E0 refers to the embedding vector of the target image, which is also the embedding vector of the input image. PT d This refers to the updated domain vector template; PT s This refers to the updated category vector template. Correspondingly, f target This refers to the target prompt word generation model. [PT] d ,PT s [DP,SP,E0] refers to the input sequence of the target prompt word generation model.

[0157] For ease of description of the image retrieval technology in the embodiments of this disclosure, see [link to relevant documentation]. Figure 6 As shown, Figure 6 This is a schematic diagram illustrating an image retrieval process according to an exemplary embodiment. Since only image features are needed for image retrieval in this embodiment, the text feature extraction model is discarded, and the prompt word generation model and image feature extraction model are retained. Specifically, 601 is the embedding vector of the input image, which is also the embedding vector of the target image. 602 is a set of multiple updated reference neighborhood vectors, 603 is a set of multiple updated reference category vectors, 604 is the updated neighborhood vector template and category vector template, and 605 is the parameter used in the image extraction process. 606 is the target neighborhood vector and target category vector generated by the target prompt word generation model. Based on the image features of the target image output by the image feature extraction model, a search is performed in the image feature library to determine the retrieval result. The image feature library stores the image features corresponding to images in the image database.

[0158] In step S311, the target neighborhood vector, the target category vector, and the embedding vector of the target image are input into the image feature extraction model for feature extraction to obtain the image features of the target image. The image features of the target image include the neighborhood features of the image neighborhood to which the target image belongs and the category features of the image category to which it belongs.

[0159] In this embodiment of the disclosure, using the target domain vector and the target category vector, feature extraction is performed using the same image feature extraction model as in step S302 to obtain image features of the target image that can reflect the domain features and category features, and these features are used to finally determine the retrieval results. This will not be elaborated further here.

[0160] For example, the image feature of the target image is called x, as shown in the following formula.

[0161] x = f image ([x0,tp d ,tp s ,E0])

[0162] Among them, tp d It refers to the target neighborhood vector, tp s This refers to the target category vector. x0 refers to the parameters used in the image feature extraction process. E0 refers to the embedding vector of the target image, which is also the embedding vector of the input image. Correspondingly, f image This refers to an image feature extraction model. [x0,tp d ,tp s [E0] refers to the input sequence of the image feature extraction model.

[0163] In step S312, based on the image features of the target image, the retrieval results are determined from the image database. The image database stores images from various image domains and images of various image types. The retrieval results include at least one image, and the similarity between the at least one image and the target image is greater than the similarity between other images in the image database and the target image.

[0164] In this embodiment of the disclosure, image features are extracted from both the target image and the images stored in the image database to obtain the image features of the target image and the image features of the stored images. The similarity between the image features of the target image and the image features of the stored images is determined, and then the multiple similarities are sorted to determine the retrieval results and feed them back to the user.

[0165] In some embodiments, search results are determined based on the similarity ranking between image features. Accordingly, the images in the image database are sorted in descending order of similarity between the image features of each image in the image database and the image features of the target image; the top N images are then selected as search results, where N is a positive integer. This method allows for the identification of a fixed number of images similar to the target image, even when the similarity between image features is low, facilitating comparison and review of search results by the user.

[0166] In some embodiments, search results are determined based on the relationship between the similarity between image features and a similarity threshold. Accordingly, for any image in the image database, the similarity between the image features of the image and the image features of the target image is determined; if the similarity is greater than the similarity threshold, the image is included in the search results. This method ensures that all images with a similarity greater than the similarity threshold are included in the search results, allowing users to obtain images that are similar to the target image and improving the user's search experience.

[0167] In some embodiments, the advantages of the image retrieval method proposed in this disclosure over traditional image retrieval methods are verified using the Domain Net dataset. Domain Net contains 596,006 images from six image domains (Real, Sketch, Quickdraw, Infograph, Clipart, and Painting) and 345 image categories, which are divided into training, validation, and test sets, consisting of 245, 55, and 45 image categories, respectively. The average precision and average accuracy (i.e., Prec@200 and mAP@200) of the top 200 retrieval results are used as evaluation metrics. During the training phase, data from five image domains are used for training, while one image domain is reserved as an unknown image domain for testing. Sample images from the unknown image domain and unknown image category are used as query images to retrieve images from the image database. Optionally, all image samples in the image database may come from unknown image categories, or all image samples in the image database may be a mixture of known and unknown image categories, the latter used to simulate more challenging image retrieval scenarios. During the experiment, APL was compared with two traditional methods (SnMpNet and SASA) and two CLIP-based methods (CLIP-Baseline and CLIP-Prompted).

[0168] Refer to Table 1 below, which shows the experimental results for two different image retrieval scenarios. First, APL significantly outperforms traditional methods, with an average mAP@200 improvement of 17.91% compared to the state-of-the-art SASA. This indicates that the image retrieval method of this disclosure has significant advantages in generalizing to unknown image domains and unknown image categories. Second, compared to CLIP-Prompted, APL's performance is further improved, indicating that the image retrieval method of this disclosure can further enhance CLIP's generalization ability.

[0169] Table 1. Experimental results for two different image retrieval scenarios.

[0170]

[0171] This disclosure provides an image retrieval method. It uses a target prompt word generation model to generate target neighborhood vectors and target category vectors based on the embedding vector of the input target image, representing the image neighborhood and category to which the target image belongs. Simultaneously, an image feature extraction model generates image features of the target image based on the input target neighborhood vector, target category vector, and the target image's embedding vector, and then determines the retrieval result based on these image features. When faced with target images of unknown neighborhoods and categories, this method uses the target prompt word generation model to obtain the target neighborhood vector and target category vector based on existing reference neighborhood vectors and reference category vectors, thus determining the neighborhood and category to which the target image belongs, guiding subsequent image feature extraction. Compared to traditional methods that directly extract image features from the target image, this method obtains image features including neighborhood and category features, which is more beneficial for subsequent image retrieval tasks and effectively improves the overall model's generalization performance.

[0172] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0173] Figure 7 This is a block diagram illustrating an image retrieval device according to an exemplary embodiment. Figure 7 As shown, the device includes: a first generation unit 701, a first extraction unit 702, and a first determination unit 703.

[0174] The first generation unit 701 is configured to input multiple reference neighborhood vectors, multiple reference category vectors, and the embedding vector of the target image into the target prompt word generation model for encoding, so as to obtain the target neighborhood vector and the target category vector. Different reference neighborhood vectors refer to word vectors used to represent different image neighborhoods, different reference category vectors refer to word vectors used to represent different image categories, the target neighborhood vector refers to word vectors used to represent the image neighborhood to which the target image belongs, and the target category vector refers to word vectors used to represent the image category to which the target image belongs.

[0175] The first extraction unit 702 is configured to input the target neighborhood vector, the target category vector, and the embedding vector of the target image into the image feature extraction model for feature extraction, thereby obtaining the image features of the target image. The image features of the target image include the neighborhood features of the image neighborhood to which the target image belongs and the category features of the image category to which it belongs.

[0176] The first determining unit 703 is configured to determine the retrieval result from the image database based on the image features of the target image. The image database stores images from multiple image domains and multiple image types. The retrieval result includes at least one image, and the similarity between the at least one image and the target image is greater than the similarity between other images in the image database and the target image.

[0177] In some embodiments, the first determining unit 703 is configured to sort the images in the image database according to the order of the similarity between the image features of each image in the image database and the image features of the target image in descending order; and determine the first N images as the retrieval results, where N is a positive integer.

[0178] In some embodiments, the first determining unit 703 is configured to, for any image in the image database, determine the similarity between the image features of the image and the image features of the target image; if the similarity is greater than a similarity threshold, the image is used as an image in the retrieval results.

[0179] In some embodiments, Figure 8 This is a block diagram illustrating another image retrieval device according to an exemplary embodiment, such as... Figure 8 As shown, the device also includes:

[0180] The second generation unit 801 is configured to input at least one reference neighborhood vector, at least one reference category vector, and the embedding vector of the first sample image into the prompt word generation model to be trained for encoding, so as to obtain a sample neighborhood vector and a sample category vector. The sample neighborhood vector refers to the word vector used to represent the image neighborhood to which the first sample image belongs, and the sample category vector refers to the word vector used to represent the image category to which the first sample image belongs.

[0181] The second extraction unit 802 is configured to input the sample neighborhood vector, the sample category vector, and the embedding vector of the first sample image into the image feature extraction model for feature extraction, so as to obtain the image features of the first sample image. The image features of the first sample image include the neighborhood features of the image neighborhood to which the first sample image belongs and the category features of the image category to which it belongs.

[0182] The third extraction unit 803 is configured to input the first sample text into the text feature extraction model for feature extraction, and obtain the text features of the first sample text. The first sample text is used to describe the image domain and image category to which the first sample image belongs.

[0183] Training unit 804 is configured to train the prompt word generation model to be trained using the cross-entropy loss function based on the similarity between the image features of the first sample image and the text features of the first sample text, thereby obtaining the target prompt word generation model.

[0184] In some embodiments, the apparatus further includes:

[0185] The second determining unit 805 is configured to determine the image region and image category to which the first sample image belongs;

[0186] The masking unit 806 is configured to perform a first masking process on multiple reference neighborhood vectors and multiple reference category vectors based on the image neighborhood and the image category, to obtain at least one reference neighborhood vector and at least one reference category vector. The first masking process is used to mask the reference neighborhood vector corresponding to the image neighborhood and the reference category vector corresponding to the image category.

[0187] In some embodiments, the masking unit 806 is configured to set a first type of neighborhood mask corresponding to an image neighborhood and a first type of category mask corresponding to an image category to 0, and set a first type of neighborhood mask corresponding to other image neighborhoods and a first type of category mask corresponding to other image categories to 1. The first type of neighborhood mask is used to mask a reference neighborhood vector corresponding to an image neighborhood, and the first type of category mask is used to mask a reference category vector corresponding to an image category. Based on the image neighborhood represented by each reference neighborhood vector, multiple reference neighborhood vectors are multiplied by their corresponding first type of neighborhood masks to obtain at least one reference neighborhood vector. Based on the image category represented by each reference category vector, multiple reference category vectors are multiplied by their corresponding first type of category masks to obtain at least one reference category vector.

[0188] In some embodiments, the apparatus further includes:

[0189] The second extraction unit 802 is further configured to input the reference neighborhood vector to be updated, the reference category vector to be updated, and the embedding vector of the second sample image into the image feature extraction model for feature extraction to obtain the image features of the second sample image. The reference neighborhood vector to be updated refers to the initial neighborhood vector corresponding to the second sample image, and the reference category vector to be updated refers to the initial category vector corresponding to the second sample image. The image features of the second sample image include the neighborhood features of the image neighborhood to which the second sample image belongs and the category features of the image category to which it belongs.

[0190] The third extraction unit 803 is also configured to input the second sample text into the text feature extraction model for feature extraction, and obtain the text features of the second sample text. The second sample text is used to describe the image domain and image category to which the second sample image belongs.

[0191] The update unit 807 is configured to update the reference neighborhood vector and the reference category vector to be updated using the cross-entropy loss function based on the similarity between the image features of the second sample image and the text features of the second sample text, so as to obtain the updated reference neighborhood vector and the updated reference category vector. The updated reference neighborhood vector represents the image neighborhood to which the second sample image belongs, and the updated reference category vector represents the image category to which the second sample image belongs.

[0192] In some embodiments, the apparatus further includes:

[0193] The second determining unit 805 is further configured to determine the image region and image category to which the second sample image belongs;

[0194] The masking unit 806 is further configured to perform a second masking process on multiple initial neighborhood vectors and multiple initial category vectors based on the image neighborhood and the image category to obtain a reference neighborhood vector to be updated and a reference category vector to be updated. The second masking process is used to retain the initial neighborhood vector corresponding to the image neighborhood and the initial category vector corresponding to the image category.

[0195] In some embodiments, the masking unit 806 is further configured to set the second type of neighborhood mask corresponding to the image neighborhood and the second type of category mask corresponding to the image category to 1, and set the second type of neighborhood mask corresponding to other image neighborhoods and the second type of category mask corresponding to other image categories to 0. The second type of neighborhood mask is used to retain the initial neighborhood vector corresponding to the image neighborhood, and the second type of category mask is used to retain the initial category vector corresponding to the image category. Based on the image neighborhood represented by each initial neighborhood vector, multiple initial neighborhood vectors are multiplied by their corresponding second type of neighborhood masks to obtain a reference neighborhood vector to be updated. Based on the image category represented by each initial category vector, multiple initial category vectors are multiplied by their corresponding second type of category masks to obtain a reference category vector to be updated.

[0196] This disclosure provides an image retrieval device. It uses a target prompt word generation model to generate target neighborhood vectors and target category vectors based on the embedding vector of an input target image, representing the image neighborhood and category to which the target image belongs. Simultaneously, an image feature extraction model generates image features of the target image based on the input target neighborhood vector, target category vector, and the target image's embedding vector, and then determines the retrieval result based on these image features. When faced with a target image of unknown neighborhood and category, the target prompt word generation model obtains the target neighborhood vector and target category vector based on existing reference neighborhood vectors and reference category vectors, thus determining the neighborhood and category to which the target image belongs, guiding subsequent image feature extraction. Compared to traditional methods that directly extract image features from the target image, this device obtains image features including neighborhood and category features, which is more beneficial for subsequent image retrieval tasks and effectively improves the overall model's generalization performance.

[0197] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0198] It should be noted that the image retrieval device provided in the above embodiments is only an example of the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the image retrieval device and the image retrieval method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0199] Regarding the image retrieval device in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0200] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Typically, the electronic device 900 includes a processor 901 and a memory 902.

[0201] Processor 901 may include one or more processing cores, such as a quad-core processor, a nine-core processor, etc. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0202] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one program code, which is executed by the processor 901 to implement the image retrieval method provided in the method embodiments of this disclosure.

[0203] In some embodiments, the electronic device 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, and a power supply 908.

[0204] Peripheral device interface 903 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 901 and memory 902. In some embodiments, processor 901, memory 902 and peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 901, memory 902 and peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0205] The radio frequency (RF) circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 904 can communicate with other electronic devices through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to NFC (Near Field Communication), which is not limited in this disclosure.

[0206] Display screen 905 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 905 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 901 for processing. In this case, display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 905, which serves as the front panel of electronic device 900; in other embodiments, there may be at least two display screens 905, respectively disposed on different surfaces of electronic device 900 or in a folded design; in still other embodiments, display screen 905 may be a flexible display screen, disposed on a curved or folded surface of electronic device 900. Furthermore, display screen 905 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 905 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0207] The camera assembly 906 is used to acquire images or videos. Optionally, the camera assembly 906 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device, and the rear-facing camera is located on the back of the electronic device. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0208] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 901 for processing, or input to the radio frequency circuit 904 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located in a different part of the electronic device 900. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0209] Power supply 908 is used to supply power to the various components in electronic device 900. Power supply 908 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 908 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0210] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on the electronic device 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0211] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 902 including instructions, which can be executed by a processor 901 of a terminal 900 to complete the image retrieval method described above. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0212] A computer program product includes a computer program that, when executed by a processor, implements the image retrieval method described above.

[0213] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0214] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image retrieval method, characterized in that, The method includes: Multiple reference domain vectors, multiple reference category vectors, and the embedding vector of the target image are input into the target prompt word generation model for encoding to obtain target domain vectors and target category vectors. Different reference domain vectors refer to word vectors used to represent different image domains, and different reference category vectors refer to word vectors used to represent different image categories. The target domain vector refers to the word vector used to represent the image domain to which the target image belongs, and the target category vector refers to the word vector used to represent the image category to which the target image belongs. The target neighborhood vector, the target category vector, and the embedding vector of the target image are input into an image feature extraction model for feature extraction to obtain the image features of the target image. The image features of the target image include the neighborhood features of the image neighborhood to which the target image belongs and the category features of the image category to which it belongs. Based on the image features of the target image, a retrieval result is determined from an image database. The image database stores images from multiple image domains and multiple image types. The retrieval result includes at least one image, and the similarity between the at least one image and the target image is greater than the similarity between other images in the image database and the target image. The training method for the target prompt word generation model includes: Based on the image neighborhood and image category to which the first sample image belongs, a first masking process is performed on the plurality of reference neighborhood vectors and the plurality of reference category vectors to obtain at least one reference neighborhood vector and at least one reference category vector. The first masking process is used to mask the reference neighborhood vector corresponding to the image neighborhood and the reference category vector corresponding to the image category. Based on the at least one reference neighborhood vector, the at least one reference category vector, the embedding vector of the first sample image, and the first sample text, the prompt word generation model to be trained is trained to obtain the target prompt word generation model. The first sample text is used to describe the image neighborhood and image category to which the first sample image belongs.

2. The image retrieval method according to claim 1, characterized in that, The step of determining the retrieval results from the image database based on the image features of the target image includes: The images in the image database are sorted in descending order of similarity between the image features of each image in the image database and the image features of the target image; The first N images are determined as the search results, where N is a positive integer.

3. The image retrieval method according to claim 1, characterized in that, The step of determining the retrieval results from the image database based on the image features of the target image includes: For any image in the image database, determine the similarity between the image features of the image and the image features of the target image; If the similarity is greater than the similarity threshold, the image is included in the search results.

4. The image retrieval method according to claim 1, characterized in that, The step of training the prompt word generation model based on the at least one reference neighborhood vector, the at least one reference category vector, the embedding vector of the first sample image, and the first sample text to obtain the target prompt word generation model includes: The at least one reference neighborhood vector, the at least one reference category vector, and the embedding vector of the first sample image are input into the prompt word generation model to be trained for encoding to obtain a sample neighborhood vector and a sample category vector. The sample neighborhood vector refers to the word vector used to represent the image neighborhood to which the first sample image belongs, and the sample category vector refers to the word vector used to represent the image category to which the first sample image belongs. The sample neighborhood vector, the sample category vector, and the embedding vector of the first sample image are input into the image feature extraction model for feature extraction to obtain the image features of the first sample image. The image features of the first sample image include the neighborhood features of the image neighborhood to which the first sample image belongs and the category features of the image category to which it belongs. The first sample text is input into a text feature extraction model for feature extraction to obtain the text features of the first sample text; Based on the similarity between the image features of the first sample image and the text features of the first sample text, the prompt word generation model to be trained is trained using the cross-entropy loss function to obtain the target prompt word generation model.

5. The image retrieval method according to claim 4, characterized in that, The first masking process, based on the image neighborhood and image category to which the first sample image belongs, is performed on the plurality of reference neighborhood vectors and the plurality of reference category vectors to obtain the at least one reference neighborhood vector and the at least one reference category vector, including: Set the first type of neighborhood mask corresponding to the image neighborhood and the first type of category mask corresponding to the image category to 0, and set the first type of neighborhood mask corresponding to other image neighborhoods and the first type of category mask corresponding to other image categories to 1. The first type of neighborhood mask is used to mask the reference neighborhood vector corresponding to the image neighborhood, and the first type of category mask is used to mask the reference category vector corresponding to the image category. Based on the image region represented by each reference neighborhood vector, the plurality of reference neighborhood vectors are multiplied by the corresponding first type of neighborhood mask to obtain the at least one reference neighborhood vector; Based on the image category represented by each reference category vector, the multiple reference category vectors are multiplied by the corresponding first category mask to obtain the at least one reference category vector.

6. The image retrieval method according to claim 4, characterized in that, The method further includes: The reference neighborhood vector to be updated, the reference category vector to be updated, and the embedding vector of the second sample image are input into the image feature extraction model for feature extraction to obtain the image features of the second sample image. The reference neighborhood vector to be updated refers to the initial neighborhood vector corresponding to the second sample image, and the reference category vector to be updated refers to the initial category vector corresponding to the second sample image. The image features of the second sample image include the neighborhood features of the image neighborhood to which the second sample image belongs and the category features of the image category to which it belongs. The second sample text is input into the text feature extraction model for feature extraction to obtain the text features of the second sample text. The second sample text is used to describe the image domain and image category to which the second sample image belongs. Based on the similarity between the image features of the second sample image and the text features of the second sample text, the reference neighborhood vector to be updated and the reference category vector to be updated are updated using the cross-entropy loss function to obtain the updated reference neighborhood vector and the updated reference category vector. The updated reference neighborhood vector represents the image neighborhood to which the second sample image belongs, and the updated reference category vector represents the image category to which the second sample image belongs.

7. The image retrieval method according to claim 6, characterized in that, The method further includes: Determine the image region and image category to which the second sample image belongs; Based on the image neighborhood and the image category, a second masking process is performed on multiple initial neighborhood vectors and multiple initial category vectors to obtain the reference neighborhood vector to be updated and the reference category vector to be updated. The second masking process is used to retain the initial neighborhood vector corresponding to the image neighborhood and the initial category vector corresponding to the image category.

8. The image retrieval method according to claim 7, characterized in that, The step of performing a second masking process on multiple initial neighborhood vectors and multiple initial category vectors based on the image neighborhood and the image category to obtain the reference neighborhood vector to be updated and the reference category vector to be updated includes: Set the second type of neighborhood mask corresponding to the image neighborhood and the second type of category mask corresponding to the image category to 1, and set the second type of neighborhood mask corresponding to other image neighborhoods and the second type of category mask corresponding to other image categories to 0. The second type of neighborhood mask is used to retain the initial neighborhood vector corresponding to the image neighborhood, and the second type of category mask is used to retain the initial category vector corresponding to the image category. Based on the image region represented by each initial neighborhood vector, the multiple initial neighborhood vectors are multiplied by the corresponding second-type neighborhood mask to obtain the reference neighborhood vector to be updated; Based on the image category represented by each initial category vector, the multiple initial category vectors are multiplied by the corresponding second category mask to obtain the reference category vector to be updated.

9. An image retrieval device, characterized in that, The device includes: The first generation unit is configured to input multiple reference neighborhood vectors, multiple reference category vectors, and the embedding vector of the target image into the target prompt word generation model for encoding, thereby obtaining target neighborhood vectors and target category vectors. Different reference neighborhood vectors refer to word vectors used to represent different image neighborhoods, and different reference category vectors refer to word vectors used to represent different image categories. The target neighborhood vector refers to word vectors used to represent the image neighborhood to which the target image belongs, and the target category vector refers to word vectors used to represent the image category to which the target image belongs. The first extraction unit is configured to input the target neighborhood vector, the target category vector, and the embedding vector of the target image into an image feature extraction model to extract features and obtain the image features of the target image. The image features of the target image include the neighborhood features of the image neighborhood to which the target image belongs and the category features of the image category to which it belongs. The first determining unit is configured to determine a retrieval result from an image database based on the image features of the target image. The image database stores images from multiple image domains and multiple image types. The retrieval result includes at least one image, and the similarity between the at least one image and the target image is greater than the similarity between other images in the image database and the target image. The training method for the target prompt word generation model includes: based on the image domain and image category to which the first sample image belongs, performing a first masking process on the plurality of reference domain vectors and the plurality of reference category vectors to obtain at least one reference domain vector and at least one reference category vector, wherein the first masking process is used to mask the reference domain vector corresponding to the image domain and the reference category vector corresponding to the image category; and training the prompt word generation model to be trained based on the at least one reference domain vector, the at least one reference category vector, the embedding vector of the first sample image, and the first sample text to obtain the target prompt word generation model, wherein the first sample text is used to describe the image domain and image category to which the first sample image belongs.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the image retrieval method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the image retrieval method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the image retrieval method as described in any one of claims 1 to 8.

Images