Method executed by electronic device, electronic device, storage medium and program product

Abstract

The embodiment of the invention provides a method executed by electronic equipment, the electronic equipment, a storage medium and a program product, and relates to the technical fields of artificial intelligence, deep learning, image retrieval and the like. The method comprises the following steps: extracting a first text feature from a first text input by a user; extracting a first semantic feature of a first region related to the first text from a first image input by a user; generating a second semantic feature based on the first text feature and the first semantic feature; and performing image retrieval in the candidate image set on the basis of the second semantic feature and the first texture feature of the first region, so that the retrieval result can better conform to the actual intention of the user. Optionally, the method performed by the electronic device may be performed using an artificial intelligence model.

Description

Technical Field

[0001] This disclosure relates to the field of image retrieval technology, and more specifically, to a method executed by an electronic device, an electronic device, a storage medium, and a program product. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, deep learning-based image retrieval technology has attracted increasing attention, especially the use of multimodal image retrieval, which has brought great convenience to people. Multimodal image retrieval can take descriptive text and reference images as input, aiming to find the best-matching target image from a candidate image set using multimodal understanding techniques.

[0003] However, existing technologies are lacking in accuracy in retrieval. Summary of the Invention

[0004] The purpose of this disclosure is to address the problem of how to improve the accuracy of retrieval.

[0005] According to one aspect of the present disclosure, a method performed by an electronic device is provided, the method comprising:

[0006] Extract the first text features from the first text entered by the user;

[0007] Extract the first semantic features of the first region related to the first text from the first image input by the user;

[0008] Based on the first text features and the first semantic features, generate the second semantic features;

[0009] Image retrieval is performed in the candidate image set based on the second semantic features and the first texture features of the first region.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method performed by the electronic device provided in the embodiments of the present disclosure.

[0011] According to another aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method performed by an electronic device provided in the embodiments of the present disclosure.

[0012] According to another aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method performed by an electronic device provided in the embodiments of the present disclosure. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below.

[0014] Figure 1 A flowchart illustrating a method executed by an electronic device according to an embodiment of this disclosure;

[0015] Figure 2 A schematic diagram illustrating a multimodal image retrieval method provided in an embodiment of this disclosure;

[0016] Figure 3 A schematic diagram illustrating a method for predicting two search instructions provided in an embodiment of this disclosure;

[0017] Figure 4 A schematic diagram illustrating a method for predicting a first texture feature according to an embodiment of this disclosure;

[0018] Figure 5 A schematic diagram illustrating a method for predicting a first semantic feature and a second heatmap provided in an embodiment of this disclosure;

[0019] Figure 6 A schematic diagram illustrating the execution process of a multi-head attention network according to an embodiment of this disclosure;

[0020] Figure 7 A schematic diagram illustrating a text feature extraction method provided in an embodiment of this disclosure;

[0021] Figure 8 A schematic diagram illustrating a method for extracting text query features according to an embodiment of this disclosure;

[0022] Figure 9 A schematic diagram illustrating a method for optimizing image feature words according to an embodiment of this disclosure;

[0023] Figure 10 A schematic diagram illustrating an image feature word tuple optimization method provided in this embodiment of the disclosure;

[0024] Figure 11 A schematic diagram of a process for generating visual query features provided in an embodiment of this disclosure;

[0025] Figure 12 A schematic diagram illustrating a visual query feature visualization based on a retrieval command, provided as an embodiment of this disclosure;

[0026] Figure 13 A schematic diagram of an image retrieval process provided in an embodiment of this disclosure;

[0027] Figure 14 This is a schematic diagram illustrating an application scenario of the image retrieval method provided in this embodiment of the disclosure;

[0028] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0029] The following description, with reference to the accompanying drawings, is provided to aid in a thorough understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. This description includes various specific details to aid understanding but should be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0030] The terms and wording used in the following description and claims are not limited to their dictionary meanings, but are merely used by the inventors to enable a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of this disclosure is for illustrative purposes only and not for limiting the purpose of this disclosure as defined in the appended claims and their equivalents.

[0031] It should be understood that the singular forms of “a,” “an,” and “the” can also include plural references unless the context clearly indicates otherwise. Thus, for example, the reference to “component surface” includes referring to one or more such surfaces. When we say that an element is “connected” or “coupled” to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element are connected through an intermediate element. Furthermore, the use of “connected” or “coupled” herein can include wireless connections or wireless couplings.

[0032] The terms “comprising” or “may include” refer to the presence of a corresponding disclosed function, operation, or component that may be used in the various embodiments of this disclosure, rather than limiting the presence of one or more additional functions, operations, or features. Furthermore, the terms “comprising” or “having” may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof, but should not be construed as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof.

[0033] The term "or" as used in the various embodiments of this disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items may refer to one, more, or all of the multiple items. For example, the description "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2, and A3.

[0034] Unless otherwise defined, all terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by those skilled in the art. Common terms as defined in dictionaries are to be interpreted as having a meaning consistent with their context in the relevant technical field and should not be interpreted ideally or overly formally, unless expressly defined in this disclosure.

[0035] At least some of the functions of the device or electronic device provided in this disclosure embodiment can be implemented by an AI model, such as implementing at least one module of a plurality of modules of the device or electronic device by an AI model. AI-related functions can be executed by non-volatile memory, volatile memory, and a processor.

[0036] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as central processing unit (CPU), application processor (AP), etc., or pure graphics processing unit, such as graphics processing unit (GPU), vision processing unit (VPU), and / or AI-specific processors, such as neural processing unit (NPU).

[0037] The one or more processors control the processing of input data based on predefined operating rules or artificial intelligence (AI) models stored in non-volatile and volatile memory. These predefined operating rules or AI models are provided through training or learning.

[0038] Here, "providing through learning" refers to obtaining predefined operating rules or an AI model with desired characteristics by applying a learning algorithm to multiple learning datasets. This learning can be performed within the device or electronic device itself, in which the AI ​​is executed according to the embodiment, and / or can be implemented via a separate server / system.

[0039] AI models can contain multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network computations by calculating the input data of that layer (such as the computation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks.

[0040] A learning algorithm is a method of training a predetermined target device (e.g., a robot) using multiple learning data sets to enable, allow, or control the target device to make determinations or predictions. Examples of such learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0041] The methods provided in this disclosure may relate to one or more fields in the technical fields of speech, language, image, video, or data intelligence.

[0042] Optionally, in the context of speech or language, according to this disclosure, in a method performed by an electronic device, a method for recognizing a user's speech and interpreting the user's intent may receive a speech signal as an analog signal via an acquisition device (e.g., a microphone) and convert the speech portion into computer-readable text using an Automatic Speech Recognition (ASR) model. The user's utterance intent can be obtained by interpreting the converted text using a Natural Language Understanding (NLU) model. The ASR model or NLU model can be an artificial intelligence (AI) model. The AI ​​model can be processed by a dedicated AI processor designed in a hardware architecture specified for processing the AI ​​model. The AI ​​model can be acquired through training. Here, "acquired through training" means obtaining a predefined operating rule or AI model configured to perform desired features (or purposes) by training a basic AI model with multiple training data using a training algorithm. Language understanding is a technique for recognizing and applying / processing human language / text, including, for example, natural language processing, machine translation, dialogue systems, question answering, or speech recognition / synthesis.

[0043] Optionally, in the context of images or videos, according to this disclosure, in a method executed in an electronic device, the method for image retrieval can obtain output data that identifies images or features, regions, instructions, terms, etc., within an image by using image data as input data for an artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operating rule or artificial intelligence model configured to perform desired features (or purposes) by training a basic artificial intelligence model with multiple training data using a training algorithm. The methods of this disclosure can relate to the field of visual understanding in artificial intelligence technology, which is a technology for recognizing and processing things like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, or image enhancement.

[0044] Optionally, in the context of data intelligence processing, according to this disclosure, in methods executed in electronic devices, methods for reasoning or predicting features, regions, instructions, or terms can use artificial intelligence models to recommend / execute by using image data and / or text data. The processor of the electronic device can perform preprocessing operations on the data to transform it into a form suitable for use as input to an artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operating rule or artificial intelligence model configured to execute desired features (or objectives) by training a basic artificial intelligence model with multiple training data using a training algorithm. Reasoning and prediction are techniques for logical reasoning and prediction based on determined information, including, for example, knowledge-based reasoning, optimization prediction, preference-based planning, or recommendation.

[0045] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0046] The inventors of this disclosure, combining theoretical and practical findings, have discovered that existing technologies are lacking in retrieval accuracy. This is because existing image retrieval methods simply extract high-dimensional semantic features from reference images (e.g., "dog"). Similar objects or objects of the same category often have similar high-dimensional semantic features. Therefore, relying solely on such features makes it difficult to distinguish or locate specific objects. For example, it is difficult to accurately search for specific people, animals, or objects (e.g., searching for "photos of puppy A" might return "photos of puppy B"). Furthermore, existing image retrieval methods feed the entire reference image input by the user into a visual encoder. The visual encoder can extract global semantic features from the entire image, focusing on salient global semantic features (e.g., "person," "bag," "sea," "skirt"), failing to focus on the area of ​​interest to the user (e.g., "skirt"). This can lead to interference from irrelevant features from other areas, causing the image retrieval results to not accurately match the user's actual intent.

[0047] To address at least one of the aforementioned technical problems or areas requiring improvement in related technologies, this disclosure proposes a method, electronic device, storage medium, and program product executed by an electronic device. This solution can also be referred to as an image retrieval method or a multimodal image retrieval method. Based on first text and a first image input by the user, the method "imagines" the key features of the target image that the user wants to find. Based on these "imagineed" feature expressions, the target image desired by the user can be accurately and quickly located within a candidate image set.

[0048] The following description of several optional embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0049] This disclosure provides a method executed by an electronic device, such as... Figure 1 As shown, the method includes:

[0050] Step S101: Extract the first text features from the first text input by the user.

[0051] Optionally, the first text may also be referred to as the retrieval text, used to find or locate target images containing described content in a large number of images using natural language. Optionally, the first text may be generated based on text directly input by the user, or based on the user's selection of displayed text options, or based on speech conversion of user input, etc. This embodiment of the disclosure does not specifically limit the input method of the first text.

[0052] Optionally, text features can be extracted using a text encoder or other methods, such as combining the semantics of the first image or the first region for text feature extraction. This embodiment of the present disclosure does not impose any limitations on these methods. The extracted text features may also be referred to as text feature terms or text query features.

[0053] Step S102: Extract the first semantic features of the first region related to the first text from the first image input by the user.

[0054] Optionally, the first image may also be referred to as a reference image, which can provide additional visual information for reference in image retrieval, thereby improving the accuracy of image retrieval. Optionally, the first image may be taken by the user in real time, selected by the user from the album, or received by the user from the network or other devices, etc. The embodiments of this disclosure do not specifically limit the input method and source of the first image.

[0055] Given the explosive growth of user photos, it has become extremely difficult for users to find the images they want using a single input (such as text). For example, if a user wants to search for "photos of me traveling to the beach with my favorite bag," there may be many corresponding "photos including bags," making it difficult to determine which one is "the photo of my favorite bag." In this embodiment of the disclosure, this difficulty can be addressed through multimodal image retrieval. Users can provide a first image and a first text for image retrieval. For example, a user can provide "photos including my favorite bag" as the first image to search for images corresponding to the first text "photos of me traveling to the beach with my favorite bag," thus directly finding the desired image.

[0056] In this embodiment of the disclosure, the first region in the first image related to the first text refers to the object or region in the first image that the user is concerned with or interested in, or the object or region that the user hopes or wants to refer to, which can be determined based on the user's intent corresponding to the first text. It is understood that the first region and the first object can have similar meanings. As an example, if the first text entered by the user is "wearing this long dress on the grass" and a first image is provided including "a photo taken at the beach while wearing a long dress, carrying a bag, and holding a dog", then the first region in the first image related to the first text refers to the "long dress region" (or the object "long dress").

[0057] In this embodiment of the disclosure, the first region can be indicated in various ways. For example, it can be indicated by features, segmentation maps, weight maps, heat maps, masks, text descriptions, related instructions, etc. This embodiment of the disclosure does not limit the scope of the indication.

[0058] Optionally, the first semantic feature can refer to a high-dimensional feature, which can characterize the semantic information of the object in the first region. Therefore, high-dimensional features can also be called high-dimensional semantic features or high-dimensional visual semantic features. High-dimensional features can go through more network layers than low-dimensional features. Optionally, the first semantic feature can be extracted using a visual decoder, or extracted using a new high-dimensional feature extraction network, etc., and this embodiment of the present disclosure is not limited thereto.

[0059] In this embodiment of the disclosure, based on the user's intent, only the region of interest to the user (such as the first region) can be extracted, and the first semantic features of the first region can be focused on, without the need for the global semantic features of the entire first image.

[0060] Step S103: Generate second semantic features based on the first text features and the first semantic features.

[0061] In this embodiment of the disclosure, the second semantic feature can be understood as a visual query feature that can express the target image, "imagined" according to the user's intent. Generated from the first text feature and the first semantic feature, this feature accurately describes the region the user wants to refer to, without interference from other irrelevant region features, thus generating a more accurate feature.

[0062] Step S104: Based on the second semantic features and the first texture features of the first region, perform image retrieval in the candidate image set.

[0063] Optionally, the first texture feature can refer to a low-dimensional feature. Low-dimensional features can characterize the texture information of the object in the first region, thus better representing the object's details. Texture information can reflect the uneven grooves on the object's surface, such as boundaries, contours, and patterns. Therefore, low-dimensional features can also be called low-dimensional texture features or low-dimensional visual texture features. Optionally, the extraction of the first texture feature can utilize several layers preceding the visual encoder, or employ a novel low-dimensional feature extraction network, etc. This embodiment of the present disclosure does not impose any limitations on this method.

[0064] In this embodiment of the disclosure, both the first texture feature and the second semantic feature can be understood as visual query features that can express the target image, "imagined" according to the user's intention. Their combination can be called a composite visual query feature group. For example Figure 2 As shown, taking an input image containing a puppy as an example, and the input text as "in pool," the feature set used for image retrieval contains two parts: one is the second semantic feature (e.g., high-dimensional semantic feature) that can characterize the target image, such as... Figure 2 The feature for "dog in pool" is generated from text features and first semantic features. This feature accurately describes the region the user wants to reference, without interference from other irrelevant region features, thus generating a more accurate feature. Another feature is the first texture feature (e.g., low-dimensional texture feature) of a portion of the first image (e.g., a first region) retained to represent a specific object in the target image. Figure 2 The features of "this dog" (or "this dog"), such as its short white fur, retain the texture information of the first region, accurately and meticulously depicting the object in the user's mind, making the generated features more accurate. This composite visual query feature set is not generated from the actual RGB pixels of the observed image, but rather a feature representation of the desired target image, capable of accurately and meticulously depicting the target image in the user's mind. Image retrieval based on this composite visual query feature set can accurately and quickly locate the user's desired target image in the candidate image set.

[0065] In this embodiment of the disclosure, image retrieval can also be called image search or image query. The candidate image set is the image set of the image to be retrieved by the user, such as a photo album, e-commerce platform, or web search platform. The user can quickly and accurately find the target image they want from hundreds or thousands of images in the candidate image set by giving a first image and a first text and through the above processing.

[0066] Optionally, the features of the images in the candidate image set can be compared with the distances between these features, and the candidate image corresponding to the feature with the closest distance can be used as the retrieval result. Other query methods can also be used, and this disclosure does not limit them.

[0067] The method performed by an electronic device according to embodiments of this disclosure generates second semantic features by "imagining" key features of a target image that a user wants to find. Based on these "imagined" feature expressions, the target image desired by the user in the candidate image set can be accurately and quickly located. Furthermore, in order to express a specific object in the first image, the texture information of the first region or its similar texture information is preserved, which can accurately and meticulously depict the object in the user's mind, making the generated retrieval features more accurate, thereby making the retrieval results more consistent with the user's actual intention.

[0068] In this embodiment of the disclosure, a second text feature corresponding to a first region and / or a third text feature corresponding to a target image can be determined based on a first image and a first text. The second text feature is used to determine a first semantic feature and / or a first texture feature, and the third text feature is used to determine a third texture feature (which will be described below).

[0069] Considering that the first text entered by the user is often concise and focuses on keywords, it is sometimes not easy to obtain the user's actual intent from the first text alone. It is often necessary to combine it with the first image to infer the intent. In this embodiment of the disclosure, the user's actual search intent is predicted by fusing multimodal information from the first image and the first text, and further decoupled into a second text feature corresponding to a first region for "imagining" visual query features and / or a third text feature corresponding to the target image.

[0070] Optionally, a first search instruction can be used to characterize a second text feature corresponding to a first region, describing which regions need to be retained from the perspective of text features. Similarly, a second search instruction can be used to characterize a third text feature corresponding to a target image, describing which regions and the first text constitute the target image from the perspective of text features. For example, the user's actual search intent can be decoupled into a first search instruction and / or a second search instruction for "imagining" visual query features.

[0071] The first retrieval instruction can also be understood as a feature preservation instruction, which is used to characterize which regions’ features need to be preserved (such as “this dog” in the example above). Based on the guidance of the first retrieval instruction, objects in specific regions (such as the first region) in the first image can be preserved unchanged, so that the low-dimensional texture features of the object can be extracted into the first texture features, such as the features of “this dog” in the example above, so as to retrieve target images that include the object.

[0072] The second retrieval instruction can also be understood as a feature planning instruction, used to characterize the features that need to be planned (such as the features "dog" and "in the pool" in the example above, which can be enhanced, while other features can be weakened). Based on the guidance of the second retrieval instruction, high-dimensional semantic features of the target image to be searched can be planned, such as the features of "dog in the pool" in the example above, in order to retrieve target images with similar semantics.

[0073] In this embodiment of the disclosure, an optional implementation method is provided for the step of "determining the second text features corresponding to the first region and / or the third text features corresponding to the target image based on the first image and the first text". Specifically, it may include:

[0074] Step S201: Determine the fourth semantic feature corresponding to the first image and at least one image feature word corresponding to the first image, where different image feature words represent features of different regions in the first image;

[0075] Among them, a word can also be called a feature vector. Different image feature words represent the feature vectors of different regions in the first image.

[0076] Optionally, the first image and its semantic segmentation map can be used as input to the visual encoder of a visual language model (e.g., the ViT-B (Vision Transformer Base resolution) / 32 structure, 512 channels, etc. of the CLIP (Contrastive Language-Image Pre-training) model, but not limited to this), outputting a fourth semantic feature and at least one image feature lexical unit. The fourth semantic feature can also be understood as a global image feature, representing the salient features of the entire image, such as the features related to "a dog in the grass" in the example above. At least one image feature lexical unit can also be called an image feature tuple, where each lexical unit represents the features of a different region, corresponding to different objects in the segmentation map, such as "dog" and "grass" in the example above. The image feature tuple can also be understood as expressing the high-dimensional semantic features of different regions in the first image.

[0077] Step S202: Determine at least one text feature word corresponding to the first text;

[0078] Optionally, the first text is input into the text encoder of the visual language model (e.g., the ViT-B / 32 structure of the CLIP model, 512 channels, etc., but not limited thereto) to obtain text feature lexical units.

[0079] Step S203: Determine the second text feature and / or the third text feature based on at least one text feature word, the fourth semantic feature, and at least one image feature word.

[0080] In this embodiment of the disclosure, based on the text feature words extracted from the first text and the fourth semantic feature and image feature word group extracted from the first image, the user's actual intent is obtained, the intent is decoupled, and the second text feature and / or the third text feature are obtained.

[0081] In this embodiment of the disclosure, an optional implementation method is provided for step S203, which may specifically include:

[0082] Step S2031: Based on at least one text feature word and a fourth semantic feature, a fourth cross-attention network is used to obtain a first feature, which includes information related to the user's intent;

[0083] In this embodiment of the disclosure, cross-attention is calculated based on at least one text feature word and a fourth semantic feature. The purpose is to interact significant visual information (such as the features "dog" and "grass" in the example above) with the information in the first text to obtain the user's most likely intent feature (such as "dog in the pool" in the example above). Optionally, features that are more likely to match the user's intent correspond to higher weights.

[0084] Step S2032: Based on the first feature and at least one image feature word, use the fifth cross-attention network to obtain the second feature and the third feature. The second feature represents the feature in at least one image feature word that is similar to the first feature, and the third feature represents the feature in at least one image feature word that is not similar to the first feature.

[0085] In this embodiment, the first feature is further cross-attention calculated with the image feature tuple. The purpose is to compare the user intent feature with the visual features of different regions in the first image, estimating which visual features are likely similar to the user intent (e.g., "dog"), which can also be understood as which image feature tuples are highly likely to be similar to the features in the first feature; and which are likely dissimilar to the user intent (e.g., "grass"), which can also be understood as which image feature tuples are highly likely to be dissimilar to all features in the first feature. Similar second features are used to preserve or enhance, while dissimilar third features can be used to weaken.

[0086] Step S2033: Based on the second feature and the third feature, determine the second text feature and / or the third text feature.

[0087] Optionally, based on the second feature, a first multilayer perceptron (MLP) is used to decouple the second text feature and / or the third text feature.

[0088] In this embodiment of the disclosure, an optional implementation method for step S2031 is provided, which specifically may include:

[0089] Step S301: Obtain a first query vector for generating the second text feature and / or a second query vector for generating the third text feature;

[0090] In this embodiment of the disclosure, instruction query features that can be learned in the network are set to represent the second text features corresponding to the first region and / or the third text features corresponding to the target image. For example, two instruction query vectors are set: query vector 1: keep "XXX" and query vector 2: plan "XXX", but not limited to this.

[0091] Optionally, the set instruction query features have the same dimension as the text feature words. For example, at least one of the dimensions of the first query vector and the second query vector is the same as the dimension of at least one text feature word.

[0092] Step S302: Based on at least one of the first query vector and the second query vector, and at least one text feature word, use a self-attention network to obtain text fusion features;

[0093] In this embodiment of the disclosure, text feature words and one or two query vectors are combined and fed into a self-attention network for text information fusion to obtain text fusion features.

[0094] Step S303: Based on the text fusion features and the fourth semantic features, the first feature is obtained using the fourth cross-attention network.

[0095] In this embodiment of the disclosure, cross-attention is calculated based on text fusion features and fourth semantic features to interact with significant visual information and text fusion features to obtain the user's most likely intent features.

[0096] Based on at least one of the above embodiments, in this embodiment of the disclosure, by Figure 3 This diagram illustrates a method for predicting two search instructions, which may specifically include:

[0097] 1) The reference image (the first image, such as a photo of "a dog in the grass") and its semantic segmentation map are used as input to the visual encoder of the visual language model. The output is the global image features (fourth semantic features, including the salient features of the whole image, such as the features of "a dog in the grass") and image feature tuples (at least one image feature tuple, each image feature tuple corresponding to the features of the region where an object is located in the segmentation semantic map, such as "dog", "grass", etc.). It can also output a lexical heatmap group (at least one heatmap, each heatmap corresponding to one image feature tuple in the image feature tuple, reflecting the response intensity of the image feature tuple in the reference image. The usage of the lexical heatmap group will be introduced below).

[0098] 2) Input the retrieved text (first text, such as “in the pool”) into the text encoder of the visual language model to obtain at least one text feature lexical.

[0099] 3) Estimating the user's actual intent through multimodal interaction of textual and visual information. Specifically, two instruction query vectors (with the same dimension as text feature words) that can be learned in the network are defined to represent feature preservation instructions (second text features corresponding to the first region) and feature planning instructions (third text features corresponding to the target image). The text feature words and these two query vectors are combined and fed into a self-attention network for text information fusion. Then, the text fusion features are cross-attentioned with the global image features, and significant visual information (such as the feature of "dog") interacts with the text fusion features to obtain the most likely user intent feature (first feature), such as "dog in the pool". Next, the user intent feature is further cross-attentioned with the image feature word group, comparing the user intent feature with the visual features of different regions, estimating features similar to the user intent feature (second feature, such as "dog") and dissimilar features (third feature, such as "grass") in the visual features, for planning, such as retaining similar features and weakening dissimilar features. Finally, the user intent is decoupled into feature preservation instructions and feature planning instructions through a multilayer perceptron. Feature retention instructions indicate which features of a region need to be retained (e.g., "this dog"), while feature planning instructions indicate which features need to be planned (e.g., "dog", "in the pool").

[0100] In this embodiment of the disclosure, the first texture feature can also be extracted in the first region in the following way:

[0101] Step S401: Determine at least one first heatmap corresponding to the first image. Different first heatmaps focus on different regions in the first image, and different regions correspond to different image feature words.

[0102] Optionally, the first image and its semantic segmentation map can be used as input to the visual encoder of a visual language model (e.g., the ViT-B / 32 structure of the CLIP model, 512 channels, etc., but not limited to this), outputting at least one first heatmap, which can be output together with the fourth semantic feature and at least one image feature lexical. The at least one first heatmap can also be called a lexical heatmap group, where each first heatmap focuses on different regions in the first image, corresponding to different objects in the segmentation map, or each first heatmap can correspond to an image feature lexical in the image feature lexical group, reflecting the response intensity of that image feature lexical in the reference image. For example, the extraction method of at least one first heatmap can be as follows: Figure 3 As shown.

[0103] In this embodiment of the disclosure, the heat map can also be replaced with information from other different regions, such as weight maps, mask maps, etc. Those skilled in the art can extend it according to the actual situation, and it should also be included within the protection scope of this disclosure.

[0104] Step S402: Based on the second text features corresponding to the first region and at least one first heatmap, a second heatmap is obtained, and the second heatmap focuses on the first region;

[0105] Since the second text feature can indicate the first region where the feature needs to be retained, the heat map of the two-dimensional image layer corresponding to the first region can be estimated based on the second text feature, that is, the second heat map, which can indicate which regions in the first image need to be retained. Therefore, the second heat map can also be called the image retention heat map.

[0106] Step S403: Extract the second texture features of the first image;

[0107] Optionally, the second texture feature may refer to the global low-dimensional features of the first image. For example, the second texture feature may be extracted from the first image using several layers of the visual encoder or by using a new low-dimensional feature extraction network, but it is not limited to these.

[0108] Step S404: Based on the second texture features and the second heatmap, obtain the first texture features of the first region.

[0109] Optionally, multiplying the second texture feature and the second heatmap to obtain the first texture feature of the first region can preserve the low-dimensional features related to the object identity of the user's region of interest for image retrieval.

[0110] In one example, such as Figure 4As shown, the image heatmap (second heatmap) and the low-dimensional features of the image (second texture features) are multiplied together. The low-dimensional features can be the output of the first few layers of the visual encoder, or the output of the first few layers of a new convolutional neural network, such as ResNet50, to obtain the two-dimensional features (first texture features) that need to be retained in the first image.

[0111] In this embodiment of the disclosure, the preserved two-dimensional features can be projected onto a one-dimensional vector through a projection network. For example, the projection network can employ the attention pooling operation in the CLIP model, but is not limited to this, to obtain a one-dimensional first texture feature (corresponding to...). Figure 4 The image retains query features, such as the one-dimensional low-dimensional features of "this dog" in the example above, which can improve the efficiency of feature comparison when querying images.

[0112] In this embodiment of the disclosure, an optional implementation method is provided for step S402, which may specifically include:

[0113] Step S4021: Determine at least one image feature word corresponding to the first image, where different image feature words represent features of different regions in the first image;

[0114] Optionally, the method for determining at least one image feature word corresponding to the first image can be found in the description of step S201, which will not be repeated here, or other determination methods may be used, which are not limited in this embodiment.

[0115] Optionally, this step can directly obtain at least one image feature word determined in step S201 for use to simplify the processing flow, or it can be re-determined; this embodiment of the present disclosure does not limit this.

[0116] Step S4022: Based on at least one image feature word and a first text feature, a first cross-attention network is used to obtain a first attention result. The first weight corresponding to the first attention result represents the relevance of each image feature word to the user's intent.

[0117] In this embodiment of the disclosure, image feature tuples and text feature tuples are cross-attention calculated. The first attention result can filter out image feature tuples that are less relevant to the user's intent, such as "grass" in the example above.

[0118] The first weight corresponding to the first attention result can be a similarity matrix calculated during the processing of the first cross-attention network. Smaller values ​​in the similarity matrix indicate that the image feature word is not very relevant to the user's intent. For example, the similarity of the image feature word expressing "grass" in the example above is relatively low. This calculation serves to filter out irrelevant image feature words. Based on the first weight and at least one image feature word, the first attention result can be obtained.

[0119] Step S4023: Based on the first attention result and the second text features, a second cross-attention network is used to obtain the second attention result. The second weight corresponding to the second attention result represents the correlation between each image feature word and the second text feature.

[0120] In this embodiment of the disclosure, the first attention result is cross-attention calculated with the second text feature. The resulting second attention result can focus on image feature words that are highly correlated with the second text feature, such as "dog" in the example above.

[0121] The second weight corresponding to the second attention result can be a similarity matrix calculated during the processing of the second cross-attention network. Larger values ​​in the similarity matrix indicate that the image feature word is close to the second text feature and is likely to be an image feature word that needs to be retained, such as the image feature word expressing "dog" in the example above. This step makes the image feature words related to the region of interest to the user more prominent and significant through the second text feature. Based on the second weight and the first attention result, the second attention result can be obtained.

[0122] Step S4024: Based on the first weight and / or the second weight, fuse at least one first heatmap to obtain a second heatmap.

[0123] In this embodiment of the disclosure, a second heatmap (such as an image-preserving heatmap) can be obtained by weighted summation of at least one first heatmap through sharing the weights in the cross-attention calculation.

[0124] In this embodiment of the disclosure, an optional implementation method is provided for step S4024, which may specifically include:

[0125] Step S501: Based on the first weight and / or the second weight, enhance at least one first heatmap to obtain at least one enhanced third heatmap;

[0126] In this embodiment of the disclosure, since each first heatmap corresponds to an image feature word, the first heatmaps can share the similarity matrix (first weight and / or second weight) in the attention calculation of the image feature words.

[0127] Optionally, averaging the similarity matrix along the word dimension can yield a weighted value for each first heatmap (the same number as the number of first heatmaps). This weighted value can then be applied to at least one first heatmap to enhance it.

[0128] The first weight and the second weight can be used to enhance at least one first heatmap once, or the first weight and the second weight can be merged to enhance at least one first heatmap once. This embodiment of the present disclosure does not limit the specific weights.

[0129] Step S502: Based on the second attention result, determine the third weight corresponding to at least one image feature word;

[0130] In this embodiment, linear computation can be used to further weight and fuse the calculated second attention results based on factors such as whether the image feature words belong to the user's region of interest and whether they need to be retained. Since each first heatmap corresponds to one image feature word, the first heatmaps can share the third weights from the linear computation. The calculation method for the third weights can be obtained through training.

[0131] Step S503: Based on the third weight, fuse at least one third heatmap to obtain a second heatmap.

[0132] For example, by weighting and summing at least one third heatmap based on the third weight, a second heatmap (image preservation heatmap) at the two-dimensional image level can be obtained, which indicates which regions in the first image need to be preserved.

[0133] In this embodiment of the disclosure, an optional implementation method is provided for step S102, which is "extracting the first semantic feature of the first region related to the first text in the first image input by the user". Specifically, it may include: fusing at least one image feature word based on a third weight to obtain the first semantic feature.

[0134] In this embodiment, linear computation can be used to merge multiple high-dimensional semantic features into one, based on whether the image feature words represent regions of interest to the user and whether they need to be retained. This results in a first semantic feature that represents the user's region of interest and needs to be retained in the target image. The first semantic feature corresponds to the second heatmap. Since the image feature words express the high-dimensional semantic features of different regions in the first image, the first semantic feature can also be a high-dimensional semantic feature that needs to be retained. The first semantic feature can also be called a visual focus feature; for example, in the above example, the visual focus features mainly include features related to "dog".

[0135] Based on at least one of the above embodiments, in this embodiment of the disclosure, by Figure 5 This diagram illustrates a method for predicting a first semantic feature and a second heatmap, which may specifically include:

[0136] 1) Perform cross-attention calculation on image feature tuples (at least one image feature tuple) and text feature tuples. A lower value in the calculated similarity matrix (first weight) indicates that the image feature tuple is not very relevant to the user's intent. The first attention result is calculated based on the similarity matrix and the image feature tuples. This calculation serves to filter out irrelevant image features, such as filtering out features like "grass" that are not very relevant in the example above.

[0137] 2) Perform cross-attention calculation between the first attention result and the feature preservation instruction (the second text feature corresponding to the first region). Larger values ​​in the calculated similarity matrix (second weight) indicate that the image feature word is close to the feature preservation instruction and is likely a feature that needs to be preserved, such as the feature expressing "dog" in the example above. Based on the similarity matrix and the first attention result, calculate the second attention result. This step uses the feature preservation instruction to make the image feature words related to the region of interest to the user more prominent and significant.

[0138] 3) Perform linear calculations on the second attention results. The main function is to further weight and fuse the second attention results based on whether the image feature words are in the region of interest to the user and whether they need to be retained, so as to obtain the visual focus features (first semantic features) that can represent the region of interest to the user and need to be retained in the target image. For example, in the above example, the main features that need to be retained are those related to "dog".

[0139] 4) Calculate the image-preserving heatmap (second heatmap) corresponding to the visual focusing feature. The input is a set of feature heatmaps (at least one first heatmap). Each feature heatmap corresponds to an image feature word. The feature heatmaps can share the two similarity matrices in the attention calculation and the weights in the linear calculation of the image feature words. For each similarity matrix in the two similarity matrices, the average operation is performed on each word to obtain the weighted value of each feature heatmap, which is used to enhance each feature heatmap. Then, based on the weights shared in the linear calculation, several enhanced feature heatmaps are weighted and summed to obtain the image-preserving heatmap (second heatmap) at the two-dimensional image level corresponding to the visual focusing feature.

[0140] In this embodiment of the disclosure, an optional implementation method for step S103 is provided, which specifically may include:

[0141] Step S601: Based on the first semantic features and the third text features corresponding to the target image, obtain the third semantic features of the first region.

[0142] Generally, users only want to search for target images related to a specific part of the first image, such as pets or people in the first image. Other objects, such as the background, are of no value and may even interfere with the correct image retrieval process. Guided by the second textual features, embodiments of this disclosure can accurately extract only the first semantic features of the region of interest to the user, while simultaneously obtaining a second heatmap corresponding to the region in the first image that needs to be retained. The second heatmap is used to calculate the first texture features of the first region, and the first semantic features are used in conjunction with the third textual features corresponding to the target image to calculate the third semantic features of the first region.

[0143] Among them, the third semantic feature, also known as the image planning query feature, can express the image features of the target image, such as the high-dimensional semantic feature of "dog in the pool" in the example above.

[0144] Step S602: Generate a second semantic feature based on the first textual feature and the third semantic feature.

[0145] A simple approach is to sum the first textual feature and the third semantic feature to obtain the second semantic feature, but this is not limited to this method, and other approaches can also be used. In this embodiment of the disclosure, an optional implementation method is provided for step 601, which specifically may include:

[0146] Step S6011: Based on the first semantic features and the third text features, use the third cross-attention network to obtain the third attention result.

[0147] Optionally, a multi-head attention network with shared weights is used to interact at least one head feature of the first semantic feature with the corresponding head feature of the first text feature to obtain at least one set of attention results; the at least one set of attention results are then fused to obtain a third attention result.

[0148] In this embodiment of the disclosure, the first semantic feature and the third text feature corresponding to the target image are interactively calculated through a multi-head attention network with shared weights to finally obtain the third semantic feature of the first region.

[0149] The number of attention heads (feature groups) in a multi-head attention network can be set according to actual conditions, and this embodiment does not limit it.

[0150] In this embodiment of the disclosure, the multi-head attention network performs interactive attention calculations on the first semantic feature and the third text feature, each with the same number of feature groups. Optionally, attention calculations are performed between features in the same index group to obtain multiple sets of attention results.

[0151] Optionally, a set of features in the first semantic features that is highly correlated with the third text features will be assigned a higher weight, and the same set of features in the third text features will be assigned the opposite weight. In this case, the same set of features in the first semantic features will be passed to the next step, and the same set of features in the third text features will be ignored. Similarly, at least one set of features that is less correlated with the third text features will be assigned a lower weight, and the same set of features in the third text features will be assigned the opposite weight. In this case, the same set of features in the third text features will be passed to the next step, and the same set of features in the first semantic features will be ignored.

[0152] Step S6012: Based on the third attention result, obtain the third semantic feature of the first region.

[0153] Optionally, based on the third attention result, a second multilayer perceptron is used to obtain the third semantic features of the first region.

[0154] In one example, such as Figure 6 Based on the feature planning instructions (the third text features corresponding to the target image) and visual focus features (the first semantic features), the high-dimensional semantic visual features of the target image are planned. The visual focus features and image planning instructions are interactively computed through a multi-head attention network with shared weights, ultimately yielding the image planning query features (the third semantic features). The execution process of the multi-head attention network may include:

[0155] 1) Calculate the query vector Q of the visual focusing feature and the value vector of the multi-head. Here, N represents the number of heads. For example, in the example above, the visual focus feature might include the features of "this dog" that need to be retained, which is the value vector of the heads. Both can represent characteristics related to "this dog".

[0156] 2) Further decompose the feature planning instruction into a multi-head key vector K n Sum vector V n Let n ∈ Z(1, N), where N represents the number of multi-heads. For example, in the above example, the feature planning instruction might plan features such as "dog" and "in the pool," and the key vector K of the multi-heads... n Sum vector V n These can represent different planning features. For example, K1 and V1 represent features related to "dogs," K... N and V N The use of terms related to "in a swimming pool" is intended to convey illustrative meanings only and does not constitute a limitation on the embodiments of this disclosure.

[0157] 3) Calculate Q and K respectively. n Similarity weight w n(dashed line), 1-w n (Hollow wire), w n ∈(0,1)), and use these weights on V n and Weighting is applied. As an example, V1 and... All represent characteristics related to "dog," with w1 values ​​being relatively large (e.g., close to 1). Therefore, the expression "this dog" is... After being multiplied by w1, it will be passed to the next step. Meanwhile, after being multiplied by 1-w1, V1, representing "dog," will be ignored. V, representing "pool,"... N And expressing "this dog" They represent different characteristics, w N The value of V is small (e.g., close to 0). Therefore, V, which expresses "pool",... N With 1-w N After the dot product, it will be passed to the next step, simultaneously expressing "this dog". With w N After dot product, it will be ignored. The same applies to other groups of features, which will not be elaborated further.

[0158] 4) Connect the attention results of all groups and input them into the multilayer perceptron to finally output image planning query features, such as high-dimensional semantic features expressing "dog in the pool".

[0159] In this embodiment of the disclosure, an optional implementation method for step S101 is provided, which specifically may include:

[0160] Step S701: Project the first semantic features onto the word base to obtain the second text;

[0161] Projecting the first semantic feature onto a lexical unit is equivalent to abstracting the first semantic feature into pseudo-lexical units that resemble text, thus obtaining the second text.

[0162] Step S702: Based on the first text and the second text, use a text encoder to obtain text features.

[0163] In general, text feature extraction methods can be as follows: Figure 7 As shown, visual focus feature estimation is performed based on at least one of the above embodiments to obtain visual focus features (first semantic features). These visual focus features are then projected into pseudo-words (second text), and combined with the retrieved text (first text), a text encoder is used to further generate text query features.

[0164] In this embodiment of the disclosure, such as Figure 8As shown, based on user input, the first semantic features are extracted using high-dimensional semantic features of the first region in the first image, rather than global features of the entire first image. The first semantic features only include visual features of a certain region. For example, if the input reference image (first image) is a photo of "a dog in the grass" and the input search text (first text) is "in the pool", the determined first semantic features only have features related to "dog". The abstracted second text also only has pseudo-words related to "dog". The first text and the second text are processed by a text encoder to output text query features, such as the features of "dog in the pool".

[0165] In this embodiment of the disclosure, during the text query feature generation process, the visual information of the most relevant region of the first image represented by the visual focus feature is injected into the text query feature. The features of the referenced first image are accurate and contain no irrelevant information, which can make the image retrieval results more in line with the user's intent.

[0166] In this embodiment of the disclosure, an optional implementation method is provided for step S4021, which specifically may include:

[0167] Step S801: Determine at least one first image feature word corresponding to the first image;

[0168] Optionally, the method for determining at least one first image feature word corresponding to the first image can be found in the description of step S201, which will not be repeated here, or other determination methods may also be used, which are not limited in this embodiment.

[0169] Optionally, this step can directly obtain at least one image feature word determined in step S201 for use to simplify the processing flow, or it can be re-determined; this embodiment of the present disclosure does not limit this.

[0170] Step S802: Determine at least one second image feature word corresponding to at least one second image;

[0171] In this embodiment of the disclosure, at least one second image can also be understood as an auxiliary image, used to optimize image feature tuples. The method for determining at least one second image feature tuple can be the same as the method for determining at least one first image feature tuple, and the process will not be described again.

[0172] Optionally, clustering can be performed on the first image within the candidate image set to obtain at least one second image; that is, the auxiliary image can be automatically obtained through an object clustering algorithm. The candidate image set used to determine the auxiliary image may be the same as or different from the candidate image set used for image retrieval.

[0173] Alternatively, at least one second image can be obtained based on the user's first specified operation, i.e., the auxiliary image can be manually specified by the user.

[0174] Step S803: Based on at least one second image feature word, process at least one first image feature word to obtain at least one image feature word corresponding to the first image.

[0175] In this embodiment of the disclosure, image feature tuples are further optimized using information provided by other auxiliary images. If the auxiliary image contains an object similar to or identical to the reference image, using the optimized image feature tuples is more conducive to accurate and stable image retrieval.

[0176] In this embodiment of the disclosure, an optional implementation method is provided for step S803, which may specifically include:

[0177] Step S8031: Determine the similarity between at least one first image feature word and at least one second image feature word to obtain first similarity information;

[0178] Optionally, the similarity between a first image feature word of the first image and a second image feature word of the second image can be obtained by calculating the cosine distance between the two (the two image feature words can be two vectors). High similarity indicates that the two image feature words are highly likely to represent the same object.

[0179] Optionally, the first similarity information between the image feature word tuples of the first image and a second image can be efficiently calculated in matrix form. For example, the first similarity information can be represented as a similarity matrix Ta.

[0180] Step S8032: Determine at least one text feature word corresponding to the first text, and determine the similarity between at least one text feature word and at least one second image feature word to obtain second similarity information;

[0181] Alternatively, the similarity results can be stored in vector form; for example, the second similarity information can be represented as a similarity vector Va. High similarity indicates that the image features of the second image can match the meaning described by the first text well.

[0182] Step S8033: Based on the first similarity information, the second similarity information, and at least one second image feature word, process at least one first image feature word to obtain at least one image feature word corresponding to the processed first image.

[0183] Optionally, methods such as attention calculation, multilayer perceptron, linear calculation, and weighted summation can be used to combine the second image feature words of the auxiliary image into the first image feature word group of the first image based on the calculated first similarity information and second similarity information, thereby completing the process of enhancing and optimizing the first image feature word group of the first image using the visual information related to the auxiliary image.

[0184] In one example, the weighted summation calculation method may include: fusing the first similarity information and the second similarity information to obtain a fusion weight; weighting at least one second image feature word based on the fusion weight to obtain at least one third image feature word; and fusing the at least one third image feature word with at least one first image feature word.

[0185] As an example, such as Figure 9 As shown, the similarity vector Va (second similarity information) is superimposed into a matrix along the number of image feature words in the reference image (first image). Then, it is multiplied by the similarity vector Ta (second similarity information) to obtain the fusion weight, which is used as the weighting value of each second image feature word in the auxiliary image (second image). After being weighted (multiplied by the corresponding second image feature word, it is added to the first image feature word group of the original reference image to obtain the final optimized image feature word group.

[0186] Based on at least one of the above embodiments, in this embodiment of the disclosure, by Figure 10 This diagram illustrates a method for optimizing image feature tuples, which specifically may include:

[0187] 1) Image lexical similarity calculation: Calculate the similarity between image feature lexical groups of the reference image (first image) and image feature lexical groups of the auxiliary image (second image). The similarity between an image feature lexical group of the reference image and an image feature lexical group of the auxiliary image is used to obtain a similarity matrix Ta (first similarity information). High similarity indicates that the two image feature lexical groups are highly likely to express the same object. The two dimensions of Ta represent the number of image feature lexical groups in the reference image and the number of image feature lexical groups in the auxiliary image, respectively.

[0188] 2) Text lexical similarity calculation: Calculate the similarity between the image feature lexical group of the auxiliary image and the text feature lexical group of the search text (first text) to obtain the similarity vector Va (second similarity information). High similarity indicates that the image features of the auxiliary image can match the meaning described by the search text well. The dimension of Va represents the number of image feature lexical groups of the auxiliary image.

[0189] 3) Image Feature Tag Merging: Using methods such as attention calculation, multilayer perceptron, linear computation, and weighted summation, image feature tags from the auxiliary image are merged into the image feature tag set of the reference image based on calculated similarity. This completes the process of enhancing and optimizing the image feature tag set of the reference image using relevant visual information from the auxiliary image. The optimized image feature tag set can be used for visual focus feature estimation as described above, but is not limited to this.

[0190] Based on at least one of the above embodiments, in this embodiment of the disclosure, by Figure 11 The diagram illustrates a complete process for generating visual query features, which may specifically include:

[0191] S11.1: The visual features of the input reference image (first image, such as a photo of "a dog in the grass") are fused with the input search text (first text) to predict two search instructions (retention & planning, corresponding to the first search instruction (second text features corresponding to the first region) and the second search instruction (third text features corresponding to the target image), respectively. Simultaneously, image feature tuples (at least one image feature tuple) and their corresponding heatmap sets (at least one first heatmap) are obtained. This step is mainly completed by the search instruction extraction module; for example, see the section on... Figure 3 The detailed processing steps will not be elaborated upon here.

[0192] S11.2: Further optimize the image feature tuples using information provided by other auxiliary images (second image). This step is mainly accomplished by the multi-image visual tuple enhancement module, for example, see the section on... Figure 9 , Figure 10 The detailed processing steps will not be elaborated upon here. This step is optional; for example, if a corresponding auxiliary image is not found, this step can be ignored.

[0193] S11.3: Based on the feature preservation instructions, estimate the heatmap (second heatmap) and high-dimensional semantic features (data-focused features / first semantic features) corresponding to the most relevant region (first region) in the reference image. This step is mainly implemented through the visual focus feature estimation module; for example, see the section on... Figure 5 The detailed processing steps will not be elaborated upon here.

[0194] S11.4: Guided by the retrieval instructions, for the target image being searched, low-dimensional image-preserving query features (first texture features) are retained, and high-dimensional image-planning query features (third semantic features) are planned. This step can be understood as visual query feature visualization based on the retrieval instructions. This step involves two modules: a feature preservation module and a feature planning module. For example, see the section on... Figure 4 , Figure 6The detailed processing steps will not be elaborated upon here.

[0195] S11.5: The visual information of the most relevant region (first region) of the reference image, represented by the visual focus feature, is projected and injected into the text query feature. For example, see the example of... Figure 7 , Figure 8 The detailed processing steps will not be elaborated upon here.

[0196] S11.6: Finally, the text query features and the two visual query features are fused to obtain the final query features.

[0197] The method for generating visual query features provided in this disclosure embodiment, such as Figure 12 As shown, for visual query feature visualization based on retrieval instructions, the retrieval instruction extraction module predicts feature-preserving instructions and feature-planning instructions by fusing the visual features of the input reference image and the information of the retrieval text. The feature-preserving instructions are used to generate image-preserving query features, and the feature-planning instructions are used to generate image-planning query features, which are used to characterize the user's actual retrieval intent.

[0198] Furthermore, based on the visual information of auxiliary images containing the same object in the candidate image set, the visual features of the user's object of interest extracted from a specific region of the input reference image are enhanced for visual focus feature estimation. The enhanced features are more conducive to accurate and stable image retrieval.

[0199] Furthermore, based on the instructions and the image heatmap and visual focus features estimated by visual focus feature estimation, low-dimensional texture features related to the individual identity of the object in specific regions of the reference image are preserved, and high-dimensional semantic features of the target image to be retrieved are planned. This composite set of visual query features accurately and meticulously depicts the target image in the user's mind.

[0200] And such as Figure 7 As shown, for the visual focus feature estimation of text query feature generation, the visual focus features extracted by the visual focus feature estimation module are used to further generate text query features. Because the visual focus feature accurately describes the area that the user wants to refer to, without interference from other irrelevant area features, the generated text query features are more accurate, thus making the search results more consistent with the user's actual intent.

[0201] In this embodiment of the disclosure, in order to determine the first region, another optional implementation method is provided. Specifically, the first region related to the first text can be determined in the first image based on the user's second specified operation. That is, the user can also specify the object to be retrieved in the reference image.

[0202] As an example, different regions in the first image can be generated as different options based on the semantic segmentation map of the first image, for the user to choose from. The user can click on any option to generate a second specified operation.

[0203] For example, a user can circle a region in the first image to generate a second specified operation, where the main object within the region corresponds to the first region.

[0204] Those skilled in the art will understand that the above-described designations are merely illustrative and do not constitute a limitation on the embodiments of this disclosure. Appropriate variations based on these examples may also be applied to this disclosure and should therefore be included within the scope of protection of this disclosure.

[0205] In this embodiment of the disclosure, an optional implementation method is provided for step S104, which may specifically include: for each third image in the candidate image set, extracting the third texture feature and the fifth semantic feature of the third image; and obtaining the image retrieval result based on the comparison result of the first texture feature and the third texture feature, and / or the comparison result of the second semantic feature and the fifth semantic feature.

[0206] As an example, such as Figure 13 As shown, the user inputs a reference image (first image) and search text (first text). Optionally, the user can also specify the object to be searched in the reference image (e.g., through a segmentation map). Three query features are generated through the text query feature generation process and visual query feature generation process described in at least one embodiment above. These are then processed by the query feature fusion module to generate the final query features. The query features consist of two parts: a low-dimensional query feature (i.e., image-preserved query features, corresponding to the first texture feature) and a high-dimensional query feature (the second semantic feature; a simple way to achieve this is to sum the image query features and the text query features). On the other hand, for each image (third image) in the candidate image set of the album, the image decoder obtains the image's high-dimensional query features (image global features, corresponding to the fifth semantic feature) and low-dimensional image features (output from the first few layers of the image encoder, corresponding to the third texture feature). The low-dimensional features are converted into vector form through a word projection network, i.e., the low-dimensional query features. Similarly, the low-dimensional query features and high-dimensional query features are combined to form the final candidate image features. The query features are compared with all candidate image features; the image with the smallest vector distance is the retrieved target image.

[0207] The following describes several scenarios in which the image retrieval method provided in this disclosure can be used:

[0208] 1) Search for photos of specific people / objects, such as Figure 14As shown, users can perform multimodal searches by providing a reference photo (a first image, such as a photo of someone) and inputting search text (a first text, such as "search for photos of her wearing a T-shirt in a summer park"), and quickly search the album for relevant photos of that person that meet the criteria, thus outputting search results.

[0209] 2) Generating user album stories: Users can actively generate desired album stories by simply providing a reference photo (first image) and topic text (first text, such as "a trip for two with someone") for multimodal retrieval. Specifically, the image retrieval method provided in this disclosure is used to search for candidate photos related to the reference image that match the input topic text, and then the photos are further combined to generate an album story.

[0210] 3) Utilizing multiple reference photos to search for photos of a specific object: By referencing multiple auxiliary images and the input search text, users can quickly find the target photo they want in their album. First, other candidate photos similar to a given reference photo (the first image, such as a photo of a building landmark) are found using methods such as timestamps and clustering. Optionally, users can further filter auxiliary images related to the reference image from these candidate photos. These auxiliary images are then combined with the reference image and search text (e.g., "a building surrounded by dazzling fireworks at night") as input for multimodal retrieval to obtain the final search results.

[0211] The inventors of this disclosure conducted preliminary prototype verification experiments on a publicly available dataset using the multimodal image retrieval method provided in the embodiments of this disclosure. The experimental results show that the image retrieval results using composite features are better than those of existing image retrieval methods. This indicates that this image retrieval method, which integrates textual and visual information, can effectively enhance query features and improve image retrieval accuracy.

[0212] The technical solutions provided in this disclosure can be applied to various electronic devices, including but not limited to mobile terminals and smart terminals, such as smartphones, tablets, laptops, smart wearable devices (e.g., watches, glasses), smart speakers, in-vehicle terminals, personal digital assistants, portable multimedia players, navigation devices, etc., but are not limited thereto. Those skilled in the art will understand that, in addition to components specifically designed for mobile purposes, the constructions according to the embodiments of this disclosure can also be applied to fixed-type terminals, such as digital televisions and desktop computers.

[0213] The technical solutions provided in this disclosure can also be applied to image retrieval in servers, such as independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0214] This disclosure also provides an electronic device including a processor, and optionally, a transceiver and / or memory coupled to the processor, the processor being configured to perform the steps of the method provided in any optional embodiment of this disclosure.

[0215] Figure 15 The diagram shows a structural schematic of an electronic device to which an embodiment of the present invention applies, such as... Figure 15 As shown, Figure 15 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this disclosure. Optionally, the electronic device may be a first network node, a second network node, or a third network node.

[0216] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0217] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 15 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0218] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0219] The memory 4003 is used to store computer programs that execute embodiments of the present disclosure, and is controlled by the processor 4001 to execute them. The processor 4001 is used to execute the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0220] This disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0221] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0222] The terms “first,” “second,” “third,” “fourth,” “1,” “2,” etc. (if present) in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in a sequence other than that shown in the figures or text.

[0223] It should be understood that although arrows indicate various operation steps in the flowcharts of the embodiments of this disclosure, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of the embodiments of this disclosure, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of this disclosure do not limit this.

[0224] The above text and accompanying drawings are provided as examples only to help the reader understand this disclosure. They are not intended and should not be construed as limiting the scope of this disclosure in any way. Although certain embodiments and examples have been provided, it will be apparent to those skilled in the art, based on the content disclosed herein, that changes can be made to the illustrated embodiments and examples, and other similar implementations based on the technical concept of this disclosure can be adopted without departing from the scope of this disclosure, and these modifications and modifications are also within the protection scope of the embodiments of this disclosure.

Claims

1. A method performed by an electronic device, characterized in that, include: Extract the first text features from the first text entered by the user; Extract the first semantic features of the first region related to the first text from the first image input by the user; Based on the first text features and the first semantic features, a second semantic feature is generated; Image retrieval is performed in the candidate image set based on the second semantic feature and the first texture feature of the first region.

2. The method according to claim 1, characterized in that, Also includes: The first texture feature is extracted from the first region using the following method: At least one first heatmap is determined corresponding to the first image, and different first heatmaps focus on different regions in the first image, and the different regions correspond to different image feature words; Based on the second text features corresponding to the first region and the at least one first heatmap, a second heatmap is obtained, and the second heatmap focuses on the first region; Extract the second texture features from the first image; Based on the second texture feature and the second heatmap, the first texture feature of the first region is obtained.

3. The method according to claim 2, characterized in that, The step of obtaining a second heatmap based on the second text features corresponding to the first region and the at least one first heatmap includes: Determine at least one image feature word corresponding to the first image, where different image feature words correspond to different regions in the first image; Based on the at least one image feature word and the first text feature, a first attention result is obtained using a first cross-attention network, and the first weight corresponding to the first attention result represents the relevance of each image feature word to the user intent; Based on the first attention result and the second text feature, a second cross-attention network is used to obtain a second attention result. The second weight corresponding to the second attention result represents the correlation between each image feature word and the second text feature. Based on the first weight and / or the second weight, the at least one first heatmap is fused to obtain the second heatmap.

4. The method according to claim 3, characterized in that, The step of fusing the at least one first heatmap based on the first weight and / or the second weight to obtain the second heatmap includes: Based on the first weight and / or the second weight, the at least one first heatmap is enhanced to obtain at least one enhanced third heatmap; Based on the second attention result, a third weight is determined for each of the at least one image feature word. Based on the third weight, the at least one third heatmap is fused to obtain the second heatmap.

5. The method according to claim 4, characterized in that, Extracting the first semantic feature of the first region related to the first text from the first image input by the user includes: Based on the third weight, the at least one image feature word is fused to obtain the first semantic feature.

6. The method according to claims 1-5, characterized in that, The step of generating a second semantic feature based on the first text feature and the first semantic feature includes: Based on the first semantic feature and the third text feature corresponding to the target image, the third semantic feature of the first region is obtained; A second semantic feature is generated based on the first text feature and the third semantic feature.

7. The method according to claim 6, characterized in that, The step of obtaining the third semantic feature of the first region based on the first semantic feature and the third text feature corresponding to the target image includes: Based on the first semantic feature and the third text feature, a third cross-attention network is used to obtain the third attention result; Based on the third attention result, the third semantic feature of the first region is obtained.

8. The method according to any one of claims 1-7, characterized in that, Extracting text features from the first text input by the user includes: The second text is obtained by projecting the first semantic features onto word segments. Based on the first text and the second text, the text features are obtained using a text encoder.

9. The method according to claim 6 or 7, characterized in that, Also includes: Based on the first image and the first text, a second text feature corresponding to the first region and / or a third text feature corresponding to the target image are determined, wherein the second text feature is used to determine the first semantic feature and / or the first texture feature.

10. The method according to claim 9, characterized in that, The step of determining the second text features corresponding to the first region and / or the third text features corresponding to the target image based on the first image and the first text includes: Determine the fourth semantic feature corresponding to the first image and at least one image feature word corresponding to the first image, wherein different image feature words represent the semantic features of different regions in the first image; Determine at least one text feature word corresponding to the first text; Based on the at least one text feature word, the fourth semantic feature, and the at least one image feature word, the second text feature and / or the third text feature are determined.

11. The method according to claim 10, characterized in that, The step of determining the second text feature and / or the third text feature based on the at least one text feature word, the fourth semantic feature, and the at least one image feature word includes: Based on the at least one text feature word and the fourth semantic feature, a first feature is obtained using a fourth cross-attention network, the first feature including information related to user intent; Based on the first feature and the at least one image feature word, a fifth cross-attention network is used to obtain a second feature and a third feature. The second feature represents a feature in the at least one image feature word that is similar to the first feature, and the third feature represents a feature in the at least one image feature word that is not similar to the first feature. Based on the second feature and the third feature, the second text feature and / or the third text feature are determined.

12. The method according to claim 11, characterized in that, The first feature is obtained by using a fourth cross-attention network based on the at least one text feature word and the fourth semantic feature, including: Obtain a first query vector for generating the second text feature, and / or a second query vector for generating the third text feature; Based on at least one of the first query vector and the second query vector, and the at least one text feature word, a self-attention network is used to obtain text fusion features; Based on the text fusion features and the fourth semantic features, the first feature is obtained using the fourth cross-attention network.

13. The method according to claim 12, characterized in that, At least one of the dimensions of the first query vector and the second query vector is the same as the dimension of the at least one text feature word.

14. The method according to claim 3, characterized in that, Determining at least one image feature word corresponding to the first image includes: Determine at least one first image feature word corresponding to the first image; Determine at least one second image feature word corresponding to at least one second image; Based on the at least one second image feature word, the at least one first image feature word is processed to obtain at least one image feature word corresponding to the first image.

15. The method according to claim 14, characterized in that, The step of processing the at least one first image feature word based on the at least one second image feature word to obtain at least one image feature word corresponding to the first image includes: Determine the similarity between the at least one first image feature word and the at least one second image feature word to obtain first similarity information; Determine at least one text feature word corresponding to the first text, and determine the similarity between the at least one text feature word and the at least one second image feature word to obtain second similarity information; Based on the first similarity information, the second similarity information, and the at least one second image feature word, the at least one first image feature word is processed to obtain at least one image feature word corresponding to the processed first image.

16. The method according to claim 15, characterized in that, The step of processing the at least one first image feature word based on the first similarity information, the second similarity information, and the at least one second image feature word includes: The first similarity information and the second similarity information are fused to obtain the fusion weight; Based on the fusion weight, the at least one second image feature word is weighted to obtain at least one third image feature word; The at least one third image feature word is fused with the at least one first image feature word.

17. The method according to claim 14, characterized in that, It also includes at least one of the following: Clustering is performed on the first image in the candidate image set to obtain the at least one second image; Based on the user's first specified operation, the at least one second image is obtained.

18. The method according to any one of claims 1-17, characterized in that, Also includes: The first region is determined in the first image based on a second specified operation by the user.

19. The method according to any one of claims 1-18, characterized in that, Based on the second semantic feature and the first texture feature of the first region, image retrieval is performed in the candidate image set, including: For each third image in the candidate image set, extract the third texture feature and the fifth semantic feature of the third image; Based on the comparison results of the first texture feature and the third texture feature, and / or the comparison results of the second semantic feature and the fifth semantic feature, the image retrieval result is obtained.

20. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-19.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-19.

22. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-19.

Images