Method, apparatus, storage medium, and program product for image retrieval

Abstract

The application discloses a kind of image retrieval method, equipment, storage medium and program product, belong to image recognition technical field.The method comprises: obtaining image to be searched, image to be searched includes target object, by the feature extraction layer in visual transformation model, the feature extraction of image to be searched is carried out, obtains the first abstract semantic feature of image to be searched, by the mask layer in visual transformation model, the first abstract semantic feature is handled, obtains the mask information of image to be searched, the mask information is used to mark the region where the target object is located, by the feature extraction layer, the first abstract semantic feature and mask information are handled, and the representation vector of image to be searched is obtained, based on the representation vector, determine the image matched with target object.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to an image retrieval method, device, storage medium, and program product. Background Technology

[0002] With the development of network and computer technologies, many platforms use images of the target object to search for images of the target object in image libraries, providing image search functionality. For example, on e-commerce platforms, users upload images of target products, and the platform searches for matching product images based on those images, then returns a list of products to the user.

[0003] In related technologies, the platform uses a CNN (convolutional neural network) model to extract features from images, and then calculates the vector similarity between these features and images in an image library to determine if images of the target object also exist.

[0004] Because CNN models have relatively weak local perception capabilities, this may result in the retrieved image not matching the target object. Summary of the Invention

[0005] This application provides an image retrieval method, device, storage medium, and program product, which can improve the probability of the retrieved image matching the desired search object. The technical solution is as follows: Firstly, this application provides an image retrieval method, the method comprising: Obtain the image to be retrieved, wherein the image to be retrieved includes the target object; The first abstract semantic features of the image to be retrieved are obtained by extracting features from the feature extraction layer in the visual transformation model. The first abstract semantic features are processed by the mask layer in the visual transformation model to obtain the mask information of the image to be retrieved, wherein the mask information is used to mark the region where the target object is located; The first abstract semantic features and the mask information are processed by the feature extraction layer to obtain the representation vector of the image to be retrieved; Based on the representation vector, an image matching the target object is determined.

[0006] In one optional approach, the feature extraction layer comprises N+1 visual transformation sublayers, where N is greater than 1. The feature extraction layer in the visual transformation model extracts features from the image to be retrieved to obtain the first abstract semantic features of the image to be retrieved, including: The first abstract semantic feature is obtained by extracting features from the image to be retrieved through the first N visual transformation sub-layers of the N+1 visual transformation sub-layers. The step of processing the first abstract semantic features and the mask information through the feature extraction layer to obtain the representation vector of the image to be retrieved includes: Based on the last visual transformation sublayer among the N+1 visual transformation sublayers, the first abstract semantic feature, and the mask information, the representation vector of the image to be retrieved is determined.

[0007] In this way, since the first abstract semantic features are extracted through multiple visual transformation sub-layers, they can include higher-order semantics. Using higher-order semantics to determine mask information can accurately determine mask information, thereby obtaining a representation vector that does not include non-subject region information as much as possible.

[0008] In an optional embodiment, the feature extraction layer further includes a feature fusion sublayer, wherein determining the representation vector of the image to be retrieved based on the last visual transformation sublayer among the N+1 visual transformation sublayers, the first abstract semantic feature, and the mask information includes: The first abstract semantic feature is processed based on the mask information through the last visual transformation sub-layer to obtain the second abstract semantic feature; The feature fusion sublayer is used to fuse the second abstract semantic feature with the features output by at least one of the first N visual transformation sublayers to obtain the representation vector.

[0009] In this way, since features extracted from other visual transformation sub-layers are also integrated, feature representations from different receptive fields can be aggregated, improving the retrieval accuracy of the visual transformation model for multi-scale targets.

[0010] In one alternative approach, the first abstract semantic feature includes spatial markers for each block of the image to be retrieved, and the processing of the first abstract semantic feature based on the mask information to obtain the second abstract semantic feature includes: Based on the mask information, multi-head self-attention scores are calculated on the first abstract semantic features to obtain an attention score matrix. In the process of multi-head self-attention score calculation, a preset negative penalty is applied to the attention scores of spatial markers identified as backgrounds participating in the calculation. The second abstract semantic feature is calculated based on the attention score matrix.

[0011] In this way, since a preset negative penalty is applied to the attention score of the spatial marker identified as background during the multi-attention head score calculation, the representation vector of the image to be retrieved can be made to minimize the reflection of features of non-subject regions.

[0012] In one alternative embodiment, the at least one visual transformation sublayer includes a first visual transformation sublayer and a second visual transformation sublayer, wherein the first visual transformation sublayer and the last visual transformation sublayer are separated from the second visual transformation sublayer by M visual transformation sublayers, where M is greater than 2.

[0013] This method integrates shallow, medium, and deep features. Shallow features correspond to small targets, medium features to medium-sized targets, and deep features to large targets, thereby further improving the model's retrieval accuracy for multi-scale targets.

[0014] In one alternative approach, the visual transformation sublayer includes a multi-head self-attention module and an activation module, wherein the activation module includes multiple feedforward activation units and stacking units; The first abstract semantic feature is obtained by extracting features from the image to be retrieved through the first N visual transformation sub-layers out of the N+1 visual transformation sub-layers, including: In the i-th visual transformation sub-layer among the first N visual transformation sub-layers, the output features of the (i-1)-th visual transformation layer are processed by the multi-head self-attention module to obtain the first residual feature, where i takes values ​​from 2 to N; The first residual feature is processed by the multiple feedforward activation units to obtain multiple fully connected features; The multiple fully connected features are processed by the superposition unit to obtain the output feature of the i-th visual transformation sub-layer, wherein the output feature of the last visual transformation sub-layer among the first N visual transformation sub-layers is the first abstract semantic feature.

[0015] In this way, by using multiple feedforward activation units, and since different feedforward activation units have different focuses, the first abstract semantic features are richer.

[0016] In one alternative approach, the first abstract semantic feature includes a global state label (CLS) and a spatial label for each block of the image to be retrieved. The step of processing the first abstract semantic feature through a mask layer in the visual transformation model to obtain mask information of the image to be retrieved includes: For each spatial marker in the first abstract semantic feature, the attention distribution weight of the CLS on the spatial marker in the multi-head self-attention layer is determined through the mask layer; The attention distribution weights are averaged through the mask layer to obtain the subject salient response value corresponding to the spatial label; The mask information is determined based on the relationship between the salient response value of the subject corresponding to each spatial marker and the truncation threshold through the mask layer.

[0017] In this way, the self-attention weight of the main area is relatively high, while the self-attention weight of the non-main area is relatively low. Therefore, by using the attention distribution weight of the self-attention layer, the mask information can be accurately determined.

[0018] In one alternative approach, the target object is a product; after determining the image matching the target object based on the representation vector, the method further includes: returning a list of products to which the image matching the target object belongs.

[0019] In this way, applying the visual transformation model to e-commerce scenarios can improve the accuracy of users' product searches, thereby increasing conversion rates.

[0020] In an alternative approach, the method further includes: A first sample set and a second sample set are obtained, wherein the first sample set includes multiple first sample images and annotation information of the first sample images, and the second sample set includes multiple second sample images and annotation information of the second sample images, wherein the multiple second sample images include images of each standardized product unit acquired under different image acquisition conditions; During the training process of the current batch, the annotation information of the first sample image of the current batch is encoded based on the text encoder to obtain the semantic vector of the first sample image of the current batch, and the first sample image of the current batch is processed based on the initial visual transformation model to obtain the representation vector of the first sample image of the current batch. Based on the semantic vector and the representation vector of the first sample image of the current batch, the initial visual transformation model is updated to obtain the intermediate visual transformation model. Based on the second sample set, the intermediate visual transformation model is trained to obtain the visual transformation model.

[0021] In this way, by training the model in two stages, the visual transformation model can fully understand the complex semantics in e-commerce scenarios.

[0022] Secondly, this application provides a method for training a visual transformation model, wherein the visual transformation model is the visual transformation model described in the first aspect or any of the optional methods above, and the method includes: A first sample set and a second sample set are obtained, wherein the first sample set includes multiple first sample images and annotation information of the first sample images, and the second sample set includes multiple second sample images and annotation information of the second sample images, wherein the multiple second sample images include images of each standardized product unit acquired under different image acquisition conditions; During the training process of the current batch, the annotation information of the first sample image of the current batch is encoded based on the text encoder to obtain the semantic vector of the first sample image of the current batch, and the first sample image of the current batch is processed based on the initial visual transformation model to obtain the representation vector of the first sample image of the current batch. Based on the semantic vector and the representation vector, the initial visual transformation model is updated to obtain an intermediate visual transformation model; Based on the second sample set, the intermediate visual transformation model is trained to obtain the visual transformation model.

[0023] Thirdly, an image retrieval apparatus is provided, the apparatus comprising at least one module, the at least one module being configured to perform operations performed by the image retrieval method described in the first aspect and any alternative manner of the first aspect, or to perform operations performed by the visual transformation model training method described in the second aspect.

[0024] Fourthly, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to perform operations performed by the image retrieval method as described in the first aspect and any alternative manner of the first aspect, or to perform operations performed by the training method of the visual transformation model as described in the second aspect.

[0025] Fifthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction that is loaded and executed by a processor to perform operations performed by the image retrieval method as described in the first aspect and any alternative manner of the first aspect, or to perform operations performed by the training method of the visual transformation model as described in the second aspect.

[0026] In a sixth aspect, a computer program product is provided, the computer program product storing at least one instruction that is loaded and executed by a processor to perform operations performed by the image retrieval method as described in the first aspect and any alternative manner of the first aspect, or to perform operations performed by the training method of the visual transformation model as described in the second aspect.

[0027] The beneficial effects of the technical solution provided in this application are: In this application, a mask layer is introduced into the visual transformation model. The mask layer uses the abstract semantic features of the image to be retrieved to determine the mask information. Since the abstract semantic features are high-order semantic features of the image to be retrieved, the mask layer can accurately separate the foreground and background, and find the main object region and non-main object regions. Therefore, when determining the representation vector of the image to be retrieved, the features of non-main object regions can be weakened, so that the representation vector includes the pure high-order semantics of the main object. Thus, when calculating similarity with other images, the interference of features of non-main object regions can be reduced, thereby increasing the probability of retrieving an image that matches the object. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of the image retrieval method provided in the embodiments of this application; Figure 2 This is an architecture diagram of the visual transformation model provided in the embodiments of this application; Figure 3 This is an architecture diagram of the visual transformation sublayer provided in the embodiments of this application; Figure 4 This is a flowchart of the training method for the visual transformation model provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the advantages of image retrieval provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the image retrieval device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the training device for the visual transformation model provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0031] With the rapid development of e-commerce, image retrieval has become a crucial function on some platforms. Currently, most image retrieval functions rely on deep learning models, such as CNN models or standard Vision Transformer (ViT) models. These deep learning models extract features from user-uploaded images (including the target object) and then calculate vector similarity to find images containing objects similar to the target object—that is, images that match the target object. For example, on an e-commerce platform, a user uploads an image containing a target product through the search box. The e-commerce platform searches for product images that match the target product based on that image and then returns a list of products to the user.

[0032] In scenarios where images are searched using images, user-uploaded images often include complex background environmental information. When deep learning models extract features from these images, due to their weak local perception capabilities, they often struggle to effectively focus on the main subject. This results in non-subject areas (such as background, models, and irrelevant accessories) severely interfering with the search results. This is especially true for complex product categories such as clothing and jewelry, where the matching rate between search results and products is relatively low.

[0033] Based on this, embodiments of this application provide an image retrieval method. In this method, a mask information recognition function is added to the visual transformation model, which enables the recognition of the mask information of the image to be retrieved (the image to be retrieved includes the target object). This reduces background information in the representation vector of the image to be retrieved, thereby increasing the probability of retrieving an image that matches the target object when matching it with images in the image library, that is, increasing the probability of retrieving an image that includes an object similar to the target object.

[0034] The image retrieval method provided in this application can be implemented by a computer device, which may be a single server or a server cluster. This method can be applied to various image retrieval scenarios, such as search scenarios in e-commerce, retrieval of similar images in AI question-answering systems, and retrieval of images that misappropriate brand logos (logotypes, logos). This application does not limit the specific application scenario.

[0035] The image retrieval method provided in the embodiments of this application will be described below with reference to the accompanying drawings. Figure 1 Steps S101 to S105 in the process. Figure 1 The method of image retrieval performed by the server is used as an example for illustration.

[0036] Step S101: Obtain the image to be retrieved, wherein the image to be retrieved includes the target object.

[0037] On the terminal, when a user wants to use the "Photo Search" function while shopping using a shopping application or through a shopping website, they can enter an image into the search box of the shopping application. This image can be a photo taken by the user while entering the image into the search box, or an image selected by the user from their photo album or image library. The terminal sends the image to the server. The server receives the image.

[0038] Alternatively, on a terminal, when a user is shopping using a shopping application or through a shopping website, if they use the "Find Similar Items" image search function for a specific product they are currently browsing, they can trigger the "Find Similar Items" function by long-pressing the product information, clicking the "Find Similar Items" option, or by other means. The terminal will then send an image of the product to the server. The server receives the image.

[0039] Alternatively, a user can input an image into an AI-powered question-answering system to find images of objects or animals that include that image on the network. The server will then receive the image.

[0040] Here, the image received by the server is called the image to be retrieved, and the objects included in the image to be retrieved are called the target objects. The target objects differ in different application scenarios. For example, in e-commerce, the target object is the product; in an AI question-answering system, the target object can be an item or an animal; and in a scenario involving the retrieval of stolen brand logos, the target object can be the brand logo itself. The following descriptions will use the product as an example to illustrate this.

[0041] Step S102: The image to be retrieved is extracted by the feature extraction layer in the visual transformation model to obtain the first abstract semantic features of the image to be retrieved.

[0042] In this embodiment, the visual transformation model is a Vision Transformer model, which includes an input embedding layer, a feature extraction layer, a mask layer, and an output layer. See [link to relevant documentation]. Figure 2 Therefore, the visual transformation model involved in the embodiments of this application not only has the function of recognizing mask information, but also has the function of determining the representation vector of the image to be retrieved based on the mask information.

[0043] The input embedding layer, serving as the starting point of the visual transformation model, converts the image to be retrieved into a sequenced high-dimensional feature token. For example, the input embedding layer includes an image segmentation unit, a linear mapping unit, and a state token concatenation unit. The image segmentation unit divides the image into blocks. The linear mapping unit performs linear mapping on the blocks to a high-dimensional space. The state token concatenation unit adds a global state token (Classification Token, CLS).

[0044] The feature extraction layer, also known as the deep vision backbone cascade module, is used to extract shallow, mid-level, and deep features from the image to be retrieved. Shallow features are primarily composed of pixel-level low-level visual primitives, including but not limited to low-level pixel details such as edges, textures, and colors. Shallow features do not involve abstract semantics and are also called basic features. Mid-level features are local components and spatial structures of objects formed by shallow textures, such as the edge of a hat. Mid-level features possess basic local semantics and can distinguish simple local components, but cannot identify complete targets or the global scene. Deep features, also known as abstract semantic features or high-level semantic features, include complete targets, the global scene, and target categories in the image. Deep features possess high-order abstract semantics, enabling them to move beyond pixel details and focus on content recognition of the image. Here, "target" in this embodiment refers to an object.

[0045] Mask layers are used to determine mask information for an image based on abstract semantic features, thereby isolating the target subject from the background. The target subject is the object of interest in the image, such as a product, item, or animal; therefore, the target subject is also called the object subject.

[0046] The output layer is used to perform pooling and other processing on the features output by the feature extraction layer.

[0047] The server inputs the image to be retrieved into the input embedding layer. The image patching unit uses an n×n sliding window for non-overlapping patching, dividing the image into multiple blocks, with the value of n set empirically. The linear mapping unit performs linear mapping on each block to project it into a high-dimensional space, obtaining a feature sequence. Here, when the image has three channels (red, green, blue, RGB), the three channels are first concatenated to obtain a one-dimensional vector before linear mapping. The state label concatenation unit concatenates a CLS (Category Class Sequence) for aggregating global information at the beginning of the feature sequence, and then outputs a sequence of label matrices with the CLS added.

[0048] For example, if the image to be retrieved is 336 x 336 x 3, the image segmentation unit uses a 14x14 sliding window to perform non-overlapping segmentation, resulting in 576 segments. The linear mapping unit projects each segment onto a high-dimensional space of 1024. The state label concatenation unit adds a CLS to the beginning of the feature sequence of each segment, ultimately outputting a 577×1024 label matrix sequence.

[0049] Optionally, the feature extraction layer is built on the Vision Transformer architecture, performing deep feature extraction through N+1 vertically cascaded visual transformation sub-layers. These visual transformation sub-layers can be understood as VisionTransformer blocks (VT blocks). See also... Figure 2 The N+1 visual transformation sublayers include the 1st to the N+1th visual transformation sublayers, and the ith visual transformation sublayer receives the features output by the (i-1)th visual transformation sublayer and inputs features to the (i+1)th visual transformation sublayer. Different visual transformation sublayers can have the same architecture, but their weight parameters are generally different.

[0050] The sequence of labeled matrices is input into the first visual transformation sub-layer of the N+1 visual transformation sub-layers. The extracted feature values ​​from the first visual transformation sub-layer are then passed to the second visual transformation sub-layer, and so on, until the Nth visual transformation sub-layer outputs its feature. The output feature of the Nth visual transformation sub-layer is called the first abstract semantic feature.

[0051] Optionally, the visual transformation sublayer employs a sub-layer layer normalization (sub-LN) module, a global topological flow with dual residuals, and a multi-head self-attention (MHSA) module.

[0052] In one example, see Figure 3 The visual transformation sublayer includes a multi-head self-attention module and an activation module. The multi-head self-attention module includes a first sub-LN, a multi-head self-attention unit, and a first adder. The multi-head self-attention unit is an MHSA module that integrates 2D Rotary Position Embedding (2DRoPE). The activation module includes multiple feedforward activation units and a stacking unit. The multiple feedforward activation units are all Swish-Gated Linear Units (SwiGLUs), and the stacking unit includes a second sub-LN, a multiplier, a third sub-LN, a fully connected (FC) network, and a second adder. Here, the architecture of these multiple feedforward activation units is the same, but the weight parameters are different. To reduce computational cost, this embodiment uses an example of multiple feedforward activation units including two feedforward activation units for illustration.

[0053] In the first N visual transformation sub-layers, the processing performed in each visual transformation sub-layer is the same, except that the input of the first visual transformation sub-layer is the sequence of label matrices, while the input of the other visual transformation sub-layers is the output features of the previous visual transformation sub-layer. Here, we will take the processing performed in the i-th visual transformation sub-layer as an example for explanation.

[0054] The i-th visual transform sublayer receives the output features of the (i-1)-th visual transform sublayer. These output features include a CLS (Conditional Language Scaling) and multiple spatial markers. The CLS aggregates global information of the image to be retrieved. Each spatial marker corresponds one-to-one with a block of the image; that is, each block corresponds to a spatial marker, and each spatial marker records the features of its corresponding block. Each spatial marker is a token, and the CLS is also a token, which itself is a multi-dimensional vector. Thus, the output features of the (i-1)-th visual transform sublayer consist of multiple tokens.

[0055] The first sub-LN first performs sub-layer normalization on the output features of the (i-1)th visual transformation sub-layer to obtain normalized features, thus stabilizing the activation distribution and accelerating training. The multi-head self-attention unit performs multi-head self-attention calculation on the normalized features to obtain multi-head self-attention features. The multi-head self-attention calculation process is as follows: First, through linear projection, the normalized features are mapped into three sets of feature matrices, resulting in a query (Q) matrix, a key (K) matrix, and a value (V) matrix. Q represents the feature being sought, K represents the feature provided by each token, and V represents the actual feature content carried by each token. Then, the Q, K, and V matrices are split into multiple attention matrices to obtain the query matrix, key matrix, and value matrix corresponding to each attention head. The query matrix is ​​constructed by concatenating the query vectors corresponding to each token as row vectors, the key matrix is ​​constructed by concatenating the key vectors corresponding to each token as row vectors, and the value matrix is ​​constructed by concatenating the value vectors corresponding to each token as row vectors. Under each attention head, for any token, a relevance score is calculated using its own query vector and the key vectors of all tokens. The matrix composed of these relevance scores is the attention score matrix for that attention head, where each element is a relevance score. Then, a soft-maximization function is applied row-by-row to the attention score matrix. Normalization yields the attention weight matrix for this attention head, with a size of S×S, where S is the number of tokens. Then, for each token, the value vectors of all tokens are weighted and summed using the attention weights as coefficients to obtain the updated vector for each token. Here, the calculation process for a single attention head is shown in formula (1).

[0056] (1) In formula (1), This represents the self-attention operation. For the dimension of the K vector within a single attention head, This represents the product of the query matrix and the transpose of the key matrix, which represents the pairwise similarity of all tokens. This indicates normalization, and V represents the matrix composed of key vectors.

[0057] Then, the vectors updated with tokens calculated by each attention head are concatenated along the feature dimension, and the concatenation result is processed through a linear layer to obtain multi-head self-attention features.

[0058] The first adder adds the multi-head self-attention feature to the output feature of the (i-1)th visual transformation sub-layer to obtain the first residual feature.

[0059] The second sub-LN performs sub-layer normalization on the first residual feature, obtaining the sub-layer normalized feature, which stabilizes the activation distribution and accelerates training. Two feedforward activation units process the sub-layer normalized feature, obtaining the first fully connected feature and the second fully connected feature. Assuming the sub-layer normalized feature is d-dimensional, each feedforward activation unit first maps the d-dimensional feature to a 4d dimension, then activates the 4d-dimensional feature to obtain the fully connected feature. A multiplier performs a dot product on the first and second fully connected features to obtain the dot product feature. The third sub-LN performs sub-layer normalization on the dot product feature, obtaining the sub-layer normalized feature. The fully connected network reduces the sub-layer normalized feature to d-dimensionality, obtaining the reduced-dimensional feature. A second adder adds the reduced-dimensional feature to the first residual feature to obtain the second residual feature. The second residual feature is the output feature of the i-th visual transformation sub-layer. The second residual feature is output to the next visual transformation sublayer.

[0060] Thus, after the label matrix sequence of the image to be retrieved is processed by N cascaded visual transform sub-layers, the output feature of the Nth visual transform sub-layer is obtained. This output feature is the first abstract semantic feature. The Nth visual transform sub-layer outputs the first abstract semantic feature to the last visual transform sub-layer among the N+1 visual transform sub-layers, and also outputs the first abstract semantic feature to the mask layer.

[0061] Step S103: The first abstract semantic feature is processed by the mask layer in the visual transformation model to obtain the mask information of the image to be retrieved.

[0062] After receiving the first abstract semantic feature, the mask layer processes the first abstract semantic feature to obtain the mask information of the image to be retrieved.

[0063] Optionally, the process of determining the mask information is as follows: As described above, the first abstract semantic feature includes the CLS and the spatial labels of each block. For the target spatial label, which is the spatial label of any block, the mask layer calculates the attention distribution weights of the CLS for that target spatial label in the multi-head self-attention layer. These attention distribution weights are also called attention weights. The calculation process for the attention distribution weights is as follows: Under the s-th attention head, a linear projection is performed through the projection matrix corresponding to the s-th attention head (s ranges from 1 to the number of attention heads) to obtain the Q-vector of the CLS and the K-vectors of each spatial label. The dot product of the Q-vector and each K-vector is calculated to obtain the dot product result. The dot product result is divided by a scaling factor to obtain the relevance score, where the scaling factor is the square root of the dimension of the K-vectors. The relevance scores of the CLS and all tokens in the sequence (the sequence composed of the CLS and the spatial labels) are uniformly softmax normalized, and the weights corresponding to each spatial label after normalization are extracted to constitute the attention distribution weights of the CLS for each spatial label under the s-th attention head. The attention distribution weights of each spatial label in each attention head are obtained in this way. Since the projection parameters of each attention head are independent of each other, different attention heads can obtain different attention distribution weights.

[0064] For a target spatial marker, the mask layer sums the weights of all attention distributions for that marker. This sum is then divided by the total number of attention heads to obtain the average weight of the attention distributions, which is the subject salient response value corresponding to that target spatial marker. This method can be used to obtain the subject salient response values ​​for all spatial markers.

[0065] Then, the relationship between the salient response value of the subject and the truncation threshold is determined. If the salient response value of the subject is greater than or equal to the truncation threshold, the block corresponding to the target spatial label is determined as the foreground, i.e., it is identified as the subject. If the salient response value of the subject is less than the truncation threshold, the block corresponding to the target spatial label is determined as the background. For example, in the image to be retrieved, street scenes and model limbs are identified as the background, while hats are identified as the foreground. Based on the judgment results, the mask information of the image to be retrieved can then be obtained. This mask information includes a one-dimensional binary physical route mask value, where a physical route mask value of 1 indicates foreground and a physical route mask value of 0 indicates background.

[0066] The cutoff threshold here is an empirical value, obtained by technicians through extensive simulation analysis.

[0067] Step S104: The first abstract semantic feature and the mask information are processed by the feature extraction layer to obtain the representation vector of the image to be retrieved.

[0068] After obtaining the mask information, the feature extraction layer uses the mask information to block background noise as much as possible during the processing of the first abstract semantic features. This is to ensure that the final representation vector of the image to be retrieved does not include the abstract semantic features of the background as much as possible, but includes the abstract semantic features of the object as much as possible, that is, it includes the higher-order semantics of the object as much as possible.

[0069] Optionally, when the feature extraction layer includes N+1 visual transformation sub-layers, the representation vector of the image to be retrieved can be determined based on the last visual transformation sub-layer, the first abstract semantic features, and the mask information among the N+1 visual transformation sub-layers.

[0070] In one example, the first abstract semantic feature and mask information are input into the last visual transform sublayer, and the output feature of the last visual transform sublayer is called the second abstract semantic feature. Then, the output layer of the visual transform model can refold and map the second abstract semantic feature into a three-dimensional spatial feature vector, and perform global average pooling (GAP) on the three-dimensional spatial feature vector to obtain a one-dimensional feature, which is the representation vector of the image to be retrieved.

[0071] See another example, see here. Figure 2 The feature extraction layer also includes a feature fusion sublayer. The first abstract semantic feature and mask information are input into this last visual transformation sublayer, and the output feature of this last visual transformation sublayer is called the second abstract semantic feature. Then, the feature fusion sublayer extracts the output feature of at least one of the first N visual transformation sublayers through a skip connection, and fuses the second abstract semantic feature with the output feature of this at least one visual transformation sublayer to obtain the fused feature. The output layer of the visual transformation model performs GAP processing on this fused feature to compress the spatial dimension, obtaining a one-dimensional feature, which is the representation vector of the image to be retrieved. In this way, feature representations from different receptive fields can be aggregated, thereby improving the retrieval accuracy of the visual transformation model for multi-scale targets.

[0072] It should be understood that feature fusion is for aggregating feature representations from different receptive fields. Therefore, the features extracted by the at least one visual transformation sub-layer are mid-level or shallow features of the image to be retrieved. This application does not limit which mid-level or shallow features output by the visual transformation sub-layer are selected.

[0073] The at least one visual transformation sublayer may include two visual transformation sublayers, namely a first visual transformation sublayer and a second visual transformation sublayer. The first visual transformation sublayer and the second visual transformation sublayer are separated by M visual transformation sublayers. The last visual transformation sublayer in the N+1 visual transformation sublayers is separated from the second visual transformation sublayer by M visual transformation sublayers, where M is greater than 2. The second visual transformation sublayer is used to extract shallow features of the image to be retrieved, the first visual transformation sublayer is used to extract mid-level features of the image to be retrieved, and the last visual transformation sublayer is used to extract deep features of the image to be retrieved.

[0074] The value of M can be an empirical value, such as N being 23 and M being 7. In this way, the first visual transformation sub-layer is the 16th visual transformation sub-layer, the second visual transformation sub-layer is the 8th visual transformation sub-layer, and the last visual transformation sub-layer is the 24th visual transformation sub-layer.

[0075] It should be understood that this example illustrates the fusion of shallow and medium-level features. It is also possible to fuse only shallow features or only medium-level features, or to fuse more shallow features and / or more medium-level features.

[0076] Optionally, taking the output features of the 8th and 16th visual transformation sub-layers as examples, the feature fusion process is illustrated as follows: Assume the features output by the 8th, 16th, and 24th visual transformation sublayers are labeled as matrices X8, X... 16 and X 24 First, the CLS at the beginning of each matrix is ​​removed. Then, the remaining matrix A×B is refolded and mapped to a three-dimensional spatial eigenvector X of C×C×B. i reshape The value of i can be 8, 16, or 24. For example, the image to be retrieved consists of 24×24 blocks, and the dimension of the features input to the embedding layer is 577×1024, where X8, X... 16 and X 24 Both are 577×1024. After removing the CLS, 576 tokens are obtained, each with a dimension of 1024. A equals 576, B equals 1024. The 576 tokens unfolded in one dimension are restored to a two-dimensional spatial grid arrangement, i.e., 24 rows × 24 columns. i reshape In this context, the location index directly corresponds to the spatial coordinates in the image to be retrieved.

[0077] Next, for X with i=8 i reshape Using a set of 1×1 convolution kernels on X i reshape Non-linear filtering and feature alignment along the channel dimension are performed to obtain the processed X.i reshape The processing for i=16 and 24 is the same as for i=8, except that different convolution kernels are used. Assuming the number of 1×1 convolution kernels in this set is H, then the processed X... i reshape The dimensions are 24×24×H.

[0078] Next, we use a set of scale confidence weights α i (where i = {8, 16, 24}, and Σα) i =1), for these three processed X i reshape Weighted summation and orthogonal fusion are performed to obtain the fused features. Specifically, for each processed X... i reshape α i The number is H. α i It is obtained through adaptive learning during the training process of the visual transformation model.

[0079] In this way, since shallow features are high-resolution, fine-grained local textures, they can ensure the ability to distinguish small targets; medium-level features can ensure the ability to distinguish medium-sized targets; and deep features are large-scale global semantic information, they can ensure the ability to distinguish large targets. Therefore, by fusing multi-scale features, the retrieval accuracy of multi-scale targets can be improved.

[0080] Optionally, the N+1 visual transformation sub-layers have the same structure. In the last visual transformation sub-layer, the first sub-LN first performs sub-layer normalization on the features output by the previous visual transformation sub-layer to obtain the sub-layer normalized features. Then, the multi-head self-attention unit uses mask information to perform multi-head self-attention calculation on the sub-layer normalized features to obtain multi-head self-attention features. After obtaining the multi-head self-attention features, the subsequent processing of the multi-head self-attention features is the same as that of the visual transformation sub-layers mentioned above, and will not be repeated here.

[0081] In one example, the process of a multi-head self-attention unit performing multi-head self-attention computation is as follows: The multi-head self-attention unit converts the mask information into a spatial mask for multi-head self-attention calculation. That is, for a spatial marker with a physical routing mask value of 0, a preset negative penalty is set, such as negative infinity. The calculation idea here is basically the same as that in formula (1), except that: if the physical routing mask value corresponding to the spatial marker is 1, then the multi-head self-attention calculation process under a single attention head is as shown in formula (1). If the physical routing mask value corresponding to the spatial marker is 0, then the multi-head self-attention calculation process under a single attention head is as shown in formula (2).

[0082] (2) In formula (2), an addition is made to formula (1). , This indicates a preset negative penalty. This is equivalent to adding a preset negative penalty to the background, thus weakening the output of background noise.

[0083] Then, the vectors updated with tokens calculated by each attention head are concatenated, and the concatenation result is processed through a linear layer to obtain multi-head self-attention features.

[0084] Step S105: Based on the representation vector, determine the image that matches the target object.

[0085] After obtaining the representation vector of the image to be retrieved, the representation vectors of each image in the image database can be obtained. The similarity between the representation vector of the image to be retrieved and the representation vectors of each of the other images is calculated, and the order of the images from high to low similarity is determined.

[0086] In scenarios like "photo-to-buy" or "find similar items," a list of products corresponding to images can be returned to the user in descending order of similarity. The terminal then receives and displays this product list.

[0087] In scenarios where similar images are searched, the user can be presented with K images sorted from highest to lowest similarity. K can be an empirical value or a value set by the user.

[0088] It should be understood that, in the preceding description, the second abstract semantic feature directly participates in the feature fusion process. In another implementation, the second abstract semantic feature can be processed by one or more visual transformation sub-layers before participating in the feature fusion process.

[0089] It should also be understood that, in the foregoing description, the visual transformation sublayers that output the first abstract semantic feature and the second abstract semantic feature are adjacent, but in other embodiments, they may not be adjacent.

[0090] The above describes the usage of the visual transformation model; the following describes the process of training the visual transformation model. Figure 4 The training method and flow for the visual transformation model are shown below. (See attached image.) Figure 4 Steps S201 to S204 in the process. The device for training the visual transformation model and the device for using the visual transformation model may or may not be the same, and this application embodiment does not limit this.

[0091] Step S201: Obtain a first sample set and a second sample set, wherein the first sample set includes multiple first sample images and annotation information of the first sample images, and the second sample set includes multiple second sample images and annotation information of the second sample images, wherein the multiple second sample images include images of each standardized product unit acquired under different image acquisition conditions.

[0092] The server obtains a first sample set and a second sample set. The first sample set is used for the first stage of training, which enables the visual transformation model to have a strong generalization feature extraction capability. The second sample set is used for the second stage of training, which enables the visual transformation model to be finely tuned for specific retrieval ranking metrics.

[0093] The first sample set includes multiple first sample images and their annotation information; it can also be understood as a set of image-text pairs. These multiple first sample images cover a large amount of data from the application scenarios of the visual transformation model. For example, in an e-commerce scenario, these multiple first sample images include product background images, images of models wearing the product, and real-life photos taken by buyers when reviewing the product. The descriptive information of the first sample images can be rich text information including product titles and attributes, which may include, but are not limited to, color, material, and occasion for wearing the product.

[0094] The second sample set includes multiple second sample images and their annotation information. These multiple second sample images include images that are difficult to distinguish. For example, in an e-commerce scenario, the second sample images belong to categories such as clothing and jewelry, which are also difficult to distinguish. Furthermore, these multiple second sample images also include images acquired under different conditions.

[0095] The second sample set includes images of multiple precisely labeled Standard Product Units (SPUs), and the images of the same SPU include real-world images taken under different image acquisition conditions. These image acquisition conditions include, but are not limited to, lighting, background, and viewing angle. In this way, images under the same SPU form positive sample pairs, and images under different SPUs form negative sample pairs, making it possible to construct a positive-negative sample matrix that is difficult to distinguish.

[0096] In the second sample set, the annotation information is the name of the SPU.

[0097] The number of SPUs in this application embodiment can be set according to empirical values, such as 10,000.

[0098] The number of first sample images is greater than the number of second sample images. This application does not limit the number of first and second sample images, and can be set according to empirical values, such as 10 billion first sample images and 100,000 second sample images.

[0099] Step S202: During the training process of the current batch, the annotation information of the first sample image of the current batch is encoded based on the text encoder to obtain the semantic vector of the first sample image of the current batch, and the first sample image of the current batch is processed based on the initial visual transformation model to obtain the representation vector of the first sample image of the current batch.

[0100] In the first stage, a CLIP (Contrastive Language-Image Pre-training) dual-stream mode is introduced. The dual-stream mode includes a visual branch and a text branch.

[0101] The visual branch uses the visual transformation model as an image encoder. The untrained model corresponding to the visual transformation model is called the initial visual transformation model, which is started with random initialization or basic visual pre-trained weights.

[0102] The text branch introduces a text encoder based on the standard Transformer architecture. To significantly reduce the training cost of multimodal cold start and accelerate semantic space alignment, this embodiment does not train the text encoder from scratch. Instead, it extracts and reuses the pre-trained weights of a mature open-source large-scale graph-text model for text encoder initialization. This way, the text encoder possesses powerful natural language understanding and commonsense feature extraction capabilities from the start of training.

[0103] At the start of the training process, hyperparameters and training cycles are configured. Hyperparameters include, but are not limited to, dynamic learning decay strategies, and training cycles include, but are not limited to, the number of training epochs, which is a first preset number, etc.

[0104] During training, multiple first sample images and their annotation information for the current batch are acquired from the first sample set. The initial visual transformation model extracts the representation vector for each first sample image. The text encoder extracts the semantic vector for each annotation. Then, the representation vector and semantic vector are non-linearly projected to map them into a feature space of the same dimension.

[0105] Step S203: Based on the semantic vector and the representation vector, update the initial visual transformation model to obtain the intermediate visual transformation model.

[0106] In the current batch, each first sample image and its annotation information are positive sample pairs, and the annotation information of each first sample image and other first sample images are negative sample pairs.

[0107] The loss value is calculated by using the symmetric cross-entropy contrastive estimation loss function (InfoNCE Loss) to maximize the cosine similarity of positive sample pairs in the current batch while minimizing the similarity of all negative sample pairs in the current batch. The parameters of the initial visual transformation model are then updated based on the loss value to obtain the updated initial visual transformation model.

[0108] Then, the next batch is obtained and used as the current batch. Steps S202 to S203 are then executed again until the updated visual transformation model meets the performance requirements, or the first preset number of epochs of full training are completed. The updated visual transformation model is then determined as the intermediate visual transformation model. The intermediate visual transformation model is the visual transformation model obtained after the first stage of training. Its internal mask layer learns to adaptively ignore irrelevant backgrounds based on product text prompts. At the same time, the intermediate visual transformation model has extremely strong generalization feature extraction capabilities.

[0109] Step S204: Based on the second sample set, train the intermediate visual transformation model to obtain the visual transformation model.

[0110] While intermediate visual transformation models possess generalization capabilities, they should also exhibit extremely high "instance-level" fine-grained discrimination ability in subsequent real-world applications. In e-commerce scenarios, instances refer to different styles of products within the same category; in AI question-answering systems retrieving similar images, instances could be the appearance of the same animal at different age stages. Therefore, the second stage involves specific fine-tuning of the search ranking metrics. Because the training in this second stage is based on images of highly similar products of the same style or category within the application scenario, the visual transformation model gains fine-grained feature recognition capabilities.

[0111] The intermediate visual transformation model is retained, and its output is connected to a normalization feature layer customized for the image retrieval task, ensuring that the output vector is strictly distributed on the unit hypersphere. This normalization feature layer includes, but is not limited to, a Euclidean (L2) normalization layer.

[0112] Before training, configure the training cycle, such as the second preset number of epochs.

[0113] During training, the current batch is obtained from the second sample set, and the current batch includes positive sample pairs and negative sample pairs. Each second sample image in the current batch is input into the intermediate visual transformation model for forward inference to obtain the representation vector of each second sample image. The similarity (e.g., cosine similarity) of the representation vectors of every two second sample images in the current batch is calculated.

[0114] The loss is calculated using the Smooth Average Precision (Smooth-AP) loss function. For example, this Smooth-AP loss function utilizes a sigmoid activation function smooth approximation technique, replacing the non-differentiable indicator function with a differentiable continuous function. It performs a continuous smooth approximation on discrete ranking judgments where the similarity of negative sample pairs is higher than that of positive sample pairs (discrete ranking judgment refers to comparing sample similarity using a binary method; if the similarity of a negative sample pair is higher than that of a positive sample pair, it is judged as 1, otherwise as 0), resulting in a differentiable loss expression. By minimizing the Smooth-AP loss during backpropagation, the parameters of the intermediate visual transformation model are updated to continuously reduce the probability that the similarity of negative sample pairs exceeds that of positive sample pairs, thus reducing the occurrence of negative sample pairs ranking before positive sample pairs in the retrieval results. This allows the gradient update direction of the network during backpropagation to be aligned as closely as possible with the ranking accuracy of the global list.

[0115] After a second set number of epochs of smoothing and optimization, the visual transformation model minimizes the task gap between the classification pre-training task and the image retrieval task. When faced with 10,000 highly similar SPUs within the same category, the visual transformation model can accurately recall identical items, achieving significant breakthroughs in core retrieval metrics such as Normalized Discounted Cumulative Gain (NDCG), top-1 recall (Recall@1), and top-5 recall (Recall@5). NDCG measures the quality of the search ranking list. Recall@1 measures the proportion of the target object being the first item in the returned results list. Recall@5 refers to the proportion of the target object present in the top 5 results.

[0116] To further verify the effectiveness of feature representation in extremely complex backgrounds, a visual comparison test using feature heatmaps was conducted between the baseline model (Base) and the proposed visual transformation model. Figure 5As shown, in complex portrait shooting scenarios with severe occlusion and background interference, the attention weight distribution of traditional baseline models is extremely scattered, and a large number of non-product elements (such as the model's hands, face, and background shadows) generate erroneous high responses. In contrast, the visual transformation model of this application, with its built-in dynamic background masking mechanism, successfully performs spatial-level semantic truncation of the image. Its high-order semantic saliency highlighting area (feature focusing area) converges precisely and uniquely to the core visual features of the target product (hat) (such as brand logo (logotype, logo) and specific splicing texture), while the response weights of the hands and background areas that occlude the product are forcibly suppressed and approach zero. This comparative result provides intuitive visual evidence that the visual transformation model overcomes the structural defect of traditional retrieval models that are "susceptible to interference from non-subject elements".

[0117] It should be understood that Figure 5 This is just an example of a heatmap that can show the location of the feature focus, but it is not a true heatmap.

[0118] This application improves and optimizes the architecture and training method of the visual transformation model, enabling the model to possess better local feature extraction capabilities and resistance to background interference. The test results below demonstrate the significant advantages of this application: Test set validation: This application constructs a highly challenging test set, containing 259 query samples based on 5 stores, covering a wide range of categories such as apparel, shoes and bags, cosmetics, health products, stationery, jewelry and food.

[0119] Overall search performance has achieved a leapfrog breakthrough: NDCG index: increased from 0.729 at baseline to 0.957 (an increase of approximately 23%).

[0120] Recall@1 metric: improved from 0.616 at baseline to 0.896 (an improvement of approximately 28%).

[0121] The Recall@5 metric improved from a baseline of 0.659 to 0.950 (an improvement of approximately 29%).

[0122] The improvement in "highly difficult and complex" product categories is particularly noteworthy: For traditionally challenging product categories that are most susceptible to interference from non-product-related factors, the improved model in this application performs exceptionally well. Apparel category: Recall@1 increased by 44%.

[0123] Jewelry: Recall@1 increased by 57%.

[0124] Footwear / Bags: Recall@1 increased by 24%.

[0125] The above data fully demonstrates the disruptive effectiveness of this application in dealing with complex backgrounds and interference from non-main areas.

[0126] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0127] Based on the same technological concept, such as Figure 6 As shown in the illustration, this application also provides an image retrieval apparatus, which includes: The acquisition module 610 is used to acquire an image to be retrieved, wherein the image to be retrieved includes a target object; The feature extraction module 620 is used to: extract features from the image to be retrieved through the feature extraction layer in the visual transformation model to obtain a first abstract semantic feature of the image to be retrieved; process the first abstract semantic feature through the mask layer in the visual transformation model to obtain mask information of the image to be retrieved, wherein the mask information is used to mark the region where the target object is located; and process the first abstract semantic feature and the mask information through the feature extraction layer to obtain a representation vector of the image to be retrieved. The retrieval module 630 is used to determine an image that matches the target object based on the representation vector.

[0128] In one alternative approach, the feature extraction layer comprises N+1 visual transformation sublayers, where N is greater than 1, and the feature extraction module 620 is used for: The first abstract semantic feature is obtained by extracting features from the first N visual transformation sub-layers of the N+1 visual transformation sub-layers; the representation vector of the image to be retrieved is determined based on the last visual transformation sub-layer of the N+1 visual transformation sub-layers, the first abstract semantic feature, and the mask information.

[0129] In an alternative embodiment, the feature extraction layer further includes a feature fusion sublayer, wherein the feature extraction module 620 is used for: The first abstract semantic feature is processed based on the mask information through the last visual transformation sublayer to obtain the second abstract semantic feature; the second abstract semantic feature is fused with the feature output of at least one of the first N visual transformation sublayers through the feature fusion sublayer to obtain the representation vector.

[0130] In one alternative approach, the first abstract semantic feature includes spatial markers for each block of the image to be retrieved, and the feature extraction module 620 is used for: Based on the mask information, multi-head self-attention scores are calculated on the first abstract semantic feature to obtain an attention score matrix. During the multi-head self-attention score calculation, a preset negative penalty is applied to the attention scores of spatial markers identified as background. Based on the attention score matrix, the second abstract semantic feature is calculated.

[0131] In one alternative embodiment, the at least one visual transformation sublayer includes a first visual transformation sublayer and a second visual transformation sublayer, wherein the first visual transformation sublayer and the last visual transformation sublayer are separated from the second visual transformation sublayer by M visual transformation sublayers, where M is greater than 2.

[0132] In one alternative approach, the visual transformation sublayer includes a multi-head self-attention module and an activation module, wherein the activation module includes multiple feedforward activation units and stacking units; The feature extraction module 620 is used for: In the i-th visual transformation sub-layer among the first N visual transformation sub-layers, the output features of the (i-1)-th visual transformation layer are processed by the multi-head self-attention module to obtain the first residual feature, where i takes values ​​from 2 to N; The first residual feature is processed by the multiple feedforward activation units to obtain multiple fully connected features; The multiple fully connected features are processed by the superposition unit to obtain the output feature of the i-th visual transformation sub-layer, wherein the output feature of the last visual transformation sub-layer among the first N visual transformation sub-layers is the first abstract semantic feature.

[0133] In one alternative approach, the first abstract semantic feature includes a global state label (CLS) and a spatial label for each block of the image to be retrieved, and the feature extraction module 620 is used for: For each spatial marker in the first abstract semantic feature, the attention distribution weight of the CLS on the spatial marker in the multi-head self-attention layer is determined through the mask layer; The attention distribution weights are averaged through the mask layer to obtain the subject salient response value corresponding to the spatial label; The mask information is determined based on the relationship between the salient response value of the subject corresponding to each spatial marker and the truncation threshold through the mask layer.

[0134] In one alternative approach, the target object is a product; after determining the image matching the target object based on the representation vector, the retrieval module 630 is further configured to return a list of products to which the image matching the target object belongs.

[0135] In an alternative embodiment, the device further includes a training module for: A first sample set and a second sample set are obtained, wherein the first sample set includes multiple first sample images and annotation information of the first sample images, and the second sample set includes multiple second sample images and annotation information of the second sample images, wherein the multiple second sample images include images of each standardized product unit acquired under different image acquisition conditions; During the training process of the current batch, the annotation information of the first sample image of the current batch is encoded based on the text encoder to obtain the semantic vector of the first sample image of the current batch, and the first sample image of the current batch is processed based on the initial visual transformation model to obtain the representation vector of the first sample image of the current batch. Based on the semantic vector and the representation vector of the first sample image of the current batch, the initial visual transformation model is updated to obtain the intermediate visual transformation model. Based on the second sample set, the intermediate visual transformation model is trained to obtain the visual transformation model.

[0136] It should be noted that the image retrieval device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image retrieval device and the image retrieval method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0137] Based on the same technological concept, such as Figure 7 As shown in the embodiments of this application, an apparatus for training a visual transformation model is also provided, the apparatus comprising: The acquisition module 710 is used to acquire a first sample set and a second sample set, wherein the first sample set includes multiple first sample images and annotation information of the first sample images, and the second sample set includes multiple second sample images and annotation information of the second sample images, wherein the multiple second sample images include images of each standardized product unit acquired under different image acquisition conditions; The training module 720 is used to encode the annotation information of the first sample image of the current batch based on the text encoder during the training process of the current batch to obtain the semantic vector of the first sample image of the current batch, and to process the first sample image of the current batch based on the initial visual transformation model to obtain the representation vector of the first sample image of the current batch. Based on the semantic vector and the representation vector, the initial visual transformation model is updated to obtain an intermediate visual transformation model; Based on the second sample set, the intermediate visual transformation model is trained to obtain the visual transformation model.

[0138] It should be noted that the apparatus for training the visual transformation model provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus for training the visual transformation model provided in the above embodiments and the method embodiments for training the visual transformation model belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0139] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 800 can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 801 and one or more memories 802. The memory 802 stores at least one instruction, which is loaded and executed by the processor 801 to implement the image retrieval method or the visual transformation model training method provided in the various method embodiments described above. Of course, the computer device may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The computer device may also include other components for implementing device functions, which will not be elaborated here.

[0140] Optionally, the processor 801 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0141] Memory 802 may include volatile memory, such as random access memory (RAM). Memory 802 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0142] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a computer device to perform the image retrieval method or the visual transformation model training method in the above embodiments. The computer-readable storage medium may be non-transitory. For example, the computer-readable storage medium may be ROM, RAM, compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0143] In an exemplary embodiment, a computer program product is also provided, which stores at least one instruction that is loaded and executed by a processor to implement the operations performed by the image retrieval method or the training method of the visual transformation model described above.

[0144] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample sets involved in this application were all obtained under full authorization.

[0145] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0146] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for image retrieval, characterized in that, The method includes: Obtain the image to be retrieved, wherein the image to be retrieved includes the target object; The first abstract semantic features of the image to be retrieved are obtained by extracting features from the feature extraction layer in the visual transformation model. The first abstract semantic features are processed by the mask layer in the visual transformation model to obtain the mask information of the image to be retrieved, wherein the mask information is used to mark the region where the target object is located; The first abstract semantic features and the mask information are processed by the feature extraction layer to obtain the representation vector of the image to be retrieved; Based on the representation vector, an image matching the target object is determined.

2. The method according to claim 1, characterized in that, The feature extraction layer includes N+1 visual transformation sublayers, where N is greater than 1. The feature extraction layer in the visual transformation model extracts features from the image to be retrieved, obtaining the first abstract semantic features of the image to be retrieved, including: The first abstract semantic feature is obtained by extracting features from the image to be retrieved through the first N visual transformation sub-layers of the N+1 visual transformation sub-layers. The step of processing the first abstract semantic features and the mask information through the feature extraction layer to obtain the representation vector of the image to be retrieved includes: Based on the last visual transformation sublayer among the N+1 visual transformation sublayers, the first abstract semantic feature, and the mask information, the representation vector of the image to be retrieved is determined.

3. The method according to claim 2, characterized in that, The feature extraction layer further includes a feature fusion sublayer. The step of determining the representation vector of the image to be retrieved based on the last visual transformation sublayer among the N+1 visual transformation sublayers, the first abstract semantic feature, and the mask information includes: The first abstract semantic feature is processed based on the mask information through the last visual transformation sub-layer to obtain the second abstract semantic feature; The feature fusion sublayer is used to fuse the second abstract semantic feature with the features output by at least one of the first N visual transformation sublayers to obtain the representation vector.

4. The method according to claim 3, characterized in that, The first abstract semantic feature includes the spatial label of each block of the image to be retrieved. The second abstract semantic feature is obtained by processing the first abstract semantic feature based on the mask information, including: Based on the mask information, multi-head self-attention scores are calculated on the first abstract semantic features to obtain an attention score matrix. In the process of multi-head self-attention score calculation, a preset negative penalty is applied to the attention scores of spatial markers identified as backgrounds participating in the calculation. The second abstract semantic feature is calculated based on the attention score matrix.

5. The method according to claim 3 or 4, characterized in that, The at least one visual transformation sublayer includes a first visual transformation sublayer and a second visual transformation sublayer. The first visual transformation sublayer and the last visual transformation sublayer are separated from the second visual transformation sublayer by M visual transformation sublayers, where M is greater than 2.

6. The method according to any one of claims 2 to 4, characterized in that, The visual transformation sublayer includes a multi-head self-attention module and an activation module. The activation module includes multiple feedforward activation units and stacking units. The first abstract semantic feature is obtained by extracting features from the image to be retrieved through the first N visual transformation sub-layers out of the N+1 visual transformation sub-layers, including: In the i-th visual transformation sub-layer among the first N visual transformation sub-layers, the output features of the (i-1)-th visual transformation layer are processed by the multi-head self-attention module to obtain the first residual feature, where i takes values ​​from 2 to N; The first residual feature is processed by the multiple feedforward activation units to obtain multiple fully connected features; The multiple fully connected features are processed by the superposition unit to obtain the output feature of the i-th visual transformation sub-layer, wherein the output feature of the last visual transformation sub-layer among the first N visual transformation sub-layers is the first abstract semantic feature.

7. The method according to any one of claims 1 to 4, characterized in that, The first abstract semantic feature includes a global state label (CLS) and a spatial label for each block of the image to be retrieved. The process of processing the first abstract semantic feature through a mask layer in the visual transformation model to obtain the mask information of the image to be retrieved includes: For each spatial marker in the first abstract semantic feature, the attention distribution weight of the CLS on the spatial marker in the multi-head self-attention layer is determined through the mask layer; The attention distribution weights are averaged through the mask layer to obtain the subject salient response value corresponding to the spatial label; The mask information is determined based on the relationship between the salient response value of the subject corresponding to each spatial marker and the truncation threshold through the mask layer.

8. The method according to any one of claims 1 to 4, characterized in that, The target object is a commodity; after determining the image matching the target object based on the representation vector, the method further includes: Returns a list of products to which the image matching the target object belongs.

9. The method according to any one of claims 1 to 4, characterized in that, The method further includes: A first sample set and a second sample set are obtained, wherein the first sample set includes multiple first sample images and annotation information of the first sample images, and the second sample set includes multiple second sample images and annotation information of the second sample images, wherein the multiple second sample images include images of each standardized product unit acquired under different image acquisition conditions; During the training process of the current batch, the annotation information of the first sample image of the current batch is encoded based on the text encoder to obtain the semantic vector of the first sample image of the current batch, and the first sample image of the current batch is processed based on the initial visual transformation model to obtain the representation vector of the first sample image of the current batch. Based on the semantic vector and the representation vector of the first sample image of the current batch, the initial visual transformation model is updated to obtain the intermediate visual transformation model. Based on the second sample set, the intermediate visual transformation model is trained to obtain the visual transformation model.

10. A training method for a visual transformation model, characterized in that, The visual transformation model is the visual transformation model according to any one of claims 1 to 9, and the method includes: A first sample set and a second sample set are obtained, wherein the first sample set includes multiple first sample images and annotation information of the first sample images, and the second sample set includes multiple second sample images and annotation information of the second sample images, wherein the multiple second sample images include images of each standardized product unit acquired under different image acquisition conditions; During the training process of the current batch, the annotation information of the first sample image of the current batch is encoded based on the text encoder to obtain the semantic vector of the first sample image of the current batch, and the first sample image of the current batch is processed based on the initial visual transformation model to obtain the representation vector of the first sample image of the current batch. Based on the semantic vector and the representation vector of the first sample image of the current batch, the initial visual transformation model is updated to obtain the intermediate visual transformation model. Based on the second sample set, the intermediate visual transformation model is trained to obtain the visual transformation model.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction that is loaded and executed by the processor to perform the operations performed by the image retrieval method as claimed in any one of claims 1 to 9 or the training method for the visual transformation model as claimed in claim 10.

12. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to perform the operations performed by the image retrieval method as described in any one of claims 1 to 9 or the training method for the visual transformation model as described in claim 10.

13. A computer program product, characterized in that, The computer program product stores at least one instruction, which is loaded and executed by a processor to perform the operations performed by the image retrieval method as described in any one of claims 1 to 9 or the training method for the visual transformation model as described in claim 10.

Images