Image determination method, apparatus, electronic device, and computer readable medium

Abstract

Embodiments of the present disclosure disclose image determination methods, devices, electronic equipment and computer readable media. A specific implementation of the method includes: performing image preprocessing on a to-be-searched item image to obtain a preprocessed vector; inputting the preprocessed vector into an attention mechanism network model in an image representation vector generation model for extracting multi-level features to output a set of hierarchical feature vectors; inputting the set of hierarchical feature vectors into a multi-level feature fusion model to output a set of multi-level fusion feature vectors; generating an image representation vector corresponding to the to-be-searched item image according to the set of multi-level fusion feature vectors, as a target image representation vector; and determining, from an item image library, an item image corresponding to an image representation vector that satisfies a preset condition in similarity with the target image representation vector, as a consistent item image. The implementation is related to artificial intelligence, and can accurately determine an item image matching the to-be-searched item image from the item image library.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to image determination methods, apparatus, electronic devices, and computer-readable media. Background Technology

[0002] Currently, the development of product image retrieval has not only provided convenience for users but also promoted the development of e-commerce on mobile devices. The typical approach for image retrieval is as follows: First, a convolutional neural network is used as the backbone network for image feature extraction to calculate the feature similarity between images. Then, the product image with the highest feature similarity to the first image is retrieved from the product image database and designated as the matching image.

[0003] However, the inventors discovered that when using the above method to retrieve images, the following technical problems often arise:

[0004] The image feature extraction capability is weak, resulting in weak feature representation ability of the extracted features, which leads to inaccurate retrieved images.

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure provide image determination methods, apparatuses, electronic devices, and computer-readable media to address the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of this disclosure provide an image determination method, comprising: preprocessing an image of an item to be retrieved to obtain a preprocessed vector; inputting the preprocessed vector into an attention mechanism network model in an image representation vector generation model for extracting multi-level features to output a hierarchical feature vector set, wherein the feature information depths corresponding to the various hierarchical feature vectors in the hierarchical feature vector set are different; inputting the hierarchical feature vector set into a multi-level feature fusion model in the image representation vector generation model to output a multi-level fused feature vector set; generating an image representation vector corresponding to the image of the item to be retrieved based on the multi-level fused feature vector set using the image representation vector generation model, as a target image representation vector; and selecting item images from an item image library whose similarity between the corresponding image representation vector and the target image representation vector meets a preset condition, as matching item images.

[0009] Optionally, the aforementioned attention mechanism network model includes: at least one attention mechanism layer, at least one first feature encoding layer, at least one second feature encoding layer, and at least one third feature encoding layer; and the aforementioned attention mechanism network model for extracting multi-level features by inputting the aforementioned preprocessed vector into the image representation vector generation model to output a hierarchical feature vector set, comprising: inputting the aforementioned preprocessed vector into the aforementioned at least one attention mechanism layer to obtain an attention vector; inputting the aforementioned attention vector into the aforementioned at least one first feature encoding layer to output a first feature encoding vector; generating a low-level feature vector based on the aforementioned first feature encoding vector as a hierarchical feature vector; inputting the aforementioned first feature encoding vector into the aforementioned at least one second feature encoding layer to output a second feature encoding vector; generating a mid-level feature vector based on the aforementioned second feature encoding vector as a hierarchical feature vector; inputting the aforementioned second feature encoding vector into the aforementioned at least one third feature encoding layer to output a third feature encoding vector; and generating a high-level feature vector based on the aforementioned third feature encoding vector as a hierarchical feature vector.

[0010] Optionally, the multi-level feature fusion model includes: a multi-level feature fusion layer and multiple fusion feature processing layers; and the multi-level feature fusion model that inputs the hierarchical feature vector set into the image representation vector generation model to output a multi-level fusion feature vector set includes: inputting the hierarchical feature vector set into the multi-level feature fusion layer to output a feature fusion vector set; and inputting the feature fusion vector set into the multiple fusion feature processing layers to output the multi-level fusion feature vector set.

[0011] Optionally, the fusion feature processing layers in the aforementioned fusion feature processing layer set include: convolutional layers, pooling layers, random deactivation layers, and fully connected layers; and the inputting of the aforementioned feature fusion vector set to the aforementioned plurality of fusion feature processing layers to output the aforementioned multi-level fusion feature vector set includes: for each feature fusion vector in the aforementioned feature fusion vector set, performing a first processing step: inputting the aforementioned feature fusion vector to the convolutional layer included in the corresponding fusion feature processing layer to output a convolutional vector; inputting the aforementioned convolutional vector to the pooling layer included in the corresponding fusion feature processing layer to output a pooling vector; inputting the aforementioned pooling vector to the random deactivation layer included in the corresponding fusion feature processing layer to output a random deactivation vector; inputting the aforementioned random deactivation vector to the fully connected layer included in the corresponding fusion feature processing layer to output a fully connected vector; determining the feature level information corresponding to each feature fusion vector in the aforementioned feature fusion vector set, wherein the feature level information characterizes the feature extraction depth of the features included in the feature fusion vector; and generating the aforementioned multi-level fusion feature vector set based on the feature level information corresponding to each feature fusion vector and the obtained fully connected vector set.

[0012] Optionally, the aforementioned fusion feature processing layer further includes: a batch normalization layer set; and the generation of the aforementioned multi-level fusion feature vector set based on the feature hierarchy information corresponding to each feature fusion vector and the obtained fully connected vector set includes: sorting the fully connected vector set according to the feature hierarchy information corresponding to each feature fusion vector to obtain a fully connected vector sequence; for each fully connected vector in the fully connected vector sequence, performing a second processing step: determining the feature hierarchy information corresponding to the fully connected vector as the target feature hierarchy information; in response to determining that the fully connected vector is not a target feature in the fully connected vector sequence... The fully connected vector at a given location is identified, and there exists a fully connected vector that is adjacent to the target fully connected vector and whose corresponding feature level information is higher than the target feature level information. This fully connected vector and its adjacent fully connected vector with higher corresponding feature level information are then identified as the target fully connected vector. The fully connected vector corresponding to the target fully connected vector is then concatenated with the target fully connected vector to obtain a concatenated vector. For each concatenated vector in the obtained concatenated vector set, the concatenated vector is input into the corresponding batch normalization layer in the batch normalization layer set to obtain a batch normalized vector, which serves as the multi-level fusion feature vector.

[0013] Optionally, generating the image representation vector corresponding to the image of the item to be retrieved, based on the multi-level fused feature vector set, as the target image representation vector, includes: inputting the preprocessed vector into at least one serially connected attention mechanism layer to obtain an attention vector, wherein the attention mechanism layer includes: at least two normalization layers, an attention layer, at least two random deactivation layers, and multiple support vector machine layers.

[0014] Optionally, the above-mentioned image preprocessing of the image of the item to be retrieved to obtain a preprocessing vector includes: adjusting the image size of the image of the item to be retrieved to obtain an adjusted image; and performing a linear convolution transformation on the adjusted image to output an image of the target dimension as a preprocessing vector.

[0015] Optionally, at least one loss function corresponding to the above image representation vector generation model includes: a multiple similarity loss function and a second-order relation loss function.

[0016] Secondly, some embodiments of this disclosure provide an image determination apparatus, comprising: a preprocessing unit configured to perform image preprocessing on an image of an item to be retrieved to obtain a preprocessed vector; a first input unit configured to input the preprocessed vector into an attention mechanism network model in an image representation vector generation model for extracting multi-level features, to output a hierarchical feature vector set, wherein the feature information depths corresponding to the various hierarchical feature vectors in the hierarchical feature vector set are different; a second input unit configured to input the hierarchical feature vector set into a multi-level feature fusion model in the image representation vector generation model, to output a multi-level fused feature vector set; a generation unit configured to generate an image representation vector corresponding to the image of the item to be retrieved based on the multi-level fused feature vector set and using the image representation vector generation model, as a target image representation vector; and a determination unit configured to determine, from an item image library, an item image whose similarity to the target image representation vector satisfies a preset condition, as a matching item image.

[0017] Optionally, the above-mentioned attention mechanism network model includes: at least one attention mechanism layer, at least one first feature encoding layer, at least one second feature encoding layer, and at least one third feature encoding layer; and the first input unit can be configured to: input the preprocessed vector into the at least one attention mechanism layer to obtain an attention vector; input the attention vector into the at least one first feature encoding layer to output a first feature encoding vector; generate a low-level feature vector based on the first feature encoding vector as a hierarchical feature vector; input the first feature encoding vector into the at least one second feature encoding layer to output a second feature encoding vector; generate a mid-level feature vector based on the second feature encoding vector as a hierarchical feature vector; input the second feature encoding vector into the at least one third feature encoding layer to output a third feature encoding vector; and generate a high-level feature vector based on the third feature encoding vector as a hierarchical feature vector.

[0018] Optionally, the above-mentioned multi-level feature fusion model includes: a multi-level feature fusion layer and multiple fusion feature processing layers; and the second input unit can be configured to: input the above-mentioned hierarchical feature vector set to the above-mentioned multi-level feature fusion layer to output a feature fusion vector set; and input the above-mentioned feature fusion vector set to the above-mentioned multiple fusion feature processing layers to output the above-mentioned multi-level fusion feature vector set.

[0019] Optionally, the fusion feature processing layers in the aforementioned fusion feature processing layer set include: convolutional layers, pooling layers, random deactivation layers, and fully connected layers; and the second input unit can be configured to: for each feature fusion vector in the aforementioned feature fusion vector set, perform a first processing step: input the aforementioned feature fusion vector to the convolutional layer included in the corresponding fusion feature processing layer to output a convolutional vector; input the aforementioned convolutional vector to the pooling layer included in the corresponding fusion feature processing layer to output a pooling vector; input the aforementioned pooling vector to the random deactivation layer included in the corresponding fusion feature processing layer to output a random deactivation vector; input the aforementioned random deactivation vector to the fully connected layer included in the corresponding fusion feature processing layer to output a fully connected vector; determine the feature level information corresponding to each feature fusion vector in the aforementioned feature fusion vector set, wherein the feature level information characterizes the feature extraction depth of the features included in the feature fusion vector; and generate the aforementioned multi-level fusion feature vector set based on the feature level information corresponding to each feature fusion vector and the obtained fully connected vector set.

[0020] Optionally, the aforementioned fusion feature processing layer further includes: a batch normalization layer set; and the second input unit can be configured to: sort the aforementioned fully connected vector set according to the feature level information corresponding to each feature fusion vector to obtain a fully connected vector sequence; for each fully connected vector in the aforementioned fully connected vector sequence, perform a second processing step: determine the feature level information corresponding to the aforementioned fully connected vector as the target feature level information; in response to determining that the aforementioned fully connected vector is not a fully connected vector at the target position in the aforementioned fully connected vector sequence, and that there exists a fully connected vector and a neighboring fully connected vector whose corresponding feature level information is higher than the target feature level information, determine the aforementioned fully connected vector and the neighboring fully connected vector whose corresponding feature level information is higher than the target feature level information as the target fully connected vector; concatenate the fully connected vector corresponding to the target fully connected vector with the aforementioned fully connected vector to obtain a concatenated vector; for each concatenated vector in the obtained concatenated vector set, input the concatenated vector to the corresponding batch normalization layer in the aforementioned batch normalization layer set to obtain a batch normalized vector, which serves as a multi-level fusion feature vector.

[0021] Optionally, the generation unit can be configured to: input the preprocessed vector above to at least one serially connected attention mechanism layer to obtain an attention vector, wherein the attention mechanism layer includes: at least two normalization layers, an attention layer, at least two random deactivation layers, and multiple support vector machine layers.

[0022] Optionally, the preprocessing unit can be configured to: resize the image of the item to be retrieved to obtain an adjusted image; and perform a linear convolution transformation on the adjusted image to output an image of the target dimension as a preprocessing vector.

[0023] Optionally, at least one loss function corresponding to the above image representation vector generation model includes: a multiple similarity loss function and a second-order relation loss function.

[0024] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0025] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0026] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0027] The above embodiments of this disclosure have the following beneficial effects: the image determination method of some embodiments of this disclosure can accurately determine the item image matching the item image to be retrieved from the item image library. Specifically, the reason for the inaccuracy of image retrieval is that the image feature extraction capability is weak, resulting in weak feature representation capability of the extracted features, leading to inaccurate retrieved images. Based on this, the image determination method of some embodiments of this disclosure firstly preprocesses the item image to be retrieved to obtain a preprocessed vector. Here, by preprocessing the item image to be retrieved, not only can the main content information of the item in the image be enhanced, but it is also convenient to input it into the attention mechanism network model. Then, the above preprocessed vector is input into the attention mechanism network model in the image representation vector generation model for extracting multi-level features to output a hierarchical feature vector set, wherein the feature information depths corresponding to the various hierarchical feature vectors in the above hierarchical feature vector set are different. Here, through the attention mechanism network model, multi-level and richer image features can be extracted from the preprocessed vector. Next, the above hierarchical feature vector set is input into the multi-level feature fusion model in the image representation vector generation model to output a multi-level fused feature vector set. Here, a multi-level feature fusion model is used to fuse features from different levels, which can better balance the feature information at different levels, making the feature representation vector corresponding to the image of the item to be retrieved more expressive. Furthermore, based on the multi-level fused feature vector set, the image representation vector generation model can accurately generate the image representation vector corresponding to the image of the item to be retrieved, serving as the target image representation vector. Here, the multi-level fused feature vector set can fully utilize the multifaceted feature information of the image, giving the obtained target image representation vector a strong feature representation capability. Finally, the item images whose similarity to the target image representation vector meets the preset conditions are accurately determined from the item image database and are considered matching item images. In summary, through the attention mechanism network model and the multi-level feature fusion model, not only can multi-level image feature information be extracted, but the feature representation capability of the image representation vector corresponding to the image of the item to be retrieved can also be improved through the fusion of feature information between multi-level features, enabling the accurate identification of item images from the item image database using the target image representation vector. Attached Figure Description

[0028] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0029] Figure 1This is a schematic diagram of an application scenario of an image determination method according to some embodiments of the present disclosure;

[0030] Figure 2 This is a flowchart of some embodiments of the image determination method according to the present disclosure;

[0031] Figure 3 These are flowcharts of other embodiments of the image determination method according to this disclosure;

[0032] Figure 4 This is a schematic diagram of the attention mechanism layer in some embodiments of the image determination method according to the present disclosure;

[0033] Figure 5 These are schematic diagrams illustrating the structure of some embodiments of the image determining apparatus according to this disclosure;

[0034] Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0035] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0036] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0037] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0038] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0039] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0040] Before performing any of the operations related to the collection, storage, and use of item information (such as item portraits) disclosed in this disclosure, the relevant organizations or individuals shall fulfill their obligations, including conducting an item information security impact assessment, informing the item information subject, and obtaining the item information subject's prior authorization and consent.

[0041] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] Figure 1 This is a schematic diagram of an application scenario of an image determination method according to some embodiments of the present disclosure.

[0043] exist Figure 1In this application scenario, firstly, the electronic device 101 can perform image preprocessing on the image 102 of the item to be retrieved, obtaining a preprocessed vector 103. In this application scenario, the item corresponding to the image 102 of the item to be retrieved is "Item A". The preprocessed vector 103 can be "[1,23,45,…,1,3]". Then, the electronic device 101 can input the above preprocessed vector 103 into the attention mechanism network model 104 in the image representation vector generation model, which is used to extract multi-level features, to output a hierarchical feature vector set 105. Among them, the feature information depths corresponding to the various hierarchical feature vectors in the hierarchical feature vector set 105 are different. In this application scenario, the hierarchical feature vector set 105 includes: a hierarchical feature vector 1051 corresponding to the first level feature, a hierarchical feature vector 1052 corresponding to the second level feature, and a hierarchical feature vector 1053 corresponding to the third level feature. The hierarchical feature vector 1051 can be "[3,4,2],[45,2,2],[3,5,1]". Hierarchical feature vector 1052 can be “[7,1,2],[0,21,4],[0,1,1]”. Hierarchical feature vector 1053 can be “[8,41,2],[5,0,2],[2,5,2]”. Next, electronic device 101 can input the above hierarchical feature vector set 105 into the multi-level feature fusion model 106 in the image representation vector generation model to output a multi-level fused feature vector set 107. In this application scenario, the multi-level fused feature vector set 107 can include: multi-level fused feature vector 1071, multi-level fused feature vector 1072, and multi-level fused feature vector 1073. Multi-level fused feature vector 1071 can be “[23,14,2,3],[45,2,12,2]”. Multi-level fused feature vector 1072 can be “[8,2,2,6],[30,21,14,42]”. The multi-level fusion feature vector 1073 can be "[82,4,2,12],[21,5,4,24]". Furthermore, the electronic device 101 can generate the image representation vector corresponding to the item image 102 to be retrieved, based on the multi-level fusion feature vector set 107 and the image representation vector generation model, as the target image representation vector 108. In this application scenario, the target image representation vector 108 can be "[23,114,3],[415,112,22]". Finally, the electronic device 101 can determine from the item image library 110 the item image whose similarity to the target image representation vector 108 meets a preset condition, as the matching item image 109.

[0044] It should be noted that the aforementioned electronic device 101 can be either hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0045] It should be understood that Figure 1 The number of electronic devices shown is merely illustrative. Any number of electronic devices can be used depending on the implementation requirements.

[0046] Continue to refer to Figure 2 The flowchart 200 illustrates some embodiments of an image determination method according to the present disclosure. The image determination method includes the following steps:

[0047] Step 201: Perform image preprocessing on the image of the item to be retrieved to obtain a preprocessed vector.

[0048] In some embodiments, the execution subject of the above image determination method (e.g.) Figure 1 The electronic device 101 shown can perform image preprocessing on the image of the item to be retrieved, obtaining a preprocessed vector. The image of the item to be retrieved can be an image of an item from an image library from which a matching image is to be retrieved. In practice, the image of the item to be retrieved can be a product image. For example, the image of the item to be retrieved can be an image of a top from a target brand. The image library stores multiple image of items with clearly defined related information. This related information can include, but is not limited to, at least one of the following: item name, item category, item price, and item sales volume.

[0049] As an example, the aforementioned execution entity can perform image smoothing on the item to be retrieved to obtain a preprocessed vector.

[0050] In some optional implementations of certain embodiments, the above-described image preprocessing of the image of the item to be retrieved to obtain a preprocessed vector may include the following steps:

[0051] The first step is to adjust the image size of the above-mentioned item image to obtain the adjusted image.

[0052] As an example, the aforementioned execution entity can adjust the image size corresponding to the image of the item to be retrieved to a predetermined size, thereby obtaining an adjusted image. The predetermined size is generated based on the size of the image samples in the input training image sample set of the attention mechanism network model.

[0053] The second step involves performing a linear convolution transformation on the adjusted image to output an image of the target dimension, which serves as the preprocessing vector. The target dimension can be one-dimensional. In other words, the adjusted image is converted into a one-dimensional vector.

[0054] As an example, the aforementioned execution entity can use a convolution kernel of a predetermined size and a predetermined stride to perform linear convolution on the integrated image to generate a one-dimensional vector.

[0055] Step 202: Input the preprocessed vector into the attention mechanism network model in the image representation vector generation model, which is used to extract multi-level features, to output a set of hierarchical feature vectors.

[0056] In some embodiments, the execution entity can input the preprocessed vectors into an attention mechanism network model in the image representation vector generation model, which is used to extract multi-level features, to output a hierarchical feature vector set. The attention mechanism network model can be a network model that extracts multi-level feature information through an attention mechanism. In practice, the attention mechanism network model can be multiple sequentially connected Transformer models. The feature information depths corresponding to the feature vectors at different levels in the hierarchical feature vector set are different. The image representation vector generation model can be a neural network model that generates image representation vectors. In practice, the image representation vector generation model can be multiple sequentially connected Transformer models plus multiple sequentially connected convolutional neural networks.

[0057] It is important to emphasize that the hierarchical feature vectors in the hierarchical feature vector set represent the image hierarchical feature information in the preprocessed vectors.

[0058] In some optional implementations of certain embodiments, at least one loss function corresponding to the above-mentioned image representation vector generation model includes: a multi-similarity loss function and a second-order loss function. The loss value corresponding to the image representation vector generation model is a weighted sum of the first loss value output by the multi-similarity loss function and the second loss value output by the second-order loss function.

[0059] Step 203: Input the hierarchical feature vector set into the multi-level feature fusion model in the image representation vector generation model to output a multi-level fused feature vector set.

[0060] In some embodiments, the execution entity may input the hierarchical feature vector set into a multi-level feature fusion model in the image representation vector generation model to output a multi-level fused feature vector set. The multi-level feature fusion model may be a model that fuses the feature vectors at each level of the hierarchical feature vector set.

[0061] As an example, firstly, the aforementioned execution entity can utilize the multi-level feature fusion model in the image representation vector generation model to perform vector classification on the hierarchical feature vectors in the hierarchical feature vector set according to the depth of the corresponding feature information, thereby obtaining at least one set of hierarchical feature vectors. Then, the aforementioned execution entity can utilize the multi-level feature fusion model to concatenate each hierarchical feature vector group in the at least one set of hierarchical feature vectors, thereby obtaining a hierarchical concatenated vector group, which serves as the multi-level fused feature vector set.

[0062] In some optional implementations of certain embodiments, the above-described multi-level feature fusion model includes: a multi-level feature fusion layer and multiple fusion feature processing layers. The multi-level feature fusion layer can be a network layer that fuses feature vectors from various levels within a set of hierarchical feature vectors. In practice, the multi-level feature fusion layer can be a BiFeature Pyramid Network (BiFPN). The fusion feature processing layer in the multiple fusion feature processing layers can be a network layer that performs feature processing on the output vector of the multi-level feature fusion layer. For example, the fusion feature processing layer can include: multiple sequentially connected convolutional layers + fully connected layers.

[0063] Optionally, inputting the aforementioned hierarchical feature vector set into the multi-level feature fusion model in the image representation vector generation model to output a multi-level fused feature vector set may include the following steps:

[0064] The first step is to input the above-mentioned hierarchical feature vector set into the above-mentioned multi-level feature fusion layer to output the feature fusion vector set.

[0065] The second step involves inputting the aforementioned feature fusion vector set into the multiple fusion feature processing layers to output the aforementioned multi-level fusion feature vector set. There is a one-to-one correspondence between the feature fusion vectors in the feature fusion vector set and the multi-level fusion feature vectors in the multi-level fusion feature vector set.

[0066] As an example, the aforementioned execution entity can input the aforementioned feature fusion vector set into the aforementioned multiple serially connected fusion feature processing layers to output the aforementioned multi-level fusion feature vector set.

[0067] Optionally, the fusion feature processing layers in the aforementioned set of fusion feature processing layers include: convolutional layers, pooling layers, dropout layers, and fully connected layers. In practice, the pooling layers can be generalized mean pooling layers (GeM layers). There is a one-to-one correspondence between the fusion feature processing layers in the aforementioned set of fusion feature processing layers and the feature fusion vectors in the feature fusion vector set.

[0068] Optionally, inputting the aforementioned feature fusion vector set into the aforementioned multiple fusion feature processing layers to output the aforementioned multi-level fusion feature vector set may include the following steps:

[0069] The first step is to perform the first processing step for each feature fusion vector in the above feature fusion vector set:

[0070] Sub-step 1 involves inputting the aforementioned feature fusion vector into the convolutional layer included in the corresponding fusion feature processing layer to output a convolutional vector.

[0071] Sub-step 2 involves inputting the convolution vector into the pooling layer included in the corresponding fusion feature processing layer to output a pooling vector.

[0072] Sub-step 3 involves inputting the pooling vector into the random deactivation layer included in the corresponding fusion feature processing layer to output a random deactivation vector.

[0073] Sub-step 4 involves inputting the aforementioned random deactivation vector into the fully connected layer included in the corresponding fusion feature processing layer to output a fully connected vector.

[0074] The second step is to determine the feature level information corresponding to each feature fusion vector in the aforementioned feature fusion vector set. The feature level information represents the feature extraction depth of the features included in the feature fusion vector. The feature level information can be, but is not limited to, one of the following: high-level information, mid-level information, and low-level information. The feature information corresponding to high-level information is deeper and more detailed than the feature information corresponding to mid-level information. Similarly, the feature information corresponding to mid-level information is deeper and more detailed than the feature information corresponding to low-level information.

[0075] The third step is to generate the multi-level fusion feature vector set mentioned above based on the feature hierarchy information corresponding to each feature fusion vector and the obtained fully connected vector set.

[0076] As an example, firstly, the aforementioned execution entity can sort the fully connected vector set according to the feature hierarchy information corresponding to each feature fusion vector to obtain a fully connected vector sequence. Next, every two adjacent fully connected vectors in the fully connected vector sequence are concatenated to generate a fully connected concatenated vector, resulting in a fully connected concatenated vector set, which serves as the multi-level fusion feature vector set.

[0077] Optionally, the fusion feature processing layer set mentioned above may also include a batch normalization layer.

[0078] Optionally, generating the multi-level fused feature vector set based on the feature hierarchy information corresponding to each feature fusion vector and the obtained fully connected vector set may include the following steps:

[0079] The first step is to sort the fully connected vector set according to the feature hierarchy information corresponding to each feature fusion vector to obtain the fully connected vector sequence.

[0080] As an example, the aforementioned execution entity can sort the fully connected vector set according to the feature level information from high to low to obtain a fully connected vector sequence.

[0081] The second step is to perform a second processing step for each fully connected vector in the above fully connected vector sequence:

[0082] Sub-step 1: Determine the feature hierarchy information corresponding to the fully connected vectors mentioned above, and use it as the target feature hierarchy information.

[0083] As an example, firstly, the aforementioned execution entity can determine the feature fusion vector corresponding to the fully connected vector, as the target feature fusion vector. Then, the aforementioned execution entity can determine the feature hierarchy information corresponding to the target feature fusion vector, as the target feature hierarchy information.

[0084] Sub-step 2: In response to determining that the fully connected vector is not a fully connected vector at the target position in the fully connected vector sequence, and that there exists a fully connected vector that is adjacent to the fully connected vector and whose corresponding feature level information is higher than the target feature level information, the fully connected vector that is adjacent to the fully connected vector and whose corresponding feature level information is higher than the target feature level information is determined as the target fully connected vector.

[0085] Among them, the fully connected vector at the target position corresponds to the lowest feature level information. In practice, if the fully connected vector sequence is sorted in descending order of feature level, the target position is the vector position of the last fully connected vector in the sequence. Therefore, the target fully connected vector is the second-to-last fully connected vector in the sequence. If the fully connected vector sequence is sorted in ascending order of feature level, the target position is the vector position of the first fully connected vector in the sequence. Therefore, the target fully connected vector is the second fully connected vector in the sequence.

[0086] Sub-step 3 involves concatenating the fully connected vector corresponding to the target fully connected vector with the aforementioned fully connected vector to obtain the concatenated vector.

[0087] The third step is to input each spliced ​​vector in the obtained spliced ​​vector set into the batch normalization layer in the corresponding fusion feature processing layer to obtain a batch normalized vector, which serves as a multi-level fusion feature vector.

[0088] Step 204: Based on the multi-level fusion feature vector set, use the image representation vector generation model to generate the image representation vector corresponding to the image of the item to be retrieved, and use it as the target image representation vector.

[0089] In some embodiments, the execution entity may generate an image representation vector corresponding to the image of the item to be retrieved, based on the multi-level fused feature vector set, as the target image representation vector. The target image representation vector represents the image feature information of the image of the item to be retrieved.

[0090] As an example, the aforementioned execution entity can input the multi-level fused feature vector set into the vector generation sub-model in the image representation vector generation model to output the image representation vector corresponding to the image of the item to be retrieved. The vector generation sub-model can be a model that generates image representation vectors. The input to the vector generation sub-model is the multi-level fused feature information of the image (i.e., the multi-level fused feature vector set). The aforementioned vector generation sub-model can be a Convolutional Neural Network (CNN) model.

[0091] As another example, the aforementioned execution entity can utilize the vector splicing sub-model included in the aforementioned image representation vector generation model to splice the various multi-level fusion vectors in the aforementioned multi-level fusion feature vector set to obtain the target image representation vector.

[0092] Step 205: Identify from the item image library the item images whose similarity between the corresponding image representation vector and the target image representation vector meets the preset conditions, and use them as matching item images.

[0093] In some embodiments, the execution entity may determine, from the item image library, an item image whose similarity to the target image representation vector satisfies a preset condition, and use it as a matching item image. The preset condition may be that the item image is the image with the highest similarity in the item image library.

[0094] As an example, firstly, the aforementioned execution entity can obtain the image representation vector library corresponding to the item image library. There is a one-to-one correspondence between the item images in the item image library and the image representation vectors in the image representation vector library. Then, the image representation vector with the smallest cosine distance to the target image representation vector is selected from the image representation vector library as a candidate image representation vector. Finally, the item image corresponding to the candidate image representation vector is determined as the matching item image.

[0095] The above embodiments of this disclosure have the following beneficial effects: the image determination method of some embodiments of this disclosure can accurately determine the item image matching the item image to be retrieved from the item image library. Specifically, the reason for the inaccuracy of image retrieval is that the image feature extraction capability is weak, resulting in weak feature representation capability of the extracted features, leading to inaccurate retrieved images. Based on this, the image determination method of some embodiments of this disclosure firstly preprocesses the item image to be retrieved to obtain a preprocessed vector. Here, by preprocessing the item image to be retrieved, not only can the main content information of the item in the image be enhanced, but it is also convenient to input it into the attention mechanism network model. Then, the above preprocessed vector is input into the attention mechanism network model in the image representation vector generation model for extracting multi-level features to output a hierarchical feature vector set, wherein the feature information depths corresponding to the various hierarchical feature vectors in the above hierarchical feature vector set are different. Here, through the attention mechanism network model, multi-level and richer image features can be extracted from the preprocessed vector. Next, the above hierarchical feature vector set is input into the multi-level feature fusion model in the image representation vector generation model to output a multi-level fused feature vector set. Here, a multi-level feature fusion model is used to fuse features from different levels, which can better balance the feature information at different levels, making the feature representation vector corresponding to the image of the item to be retrieved more expressive. Furthermore, based on the multi-level fused feature vector set, the image representation vector generation model can accurately generate the image representation vector corresponding to the image of the item to be retrieved, serving as the target image representation vector. Here, the multi-level fused feature vector set can fully utilize the multifaceted feature information of the image, giving the obtained target image representation vector a strong feature representation capability. Finally, the item images whose similarity to the target image representation vector meets the preset conditions are accurately determined from the item image database and are considered matching item images. In summary, through the attention mechanism network model and the multi-level feature fusion model, not only can multi-level image feature information be extracted, but the feature representation capability of the image representation vector corresponding to the image of the item to be retrieved can also be improved through the fusion of feature information between multi-level features, enabling the accurate identification of item images from the item image database using the target image representation vector.

[0096] Further reference Figure 3 The diagram illustrates a flow 300 of another embodiment of the image determination method according to the present disclosure. This image determination method includes the following steps:

[0097] Step 301: Perform image preprocessing on the image of the item to be retrieved to obtain a preprocessed vector.

[0098] Step 302: Input the preprocessed vector into at least one attention mechanism layer to obtain the attention vector.

[0099] In some embodiments, the executing entity (e.g. Figure 1 The electronic device 101 shown can input the preprocessed vector to the at least one attention mechanism layer to obtain an attention vector. The attention mechanism network model includes: at least one attention mechanism layer, at least one first feature encoding layer, at least one second feature encoding layer, and at least one third feature encoding layer. In practice, the at least one first feature encoding layer can be a series-connected series of at least one first feature encoding layer. The first feature encoding layer can be a Transformer encoding layer. For example, the at least one first feature encoding layer includes four first feature encoding layers. The at least one second feature encoding layer can be a series-connected series of at least one second feature encoding layer. The second feature encoding layer can be a Transformer encoding layer. For example, the at least one second feature encoding layer includes eight second feature encoding layers. The at least one third feature encoding layer can be a series-connected series of at least one third feature encoding layer. The third feature encoding layer can be a Transformer encoding layer. For example, the at least one third feature encoding layer includes four third feature encoding layers. The attention mechanism layer in the at least one attention mechanism layer can be an Outlooker Layer. The Outlooker Layer includes: an Outlook attention layer for spatial information encoding and a multilayer perceptron (MLP) for inter-channel information interaction. At least one first feature coding layer can be a serially connected first feature coding layer. The first feature coding layer can be a Transformer coding layer.

[0100] In some optional implementations of some embodiments, the preprocessed vector is input to at least one serially connected attention mechanism layer to obtain an attention vector, wherein the attention mechanism layer includes: at least two normalization layers, an attention layer, at least two random deactivation layers, and multiple support vector machine (SVM) layers.

[0101] See Figure 4 The network structure of the attention mechanism layer is shown.

[0102] First, the input vector to the attention mechanism layer is fed into the first normalization layer (one of at least two normalization layers) to obtain a first normalized vector. Then, the first normalized vector is fed into the attention layer to obtain the output vector. Next, the output vector is fed into the first random deactivation layer (one of at least two random deactivation layers) to output a first random deactivation vector. Then, the first random deactivation vector and the input vector are concatenated to obtain a first concatenated vector. Next, the first concatenated vector is fed into the second normalization layer (one of at least two normalization layers) to obtain a second normalized vector. Further, the second normalized vector is fed into a multi-layer support vector machine (SVM) layer to obtain a support vector machine (SVM) vector. Then, the SVM vector is fed into the second random deactivation layer (one of at least two random deactivation layers) to output a second random deactivation vector. Finally, the second random deactivation vector and the first concatenated vector are concatenated to obtain a second concatenated vector, which serves as the output vector of the attention mechanism layer.

[0103] Step 303: Input the attention vector into at least one first feature encoding layer to output the first feature encoding vector.

[0104] In some embodiments, the execution entity may input the attention vector into the at least one first feature encoding layer to output a first feature encoding vector.

[0105] As an example, the aforementioned execution entity can input the attention vector into at least one serially connected first feature encoding layer to output a first feature encoding vector.

[0106] Step 304: Generate a low-level feature vector based on the first feature encoding vector, which serves as the hierarchical feature vector.

[0107] In some embodiments, the executing entity may generate a low-level feature vector as a hierarchical feature vector based on the first feature encoding vector. The low-level feature vector contains less rich and expressive image feature information than the mid-level feature vector. The mid-level feature vector contains less rich and expressive image feature information than the high-level feature vector.

[0108] As an example, the aforementioned execution entity can input the first feature encoding vector into the convolutional neural network layer to output a low-level feature vector as a hierarchical feature vector.

[0109] Step 305: Input the first feature encoding vector into at least one second feature encoding layer to output the second feature encoding vector.

[0110] In some embodiments, the execution entity may input the first feature encoding vector into the at least one second feature encoding layer to output the second feature encoding vector.

[0111] As an example, the aforementioned execution entity can input the aforementioned first feature encoding vector into the aforementioned at least one serially connected second feature encoding layer to output the second feature encoding vector.

[0112] Step 306: Generate a mid-level feature vector based on the second feature encoding vector described above, which will serve as the hierarchical feature vector.

[0113] In some embodiments, the execution entity may generate a mid-level feature vector based on the second feature encoding vector, which serves as a hierarchical feature vector.

[0114] As an example, the aforementioned execution entity can input the second feature encoding vector into the convolutional neural network layer to output a mid-level feature vector as a hierarchical feature vector.

[0115] Step 307: Input the second feature encoding vector into at least one third feature encoding layer to output the third feature encoding vector.

[0116] In some embodiments, the execution entity may input the second feature encoding vector into the at least one third feature encoding layer to output the third feature encoding vector.

[0117] As an example, the aforementioned execution entity can input the aforementioned second feature encoding vector into the aforementioned at least one serially connected third feature encoding layer to output the third feature encoding vector.

[0118] Step 308: Generate a high-level feature vector based on the third feature encoding vector, which serves as the hierarchical feature vector.

[0119] In some embodiments, the execution entity may generate a high-level feature vector based on the third feature encoding vector, which serves as a hierarchical feature vector.

[0120] As an example, the aforementioned execution entity can input the third feature encoding vector into the convolutional neural network layer to output a high-level feature vector as a hierarchical feature vector.

[0121] Step 309: Input the hierarchical feature vector set into the multi-level feature fusion model in the image representation vector generation model to output the multi-level fused feature vector set.

[0122] Step 310: Based on the multi-level fusion feature vector set, use the image representation vector generation model to generate the image representation vector corresponding to the image of the item to be retrieved, and use it as the target image representation vector.

[0123] Step 311: Identify from the item image library the item image whose similarity between the corresponding image representation vector and the target image representation vector meets the preset conditions, and use them as matching item images.

[0124] In some embodiments, the specific implementation of steps 301, 309-311 and their resulting technical effects can be found in [reference needed]. Figure 2 Steps 201 and 203-205 in the corresponding embodiments will not be repeated here.

[0125] from Figure 3 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 3 In some corresponding embodiments, the image determination method process 300 uses an attention mechanism network model that includes at least one attention mechanism layer, at least one first feature encoding layer, at least one second feature encoding layer, and at least one third feature encoding layer to accurately generate diverse hierarchical feature vectors. This allows the image representation vector corresponding to the image of the item to be retrieved to have more comprehensive image feature information, thereby making the feature expression capability of the target image representation vector stronger.

[0126] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an image determination apparatus, which are similar to... Figure 2 Corresponding to the method embodiments shown, this image determining device can be specifically applied to various electronic devices.

[0127] like Figure 5 As shown, an image determination device 500 includes: a preprocessing unit 501, a first input unit 502, a second input unit 503, a generation unit 504, and a determination unit 505. The system includes a preprocessing unit 501 configured to preprocess the image of the item to be retrieved to obtain a preprocessed vector; a first input unit 502 configured to input the preprocessed vector into an attention mechanism network model in the image representation vector generation model for extracting multi-level features, to output a hierarchical feature vector set, wherein the feature information depths corresponding to the various hierarchical feature vectors in the hierarchical feature vector set are different; a second input unit 503 configured to input the hierarchical feature vector set into a multi-level feature fusion model in the image representation vector generation model, to output a multi-level fused feature vector set; a generation unit 504 configured to generate an image representation vector corresponding to the image of the item to be retrieved based on the multi-level fused feature vector set and using the image representation vector generation model, as the target image representation vector; and a determination unit 505 configured to determine, from the item image library, an item image whose similarity to the target image representation vector meets a preset condition, as a matching item image.

[0128] In some optional implementations of certain embodiments, the above-mentioned attention mechanism network model includes: at least one attention mechanism layer, at least one first feature encoding layer, at least one second feature encoding layer, and at least one third feature encoding layer; and the first input unit 502 may be further configured to: input the preprocessed vector to the at least one attention mechanism layer to obtain an attention vector; input the attention vector to the at least one first feature encoding layer to output a first feature encoding vector; generate a low-level feature vector based on the first feature encoding vector as a hierarchical feature vector; input the first feature encoding vector to the at least one second feature encoding layer to output a second feature encoding vector; generate a mid-level feature vector based on the second feature encoding vector as a hierarchical feature vector; input the second feature encoding vector to the at least one third feature encoding layer to output a third feature encoding vector; and generate a high-level feature vector based on the third feature encoding vector as a hierarchical feature vector.

[0129] In some optional implementations of some embodiments, the multi-level feature fusion model includes: a multi-level feature fusion layer and multiple fusion feature processing layers; and the second input unit 503 can be further configured to: input the multi-level feature vector set to the multi-level feature fusion layer to output a feature fusion vector set; and input the feature fusion vector set to the multiple fusion feature processing layers to output the multi-level fusion feature vector set.

[0130] In some optional implementations of some embodiments, the fusion feature processing layers in the above-mentioned fusion feature processing layer set include: convolutional layers, pooling layers, random deactivation layers, and fully connected layers; and the second input unit 503 can be further configured to: for each feature fusion vector in the above-mentioned feature fusion vector set, perform a first processing step: input the above-mentioned feature fusion vector to the convolutional layer included in the corresponding fusion feature processing layer to output a convolutional vector; input the above-mentioned convolutional vector to the pooling layer included in the corresponding fusion feature processing layer to output a pooling vector; input the above-mentioned pooling vector to the random deactivation layer included in the corresponding fusion feature processing layer to output a random deactivation vector; input the above-mentioned random deactivation vector to the fully connected layer included in the corresponding fusion feature processing layer to output a fully connected vector; determine the feature level information corresponding to each feature fusion vector in the above-mentioned feature fusion vector set, wherein the feature level information characterizes the feature extraction depth of the features included in the feature fusion vector; generate the above-mentioned multi-level fusion feature vector set based on the feature level information corresponding to each feature fusion vector and the obtained fully connected vector set.

[0131] In some optional implementations of some embodiments, the fusion feature processing layer in the above-mentioned fusion feature processing layer set further includes: a batch normalization layer; and the generation unit 504 can be further configured to: sort the above-mentioned fully connected vector set according to the feature level information corresponding to each feature fusion vector to obtain a fully connected vector sequence; for each fully connected vector in the above-mentioned fully connected vector sequence, perform a second processing step: determine the feature level information corresponding to the above-mentioned fully connected vector as the target feature level information; in response to determining that the above-mentioned fully connected vector is not a fully connected vector at the target position in the above-mentioned fully connected vector sequence... The fully connected vector is identified as a target fully connected vector, provided that there exists a neighboring fully connected vector whose corresponding feature level information is higher than the target feature level information. The fully connected vector corresponding to the target fully connected vector is then concatenated with the target fully connected vector to obtain a concatenated vector. For each concatenated vector in the obtained set, it is input into the batch normalization layer in the corresponding fusion feature processing layer to obtain a batch normalized vector, which serves as the multi-level fusion feature vector.

[0132] In some optional implementations of some embodiments, the first input unit 502 may be further configured to: input the above-mentioned preprocessed vector to at least one serially connected attention mechanism layer to obtain an attention vector, wherein the attention mechanism layer includes: at least two normalization layers, an attention layer, at least two random deactivation layers, and a multi-layer support vector machine.

[0133] In some optional implementations of some embodiments, the preprocessing unit 501 may be further configured to: adjust the image size of the above-mentioned image of the item to be retrieved to obtain an adjusted image; and perform a linear convolution transformation on the adjusted image to output an image of the target dimension as a preprocessing vector.

[0134] In some optional implementations of certain embodiments, at least one loss function corresponding to the above image representation vector generation model includes: a multiple similarity loss function and a second-order relation loss function.

[0135] It is understandable that the units described in the image determining device 500 are related to the reference. Figure 2 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the image determining apparatus 500 and the units contained therein, and will not be repeated here.

[0136] The following is for reference. Figure 6 It illustrates electronic devices suitable for implementing some embodiments of this disclosure (e.g., Figure 1A schematic diagram of the structure of electronic device 101)600 in the middle. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0137] like Figure 6 As shown, the electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory 602 or a program loaded from a storage device 608 into a random access memory 603. The random access memory 603 also stores various programs and data required for the operation of the electronic device 600. The processing unit 601, the read-only memory 602, and the random access memory 603 are interconnected via a bus 604. An input / output interface 605 is also connected to the bus 604.

[0138] Typically, the following devices can be connected to the input / output interface 605: input devices 606 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 608 including, for example, magnetic tape, hard disk, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0139] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a read-only memory 602. When the computer program is executed by the processing device 601, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0140] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0141] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0142] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: perform image preprocessing on the image of the item to be retrieved, obtaining a preprocessed vector; input the preprocessed vector into an attention mechanism network model in an image representation vector generation model for extracting multi-level features, to output a hierarchical feature vector set, wherein the feature information depths corresponding to the various hierarchical feature vectors in the hierarchical feature vector set are different; input the hierarchical feature vector set into a multi-level feature fusion model in the image representation vector generation model, to output a multi-level fused feature vector set; based on the multi-level fused feature vector set, use the image representation vector generation model to generate an image representation vector corresponding to the image of the item to be retrieved, as a target image representation vector; and select from the item image library item images whose corresponding image representation vectors and the target image representation vectors have a similarity that meets a preset condition, as matching item images.

[0143] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0145] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a preprocessing unit, a first input unit, a second input unit, a generation unit, and a determination unit. The names of these units do not necessarily limit the specific unit; for example, a preprocessing unit may also be described as "a unit that performs image preprocessing on an image of an item to be retrieved to obtain a preprocessed vector."

[0146] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0147] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the image determination methods described above.

[0148] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An image determination method, comprising: Image preprocessing is performed on the image of the item to be retrieved to obtain a preprocessed vector; The preprocessed vector is input into the attention mechanism network model in the image representation vector generation model for extracting multi-level features to output a hierarchical feature vector set. The feature information depths corresponding to the various hierarchical feature vectors in the hierarchical feature vector set are different. The hierarchical feature vector set includes: hierarchical feature vectors corresponding to the first level features, hierarchical feature vectors corresponding to the second level features, and hierarchical feature vectors corresponding to the third level features. The process of inputting the hierarchical feature vector set into the multi-level feature fusion model in the image representation vector generation model to output a multi-level fused feature vector set includes: using the multi-level feature fusion model in the image representation vector generation model to classify the hierarchical feature vectors in the hierarchical feature vector set according to the depth of the corresponding feature information, to obtain at least one hierarchical feature vector set; and using the multi-level feature fusion model to concatenate each hierarchical feature vector set in the at least one hierarchical feature vector set to obtain a hierarchical concatenated vector set, which serves as the multi-level fused feature vector set. Using the vector splicing sub-model included in the image representation vector generation model, the various multi-level fusion vectors in the multi-level fusion feature vector set are spliced ​​together to generate the image representation vector corresponding to the image of the item to be retrieved, which serves as the target image representation vector. The image of an item is identified from the image database if the similarity between its corresponding image representation vector and the target image representation vector meets a preset condition. This image is then used as the matching image.

2. The method according to claim 1, wherein, The attention mechanism network model includes: at least one attention mechanism layer, at least one first feature encoding layer, at least one second feature encoding layer, and at least one third feature encoding layer; and The step of inputting the preprocessed vector into the attention mechanism network model in the image representation vector generation model for extracting multi-level features, to output a hierarchical feature vector set, includes: The preprocessed vector is input into the at least one attention mechanism layer to obtain an attention vector; The attention vector is input into the at least one first feature encoding layer to output a first feature encoding vector; Based on the first feature encoding vector, a low-level feature vector is generated as a hierarchical feature vector. The first feature encoding vector is input into the at least one second feature encoding layer to output the second feature encoding vector; Based on the second feature encoding vector, a mid-level feature vector is generated as a hierarchical feature vector; The second feature encoding vector is input into the at least one third feature encoding layer to output the third feature encoding vector; Based on the third feature encoding vector, a high-level feature vector is generated as a hierarchical feature vector.

3. The method according to claim 1, wherein, The multi-level feature fusion model includes: a multi-level feature fusion layer and multiple fusion feature processing layers; and The step of inputting the hierarchical feature vector set into the multi-level feature fusion model in the image representation vector generation model to output a multi-level fused feature vector set includes: The hierarchical feature vector set is input into the multi-level feature fusion layer to output a feature fusion vector set; The feature fusion vector set is input into the multiple fusion feature processing layers to output the multi-level fusion feature vector set.

4. The method according to claim 3, wherein, The fusion feature processing layers in the set include: convolutional layers, pooling layers, random deactivation layers, and fully connected layers; and The step of inputting the feature fusion vector set into the plurality of fusion feature processing layers to output the multi-level fusion feature vector set includes: For each feature fusion vector in the feature fusion vector set, perform the first processing step: The feature fusion vector is input into the convolutional layer included in the corresponding fusion feature processing layer to output a convolutional vector; The convolution vector is input to the pooling layer included in the corresponding fusion feature processing layer to output a pooling vector; The pooling vector is input to the random deactivation layer included in the corresponding fusion feature processing layer to output a random deactivation vector; The random deactivation vector is input to the fully connected layer included in the corresponding fusion feature processing layer to output a fully connected vector; Determine the feature hierarchy information corresponding to each feature fusion vector in the feature fusion vector set, wherein the feature hierarchy information characterizes the feature extraction depth of the features included in the feature fusion vector; The multi-level fusion feature vector set is generated based on the feature hierarchy information corresponding to each feature fusion vector and the obtained fully connected vector set.

5. The method according to claim 4, wherein, The fusion feature processing layer set further includes: a batch normalization layer; and The step of generating the multi-level fused feature vector set based on the feature hierarchy information corresponding to each feature fusion vector and the obtained fully connected vector set includes: Based on the feature hierarchy information corresponding to each feature fusion vector, the fully connected vector set is sorted to obtain a fully connected vector sequence; For each fully connected vector in the fully connected vector sequence, perform the second processing step: Determine the feature hierarchy information corresponding to the fully connected vector, and use it as the target feature hierarchy information; In response to determining that the fully connected vector is not a fully connected vector at the target position in the fully connected vector sequence, and that there exists a fully connected vector that is adjacent to the fully connected vector and whose corresponding feature level information is higher than the target feature level information, the fully connected vector that is adjacent to the fully connected vector and whose corresponding feature level information is higher than the target feature level information is determined as the target fully connected vector; The fully connected vector corresponding to the target fully connected vector is concatenated with the fully connected vector to obtain the concatenated vector. For each spliced ​​vector in the obtained spliced ​​vector set, the spliced ​​vector is input into the batch normalization layer in the corresponding fusion feature processing layer to obtain a batch normalized vector, which serves as a multi-level fusion feature vector.

6. The method according to claim 2, wherein, The step of inputting the preprocessed vector into the at least one attention mechanism layer to obtain an attention vector includes: The preprocessed vector is input to at least one serially connected attention mechanism layer to obtain an attention vector. The attention mechanism layer includes: at least two normalization layers, an attention layer, at least two random deactivation layers, and multiple support vector machine layers.

7. The method according to claim 1, wherein, The image preprocessing of the image of the item to be retrieved, to obtain a preprocessed vector, includes: The image size of the item to be retrieved is adjusted to obtain the adjusted image; The adjusted image is subjected to a linear convolution transformation to output the target dimension image as a preprocessing vector.

8. The method according to claim 1, wherein, At least one loss function corresponding to the image representation vector generation model includes: a multiple similarity loss function and a second-order relation loss function.

9. An image determining device, comprising: The preprocessing unit is configured to perform image preprocessing on the image of the item to be retrieved, and obtain a preprocessed vector. The first input unit is configured to input the preprocessed vector into an attention mechanism network model in the image representation vector generation model for extracting multi-level features, so as to output a hierarchical feature vector set, wherein the feature information depths corresponding to the various hierarchical feature vectors in the hierarchical feature vector set are different, and the hierarchical feature vector set includes: hierarchical feature vectors corresponding to the first level features, hierarchical feature vectors corresponding to the second level features, and hierarchical feature vectors corresponding to the third level features. The second input unit is configured to input the hierarchical feature vector set into the multi-level feature fusion model in the image representation vector generation model to output a multi-level fused feature vector set, including: using the multi-level feature fusion model in the image representation vector generation model to perform vector classification on the hierarchical feature vectors in the hierarchical feature vector set according to the depth of the corresponding feature information, to obtain at least one hierarchical feature vector set; using the multi-level feature fusion model to concatenate each hierarchical feature vector set in the at least one hierarchical feature vector set to obtain a hierarchical concatenated vector set as the multi-level fused feature vector set; The generation unit is configured to use the image representation vector generation model to generate the image representation vector corresponding to the image of the item to be retrieved, and use it as a vector splicing sub-model included in the target image representation vector to splice the various multi-level fusion vectors in the multi-level fusion feature vector set. The determining unit is configured to determine, from the item image library, an item image whose similarity to the target image representation vector satisfies a preset condition, and use it as a matching item image.

10. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images