Image retrieval method, device, electronic device, and computer-readable storage medium

Abstract

The present application provides an image retrieval method, apparatus, electronic device, and computer-readable storage medium. The method comprises: extracting features from an image to be processed to obtain first feature information; performing feature information processing on the first feature information to obtain second feature information corresponding to a target region of the image to be processed; and performing image retrieval processing on the image to be processed based on the second feature information. The present application processes the feature information of the first feature information to obtain second feature information corresponding to the target region of the image to be processed. This allows the feature information of the target region of the image to be processed to be extracted from the first feature information, filtering out cluttered interference regions, and then performing image retrieval based on the features of the target region of the image to be processed, thereby improving the accuracy of the retrieval results.

Description

Technical Field

[0001] The present application relates to the field of image processing technology. Specifically, the present application relates to an image retrieval method, device, electronic device and computer-readable storage medium. Background Art

[0002] Image retrieval is increasingly used in a wide range of applications. However, image databases typically used for retrieval often contain massive amounts of image data, potentially reaching millions or even tens of millions. Accurately and quickly finding images that are identical or similar to the image being retrieved within this vast amount of image data requires extracting discriminative, robust, and accurate feature information from the image. Therefore, feature extraction is a critical step in the image retrieval process.

[0003] Figure 1 It is the feature extraction process in the prior art, such as Figure 1 As shown in the figure, the results of the feature extraction network can be divided into two parts: the backbone network and the feature representation network. The backbone network extracts feature information from the image to be processed and outputs a feature map. The feature representation network can further include a two-layer structure. The first layer averages the feature matrix of each channel in the feature map output by the backbone network, thereby obtaining an eigenvalue for each channel. Multiple channels form a feature vector, which is then fully connected through the second layer's fully connected (FC) layer. The database is retrieved based on the final feature information obtained to obtain the image retrieval result.

[0004] The existing feature representation network uses the method of averaging the feature matrix, which does not distinguish between the background and the object in the image. As a result, the background information will be included in the final feature vector, making the extracted features unable to accurately describe the target features, greatly interfering with accurate retrieval, and thus leading to inaccurate retrieval results. Summary of the Invention

[0005] In order to overcome the above technical problems or at least partially solve the above technical problems, the following technical solutions are proposed:

[0006] In a first aspect, the present application provides an image retrieval method, the method comprising:

[0007] Perform feature extraction on the image to be processed to obtain first feature information;

[0008] Performing characteristic information processing on the first feature information to obtain second feature information corresponding to the target area of the image to be processed;

[0009] According to the second feature information, image retrieval processing is performed on the image to be processed

[0010] In a second aspect, the present application provides an image retrieval device, the device comprising:

[0011] A feature extraction module, configured to extract features from the image to be processed to obtain first feature information;

[0012] a feature information processing module, configured to perform feature information processing on the first feature information to obtain second feature information corresponding to a target area of the image to be processed;

[0013] The image retrieval module is used to perform image retrieval processing on the image to be processed based on the second feature information.

[0014] In a third aspect, the present application provides an electronic device, comprising:

[0015] A processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set being loaded and executed by the processor to implement the method as shown in the first aspect of the present application.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, which is used to store computer instructions, programs, code sets or instruction sets, which, when run on a computer, enable the computer to execute the method shown in the first aspect of the present application.

[0017] The image retrieval method, device, electronic device and computer-readable storage medium provided in the present application obtain second feature information corresponding to the target area of the image to be processed by processing the characteristic information of the first feature information. That is, the feature information of the target area of the image to be processed can be extracted from the first feature information, and the messy interference area can be filtered out. Then, image retrieval can be performed based on the features of the target area of the image to be processed, thereby improving the accuracy of the retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for describing the embodiments of the present application.

[0019] Figure 1 Schematic diagram of the feature extraction process in the prior art;

[0020] Figure 2 A schematic diagram of the flow of the image retrieval method provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of an ASAU structure provided in an embodiment of the present application;

[0022] Figure 4a Schematic diagram of another ASAU structure provided in the embodiment of the present application Figure 1 ;

[0023] Figure 4b Schematic diagram of another ASAU structure provided in the embodiment of the present application Figure 2 ;

[0024] Figure 5 A schematic diagram of convolution processing parameters of different scales provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of an ASAU processing process provided in an embodiment of the present application;

[0026] Figure 7 Schematic diagram of at least two ASAU processing processes provided in the embodiment of the present application Figure 1 ;

[0027] Figure 8 Schematic diagram of at least two ASAU processing processes provided in the embodiment of the present application Figure 2 ;

[0028] Figure 9 Schematic diagram of at least two ASAU processing processes provided in an embodiment of the present application;

[0029] Figure 10 A schematic diagram of at least two ASAUs processing clothing images provided in an embodiment of the present application;

[0030] Figure 11 A schematic diagram of the MGMN processing process provided in an embodiment of the present application;

[0031] Figure 12 A schematic diagram of the process of adding spatial constraints provided in an embodiment of the present application;

[0032] Figure 13 A schematic diagram of the SCAN processing process provided in an embodiment of the present application;

[0033] Figure 14a A schematic diagram of the MGMN plus SCAN processing process provided in an embodiment of the present application;

[0034] Figure 14b A schematic diagram of offline training of a feature representation network provided in an embodiment of the present application;

[0035] Figure 15 A schematic diagram of multiplexing ASAUs during MGMN plus SCAN processing provided in an embodiment of the present application;

[0036] Figure 16a A schematic diagram of a clothing search scene search function provided in an embodiment of the present application;

[0037] Figure 16b A schematic diagram of a clothing retrieval process provided in an embodiment of the present application;

[0038] Figure 16c A schematic diagram of clothing retrieval results without considering spatial constraints provided in an embodiment of the present application;

[0039] Figure 16d A schematic diagram of the clothing retrieval and evaluation effect provided in an embodiment of the present application;

[0040] Figure 17 A schematic diagram of the structure of the image retrieval device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0042] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0043] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0044] To address the problems of the existing technology, this application proposes a multi-granularity mining network based on the attention mechanism and its application in image retrieval. This application solution can be applied to the field of clothing retrieval. This application solution can be implemented through a deep learning network. The network can highlight the area of interest in the input image, extract the features of the target area, ignore the background area, and mine useful information from different granularities. Finally, this information is synthesized to achieve the purpose of more accurate recognition.

[0045] The present application embodiment provides an image retrieval method, such as Figure 2 As shown, the method includes:

[0046] Step S101: extracting features from the image to be processed to obtain first feature information;

[0047] Step S102: performing characteristic information processing on the first feature information to obtain second feature information corresponding to the target area of the image to be processed;

[0048] Step S103: performing image retrieval processing on the image to be processed according to the second feature information.

[0049] In the embodiment of the present application, feature information processing can also be called feature representation processing. Similarly, the feature representation network can also be called a feature processing network.

[0050] Specifically, in step S101, a backbone network may be used to extract features from the image to be processed. In the embodiment of the present application, the backbone network comprises multiple layers of modules. The bottom layer modules are used to extract basic structural information such as boundaries and intersections, the middle layer modules are used to extract higher-level shape information, and the top layer modules are used to extract semantic information (semantic information contains more detailed information, such as position feature information).

[0051] The backbone network outputs the first feature information, which can be a multi-channel feature map, such as a 512-channel feature map, where each channel corresponds to a feature matrix, such as a 7*7 feature matrix. That is, the backbone network can output a 512*7*7 feature map. The first feature information can also be other feature information.

[0052] In the embodiment of the present application, the feature map, feature matrix, and feature vector are all feature information.

[0053] In step S102, the first feature information obtained in step S101 is processed to obtain second feature information corresponding to the target area of the image to be processed. The second feature information may be a one-dimensional feature vector. Figure 1 The (0.4, 1.2, ..., 3.2, 0.1) in the image may also be a multi-dimensional feature vector, such as a 512-dimensional or 1024-dimensional feature vector obtained by converting a 512*7*7 feature map, or other feature information.

[0054] In step S103, image retrieval processing is performed on the image to be processed based on the second feature information obtained in step S102. That is, the feature information corresponding to the target area of the image to be processed is matched with the feature information of each image in the database to determine an image that matches the image to be processed.

[0055] Among them, in the process of matching the feature information of each image in the database, the cosine distance or Euclidean distance between the feature information of each image in the database and the feature information of the image to be processed can be calculated first, and then sorted according to the calculated distances, and the image in the database with the smallest distance is obtained, which is the image that matches the image to be processed.

[0056] The image retrieval method provided in the embodiment of the present application obtains second feature information corresponding to the target area of the image to be processed by processing the characteristic information of the first feature information. That is, it can extract the feature information of the target area of the image to be processed from the first feature information, filter out the messy interference area, and then perform image retrieval based on the features of the target area of the image to be processed, thereby improving the accuracy of the retrieval results.

[0057] In one possible implementation, step S102 of the embodiment of the present application specifically includes the following steps:

[0058] Step S1021: for at least one target area, performing corresponding characteristic information processing on the first feature information to obtain third feature information corresponding to each target area;

[0059] Step S1022: fusing the respective third feature information to obtain second feature information corresponding to the target area of the image to be processed.

[0060] Specifically, in an embodiment of the present application, for each target region, the first feature information can be processed by the region processing unit corresponding to the target region to obtain third feature information corresponding to the target region. The region processing unit can also be called an adaptive spatial attention unit (ASAU). The ASAU can extract multi-scale features. In addition, the extracted features have spatial and channel correlation.

[0061] When the ASAU corresponding to any target area performs further characteristic information processing on the first feature information, it performs a convolution operation based on the first convolution processing parameter corresponding to the target area, that is, convolution processing is performed on the first feature information to obtain the third feature information corresponding to the target area, which is also the third feature information corresponding to the ASAU. The third feature information mainly includes the feature information corresponding to the target area of the image to be processed extracted from the first feature information, and filters out messy interference areas, such as background areas, etc., so that image retrieval can be performed based on the feature information of the target area of the image to be processed, thereby improving the accuracy of the retrieval results.

[0062] like Figure 3 As shown, the embodiment of the present application provides a feasible ASAU structure:

[0063] This ASAU structure incorporates a spatial scale processing method, namely a convolution operation. The first feature information is convolved with the learned first convolution processing parameters to obtain feature information that highlights the primary target area. The convolution processing parameters include, but are not limited to, at least one of the convolution kernel, kernel size, expansion factor, and convolution step size. In practical applications, the feature information after convolution processing can be a feature map.

[0064] Furthermore, based on the first convolution processing parameter corresponding to the target area, the first feature information is convolved to obtain third feature information corresponding to the target area, specifically including:

[0065] Performing convolution processing on the first feature information based on the first convolution processing parameter corresponding to the target area, wherein the first feature information may be a feature map of size H*W*C, and feature information obtained after the convolution processing on the first feature information may be a feature map of size H*W*1;

[0066] Normalize the feature information obtained after convolution processing. For example, the sigmoid operation is used to normalize the feature map after convolution processing, that is, the value of the feature map is normalized to the interval [0, 1], and a feature map of size H*W*1 can be obtained.

[0067] Perform a stacking operation on the normalized feature information. Through the stacking operation, the same number of feature maps as the number of channels of the first feature information are copied, that is, a feature map of size H*W*C can be obtained. Continuing with the previous example, if 512 feature maps of the same size are copied, the obtained feature maps form a feature map of the same size as the first feature information, such as a 512*7*7 feature map;

[0068] The feature information after the stacking operation is fused with the first feature information, such as a dot product operation, to obtain the third feature information corresponding to the target area, which is also the third feature information corresponding to the ASAU. The third feature information can be a feature map with a size of H*W*C.

[0069] like Figure 4a As shown, the embodiment of the present application also provides another feasible ASAU structure:

[0070] In this ASAU structure, an ASAU can include feature extraction processes for at least two scales. Since the image to be retrieved may contain detailed information at various scales, taking clothing retrieval as an example, a typical clothing image also includes details such as pockets, collars, and buttons. The scale of pockets is generally larger than that of collars, which in turn is larger than that of buttons. Therefore, the ASAU can further process the different scales separately to obtain feature information for multiple scales. This multi-scale feature information is then weighted and fused, so that the final feature map can highlight objects of different scales in the image to be processed that belong to the target area (such as pockets, collars, and buttons in clothing images). This allows the target area to have feature information that reflects various details, further improving the accuracy of the retrieval results.

[0071] Specifically, different scales are achieved through different forms of convolution processing parameters, that is, step SA is used to implement ASAU to process the first feature information based on different scales.

[0072] Step SA: Based on the first convolution processing parameters corresponding to the target area, perform convolution processing on the first feature information at least twice to obtain each fourth feature information corresponding to the target area, wherein the first convolution processing parameters used in each convolution processing are different.

[0073] That is to say, the convolution processing parameters used in each convolution process are different, specifically, the first convolution processing parameters corresponding to the target area used in each convolution process are different, then the scale of processing the first feature information will be different. In an embodiment of the present application, the convolution processing parameters that characterize different scales may include but are not limited to dilation factors (DF, Dilation Factor), such as different dilation rates, corresponding to different scales, that is, when the first feature information is convolved at least twice in ASAU, the dilation rates used are different. In this case, other convolution processing parameters except the dilation rate may be the same or different, for example, the convolution kernel, the convolution step size, etc. may be the same or different. That is, in an embodiment of the present application, the convolution processing parameters used in each convolution process include at least one of the following: convolution kernel, convolution kernel size, dilation factor, and convolution step size.

[0074] Figure 4a Type 1, Type 2, and Type 3 represent three types of convolution processing parameters, representing three different scales and three different sets of convolution processing parameters. This means that three convolution operations are performed on the first feature information based on the three different sets of first convolution processing parameters corresponding to the target area. If the first feature information is a feature map of size H*W*C, then three convolution operations on the first feature information yield three feature maps of size H*W*1.

[0075] As an example, Figure 5 Three types of convolution processing parameters are shown, including kernel size ( Figure 5 K in), expansion factor ( Figure 5 The expansion factor (DF) and other parameters in the image are included. The size of the expansion factor reflects the change in scale. For example, when DF = 1, it is a smaller scale, which can capture richer details in the image. When DF = 2, it is a medium scale, which may weaken the extraction of details but can extract the main outline information. The effect is similar when DF = 4. In general, the larger the DF value, the more it tends to extract the main outline of the object in the image, while gradually weakening the expression of detailed information. Therefore, in order to obtain more complete and rich feature information, it is necessary to express information at different scales together through step SB, thereby expressing more accurate feature information.

[0076] Step SB: Perform fusion processing based on each fourth feature information to obtain the third feature information corresponding to the target area.

[0077] Specifically, after obtaining the fourth feature information corresponding to each scale in step SA, if the contribution of the feature information of each scale to the final processing result is different, a weight calculation method can be introduced in step SB, that is, based on the weight corresponding to each fourth feature information, each fourth feature information is fused.

[0078] Before this, the weight calculation method needs to use the second convolution processing parameters corresponding to the target area, that is, a pre-designed convolution kernel, based on the second convolution processing parameters corresponding to the target area, perform convolution processing on the first feature information; according to the result of the convolution processing, the weights corresponding to each fourth feature information are obtained, that is, the output is a weight consistent with the number of scales (for example, if ASAU contains 3 scales, 3 weights will be output). Among them, the second convolution processing parameters include at least one of the following: convolution kernel, convolution kernel size, expansion factor, and convolution step size. In practical applications, the weights can be calculated by minimizing the loss function using the backpropagation algorithm widely used in neural network training.

[0079] In practical applications, based on the result of the convolution process, the weights corresponding to the respective fourth feature information are obtained, specifically including: normalizing the result of the convolution process, for example, using the sigmoid operation to normalize the weights corresponding to the respective fourth feature information to the interval [0, 1], and obtaining the weights corresponding to the respective fourth feature information. If the first feature information is a feature map of size H*W*C, such as the number of channels C=3, then the result of the convolution process can be a feature map of size 1*1*3, and the size of the normalized feature map can be 1*1*3. In this feature map, each eigenvalue represents the weight of the corresponding scale, such as Figure 4a shown.

[0080] Furthermore, after obtaining the fourth feature information of different scales and the corresponding weights obtained in step SA, a fusion process is performed in step SB.

[0081] One feasible approach is to multiply the weight by the fourth feature information of the corresponding scale, and then add the products to obtain the result of deep fusion, which is the third feature information with the same size and quantity as the first feature information, such as a feature map of size H*W*C, which is the processing result of this ASAU structure.

[0082] like Figure 4a The example of ASAU with three scales should not be construed as limiting the number of scales. The operations depicted by the dashed lines are intended to calculate weights, reflecting the contribution of feature information at different scales to the final processing result. The processes depicted by the solid lines are intended to perform convolution processing on the first feature information at different scales, thereby obtaining the primary target region in the processed image and ignoring irrelevant regions.

[0083] In the embodiment of this application, Figure 4a As can be seen, step SA specifically includes:

[0084] Based on the first convolution processing parameters corresponding to the target area, convolution processing is performed on the first feature information at least twice, wherein the first convolution processing parameters used in each convolution processing are different. If the first feature information is a feature map of size H*W*C, then the feature information obtained after the convolution processing may be a feature map of size H*W*1;

[0085] For the feature information obtained after each convolution process, the following processing is performed respectively:

[0086] Normalize the feature information obtained after convolution processing. For example, the sigmoid operation is used to normalize the feature map after convolution processing, that is, the value of the feature map is normalized to the interval [0, 1]. The normalized feature information can be a feature map of H*W*1;

[0087] Perform a stacking operation on the normalized feature information. Through the stacking operation, the same number of feature maps as the number of channels of the first feature information are copied. Continuing with the previous example, if 512 feature maps of the same size are copied, the obtained feature maps form a feature map of the same size as the first feature information, such as a feature map of size H*W*C, for example, a feature map of 512*7*7;

[0088] The feature information after the stacking operation is fused with the first feature information, such as a dot product operation, to obtain the fourth feature information corresponding to each convolution operation. The size of the fourth feature information is consistent with the size of the first feature information, such as a feature map of size H*W*C.

[0089] That is to say, each scale in ASAU is processed in the same way, that is, the first feature information is subjected to convolution, sigmoid, stacking and fusion to obtain the corresponding fourth feature information, which is then fused with the corresponding weight to obtain the third feature information corresponding to the ASAU.

[0090] Another feasible way is to multiply the weight by the fourth feature information of the corresponding scale, and then add each product to obtain the result of deep fusion. After stacking and fusion, the third feature information with the same size and quantity as the first feature information is obtained, that is, the processing result of this ASAU structure.

[0091] like Figure 4b The example of ASAU with three scales should not be construed as limiting the number of scales. The operations depicted by the dashed lines are intended to calculate weights, reflecting the contribution of feature information at different scales to the final processing result. The processes depicted by the solid lines are intended to perform convolution processing on the first feature information at different scales, thereby obtaining the primary target region in the processed image and ignoring irrelevant regions.

[0092] In the embodiment of this application, Figure 4b As can be seen, step SA specifically includes:

[0093] Based on the first convolution processing parameters corresponding to the target area, the first feature information (such as the feature map of size H*W*C) is convolved at least twice, wherein the first convolution processing parameters used in each convolution processing are different. Figure 4b In the figure, type 1, type 2, and type 3 represent three types of convolution processing parameters, which are three different scales and three different sets of convolution processing parameters. That is, based on the three different sets of first convolution processing parameters corresponding to the target area, the first feature information is convolved three times, and three feature maps of size H*W*1 can be obtained.

[0094] For each convolution operation, the feature information obtained after the convolution operation is normalized. For example, the feature map after the convolution operation is normalized by a sigmoid operation, that is, the value of the feature map is normalized to the interval [0, 1] to obtain the corresponding fourth feature information. The fourth feature information can be a feature map with a size of H*W*1.

[0095] Furthermore, step SB specifically includes:

[0096] A fusion process is performed on each fourth feature information. Specifically, a fusion process is performed on each fourth feature information based on the weight corresponding to each fourth feature information. The fused feature information can be a feature map with a size of H*W*1, which can also be called an attention map.

[0097] Perform a stacking operation on the feature information obtained after fusion. Through the stacking operation, the feature maps with the same number of channels as the first feature information are copied. Continuing with the previous example, if 512 feature maps of the same size are copied, the obtained feature maps form a feature map with the same size as the first feature information, such as a feature map of size H*W*C, for example, a feature map of 512*7*7;

[0098] The feature information after the stacking operation is fused with the first feature information, such as a dot product operation, to obtain the third feature information corresponding to the target area. The third feature information and the first feature information have the same size, for example, both are H*W*C feature maps.

[0099] That is to say, at each scale processed in ASAU, the first feature information will be processed in the same way through convolution and sigmoid to obtain the corresponding fourth feature information, which will be fused with the corresponding weights. After stacking and fusion, the third feature information corresponding to the ASAU will be obtained.

[0100] In practical applications, those skilled in the art may select any of the above-mentioned ASAU structures for application based on practical circumstances, and this embodiment of the present application is not limiting. Furthermore, step S1022 may specifically include performing a full connection operation based on each piece of third feature information to obtain second feature information corresponding to the target region of the image to be processed. The implementation of ASAU in steps S1021 and S1022 will be described in detail below.

[0101] In a feasible implementation, Figure 6As shown, in step S1021, for the target area of the image to be processed, the first feature information is processed by an ASAU to obtain third feature information corresponding to the ASAU, that is, the third feature information corresponding to the target area of the image to be processed. Then, in step S1022, a full connection operation is performed on the third feature information to obtain second feature information corresponding to the target area of the image to be processed. The second feature information is used to perform image retrieval in the database to obtain the retrieved image. Figure 6 The feature representation network in can also be called a feature processing network.

[0102] Specifically, during the feature extraction process for image retrieval, an ASAU can be used to process the first feature information output by the backbone network. The specific processing process is described above for the ASAU and will not be repeated here. The ASAU results are then connected through a fully connected layer (FC) to obtain second feature information corresponding to the target region of the image to be processed, for example, a 512-dimensional feature vector. An ASAU and a fully connected layer (FC) can be included in a feature representation network, meaning that the output of the feature representation network is the second feature information corresponding to the target region of the image to be processed.

[0103] Another possible implementation is as follows: Figure 7 As shown, in step S1021, for at least two target regions, the first feature information is processed by at least two ASAUs to obtain third feature information corresponding to each ASAU. Different third feature information can represent different regions of the image to be processed. Then, in step S1022, a full connection operation is performed on each third feature information to obtain second feature information corresponding to the target region of the image to be processed.

[0104] Specifically, during the feature extraction process for image retrieval, at least two ASAUs can first process the first feature information output by the backbone network. The specific processing steps for each ASAU can be found in the above description of ASAUs and will not be repeated here. The results of these multiple ASAUs are then connected via a fully connected layer (FC) to obtain second feature information corresponding to the target region of the image being processed. Each ASAU focuses on a different region of the image being processed. Continuing with the clothing retrieval scenario above, for example, ASAU 1 might focus primarily on the collar region of the clothing image, while ASAU 2 might focus primarily on the pocket region, and so on.

[0105] In the embodiment of this application, Figure 8 As shown in , at least two ASAUs and one fully connected layer (FC) can be included in a feature representation network, i.e. Figure 7It represents the processing process of a feature representation network, wherein the output of the feature representation network is the second feature information corresponding to the target area of the image to be processed.

[0106] The feature representation network can be used to further extract the feature information of the target area from the first feature information output by the backbone network. Compared with the structure of the feature representation network in the prior art that first performs averaging processing and then performs full connection processing, the technical solution provided by the embodiment of the present application can obtain the feature information of each local area in the target area of the image to be processed. Since the local features are relatively stable and are not greatly affected by changes in the shape or angle of the object, more robust features can be obtained, which greatly improves the accuracy of the image retrieval results.

[0107] In another feasible implementation, the image to be processed corresponds to at least two target area groups, each target area group includes at least one target area, wherein each target area group can represent the target area of the image to be processed. Figure 9 As shown, in step S1021, for each target area, the first feature information is processed by the corresponding ASAU to obtain the third feature information corresponding to the ASAU (i.e., the third feature information corresponding to the target area), and then in step S1022, a full connection operation is performed on each third feature information to obtain the second feature information corresponding to the target area of the image to be processed. In step S1022, the third feature information corresponding to each target area in each target area group is specifically fused to obtain the fifth feature information corresponding to each target area group. As described above, each fifth feature information can represent the target area of the image to be processed; each fifth feature information is then fused to obtain the second feature information corresponding to the target area of the image to be processed. Specifically, a full connection operation can be performed based on each fifth feature information to obtain the second feature information corresponding to the target area of the image to be processed. The embodiment of the present application proposes that before the fifth feature information is fused, a global average pooling (GAP) operation can be performed on each fifth feature information, and the feature information obtained after pooling can be further subjected to a full connection operation.

[0108] That is, in the feature extraction process of image retrieval, the first feature information output by the backbone network can be processed by at least two groups of ASAUs, each group including at least one ASAU. Optionally, each group can include a different number of ASAUs. The specific processing process of each ASAU can be found in the above introduction to ASAU and will not be repeated here. Figure 9As shown, it can also be understood that the first feature information output by the backbone network is processed through a multi-branch structure, each branch is the above-mentioned target area group, each branch contains at least one ASAU, and each branch represents a local area of different scales, which can also be considered as a local area of different granularity.

[0109] For example, a branch structure is set up with one ASAU, which can be considered to focus on global information. Two other branch structures are set up, containing two and three ASAUs respectively. The more ASAUs included, the finer the processing granularity of the branch. The results of each ASAU in each branch are then fused. That is, the information output by each ASAU is combined to obtain higher-level semantic information, resulting in a more concise expression for that branch. The fused results of each branch are then fused together, and the total fused results of multiple branches are subsequently connected through a fully connected layer (FC) to obtain second feature information corresponding to the target area of the image to be processed.

[0110] Since the local areas contained in the target area of each image to be processed can also be considered to be multi-scale, through this multi-branch processing result, we can focus on local areas of different scales, thereby obtaining more comprehensive feature information and making the retrieval results more accurate.

[0111] For example Figure 9 In this example, three branches are used, each containing a different number of ASAU units. The branch containing one ASAU is used to extract the target region of the image being processed, focusing on describing more holistic feature information such as the target region's outline. In addition to overall feature information, the target region also contains richer detailed information. Therefore, in addition to the branch containing one ASAU, two branches containing two and three ASAUs are added, respectively. These branches can be used to extract detailed information of the target region. For example, continuing with the clothing retrieval scenario in the above example, this includes small areas such as collars and pocket buttons in clothing images.

[0112] As an example, Figure 10 The figure shows the process of feature extraction in a clothing image. First, the feature information of the entire target area is extracted through the dotted line portion to obtain the contour information. That is, the dotted line portion represents a branch containing an ASAU. As mentioned above, an ASAU can be a multi-scale processing process. In the embodiment of the present application, a feature map obtained after convolution processing based on one scale is extracted. Based on a determined threshold, the values in the feature map greater than the threshold are changed to 1, otherwise to 0. The modified feature map is then proportionally mapped to the image to be processed. After mapping, it can be seen that the target area is enhanced, while the background area is basically eliminated.

[0113] Furthermore, the feature information of the local area in the target area is extracted through the solid line part. The solid line part contains three ASAU branches as an example. The mapping method of the local area can be seen in the description of the dotted line part, which will not be repeated here. After the processing of this branch, three parts of the results can be obtained. The top part extracts the collar part. Figure 10 In the figure, the bottom portion extracts the characteristic information of the black-and-white spacing between the sleeves and the bottom edge of the garment. The middle portion, in particular, not only extracts the buttons, a local object in the garment (an object can also be called an object, a component, etc.), but also extracts the trademark, a small and easily overlooked detail, demonstrating the advantages of the multi-scale ASAU structure. The solid line portion, containing two ASAU branches, is used as an example for illustration. The mapping method for the local area is described in the dashed line portion and will not be repeated here. The processing of this branch yields two results: the top portion extracts detailed information such as the buttons, while the bottom portion extracts the characteristic information of the black-and-white spacing between the sleeves and the bottom edge of the garment.

[0114] exist Figure 10 In the figure, the dotted part can be considered to extract global information, and the solid part can be considered to extract small and easily ignored local information.

[0115] In the embodiment of this application, Figure 11 As shown, at least two ASAUs from multiple branches can be included in a multi-granularity feature mining network (MGMN). Furthermore, an MGMN and a fully connected layer (FC) can be included in a feature representation network. The MGMN's input is the first feature information output by the backbone network. The MGMN's output (i.e., the fusion output of each branch) is connected through the fully connected layer to obtain second feature information corresponding to the target region of the image to be processed, i.e., the feature information output by the feature representation network.

[0116] In combination with the above, the MGMN in the embodiments of this application has two major advantages:

[0117] 1) It can fuse information of different granularities. For example, for a clothing image, different levels of information can be obtained by viewing different sized regions. For example, focusing on the entire image will only reveal the overall outline of the garment, while ignoring more detailed information. However, focusing on a specific area, such as the upper part of the garment, can extract the collar style and even smaller details such as texture. If the area containing the trademark is further focused, the trademark itself will also be extracted, allowing for more accurate description and expression.

[0118] 2) It can automatically calculate image regions of different scales. For example, for a clothing image, not all areas in the image contribute equally to the final processing result. In fact, some areas are large and some are small, and they are not fixed. For example, if two pieces of clothing have buttons and one does not, but are very similar in other places, then it is obvious that the button feature information will play a greater role in image retrieval and should therefore be given a higher weight. The ASAU in the MGMN of the embodiment of the present application includes a learning process for weights of different scales, which can automatically learn which area has a greater role and assign a larger weight, making the final feature representation more robust and accurate.

[0119] As can be seen from the above, when the first feature information output by the backbone network is processed by at least two ASAUs, each ASAU focuses on a different area of the image to be processed. Considering that the different areas focused on by each ASAU may be related to each other, the embodiment of the present application provides a feasible implementation method, such as Figure 12 As shown in the figure, the LSTM mechanism is used to simulate the relative position relationship between each region, and the feature information output by each ASAU is jointly enhanced to strengthen the spatial constraints between them, so that the processed feature information can reflect the spatial relationship between each region. Subsequently, the enhanced results are connected through the fully connected layer (FC) to obtain the second feature information corresponding to the target region of the image to be processed.

[0120] Specifically, after step S1021, that is, after the first feature information is processed by at least two ASAUs to obtain the third feature information corresponding to each ASAU, the third feature information corresponding to each target area is input into the preset long short-term memory (LSTM) network in sequence according to the positional relationship of each target area, and the third feature information after relationship enhancement is output.

[0121] Specifically, the input order of each third feature information is determined based on the positional relationship of each target area and the centroid information of the third feature information corresponding to each target area. For example, the positional relationship of each target area can be from top to bottom, from left to right, from inside to outside, etc. This embodiment of the application is not limited here. In actual application, it can be Figure 12 The adjustment module in determines the input order of each third feature information.

[0122] Furthermore, according to the determined input order, each third feature information is sequentially input into a preset LSTM, and feature information with enhanced relationships is output. LSTM has been widely used in natural language processing (NLP) because language is sequential and there are natural relationships between words, and these relationships can be organized into a meaningful sentence. In an embodiment of the present application, when LSTM is applied to image processing, the image can be simply segmented into different parts in spatial order, and then the feature information of multiple parts can be input into the LSTM.

[0123] Following the determined input order, each third feature is sequentially input into the LSTM. In the feature information output by the LSTM, subsequent features are dependent on previous ones, thus modeling the relationships between the features. In other words, the LSTM jointly enhances the feature information output by each ASAU, strengthening spatial constraints so that the processed features reflect the spatial relationships between regions. Ultimately, the LSTM outputs a corresponding number of features, which are enhanced by considering their interrelationships, making the features more distinctive.

[0124] In the embodiment of this application, Figure 13 As shown, at least two ASAUs, an adjustment module, and an LSTM network can be included in a spatially constrained attention network (SCAN). A SCAN and a fully connected layer (FC) can be included in a feature representation network. The SCAN input is the first feature information output by the backbone network. The SCAN output is the relationship-enhanced feature information, which is input into the fully connected layer. The fully connected layer performs a fully connected operation on the relationship-enhanced feature information to obtain second feature information corresponding to the target area of the image to be processed.

[0125] For example, consider processing through three ASAUs. For example, if the input image to be processed is a garment, the backbone network generates a feature map. Then, the three ASAUs extract feature information for different parts of the garment. Specifically, one extracts features for the collar, another for the buttons, and yet another for the sleeves. However, the relationships between these parts are uncertain. The Adjustment module then determines the order in which the features are input, resulting in a top-to-bottom dependency order for the target regions. Typically, the collar is above the buttons, and the buttons are above the bottom edge of the garment. These features are then sequentially fed into the LSTM. After processing, the LSTM outputs enhanced feature information reflecting the relationships between the various objects. This unique feature information is more discriminative, resulting in more accurate retrieval results.

[0126] In combination with the above content, the use of SCAN in the embodiments of this application has the following advantages:

[0127] 1) It is possible to extract complete object information and simulate the relationship between them. For example, in order to simulate the relationship between objects in the image from top to bottom structure, there is also a division method that is to divide the image to be processed evenly, and then obtain feature information for each divided area, and then input it into LSTM for processing. However, this rigid division method does not take into account the area size of each object in the image, and may cut a complete object apart, resulting in the final feature information not being able to correctly correspond to the features of the image, and the enhanced feature information is also inaccurate. The SCAN network provided in the embodiment of the present application focuses on the regional position of the object and extracts the complete area of the object. The feature information obtained is more targeted, and the feature information of the enhanced relationship is also more accurate.

[0128] 2) An adjustment module has been introduced to automatically align the relative positions of objects in an image. Each ASAU extracts feature information for different objects. The adjustment module then arranges these objects according to their original positions, ensuring that the relationship-enhanced feature information is calculated based on the correct positional relationships. The result is feature information that reflects the relative positions of each object and enhances the relationship, significantly improving search performance.

[0129] The embodiment of the present application provides a feasible implementation method, such as Figure 14a As shown, the MGMN and SCAN may also be included at the same time. MGMN and SCAN may respectively deploy independent ASAUs or share ASAUs.

[0130] Specifically, when independent ASAUs are deployed in the MGMN and SCAN, that is, in addition to the MGMN, the first feature information must be reprocessed using the SCAN. As mentioned above, in the MGMN, the processed image corresponds to at least two target region groups, each containing at least one target region. In this case, at least one target region group in the MGMN, i.e., at least one of the multiple branches, can be processed using the corresponding SCAN (one target region group requires one SCAN for joint enhancement, i.e., the number of SCANs corresponds to the number of target region groups requiring enhancement). This can also achieve a similar effect of enhancing the relationships of all branches in the MGMN, while saving computing resources to a certain extent.

[0131] Therefore, in the embodiment of the present application, at least one target area group is reacquired using a corresponding number of SCANs. The specific acquisition method can be found in the above description of SCANs and will not be repeated here. Then, for each of the at least one target area group reacquired using a corresponding number of SCANs, the following steps are performed for each target area group: based on the positional relationship between the target areas, the third feature information corresponding to each target area is sequentially input into a preset long short-term memory network, and the third feature information after the relationship enhancement is output.

[0132] Specifically, the input order of each third feature information is determined based on the positional relationship of each target area and the centroid information of the third feature information corresponding to each target area. For example, the positional relationship of the target areas can be from top to bottom, from left to right, from inside to outside, etc., which is not limited in this embodiment of the present application. In actual applications, the input order of each third feature information can be determined by the Adjustment module.

[0133] Furthermore, the feature information of each set of relationship enhancement is fused, specifically, a full connection operation is performed on the feature information of each set of relationship enhancement, that is, Figure 14a As shown, at least one SCAN ( Figure 14a In the example above, only one target region group is subjected to joint enhancement processing. In this case, only one SCAN output result is required to perform a full connection operation to obtain a first intermediate full connection result.

[0134] In addition, in MGMN, the feature information of each branch needs to be fused, and the fusion processing result needs to be fully connected through a fully connected layer (FC_MGMN) to obtain the second intermediate fully connected result.

[0135] Subsequently, the first intermediate fully connected result and the second fully connected result are input into a fully connected layer (FC), and a fully connected operation is performed on the first intermediate fully connected result and the second fully connected result through the fully connected layer to obtain second feature information corresponding to the target area of the image to be processed.

[0136] In short, the backbone network extracts features from the processed image, and the first feature information output is processed by MGMN and SCAN respectively. The results of each branch output by MGMN can be passed through a fully connected layer ( Figure 14a The feature information output by SCAN can also be connected through a fully connected layer ( Figure 14a The results of FC-MGMN and FC-SCAN are then connected through the fully connected layer ( Figure 14a FC in the image) is connected to obtain the second feature information corresponding to the target area of the image to be processed.

[0137] Specifically, when ASAUs are shared between the MGMN and SCAN, the SCAN can directly determine the input order of the feature information output by each ASAU in the MGMN branch, which serves as the input to the LSTM in the SCAN, improving processing efficiency. In practical applications, at least one of the multiple branches of the MGMN can be processed using the corresponding SCAN (one target region group requires one SCAN for joint enhancement, i.e., the number of SCANs corresponds to the number of target region groups requiring enhancement). This can also achieve similar relationship enhancement for all branches in the MGMN and save computing resources to a certain extent. When selecting reusable ASAUs, all ASAUs in a branch can be selected.

[0138] As can be seen from the above, in the MGMN, the processed image corresponds to at least two target area groups, each of which contains at least one target area. Specifically, for at least one target area group in the MGMN, the following steps are performed: based on the positional relationship between the target areas, the third feature information corresponding to each target area is sequentially input into a preset long short-term memory network, and the third feature information with enhanced relationship is output.

[0139] Specifically, the input order of each third feature information is determined based on the positional relationship of each target area and the centroid information of the third feature information corresponding to each target area. For example, the positional relationship of the target areas can be from top to bottom, from left to right, from inside to outside, etc., which is not limited in this embodiment of the present application. In actual applications, the input order of each third feature information can be determined by the Adjustment module.

[0140] Furthermore, the feature information of each set of relationship enhancement is fused, specifically, a full connection operation is performed on the feature information of each set of relationship enhancement, that is, Figure 14a As shown, a fully connected layer FC_SCAN is used to connect at least one SCAN( Figure 14a In the example above, only one target region group is subjected to joint enhancement processing. In this case, only one SCAN output result is required to perform a full connection operation to obtain a first intermediate full connection result.

[0141] In addition, in MGMN, the feature information of each branch needs to be fused, and a fully connected operation is performed on the fusion processing result through a fully connected layer FC_MGMN to obtain the second intermediate fully connected result.

[0142] Subsequently, the first intermediate fully connected result and the second fully connected result are input into a fully connected layer (FC), and a fully connected operation is performed on the first intermediate fully connected result and the second fully connected result through the fully connected layer to obtain second feature information corresponding to the target area of the image to be processed.

[0143] As an example, Figure 15 As shown, the MGMN contains a branch with three ASAUs. All ASAUs in this branch are selected for reuse. After the ASAU processing results in this branch are processed and the input order is determined by the adjustment module, they can be directly used as input to the LSTM in SCAN, improving processing efficiency.

[0144] In an embodiment of the present application, a MGMN, a SCAN, and a fully connected layer (FC) may be included in a feature representation network. The inputs to both the MGMN and the SCAN are the first feature information output by the backbone network, or the input to the MGMN is the first feature information output by the backbone network, while the input to the SCAN is information shared by the MGMN (i.e., information shared by the ASAUs in the MGMN). The outputs of the MGMN and the SCAN are each fully connected and then input to the fully connected layer, which outputs the second feature information corresponding to the target region of the image to be processed.

[0145] It should be noted that the above-mentioned fusion processes in the embodiments of the present application can adopt any fusion method and can also be processed through a fully connected layer (FC).

[0146] In the embodiment of the present application, the convolution processing parameters during the convolution processing can be obtained by pre-learning or training.

[0147] The embodiment of the present application provides an offline training method for a feature representation network, such as Figure 14b As shown, the MGMN and SCAN can be included at the same time. In particular, the MGMN and SCAN can deploy independent ASAUs respectively or share ASAUs.

[0148] For each branch in the MGMN, the output results of each ASAU in the branch are fused (e.g., concatenated), and then the network is optimized based on the obtained feature information and various target loss functions. The target loss functions may include: SoftMax loss function, Lifted Structured Feature Embedding (LSFE) loss function, and Euclidean loss function.

[0149] For the SoftMax loss function and the LSFE loss function, the feature information obtained after fusion can be subjected to a global average pooling (GAP) operation. The sizes of the feature information obtained after pooling of each branch are 1*512, 2*512, and 3*512 respectively.

[0150] The obtained feature information is fully connected through the fully connected layer to obtain the corresponding feature information, such as Figure 14b As shown, the feature information corresponding to each branch can be f1 g , f1 p2 , f1 p3 , whose sizes are 1*256, 1*256, and 1*512 respectively.

[0151] For the above f1 g , f1 p2 , f1 p3 For each feature information in , two full connection operations can be performed separately to obtain feature information of different dimensions. For example, a feature map of size 1*N can be obtained, where N can be the number of object categories contained in the database, or a feature map of size 1*512 can be obtained.

[0152] When optimizing the network based on the obtained 1*N feature map and SoftMax loss function, the 1*N feature map is matched with the true value of the corresponding feature information. If the match is consistent, the target loss function is minimized. If it is not minimized, the parameters in the network are adjusted.

[0153] When optimizing the network based on the obtained 1*512 feature map and LSFE loss function, the 1*512 feature map is matched with the true value of the corresponding feature information. If the match is consistent, the target loss function is minimized. If it is not minimized, the parameters in the network are adjusted.

[0154] For the Euclidean loss function, when fusing the results of the outputs of each ASAU, the attention maps obtained by each ASAU can be fused, and the fused feature information can be matched with the true value of the corresponding feature information. If the match is consistent, the target loss function is minimized. If it is not minimized, the parameters in the network are adjusted.

[0155] If SCAN reuses a branch containing three ASAUs in MGMN, the results of each ASAU output are GAPed separately, and the obtained feature information is input into the LSTM network in the adjusted order to obtain three feature information, such as three feature maps of size 1*1*U. The three feature information are then fused (such as splicing) to obtain concatenated feature information, such as a feature map of size H*U. For the obtained feature information, two full connection operations can be performed respectively to obtain feature information of different dimensions, for example, a feature map of size 1*N can be obtained, where N can be the number of object categories contained in the database, or a feature map of size 1*512.

[0156] When optimizing the network based on the obtained 1*N feature map and SoftMax loss function, the 1*N feature map is matched with the true value of the corresponding feature information. If the match is consistent, the target loss function is minimized. If it is not minimized, the parameters in the network are adjusted.

[0157] When optimizing the network based on the obtained 1*512 feature map and LSFE loss function, the 1*512 feature map is matched with the true value of the corresponding feature information. If the match is consistent, the target loss function is minimized. If it is not minimized, the parameters in the network are adjusted.

[0158] Figure 14b For online processing, the above f1 g , f1 p2 , f1 p3 The fused feature information can be the feature information finally output by MGMN. If SCAN is not set during online processing, the feature information finally output by MGMN is the feature information corresponding to the image to be processed, and this feature information is used for image retrieval.

[0159] exist Figure 14b In the figure, the upper solid-line box represents the MGMN, and the lower dashed-line box represents the SCAN. The SCAN can be deployed during the offline training phase to optimize the MGMN network. However, during the online processing phase, only the MGMN can be deployed without the SCAN.

[0160] To learn more discriminative features, we used the SoftMax loss function for classification and the LSFE loss function for metric learning during offline training. In addition, we designed a Part Partition Loss (PPL) function to extract regions with semantic pattern information.

[0161] Many methods have previously shown that semantic descriptions of clothing can help with clothing retrieval tasks, such as clothing information such as texture and type. In this application, we use this clothing semantic information and use a SoftMax loss function for each clothing attribute to predict accurate attributes.

[0162] In addition, to learn the similarity of the appearance of identical garments, existing methods organize training data into pairs, while others organize training data into triplets. However, neither approach fully exploits the relationships within the training data and is difficult to optimize. Therefore, we use the LSFE loss function, which maximizes the proximity of similar garments and distances dissimilar garments.

[0163] When performing image retrieval, more local information helps improve retrieval performance. Existing methods design group clustering loss functions to group similar and nearby specific components together. In addition, some existing methods use the mean square error function to constrain the Euclidean distance between the predicted attention map and the true attention map, thereby achieving the purpose of predicting the position distribution of each annotation point.

[0164] To enable the ASAU in the MGMN to focus on each region containing specific semantics, we also designed a mean square error function, also known as PPL. The actual attention map in this loss function can be calculated from the existing feature map.

[0165] More specifically, this loss function is designed as follows:

[0166]

[0167] Among them, L ppl is the PPL loss function; N is the number of training data used in each iteration; u and v are the rows and columns of the attention map respectively; the prediction map S N (u,v) is the attention map in ASAU, with a size of 7*7; the real picture The calculation is as follows: first find the maximum point in the attention map, then apply a 2D Gaussian filter. Note that in the MGMN, branches containing two ASAUs use PPL optimization, and the two maximum points are found in rows [0-3] and [4-6], respectively. For branches containing three ASAUs, the three maximum points are found in rows [0-2], [3-4], and [5, 6], respectively.

[0168] In the embodiment of the present application, the offline training process of the feature representation network includes:

[0169] The feature representation network can be trained based on the training samples first. The training process is consistent with the above online process. The feature information extracted for the training samples is output. According to the target loss function (such as the SoftMax function), the extracted feature information is matched with the training samples. If the match is consistent, the target loss function is minimized.

[0170] In the embodiment of the present application, each step of the image retrieval method can be executed by a server device, or can be executed locally on a terminal device, or some steps can be executed on the server and some steps can be executed on the terminal device.

[0171] In general, the image retrieval method provided in the embodiments of the present application has the following advantages:

[0172] 1) It not only extracts global information about the target region of the image being processed, but also captures rich, detailed information at varying granularities. Typically, viewing an image from different regions reveals different information. While a holistic view might reveal the overall outline of an object, delving deeper into a specific part reveals more specific details. To obtain richer feature information from an image, it's necessary to combine information at different granularities. Finally, a fusion process yields more accurate feature information, thereby improving retrieval accuracy.

[0173] 2) The relative position relationship of different objects in the image is considered and simulated, so that the objects in the image become a tight whole, so that the enhanced feature information is more distinctive and the retrieval effect is better.

[0174] The image retrieval method provided in the embodiment of the present application will be introduced below in conjunction with the application scenario of clothing retrieval.

[0175] When people see an item of clothing that interests them in a real-life scenario, they may want to search online for similar clothing to purchase or collect, so the terminal device needs to provide a convenient and accurate clothing retrieval function. For example, when people see a top that they are very interested in in a real-life scenario, they can use the terminal device to take a picture of the top. The clothing retrieval function will retrieve pictures or product information of clothing that is consistent / similar to the clothing in the picture taken by the user from a database, e-commerce website, or search website based on the picture taken by the user. In addition, the terminal device can further provide the user with a clothing recommendation function. For example, if a user takes a picture of a piece of clothing in reality, the terminal device uses the clothing retrieval function to retrieve consistent or similar clothing, and then determines other clothing that is pre-set for the clothing or automatically determined in real time, and provides them to the user for reference, such as Figure 16a As shown, the user takes a picture of a top, and the terminal device provides the user with pictures of bags and skirts that match the top.

[0176] Specifically, when a user uses a terminal device, they use the camera to take a real-time picture of the clothing they are interested in. The terminal device provides a search function on the interface. If the user clicks on the search function, it means that the user wants to search for the clothing in the picture. The terminal device then uploads the picture taken by the user to the cloud server.

[0177] The process of performing clothing retrieval may include the steps of feature extraction and retrieval mentioned in the above embodiment. Figure 16b As shown in the figure, the process of clothing retrieval can specifically include four steps: clothing classification, object detection, feature extraction, and retrieval. The implementation of each step is described in detail below:

[0178] Step 1: Classify the clothing of the image input by the user. This step can be performed by the cloud server.

[0179] The user's input image can be an image captured in real time by the user using a terminal device (such as an image capture device capturing clothing images in real scenes in real time), or the user can also use an already captured image (such as an image stored in the terminal device, or an image downloaded by the user online, etc.).

[0180] When using a terminal device, the user can use the camera to take real-time pictures of the clothing of interest. The terminal device provides an entrance to the search function on the interface. If the user clicks on the entrance, it means that the user wants to retrieve the clothing in the picture. The terminal device uploads the picture taken by the user to the cloud server.

[0181] First, use a classification algorithm to classify the clothing in the image and determine the category to which the clothing belongs. Clothing categories include tops, bags, pants, skirts, shoes, etc. Among them, classification algorithms such as Googlenet and VGGnet can be used to classify clothing in the image input by the user.

[0182] According to the above classification algorithm, the category of the clothing in the image, such as skirt, can be obtained. Of course, if there are multiple clothing items in the image, this step can also obtain multiple clothing categories.

[0183] Step 2: Based on the obtained clothing category, further determine the specific location of the clothing in the image, that is, object detection. This step can be performed by the cloud server.

[0184] Among them, each category can correspond to an object detector, for example, the object detector includes a top detector, a bag detector, a pants detector, a skirt detector, a shoe detector, and so on.

[0185] Based on the category of clothing in the image obtained in step 1, object detection is performed using the detector corresponding to the category to obtain the specific location of the clothing in the image. This specific location can represent the specific location of the clothing area in the image, such as (x, y, w, h), where x and y can be the coordinates of the upper left corner of the clothing area (also called clothing detection frame) in the image, w is the width of the clothing area, and h is the height of the clothing area.

[0186] Of course, if there are multiple garments in the image, object detection can be performed on each garment separately in this step to obtain the specific location of each garment.

[0187] Among them, the specific position of the clothing can be automatically detected for the user to select and confirm, or the user can correct the specific position of the automatically detected clothing to obtain more accurate position information; if the positions of multiple clothing are detected, the user can further select the clothing to be retrieved from the multiple clothing, that is, the clothing to be retrieved.

[0188] Step 3: Based on the specific location of the clothing to be retrieved output in step 2, a feature extraction network is used to extract features from the regional image of the clothing to be retrieved, and feature information is output. The feature information can be a one-dimensional feature vector, i.e., feature extraction. This step can be performed by a cloud server.

[0189] The overall structure of the feature extraction network can be composed of the feature representation network and the backbone network provided in any of the above embodiments. The backbone network receives the detection results output by the object detection and obtains a regional image of the garment to be retrieved based on the specific location of the garment to be retrieved. The feature representation network maps the feature map output by the backbone network into a fixed-length feature vector, ultimately obtaining a 512-dimensional or 1024-dimensional feature vector. This feature vector generally contains higher-level semantic information and better represents the characteristics of the garment to be retrieved.

[0190] Step 4, using the feature information of the clothing to be retrieved extracted in step 3, matches it with the feature information of each clothing image in the clothing database, and determines the clothing image that matches the clothing to be retrieved in the clothing database. This step can be performed by the cloud server.

[0191] Step 5: Provide the matched clothing image and related information to the user. The above related information can be the link information of the clothing, or the matching clothing image, etc. This step can be performed by the terminal device.

[0192] The image retrieval method provided in the embodiments of this application is applied to clothing retrieval scenarios, and compared with the existing technology, it can solve the following problems:

[0193] 1) In existing technologies, background clutter significantly interferes with accurate retrieval. Existing feature representation networks fail to distinguish between background and object information, incorporating background information into the final feature information. This results in the extracted features failing to accurately describe the clothing, creating increased interference and leading to inaccurate retrieval results.

[0194] In the embodiment of the present application, by processing at least one ASAU, convolution processing can be used to extract the characteristic information of the main target area and filter out cluttered interference areas, such as background areas, thereby improving the accuracy of the retrieval results.

[0195] 2) In the prior art, the shape change of clothing caused by the wearer's posture or perspective has a great impact on the performance of the retrieval results.

[0196] The inventors of this application discovered that even if the shape of clothing changes significantly, its local features are relatively stable. For example, even if the shape of clothing changes, the position and style of its buttons do not change much, or the style of the collar does not change much. If these local features can be used to represent clothing, more accurate retrieval results can be obtained.

[0197] Therefore, in the embodiment of the present application, the feature representation network can be processed by at least two ASAUs, each ASAU focuses on different areas of the clothing (for example, ASAU1 mainly focuses on the area where the collar is located, and ASAU2 mainly focuses on the area where the pockets are located), that is, further extracting the local feature information of each area of the clothing. Since the local features are relatively stable and are not greatly affected by the shape changes of the clothing, more robust features can be obtained, which greatly improves the accuracy of the retrieval results.

[0198] Furthermore, in the embodiment of the present application, each ASAU is processed separately based on different scales, and feature information of a local area of clothing can be obtained that can reflect various detail information, such as more specific detail information such as the collar style or pocket style of the clothing.

[0199] Furthermore, in an embodiment of the present application, the feature representation network can be a multi-branch structure, each branch contains at least one ASAU, and each branch represents a local area of a different scale. For example, a branch containing one regional processing unit can be considered to focus on the global information of the clothing, and a branch containing two or three ASAUs focuses on the detailed information of the clothing, thereby obtaining more comprehensive feature information and making the retrieval results more accurate.

[0200] 3) The inventors of this application discovered that the feature information output by the feature extraction network of the prior art lacks spatial constraints. When searching in a database with a huge number of clothing images, since many clothing images may have only slight differences, if the relative relationship between each object in the clothing, that is, the spatial constraint, is not taken into account, the accuracy of the retrieval results will be greatly reduced.

[0201] For example, Figure 16c As shown, Figure 16c The two images above are the images input by the user. The images contain the clothing to be retrieved, so the two images above can also be called clothing images to be retrieved. Figure 16c The two images at the bottom of the figure are search results generated without considering spatial constraints. As you can see, without considering spatial constraints, the search result for striped shorts is an image composed of multiple stockings. This image also contains striped regions, but is composed of multiple items of clothing. Furthermore, the search result for an image of both a top and pants is a single top with a color boundary. Therefore, when the extracted feature information lacks spatial constraints, the search results are less accurate and may yield erroneous results.

[0202] In the embodiment of the present application, an adjustment module and an LSTM mechanism are introduced to consider and simulate the relative positional relationship of different objects on clothing. Because clothing actually contains several local objects, such as collars, buttons, sleeves and pockets, etc., and these objects have a natural positional relationship, such as the collar will be above the pocket, and the pocket will be next to the button, etc. The adjustment module can automatically arrange the relative positional relationship of each object on the clothing. The feature information of different objects can be extracted through multiple ASAUs, and the adjustment module is designed to determine the input order of each feature information. The LSTM mechanism is then used to simulate the relative positional relationship between each region, and the feature information output by each ASAU is jointly enhanced to enhance the spatial constraint so that the processed feature information can reflect the spatial relationship between each region, which can greatly improve the retrieval effect of clothing.

[0203] In general, applying the image retrieval method provided in the embodiments of the present application to clothing retrieval scenarios can improve the accuracy of clothing retrieval and enable users to obtain more satisfactory clothing information.

[0204] The evaluation results of the performance improvement in this scenario are given below. The embodiment of the present application performs a performance evaluation on the above-mentioned solution based on an existing clothing image database. The database includes three types of clothing: tops, skirts, and pants. The number of images in each category and the number of images to be retrieved by users are shown in Table 1.

[0205] Jacket skirt Pants The number of images to be retrieved by the user 563 366 289 Number of images in the database 648153 396702 372250

[0206] Table 1

[0207] Table 2 shows the objective evaluation results of three schemes on this dataset. The retrieval effect can be expressed by retrieval accuracy. These three schemes are: the existing technology scheme, the MGMN scheme proposed in the embodiment of this application, and the MGMN+SCAN scheme proposed in the embodiment of this application.

[0208] As can be seen from Table 2, the MGMN solution proposed in this embodiment of the present application achieves approximately 2% better retrieval performance than the prior art solution. This is because, for clothing feature information, global information can only roughly describe the overall contour of the clothing, weakening or even ignoring the characteristics of local information. The multiple ASAU branches included in the MGMN solution can not only extract global information but also extract multi-granular local information, thereby obtaining a more complete clothing feature representation and improving the accuracy of clothing retrieval.

[0209] Furthermore, Table 2 shows that the MGMN+SCAN solution achieves approximately 2% better retrieval performance than the MGMN solution. This is due to the relative positional relationships between individual objects within a garment, i.e., the necessity of spatial constraints. If spatial constraints between objects are ignored, the relationships between them are loose. During clothing retrieval, two similar but incompletely matching garments may achieve a high degree of match. However, by taking into account the relative positional relationships between objects and treating them as a whole, similar but incompletely matching garments can be eliminated.

[0210]

[0211]

[0212] Table 2

[0213] like Figure 16d As shown in the figure, the subjective evaluation effect of the image retrieval scheme proposed in the embodiment of the present application and the prior art scheme is that by inputting two clothing pictures to be retrieved, only two retrieval results are returned for each picture. From the first retrieval result, the image retrieval scheme proposed in the embodiment of the present application can find clothing with the same collar type as the clothing picture to be retrieved because it takes into account more detailed information, while the prior art scheme can only retrieve clothing with relatively similar appearance. In the second retrieval result, although both schemes can find clothing with grids or lattices, since the image retrieval scheme proposed in the embodiment of the present application takes into account the dependency relationship of each local object of clothing, it can find clothing images with a high degree of match with the clothing picture to be retrieved, while the prior art scheme only finds partially similar clothing pictures.

[0214] It should be noted that the above-mentioned clothing retrieval scenarios are merely examples and should not be construed as limiting the present application. Appropriate variations based on these examples are also applicable to the present application. For example, those skilled in the art can easily transfer the retrieval method provided in the embodiments of the present application to scenarios such as video surveillance, which should also be included in the scope of protection of the present application. For example, the technical solution proposed in the embodiments of the present application can be used to extract the features of pedestrians in a video. After obtaining the pedestrian's feature information, the user can search all videos in the database based on the features of the pedestrian in the input video, and finally, through matching calculations, determine whether the video data of the pedestrian exists in the database.

[0215] The present application also provides an image retrieval device, such as Figure 17 As shown, the image retrieval device 170 may include: a feature extraction module 1701, a feature information processing module 1702 and an image retrieval module 1703, wherein:

[0216] The feature extraction module 1701 is used to extract features from the image to be processed to obtain first feature information;

[0217] The feature information processing module 1702 is used to perform feature information processing on the first feature information to obtain second feature information corresponding to the target area of the image to be processed;

[0218] The image retrieval module 1703 is used to perform image retrieval processing on the image to be processed according to the second feature information.

[0219] In an optional implementation, the feature information processing module 1702 is specifically used to perform corresponding characteristic information processing on the first feature information for at least one target area to obtain third feature information corresponding to each target area; and to fuse each third feature information to obtain second feature information corresponding to the target area of the image to be processed.

[0220] In an optional implementation, the feature information processing module 1702 is specifically configured to perform convolution processing on the first feature information based on a first convolution processing parameter corresponding to the target area to obtain third feature information corresponding to the target area.

[0221] In an optional implementation, the feature information processing module 1702 is specifically used to perform convolution processing on the first feature information based on the first convolution processing parameters corresponding to the target area; normalize the feature information obtained after the convolution processing; perform a stacking operation on the normalized feature information; and fuse the feature information after the stacking operation with the first feature information to obtain third feature information corresponding to the target area.

[0222] In an optional implementation, the feature information processing module 1702 is specifically used to perform convolution processing on the first feature information at least twice based on the first convolution processing parameters corresponding to the target area, and obtain each fourth feature information corresponding to the target area, wherein the first convolution processing parameters used in each convolution processing are different; and fusion processing is performed based on each fourth feature information to obtain the third feature information corresponding to the target area.

[0223] In an optional implementation, the feature information processing module 1702 is specifically used to perform at least two convolution processes on the first feature information based on the first convolution processing parameters corresponding to the target area, wherein the first convolution processing parameters adopted for each convolution process are different; for the feature information obtained after each convolution process, the following processing is performed respectively: normalizing the feature information obtained after the convolution process, and performing a stacking operation on the feature information after the normalization process, and fusing the feature information after the stacking operation with the first feature information to obtain the corresponding fourth feature information.

[0224] In an optional implementation, the feature information processing module 1702 is specifically configured to perform at least two convolution processes on the first feature information based on the first convolution processing parameters corresponding to the target area, wherein the first convolution processing parameters used in each convolution process are different; and perform normalization processing on the feature information obtained after each convolution process to obtain corresponding fourth feature information.

[0225] In addition, the feature information processing module 1702 is specifically used to perform fusion processing based on each fourth feature information; perform a stacking operation on the feature information obtained after the fusion; and fuse the feature information after the stacking operation with the first feature information to obtain the third feature information corresponding to the target area.

[0226] In an optional implementation, the feature information processing module 1702 is specifically configured to perform fusion processing on each piece of fourth feature information based on the weight corresponding to each piece of fourth feature information.

[0227] In an optional implementation, the feature information processing module 1702 is further configured to perform convolution processing on the first feature information based on a second convolution processing parameter corresponding to the target area; and obtain weights corresponding to each fourth feature information according to the result of the convolution processing.

[0228] In an optional implementation, the feature information processing module 1702 is specifically configured to perform normalization processing on the result of the convolution processing to obtain weights corresponding to respective pieces of fourth feature information.

[0229] In an optional implementation, the convolution processing parameters include at least one of the following: a convolution kernel, a convolution kernel size, an expansion factor, and a convolution step size.

[0230] In an optional implementation, the feature information processing module 1702 is specifically configured to perform a full connection operation based on each piece of third feature information to obtain second feature information corresponding to the target area of the image to be processed.

[0231] In an optional implementation, the image to be processed corresponds to at least two target area groups, each target area group contains at least one target area; the feature information processing module 1702 is specifically used to fuse the third feature information corresponding to each target area in each target area group to obtain the fifth feature information corresponding to each target area group; and fuse each fifth feature information to obtain the second feature information corresponding to the target area of the image to be processed.

[0232] In an optional implementation, the feature information processing module 1702 is specifically configured to perform a full connection operation based on each fifth feature information to obtain second feature information corresponding to the target area of the image to be processed.

[0233] In an optional implementation, the feature information processing module 1702 is specifically configured to perform a full connection operation based on each fifth feature information to obtain second feature information corresponding to the target area of the image to be processed.

[0234] In an optional implementation, the feature information processing module 1702 is further used to input the third feature information corresponding to each target area into a preset long short-term memory network in sequence according to the positional relationship of each target area, and output the third feature information after the relationship is enhanced.

[0235] In an optional implementation, the image to be processed corresponds to at least two target area groups, each target area group contains at least one target area; the feature information processing module 1702 is also used to execute, for at least one target area group: according to the relative position relationship of each target area, the third feature information corresponding to each target area is input into a preset long short-term memory network in sequence, and the third feature information with enhanced relationship is output.

[0236] In an optional implementation, the feature information processing module 1702 is specifically used to determine the input order of each third feature information based on the positional relationship of each target area and the centroid information of the third feature information corresponding to each target area; and according to the determined input order, each third feature information is input into the preset long short-term memory network in sequence.

[0237] Those skilled in the art can clearly understand that the image retrieval device provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the convenience and conciseness of description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment, and no further details will be given here.

[0238] An embodiment of the present application also provides an electronic device (e.g., a terminal device), comprising: a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, at least one program, a code set, or an instruction set being loaded and executed by the processor to implement the corresponding content in the aforementioned method embodiment.

[0239] Optionally, the electronic device may further include a transceiver. The processor and the transceiver are connected, for example, via a bus. It should be noted that in actual applications, the number of transceivers is not limited to one, and the structure of the electronic device does not constitute a limitation on the embodiments of the present application.

[0240] The processor may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. A processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0241] The bus may include a path for transmitting information between the above components. The bus may be a PCI bus or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory may be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, a CD-ROM or other optical disk storage, an optical disk storage (including a compact disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0242] An embodiment of the present application further provides a computer-readable storage medium, which is used to store computer instructions. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content of the aforementioned method embodiment.

[0243] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0244] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method performed by an electronic device, characterized in that: include: Based on the first neural network, feature extraction is performed on the image to be processed to obtain first feature information, wherein the image to be processed includes at least two target areas; Processing the first feature information based on a second neural network and filtering operations at at least two scales to obtain at least two third feature information corresponding to the at least two target areas of the image to be processed; inputting the at least two third feature information into a third neural network in sequence according to the positional relationship between the at least two target areas to obtain second feature information; performing image retrieval processing on the image to be processed according to the second feature information; The at least two third feature information are obtained, including: performing convolution processing on the first feature information at least twice based on the first convolution processing parameters corresponding to the at least two target areas, to obtain a plurality of fourth feature information corresponding to the at least two target areas, respectively, wherein the first convolution processing parameters used in each convolution processing are different; At least two pieces of third feature information are obtained based on the plurality of fourth feature information.

2. The method according to claim 1, characterized in that The third feature information includes feature information corresponding to the target area of the image to be processed and filtering out interference areas; The target area is a region of interest obtained using a deep learning network, and the interference area is a background area of the target area.

3. The method according to claim 1, characterized in that Perform feature extraction on the image to be processed to obtain first feature information, including: By inputting the image to be processed into the first neural network, first feature information including boundary information of at least two target areas in the image to be processed is obtained.

4. The method according to claim 1, wherein The performing image retrieval processing on the image to be processed according to the second feature information includes: An image corresponding to the image to be processed is identified according to the second feature information.

5. The method according to claim 1, wherein The performing image retrieval processing on the image to be processed according to the second feature information includes: By comparing the second feature information with feature information of a plurality of images, an image corresponding to the image to be processed is searched in a database storing the plurality of images.

6. The method according to claim 5, characterized in that The comparing the second feature information with feature information of a plurality of images includes: The distance between the second feature information and feature information of a plurality of images is determined.

7. The method according to claim 6, characterized in that Also includes: The determined distances are sorted to identify the minimum distance.

8. The method according to claim 6, characterized in that The distance between the second feature information and the feature information of the plurality of images is a cosine distance between the second feature information and the feature information of the plurality of images.

9. The method according to claim 5, characterized in that The plurality of images stored in the database and / or the images to be processed include images taken by a user.

10. The method according to claim 5, characterized in that Before searching the database, also include: Based on the feature information respectively obtained for each of the plurality of images, feature vectors of the plurality of images are obtained.

11. The method according to claim 5, characterized in that Also includes: When the image to be processed includes a plurality of searchable objects related to the at least two target areas, a target search object is selected from the plurality of searchable objects.

12. The method according to claim 11, characterized in that The selection of target search objects includes: A user selection of a target retrieval object from among the plurality of retrieval objects included in the image to be processed is received.

13. The method according to claim 1, wherein The first feature information includes basic structural information of the image to be processed; The basic structure information includes: Boundary information, intersection and shape information.

14. The method according to claim 1, wherein The first feature information includes information characterizing a spatial relationship between the at least two target areas.

15. The method according to claim 1, wherein The extracting features of the image to be processed to obtain first feature information includes: Perform multi-layer feature extraction on the image to be processed to obtain the first feature information.

16. The method according to claim 1, wherein The at least two third feature information have different scales.

17. The method according to claim 1, wherein Also includes: At least two pieces of third feature information are fused to obtain second feature information.

18. The method according to claim 17, characterized in that The fusion process includes a dot multiplication operation.

19. The method according to claim 1, wherein At least two third feature information are obtained, including: Based on a second neural network and filtering operations at at least two scales, convolution processing is performed on the first feature information to obtain at least two third feature information corresponding to the at least two target areas of the image to be processed.

20. The method according to claim 1, wherein At least two third feature information are obtained, including: Based on at least two different first convolution processing parameters, convolution processing is performed on the first feature information to obtain at least two third feature information corresponding to the at least two target areas of the image to be processed.

21. The method according to claim 20, characterized in that Each first convolution processing parameter includes expansion rates corresponding to different scales.

22. The method according to claim 1, wherein At least two third feature information are obtained, including: performing convolution processing on the first feature information based on first convolution processing parameters corresponding to the at least two target areas; Normalize the feature information obtained after convolution processing; Perform stacking operation on the normalized feature information; The feature information after the stacking operation is fused with the first feature information to obtain at least two third feature information corresponding to the at least two target areas of the image to be processed.

23. The method according to claim 1, wherein The filtering operations at the at least two scales correspond to first convolution processing parameters corresponding to the at least two target regions.

24. The method according to claim 1, wherein Based on the first convolution processing parameters corresponding to the at least two target areas, convolution processing is performed on the first feature information at least twice to obtain a plurality of fourth feature information corresponding to the at least two target areas, respectively, including: performing convolution processing on the first feature information at least twice based on the first convolution processing parameters corresponding to the at least two target areas, wherein the first convolution processing parameters used in each convolution processing are different; For the feature information obtained after each convolution process, the following processing is performed respectively: the feature information obtained after the convolution process is normalized, and the normalized feature information is stacked, and the feature information after the stacking operation is fused with the first feature information to obtain the corresponding fourth feature information.

25. The method according to claim 1, wherein Based on the first convolution processing parameters corresponding to the at least two target areas, convolution processing is performed on the first feature information at least twice to obtain a plurality of fourth feature information corresponding to the at least two target areas, respectively, including: performing convolution processing on the first feature information at least twice based on the first convolution processing parameters corresponding to the at least two target areas, wherein the first convolution processing parameters used in each convolution processing are different; For each feature information obtained after the convolution process, normalize the feature information obtained after the convolution process to obtain corresponding fourth feature information; At least two pieces of third feature information are obtained based on the plurality of fourth feature information, including: Perform fusion processing based on each fourth feature information; Perform stacking operation on the fused feature information; The feature information after the stacking operation is fused with the first feature information to obtain at least two third feature information.

26. The method according to claim 1, wherein At least two pieces of third feature information are obtained based on the plurality of fourth feature information, including: Based on the weights respectively corresponding to the multiple fourth feature information, the multiple fourth feature information are fused to obtain at least two third feature information.

27. The method according to claim 26, characterized in that Before fusing the plurality of fourth feature information based on the weights respectively corresponding to the plurality of fourth feature information, the method further includes: performing convolution processing on the first feature information based on second convolution processing parameters corresponding to the at least two target areas; According to the result of the convolution process, weights corresponding to the plurality of fourth feature information are obtained.

28. The method according to claim 27, characterized in that According to the result of the convolution processing, weights corresponding to the plurality of fourth feature information are obtained, including: The convolution result is normalized to obtain weights corresponding to the plurality of fourth feature information.

29. The method according to any one of claims 20 to 28, characterized in that The convolution processing parameters include at least one of the following: Convolution kernel, convolution kernel size, expansion factor, convolution step size.

30. The method according to claim 17, wherein The at least two pieces of third feature information are fused to obtain second feature information, including: A full connection operation is performed based on the at least two third feature information to obtain second feature information.

31. The method according to claim 1, wherein The image to be processed corresponds to at least two target area groups, and each target area group includes at least one target area.

32. The method according to claim 31, characterized in that Also includes: Acquire fifth characteristic information corresponding to the at least two target area groups respectively to obtain second characteristic information.

33. The method according to claim 17, wherein The at least two pieces of third feature information are fused to obtain second feature information, including: performing fusion processing on the third feature information corresponding to each target area in the at least two target area groups to obtain fifth feature information corresponding to the at least two target area groups; At least two fifth feature information are fused to obtain second feature information.

34. The method according to claim 33, wherein Fusing at least two fifth feature information to obtain second feature information includes: A full connection operation is performed based on at least two pieces of fifth feature information to obtain second feature information.

35. The method according to claim 1, wherein Also includes: determining an input order of at least two pieces of third feature information according to a positional relationship between the at least two target areas; According to the input order, at least two third feature information are sequentially input into the third neural network, and the third feature information with enhanced relationship is output.

36. The method according to claim 1, wherein The image to be processed corresponds to at least two target area groups, each target area group includes at least one target area; The method further includes: For at least one target area group, respectively, the following are performed: according to the positional relationship of each target area, the third feature information corresponding to each target area is sequentially input into the third neural network, and the third feature information after the relationship is enhanced is output.

37. The method according to claim 1, wherein Also includes: determining an input order of the at least two pieces of third feature information according to a positional relationship between the at least two target areas and centroid information of the third feature information respectively corresponding to the at least two target areas; According to the determined input order, at least two third feature information are sequentially input into the third neural network.

38. The method according to claim 1, wherein The first neural network and the second neural network include convolutional neural networks.

39. The method according to any one of claims 35 to 38, wherein: The third neural network includes a long short-term memory network.

40. The method according to claim 1, wherein Also includes: Use at least one adaptive spatial attention unit.

41. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 40.

42. A computer-readable storage medium, characterized in that The computer storage medium is used to store computer instructions, programs, code sets or instruction sets, which, when run on a computer, enable the computer to perform the method according to any one of claims 1 to 40.

Images