Image retrieval method based on visual attention mechanism and symbiotic feature integration

Abstract

The invention discloses an image retrieval method based on a visual attention mechanism and symbiotic feature integration, and relates to the technical field of image retrieval. The method comprises the following steps: firstly, combining Tamura visual statistic features with depth features, and mining a potential symbiosis key mode learned in a depth model from a data set end side; then, on the basis that a two-dimensional Gabor primary function can approximate receptive field characteristics of visual neurons of mammals, global texture information of the image is extracted by using a multi-direction Gabor filter; besides, in order to relieve the texture bias phenomenon of the depth feature, the geometric moment feature and the depth feature are fused, the depth moment feature is innovatively proposed, and the feature can be used for representing the regional shape attribute of the image. Finally, the multi-direction global features, the depth distance features and the depth semantic features are effectively integrated, and a compact, efficient and universal image representation is constructed and used for an image retrieval task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image retrieval, and particularly relates to an image retrieval method based on a visual attention mechanism and a co-occurrence feature integration. BACKGROUND

[0002] With the wide popularity of mobile devices and the rapid development of self-media platforms, picture and video data are growing explosively on the Internet platform, and how to quickly and efficiently retrieve effective information has become a difficulty and challenge in the field of computer vision and multimedia. The image retrieval technology based on content has been one of the research hotspots in the field of computer vision and multimedia.

[0003] At present, two major methods in the field of image retrieval include a model fine-tuning-based method and a multi-feature fusion-based method. The model fine-tuning-based method can obtain excellent retrieval performance on a specific data set, but the method has high demand for computing resources and limited generalization performance. Under this background, the multi-feature fusion-based image retrieval method has become another important research hotspot in the field.

[0004] With the in-depth research, scholars have successively proposed a variety of innovative multi-feature fusion methods, which have achieved remarkable results in improving the retrieval performance. However, despite this, the method still faces two key problems to be solved: on the one hand, most of the existing methods fail to realize the early fusion of multi-features from the spatial structure level, which limits the complementarity and synergistic effect between features; on the other hand, these methods usually pay insufficient attention to the “texture bias” problem existing in deep features, thereby affecting the universality and generalization ability. As is known to all, the visual attention mechanism of the human visual system shows significant efficiency and selectivity when dealing with complex scenes. It can quickly locate and analyze important target areas in detail, while other areas are roughly analyzed or directly ignored. The visual attention mechanism embodies the characteristics of the human visual system in actively selecting and concentrating on the content. Jointing the visual attention mechanism and deep features is expected to improve the focusing ability and recognition degree of deep features on target objects, thereby alleviating the “texture bias” problem. However, the implementation of this mode still faces many challenges. SUMMARY

[0005] One of the purposes of the present application is to provide an image retrieval method based on a visual attention mechanism and a co-occurrence feature integration, so as to solve the compatibility problem between a plurality of different types of features and the texture bias problem existing in deep semantic features when representing images.

[0006] In order to solve the above technical problems, the present application adopts the following technical scheme: an image retrieval method based on a visual attention mechanism and a co-occurrence feature integration, comprising the following steps:

[0007] S1, extract the depth feature of each sample image in the target image set and the auxiliary image set respectively, and calculate the corresponding Tamura contrast statistical quantity feature;

[0008] S2, for the target image set and the auxiliary data set, calculate the average value of the contrast statistical quantity of each channel respectively, and obtain the mean contrast vector of the two data sets;

[0009] S3, sort the mean contrast vector in descending order to obtain the sorted vector and its corresponding channel index list; select the channels ranking in the front and commonly appearing in the two data sets to construct the co-occurring key feature;

[0010] S4, extract the global multi-directional information of the co-occurring key feature, and fuse it with the depth feature to obtain the multi-directional global feature, so as to enhance the directional representation ability of the feature;

[0011] S5, use the geometric moment of the sample image to extract the depth moment feature, and fuse it into the depth feature, to further improve the ability of the feature to represent shape information, so as to alleviate the texture bias problem of the depth feature;

[0012] S6, integrate the global feature vector of each sample image to construct the final representation vector of each sample image;

[0013] S7, based on the final representation vector, calculate the similarity between the image to be searched and the sample image, and return the final search result.

[0014] Preferably, step S1 specifically comprises:

[0015] S1.1, input each image in the image set into the depth model to extract the depth semantic feature of each sample image, and the depth feature of the image can be represented as , wherein and represent the width and height of the feature map respectively, represents the total number of feature maps; the depth feature sets of the target image set and the auxiliary image set are represented as and respectively;

[0016] S1.2, flatten each feature map of the depth feature into a one-dimensional vector form, calculate the Tamura contrast statistical quantity feature of the depth feature corresponding to each image, and the calculation formula is as follows:

[0017] (1);

[0018] wherein, and represent the standard deviation and variance of the one-dimensional vector respectively, denote their mean fourth-order moments, and parameters Through the above formula, the contrast statistic set of all images in the above two data sets can be obtained, denoted as and .

[0019] More preferably, in step S2, the mean contrast vectors of the two data sets are and respectively, and the corresponding representations are:

[0020] (2).

[0021] (3).

[0022] wherein n denotes the total number of images in the target image set; m denotes the total number of images in the auxiliary image set; denotes the contrast statistic feature vector of the th image in the target image set; denotes the contrast statistic feature vector of the th image in the auxiliary image set.

[0023] More preferably, step S3 specifically comprises:

[0024] First, the sorted channel index lists and are obtained by sorting; then, the channel index lists are subdivided into multiple smaller subsets according to the percentage, thereby obtaining the sets and ; wherein and , and ;

[0025] Then, the index set of the key channels is obtained by identifying the common channels in the two data sets , and the calculation formula is as follows:

[0026] (4).

[0027] wherein the value range of the parameter is , denotes the number of feature maps;

[0028] Finally, for each image, the co-occurring key features are generated by aggregating the key channels according to the obtained key channel index set; and the calculation formula is as follows:

[0029] (5).

[0030] wherein, is the feature map of the i-th channel in the original depth feature F.

[0031] More preferably, step S4 specifically comprises:

[0032] Firstly, a variant of two-dimensional Gabor function is adopted to extract the multi-directional detail information of the co-occurrence key feature, and the formula definition of the variant is as follows:

[0033] (6) ;

[0034] wherein,

[0035] ;

[0036] The parameters of represent the center of the Gabor function receptive field, and the parameters of and respectively represent the standard deviation of the receptive field in the axis and the axis direction, which determines the size of the receptive field, and the ratio represents the spatial aspect ratio, which determines the ellipticity of the Gabor function. In the present application, is set to , and , wherein and respectively represent the width and height of the feature map;

[0037] Then, the generated Gabor filter is used to filter the co-occurrence key feature to extract its multi-directional detail information, and the calculation formula is as follows:

[0038] (7) ;

[0039] Finally, after the above features are normalized, they are fused with the original depth feature to obtain the multi-directional global feature ; the calculation formula is as follows:

[0040] (8) ;

[0041] wherein, represents L2 normalization.

[0042] More preferably, step S5 specifically comprises:

[0043] Firstly, for each feature map of the multi-directional global feature , its two-dimensional order geometric moment is defined as follows:

[0044] (9);

[0045] wherein, ., ;

[0046] Then, the obtained geometric moments are normalized to obtain the depth moment features ; the calculation formula is as follows:

[0047] (10);

[0048] wherein, is a balance factor, which is a positive number, and is used to ensure the stability of the calculation; the parameter is a normalization factor, which has a value range of , and is used to adjust the degree of normalization;

[0049] Finally, the multi-direction global feature is fused with the depth moment feature to construct a fused global feature vector ; the calculation formula is as follows:

[0050] (11);

[0051] wherein, denotes the multi-direction global feature after pooling; and:

[0052] (12).

[0053] More preferably, in step S6, the integrated global feature vector of each sample image is sequentially subjected to L2 normalization, PCA whitening, PCA, and again L2 normalization operation to obtain the final feature vector of each sample image:

[0054] (13);

[0055] wherein, denotes an L2 normalization function, denotes PCA whitening, denotes principal component analysis; wherein the parameter is a whitening parameter matrix, denotes the final selected feature dimension.

[0056] More preferably, step S7 specifically comprises:

[0057] First, the same processing as steps S3-S6 is performed on the current image to be searched to obtain the final feature vector of the current image to be searched;

[0058] Then, the Euclidean distance between the final representation vector of the current image to be retrieved and the final representation vector of each sample image in the retrieval sample image library is calculated, and the final retrieval results are returned in ascending order of Euclidean distance.

[0059] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0060] 1. This invention introduces Tamura contrast statistics and collaboratively utilizes information from both target and auxiliary datasets to filter key feature channels. This allows for precise focusing on the most discriminative content in an image, effectively simulating the attention mechanism of human vision. This method avoids interference from irrelevant background information, thereby significantly improving the accuracy and relevance of retrieval.

[0061] 2. By integrating geometric moment features into depth features, depth moment features that can characterize the shape attributes of a region are formed. This innovation directly addresses the texture bias problem existing in current depth features, enhancing the feature's ability to depict the essential shape of an object. This makes the retrieval system less likely to be misled by changes in the texture of the object's surface, thus exhibiting stronger robustness and generalization ability when facing complex scenes such as changes in lighting, occlusion, and complex backgrounds.

[0062] 3. The invention does not rely solely on a single feature, but rather constructs a multi-layered, multi-faceted feature integration framework. This method effectively fuses semantic information derived from deep models, global multi-directional texture information extracted through Gabor filters, and geometric moment information representing shape, generating a final image feature that is complementary and comprehensively representative. This feature can describe image content from multiple dimensions, greatly enhancing the feature's discriminative power and ensuring high-quality retrieval results.

[0063] 4. The image representation vector finally constructed by this invention is compact and low-dimensional. Compared with directly using high-dimensional original deep features or complex multi-model fusion methods, the representation vector generated by this method greatly reduces storage costs and computational overhead, making it faster and more responsive when calculating the similarity between the image to be retrieved and samples in a massive image database. This makes it very suitable for large-scale image retrieval applications. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the experimental detection results in the example. Detailed Implementation

[0065] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0066] An image retrieval method based on visual attention mechanisms and symbiotic feature integration includes the following steps:

[0067] S1. Extract the depth features of each sample image in the target image set and the auxiliary image set, and calculate the corresponding Tamura contrast statistics; specifically:

[0068] S1.1 Input each image from each image set into the depth model and extract the depth semantic features of each sample image. The depth features of this image can be represented as follows: ,in and These represent the width and height of the feature map, respectively. The total number of feature maps is represented by: The depth feature sets of the target image set and the auxiliary image set are represented as: and ;

[0069] S1.2 Flatten each feature map of the depth features into a one-dimensional vector, and calculate the Tamura contrast statistic feature of the corresponding depth feature for each image. The calculation formula is as follows:

[0070] (1);

[0071] in, and Let these represent the standard deviation and variance of a one-dimensional vector, respectively. It represents its mean fourth moment, parameter Using the above formulas, we can obtain the set of contrast statistics for all images in the two datasets mentioned above, which are expressed as follows: and .

[0072] S2. For the target image set and the auxiliary dataset, calculate the average value of the contrast statistics for each channel to obtain the mean contrast vector of the two datasets; specifically:

[0073] The mean contrast vectors of the two datasets are respectively and The corresponding representation is:

[0074] (2);

[0075] (3);

[0076] Where n represents the total number of images in the target image set; m represents the total number of images in the auxiliary image set; Indicates the first image in the target image set The contrast statistics feature vector of the image; Represents the first image in the auxiliary image set. The contrast statistics feature vector of the image.

[0077] S3. Sort the mean contrast vectors in descending order to obtain the sorted vectors and their corresponding channel index lists; select the top-ranking and co-occurring channels from the two datasets to construct co-occurring key features; specifically:

[0078] First, sort the channel indices to obtain a sorted list. and Then, the channel index list is subdivided into several smaller subsets based on percentage, thereby obtaining separate sets. and set ;in, and ,and ;

[0079] Then, by identifying channels that co-occur in both datasets, an index set of key channels is obtained. The calculation formula is as follows:

[0080] (4);

[0081] Among them, parameters The range of values ​​is , Indicates the number of feature maps;

[0082] Finally, for each image, based on the obtained key channel index set, these key channels are aggregated to generate its co-occurring key features. The calculation formula is as follows:

[0083] (5);

[0084] In the formula, It is the feature map of the i-th channel in the original depth feature F.

[0085] S4. Extract global multi-directional information from the co-occurring key features and fuse it with the deep features to obtain multi-directional global features, thereby enhancing the directional representation capability of the features; specifically:

[0086] First, a variant of the two-dimensional Gabor function is used to extract multi-directional detail information of co-occurring key features. The formula for this variant is defined as follows:

[0087] (6);

[0088] in,

[0089] ;

[0090] Parameter pair This represents the center of the receptive field of the Gabor function, and the parameters are... and They represent the feeling of being in the wild. shaft and The standard deviation along the axis determines the size of the receptive field, and the ratio... The aspect ratio represents the spatial dimensions and determines the ellipticity of the Gabor function. In this invention, Set as ,and ,in and These represent the width and height of the feature map, respectively.

[0091] Then, the generated Gabor filter is used to filter the co-occurring key features to extract their multi-directional detailed information. The calculation formula is as follows:

[0092] (7);

[0093] Finally, after normalizing the above features, they are fused with the original deep features to obtain multi-directional global features. The calculation formula is as follows:

[0094] (8);

[0095] in, This indicates L2 normalization.

[0096] S5. Extract depth moment features using the geometric moments of the sample image and integrate them into the depth features to further enhance the ability of the features to represent shape information, thereby alleviating the texture bias problem of the depth features; specifically:

[0097] First, for multi-directional global features Each feature map defines its two-dimensional... The first geometric moments are as follows:

[0098] (9);

[0099] in, ., ;

[0100] Then, the obtained geometric moments are normalized to obtain the depth moment features. The calculation formula is as follows:

[0101] (10);

[0102] in, It is a balancing factor, taking the value of a positive number, used to ensure the stability of the calculation; parameter It is the normalization factor, and its value range is... , used to adjust the degree of normalization;

[0103] Finally, multi-directional global features It is fused with depth moment features to construct a fused global feature vector. The calculation formula is expressed as follows:

[0104] (11);

[0105] in, Representation and pooling of multi-directional global features; and:

[0106] (12).

[0107] S6. Integrate the global feature vectors of each sample image to construct the final representation vector for each sample image; specifically:

[0108] The integrated global feature vector of each sample image is sequentially subjected to L2 normalization, PCA whitening, PCA, and another L2 normalization operation to obtain the final representation vector of each sample image:

[0109] (13);

[0110] In the formula, This represents the L2 normalization function. This indicates PCA whitening. This represents principal component analysis; where the parameters are... The whitening parameter matrix, This indicates the final selected feature dimension.

[0111] S7. Perform the same processing as steps S3 to S6 on the current image to be retrieved to obtain the final representation vector of the current image to be retrieved.

[0112] S8. Calculate the Euclidean distance between the final representation vector of the current image to be retrieved and the final representation vector of each sample image in the retrieval sample image library, and return the final retrieval results in ascending order of Euclidean distance.

[0113] Experimental Test

[0114] To verify the effectiveness of the method provided in the above embodiments, validation tests were conducted on the publicly available image retrieval datasets Oxford Landmarks Dataset (Oxford5K) and Paris Landmarks Dataset (Paris6K). The specific tests included the following steps:

[0115] Step 1: Input the images from the test dataset (Oxford5K) and the auxiliary dataset (Paris6K) into the depth model to extract their depth features; and calculate the contrast vector of each image according to formula (1).

[0116] Step 2: Obtain the potential key pattern channel index of the deep model by combining the two datasets. Based on the key pattern channel index, co-occurring key features can be generated. Then, generate the CFID descriptor of each image according to formulas (6)-(13).

[0117] Step 3: For the test sample image, perform the same steps to obtain its corresponding CFID descriptor; then, perform similarity matching with the test dataset images, and return the final retrieval results in ascending order of Euclidean distance. The results show that the method proposed in the patent achieves an average precision of 85.8% on the Oxford5K dataset (using 128-dimensional image vectors), far exceeding the best method of the same category, and an average precision of 92.4% on the Paris6K dataset (using 128-dimensional image vectors). The method provided in the above embodiments demonstrates significant advantages in both accuracy and time efficiency (low vector dimension).

[0118] Step 4: The effectiveness of the method was tested through qualitative comparative experiments. In the experiments, three query images were randomly selected from the Oxford5k dataset, and their corresponding raw depth features and CFID features were extracted respectively. Then, instance retrieval was performed using the raw depth features and CFID features respectively. The results are as follows: Figure 1 As shown.

[0119] The qualitative experimental results show that the image descriptors generated by the method provided in the above embodiments have very strong distinguishability and excellent retrieval results.

[0120] This invention creatively integrates deep semantic features, global multi-directional texture features, and geometric rectangular features of images by simulating human visual attention mechanisms, constructing a compact and powerful image representation vector. This method not only significantly improves the accuracy of image retrieval, but more importantly, effectively overcomes the inherent texture bias of deep learning models, enhancing the system's ability to recognize the essential shape of objects. This results in excellent robustness and generalization ability in complex scenes. Simultaneously, its efficient feature integration method ensures retrieval speed in large-scale applications, achieving a balance between accuracy and efficiency.

[0121] To facilitate understanding by those skilled in the art of the improvements of this invention over the prior art, some of the accompanying drawings and descriptions have been simplified. The above embodiments are preferred implementations of this invention. In addition, this invention can be implemented in other ways. Any obvious substitutions without departing from the concept of this technical solution are within the protection scope of this invention.

Claims

1. An image retrieval method based on visual attention mechanism and salient feature integration, characterized in that, The method comprises the following steps: S1, respectively extracting the depth features of each sample image in the target image set and the auxiliary image set, and calculating the corresponding Tamura contrast statistical quantity features; S2, calculating the average value of the contrast statistical quantity of each channel of the target image set and the auxiliary data set respectively, to obtain the mean contrast vector of the two data sets; S3, sorting the mean contrast vector in descending order to obtain the sorted vector and its corresponding channel index list; selecting the channels ranking in the front and commonly appearing in the two data sets to construct the co-occurring key feature; S4, extracting the global multi-directional information of the co-occurring key feature, and fusing it with the depth feature to obtain a multi-directional global feature, so as to enhance the directional representation ability of the feature; S5, using the geometric moment of the sample image to extract the depth moment feature, and fusing it into the depth feature, to further improve the ability of the feature to represent shape information, so as to alleviate the texture bias problem of the depth feature; S6, integrating the global feature vector of each sample image to construct the final representation vector of each sample image; S7, based on the final representation vector, calculating the similarity between the image to be searched and the sample image, and returning the final search result. 2.The image retrieval method based on visual attention mechanism and saliency feature integration according to claim 1, characterized in that, Step S1 specifically comprises: S1.1, input each image in each image set into a deep model, extract the deep semantic feature of each sample image, the deep feature of the image can be expressed as wherein and respectively represent the width and height of the feature map, the total number of feature maps is represented, and the deep feature sets of the target image set and the auxiliary image set are respectively represented as and ; S1.2, flattening each feature map of the depth feature into a one-dimensional vector form, calculating the Tamura contrast statistical quantity feature corresponding to the depth feature of each image, and the calculation formula is as follows: (1)； wherein, and respectively represent the standard deviation and variance of one-dimensional vector, represent the mean fourth moment, the parameter ; through the above formula, the contrast statistics set of all images in the above two data sets can be obtained, respectively represented as and . 3.The image retrieval method based on visual attention mechanism and saliency feature integration according to claim 2, characterized in that: In step S2, the mean contrast vectors of the two data sets are respectively and ; the corresponding representation is (2)； (3)； Wherein, n represents the total number of images in the target image set; m represents the total number of images in the auxiliary image set; The contrast statistical quantity feature vector of the mth image in the target image set is represented as The contrast statistical quantity feature vector of the mth image in the target image set is represented as The contrast statistical quantity feature vector of the mth image in the auxiliary image set is represented as The contrast statistical quantity feature vector of the mth image in the auxiliary image set is represented as 4. The image retrieval method based on visual attention mechanism and salient feature integration according to claim 3, characterized in that, Step S3 specifically comprises: First, the sorted channel index list is obtained by sorting and respectively; then, the channel index list is subdivided into multiple smaller subsets according to the percentage, so as to obtain the set and the set respectively; wherein, and , and . Then, by identifying the common channels appearing in both datasets, the index set of the key channels is obtained The calculation formula is as follows: (4)； wherein the parameter has a value range of , represents the number of feature maps; Finally, for each image, the key channels are aggregated to generate its co-occurring key features according to the obtained key channel index set ; its calculation formula is as follows: (5)； In the formula, is the feature map of the i-th channel in the original deep feature F.

5. The image retrieval method based on visual attention mechanism and salient feature integration according to claim 4, characterized in that, Step S4 specifically comprises: First, a variant of the two-dimensional Gabor function is used to extract the multi-directional detail information of the co-occurring key feature, and the formula definition of the variant is as follows: (6)； Wherein, ； Parameter pair denotes the center of the Gabor function receptive field, parameter and denote the standard deviation of the receptive field in the axis and axis direction, respectively, ratio denotes the spatial aspect ratio; Then, the generated Gabor filter is used to filter the co-occurring key feature to extract its multi-directional detail information, and the calculation formula is as follows: (7)； Finally, the above features are normalized and fused with the original depth features to obtain multi-directional global features The calculation formula is as follows: (8)； wherein, represents L2 normalization.

6. The image retrieval method based on visual attention mechanism and salient feature integration according to claim 5, characterized in that, Step S5 specifically comprises: First, for each feature map of multi-direction global features its two-dimensional order geometric moments are defined as follows: (9)； wherein ., ; Then, the obtained geometric moments are normalized, so as to obtain the depth moment features ; the calculation formula is as follows: (10)； wherein, is a balancing factor, taking a positive value, for ensuring stability of the calculation; the parameter is a normalizing factor, taking a value in the range , for adjusting the degree of normalization; Finally, the multi-direction global feature is fused with the depth matrix feature to construct a fused global feature vector ; the calculation formula is as follows: (11)； wherein, represents and pool the multi-directional global features; and: (12)。 7. The image retrieval method based on visual attention mechanism and salient feature integration according to claim 6, characterized in that: In step S6, the integrated global feature vector of each sample image is sequentially subjected to L2 normalization, PCA whitening, PCA and L2 normalization again to obtain the final representation vector of each sample image: (13)； wherein, denotes the L2 normalization function, denotes PCA whitening, denotes principal component analysis; wherein the parameters is a whitening parameter matrix, denotes the final selected feature dimension.

8. The image retrieval method based on visual attention mechanism and salient feature integration according to claim 7, characterized in that, Step S7 specifically comprises: First, the same processing as steps S3-S6 is performed on the current image to be searched to obtain the final representation vector of the current image to be searched; Then, the Euclidean distance between the final representation vector of the current image to be searched and the final representation vector of each sample image in the search sample image library is calculated, and the final search result is returned in order of the Euclidean distance from small to large.

Images