Image retrieval method based on depth edge information

Abstract

The invention discloses an image retrieval method based on depth edge information, which comprises the following steps: firstly, extracting depth features of an image from a deep neural network, and constructing a target information feature map through color information and spatial information of the image; secondly, extracting a horizontal edge graph and a vertical edge graph from the target information feature graph and the depth feature graph by using a Prewitt operator to construct an edge information weight and an edge information feature graph; then, weighting the edge information feature map by using an edge information weight, and obtaining a spatial aggregation feature vector through spatial aggregation; and calculating a channel difference weight value according to the edge information feature map and the spatial aggregation feature value, and weighting the spatial aggregation feature vector by using the channel difference weight to obtain a final feature vector. And principal component analysis and feature dimension compression are carried out on the final feature vector to obtain the final representation of the image. And performing similarity calculation by utilizing the final representation of the image during retrieval to obtain a retrieval result. According to the method, the image retrieval performance of the depth features can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image retrieval, and in particular to an image retrieval method based on depth edge information. Background Art

[0002] With the rapid development of information technology, image data has become an essential component of the Internet and various information systems, and image retrieval technology has gradually become a research hotspot. Traditional image retrieval techniques primarily rely on text-based descriptive information (such as keyword tags) for search. However, this approach suffers from incomplete information representation, strong subjectivity, and high annotation costs, making it difficult to meet the high accuracy and efficiency requirements of practical applications. Therefore, content-based image retrieval has become a major research direction. Initially, researchers extracted low-level features such as color, texture, and shape to construct image representations for image retrieval. Subsequently, deep learning methods were applied to image retrieval. Researchers used deep neural networks to extract deep features and construct image representations for image retrieval, achieving excellent performance. However, deep features extracted from deep neural networks still suffer from excessive background noise, resulting in poor discriminability of target content, and the neglect of low-level features. Therefore, extracting features with stronger target information from deep features and using them to construct more discriminative image representations remains a major challenge. Summary of the Invention

[0003] The present invention aims to solve the problem of how to effectively extract target information of depth features and construct a more distinguishable image representation based on this information, and provides an image retrieval method based on depth edge information.

[0004] To solve the above problems, the present invention is achieved through the following technical solutions:

[0005] An image retrieval method based on depth edge information comprises the following steps:

[0006] Step 1: Input the image into the deep convolutional neural network model to extract the deep features of the image;

[0007] Step 2: Use the SLIC superpixel segmentation method to segment the image into different regions;

[0008] Step 3: Calculate the target information of each region using the color mean and spatial coordinates of each region and the center point coordinates of the image, and construct the target information feature map of the image using the target information of all regions;

[0009] Step 4: Use the Prewitt operator to extract the horizontal edge map and vertical edge map of the target information feature map, and integrate the horizontal edge map and vertical edge map of the target information feature map to construct the edge information weight;

[0010] Step 5: For each depth feature map of the depth feature, use the Prewitt operator to extract the horizontal edge map and the vertical edge map of the depth feature map, and integrate the horizontal edge map and the vertical edge map of the depth feature map to construct an edge information feature map;

[0011] Step 6: For each depth feature map of the depth feature, the edge information feature map is weighted by the edge information weight to obtain a weighted edge information feature map, and then the weighted edge information feature map is aggregated in the spatial direction to obtain a spatial aggregation feature value;

[0012] Step 7: For each depth feature map of the depth feature, the channel difference weight value is calculated according to the gray level co-occurrence matrix and the spatial aggregation eigenvalue of the edge information feature map;

[0013] Step 8: For each depth feature map of the depth feature, use the channel difference weight value to weight the spatial aggregation feature value to obtain the final feature value;

[0014] Step 9: Use all the final eigenvalues ​​to construct the final eigenvector of the image, and perform principal component analysis and dimensionality compression on the final eigenvector of the image to generate the final representation of the image;

[0015] Step 10: Perform steps 1 to 9 on the retrieval image in the image database to obtain a final representation of the retrieval image, and perform steps 1 to 9 on the query image to obtain a final representation of the query image; then, calculate the similarity between the final representation of the query image and the final representation of each retrieval image in the image database, and use the retrieval image with the highest similarity as the retrieval result.

[0016] In step 3 above, area a i Target information i for:

[0017]

[0018] Where, Represents region a of image I i The color mean, Represents region a of image I j The color mean of Represents region a of image I i The spatial coordinates of Represents region a of image I j The spatial coordinates of Represents the coordinates of the center point of image I; W and H represent the width and height of image I respectively; i, j = 1, 2, ..., N, where N represents the number of regions in image I; ‖*‖ 2 represents the L2 distance.

[0019] In step 4 above, the edge information weight for:

[0020]

[0021] Where, and ef y They represent the horizontal edge map and vertical edge map of the target information feature map O respectively.

[0022] In the above step 5, the edge information feature map E k for:

[0023]

[0024] Where, and Represents the depth feature map x in the depth feature k The horizontal edge map and vertical edge map of , k = 1, 2, ..., K, K represents the number of depth feature maps in the depth feature.

[0025] In the above step 6, the weighted edge information feature map x′ k for:

[0026]

[0027] Where, E k Represents the deep feature map x k The edge information feature map, is the edge information weight of the target information feature map O, k = 1, 2, ..., K, K represents the number of depth feature maps in the depth feature, and ⊙ represents the Hadamard product.

[0028] In step 6 above, the spatial aggregation eigenvalue for:

[0029]

[0030] Where x′ k (i,j) represents the weighted edge information feature map x′ k Coordinate representation, W and H represent the weighted edge information feature map x′ K The width and height of , k = 1, 2, ..., K, K represents the number of depth feature maps in the depth feature.

[0031] The specific process of step 7 above is as follows:

[0032] First, calculate the energy value e k :

[0033]

[0034] Then, calculate the element value ratio c k :

[0035]

[0036] Finally, calculate the channel difference weight value

[0037]

[0038] In the above formula, p k (i,j) represents the edge information feature map E k The coordinate representation of the gray level co-occurrence matrix, M represents the gray level, represents the spatial aggregation feature value, and K represents the number of deep feature maps in the deep feature.

[0039] The specific process of step 8 above is as follows:

[0040] Final eigenvalue for:

[0041]

[0042] Where, represents the channel difference weight value, Represents the spatial aggregation feature value.

[0043] The basic idea of ​​this invention is to input an image into a deep neural network model and extract deep features. These deep features consist of a set of deep feature maps, each representing different target information. To further enhance target information, horizontal and vertical edge maps are extracted from the image and deep feature maps and effectively combined to improve the discriminability of the deep features.

[0044] Compared with the existing technology, the present invention proposes to use manual features to enhance the representation ability of depth features based on the visual application of manual features and depth features of images, uses low-level features to enhance the target information in the depth features, and filters the background noise in the depth features, obtaining a new image representation based on depth edge information, which can highlight the target objects in the image, improve the resolution of the depth feature map, and thus effectively improve the image retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flowchart of an image retrieval method based on depth edge information. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific examples and the accompanying drawings.

[0047] An image retrieval method based on deep edge information, such as Figure 1 As shown, the steps include:

[0048] Step 1: Input the image I into the deep neural network model to extract the deep features X of this image.

[0049] Single image I∈R W×H It is an RGB three-channel image, and its depth feature is That is, the depth feature X of each image I is composed of K depth feature maps x k W and H represent the image I and the depth feature map x respectively. k The width and height of the image I and the depth feature map x are k The space dimensions W×H remain consistent.

[0050] Step 2: Use the SLIC superpixel segmentation method to divide the image I into different regions

[0051] The calculation formula of the SLIC superpixel segmentation method is as follows:

[0052] A=superpixds(I,N)

[0053] Where superpixds(*) represents the SLIC segmentation method, I represents the image, and N represents the desired number of segmentations, which is set to 100 here. A represents the segmented image, where Each area a i They are all irregular shaped areas.

[0054] Step 3: Calculate the target information of each region using the color mean and spatial coordinates of each region and the center coordinates of the image. i , and the target information of all regions o i As the target information feature map of the image

[0055] Step 3.1. Calculate the color mean of each area Unify the color information of each area.

[0056] The color space of image I is represented as CO = {co s}, s∈(R,G,B), then region a i The color mean The calculation formula is as follows:

[0057]

[0058] In the formula, numel(*) represents area a i The number of pixels in co s (a i ) represents area a i The pixel value in color channel s.

[0059] Step 3.2: Calculate the spatial coordinates of each region Build spatial connections between areas.

[0060] Area a i The spatial coordinates of The calculation formula is as follows:

[0061]

[0062] In the formula, round(*) means rounding. and Respectively represent area a i The width and height of image I are represented by W and H respectively.

[0063] Step 3.3, calculate the center point coordinates of the image

[0064] The coordinates of the center point of image I The calculation formula is as follows:

[0065]

[0066] Where round(*) indicates rounding, W and H represent the width and height of image I, respectively.

[0067] Step 3.4: Calculate the target information ω of each region based on the color mean and spatial coordinates of each region and the center coordinates of the image. i .

[0068] Area a i Target information i The calculation formula is as follows:

[0069]

[0070] Where, Indicates area a i The color mean, Indicates area a j The color mean, ‖*‖ 2 Represents the L2 distance, and we use this method to express the color difference of the RGB three channels between regions. Indicates area a i with a jThe spatial distance between Indicates area a i The coordinates of the center point of the image W and H represent the width and height of image I respectively. Indicates area a i The spatial coordinates of Indicates area a j The spatial coordinates of Represents the coordinates of the center point of the image. i,j = 1, 2, ..., N, where N is the number of regions. ln(*) represents the logarithmic function with base e.

[0071] Step 3.5: Utilize target information of all regions i Constructing target information feature map

[0072] Step 4: Use the Prewitt operator to extract the horizontal edge map of the target information feature map O and vertical edge maps And integrate the horizontal edge map of the target information feature map and vertical edge maps Constructing edge information weights

[0073] Horizontal edge map of target information feature map O and vertical edge maps for:

[0074]

[0075] Where Prewitt(*) represents the Prewitt operator.

[0076] Edge information weight The calculation formula is as follows:

[0077]

[0078] Edge information weight Is a spatial matrix whose spatial size is the same as the depth feature map x k consistent.

[0079] Step 5: For each depth feature map x of the depth feature k , use the Prewitt operator to extract the horizontal edge map of the depth feature map and vertical edge maps Horizontal edge map integrating deep feature maps and vertical edge maps Construct edge information feature map E k .

[0080] In order to improve the compatibility of deep features and manual features and reduce the semantic gap between them, we use Prewitt to extract each deep feature map x k Horizontal edge map and vertical edge maps The calculation formula is as follows:

[0081]

[0082] Where Prewitt(*) represents the Prewitt operator.

[0083] Edge information feature map E k The calculation formula is as follows:

[0084]

[0085] Edge information feature map E k The spatial size and depth feature map x k consistent.

[0086] Step 6: For each depth feature map x of the depth feature k , using edge information weight Edge information feature map E k Weighted to obtain the weighted edge information feature map x′ k , and then the weighted edge information feature map x′ k Perform spatial direction aggregation to obtain spatial aggregation feature vector

[0087] Weighted edge information feature map x′ k The calculation formula is as follows:

[0088]

[0089] Where ⊙ represents the Hadamard product, which is the multiplication of corresponding elements. The weighted edge information feature map x′ k With the depth feature map x k The spatial scale is consistent.

[0090] Spatial aggregation eigenvector The calculation formula is as follows:

[0091]

[0092] Where x′ k (i,j) represents the weighted edge information feature map x′ k Coordinate representation, W and H represent the weighted edge information feature map x′ k width and height.

[0093] Step 7. For each depth feature map x of the depth feature k , according to the edge information feature map E k Gray-level co-occurrence matrix and spatial aggregation eigenvalue of Calculate channel difference weight value

[0094] Step 7.1. Calculate the energy value e k .

[0095] Edge information feature map E k The gray-level co-occurrence matrix p k The calculation formula is as follows:

[0096] p k =GLCM(E k )

[0097] Where GLCM(*) represents the gray-level co-occurrence matrix.

[0098] So the energy value e k The calculation formula is as follows:

[0099]

[0100] Where p(i, j) represents the coordinate representation of the gray-level co-occurrence matrix p of the edge information feature map; M represents the gray level, which is generally 8, indicating that the gray-level co-occurrence matrix is ​​an 8×8 matrix.

[0101] Step 7.2: Calculate the element value ratio c k .

[0102] Element value ratio c k The calculation formula is as follows:

[0103]

[0104] Where, represents the spatial aggregated feature value, and K represents the number of deep feature maps.

[0105] Step 7.3: Calculate channel difference weight value

[0106] Channel difference weight value The calculation formula is as follows:

[0107]

[0108] Wherein, ln(*) represents the logarithmic function with base e.

[0109] Step 8. For each depth feature map x of the depth feature k, using channel difference weight values Spatial aggregation eigenvalues Weighted to obtain the final eigenvalue

[0110] Final eigenvalue The calculation formula is as follows:

[0111]

[0112] Where, represents the channel difference weight value, Represents the spatial aggregation feature value.

[0113] Step 9: Utilize all final eigenvalues Construct the final feature vector of image I The final feature vector of the combined image I Perform principal component analysis and dimension compression to generate the final representation V of image I.

[0114] The final representation V of image I is calculated as follows:

[0115]

[0116] Where, Represents the final eigenvector of the image, pca(*) represents principal component analysis of the eigenvector, and whiten(*) represents whitening and dimensionality reduction processing of the analyzed data.

[0117] Step 10: Perform steps 1 to 9 on the retrieval image in the image database to obtain a final representation of the retrieval image, and perform steps 1 to 9 on the query image to obtain a final representation of the query image; then, calculate the similarity between the final representation of the query image and the final representation of each retrieval image in the image database, and use the retrieval image with the highest similarity as the retrieval result.

[0118] In the present invention, the similarity between the final representation of the query image and the final representation of the retrieval image can be calculated using the L2 distance.

[0119] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention. Therefore, the present invention is not limited to the above-mentioned specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.

Claims

1. An image retrieval method based on depth edge information, characterized in that: The steps are as follows: Step 1: Input the image into the deep convolutional neural network model to extract the deep features of the image; Step 2: Use the SLIC superpixel segmentation method to segment the image into different regions; Step 3: Calculate the target information of each region using the color mean and spatial coordinates of each region and the center point coordinates of the image, and construct the target information feature map of the image using the target information of all regions; Step 4: Use the Prewitt operator to extract the horizontal edge map and vertical edge map of the target information feature map, and integrate the horizontal edge map and vertical edge map of the target information feature map to construct the edge information weight; Step 5: For each depth feature map of the depth feature, use the Prewitt operator to extract the horizontal edge map and the vertical edge map of the depth feature map, and integrate the horizontal edge map and the vertical edge map of the depth feature map to construct an edge information feature map; Step 6: For each depth feature map of the depth feature, the edge information feature map is weighted by the edge information weight to obtain a weighted edge information feature map, and then the weighted edge information feature map is aggregated in the spatial direction to obtain a spatial aggregation feature value; Step 7: For each depth feature map of the depth feature, the channel difference weight value is calculated according to the gray level co-occurrence matrix and the spatial aggregation eigenvalue of the edge information feature map; Step 8: For each depth feature map of the depth feature, use the channel difference weight value to weight the spatial aggregation feature value to obtain the final feature value; Step 9: Use all the final eigenvalues ​​to construct the final eigenvector of the image, and perform principal component analysis and dimensionality compression on the final eigenvector of the image to generate the final representation of the image; Step 10: Perform steps 1 to 9 on the retrieval image in the image database to obtain a final representation of the retrieval image, and perform steps 1 to 9 on the query image to obtain a final representation of the query image; then, calculate the similarity between the final representation of the query image and the final representation of each retrieval image in the image database, and use the retrieval image with the highest similarity as the retrieval result.

2. The image retrieval method based on depth edge information according to claim 1, characterized in that: In step 3, area a i Target information i for: Where, Represents region a of image I i The color mean, Represents region a of image I j The color mean of Represents region a of image I i The spatial coordinates of Represents region a of image I j The spatial coordinates of Represents the coordinates of the center point of image I; W and H represent the width and height of image I respectively; i, j = 1, 2, ..., N, where N represents the number of regions in image I; ‖*‖ 2 represents the L2 distance.

3. The image retrieval method based on depth edge information according to claim 1, characterized in that: In step 4, the edge information weight for: Where, ef x and ef y They represent the horizontal edge map and vertical edge map of the target information feature map O respectively.

4. The image retrieval method based on depth edge information according to claim 1, characterized in that: In step 5, the edge information feature map E k for: Where, ef x,k and Represents the depth feature map x in the depth feature k The horizontal edge map and vertical edge map of , k = 1, 2, ..., K, K represents the number of depth feature maps in the depth feature.

5. The image retrieval method based on depth edge information according to claim 1, characterized in that: In step 6, the weighted edge information feature map x ′ k for: Where, E k Represents the deep feature map x k The edge information feature map, is the edge information weight of the target information feature map O, k = 1, 2, ..., K, K represents the number of depth feature maps in the depth feature, and ⊙ represents the Hadamard product.

6. The image retrieval method based on depth edge information according to claim 1, characterized in that: In step 6, spatial aggregation eigenvalues for: Where x ′ k (i,j) represents the weighted edge information feature map x ′ k Coordinate representation, W and H represent the weighted edge information feature map x ′ k The width and height of , k = 1, 2, ..., K, K represents the number of depth feature maps in the depth feature.

7. The image retrieval method based on depth edge information according to claim 1, characterized in that: The specific process of step 7 is as follows: First, calculate the energy value e k : Then, calculate the element value ratio c k : Finally, calculate the channel difference weight value In the above formula, p k (i,j) represents the edge information feature map m k The coordinate representation of the gray level co-occurrence matrix, M represents the gray level, represents the spatial aggregation feature value, and K represents the number of deep feature maps in the deep feature.

8. The image retrieval method based on depth edge information according to claim 1, characterized in that: The specific process of step 8 is as follows: Final eigenvalue for: Where, represents the channel difference weight value, Represents the spatial aggregation feature value.

Images