A cross-modal image retrieval method based on feature aggregation

Highly characterized features of dust footprints and grayscale footprints are extracted through feature aggregation network and attention module, and cross-modal differences are solved, efficient and intelligent retrieval is achieved, and the accuracy and efficiency of the comparison are improved.

CN117891964BActive Publication Date: 2024-11-22ANHUI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410059094.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-16
Publication Date
2024-11-22
Estimated Expiration
2044-01-16

AI Technical Summary

Technical Problem

There is a large modal difference between dust footprints and grayscale footprints, resulting in low accuracy of cross-modal image retrieval and low manual comparison efficiency and accuracy, making it difficult to effectively extract high characterization features.

Method used

Using a cross-modal image retrieval method based on feature aggregation, the high characterization features of dust footprints and grayscale footprints are extracted through multi-stage feature aggregation network, mixed attention modules, partial attention modules and other technologies, and the similarity is calculated using Manhattan distance. , realize intelligent cross-modal retrieval.

Benefits of technology

The cross-modal retrieval accuracy of dust footprints and grayscale footprints is improved, labor costs are reduced, efficient feature extraction and comparison are achieved, and the comparison efficiency and accuracy are significantly improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117891964B_ABST
    Figure CN117891964B_ABST
Patent Text Reader

Abstract

The present invention discloses a cross-modal image retrieval method based on feature aggregation, comprising the following steps: processing the collected footprint image with a CPU processor; sending the footprint data set into a multi-stage feature aggregation network to optimize and load the grayscale footprint image in the retrieval library; obtaining the dust footprint image to be queried; calculating the similarity between the dust footprint image to be queried and the grayscale footprint image in the retrieval library; outputting the personnel information of the grayscale footprint image in the retrieval library that is most similar to the dust footprint image to be queried. The present invention relates to the field of image processing, and the cross-modal image retrieval method based on feature aggregation effectively reduces the modal difference between dust footprints and grayscale footprints, and improves the accuracy of cross-modal footprint image retrieval.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a cross-modal image retrieval method based on feature aggregation. Background Art

[0002] At present, a grayscale footprint database has been established for criminals. The dust footprints left at the scene are compared with the footprint database. The results of this method depend largely on expert experience. In addition, manual comparison requires a lot of human resources and time costs. Manual comparison is not efficient and the actual effect is not ideal. Therefore, in order to improve the comparison efficiency and accuracy, a cross-modal image retrieval method for dust footprints and grayscale footprints is needed.

[0003] However, there are not only large modal differences between the dust footprint and grayscale footprint images, but also similarities between footprint images of different objects and the same modality. In addition, the cluttered background of dust footprints is not conducive to the extraction of high-representation features, which brings challenges to the cross-modal retrieval of footprint images. Therefore, extracting high-representation features of the same object and different modalities is of great help to footprint retrieval. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the shortcomings of the prior art, the present invention provides a cross-modal image retrieval method based on feature aggregation, which effectively reduces the modal difference between dust footprints and grayscale footprints and improves the accuracy of cross-modal footprint image retrieval.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a cross-modal image retrieval method based on feature aggregation, comprising the following steps:

[0008] S1: Processing the collected footprint image with a CPU processor;

[0009] S2: Feed the footprint dataset into a multi-stage feature aggregation network for optimization and load the grayscale footprint images in the retrieval library;

[0010] S3: Obtain the dust footprint image to be queried;

[0011] S4: Calculate the similarity between the dust footprint image to be queried and the grayscale footprint image in the retrieval library;

[0012] S5: outputting the grayscale footprint image person information that is most similar to the dust footprint image to be queried in the retrieval database;

[0013] Wherein, step S2 comprises the following steps:

[0014] The first step is to preprocess the collected footprint images: barefoot dust footprints and grayscale footprint images are collected by photographing the footprints of the scene environment and optical sensors; image segmentation is used to extract high-characteristic features for dust footprints, and barefoot footprints are segmented from the background to obtain a dust footprint dataset; the footprint images are resized to 512×512, and data augmentation is performed on the grayscale footprint image dataset;

[0015] The second step is network model training: the footprint images of seven out of the nine people are used as training samples to input into the network model for training, and the footprint images of the remaining two out of the nine are used as test samples, and a retrieval library is established based on the grayscale footprint images;

[0016] The specific training steps for training the network model in the second step are as follows:

[0017] (1) Constructing a hybrid attention module: ResNet50 is used as the backbone network. The input of the hybrid attention module is the low-level feature map before each layer of the backbone network and the high-level feature map after each layer. The low-level feature map and the high-level feature map are first sent to two 1×1 convolutional layers respectively. Then, the outputs of the two convolutional layers are used to calculate the channel similarity through matrix multiplication and softmax function. The low-level feature map is then sent to a 1×1 convolutional layer and the channel similarity matrix is ​​used to enhance the channel feature representation through matrix multiplication. Finally, a 1×1 convolutional layer is used to convert the feature to the size of the original high-level feature map and the output is obtained by adding it to the original high-level feature map. Similar operations with the low-level feature map can enhance the spatial feature representation.

[0018] (2) Constructing feature aggregation module: The feature aggregation module is constructed by fusing hybrid attention modules between each layer of the backbone network. Layer 4 is not used. One and two hybrid attention modules are fused after Layer 1 and Layer 2 respectively. The first two modules input the high-level feature maps and low-level feature maps before and after each layer. The low-level feature map input to the final fused hybrid attention module is the original feature before Layer 1, and the high-level feature map input is the output of the previous hybrid attention module.

[0019] (3) Constructing a partial attention module: A partial attention module is added after the backbone network to focus on fine-grained partial features. The features after Layer 3 are divided into three non-overlapping parts through an adaptive average pooling function. Each part is sent to three 1×1 convolutional layers. The outputs of the first two convolutional layers are matrix multiplied and then passed through a softmax activation function. The matrix multiplication is then performed with the output of the third convolutional layer to obtain a fine-grained partial feature. The weighted sum is then performed with a weight matrix normalized by a softmax activation function to obtain an attention-enhanced partial feature. The input feature is passed through a global average pooling layer and a batch normalization layer to obtain a feature vector. The features of the two parts are then added together to obtain the output feature.

[0020] As training, in step S3, the Manhattan distance between the dust footprint image to be queried and the grayscale footprint image in the retrieval library is calculated using the trained network model and the metric function, and the distance is used to measure their similarity.

[0021] As training, when collecting footprint images in the first step, each object contains 42 barefoot dust footprint images and 6 barefoot grayscale footprint images.

[0022] As training, the grayscale footprint data is augmented in the first step by horizontal flipping, 10° clockwise rotation, and 10° counterclockwise rotation.

[0023] (III) Beneficial effects

[0024] The present invention provides a cross-modal image retrieval method based on feature aggregation, which has the following beneficial effects:

[0025] The present invention uses barefoot dust footprint images and barefoot grayscale footprint images to extract high-characteristic features, and realizes intelligent cross-modal footprint retrieval through deep learning. This method can extract features and calculate similarities at low cost, solving problems such as cross-modal differences in footprint images. Compared with manual retrieval, this method can improve the comparison efficiency and accuracy to a certain extent, and effectively realize cross-modal image retrieval of dust footprints and grayscale footprints. With the help of artificial intelligence technology, this invention has positive significance for dust footprint comparison and identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the image retrieval method of the present invention;

[0027] Figure 2 A framework diagram of network optimization and cross-modal retrieval of the present invention;

[0028] Figure 3 It is a framework diagram of the feature aggregation module of the present invention;

[0029] Figure 4This is a framework diagram of the hybrid attention module of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] See also Figure 1-4 , the present invention provides a technical solution:

[0032] like Figure 2 As shown in the figure, network optimization and cross-modal retrieval are completed in the following steps:

[0033] The first step is to preprocess the collected footprint images: barefoot dust footprints and grayscale footprint images are collected by photographing the footprints of the scene environment and optical sensors. Each object contains 42 barefoot dust footprint images and 6 barefoot grayscale footprint images. Image segmentation is used to extract high-representation features from dust footprints, and barefoot footprints are separated from the background to obtain a dust footprint dataset. The footprint images are resized to 512×512, and the grayscale footprint image dataset is augmented. The augmentation methods include horizontal flipping, clockwise rotation of 10°, and counterclockwise rotation of 10°.

[0034] The second step is network model training: the footprint images of seven out of the nine people are used as training samples to input into the network model for training, and the footprint images of the remaining two out of the nine are used as test samples. A retrieval library is established based on the grayscale footprint images. The dust footprint images to be queried are input into the network, and the network will extract the query features and compare them with the features in the retrieval library for similarity. The similarity is determined by calculating the Manhattan distance. The smaller the distance, the more similar it is. Finally, the prediction accuracy is output.

[0035] like Figure 4 As shown, the specific training steps for the network model are as follows:

[0036] (1) Constructing a hybrid attention module: ResNet50 is used as the backbone network. The input of the hybrid attention module is the low-level feature map before each layer of the backbone network and the high-level feature map after each layer. The low-level feature map and the high-level feature map are first sent to two 1×1 convolutional layers respectively. Then, the outputs of the two convolutional layers are used to calculate the channel similarity through matrix multiplication and softmax function. The low-level feature map is then sent to a 1×1 convolutional layer and the channel similarity matrix is ​​used to enhance the channel feature representation through matrix multiplication. Finally, a 1×1 convolutional layer is used to convert the feature to the size of the original high-level feature map and the output is obtained by adding it to the original high-level feature map. Similar operations with the low-level feature map can enhance the spatial feature representation.

[0037] (2) Construct feature aggregation module: Figure 3 As shown in the figure, adding a hybrid attention module between each layer of the backbone network constitutes a feature aggregation module. Layer4 is not used. A hybrid attention module is added after Layer1, which uses the high-level feature maps and low-level feature maps before and after Layer1. Two hybrid attention modules are added after Layer2. The first one uses the features before and after Layer2. The low-level feature map input to the last hybrid attention module is the original feature before Layer1, and the high-level feature map input is the output of the previous hybrid attention module.

[0038] (3) Constructing a partial attention module: A partial attention module is added after the backbone network to focus on fine-grained partial features. The features after Layer 3 are divided into three non-overlapping parts through an adaptive average pooling function. Each part is sent to three 1×1 convolutional layers. The outputs of the first two convolutional layers are matrix multiplied and then passed through a softmax activation function. The matrix multiplication is then performed with the output of the third convolutional layer to obtain a fine-grained partial feature. The weighted sum is then performed with a weight matrix normalized by a softmax activation function to obtain an attention-enhanced partial feature. The input feature is passed through a global average pooling layer and a batch normalization layer to obtain a feature vector. The features of the two parts are then added together to obtain the output feature.

[0039] The third step is cross-modal footprint retrieval: When we need to find an object in the footprint retrieval library that matches the footprint image of the unknown identity to be queried, we first provide the footprint image to be queried to the trained network model. Then, the Manhattan distance between the image and each image in the retrieval library is calculated using the metric function of the network model. This distance is used as a basis for measuring the similarity between the image to be tested and the image in the retrieval library. Finally, the information of the person with the closest distance is returned to the user as the output result. It can help find possible matching objects in the scenario of footprint image comparison.

[0040] Based on the above steps, Figure 1 As shown in FIG. 1 , a cross-modal image retrieval method based on feature aggregation is shown in FIG. 1 . The specific operation steps are as follows:

[0041] S1: Processing the collected footprint image with a CPU processor;

[0042] S2: Feed the footprint dataset into a multi-stage feature aggregation network for optimization and load the grayscale footprint images in the retrieval library;

[0043] S3: Obtain the dust footprint image to be queried;

[0044] S4: Calculate the similarity between the dust footprint image to be queried and the grayscale footprint image in the retrieval library;

[0045] S5: Output the person information of the grayscale footprint image that is most similar to the dust footprint image to be queried in the retrieval library.

[0046] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A cross-modal image retrieval method based on feature aggregation, characterized by: The following steps are involved: S1: Processing the collected footprint image with a CPU processor; S2: Feed the footprint dataset into a multi-stage feature aggregation network for optimization and load the grayscale footprint images in the retrieval library; S3: Obtain the dust footprint image to be queried; S4: Calculate the similarity between the dust footprint image to be queried and the grayscale footprint image in the retrieval library; S5: outputting the grayscale footprint image person information that is most similar to the dust footprint image to be queried in the retrieval database; Wherein, step S2 comprises the following steps: The first step is to preprocess the collected footprint images: barefoot dust footprints and grayscale footprint images are collected by photographing the footprints of the scene environment and optical sensors; image segmentation is used to extract high-characteristic features for dust footprints, and barefoot footprints are segmented from the background to obtain a dust footprint dataset; the footprint images are resized to 512×512, and data augmentation is performed on the grayscale footprint image dataset; The second step is network model training: the footprint images of seven out of the nine people are used as training samples to input into the network model for training, and the footprint images of the remaining two out of the nine are used as test samples, and a retrieval library is established based on the grayscale footprint images; The specific training steps for training the network model in the second step are as follows: (1) Constructing a hybrid attention module: ResNet50 is used as the backbone network. The input of the hybrid attention module is the low-level feature map before each layer of the backbone network and the high-level feature map after each layer. The low-level feature map and the high-level feature map are first sent to two 1×1 convolutional layers respectively. Then, the outputs of the two convolutional layers are used to calculate the channel similarity through matrix multiplication and softmax function. The low-level feature map is then sent to a 1×1 convolutional layer and the channel similarity matrix is ​​used to enhance the channel feature representation through matrix multiplication. Finally, a 1×1 convolutional layer is used to convert the feature to the size of the original high-level feature map and the output is obtained by adding it to the original high-level feature map. Similar operations with the low-level feature map can enhance the spatial feature representation. (2) Constructing feature aggregation module: The feature aggregation module is constructed by fusing hybrid attention modules between each layer of the backbone network. Layer 4 is not used. One and two hybrid attention modules are fused after Layer 1 and Layer 2 respectively. The first two modules input the high-level feature maps and low-level feature maps before and after each layer. The low-level feature map input to the final fused hybrid attention module is the original feature before Layer 1, and the high-level feature map input is the output of the previous hybrid attention module. (3) Constructing a partial attention module: A partial attention module is added after the backbone network to focus on fine-grained partial features. The features after Layer 3 are divided into three non-overlapping parts through an adaptive average pooling function. Each part is sent to three 1×1 convolutional layers. The outputs of the first two convolutional layers are matrix multiplied and then passed through a softmax activation function. The matrix multiplication is then performed with the output of the third convolutional layer to obtain a fine-grained partial feature. The weighted sum is then performed with a weight matrix normalized by a softmax activation function to obtain an attention-enhanced partial feature. The input feature is passed through a global average pooling layer and a batch normalization layer to obtain a feature vector. The features of the two parts are then added together to obtain the output feature.

2. The cross-modal image retrieval method based on feature aggregation according to claim 1, characterized in that: In step S3, the cross-modal retrieval function is to provide the footprint image to be queried to the trained network model, use the trained network model and metric function to calculate the Manhattan distance between the dust footprint image to be queried and the grayscale footprint image in the retrieval library, and use the distance to measure their similarity.

3. The cross-modal image retrieval method based on feature aggregation according to claim 1, characterized in that: When collecting footprint images in the first step, each object includes 42 barefoot dust footprint images and 6 barefoot grayscale footprint images.

4. The cross-modal image retrieval method based on feature aggregation according to claim 1, characterized in that: In the first step, the grayscale footprint data is augmented by horizontal flipping, 10° clockwise rotation, and 10° counterclockwise rotation.