A footprint cross-modal retrieval method based on pressure information

Abstract

The present application relates to the field of footprint analysis, and more particularly to a footprint cross-modal retrieval method based on pressure information, comprising: collecting inked footprint images on site, and converting the inked footprint images into a footprint dataset according to the footprint pressure; performing color conversion according to the gray scale of each pixel point in the footprint dataset to obtain a color image; extracting global features and local features of the color image by using a deep convolutional neural network, and dynamically adjusting the global features and the local features; and fusing the adjusted global features and local features to realize multi-level feature retrieval. The present application combines global features and local features, and uses an attention mechanism for dynamic adjustment, so that when identifying and searching footprint images containing a large amount of detailed information, the real identity information of the footprint can be more accurately identified and searched.

Description

Technical Field

[0001] This invention relates to the field of footprint analysis, and more particularly to a method for cross-modal footprint retrieval based on pressure information. Background Technology

[0002] Footprints are the marks left by the feet of a human body during movement, formed by their contact with surfaces such as the ground. As a common biological characteristic, footprints are frequently found at crime scenes and can reflect human behavioral characteristics. Therefore, they play a special role in locating criminals and narrowing down the scope of investigations.

[0003] Because footprint images belong to a fine-grained domain with minimal differences in features between individuals, cross-modal footprint retrieval research is challenging. Cross-modal footprint images suffer from differences in features both within and between modalities: within a modality, the differences between categories are small, mainly because visual differences in footprints are not obvious and are easily masked by similar contours, textures, and other factors; while between different modalities, differences within the same category are mainly due to semantic gaps caused by the characteristics of different modalities. To address this issue, it is urgent to design a cross-modal footprint retrieval method based on pressure information. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides the following technical solution:

[0005] A method for cross-modal footprint retrieval based on stress information includes:

[0006] S1: Collect images of ink-printed footprints on site and convert them into a footprint dataset based on the footprint pressure.

[0007] S2: Perform color conversion on each pixel in the footprint dataset to obtain a color image;

[0008] S3: Use a deep convolutional neural network to extract global and local features of color images, and dynamically adjust the global and local features;

[0009] S4: The adjusted global and local features are fused to achieve multi-level feature retrieval.

[0010] As an improvement to the above technical solution, step S1 includes:

[0011] S11: Collect images of ink-printed footprints using ink-printed paper.

[0012] S12: Use a footprint pressure acquisition device to collect footprint pressure information of ink-printed footprint images to obtain pressure-colored footprint data of a row, with color reflecting the pressure distribution.

[0013] S13: The target recognition algorithm of YOLOV5 is used to identify targets in the ink imprint footprint images and pressure color footprint data, and then segment them to obtain a dataset of individual footprint images.

[0014] As an improvement to the above technical solution, after step S13 is performed, the following steps also need to be performed:

[0015] S131: The dataset of footprint images is filtered using median filtering.

[0016] S132: Perform data augmentation on the dataset of filtered footprint images.

[0017] As an improvement to the above technical solution, the following steps need to be performed before step S2:

[0018] S14: The dataset of a single footprint image is fed into a deep convolutional neural network through the foreground segmentation module to estimate the accurate foreground object mask of the input image.

[0019] S15: Extract the image of the actual single ink imprint by filtering scene pixels and setting a threshold.

[0020] As an improvement to the above technical solution, step S2 includes the following steps:

[0021] S21: Calculate the score of each pixel in a single ink imprint footprint image using the modal conversion module, and set a threshold.

[0022] S22: Divide the grayscale image of a single ink imprint footprint into three component images: R, G, and B. Calculate the grayscale intensity of each component image to display a specific color, thereby converting the ink imprint footprint image into a color image.

[0023] As an improvement to the above technical solution, step S3 includes the following steps:

[0024] S31: Use at least two deep convolutional neural networks to extract global and local features of the color image respectively.

[0025] S32: Use a max pooling layer to obtain a discriminative global core feature f1 from the global features.

[0026] S33: Use average pooling to obtain the global average feature f2.

[0027] S34: Convolve f1 and f2 and then subtract them to obtain the global feature F, where F represents the difference between the average feature and the core feature.

[0028] As an improvement to the above technical solution, the deep convolutional neural network includes a CBAM module:

[0029] The CBAM module includes a channel attention module and a spatial attention module. The channel attention module obtains the importance weight of each channel and then applies it to each channel of the local features to enhance the feature representation capability of the channel dimension. The spatial attention module obtains the importance weight of each spatial location and then applies it to each spatial location of the local features to enhance the feature representation capability of the spatial dimension.

[0030] As an improvement to the above technical solution, the deep convolutional neural network adopts a progressive training method, specifically including the following steps:

[0031] S311: Iterate through each layer of the deep convolutional neural network and use CE loss and Hard Triplet Loss to constrain the training, gradually increasing the depth and complexity of the network until the entire network is trained.

[0032] As an improvement to the above technical solution, the following steps need to be performed before step S4 is executed;

[0033] S35: Further process the features of each layer at different depths output by the deep convolutional neural network through convolution and pooling to obtain local features at different levels. Use the local features at different levels to distinguish the identity ID and calculate the score.

[0034] As an improvement to the above technical solution, before step S1 is executed, the following steps also need to be performed:

[0035] S101: Use the Flask framework to build an HTTP service and establish a cross-modal footprint retrieval server.

[0036] The server is used to interact with the client for data.

[0037] The beneficial effects of this invention are:

[0038] By using modal conversion, an image in one modality is given RGB information in another modality, which adds rich RGB three-channel information while preserving the original rich texture information.

[0039] By combining global and local features and using an attention mechanism for dynamic adjustment, multi-level and multi-scale features are obtained for discrimination. For footprint images rich in fine-grained information, more attention is paid to the extraction and matching of details within and in local areas of the footprint image. When recognizing and searching footprint images containing a large amount of detailed information, the system can more accurately identify and retrieve the true identity information of the footprints. Attached Figure Description

[0040] Figure 1 This is a flowchart of the present invention;

[0041] Figure 2 This is a flowchart of the HTTP service process of the present invention; Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0043] Because footprint images belong to a fine-grained domain with minimal differences in features between individuals, cross-modal footprint retrieval research is challenging. Cross-modal footprint images suffer from differences in features both within and between modalities: within a modality, the differences between categories are small, mainly because visual differences in footprints are not obvious and are easily masked by similar contours, textures, and other factors; while between different modalities, differences within the same category are mainly due to semantic gaps caused by the characteristics of different modalities. To address this issue, it is urgent to design a cross-modal footprint retrieval method based on pressure information.

[0044] To address this issue, a cross-modal footprint retrieval method based on stress information is provided, including:

[0045] S1: Collect images of ink-printed footprints on site and convert them into a footprint dataset based on the footprint pressure.

[0046] Among them, a complete set of footprints refers to the footprints formed after completing one walking cycle.

[0047] Specifically, step S1 includes:

[0048] S11: Collect images of ink-printed footprints using ink-printed paper.

[0049] The footprint images here are images of footprints on ink-printed paper, which are collected by ink-printing paper. Because the images are ink-printed, the image quality may not be high, and it is difficult for ordinary people to directly obtain information from ink-printed images.

[0050] S12: Use a footprint pressure acquisition device to collect footprint pressure information of ink-printed footprint images to obtain pressure-colored footprint data of a row, with color reflecting the pressure distribution.

[0051] Footprint pressure information can reflect and distinguish information such as gait, weight and walking habits. Therefore, in addition to obtaining image information, it is also necessary to obtain the distribution information of footprint pressure.

[0052] S13: The target recognition algorithm of YOLOV5 is used to identify targets in the ink imprint footprint images and pressure color footprint data, and then segment them to obtain a dataset of individual footprint images.

[0053] This step is to identify and separate each footprint to facilitate subsequent processing of this footprint data. To further improve image quality, the following steps are required after step S13:

[0054] S131: The dataset of footprint images is filtered using median filtering.

[0055] Median filtering is used to remove noise from an image and enhance its smoothness.

[0056] S132: Perform data augmentation on the dataset of filtered footprint images.

[0057] Data augmentation can improve the model's generalization ability to process footprint image datasets and reduce overfitting.

[0058] Furthermore, since other factors may influence the image of the ink-printed paper, such as shadows in the ink print footprint, it is necessary to obtain an image of the actual single ink print footprint to reduce these effects. Specifically, the following steps need to be performed before step S2:

[0059] S14: The dataset of a single footprint image is fed into a deep convolutional neural network through the foreground segmentation module to estimate the accurate foreground object mask of the input image.

[0060] S15: Extract the image of the actual single ink imprint by filtering scene pixels and setting a threshold.

[0061] In step S14, the grayscale histogram of the dataset of a single footprint image is calculated to represent the number of pixels with a certain grayscale level in the image, reflecting the frequency of each grayscale value in the image. According to specific experimental results, the maximum frequency of grayscale values ​​in the grayscale histogram plus 20 is a suitable threshold. This threshold is used as the accurate foreground object mask for prediction. Then, based on this threshold, the part of the scene pixels that is greater than the threshold is filtered out and separated to complete the segmentation and obtain the actual image of a single ink imprint footprint.

[0062] Steps S14 and S15 are used to obtain more accurate images of individual ink-printed footprints. Since these images are printed with ink, although they have undergone image processing, they are not intuitive enough for footprint recognition. In order to enhance the visual intuitiveness, step S2 is executed.

[0063] S2: Perform color conversion on each pixel in the footprint dataset to obtain a color image.

[0064] The footprint dataset, which was originally presented in grayscale, is converted into color images. Color images allow for a more intuitive understanding of the information contained within them, without requiring the observer's experience to make judgments. Observers can intuitively perceive the information from the images without needing professional training.

[0065] Specifically, step S2 includes the following steps:

[0066] S21: Calculate the score of each pixel in a single ink imprint footprint image using the modal conversion module, and set a threshold.

[0067] Because footprint images may contain other areas besides footprints that are not properly segmented, such as messy ink or dirt spots, and these ink or dirt spots may also have the same gray level, which can lead to incorrect labeling, it is necessary to select the optimal gray level range to reduce such areas.

[0068] The threshold here is calculated based on the maximum value of the gray-level histogram of the image. This threshold value is not only used for screening, but also directly participates as a parameter in the calculation of the gray-level intensity of the three component images R, G, and B in step S22.

[0069] S22: Divide the grayscale image of a single ink imprint footprint into three component images: R, G, and B. Calculate the grayscale intensity of each component image to display a specific color, thereby converting the ink imprint footprint image into a color image.

[0070] S3: Use a deep convolutional neural network to extract global and local features of color images, and dynamically adjust the global and local features.

[0071] Specifically, step S3 includes the following steps:

[0072] S31: Use at least two deep convolutional neural networks to extract global and local features of the color image respectively.

[0073] Each deep convolutional neural network needs to share a portion of its parameters, with the specific parameters depending on the global and local features in the actual computation process.

[0074] S32: Use a max pooling layer to obtain a discriminative global core feature f1 from the global features;

[0075] S33: Use average pooling to obtain the global average feature f2;

[0076] S34: Convolve f1 and f2 and then subtract them to obtain the global feature F, where F represents the difference between the average feature and the core feature. The difference feature obtained here has a certain degree of discriminative power.

[0077] The deep convolutional neural network includes a CBAM module:

[0078] The CBAM module includes a channel attention module and a spatial attention module;

[0079] The channel attention module obtains the importance weight of each channel and then applies it to each channel of the local features to enhance the feature representation capability of the channel dimension.

[0080] The spatial attention module obtains the importance weight of each spatial location and then applies it to each spatial location on the local features to enhance the feature representation capability of the spatial dimension.

[0081] The CBAM module, or Convolutional Attention Mechanism module, can adaptively select important information in feature maps and dynamically weight and adjust the channels and spatial dimensions in feature maps, thereby improving the model's expressiveness and generalization ability.

[0082] By combining global and local features and using an attention mechanism for weighting, the network can focus more on important features and identify footprints more accurately.

[0083] In addition, to improve the training efficiency of the network, the deep convolutional neural network adopts a progressive training method, which specifically includes the following steps:

[0084] S311: Iterate through each layer of the deep convolutional neural network and use CE loss and Hard Triplet Loss to constrain the training, gradually increasing the depth and complexity of the network until the entire network is trained.

[0085] Specifically, the network can be iterated over at each layer, training only a small portion of the network in each iteration, and then the network's depth and complexity can be gradually increased until the entire network is trained. This allows the network to gradually adapt to complex features, avoiding problems such as vanishing gradients and overfitting, while also improving training speed and stability.

[0086] The training is constrained using CEloss and Hard Triplet Loss. For each step of progressive training, the output features are passed through a linear layer to obtain ID scores, which are compared with the actual real ID information to calculate cross-entropy. Hard triplet loss is also used to constrain the training. Unlike traditional triplet loss, hard triplet loss strengthens the training by selecting negative samples of moderate difficulty, thereby improving the robustness and generalization ability of the model.

[0087] Among them, the implementation of methods for constraining training using CEloss and Hard Triplet Loss relies on the literature [Hermans A, Beyer L, Leibe B. In defense of the triplet loss for person re-identification[J].arXiv preprint arXiv:1703.07737,2017].

[0088] In order to obtain more discriminative local features at different levels, the following steps need to be performed before step S4 is executed;

[0089] S35: Further process the features of each layer at different depths output by the deep convolutional neural network through convolution and pooling to obtain local features at different levels. Use the local features at different levels to distinguish the identity ID and calculate the score.

[0090] Here, a score is calculated for rank ranking. The probability that the top n images (with the highest confidence) in the search results contain the correct result is calculated. Specifically, in this task, when we input a footprint image from the query set, we search for the most similar footprint image in the gallery set. For each feature of an image in the query set, we calculate its similarity to all images in the gallery set. Rank-k represents the accuracy rate at which the top k images, sorted by similarity, contain images with the same ID as the query image.

[0091] S4: The adjusted global and local features are fused to achieve multi-level feature retrieval.

[0092] The Concat feature fusion method is used to fuse global and local features by directly concatenating two features and stitching together multiple feature maps to obtain a richer feature representation, thereby achieving multi-level feature retrieval. For example, if the dimensions of two input features x and y are p and q, respectively, and the dimension of the output feature z is p+q, then stitching together multiple feature maps in this way can yield a richer feature representation.

[0093] In addition, to provide a more efficient data processing method, the following steps need to be performed before step S1:

[0094] S101: Use the Flask framework to build an HTTP service and establish a cross-modal footprint retrieval server.

[0095] The server is used to interact with the client for data.

[0096] The client encodes the ink imprint footprint image into byte information and sends this byte information to a specified HTTP URL and port number. The server then retrieves the byte information from the specified HTTP URL and port number and sends the image to a modality conversion module for processing to obtain the footprint detection result. Specifically, the modality conversion module converts the uploaded footprint image into the same modality as the training set. A deep convolutional neural network is then used to extract useful feature information and perform footprint detection. The detection results are packaged into a corresponding data format and returned to the client. The detection results may include the footprint's specific ID information or pseudo-color information, allowing the client to perform further processing and analysis.

[0097] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it.

Claims

1. A method for cross-modal footprint retrieval based on pressure information, characterized in that, include: S1: Collect images of ink-printed footprints on site and convert them into a footprint dataset based on the footprint pressure. S2: Perform color conversion on each pixel in the footprint dataset to obtain a color image; S3: Use a deep convolutional neural network to extract global and local features of color images, and dynamically adjust the global and local features; S4: Merge the adjusted global and local features to achieve multi-level feature retrieval; Step S3 includes the following steps: S31: Use at least two deep convolutional neural networks to extract global and local features of the color image respectively; S32: Use a max pooling layer to obtain a discriminative global core feature f1 from the global features; S33: Use average pooling to obtain the global average feature f2; S34: Convolve f1 and f2 and then subtract them to obtain the global feature F, where F represents the difference between the average feature and the core feature; The deep convolutional neural network includes a CBAM module: The CBAM module includes a channel attention module and a spatial attention module; The channel attention module obtains the importance weight of each channel and then applies it to each channel of the local features to enhance the feature representation capability of the channel dimension. The spatial attention module obtains the importance weight of each spatial location and then applies it to each spatial location on the local features to enhance the feature representation capability of the spatial dimension. The deep convolutional neural network employs a progressive training method, specifically including the following steps: S311: Iterate through each layer of the deep convolutional neural network and use CE loss and Hard Triplet Loss to constrain the training, gradually increasing the depth and complexity of the network until the entire network is trained.

2. The method for cross-modal footprint retrieval based on pressure information according to claim 1, characterized in that: Step S1 includes: S11: Collect images of ink-printed footprints using ink-printing paper; S12: Use a footprint pressure acquisition device to collect footprint pressure information of ink-printed footprint images to obtain pressure-colored footprint data of a single footprint, with color reflecting the pressure distribution. S13: The target recognition algorithm of YOLOV5 is used to identify targets in the ink imprint footprint images and pressure color footprint data, and then segment them to obtain a dataset of individual footprint images.

3. The method for cross-modal footprint retrieval based on pressure information according to claim 2, characterized in that: After step S13 is executed, the following steps also need to be performed: S131: The dataset of footprint images is filtered using median filtering; S132: Perform data augmentation on the dataset of filtered footprint images.

4. The method for cross-modal footprint retrieval based on pressure information according to claim 1, characterized in that: Before step S2 is executed, the following steps also need to be performed: S14: The dataset of a single footprint image is fed into a deep convolutional neural network through the foreground segmentation module to estimate the accurate foreground object mask of the input image; S15: Extract the image of the actual single ink imprint by filtering scene pixels and setting a threshold.

5. The method for cross-modal footprint retrieval based on pressure information according to claim 4, characterized in that: Step S2 includes the following steps: S21: Calculate the score of each pixel in a single ink imprint footprint image using the modal conversion module, and set a threshold; S22: Divide the grayscale image of a single ink imprint footprint into three component images: R, G, and B. Calculate the grayscale intensity of each component image to display a specific color, thereby converting the ink imprint footprint image into a color image.

6. The method for cross-modal footprint retrieval based on pressure information according to claim 1, characterized in that: Before step S4 is executed, the following steps also need to be performed; S35: Further process the features of each layer at different depths output by the deep convolutional neural network through convolution and pooling to obtain local features at different levels. Use the local features at different levels to distinguish the identity ID and calculate the score.

7. A method for cross-modal footprint retrieval based on pressure information according to any one of claims 1-6, characterized in that: Before step S1 is executed, the following steps also need to be performed: S101: Use the Flask framework to build an HTTP service and establish a cross-modal footprint retrieval server; The server is used to interact with the client for data.

Images