A method for cross-domain retrieval of on-site footprints

Abstract

This invention relates to the field of footprint recognition, and more particularly to a cross-domain retrieval method for on-site footprints. The method includes: obtaining a foreground footprint mask from a footprint image dataset using a foreground extraction module; obtaining footprint images based on the foreground footprint mask and the footprint image dataset, and feeding the footprint images into a momentum dual-branch network to obtain feature descriptors, thereby enhancing the discriminative power of the feature descriptors by increasing the inter-class distance and decreasing the intra-class distance in the feature space; calculating Manhattan distance scores between the feature descriptors and feature information in the feature database, and sorting them according to the distance scores to determine the returned retrieval personnel information. To obtain more discriminative features, this invention constructs a momentum dual-branch network for feature extraction, maintains a momentum-updated feature descriptor queue, and uses KL divergence as the loss to achieve the goal of increasing the inter-class distance and decreasing the intra-class distance in the feature space.

Description

Technical Field

[0001] This invention relates to the field of footprint recognition, and more particularly to a cross-domain retrieval method for on-site footprints. Background Technology

[0002] Currently, there are no publicly available indoor crime scene footprint datasets, and research on crime scene footprint retrieval is scarce. Although many pressure, fingerprint, and optical retrieval algorithms have achieved high retrieval accuracy, in criminal investigation, often only crime scene footprint images are available. There is a need to establish a comprehensive system for footprint extraction, cross-domain retrieval algorithms, and integrated systems based on crime scene footprints to meet the needs of actual criminal investigation.

[0003] First, for indoor on-site footprint image retrieval, there is a certain similarity between footprint images of different people, and there is a certain difference between different footprint images of the same person. In other words, there is a problem of close inter-class distance and large intra-class distance in the feature space. This problem leads to low distinguishability of footprint images, which will have a significant impact on the results of footprint retrieval. In order to solve this problem, it is urgent to provide an on-site footprint cross-domain retrieval method. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides the following technical solution:

[0005] A method for cross-domain retrieval of on-site footprints includes:

[0006] S1: Obtain the foreground footprint mask of the footprint image dataset through the foreground extraction module.

[0007] S2: Obtain footprint images based on the foreground footprint mask and footprint image dataset, and feed the footprint images into the momentum dual-branch network to obtain feature descriptors. This enhances the discriminative power of the feature descriptors by increasing the inter-class distance and decreasing the intra-class distance in the feature space.

[0008] S3: Calculate the Manhattan distance score between the feature descriptor and the feature information in the feature database, sort them according to the distance score, and determine the returned searcher information.

[0009] As an improvement to the above technical solution, the method for obtaining the footprint image dataset in step S1 includes the following steps:

[0010] S01: Obtain on-site footprint images by taking photos of the footprints at the scene.

[0011] S02: Acquire optical footprint images using an optical acquisition device.

[0012] S03: The footprint image dataset is obtained by recognizing and processing on-site footprint images and optical footprint images using the YoloV5 model.

[0013] As an improvement to the above technical solution, the optical footprint image also needs to undergo the following steps during processing:

[0014] S05: Filter the optical footprint image using median filtering.

[0015] S06: Perform data augmentation on the dataset of filtered optical footprint images.

[0016] As an improvement to the above technical solution, step S2 includes the following steps:

[0017] S21: The foreground footprint mask, the on-site footprint image dataset, and some manually extracted foreground footprint images are fed into the GauGAN network to train a cross-domain transformation model.

[0018] S22: Input all images from the on-site footprint image dataset into the cross-domain transformation model for separation to obtain the foreground footprint image.

[0019] S23: Construct a momentum dual-branch network, including a feature extraction network and a momentum feature extraction network.

[0020] S24: The momentum update method is used to update the feature descriptors obtained by the momentum feature extraction network and to establish a feature queue.

[0021] S25: When a sample is input, the feature descriptor obtained by the sample through the momentum dual-branch network is input into the feature queue, and the feature descriptor at the end of the queue is dequeued.

[0022] S26: Calculate the contrast loss based on the feature descriptor and similar or dissimilar features in the feature queue, and use the contrast loss to narrow the intra-class distance and widen the inter-class distance in the feature space to enhance the discriminative power of the feature descriptor.

[0023] As an improvement to the above technical solution, the calculation of the contrast loss depends on the following formula:

[0024]

[0025] Where q represents a single image query sample, and p represents another footprint image sample of the same person. It is the feature descriptor obtained by the feature extraction network from sample q. It is the feature descriptor obtained by the feature extraction network of sample p;

[0026] P(q) represents the feature set of images similar to it in the maintained feature queue, N(q) represents the dissimilar feature set, and i is an element in the union of feature sets P(q) and N(q);

[0027] D() represents the metric function used to measure the distance between two distributions, and τ is the scaling factor.

[0028] As an improvement to the above technical solution, the feature extraction network has the same structure as the momentum feature extraction network. The feature extraction network uses the ResNet50 network as the backbone network and obtains feature maps through several modules at the end of the ResNet50 network.

[0029] As an improvement to the above technical solution, the feature map processing method specifically includes the following steps:

[0030] S41: Pass the feature map through a max pooling layer and an average pooling layer respectively.

[0031] S42: Stack the pooled feature maps in the 0-dimensional plane, process them, and then pass them through a convolutional layer.

[0032] S43: Multiply the stacked feature map representation with the feature map representation processed by one convolution layer to obtain a weighted feature map.

[0033] S44: The weighted feature maps are convolved twice to make each weighted feature map have the same dimension, thereby obtaining feature maps containing information of different granularities. The obtained feature maps are then fused to obtain a new feature map.

[0034] As an improvement to the above technical solution, the training method for the network classifier includes the following steps:

[0035] S51: Input the several feature maps obtained in step S44 into the classifier of the ResNet50 network for classification. Each process of the classifier completing the classification is a step, and they are ordered in sequence.

[0036] S52: When several feature maps are input, each batch of input feature maps is trained starting from the first step until all steps are completed.

[0037] As an improvement to the above technical solution, the output after each step in step S52 and the final concatenated feature output training further include the following steps:

[0038] S521: Calculate the cross-entropy loss between the predicted labels obtained by the classifier and the true labels in each step.

[0039] S522: After obtaining the cross-entropy loss, perform gradient backpropagation, then proceed to the remaining steps in sequence, and take the output of the last step as the final output.

[0040] The beneficial effects of this invention are:

[0041] To obtain more discriminative features, a momentum-based dual-branch network for feature extraction was constructed. Features with varying granularities are extracted from different network stages of the main branch, and then fused to ultimately extract features containing multi-granularity information. The branch network structure and initial weights are consistent with the main network. By maintaining a momentum-updated feature descriptor queue and using KL divergence as the loss, the goal of increasing inter-class distance and decreasing intra-class distance in the feature space is achieved. Attached Figure Description

[0042] Figure 1 A plan view containing images of footprints taken at the scene;

[0043] Figure 2 A planar view of the optical footprint image;

[0044] Figure 3 This is a before-and-after comparison chart using the cross-domain conversion model;

[0045] Figure 4 This is a structural diagram of the feature extraction network of the present invention;

[0046] Figure 5 This is a structural diagram of the momentum dual-branch network of the present invention;

[0047] Figure 6 This is a diagram showing the ablation experiment results of this invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0049] Because footprint images of different people have certain similarities, and footprint images of the same person have certain differences, the problem of close inter-class distances and large intra-class distances occurs in the feature space. This problem leads to low distinguishability of footprint images and has a significant impact on the results of footprint retrieval. In order to solve this problem, it is urgent to provide a cross-domain retrieval method for on-site footprints.

[0050] To address this issue, a cross-domain retrieval method for on-site footprints is provided, including:

[0051] S1: Obtain the foreground footprint mask of the footprint image dataset through the foreground extraction module.

[0052] The purpose of obtaining the foreground footprint mask in step S1 is to facilitate the differentiation of footprint portions in the image and to separate the footprint portions from the image individually. The method for obtaining the footprint image dataset in step S1 includes the following steps:

[0053] S01: Obtain on-site footprint images by taking photos of the footprints at the scene.

[0054] S02: Acquire optical footprint images using an optical acquisition device.

[0055] S03: The footprint image dataset is obtained by recognizing and processing on-site footprint images and optical footprint images using the YoloV5 model.

[0056] Step S03 is to integrate the on-site footprint images and optical footprint images to obtain feature localization in the images and convert this information into a footprint image dataset.

[0057] In addition, since the RGB three-channel values ​​of the foreground texture in the footprint image dataset are significantly different from those of the background, but the foreground texture is small, numerous, and intertwined with the background, it is difficult to directly segment using a segmentation algorithm. In this embodiment, a custom-sized extraction box is used to find the gray value with the largest number of gray values ​​within the extraction box. This gray value ±n is regarded as the foreground. The foreground footprint mask is manually obtained by setting the thickness of the extraction box and setting the extraction threshold.

[0058] In one embodiment, the optical footprint image is further processed by performing the following steps:

[0059] S05: Filter the optical footprint image using median filtering.

[0060] Step S05 is to make the optical footprint image smoother. Furthermore, since the amount of image data acquired by the optical instrument is less than that of the indoor field footprint image, step S06 is also required to ensure generalization ability. Specifically:

[0061] S06: Perform data augmentation on the dataset of filtered optical footprint images.

[0062] Data augmentation can be used to improve the model's generalization ability to optical footprint image datasets and reduce overfitting.

[0063] S2: Obtain footprint images based on the foreground footprint mask and footprint image dataset, and feed the footprint images into the momentum dual-branch network to obtain feature descriptors. This enhances the discriminative power of the feature descriptors by increasing the inter-class distance and decreasing the intra-class distance in the feature space.

[0064] Step S2 includes the following steps:

[0065] S21: The foreground footprint mask, the on-site footprint image dataset, and some manually extracted foreground footprint images are fed into the GauGAN network to train a cross-domain transformation model.

[0066] S22: Input all images from the on-site footprint image dataset into the cross-domain transformation model for separation to obtain the foreground footprint image.

[0067] The transformed foreground footprint and optical footprint can be considered to be in a more similar domain, which facilitates the extraction of subsequent discriminative features.

[0068] Cross-domain transformation models can be trained adversarially by combining a multi-scale generator and a discriminator to enhance the accuracy of feature extraction.

[0069] S23: Construct a momentum dual-branch network, including a feature extraction network and a momentum feature extraction network.

[0070] Overall network structure (e.g.) Figure 5 As shown, a dual-branch network is adopted, and a momentum network branch Net_m is constructed by introducing the idea of ​​contrastive learning. [The establishment of this momentum network branch relies on the literature: {He,K.,Fan,H.,Wu,Y.,Xie,S.,and Girshick,R.,“Momentum Contrast for Unsupervised Visual RepresentationLearning”, arXiv e-prints [2019.doi:10.48550 / arXiv.1911.05722], its structure is completely consistent with Net (feature extraction network).

[0071] S24: The momentum update method is used to update the feature descriptors obtained by the momentum feature extraction network and to establish a feature queue.

[0072] Specifically, each time a batch of images q is input, the net obtains feature descriptors. Another footprint image of the same person is selected as a similar image p and input into the momentum feature extraction network to obtain feature descriptors.

[0073] Net_m updates its parameters using momentum updates, meaning it relies not only on the current Net's parameters but also on the Net's past parameters. The specific update method depends on the following formula:

[0074] θk=mθk+(1-m)θq

[0075] Where θk is the network parameter of the momentum feature extraction network branch, θq is the parameter of the feature extraction network branch, and m is the adjustable momentum coefficient.

[0076] While performing the above operations, a feature queue Q is also constructed, and step S24 is executed.

[0077] S25: When a sample is input, the feature descriptor obtained by the sample through the momentum dual-branch network is input into the feature queue, and the feature descriptor at the end of the queue is dequeued.

[0078] S26: Calculate the contrast loss based on the feature descriptor and similar or dissimilar features in the feature queue, and use the contrast loss to narrow the intra-class distance and widen the inter-class distance in the feature space to enhance the discriminative power of the feature descriptor.

[0079] The calculation of contrastive loss depends on the following formula:

[0080]

[0081] Where q represents a single image query sample, and p represents another footprint image sample of the same person. It is the feature descriptor obtained by the feature extraction network from sample q. It is the feature descriptor obtained by the feature extraction network of sample p;

[0082] P(q) represents the feature set of images similar to it in the maintained feature queue, N(q) represents the dissimilar feature set, and i is an element in the union of feature sets P(q) and N(q);

[0083] D() represents the metric function used to measure the distance between two distributions, and τ is the scaling factor.

[0084] By optimizing the loss term to make the numerator as small as possible and the denominator as large as possible, the goal of narrowing the intra-class distance and widening the inter-class distance in the feature space can be achieved.

[0085] When a certain input image needs to be compared with a large number of different batches of data for loss calculation, a queue can be used to store the feature descriptors that are similar or dissimilar to the input image. The queue is updated by enqueuing and dequeuing. Each new batch of images is enqueued after being output by the momentum feature extraction network, while the oldest batch of feature descriptors in the database is dequeued to achieve the update.

[0086] Specifically, when a batch is input, the obtained feature descriptor fp is added to the queue, and the last feature descriptor in the queue is dequeued. The comparison loss is calculated with features similar to or dissimilar to those in Q to achieve the goal of narrowing the intra-class distance and widening the inter-class distance in the feature space.

[0087] In one embodiment, see Figure 4 The feature extraction network uses the ResNet50 network as its backbone and obtains feature maps through several modules at the end of the ResNet50 network.

[0088] The feature map processing method specifically includes the following steps:

[0089] S41: Pass the feature map through a max pooling layer and an average pooling layer respectively.

[0090] Here, we take the feature maps obtained from the last three blocks of the ResNet50 network as an example. After the feature maps obtained from these three blocks are pooled, step S42 is executed.

[0091] S42: Stack the pooled feature maps in the 0-dimensional plane, process them, and then pass them through a convolutional layer.

[0092] The feature map after zero-dimensional stacking can be represented as [2,H,W], and after one convolution layer, it can be represented as [1,H,W]. This feature map represents the importance of each point on the original feature map.

[0093] S43: Multiply the stacked feature map representation with the feature map representation processed by one convolution layer to obtain a weighted feature map.

[0094] After multiplication in step S43, the network can adaptively focus on the areas that need attention.

[0095] S44: The weighted feature maps are convolved twice to make each weighted feature map have the same dimension, thereby obtaining feature maps containing information of different granularities. The obtained feature maps are then fused to obtain a new feature map.

[0096] Taking the aforementioned three block modules as an example, feature maps F1, F2, and F3 containing information of different granularities are obtained, and feature fusion is performed to form a new feature map F4.

[0097] In one embodiment, the method for training a network classifier includes the following steps:

[0098] S51: Input the several feature maps obtained in step S44 into the classifier of the ResNet50 network for classification. Each process of the classifier completing the classification is a step, and they are ordered in sequence.

[0099] Using the example above, step S51 inputs feature maps F1, F2, F3 and F4 into their respective classifiers for classification, and there are four processes, which are denoted as a1-a4 respectively. These processes are ordered and step S52 is executed.

[0100] The network classifier has two fully connected layers, two batch normalization (BN) layers, and one softmax layer.

[0101] S52: When several feature maps are input, each batch of input feature maps is trained starting from the first step until all steps are completed.

[0102] The feature maps input here are used as a batch of samples. Each batch of samples is used to start training sequentially from a1 to a4.

[0103] Because the receptive domain and expressive power of low-level (e.g., a1) are limited, the network will first extract discriminative information from local details (e.g., object texture). Compared to training the entire network directly, this training method allows the model to locate discriminative information from local details to global structure as feature extraction gradually progresses to higher levels, rather than learning all granularities at the same time.

[0104] In one embodiment, the training of the output after each step in step S52 and the output of the final concatenated feature further includes the following steps:

[0105] S521: Calculate the cross-entropy loss between the predicted labels obtained by the classifier and the true labels in each step.

[0106] S522: After obtaining the cross-entropy loss, perform gradient backpropagation, then proceed to the remaining steps in sequence, and take the output of the last step as the final output.

[0107] Taking the aforementioned a1-a4 as an example, in the a1 process, the predicted label obtained by the classifier is used to calculate the cross-entropy with the real label. After obtaining the cross-entropy loss, the gradient is backpropagated, and then the process enters a2, a3, and a4. Each batch of images sent into the network undergoes these four processes a1-a4, and finally the output of the a4 process is selected as the final output.

[0108] In addition to the steps mentioned above, the footprint images can be scrambled at each stage to improve robustness, based on the characteristics of the footprint feature region distribution. That is, the feature regions are randomly shuffled for data augmentation. Taking a1-a4 as an example, the 10% toe area, 40% forefoot area, 15% arch area and 35% heel area are usually randomly shuffled into four equal parts in process a1 and randomly shuffled into left and right parts in process a2.

[0109] S3: Calculate the Manhattan distance score between the feature descriptor and the feature information in the feature database, sort them according to the distance score, and determine the returned searcher information.

[0110] The Manhattan distance score is used to determine the degree of difference between different feature information. The difference size is judged based on this distance score, and the information is sorted according to the difference size. The information with the smallest difference is selected as the final information to be retrieved and returned.

[0111] The feature database is created by sequentially inputting all images from an image library into a feature extraction network to extract features, storing personnel information and feature descriptors in dictionary form. When a query image passes through this network, it outputs a feature descriptor. The Manhattan distance score is calculated between this descriptor and features in the database. The degree of difference between different feature information is determined based on the Manhattan distance score, and the features are sorted according to their magnitude. The information with the smallest difference is selected as the final retrieved information and returned.

[0112] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it.

Claims

1. A method for cross-domain retrieval of on-site footprints, characterized in that, Includes the following steps: S1: Obtain the foreground footprint mask of the footprint image dataset through the foreground extraction module; S2: Obtain footprint images based on the foreground footprint mask and footprint image dataset, and feed the footprint images into the momentum dual-branch network to obtain feature descriptors. This enhances the discriminative power of the feature descriptors by increasing the inter-class distance and decreasing the intra-class distance in the feature space. S3: Calculate the Manhattan distance score between the feature descriptor and the feature information in the feature database, sort them according to the distance score, and determine the returned searcher information; Step S2 involves obtaining footprint images based on the foreground footprint mask and footprint image dataset, and then feeding these footprint images into a momentum dual-branch network to obtain feature descriptors. The discriminative power of the feature descriptors is enhanced by increasing the inter-class distance and decreasing the intra-class distance in the feature space. This includes the following steps: S21: The foreground footprint mask, the on-site footprint image dataset, and some manually extracted foreground footprint images are fed into the GauGAN network to train a cross-domain transformation model; S22: Input all images from the on-site footprint image dataset into the cross-domain transformation model for separation to obtain the foreground footprint image; S23: Construct a momentum dual-branch network, including a feature extraction network and a momentum feature extraction network; S24: The momentum update method is used to update the feature descriptors obtained by the momentum feature extraction network and a feature queue is established; S25: When a sample is input, the feature descriptor obtained by the sample through the momentum dual-branch network is input into the feature queue, and the feature descriptor at the end of the queue is dequeued. S26: Calculate the contrast loss based on the feature descriptor and similar or dissimilar features in the feature queue, and use the contrast loss to narrow the intra-class distance and widen the inter-class distance in the feature space to enhance the discriminative power of the feature descriptor; The calculation of the contrast loss depends on the following formula: Where q represents a single image query sample, and p represents another footprint image sample of the same person. It is the feature descriptor obtained by the feature extraction network from sample q. It is the feature descriptor obtained by the feature extraction network of sample p; P(q) represents the feature set of images similar to it in the maintained feature queue, N(q) represents the dissimilar feature set, and i is an element in the union of feature sets P(q) and N(q); D() represents the metric function used to measure the distance between two distributions, and τ is the scaling factor.

2. The method for cross-domain retrieval of on-site footprints according to claim 1, characterized in that: Step S1, which involves obtaining the foreground footprint mask from the footprint image dataset using the foreground extraction module, includes the following steps: S01: Obtain on-site footprint images by taking photos of the footprints at the scene; S02: Acquire optical footprint images using an optical acquisition device; S03: The footprint image dataset is obtained by recognizing and processing on-site footprint images and optical footprint images using the YoloV5 model.

3. The method for cross-domain retrieval of on-site footprints according to claim 2, characterized in that: The optical footprint image also requires the following steps to be performed during processing: S05: Filter the optical footprint image using median filtering; S06: Perform data augmentation on the dataset of filtered optical footprint images.

4. The method for cross-domain retrieval of on-site footprints according to claim 1, characterized in that: The feature extraction network has the same structure as the momentum feature extraction network. The feature extraction network uses the ResNet50 network as the backbone network and obtains feature maps through several modules at the end of the ResNet50 network.

5. The method for cross-domain retrieval of on-site footprints according to claim 4, characterized in that: The feature map processing method specifically includes the following steps: S41: Pass the feature map through both max pooling and average pooling layers; S42: Stack the pooled feature maps in 0-dimensional space, process them, and then pass them through a convolutional layer. S43: Multiply the stacked feature map representation with the feature map representation processed by one convolution layer to obtain a weighted feature map; S44: The weighted feature maps are convolved twice to make each weighted feature map have the same dimension, thereby obtaining feature maps containing information of different granularities. The obtained feature maps are then fused to obtain a new feature map.

6. The method for cross-domain retrieval of on-site footprints according to claim 5, characterized in that: The training method for a network classifier includes the following steps: S51: Input the several feature maps obtained in step S44 into the classifier of the ResNet50 network for classification. Each process of the classifier completing the classification is a step, and they are ordered in sequence. S52: When several feature maps are input, each batch of input feature maps is trained starting from the first step until all steps are completed.

7. The method for cross-domain retrieval of on-site footprints according to claim 6, characterized in that: The training of the outputs after each step in step S52 and the final concatenated feature output also includes the following steps: S521: Calculate the cross-entropy loss between the predicted labels obtained by the classifier and the true labels in each step; S522: After obtaining the cross-entropy loss, perform gradient backpropagation, then proceed to the remaining steps in sequence, and take the output of the last step as the final output.

Images