A cross-domain footprint pressure image retrieval system based on high-order spatiotemporal relations
By constructing a cross-domain footprint pressure image retrieval system with high-order spatiotemporal relationships, using 2D blocking and spatial attention module to extract features, the problem of unconsidered time and spatial correlation of footprint features in the existing technology is solved, and the recognition is improved Accuracy and fast and convenient access control management in the security field.
Patent Information
- Application Number
- CN202410325445.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-21
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-03-21
AI Technical Summary
The existing footprint retrieval method fails to effectively consider the temporal and spatial correlation of footprint characteristics, and the footprint images of different domains are very different, resulting in low recognition accuracy. Especially when wearing different shoe types, it is difficult to extract the same one. Highly distinguishing characteristics of objects.
Using a cross-domain footprint pressure image retrieval system based on high-order spatiotemporal relationships, a network model is constructed to extract features by collecting and preprocessing footprint pressure images, calculating image similarity using metric functions, and 2D blocking and spatial attention are used. The module builds hypergraphs, aggregates time and spatial clues, and realizes cross-domain retrieval.
It improves the recognition accuracy of wearing different shoe types, saves time and labor costs through deep learning, and improves the convenience and security of access control management in the security field.
Smart Images

Figure CN118210941B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a cross-domain footprint pressure image retrieval system based on high-order spatiotemporal relations. Background Art
[0002] The application value of the whole-trip footprint identification technology in the field of security is self-evident. It does not require personnel to stop and cooperate deliberately, and can effectively complete the identity verification work by just walking normally. Compared with the single-footprint identification technology, the whole-trip footprint identification technology can obtain the correlation information between footprints, with high accuracy and not easy to disguise.
[0003] Existing methods for searching for footprints focus on temporal or spatial clues, but do not consider the temporal and spatial correlation of footprint features at the same time. In addition, due to the different types of shoes worn, footprint images vary greatly, and there is a semantic gap between footprint images in different domains, which brings challenges to the analysis and comparison of footprints. Therefore, extracting highly discriminative features of the same object and common features in footprints in different domains is of vital help to the analysis of footprints. Summary of the invention
[0004] 1. Technical issues to be resolved
[0005] In view of the shortcomings of the prior art, the present invention provides a cross-domain footprint pressure image retrieval system based on high-order spatiotemporal relationships, which solves the limitation of the existing footprint retrieval that only targets a single footprint in the same domain, further explores the spatiotemporal dependency between footprints and improves the recognition accuracy of people wearing different shoe types.
[0006] (II) Technical solution
[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a cross-domain footprint pressure image retrieval system based on high-order spatiotemporal relationship, comprising the following steps:
[0008] S1: The acquisition window obtains the shoe footprint pressure image to be queried;
[0009] S2: Send the collected image to the CPU processor for image processing;
[0010] S3: Add the best model obtained through training and load the barefoot footprint pressure image in the base library to the directory of the system;
[0011] S4: using the system to adopt a metric function to calculate the similarity between the footprint pressure image to be queried and the footprint pressure images in the base database;
[0012] S5: using the cross-domain trip footprint pressure retrieval function in the system to obtain the barefoot footprint pressure image in the base database that is most similar to the shoe footprint pressure image to be queried;
[0013] S6: Print the final search result in the display window;
[0014] The optimal model training described in step S3 includes the following steps:
[0015] The first step is to collect and preprocess the training images: the pressure sensors are used to collect the pressure images of 200 barefoot and shoe-wearing footprints, and multiple single footprint pressure images are obtained by cropping. All images are resized to 256×256, and the cropped footprint pressure images are centered and regularized.
[0016] The second step is optimal network training: construct and train the network model, and divide the data set into a training set and a test set in a ratio of 7:3. Input the training samples into the network model, train the network structure module, and obtain the trained network; for the test samples, use the pressure images of shoe-wearing footprints as the query library and the pressure images of barefoot footprints as the base library, input the query images into the trained network, and search in the base library.
[0017] Among them, the specific steps of network structure module training in the second step are as follows:
[0018] (1) Construct a backbone network for extracting footprint features: Given a sequence of footprint pressure images The backbone network uses ResNet50 pre-trained on Imagenet to obtain a collection of sequence-level features. , .in Dimension A three-dimensional tensor of is the channel size, , are the height and width of the feature map respectively. The feature map output by the backbone network is input into the three branches respectively to obtain global feature information, 2D block feature information, and spatial attention feature information;
[0019] (2) Construct a 2D block module: Divide the feature map output by the backbone network into different granularities The segmented feature map is averaged and pooled to construct a partial-level feature vector ,in For each granularity of sequence-level samples with T footprints, the number of nodes in this branch is ;
[0020] (3) Constructing the spatial attention module: Using A spatial attention learner locates a set of discriminative image regions. For each frame-level feature map , each feature is The attention learner generates the corresponding spatial attention map. After two linear transformations and a ReLU activation function, the attention map is generated. After the first linear transformation, the original data is projected to dimensional space, and the second transformation projects the feature map to dimensional space, that is, a The attention distribution of each channel is used to indicate the importance of the region in different spatial attention models. Then, the attention map is normalized to [0, 1] using softmax to obtain the normalized attention map , represents the weight of a receptive field of the attention model on the image. The feature map is weighted by the attention map, and the and Element-wise product, the features of each channel are summarized by global average pooling to obtain Used to initialize the nodes in the hypergraph. The number of nodes in this branch is ;
[0021] (4) Construct a high-order spatiotemporal relationship module: Use the nodes obtained from the 2D block and spatial attention modules to construct a hypergraph ,in Indicates a specific granularity. Include nodes and a set of related hyperedges. In order to aggregate temporal and spatial clues, three hypergraphs are constructed at each granularity to fully consider the temporal dependencies of different time lengths. An adjacency matrix is constructed, and each element in the matrix represents the relationship between the features of two hypergraph nodes. The adjacency matrix is added to its transpose to achieve symmetry of the adjacency matrix. ,in represents the number of nodes. Then normalize it to get . Calculate the cosine distance between the adjacency matrix and the node features, and use K nearest neighbors to obtain the most relevant node features within a specific time range, and use it as a hyperedge. Aggregate node features along the hyperedge and average all node features on a hyperedge except the node itself as a new node feature , generate node-level attention for each hypergraph , to select the most distinctive node features, perform weighted summation on the obtained node features, and finally obtain the hypergraph features .
[0022] As an optimization, in step S5, the cross-domain footprint pressure retrieval function is to input the footprint pressure image to be queried into the trained network model, use the metric function of the network model to calculate the distance between the query image and the image in the base library, and measure the similarity between the query image and the base library image by the distance.
[0023] As an optimization, in the first step, the footprint pressure images of a trip are cropped into multiple single footprint pressure images, and each object contains 72 shoe footprint pressure images and barefoot footprint pressure images.
[0024] As an optimization, the data preprocessing methods in the first step include centralization and regularization.
[0025] As an optimization, in order to aggregate temporal and spatial clues in the high-order spatiotemporal relationship module constructed in step (4), three hypergraphs are constructed at each granularity to fully consider the temporal dependencies of different time lengths.
[0026] (III) Beneficial effects
[0027] The present invention provides a cross-domain footprint pressure image retrieval system based on high-order spatiotemporal relationship, which has the following beneficial effects:
[0028] The present invention collects barefoot footprint pressure images and shoe footprint pressure images, extracts features and constructs a hypergraph to capture high-order spatiotemporal relationships, unifies the learning of spatial cues and temporal cues into a joint framework, improves retrieval accuracy, and solves the cross-domain problem of footprint pressure images; promotes spatial alignment of footprint areas through a 2D blocking module, alleviates the problem of structural information loss caused by fixed blocking through a spatial attention module, and adaptively learns a group of distinctive footprint salient areas to focus on, thereby obtaining more distinctive features; automatically performs footprint image matching and retrieval through deep learning, greatly saving time and labor costs; the invention has wide applicability in the field of security, can be used for rapid unlocking of entrances and exits of private places, greatly improving the convenience of access control management, and can also be used in key places to set up a database of footprints of people in key areas to improve the security of key areas. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 is a flow chart of the retrieval system of the present invention;
[0030] Figure 2 This is a framework diagram for model training and retrieval of the present invention;
[0031] Figure 3 It is a framework diagram of the high-order spatiotemporal relationship model of the present invention. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] See also Figure 1-3 , the present invention provides a technical solution:
[0034] like Figure 2 As shown, model training and retrieval are completed in the following steps:
[0035] The first step is to collect and preprocess the training images: the pressure sensors are used to collect the pressure images of 200 barefoot and shoe-wearing footprints, and multiple single footprint pressure images are obtained by cropping. All images are resized to 256×256, and the cropped footprint pressure images are centered and regularized.
[0036] The second step is optimal network training: construct and train the network model, and divide the data set into a training set and a test set in a ratio of 7:3. Input the training samples into the network model, train the network structure module, and obtain the trained network; for the test samples, use the pressure images of shoe-wearing footprints as the query library and the pressure images of barefoot footprints as the base library. Input the query images into the trained network, and compare the query features extracted by the network with the features in the feature library. The smaller the distance, the more similar they are, and finally obtain the prediction accuracy.
[0037] The specific steps for training the network structure module in the second step are as follows:
[0038] (1) Construct a backbone network for extracting footprint features: Given a sequence of footprint pressure images The backbone network uses ResNet50 pre-trained on Imagenet to obtain a collection of sequence-level features. , .in Dimension A three-dimensional tensor of is the channel size, , are the height and width of the feature map respectively. The feature map output by the backbone network is input into the three branches respectively to obtain global feature information, 2D block feature information, and spatial attention feature information;
[0039] (2) Construct a 2D block module: Divide the feature map output by the backbone network into different granularities The segmented feature map is averaged and pooled to construct a partial-level feature vector ,in For each granularity of sequence-level samples with T footprints, the number of nodes in this branch is ;
[0040] (3) Constructing the spatial attention module: Using A spatial attention learner locates a set of discriminative image regions. For each frame-level feature map , each feature is The attention learner generates the corresponding spatial attention map. After two linear transformations and a ReLU activation function, the attention map is generated. After the first linear transformation, the original data is projected to dimensional space, and the second transformation projects the feature map to dimensional space, that is, a The attention distribution of each channel is used to indicate the importance of the region in different spatial attention models. Then, the attention map is normalized to [0, 1] using softmax to obtain the normalized attention map , represents the weight of a receptive field of the attention model on the image. The feature map is weighted by the attention map, and the and Element-wise product, the features of each channel are summarized by global average pooling to obtain Used to initialize the nodes in the hypergraph. The number of nodes in this branch is ;
[0041] (4) Construct a high-order spatiotemporal relationship module: Use the nodes obtained from the 2D block and spatial attention modules to construct a hypergraph ,in Indicates a specific granularity. Include nodes and a set of related hyperedges. In order to aggregate temporal and spatial clues, three hypergraphs are constructed at each granularity to fully consider the temporal dependencies of different time lengths. An adjacency matrix is constructed, and each element in the matrix represents the relationship between the features of two hypergraph nodes. The adjacency matrix is added to its transpose to achieve symmetry of the adjacency matrix. ,in represents the number of nodes. Then normalize it to get . Calculate the cosine distance between the adjacency matrix and the node features, and use K nearest neighbors to obtain the most relevant node features within a specific time range, and use it as a hyperedge. Aggregate node features along the hyperedge and average all node features on a hyperedge except the node itself as a new node feature , generate node-level attention for each hypergraph , to select the most distinctive node features, perform weighted summation on the obtained node features, and finally obtain the hypergraph features .
[0042] The third step is the cross-domain full-trip footprint pressure retrieval function: The cross-domain full-trip footprint pressure retrieval function is the core of the system. In this step, the system is committed to finding objects that match the footprint pressure image of the unknown identity to be queried. The operation process is as follows: first, the footprint image to be queried is input into the trained network model; then, the metric function is used to calculate the similarity between the image to be tested and the image in the base library; then, this indicator is used to measure the similarity between the image to be tested and the base library image; the system finally returns the most similar image as the output result. This process aims to achieve efficient image matching, thereby helping users to accurately find the target footprint.
[0043] Based on the above steps, Figure 1 As shown in FIG. 1 , a cross-domain footprint pressure image retrieval system based on high-order spatiotemporal relations is described. The specific operation steps are as follows:
[0044] S1: The acquisition window obtains the pressure image of the shoe-wearing footprint to be queried;
[0045] S2: Send the collected image to the CPU processor for image processing;
[0046] S3: Add the best model obtained through training and load the barefoot footprint pressure image in the base library to the directory of the system;
[0047] S4: using the system to use a metric function to calculate the similarity between the shoe footprint pressure image to be queried and the barefoot footprint pressure image in the base database;
[0048] S5: using the cross-domain trip footprint retrieval function in the system to obtain the barefoot footprint pressure image in the base database that is most similar to the shoe footprint pressure image to be queried;
[0049] S6: Print out the final search results in the display window.
[0050] During the optimization process of the entire network, the present invention extracts finer-grained information and more discriminative features through a 2D blocking module and a spatial attention module, and adopts a hypergraph neural network to effectively utilize the dynamic spatiotemporal relationships and domain-invariant features in the trip footprint sequence to obtain better cross-domain trip retrieval performance.
[0051] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A cross-domain footprint pressure image retrieval method based on high-order spatiotemporal relations, characterized by: The following steps are involved: S1: The acquisition window obtains the pressure image of the shoe-wearing footprint to be queried; S2: Send the collected image to the CPU processor for image processing; S3: Add the optimal model obtained through training and load the barefoot footprint pressure image in the base library to the system directory; S4: using the system to use a metric function to calculate the similarity between the footprint image to be queried and the footprint images in the base database; S5: using the cross-domain footprint pressure retrieval function in the system to obtain the barefoot footprint pressure image in the base database that is most similar to the shoe-wearing footprint pressure image to be queried; S6: Print the final search result in the display window; The optimal model training described in step S3 includes the following steps: The first step is to collect and preprocess the training images: the pressure sensors are used to collect the pressure images of 200 barefoot and shoe-wearing footprints, and multiple single footprint pressure images are obtained by cropping. All images are resized to 256×256, and the cropped footprint pressure images are centered and regularized. The second step is optimal network training: construct and train the network model, divide the data set into training set and test set in a ratio of 7:3; input the training samples into the network model, train the network structure module, and obtain the trained network; for the test samples, use the pressure images of shoe-wearing footprints as the query library and the pressure images of barefoot footprints as the base library, input the query images into the trained network, and search in the base library; Among them, the specific steps of network structure module training in the second step are as follows: (1) Construct a backbone network for extracting footprint features: Given a sequence of footprint pressure images The backbone network uses ResNet50 pre-trained on Imagenet to obtain a collection of sequence-level features. , ,in Dimension A three-dimensional tensor of is the channel size, , are the height and width of the feature map respectively; the feature map output by the backbone network is input into the three branches respectively to obtain global feature information, 2D block feature information, and spatial attention feature information; (2) Construct a 2D block module: Divide the feature map output by the backbone network into different granularities The segmented feature map is averaged and pooled to construct a partial-level feature vector ,in ; For each granularity of sequence-level samples with T footprints, the number of nodes in this branch is ; (3) Constructing the spatial attention module: Using A spatial attention learner locates a set of distinctive image regions; for each frame-level feature map , each feature is The attention learner generates the corresponding spatial attention map, which is generated by two linear transformations and a ReLU activation function. , after the first linear transformation, the original data is projected onto dimensional space, and the second transformation projects the feature map to dimensional space, that is, a The attention distribution of each channel is used to indicate the importance of the region in different spatial attention models. Then, the attention map is normalized to [0, 1] using softmax to obtain the normalized attention map. , represents the weight of a receptive field of the attention model on the image, and the feature map is weighted by the attention map, along the channel dimension and Element-wise product, the features of each channel are summarized by global average pooling to obtain Used to initialize the nodes in the hypergraph. The number of nodes in this branch is ; (4) Construct a high-order spatiotemporal relationship module: Use the nodes obtained from the 2D block and spatial attention modules to construct a hypergraph ,in Indicates a specific granularity, Include nodes and a set of related hyperedges; construct an adjacency matrix, where each element in the matrix represents the relationship between the features of two hypergraph nodes, and add the adjacency matrix to its transpose to achieve symmetry of the adjacency matrix. ,in Represents the number of nodes, and then normalizes it to get ; Calculate the cosine distance between the adjacency matrix and the node features, use K nearest neighbors to get the most relevant node features within a specific time range, and use it as a hyperedge; Aggregate node features along the hyperedge, and average all node features on a hyperedge except the node itself as a new node feature , generate node-level attention for each hypergraph , to select the most distinctive node features, perform weighted summation on the obtained node features, and finally obtain the hypergraph features .
2. According to claim 1, a cross-domain footprint pressure image retrieval method based on high-order spatiotemporal relationship is characterized by: In step S5, the cross-domain footprint pressure retrieval function is to input the footprint pressure image to be queried into the trained network model, use the metric function of the network model to calculate the distance between the query image and the image in the base library, and measure the similarity between the image to be tested and the base library image by the distance.
3. The cross-domain footprint pressure image retrieval method based on high-order spatiotemporal relationship according to claim 1 is characterized by: In the first step, the footprint pressure images are cropped into multiple single footprint pressure images, each object contains 72 shod footprint pressure images and barefoot footprint pressure images.
4. The cross-domain footprint pressure image retrieval method based on high-order spatiotemporal relationship according to claim 1 is characterized by: In the first step, the methods for data preprocessing include centralization and regularization.
5. The cross-domain footprint pressure image retrieval method based on high-order spatiotemporal relationship according to claim 1 is characterized by: In step (4), in order to aggregate temporal and spatial clues in constructing the high-order spatiotemporal relationship module, three hypergraphs are constructed at each granularity to fully consider the temporal dependencies of different time lengths.
Citation Information
Cited By
Deep learning-based barefoot footprint comparison method and device
CN120388360A
Deep learning based barefoot footprint comparison method and apparatus
CN120388360B
