A footprint recognition method, device and storage medium

Abstract

The present invention proposes a footprint recognition method, device and storage medium, which relate to the field of computer vision technology. The method comprises: obtaining a footprint image to be recognized; performing feature extraction on the footprint image to be recognized using a shallow feature extraction model to obtain shallow image features corresponding to the footprint image to be recognized; inputting the shallow image features into a deep feature extraction model to obtain full-foot deep features and K local deep features corresponding to the footprint image to be recognized; performing feature fusion on the shallow image features, the full-foot deep features and the K local deep features to obtain fused features; and performing footprint recognition on the footprint image to be recognized based on the fused features. The present invention can effectively integrate feature information at different levels and different parts to improve the robustness and accuracy of footprint recognition in complex environments.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a footprint recognition method, device and storage medium. Background Art

[0002] Footprint recognition technology faces many limitations and challenges in today's complex crime scenarios. Traditional footprint recognition methods are limited by the limited information provided by a single footprint image, and are easily interfered by environmental factors. For example, factors such as light, perspective, and scale changes can cause the acquired image features to be unstable, leading to misjudgment.

[0003] In recent years, footprint recognition methods based on deep learning have significantly improved the accuracy and robustness of recognition and have become one of the current mainstream methods. However, footprint recognition methods based on deep learning still have many defects. For example, existing footprint recognition methods based on deep learning tend to take the entire footprint image as input to perform footprint recognition based on global image information. This often leads to insufficient local information extraction capabilities of deep learning models. Especially in applications at complex crime scenes, existing footprint recognition methods usually result in inaccurate recognition due to the lack of local feature information. At the same time, traditional deep learning models are also unable to keenly perceive the interference of environmental factors, further affecting the consistency and accuracy of recognition. Therefore, how to improve the accuracy of footprint recognition has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the present invention proposes a footprint recognition method, device and storage medium. First, a shallow feature extraction model is used to extract shallow image features of the footprint image to be identified. Then, a deep feature extraction model is used to obtain the full-foot deep features and local deep features of the footprint image to be identified. Finally, the shallow image features, the full-foot deep features and the local deep features are fused, and the fused features are used to perform footprint recognition. In this way, feature information at different levels and different parts can be effectively integrated to improve the robustness and accuracy of footprint recognition in complex environments.

[0005] A first aspect of the present invention provides a footprint recognition method, the footprint recognition method comprising:

[0006] Obtaining a footprint image to be identified;

[0007] A shallow feature extraction model is used to extract features of the footprint image to be identified, so as to obtain shallow image features corresponding to the footprint image to be identified; wherein the shallow feature extraction model uses N convolution layers to perform convolution operations on the footprint image to be identified; N is a positive integer greater than or equal to 1;

[0008] The shallow image features are input into the deep feature extraction model to obtain the full-foot deep features and K local deep features corresponding to the footprint image to be identified; wherein the deep feature extraction model includes a first feature extraction module and K second feature extraction modules; the first feature extraction module is used to determine the full-foot deep features according to the shallow image features; any one of the K second feature extraction modules is used to determine a local deep feature according to a local image feature; the local image features are obtained by segmenting the shallow image features; K is a positive integer greater than or equal to 1;

[0009] Perform feature fusion on shallow image features, full-foot deep features and K local deep features to obtain fused features;

[0010] The footprints to be identified are recognized based on the fused features.

[0011] In an optional embodiment, the shallow feature extraction model includes a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in sequence; wherein:

[0012] The first convolution layer is used to input the footprint image to be identified and output the first image feature; the first convolution layer includes a first convolution sublayer, a second convolution sublayer and a pooling sublayer; the first convolution sublayer and the second convolution sublayer have different step sizes; the pooling sublayer is used to downsample the output results of the first convolution sublayer and the second convolution sublayer to obtain the first image feature;

[0013] The second convolutional layer is used to input the first image feature and output the second image feature; the second convolutional layer includes multiple groups of first edge attention residual modules;

[0014] The third convolutional layer is used to input the second image features and output shallow image features; the third convolutional layer includes multiple groups of first edge attention residual modules and second edge attention residual modules; the resolutions of the first edge attention residual modules and the second edge attention residual modules are different.

[0015] In an optional embodiment, the first edge attention residual module includes a third convolution sublayer, a fourth convolution sublayer, a fifth convolution sublayer, a connection sublayer and a sixth convolution sublayer; the first edge attention residual module is used to:

[0016] Get input features;

[0017] The input features are convolved through the third convolution sublayer; the step size of the third convolution sublayer is 1, and the convolution kernel size of the third convolution sublayer is 1×1;

[0018] The output results of the third convolution sublayer are grouped in the channel dimension, and the grouped output results of the third convolution sublayer are input to the fourth convolution sublayer and the fifth convolution sublayer respectively; the fourth convolution sublayer is used for convolution operation, the step size of the fourth convolution sublayer is 1 and the convolution kernel size of the fourth convolution sublayer is 3×3; the fifth convolution sublayer is used for differential convolution operation, the step size of the fifth convolution sublayer is 1 and the convolution kernel size of the fifth convolution sublayer is 3×3;

[0019] The output results of the fourth convolutional sublayer and the fifth convolutional sublayer are fused using the connection sublayer;

[0020] The fusion result output by the connection sublayer is calculated using the sixth convolution sublayer; the step size of the sixth convolution sublayer is 1 and the convolution kernel size of the sixth convolution sublayer is 1×1;

[0021] The output result of the sixth convolutional sublayer and the input features are weighted to obtain the output result of the first edge attention residual module.

[0022] In an optional embodiment, K is equal to 2; and the method further comprises:

[0023] Segment the shallow image features into forefoot image features and hindfoot image features;

[0024] Inputting the forefoot image features and the hindfoot image features into two second feature extraction modules respectively to obtain the forefoot deep features corresponding to the forefoot image features and the hindfoot deep features corresponding to the hindfoot image features;

[0025] The shallow image features, the full foot deep features, and K local deep features are fused to obtain fused features, including:

[0026] The shallow image features, the whole foot deep features, the forefoot deep features and the hindfoot deep features are fused to obtain the fused features.

[0027] In an optional embodiment, the first feature extraction module and the second feature extraction module both include M convolutional layers; the M convolutional layers include a fourth convolutional layer and a fifth convolutional layer connected in sequence; M is a positive integer greater than or equal to 2. The fourth convolutional layer includes at least one group of first edge attention residual modules and at least one group of second edge attention residual modules; the fifth convolutional layer includes multiple groups of first edge attention residual modules.

[0028] In an optional embodiment, before fusing the shallow image features, the whole foot deep features, the forefoot deep features and the hindfoot deep features, the method further includes:

[0029] The dimensions and sizes of the shallow image features, the deep features of the whole foot, the deep features of the forefoot and the deep features of the hindfoot are adjusted respectively so that the dimensions and sizes of the shallow image features, the deep features of the whole foot, the deep features of the forefoot and the deep features of the hindfoot are the same.

[0030] In an optional embodiment, the method further includes:

[0031] The footprint image to be identified is subjected to image preprocessing; the image preprocessing includes one or a combination of image grayscale adjustment processing, image background separation processing, image data enhancement processing and image scaling processing.

[0032] In an optional implementation, the shallow image features, the full foot deep features and K local deep features are fused to obtain fused features, including:

[0033] A fusion model is used to fuse shallow image features, full-foot deep features and K local deep features; wherein the fusion model includes a convolution module, a spatial attention module and a channel attention module connected in sequence; the convolution module is used to reduce the dimension of shallow image features, full-foot deep features or K local deep features; the spatial attention module is used to output the spatial attention weighted features corresponding to the shallow image features, full-foot deep features or K local deep features after dimensionality reduction; the channel attention module is used to obtain connection features, which are obtained by connecting the spatial attention weighted features corresponding to the shallow image features, full-foot deep features and K local deep features in the channel dimension; and the connection features are weighted according to the channel attention; the output result of the channel attention module is used to determine the fusion features.

[0034] A second aspect of the present invention provides a footprint recognition device, the device comprising:

[0035] An acquisition unit, used for acquiring a footprint image to be identified;

[0036] A processing unit, used for performing feature extraction on the footprint image to be identified by using a shallow feature extraction model to obtain shallow image features corresponding to the footprint image to be identified; wherein the shallow feature extraction model performs convolution operation on the footprint image to be identified by using N convolution layers; N is a positive integer greater than or equal to 1;

[0037] The processing unit is further used to input the shallow image features into the deep feature extraction model to obtain the full-foot deep features and K local deep features corresponding to the footprint image to be identified; wherein the deep feature extraction model includes a first feature extraction module and K second feature extraction modules; the first feature extraction module is used to determine the full-foot deep features according to the shallow image features; any one of the K second feature extraction modules is used to determine a local deep feature according to a local image feature; the local image features are obtained by segmenting the shallow image features; K is a positive integer greater than or equal to 1;

[0038] The processing unit is further used to perform feature fusion on the shallow image features, the full foot deep features, and the K local deep features to obtain fused features;

[0039] The recognition unit is used to perform footprint recognition on the footprint image to be recognized according to the fusion features.

[0040] In an optional embodiment, the shallow feature extraction model includes a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in sequence; wherein:

[0041] The first convolution layer is used to input the footprint image to be identified and output the first image feature; the first convolution layer includes a first convolution sublayer, a second convolution sublayer and a pooling sublayer; the first convolution sublayer and the second convolution sublayer have different step sizes; the pooling sublayer is used to downsample the output results of the first convolution sublayer and the second convolution sublayer to obtain the first image feature;

[0042] The second convolutional layer is used to input the first image feature and output the second image feature; the second convolutional layer includes multiple groups of first edge attention residual modules;

[0043] The third convolutional layer is used to input the second image features and output shallow image features; the third convolutional layer includes multiple groups of first edge attention residual modules and second edge attention residual modules; the resolutions of the first edge attention residual modules and the second edge attention residual modules are different.

[0044] In an optional embodiment, the first edge attention residual module includes a third convolution sublayer, a fourth convolution sublayer, a fifth convolution sublayer, a connection sublayer and a sixth convolution sublayer; the first edge attention residual module is used to:

[0045] Get input features;

[0046] The input features are convolved through the third convolution sublayer; the step size of the third convolution sublayer is 1, and the convolution kernel size of the third convolution sublayer is 1×1;

[0047] The output results of the third convolution sublayer are grouped in the channel dimension, and the grouped output results of the third convolution sublayer are input to the fourth convolution sublayer and the fifth convolution sublayer respectively; the fourth convolution sublayer is used for convolution operation, the step size of the fourth convolution sublayer is 1 and the convolution kernel size of the fourth convolution sublayer is 3×3; the fifth convolution sublayer is used for differential convolution operation, the step size of the fifth convolution sublayer is 1 and the convolution kernel size of the fifth convolution sublayer is 3×3;

[0048] The output results of the fourth convolutional sublayer and the fifth convolutional sublayer are fused using the connection sublayer;

[0049] The fusion result output by the connection sublayer is calculated using the sixth convolution sublayer; the step size of the sixth convolution sublayer is 1 and the convolution kernel size of the sixth convolution sublayer is 1×1;

[0050] The output result of the sixth convolutional sublayer and the input features are weighted to obtain the output result of the first edge attention residual module.

[0051] In an optional embodiment, K is equal to 2;

[0052] The processing unit is further used to segment the shallow image features into forefoot image features and hindfoot image features; input the forefoot image features and the hindfoot image features into two second feature extraction modules respectively to obtain the forefoot deep features corresponding to the forefoot image features and the hindfoot deep features corresponding to the hindfoot image features;

[0053] The processing unit is specifically used to fuse the shallow image features, the deep features of the whole foot, the deep features of the forefoot and the deep features of the hindfoot to obtain the fused features.

[0054] In an optional embodiment, the first feature extraction module and the second feature extraction module both include M convolutional layers; the M convolutional layers include a fourth convolutional layer and a fifth convolutional layer connected in sequence; M is a positive integer greater than or equal to 2. The fourth convolutional layer includes at least one group of first edge attention residual modules and at least one group of second edge attention residual modules; the fifth convolutional layer includes multiple groups of first edge attention residual modules.

[0055] In an optional embodiment, before the processing unit performs feature fusion on the shallow image features, the whole foot deep features, the forefoot deep features and the hindfoot deep features,

[0056] The processing unit is also used to adjust the dimensions and sizes of the shallow image features, the deep features of the entire foot, the deep features of the forefoot and the deep features of the hindfoot, respectively, so that the dimensions and sizes of the shallow image features, the deep features of the entire foot, the deep features of the forefoot and the deep features of the hindfoot are the same.

[0057] In an optional embodiment, the processing unit is further used to perform image preprocessing on the footprint image to be identified; the image preprocessing includes one or more combinations of image grayscale adjustment processing, image background separation processing, image data enhancement processing and image scaling processing.

[0058] In an optional embodiment, the processing unit is specifically used to use a fusion model to perform feature fusion on shallow image features, full-foot deep features and K local deep features; wherein the fusion model includes a convolution module, a spatial attention module and a channel attention module connected in sequence; the convolution module is used to reduce the dimension of shallow image features, full-foot deep features or K local deep features; the spatial attention module is used to output the spatial attention weighted features corresponding to the shallow image features, full-foot deep features or K local deep features after dimensionality reduction; the channel attention module is used to obtain connection features, which are obtained by connecting the spatial attention weighted features corresponding to the shallow image features, full-foot deep features and K local deep features in the channel dimension; and the connection features are weighted according to the channel attention; the output result of the channel attention module is used to determine the fusion features.

[0059] The third aspect of the present invention provides a footprint recognition device, which includes a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, and the computer instructions are loaded and executed by the processor, so that the footprint recognition device implements the method described in the first aspect and any one of the embodiments of the first aspect.

[0060] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions; the computer instructions enable the computer to execute the method described in the first aspect and any one of the embodiments of the first aspect.

[0061] The footprint recognition method, device and storage medium provided by the present invention have the following beneficial effects compared with the prior art:

[0062] (1) The present invention improves the deep learning architecture. First, a shallow feature extraction model is used to extract shallow features of the footprint image to be identified, and shallow image features with richer texture information are obtained. Then, a deep feature extraction model is used to further extract the shallow image features, and local features are separated during the feature extraction process to obtain a full set of deep features and multiple local deep features. In this way, the force differences of different parts of the footprint can be refined, and the local texture shape features of the footprint can be captured more accurately, avoiding the problem of differential information loss that may occur when only the overall image is processed in the traditional way. The recognition accuracy is greatly improved.

[0063] (2) In the feature extraction process of the present invention, a new edge attention residual module is used for feature extraction, which can more accurately extract and enhance the edge texture features of the shoe print image. The new residual module selectively fuses the ordinary convolution features and the differential convolution features, enriching the shoe print edge texture features while avoiding the situation where a single feature is too strong, which helps to enhance the robustness and recognizability of the shoe print image features. At the same time, it also improves its robustness to lighting, visual changes, etc., further improving the accuracy and reliability of the recognition results.

[0064] (3) Before fusing multiple features, the present invention unifies the dimensions and sizes of the multiple features, which can restore the spatial characteristics of local features and avoid mixing of unrelated features. This ensures that the refined local deep features are retained, which is conducive to better fusion of diverse and complex features.

[0065] (4) The present invention uses a fusion model (Multi-CBAM module) modified by spatial attention (SAM) and channel attention (CAM) to perform multi-feature fusion, which can effectively integrate feature information at different levels and different parts. The multi-feature selective fusion method enables the system to maintain efficient recognition capabilities when processing diverse and complex footprint images, and improves the accuracy and robustness of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 A schematic diagram of a flow chart of a footprint recognition method provided by the present invention;

[0068] Figure 2 A system architecture diagram of a footprint recognition system provided by the present invention;

[0069] Figure 3 A schematic diagram of a strategy for integrating multiple types of feature sizes provided by the present invention;

[0070] Figure 4 A structural schematic diagram of a fusion model provided by the present invention;

[0071] Figure 5 A schematic diagram of the structure of a first edge attention residual module provided by the present invention;

[0072] Figure 6A schematic structural diagram of a footprint recognition device provided by the present invention;

[0073] Figure 7 A schematic structural diagram of a footprint recognition device provided by the present invention. DETAILED DESCRIPTION

[0074] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] Footprint recognition technology faces many limitations and challenges in today's complex crime scenarios. Traditional footprint recognition methods are limited by the limited information provided by a single footprint image, and are easily interfered by environmental factors. For example, factors such as light, perspective, and scale changes can cause the acquired image features to be unstable, leading to misjudgment.

[0076] In recent years, the footprint recognition method based on deep learning has significantly improved the accuracy and robustness of recognition and has become one of the current mainstream methods. However, the footprint recognition method based on deep learning still has many defects. For example:

[0077] 1. Insufficient local information extraction capability: Existing footprint recognition methods based on deep learning tend to take the entire footprint image as input. Although this can fully consider the information of the image, it is often unable to effectively extract local shoe print texture information when faced with complex scene backgrounds. This is because these methods lack targeted processing of local features and are easily affected by background interference or non-critical information. This will easily lead to inaccurate recognition due to the lack of local feature information.

[0078] 2. Interference from environmental factors: Perspective, illumination, and scale changes are the most common interference factors in obtaining footprint images from crime scenes. However, the convolution operation in existing deep learning is not sensitive to these changes, which will cause the pixel features under the same label to show differences, affecting the consistency and accuracy of recognition. Therefore, how to improve the accuracy and reliability of footprint recognition has become an urgent problem to be solved.

[0079] In response to these problems, the present invention proposes a footprint recognition method, device and storage medium. First, through an improved deep learning architecture, in the feature extraction process, a shallow feature extraction model is first used to extract shallow image features of the footprint image to be identified, and then a deep feature extraction model is used to obtain the full-foot deep features and local deep features of the footprint image to be identified. The deep feature extraction model can refine the force differences of different parts of the footprint, and perform local feature separation on the shallow image features instead of simple full-image processing. Finally, feature fusion is performed on the shallow image features, the full-foot deep features and the local deep features, and the fused features are used to perform footprint recognition. In this way, the feature information of different levels and different parts can be effectively integrated, the feature representation is effectively optimized, and the robustness and accuracy of footprint recognition in complex environments are improved.

[0080] Figure 1 A schematic diagram of a flow chart of a footprint recognition method provided in an embodiment of the present application. Figure 1 As shown, the footprint recognition method includes the following steps:

[0081] 101. Obtain a footprint image to be identified, and perform image preprocessing on the footprint image to be identified;

[0082] First, the footprint image to be identified is preprocessed; image preprocessing includes one or more combinations of image grayscale adjustment, image background separation, image data enhancement, and image scaling. For example, the footprint image to be identified can be converted into a grayscale image, and then background separation and data enhancement are performed. Finally, the grayscale image after background separation and data enhancement is adaptively scaled, such as cropping the grayscale image to a size of 384×144 to serve as the input of the shallow feature extraction model.

[0083] It is understandable that in the process of footprint recognition, it is necessary to ensure the consistency of the identified feature data and the data collected by the footprint library, and the image of 384×144 size meets the proportions of most footprints, and the image of this size, after passing through the shallow feature extraction model, the feature map aspect ratio obtained is 48:18, and after passing through the deep feature extraction model, the output global deep feature map aspect ratio will be 24:9, which can ensure that the deep features can be realized in a variety of refinement methods such as 2:1, 2 equal divisions, 3 equal divisions, and 6 equal divisions. If the size of the data collected by the footprint library changes, the cropping method can be adapted to it, and the embodiment of the present application does not limit it.

[0084] 102. Use a shallow feature extraction model to extract features from the preprocessed footprint image to be identified, so as to obtain shallow image features corresponding to the footprint image to be identified.

[0085] Figure 2 This is a system architecture diagram of the footprint recognition system provided in the embodiment of the present application. Figure 2 As shown, the preprocessed footprint image to be identified needs to be first input into a shallow feature extraction model for feature extraction to obtain shallow image features corresponding to the footprint image to be identified.

[0086] Optionally, the shallow feature extraction model includes N convolution layers. It uses N convolution layers to perform convolution operations on the footprint image to be identified, where N is a positive integer greater than or equal to 1. For example, Figure 2 As shown, N is equal to 3, that is, the shallow feature extraction model can be composed of the first convolution layer, the second convolution layer and the third convolution layer connected in sequence.

[0087] Among them, the input of the first convolution layer is the footprint image to be identified, and its output is the input of the second convolution layer; the output of the second convolution layer is the input of the third convolution layer. Finally, the third convolution layer completes the extraction of shallow features and outputs the final shallow image features.

[0088] The first convolution layer can be composed of a first convolution sublayer, a second convolution sublayer, and a pooling sublayer. The first convolution sublayer and the second convolution sublayer have different step sizes, and the output results of the first convolution sublayer and the second convolution sublayer are used to input into the pooling sublayer for downsampling. Exemplarily, the first convolution layer can use a set of 3×3 convolution sublayers with a step size of 1, 3×3 with a step size of 2, and 3×3 with a step size of 1 to extract features, and then send the extracted results to a 3×3 maximum pooling layer with a step size of 2 for downsampling to output the first image features.

[0089] The second convolutional layer inputs the first image feature output by the first convolutional layer, and outputs the second image feature. Exemplarily, the second convolutional layer may include multiple groups of first edge attention residual modules. Exemplarily, three groups of first edge attention residual modules may be used to extract features. The first edge attention residual module will be introduced in detail below, so it will not be described here.

[0090] The third convolutional layer is stacked by the first edge attention residual module and the second edge attention residual module with reduced resolution. Exemplarily, the third convolutional layer can extract features using a set of second edge attention residual modules with reduced resolution and three sets of first edge attention residual modules. Similarly, the second edge attention residual module will be introduced in detail below, so it will not be repeated here.

[0091] For example, the footprint image to be identified with a size of 384×144 is input into the first convolutional layer to the third convolutional layer for feature extraction in sequence, and the shallow image feature F is obtained. s , whose dimensions are 512×48×18.

[0092] 103. Input the shallow image features into the deep feature extraction model to obtain the full-foot deep features and K local deep features corresponding to the footprint image to be identified.

[0093] After obtaining the shallow image feature F s After that, it needs to be input into the deep feature extraction model for deep feature extraction. The deep feature extraction model includes a first feature extraction module for extracting the deep features of the whole foot and K second feature extraction modules for extracting local deep features.

[0094] The input of the first feature extraction module is the shallow image feature F s , and the input of the second feature extraction module is the shallow image feature F s The local image features obtained after segmentation. Wherein, the first feature extraction module and the second feature extraction module both include M convolution layers, where M is a positive integer greater than or equal to 2.

[0095] For example, Figure 2 As shown in the figure, the deep feature extraction model includes a first feature extraction module and two second feature extraction modules (forefoot feature extraction module and rearfoot feature extraction module). All modules include two convolutional layers, namely the fourth convolutional layer and the fifth convolutional layer connected in sequence. Among them, the first feature extraction module is used to output the deep features of the whole foot based on the shallow image features. Among them, the shallow image feature F with a dimension of 512×48×18 s After inputting into the first feature extraction module, the deep feature F0 of the whole foot feature of 2048×24×9 is obtained. s After a 2:1 segmentation according to the feature map height dimension, a 512×32×18 forefoot image feature and a 512×16×18 hindfoot image feature are obtained. Then, the forefoot image feature is input into the forefoot feature extraction module to obtain a 2048×16×9 forefoot deep feature F1, and the hindfoot image feature is input into the hindfoot feature extraction module to obtain a 2048×8×9 hindfoot deep feature F2.

[0096] Among them, the fourth convolutional layer can be composed of at least one group of first edge attention residual modules and at least one group of second edge attention residual modules with reduced resolution, and the fifth convolutional layer can be composed of multiple groups of first edge attention residual modules. Exemplarily, the fourth convolutional layer can use 1 group of second edge attention residual modules with reduced resolution and 5 groups of first edge attention residual modules to extract features. The fifth convolutional layer can use 3 groups of first edge attention residual modules to extract features. Similarly, the first edge attention residual module and the second edge attention residual module will be introduced in detail below, so they will not be repeated here.

[0097] It can be understood that this embodiment improves the deep learning architecture. First, the shallow feature extraction model is used to extract shallow features of the footprint image to be identified, and shallow image features with richer texture information are obtained; then the deep feature extraction model is used to further extract the shallow image features, and local features are separated during the feature extraction process to obtain a full set of deep features and multiple local deep features. In this way, the force differences of different parts of the footprint can be refined, and the local texture shape features of the footprint can be captured more accurately, avoiding the problem of differential information loss that may occur when only the overall image is processed in the traditional way. The accuracy of recognition is greatly improved.

[0098] 104. Feature fusion is performed on the shallow image features, the full foot deep features, and K local deep features to obtain fused features.

[0099] like Figure 2 As shown, the shallow image feature F s , full foot deep feature F0, K local deep features F1~F K (Forefoot depth features and hindfoot depth features) are input into the multi-feature fusion module for feature fusion. Before fusion, in order to restore the spatial characteristics of local features and avoid the mixing of unrelated features to ensure that the refined local deep features are preserved, it is necessary to process the dimensions and sizes of multiple types of features to obtain features of the same size E1~E g , where g=K+2. This is conducive to better integration of diverse and complex features. It can be understood that in the above example, K is 2, where the two local deep features are the forefoot depth feature and the hindfoot depth feature.

[0100] Exemplarily, the integration process of multiple feature dimensions and sizes can be:

[0101] First, the full-foot feature F0 is not processed and recorded as feature E1, whose dimension is 2048×24×9.

[0102] Then, the shallow image feature F of 512×48×18 s A 1×1 ordinary convolution with a channel number of 2048 is used to increase the dimension, and a maximum feature pooling layer with a stride of 2 and a convolution kernel of 2×2 is used for downsampling to obtain feature E2 of the same size as E1.

[0103] Next, the 2048×16×9 forefoot deep feature F1 is filled with blanks in the hindfoot part to obtain feature E3 of the same size as E1.

[0104] Finally, the 2048×8×9 hindfoot deep feature F2 is used to fill the forefoot part with blanks, and the feature E3 of the same size as E1 is obtained. As you can understand, the integration process can be referred to Figure 3 The structure diagram is shown.

[0105] Then, E1~E4 of the same size are sent to the multi-input convolutional block attention module (Multi-CBAM), where the fusion model includes multiple convolutional modules, multiple spatial attention modules SAM and channel attention modules CAM. Among them, multiple spatial attention modules SAM are used to calculate the relationship between each pixel in the input feature and perform weighting, while the channel attention module CAM calculates the relationship between channels and performs weighting.

[0106] like Figure 4 As shown in the figure, first, multiple features after size integration are input into the Multi-CBAM module to obtain the fused attention feature E with channel dimension C, and then E is subjected to global average pooling and BN layer operations to obtain the fused feature F.

[0107] Specifically, features E1~E4 are reduced in dimension through 1×1 ordinary convolution, reducing the channel dimension from C=2048 to c=C / g=512, and then the SAM module is used to independently perform attention weighting on each feature in the pixel space dimension to obtain spatial attention features S1~S k .

[0108] Then, the spatial attention features S1~S k After connecting in the channel dimension and obtaining the connection feature with the dimension of 2048×24×9, it is sent to the CAM module. Different features are screened by using the characteristics of channel attention, and the weight of useful features is increased to obtain the fused attention feature E with the channel dimension C=2048, which has the dimension of 2048×24×9.

[0109] It can be understood that this embodiment uses the Multi-CBAM module transformed by spatial attention (SAM) and channel attention (CAM) to perform multi-feature fusion, which can effectively integrate feature information of different levels and different parts. The multi-feature selective fusion method enables the system to maintain efficient recognition capabilities when processing diverse and complex footprint images, and improves the accuracy and robustness of recognition.

[0110] 105. Perform footprint recognition on the footprint image to be recognized based on the fused features.

[0111] The final fused feature F with a dimension of 2048×216 is compared with the feature database to obtain the person whose footprint is most similar to the initial input footprint image and their similarity.

[0112] The first edge attention residual module and the second edge attention residual module are described in detail below. Figure 5, the first edge attention residual module includes a third convolution sublayer, a fourth convolution sublayer, a fifth convolution sublayer, a connection sublayer and a sixth convolution sublayer. Specifically, the feature extraction process of the first edge attention residual module is:

[0113] First, the input feature C0 is sent to the third convolutional sublayer. The stride of the third convolutional sublayer is 1 and the convolution kernel size is 1×1. Therefore, the input feature C0 is extracted through a 1×1 ordinary convolution to obtain feature C1.

[0114] Next, feature C1 is grouped in the channel dimension and feature C1 is divided into feature C 21 and C 22 . It can be understood that this embodiment takes the first edge attention residual module including a convolution sublayer and a differential convolution sublayer as an example to obtain the residual features of the input features. Therefore, feature C1 needs to be divided into two groups in the channel dimension. The first edge attention residual module can also include a differential convolution sublayer, so that the number of groupings of feature C1 in the channel dimension needs to be determined by the number of convolution sublayers and differential convolution sublayers, and there is no limitation on this.

[0115] In the above example, C 21 It is sent to the fourth convolutional sublayer, which is a 3×3 ordinary convolution with a stride of 1, and the ordinary feature R1 is obtained. 22 It is sent to the fifth convolution sublayer, that is, a 3×3 differential convolution with a step size of 1 to obtain the feature R2 with edge information.

[0116] Next, R1 and R2 are sent to the connection sublayer. Specifically, R1 and R2 are sent to the Split-Attention layer to be connected and selectively fused using the channel mapping relationship to obtain the edge information selective fusion feature C3.

[0117] Then, C3 is operated by the sixth convolution sublayer, where the step size of the sixth convolution sublayer is 1 and the convolution kernel size is 1×1. That is, through a layer of 1×1 ordinary convolution, the attention feature C4 with edge feature attention weighting is obtained.

[0118] Finally, the attention feature C4 and the input feature C0 are weighted to obtain the output result of the first edge attention residual module, namely, the residual feature C5.

[0119] It can be understood that in the feature extraction process of this embodiment, a new edge attention residual module is used for feature extraction, which can more accurately extract and enhance the edge texture features of the shoe print image. The new residual module selectively fuses the ordinary convolution features and the differential convolution features, enriching the shoe print edge texture features while avoiding the situation where a single feature is too strong, which helps to enhance the robustness and recognizability of the shoe print image features. At the same time, it also improves its robustness to lighting, visual changes, etc., further improving the accuracy and reliability of the recognition results.

[0120] It can be understood that the structure of the second edge attention residual module is similar to that of the first edge attention residual module, and the second edge attention residual module also includes a third convolution sublayer, a fourth convolution sublayer, a fifth convolution sublayer, a connection sublayer and a sixth convolution sublayer. Specifically, the feature extraction process of the second edge attention residual module is:

[0121] First, the input feature C0 is sent to the third convolutional sublayer. The stride of the third convolutional sublayer is 1 and the convolution kernel size is 1×1. Therefore, the input feature C0 is extracted through a 1×1 ordinary convolution to obtain feature C1.

[0122] Next, similarly, feature C1 is grouped in the channel dimension. Similarly, this embodiment takes the second edge attention residual module including a convolution sublayer and a differential convolution sublayer as an example to obtain the residual feature of the input feature. Therefore, feature C1 is divided into C 21 and C 22。 The second edge attention residual module can also include differential convolution sublayers, so the number of groupings of feature C1 in the channel dimension needs to be determined by the number of convolution sublayers and differential convolution sublayers, and there is no limitation on this.

[0123] In this way, C 21 It is sent to the fourth convolutional sublayer, which is a 3×3 ordinary convolution with a stride of 2, and the ordinary feature R1 is obtained. 22 It is sent to the fifth convolution sublayer, that is, a 3×3 differential convolution with a step size of 2 to obtain the feature R2 with edge information.

[0124] Next, R1 and R2 are sent to the connection sublayer. Specifically, R1 and R2 are sent to the Split-Attention layer to be connected and selectively fused using the channel mapping relationship to obtain the edge information selective fusion feature C3.

[0125] Then, the sixth convolution sublayer is used to operate C3, where the step size of the sixth convolution sublayer is 1 and the convolution kernel size is 1×1. That is, through a layer of 1×1 ordinary convolution, the attention feature C4 with edge feature attention weighting is obtained.

[0126] Finally, the attention feature C4 and the input feature C0 are weighted to obtain the output result of the first edge attention residual module, namely, the residual feature C5.

[0127] Based on the above description, Figure 6 This is a schematic diagram of the structure of a footprint recognition device provided in an embodiment of the present application. Figure 6 As shown, the device comprises:

[0128] An acquisition unit 601 is used to acquire a footprint image to be identified;

[0129] The processing unit 602 is used to perform feature extraction on the footprint image to be identified using a shallow feature extraction model to obtain shallow image features corresponding to the footprint image to be identified; wherein the shallow feature extraction model uses N convolution layers to perform convolution operations on the footprint image to be identified; N is a positive integer greater than or equal to 1;

[0130] The processing unit 602 is further used to input the shallow image features into the deep feature extraction model to obtain the full foot deep features and K local deep features corresponding to the footprint image to be identified; wherein the deep feature extraction model includes a first feature extraction module and K second feature extraction modules; the first feature extraction module is used to determine the full foot deep features according to the shallow image features; any one of the K second feature extraction modules is used to determine a local deep feature according to a local image feature; the local image features are obtained by segmenting the shallow image features; K is a positive integer greater than or equal to 1;

[0131] The processing unit 602 is further used to perform feature fusion on the shallow image features, the full foot deep features, and the K local deep features to obtain fused features;

[0132] The recognition unit 603 is used to perform footprint recognition on the footprint image to be recognized according to the fusion features.

[0133] In an optional embodiment, the shallow feature extraction model includes a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in sequence; wherein:

[0134] The first convolution layer is used to input the footprint image to be identified and output the first image feature; the first convolution layer includes a first convolution sublayer, a second convolution sublayer and a pooling sublayer; the first convolution sublayer and the second convolution sublayer have different step sizes; the pooling sublayer is used to downsample the output results of the first convolution sublayer and the second convolution sublayer to obtain the first image feature;

[0135] The second convolutional layer is used to input the first image feature and output the second image feature; the second convolutional layer includes multiple groups of first edge attention residual modules;

[0136] The third convolutional layer is used to input the second image features and output shallow image features; the third convolutional layer includes multiple groups of first edge attention residual modules and second edge attention residual modules; the resolutions of the first edge attention residual modules and the second edge attention residual modules are different.

[0137] In an optional embodiment, the first edge attention residual module includes a third convolution sublayer, a fourth convolution sublayer, a fifth convolution sublayer, a connection sublayer and a sixth convolution sublayer; the first edge attention residual module is used to:

[0138] Get input features;

[0139] The input features are convolved through the third convolution sublayer; the step size of the third convolution sublayer is 1, and the convolution kernel size of the third convolution sublayer is 1×1;

[0140] The output results of the third convolution sublayer are grouped in the channel dimension, and the grouped output results of the third convolution sublayer are input to the fourth convolution sublayer and the fifth convolution sublayer respectively; the fourth convolution sublayer is used for convolution operation, the step size of the fourth convolution sublayer is 1 and the convolution kernel size of the fourth convolution sublayer is 3×3; the fifth convolution sublayer is used for differential convolution operation, the step size of the fifth convolution sublayer is 1 and the convolution kernel size of the fifth convolution sublayer is 3×3;

[0141] The output results of the fourth convolutional sublayer and the fifth convolutional sublayer are fused using the connection sublayer;

[0142] The fusion result output by the connection sublayer is calculated using the sixth convolution sublayer; the step size of the sixth convolution sublayer is 1 and the convolution kernel size of the sixth convolution sublayer is 1×1;

[0143] The output result of the sixth convolutional sublayer and the input features are weighted to obtain the output result of the first edge attention residual module.

[0144] In an optional embodiment, K is equal to 2;

[0145] The processing unit 602 is further used to segment the shallow image features into forefoot image features and hindfoot image features; input the forefoot image features and the hindfoot image features into two second feature extraction modules respectively to obtain the forefoot deep features corresponding to the forefoot image features and the hindfoot deep features corresponding to the hindfoot image features;

[0146] The processing unit 602 is specifically used to perform feature fusion on the shallow image features, the whole foot deep features, the forefoot deep features and the hindfoot deep features to obtain fused features.

[0147] In an optional embodiment, the first feature extraction module and the second feature extraction module both include M convolutional layers; the M convolutional layers include a fourth convolutional layer and a fifth convolutional layer connected in sequence; M is a positive integer greater than or equal to 2; the fourth convolutional layer includes at least one group of first edge attention residual modules and at least one group of second edge attention residual modules; the fifth convolutional layer includes multiple groups of first edge attention residual modules.

[0148] In an optional embodiment, before the processing unit 502 performs feature fusion on the shallow image features, the whole foot deep features, the forefoot deep features and the rear foot deep features,

[0149] The processing unit 602 is also used to adjust the dimensions and sizes of the shallow image features, the deep features of the entire foot, the deep features of the forefoot and the deep features of the hindfoot, respectively, so that the dimensions and sizes of the shallow image features, the deep features of the entire foot, the deep features of the forefoot and the deep features of the hindfoot are the same.

[0150] In an optional embodiment, the processing unit 602 is further used to perform image preprocessing on the footprint image to be identified; the image preprocessing includes one or more combinations of image grayscale adjustment processing, image background separation processing, image data enhancement processing and image scaling processing.

[0151] In an optional embodiment, the processing unit 602 is specifically used to use a fusion model to perform feature fusion on shallow image features, full-foot deep features and K local deep features; wherein the fusion model includes a convolution module, a spatial attention module and a channel attention module connected in sequence; the convolution module is used to reduce the dimension of shallow image features, full-foot deep features or K local deep features; the spatial attention module is used to output the spatial attention weighted features corresponding to the shallow image features, full-foot deep features or K local deep features after dimensionality reduction; the channel attention module is used to obtain connection features, which are obtained by connecting the spatial attention weighted features corresponding to the shallow image features, full-foot deep features and K local deep features in the channel dimension; and the connection features are weighted according to the channel attention; the output result of the channel attention module is used to determine the fusion features.

[0152] See also Figure 7 The embodiment of the present application also shows a footprint recognition device 1000, which may include one or more central processors 1001 and a communication interface 1003. Optionally, a memory 1002 may also be included.

[0153] The memory 1002 may be a temporary storage or a permanent storage. Furthermore, the central processor 1001 may be configured to communicate with the memory 1002 and execute a series of instruction operations in the memory 1002 on the footprint recognition device.

[0154] In this embodiment, the CPU 1001 can execute the aforementioned Figures 1 to 5 The steps performed in the illustrated embodiment will not be described in detail here.

[0155] In this embodiment, the specific functional module division in the central processing unit 1001 can be the same as the above Figure 6 The division method of each module described in is similar and will not be repeated here.

[0156] It should be noted that the information execution process of the central processor 1001 of the footprint recognition device 1000 and other contents can be found in the aforementioned Figures 1 to 5 The description in the method embodiment shown will not be repeated here.

[0157] The present invention also provides a computer-readable storage medium, which stores computer instructions; the computer instructions enable the computer to execute the above Figures 1 to 5 The steps performed in any one of the embodiments shown.

[0158] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A footprint recognition method, characterized in that: The footprint recognition method comprises: Obtaining a footprint image to be identified; Using a shallow feature extraction model to extract features from the footprint image to be identified, so as to obtain shallow image features corresponding to the footprint image to be identified; wherein the shallow feature extraction model uses N convolution layers to perform convolution operations on the footprint image to be identified; N is a positive integer greater than or equal to 1; The shallow image features are input into the deep feature extraction model to obtain the full-foot deep features and K local deep features corresponding to the footprint image to be identified; wherein the deep feature extraction model includes a first feature extraction module and K second feature extraction modules; the first feature extraction module is used to determine the full-foot deep features according to the shallow image features; any one of the K second feature extraction modules is used to determine a local deep feature according to a local image feature; the local image features are obtained by segmenting the shallow image features; K is a positive integer greater than or equal to 1; Performing feature fusion on the shallow image features, the full foot deep features and the K local deep features to obtain fused features; Performing footprint recognition on the footprint image to be recognized according to the fusion feature; The shallow feature extraction model includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence; wherein: The first convolution layer is used to input the footprint image to be identified and output the first image feature; the first convolution layer includes a first convolution sublayer, a second convolution sublayer and a pooling sublayer; the first convolution sublayer and the second convolution sublayer have different step sizes; the pooling sublayer is used to downsample the output results of the first convolution sublayer and the second convolution sublayer to obtain the first image feature; The second convolutional layer is used to input the first image feature and output the second image feature; the second convolutional layer includes multiple groups of first edge attention residual modules; The third convolutional layer is used to input the second image feature and output the shallow image feature; the third convolutional layer includes multiple groups of the first edge attention residual modules and the second edge attention residual modules; the first edge attention residual modules and the second edge attention residual modules have different resolutions; The first edge attention residual module includes a third convolution sublayer, a fourth convolution sublayer, a fifth convolution sublayer, a connection sublayer and a sixth convolution sublayer; the first edge attention residual module is used to: Get input features; Performing a convolution operation on the input feature through a third convolution sublayer; the step size of the third convolution sublayer is 1, and the convolution kernel size of the third convolution sublayer is 1×1; The output results of the third convolution sublayer are grouped in the channel dimension, and the grouped output results of the third convolution sublayer are respectively input to the fourth convolution sublayer and the fifth convolution sublayer; the fourth convolution sublayer is used to perform a convolution operation, the step size of the fourth convolution sublayer is 1, and the convolution kernel size of the fourth convolution sublayer is 3×3; the fifth convolution sublayer is used to perform a differential convolution operation, the step size of the fifth convolution sublayer is 1, and the convolution kernel size of the fifth convolution sublayer is 3×3; Using the connection sublayer to fuse the output results of the fourth convolution sublayer and the fifth convolution sublayer; The fusion result output by the connection sublayer is calculated by using the sixth convolution sublayer; the step size of the sixth convolution sublayer is 1 and the convolution kernel size of the sixth convolution sublayer is 1×1; Performing a weighted operation on the output result of the sixth convolutional sublayer and the input feature to obtain an output result of the first edge attention residual module; K is equal to 2; the method further comprises: Segmenting the shallow image features into forefoot image features and hindfoot image features; Inputting the forefoot image feature and the hindfoot image feature into two second feature extraction modules respectively to obtain the forefoot deep feature corresponding to the forefoot image feature and the hindfoot deep feature corresponding to the hindfoot image feature; The step of fusing the shallow image features, the full foot deep features, and the K local deep features to obtain fused features includes: The shallow image features, the whole foot deep features, the forefoot deep features and the hindfoot deep features are fused to obtain the fused features.

2. The method according to claim 1, characterized in that The first feature extraction module and the second feature extraction module both include M convolutional layers; the M convolutional layers include a fourth convolutional layer and a fifth convolutional layer connected in sequence; M is a positive integer greater than or equal to 2; Among them, the fourth convolutional layer includes at least one group of first edge attention residual modules and at least one group of second edge attention residual modules; the fifth convolutional layer includes multiple groups of the first edge attention residual modules.

3. The method according to claim 2, characterized in that Before fusing the shallow image features, the whole foot deep features, the forefoot deep features, and the hindfoot deep features, the method further includes: The dimensions and sizes of the shallow image features, the full foot deep features, the forefoot deep features and the hindfoot deep features are adjusted respectively so that the dimensions and sizes of the shallow image features, the full foot deep features, the forefoot deep features and the hindfoot deep features are the same.

4. The method according to claim 3, characterized in that The method further comprises: The footprint image to be identified is subjected to image preprocessing; the image preprocessing includes one or a combination of image grayscale adjustment processing, image background separation processing, image data enhancement processing and image scaling processing.

5. The method according to any one of claims 1 to 4, characterized in that: The step of fusing the shallow image features, the full foot deep features and the K local deep features to obtain fused features includes: A fusion model is used to perform feature fusion on the shallow image features, the full-foot deep features and the K local deep features; wherein the fusion model includes a convolution module, a spatial attention module and a channel attention module connected in sequence; the convolution module is used to reduce the dimension of the shallow image features, the full-foot deep features or the K local deep features; the spatial attention module is used to output the spatial attention weighted features corresponding to the shallow image features, the full-foot deep features or the K local deep features after dimension reduction; the channel attention module is used to obtain connection features, which are obtained by connecting the spatial attention weighted features corresponding to the shallow image features, the full-foot deep features and the K local deep features in the channel dimension; and the connection features are weighted according to the channel attention; the output result of the channel attention module is used to determine the fusion features.

6. A footprint recognition device, characterized in that: The footprint recognition device includes a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, and the computer instructions are loaded and executed by the processor so that the footprint recognition device implements the method described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions; the computer instructions enable the computer to execute the method according to any one of claims 1 to 5.

Images