Human biometrics estimation method from footprint images based on multi-task interactive enhancement

Abstract

The present invention relates to a method for estimating human biometrics from footprint images based on multi-task interactive enhancement. Compared with the prior art, this method solves the drawbacks of single-prediction and multi-task conflicts when predicting human biometrics from footprints. The present invention includes the following steps: collecting and preprocessing a dataset; constructing a footprint multi-task interactively enhanced biometric estimation network; training the footprint multi-task interactively enhanced biometric estimation network; acquiring and preprocessing the footprint image to be estimated; and obtaining the human biometric estimation results from the footprint image. The footprint multi-task interactively enhanced biometric estimation network involved in the present invention not only has strong feature expression and prediction capabilities, but also solves the problem of conflicts among multi-task training tasks to a certain extent, thereby improving the accuracy of footprint biometric prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of footprint image analysis, and in particular to a method for estimating human biometric features from footprint images based on multi-task interactive enhancement. Background Art

[0002] Biometric recognition technology is used in crime detection, medical monitoring, and social services. Footprints, as important biometric features, are specific and relatively stable, making them extremely valuable in determining individual identity. Footprint features extracted from footprint images can be used for both identity recognition and biometric prediction.

[0003] Methods for predicting human attributes from footprints can be categorized into traditional manual measurement and deep learning model estimation. Traditional algorithms for predicting human attributes from footprints are based on manually defined footprint features. These algorithms extract these features and establish regression equations linking them to the target human attributes. However, manually defined features are subjective, and the resulting regression equations have poor generalization capabilities. Deep learning models automatically extract features from footprint images and use these features to predict human attributes. Deep learning methods make it easier to obtain footprint features and improve prediction accuracy. However, most of these methods predict only one or two of the following features, such as height, age, and weight, from the footprint.

[0004] Since footprint trace information comes from a person's overall behavior and has a specific connection with biological characteristics such as height, body shape, gender, age and walking habits, how to effectively model a person's multi-dimensional biological characteristics through footprint images has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The purpose of the present invention is to solve the defects of the existing technology in predicting human biometrics through footprints, which is based on single prediction and multi-task conflict. A method for estimating human biometrics from footprint images based on multi-task interaction enhancement is provided to solve the above problems.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] A method for estimating human biometric features from footprint images based on multi-task interactive enhancement includes the following steps:

[0008] Dataset collection and preprocessing: An optical footprint collector was used to collect ten barefoot images of each person, each with their corresponding labels. The labels contained biometric information corresponding to the left and right barefoot images of each person, forming the raw dataset. Preprocessing was performed on the raw dataset, including inverting the grayscale footprint images, removing the scale from the images, rotating the footprints according to their circumscribed rectangles, moving them to the center of the image, and zero-padding the images to 3136 x 3136 pixels. Outliers were removed, resulting in a total of W barefoot images of Z people.

[0009] Constructing a footprint multi-task interaction enhanced biometric feature estimation network: Constructing a footprint multi-task interaction enhanced biometric feature estimation network based on the contour feature extraction branch, the multi-task interaction branch, the task feature enhancement module and the task output head;

[0010] Training of the footprint multi-task interaction enhanced biometric estimation network: The pre-processed collected dataset is input into the footprint multi-task interaction enhanced biometric estimation network for training;

[0011] Acquisition and preprocessing of the footprint image to be estimated: Acquire the barefoot image to be estimated and preprocess it;

[0012] Obtaining the human biometric estimation results of footprint images: The preprocessed barefoot image to be estimated is input into the trained footprint multi-task interactive enhanced biometric estimation network to obtain the human biometric estimation results of the footprint image.

[0013] The construction of the footprint multi-task interactive enhanced biometric feature estimation network includes the following steps:

[0014] The footprint multi-task interaction enhanced biometric feature estimation network is set to include a contour feature extraction branch, a multi-task interaction branch, a task feature enhancement module and a task output head;

[0015] Construct contour feature extraction branch:

[0016] The Laplacian convolution kernel with stride=1, 2, and 4 is used to extract the initial footprint contour images of different sizes. The size is unified to 224×224 by upsampling. The multi-scale contour images are fused into a single-channel image using 1×1 convolution. The image is binarized with a threshold of X=0.1 to obtain the footprint contour image.

[0017] Six convolutional layers are set to extract contour features from the footprint contour map. The convolution kernel size of each convolutional layer is 3×3. Every two convolutional layers are followed by a four-fold downsampling layer. The number of channels of the six convolutional layers are 128, 128, 512, 512, 1024, and 1024 respectively.

[0018] The contour features extracted by 6 convolutional layers are represented as fcnt , which will be used as the input of the task feature enhancement module later;

[0019] Construct multi-task interactive branches:

[0020] Setting the multi-task interaction branch includes a shared trunk branch and a task interaction module;

[0021] The shared backbone consists of 7 convolutional layers, each with a 3×3 kernel size and 64, 64, 128, 128, 256, 256, and 256 channels respectively. Every two convolutional layers in the first four convolutional layers are followed by a maximum pooling layer with a stride of 2, and the last three convolutional layers are also followed by a maximum pooling layer with a stride of 2.

[0022] The task interaction module consists of four task branches. Each branch selects features of interest to each task from shared features using the SAM spatial attention mechanism. The four task branches interact with each other through a multi-head self-attention mechanism. Six convolutional layers with a kernel size of 3×3 and 512 channels are then used to extract deep features after interaction. Every three convolutional layers are followed by a maximum pooling layer with a stride of 2.

[0023] Construct task feature enhancement module:

[0024] The shared backbone features of the multi-task interaction branch are orthogonally fused with the features extracted by the four task branches of its task interaction module. The orthogonally fused features are then orthogonally fused with the feature vectors of the contour feature extraction branch to obtain task-specific features after four task feature enhancements, namely age task, gender task, height task, and weight task.

[0025] Construct the task output header:

[0026] The feature vector after enhancing each task feature is passed through the fully connected layer to obtain the prediction result. The output dimension of the fully connected layer for the age task is 100, the output dimension of the fully connected layer for the gender task is 2, and the output dimension of the fully connected layer for the height task and weight task is 1.

[0027] The training of the footprint multi-task interaction enhanced biometric feature estimation network includes the following steps:

[0028] The preprocessed data set is input into the contour feature extraction branch, and the contour feature f is output through the contour feature extraction branch. cnt ;

[0029] The preprocessed data set is input into the shared trunk branch, which adaptively selects shared features through the SAM attention module. The extracted shared features f S , Task T iThe feature weights learned by the SAM attention module are w i , Task T i For gender, age, height, and weight prediction tasks, the task-specific features after feature selection are flattened in the spatial dimension, and the single task feature is represented as:

[0030]

[0031] Among them, Reshape() is a flattening operation;

[0032] The task interaction module uses multi-head self-attention MHSA and a small multi-layer perceptron sMLP to interact between tasks, Splicing in the spatial dimension to get f T , as Q, K, V respectively, the expression of feature interaction through multi-head self-attention and small multi-layer perceptron is as follows:

[0033]

[0034]

[0035] Among them, Concat() is feature concatenation, is the splicing feature after interaction;

[0036] Splicing features after interaction After splitting in the spatial dimension and restoring the features of each task to the original feature map size, the output of the task interaction module is as follows:

[0037]

[0038] Among them, Split() is feature splitting, Reshape() is flattening operation, and i is one of the four tasks;

[0039] The task feature enhancement module converts the adjusted shared features f′ into S and task characteristics Orthogonal fusion: shared features f S After adjusting the number of channels through 1*1 convolution, f′ is obtained through the pooling layer. S , task characteristics after interaction After 6 convolution layers with a kernel size of 3 and a channel number of 512, each of which is followed by a maximum pooling layer with two times downsampling, and finally a fully connected layer, the network is obtained. The orthogonal fusion expression is as follows:

[0040]

[0041] in, is the task-specific feature of the first orthogonal fusion, F orth It is an orthogonal fusion algorithm;

[0042] F orth (f1, f2)=FC(Concat((f1-proj(f1, f2)), f2))

[0043]

[0044] In the formula, FC is the fully connected layer, Concat() is the feature concatenation, f1 and f2 are the two input vectors of orthogonal fusion, and F proj Project vector f1 onto vector f1;

[0045] The contour feature f cnt Task characteristics with the first orthogonal fusion Fuse again, and the second orthogonal fusion is as follows:

[0046]

[0047] in, It is the task characteristic feature of the second orthogonal fusion;

[0048] In the task output header, the output of each task feature enhancement module The final prediction result is obtained through the task output head; the age task is passed through the fully connected layer and softmax layer with a dimension of 100 to obtain the predicted age probability distribution D pre =[a1, a2, ..., a n ], the predicted age value obtained according to the expectation formula

[0049]

[0050] The gender task passes through a fully connected layer and a softmax layer with a dimension of 2, and the one with the highest prediction probability is taken as the prediction result;

[0051] The output dimension of the fully connected layer for both the height and weight tasks is 1, and the predicted value is obtained by multiplying it by 255;

[0052] Set the training end condition to S epochs to end the training process.

[0053] Beneficial effects

[0054] Compared with the prior art, the human biometric feature estimation method for footprint images based on multi-task interaction enhancement of the present invention has a contour feature extraction branch that enhances the feature representation capability and a multi-task interaction enhancement branch that strengthens the interaction between tasks, thereby solving the problem of multi-task conflict to a certain extent.

[0055] The footprint multi-task interactive enhanced biometric feature estimation network involved in the present invention not only has strong feature expression and prediction capabilities, but also solves the problem of conflict between multi-task training tasks to a certain extent, thereby improving the accuracy of footprint biometric feature prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a method sequence diagram of the present invention;

[0057] Figure 2 This is a fitting curve diagram of the age regression task using the method of the present invention;

[0058] Figure 3 This is a fitted curve diagram of the height regression task using the method of the present invention;

[0059] Figure 4 The figure is a fitted curve diagram of the weight regression task using the method of the present invention. DETAILED DESCRIPTION

[0060] In order to provide a further understanding and appreciation of the structural features and effects achieved by the present invention, a detailed description is provided with reference to preferred embodiments and accompanying drawings as follows:

[0061] like Figure 1 As shown, the method for estimating human biometric features from footprint images based on multi-task interactive enhancement of the present invention comprises the following steps:

[0062] The first step is to collect and preprocess the dataset: an optical footprint collector is used to collect ten barefoot images of each person, each with their corresponding labels. The labels contain the biometric information corresponding to each person's barefoot images, forming the original dataset. The collected raw dataset is then preprocessed, including inverting the grayscale footprint images, removing the scale from the image, rotating the footprints according to their circumscribed rectangle, moving them to the center of the image, and unifying the image to 3136*3136 pixels through zero padding. Abnormal samples are removed, resulting in a total of W processed barefoot images of Z people.

[0063] The second step is to construct a footprint multi-task interaction enhanced biometric feature estimation network: construct a footprint multi-task interaction enhanced biometric feature estimation network based on the contour feature extraction branch, multi-task interaction branch, task feature enhancement module and task output head.

[0064] The contour feature extraction branch uses the Laplacian convolution kernel to extract the contour map of the footprint, which does not significantly increase the number of network parameters while enabling the network to learn supplementary information about the contour; the soft parameter sharing structure and task interaction module in the multi-task interaction branch enable the network to learn four tasks simultaneously and alleviate conflicts between tasks; the age task in the task output head alleviates the label ambiguity problem by predicting the age probability distribution.

[0065] The specific steps are as follows:

[0066] (1) The footprint multi-task interaction enhanced biometric feature estimation network is set to include a contour feature extraction branch, a multi-task interaction branch, a task feature enhancement module and a task output head.

[0067] (2) Constructing contour feature extraction branch:

[0068] The Laplacian convolution kernel with stride=1, 2, and 4 is used to extract the initial footprint contour images of different sizes. The size is unified to 224×224 by upsampling. The multi-scale contour images are fused into a single-channel image using 1×1 convolution. The image is binarized with a threshold of X=0.1 to obtain the footprint contour image.

[0069] Six convolutional layers are set to extract contour features from the footprint contour map. The convolution kernel size of each convolutional layer is 3×3. Every two convolutional layers are followed by a four-fold downsampling layer. The number of channels of the six convolutional layers are 128, 128, 512, 512, 1024, and 1024 respectively.

[0070] The contour features extracted by 6 convolutional layers are represented as f cnt , which will be used as the input of the task feature enhancement module later; the Laplace operator used is:

[0071]

[0072] (3) Constructing a multi-task interaction branch: This branch adopts a soft-sharing structure, with a shared backbone and four task branches. The shallow layer of the network is the shared backbone, which is used to extract shared features of the four tasks. The deep layer of the network sets a feature extraction backbone for each task, which is used to extract higher-level semantic information of each task. In order to reduce the mutual conflict between tasks and strengthen the interaction of useful information between tasks, a task interaction module based on the attention mechanism and a task feature enhancement module based on orthogonal fusion are set up.

[0073] Setting the multi-task interaction branch includes a shared trunk branch and a task interaction module;

[0074] The shared backbone consists of 7 convolutional layers, each with a 3×3 kernel size and 64, 64, 128, 128, 256, 256, and 256 channels respectively. Every two convolutional layers in the first four convolutional layers are followed by a maximum pooling layer with a stride of 2, and the last three convolutional layers are also followed by a maximum pooling layer with a stride of 2.

[0075] The task interaction module consists of four task branches. Each branch selects features of interest to each task from shared features using the SAM spatial attention mechanism. The four task branches interact with each other through a multi-head self-attention mechanism. Six convolutional layers with a kernel size of 3×3 and 512 channels are then used to extract deep features after interaction. Every three convolutional layers are followed by a maximum pooling layer with a stride of 2.

[0076] (4) Constructing task feature enhancement module:

[0077] The shared backbone features of the multi-task interaction branch are orthogonally fused with the features extracted by the four task branches of its task interaction module. The orthogonally fused features are then orthogonally fused with the feature vectors of the contour feature extraction branch to obtain task-specific features after four task feature enhancements, namely age task, gender task, height task and weight task.

[0078] (5) Construct the task output header:

[0079] The feature vector after enhancing each task feature is passed through the fully connected layer to obtain the prediction result. The output dimension of the fully connected layer for the age task is 100, the output dimension of the fully connected layer for the gender task is 2, and the output dimension of the fully connected layer for the height task and weight task is 1.

[0080] The third step is to train the footprint multi-task interactive enhanced biometric estimation network: the pre-processed data set is input into the footprint multi-task interactive enhanced biometric estimation network for training. The specific steps are as follows:

[0081] (1) Input the preprocessed data set into the contour feature extraction branch, and output the contour feature f through the contour feature extraction branch cnt ;

[0082] (2) The preprocessed data set is input into the shared trunk branch, which adaptively selects shared features through the SAM attention module. The extracted shared features f S , Task T i The feature weights learned by the SAM attention module are w i , Task T iFor gender, age, height, and weight prediction tasks, the task-specific features after feature selection are flattened in the spatial dimension, and the single task feature is represented as:

[0083]

[0084] Among them, Reshape() is a flattening operation;

[0085] (3) The task interaction module uses multi-head self-attention MHSA and a small multi-layer perceptron sMLP to interact between tasks, Splicing in the spatial dimension to get f T , as Q, K, V respectively, the expression of feature interaction through multi-head self-attention and small multi-layer perceptron is as follows:

[0086]

[0087]

[0088] Among them, Concat() is feature concatenation, is the splicing feature after interaction;

[0089] Splicing features after interaction After splitting in the spatial dimension and restoring the features of each task to the original feature map size, the output of the task interaction module is as follows:

[0090]

[0091] Among them, Split() is feature splitting, Reshape() is flattening operation, and i is one of the four tasks;

[0092] (4) The task feature enhancement module takes the adjusted shared feature f′ S and task characteristics Orthogonal fusion: shared features f S After adjusting the number of channels through 1*1 convolution, f′ is obtained through the pooling layer. S , task characteristics after interaction After 6 convolution layers with a kernel size of 3 and a channel number of 512, each of which is followed by a maximum pooling layer with two times downsampling, and finally a fully connected layer, the network is obtained. The orthogonal fusion expression is as follows:

[0093]

[0094] in, is the task-specific feature of the first orthogonal fusion, F orth It is an orthogonal fusion algorithm;

[0095] F orth (f1, f2)=FC(Concat((f1-proj(f1, f2)), f2))

[0096]

[0097] In the formula, FC is the fully connected layer, Concat() is the feature concatenation, f1 and f2 are the two input vectors of orthogonal fusion, and F proj Project vector f1 to vector f2;

[0098] (5) The contour feature f cnt Task characteristics with the first orthogonal fusion Fuse again, and the second orthogonal fusion is as follows:

[0099]

[0100] in, It is the task characteristic feature of the second orthogonal fusion;

[0101] (6) In the task output header, the output of each task feature enhancement module The final prediction result is obtained through the task output head; the age task is passed through the fully connected layer and softmax layer with a dimension of 100 to obtain the predicted age probability distribution D pre =[a1, a2, ..., a n ], the predicted age value obtained according to the expectation formula

[0102]

[0103] The gender task passes through a fully connected layer and a softmax layer with a dimension of 2, and the one with the highest prediction probability is taken as the prediction result;

[0104] The output dimension of the fully connected layer for both the height and weight tasks is 1, and the predicted value is obtained by multiplying it by 255;

[0105] (7) Set the training end condition to S epochs and end the training process.

[0106] The fourth step is to obtain and preprocess the footprint image to be estimated: obtain the barefoot image to be estimated and preprocess it.

[0107] The fifth step is to obtain the human biometric estimation results of the footprint image: the preprocessed barefoot image to be estimated is input into the trained footprint multi-task interactive enhanced biometric estimation network to obtain the human biometric estimation results of the footprint image.

[0108] Compare the predicted results of the regression task with the true value, and the results are as follows Figure 2 、 Figure 3 and Figure 4 As shown in the figure, the network's prediction error is large when the true age is between 40 and 50. This is attributed to the dataset's concentrated age distribution between 20 and 30, with fewer samples of older ages, which results in insufficient network learning. Height prediction yields a good fit and high accuracy. Weight prediction generally conforms to the true weight distribution, but the prediction error is large when the true weight is around 100 kg. This is due to two factors: the dataset contains only a small number of data with weights between 90 and 100 kg, and the low feature diversity among individuals with excessively large weights.

[0109] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for estimating human biometric features from footprint images based on multi-task interactive enhancement, characterized in that: The following steps are involved: 11) Dataset collection and preprocessing: An optical footprint collector is used to collect ten barefoot images of each person, each with their corresponding labels. The labels contain the biometric information corresponding to each barefoot image of each person, forming the raw dataset. Preprocessing operations are performed on the collected raw dataset, including: inverting the grayscale footprint images, removing the scale from the image, rotating the footprints according to the footprint circumscribed rectangle, moving the footprints to the center of the image, unifying the image through zero padding, and removing abnormal samples to obtain the processed image. 12) Constructing a footprint multi-task interactive enhanced biometric feature estimation network: Constructing a footprint multi-task interactive enhanced biometric feature estimation network based on the contour feature extraction branch, the multi-task interactive branch, the task feature enhancement module, and the task output head; The construction of the footprint multi-task interactive enhanced biometric feature estimation network includes the following steps: 121) Setting a footprint multi-task interaction enhanced biometric feature estimation network includes a contour feature extraction branch, a multi-task interaction branch, a task feature enhancement module and a task output head; 122) Construct contour feature extraction branch: Set the Laplacian convolution kernel to extract initial footprint contours of different sizes, unify the size by upsampling, use convolution to fuse the multi-scale contours into a single-channel image, perform image binarization processing, and obtain the footprint contour map; The contour features extracted by 6 convolutional layers, It will be used as input to the task feature enhancement module later; 123) Constructing multi-task interactive branches: Setting the multi-task interaction branch includes a shared trunk branch and a task interaction module; The shared backbone consists of 7 convolutional layers. In the first four convolutional layers, every two convolutional layers are followed by a maximum pooling layer, and the last three convolutional layers are also followed by a maximum pooling layer. The task interaction module consists of four task branches. Each branch selects the features of interest to each task from the shared features through the SAM spatial attention mechanism. The four task branches interact with each other through the multi-head self-attention mechanism to extract the deep features after interaction. Every three convolutional layers are followed by a maximum pooling layer. 124) Constructing task feature enhancement module: The shared backbone features of the multi-task interaction branch are orthogonally fused with the features extracted by the four task branches of its task interaction module. The orthogonally fused features are then orthogonally fused with the feature vectors of the contour feature extraction branch to obtain the task-specific features after the four task features are enhanced. 125) Construct task output header: The feature vector after each task feature enhancement is passed through the fully connected layer to obtain the prediction result; 13) Training of the footprint multi-task interactive enhanced biometric estimation network: The pre-processed collected data set is input into the footprint multi-task interactive enhanced biometric estimation network for training; 14) Acquisition and preprocessing of the footprint image to be estimated: Acquisition of the barefoot image to be estimated and preprocessing; 15) Obtaining the human biometric estimation results of the footprint image: The preprocessed barefoot image to be estimated is input into the trained footprint multi-task interactive enhanced biometric estimation network to obtain the human biometric estimation results of the footprint image.

2. The method for estimating human biometric features from footprint images based on multi-task interactive enhancement according to claim 1, characterized in that: The training of the footprint multi-task interaction enhanced biometric feature estimation network includes the following steps: 21) Input the preprocessed data set into the contour feature extraction branch, and output the contour feature f through the contour feature extraction branch cnt ; 22) The pre-processed data set is input into the shared trunk branch, and the shared trunk branch adaptively selects shared features through the SAM attention module. The extracted shared features f S , Task T i The feature weights learned by the SAM attention module are w i , Task T i For gender, age, height, and weight prediction tasks, the task-specific features after feature selection are flattened in the spatial dimension, and the single task feature is represented as: Among them, Reshape() is a flattening operation; 23) The task interaction module uses multi-head self-attention MHSA and a small multi-layer perceptron sMLP to interact between tasks, Splicing in the spatial dimension to get f T , as Q, K, V respectively, the expression of feature interaction through multi-head self-attention and small multi-layer perceptron is as follows: Among them, Concat() is feature concatenation, is the splicing feature after interaction; Splicing features after interaction After splitting in the spatial dimension and restoring the features of each task to the original feature map size, the output of the task interaction module is as follows: Among them, Split() is feature splitting, Reshape() is flattening operation, and i is one of the four tasks; 24) The task feature enhancement module takes the adjusted shared feature f′ S and task characteristics Orthogonal Fusion: Shared feature f S After adjusting the number of channels through 1*1 convolution, f′ is obtained through the pooling layer. S , task characteristics after interaction After 6 convolution layers with a kernel size of 3 and a channel number of 512, each of which is followed by a maximum pooling layer with two times downsampling, and finally a fully connected layer, the network is obtained. The orthogonal fusion expression is as follows: in, is the task-specific feature of the first orthogonal fusion, F orth It is an orthogonal fusion algorithm; F orth (f1,f2)=FC(Concat((f1-proj(f1,f2)),f2)) In the formula, FC is the fully connected layer, Concat() is the feature concatenation, f1 and f2 are the two input vectors of orthogonal fusion, and F proj Project vector f1 to vector f2; 25) The contour feature f cnt Task characteristics with the first orthogonal fusion Fuse again, and the second orthogonal fusion is as follows: in, It is the task characteristic feature of the second orthogonal fusion; 26) In the task output header, the output of each task feature enhancement module The final prediction result is obtained through the task output head; the age task is passed through the fully connected layer and softmax layer with a dimension of 100 to obtain the predicted age probability distribution D pre =[a1,a2,…,a n ], the predicted age value obtained according to the expectation formula The gender task passes through a fully connected layer and a softmax layer with a dimension of 2, and the one with the highest prediction probability is taken as the prediction result; The output dimension of the fully connected layer for both the height and weight tasks is 1, and the predicted value is obtained by multiplying it by 255; 27) Set the training end condition to S epochs to end the training process.

Images