A plantar pressure automatic classification method and system based on multi-task learning framework

Abstract

The present invention discloses a method and system for automatic plantar pressure classification based on a multi-task learning framework, which belongs to the field of plantar pressure data research. The steps include: S1. Collecting plantar pressure data; S2. Preprocessing the plantar pressure data: Plantar pressure extraction algorithm: Segmenting the plantar pressure data and extracting to generate a footprint sequence through the DBSCAN clustering algorithm; Footprint image representation: Accumulating and summing the extracted plantar pressure sequence to reduce the length, and suppressing the noise of the image through Gaussian filtering; then performing data enhancement; S3. Introducing the channel and spatial attention modules into the Inception block of GoogLeNet, establishing the Inception‑CBAM block, and constructing a multi-task learning model; The multi-task learning model is used to classify and process the footprint image; S4. Obtaining plantar pressure classification data. The method proposed in the present invention is not only applicable to barefoot data, but also to shoe-wearing data with complex pressure areas, laying the foundation for data preparation for automated gait analysis and recognition based on plantar pressure.

Description

Technical Field

[0001] The present invention relates to the field of plantar pressure data research, and in particular to a plantar pressure automatic classification method and system based on a multi-task learning framework. Background Art

[0002] Plantar pressure refers to the pressure field exerted on the sole of the foot and the supporting surface during daily activities. Plantar pressure distribution measurement systems can accurately record the dynamic pressure distribution on the sole of the foot with high temporal and spatial resolution, thereby reflecting the characteristics of a person's gait. Plantar pressure data can be used to determine the dynamic characteristics of different anatomical regions of the sole of the foot.

[0003] Due to varying foot shapes, shoe sole textures, and individual walking abilities, plantar pressure creates significant variations in footprint patterns, potentially resulting in both continuous and disconnected footprints. Accurately identifying left and right feet and their integrity is challenging. Furthermore, because shoe samples may carry noise, they require filtering before identifying left and right feet and their integrity. Since the sample size of the 60 individuals collected was unequal and their stride angles varied, increasing the diversity of the sample size was necessary to avoid imbalanced footprint data that would bias the trained model towards the footprint patterns of individuals with a larger sample size. Some footprints with disconnected heels, as well as those with high arches and flat feet, differ from normal footprints, making manual differentiation difficult.

[0004] In addition, most current work has studied plantar pressure samples taken barefoot or with socks, and it is unclear how these techniques will be applied to different types of footwear. Prior to this study, only MACDONALD et al. used plantar pressure data collected while wearing shoes to classify left and right feet. However, the footprints collected by MACDONALD were relatively complete, without taking into account the incomplete footprints that may occur during daily walking activities. Furthermore, these applications may involve plantar pressure morphology after the subject wears shoes, which is more complex in spatial morphology than barefoot pressure data. This may bring additional difficulties to simple comparison-based algorithm annotation (such as the angle between the two feet, the number of pixels in different parts of the foot, and the similarity between the footprint and the template). Summary of the Invention

[0005] The purpose of the present invention is to overcome one or more deficiencies of the prior art and to provide a method and system for automatic plantar pressure classification based on a multi-task learning framework.

[0006] The object of the present invention is achieved through the following technical solutions:

[0007] A method for automatic plantar pressure classification based on a multi-task learning framework, the method comprising the following steps:

[0008] S1. Collect plantar pressure data;

[0009] S2. Preprocessing of plantar pressure data:

[0010] Plantar pressure extraction algorithm: The plantar pressure data is segmented and extracted using the DBSCAN clustering algorithm to generate a plantar pressure sequence;

[0011] Footprint image representation: The extracted plantar pressure sequence is accumulated and summed to reduce the length and represented as a single frame image;

[0012] Use Gaussian filtering to suppress noise in the image, and then perform data enhancement;

[0013] S3. Introducing the channel and spatial attention modules into the Inception block of GoogLeNet, establishing the Inception-CBAM block, and constructing a multi-task learning model. The multi-task learning model is used to classify footprint images.

[0014] S4. Obtain plantar pressure classification data.

[0015] Furthermore, in step S2, plantar pressure data is extracted using the DBSCAN algorithm and the contour merging algorithm. The measurement formula of the DBSCAN algorithm is:

[0016] ;

[0017] in, is the Euclidean distance matrix, is the Manhattan distance matrix, α and β are weight parameters.

[0018] Furthermore, in step S2, the footprint image represents: the extracted plantar pressure sequence is accumulated and summed to reduce the length. By reducing the length of the segmented pressure sequence, the model convergence speed can be accelerated; the single footprint sequence s i Represented as a single-frame cumulative pressure image , expressed as:

[0019] ;

[0020] in, 、 、 and Then they are respectively represented as the pressure values ​​at their respective positions in the pressure matrix sequence; is the same in the pressure matrix sequence The cumulative sum of the pressure values ​​at a location is expressed as follows:

[0021] ;

[0022] in, is the maximum value of the valid sampling of the valid plantar pressure sequence, It is the pressure exerted on the sensor in the nth row and mth column of the sensor matrix at time t.

[0023] Furthermore, in step S2, the image is subjected to noise suppression processing by Gaussian filtering, which is expressed as:

[0024] ;

[0025] Among them, k is 1, σ is 1, i and j both range from -1 to 1. After calculation, the template needs to be processed in one step: all coefficients of the obtained Gaussian template are normalized, that is, each coefficient is divided by the sum of the template coefficients.

[0026] Furthermore, in step S3, a multi-task learning model is established: the single classification heads of the two auxiliary classifiers of GoogLeNet are modified into dual classification heads, which are defined as left and right foot classification heads and integrity classification heads, respectively, for processing left and right foot predictions and integrity predictions; the single classification head structure of the main classifier is modified into a dual classification head, and the rest of GoogLeNet is used as a hard shared parameter structure.

[0027] Furthermore, in step S3, a focal loss function is added to the multi-task learning model training to adjust the sample loss weight during the model training process. The focal loss function formula is:

[0028] ;

[0029] ;

[0030] in, is the focusing parameter, is the modulation factor, is the predicted probability, , when predicting the left and right feet, the left foot is 1 and the right foot is 0; when predicting the completeness, the complete one is 1 and the incomplete one is 0.

[0031] Furthermore, in step S3, the total loss of the two types of tasks is calculated by the focal loss function and then weighted. The two types of tasks are the left and right foot recognition task and the integrity recognition task, respectively, and are expressed as:

[0032] ;

[0033] in, is the weight parameter of the left and right foot recognition task, Identify task weight parameters for completeness, For the left and right foot recognition prediction task loss, In order to completely identify the prediction task loss, both types of tasks use the focal loss function to calculate the corresponding losses.

[0034] This paper provides an automatic plantar pressure classification system based on a multi-task learning framework, which is used to collect, process, classify and analyze plantar pressure.

[0035] The beneficial effects of the present invention are:

[0036] (1) The custom distance metric not only considers the spatial distance between force sensors, but also the relationship between the force in time sequence. Compared with the existing method of extracting and locating the left and right feet from array sensors, it is not only applicable to barefoot data, but also to shoe-wearing data with complex pressure areas. Ablation studies have shown that the features extracted by combining shallow, middle and deep Inception blocks have the strongest expressive power. The convolutional attention module can improve the model recognition performance. The focal loss adjusts the loss weights of simple samples and sparse difficult samples to improve the model generalization ability.

[0037] (2) Using existing methods and deep learning methods, it was demonstrated that barefoot data does not require filtering, while shoe-wearing data does. The accuracy of left and right foot recognition in barefoot and shoe-wearing data reached 99.78% and 98.59%, respectively, and the reliability was verified through cross-population testing.

[0038] (3) By collecting plantar pressure data from more patients, combining this scheme, optimizing the existing algorithm, and integrating it into automated evaluation software, further research will be facilitated. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flowchart of the steps of the method;

[0040] Figure 2 It is a difficult sample graph;

[0041] Figure 3 Schematic diagram of experimental data acquisition, (a) is barefoot walking data acquisition, (b) is shoe walking data acquisition;

[0042] Figure 4 This is the data preprocessing structure diagram;

[0043] Figure 5 It is a footprint sequence diagram;

[0044] Figure 6 This is a diagram of the multi-task learning model structure;

[0045] Figure 7 Schematic diagram of the Inception-CBAM block structure;

[0046] Figure 8 Schematic diagram of the basic structural unit of the network, (a) is the basic convolution block, (b) is the main classifier and (c) is the auxiliary classifier. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0048] See Figure 1 , provides an automatic plantar pressure classification method based on a multi-task learning framework, characterized in that the method steps include:

[0049] S1. Collect plantar pressure data;

[0050] S2. Preprocessing of plantar pressure data:

[0051] Plantar pressure extraction algorithm: The plantar pressure data is segmented and extracted using the DBSCAN clustering algorithm to generate a plantar pressure sequence;

[0052] Footprint image representation: The extracted plantar pressure sequence is accumulated and summed to reduce the length and represented as a single frame image;

[0053] Use Gaussian filtering to suppress noise in the image, and then perform data enhancement;

[0054] S3. Introducing the channel and spatial attention modules into the Inception block of GoogLeNet, establishing the Inception-CBAM block, and constructing a multi-task learning model. The multi-task learning model is used to classify footprint images.

[0055] S4. Obtain plantar pressure classification data.

[0056] We recruited 45 male and 15 female subjects for data collection. The subjects were primarily students (age 23.17 ± 1.3 years, height 169.075 ± 9.19 cm, weight 62.93 ± 11.76 kg, shoe size 39.3 ± 2.29). Their shoes included basketball shoes, sneakers, casual shoes, athletic shoes, and Martin boots. The pressure platform parameters for the data collection are shown in Table 1:

[0057] Table 1

[0058]

[0059] See Figure 3The experimenters used the Mid-step protocol, that is, each experimenter walked at least three steps, stepped on the pressure test plate, and then walked at least 3 steps and stopped. The data collected in this way is obtained in the most stable state of walking, because the data obtained from the same person at different times are more repeatable. For each experimenter, complete and incomplete data were obtained when wearing shoes and complete and incomplete data when barefoot. The experimental data were obtained as follows Figure 3 shown.

[0060] Data preprocessing:

[0061] See Figure 2 , Figure 4 In this study, the distance metric of DBSCAN was improved according to the temporal and spatial relationship of the sensor force points, and the footprints were automatically merged into the same category after being clustered into pressure areas, so as to realize automatic sample extraction. At the same time, this embodiment modified GoogLeNet into a multi-task network and embedded CBAM into it, and used focalloss to adjust the loss weights of difficult-to-identify samples and easy-to-identify samples during the training phase. The effectiveness of the proposed method was verified by collecting complete and incomplete plantar pressure data of 60 people and conducting cross-population tests. The results showed that the recognition accuracy was much higher than that of the recent methods. The data processing framework of this embodiment is as follows: Figure 4 As shown. In the data preprocessing stage, the plantar pressure data collected by the Footscan system is first saved, and the left and right footprints and their integrity are annotated based on the self-developed software. For the extraction of footprints, an improved DBSCAN algorithm is used to automatically extract footprints. Considering that the plantar pressure images collected when wearing shoes contain more high-frequency information, this information may not serve the classification of footprints and the judgment of their integrity well, so a Gaussian filtering algorithm is used to process the high-frequency information. Due to the imbalance in the number of samples collected in each category, this embodiment also uses a variable step direction angle to achieve data enhancement.

[0062] Plantar pressure extraction algorithm:

[0063] Barefoot pressure data can take on different shapes depending on foot shape. For example, someone with high arches may have a split front and back of the foot. Footprint pressure data collected while wearing shoes can create more complex patterns due to shoe shape, creating pressure zones that may appear continuous or discontinuous. Furthermore, stride length varies from person to person, and everyone starts their step at a different point when collecting data, making it difficult to locate footprints simply by presetting the number of footprints.

[0064] In this embodiment, a clustering algorithm is used to merge pressure areas that are close to each other to form a footprint, and an automatic sequence extraction is achieved based on the footprint position. Based on the above considerations, an improved DBSCAN algorithm and a contour merging algorithm are used to complete this work.

[0065] First, according to the three-dimensional plantar pressure matrix Determine the feature array, that is, the effective position index matrix of the footprint on the sensor matrix , and calculate the Euclidean distance matrix and The Manhattan distance matrix of the initial force time At the same time, and Multiply the weight coefficients α and β respectively, and set the initial values ​​of the two parameters to 0.1 and 0.1 respectively. Then search for the optimal parameters, and set the search step to 0.01. The optimal α and β obtained after the search are 0.95 and 0.05 respectively. The two matrices are calculated according to the formula to obtain the DBSCAN metric:

[0066] ;

[0067] The clustering and extraction process is as follows:

[0068] Algorithm 1: Plantar pressure extraction;

[0069] enter , pressure point scanning radius eps: 0.2, minimum number of pressure points min_samples: 3, distance matrix Output segmented footprint sequence:

[0070] 1) Determined in Mark as unvisited;

[0071] 2) Do;

[0072] 3) Randomly select an unvisited ;

[0073] 4) Marking for visited;

[0074] 5) if The number of pressure points in eps ≥ min_samples;

[0075] 6) Create a new cluster , while Add the points inside eps to ;

[0076] 7) Add the points within eps to middle;

[0077] 8) Traversal (1≤k≤max), ;

[0078] 9) if for unvisited;

[0079] 10) Marking for visited;

[0080] 11) if The pressure points in eps are ≥ min_samples, and these pressure points are added to ;

[0081] 12) if If it has not been assigned to any cluster, it will be assigned to the current cluster. ;

[0082] 13) End traversal;

[0083] 14) else Recorded as noise;

[0084] 15) until there is no mark as unvisited pressure point;

[0085] 16) Use the labels_ method of DBSCAN to extract cluster labels after clustering;

[0086] 17) for each cluster;

[0087] 18) Calculate the cluster center label category and merge clusters with the same label;

[0088] 19) end for;

[0089] 20) Searching for footprint location information on the clustered footprint array;

[0090] 21) Determine the location of the dynamic sequence footprints based on the array footprint positions;

[0091] 22) Calculate the shortest width and height rectangle that encloses the footprint;

[0092] 23) Determine the index of the footprint in the sequence based on the length, width and coordinates to complete the segmentation;

[0093] 24) Output the segmented footprint sequence;

[0094] Footprint image representation:

[0095] After extracting the footprint data using the proposed plantar pressure extraction algorithm, a representative footprint sequence is visualized as follows: Figure 5 As shown. There are still many similar frames in the time dimension. If the sequence is input into the network, a large number of image features need to be extracted for each training, which will slow down the network convergence. By reducing the length of the segmented pressure sequence, the model convergence speed can be accelerated. In this embodiment, a single footprint sequence s i Represented as a single-frame cumulative pressure image .

[0096] ;

[0097] in, 、 、 and Then they are respectively represented as the pressure values ​​at their respective positions in the pressure matrix sequence; is the same in the pressure matrix sequence The cumulative sum of the pressure values ​​at a location is expressed as follows:

[0098] ;

[0099] in, is the maximum value of the valid sampling of the valid plantar pressure sequence, is the pressure exerted on the sensor in the nth row and mth column of the sensor matrix at time t. After this step, a footprint sequence of dozens of frames is represented as one frame.

[0100] Gaussian filtering:

[0101] For the plantar pressure data of shoes, the texture of the soles and debris will lead to the existence of high-frequency information. After a single footprint sequence s is represented as a single-frame pressure image img, the high-frequency information in the sequence still exists, which is not conducive to model classification and integrity judgment. Therefore, it is necessary to filter the single-frame pressure image. The Gaussian filter is a linear filter that can effectively suppress noise and smooth the image. Its function and principle are similar to those of the mean filter, and both take the mean of the pixels in the filter window as the output. Before processing the single-frame cumulative pressure image, it is necessary to discretize the continuous Gaussian function to obtain a Gaussian filter template. This embodiment generates a 3×3 template through parameters, and uses the center point of this template as the origin for sampling. The value obtained by substituting the value of each position into the Gaussian function is the coefficient of the Gaussian template. The calculation method of the Gaussian template in this embodiment is as follows:

[0102] ;

[0103] Here k is 1, σ is 1, i and j both range from -1 to 1. After calculation, the template needs to be further processed: all coefficients of the obtained Gaussian template are normalized, that is, each coefficient is divided by the sum of the template coefficients.

[0104] Data enhancement: This embodiment addresses the problem of extremely unbalanced sample sizes between subjects and proposes a data enhancement method based on step angle to improve the robustness of the model in footprint pattern recognition. Under normal circumstances, there are natural differences in step angles between subjects due to differences in physiological characteristics and walking habits. To avoid excessive bias towards subjects with large sample sizes during model training, this embodiment first identifies the maximum and minimum step angles of the left and right feet in the plantar pressure image set of each subject (denoted as , and , ). Then, each left and right foot sample is rotated, and the rotation angle is the original step angle plus a random angle α (in and ), simulating the natural variation of the actual step angle. If α is greater than the current angle, the step angle is increased; otherwise, the step angle is decreased to ensure that the enhanced data fits the real sample distribution.

[0105] Table 2 Sample distribution table

[0106]

[0107] Table 2 shows the sample distribution before and after data augmentation. This method effectively alleviates the sample imbalance problem and improves the generalization ability of the model. The sample augmentation algorithm process is as follows:

[0108] Algorithm 2: Sample enhancement:

[0109] Enter the experimenter sample:

[0110] Output the newly generated sample:

[0111] 1) for in :;

[0112] 2) Search Pressure area contours;

[0113] 3) if ==1:

[0114] 4) The minimum enclosing rectangle calculates the rectangular angle of the outline;

[0115] 5) else: merge ;

[0116] 6) The minimum enclosing rectangle calculates the rectangular angle of the outline;

[0117] 7) end if;

[0118] 8) if : Add to ;

[0119] 9) else: add to ;

[0120] 10) end if;

[0121] 11) end for;

[0122] 12) Calculate the maximum and minimum angles of the left and right feet;

[0123] 13) Randomly generate angle values ​​between the maximum and minimum values;

[0124] 14) Calculate the coordinates of the image center;

[0125] 15) Generate a rotation matrix based on the angle and center coordinates;

[0126] 16) Affine transformation of the image;

[0127] Multi-task learning model:

[0128] This embodiment focuses on the classification of left and right feet and integrity of plantar pressure (respectively referred to as Task 1 and Task 2). Because both left and right foot attributes and integrity attributes exist, the two tasks are correlated, allowing the two recognition tasks to be performed in parallel. Compared to single-task models, multi-task models can leverage the correlation between left and right foot classification and integrity classification to learn more general and richer feature representations. This shared learning mechanism also prevents the model from relying too heavily on the specific features of a single task, reducing the risk of overfitting.

[0129] The Inception block of GoogLeNet can process feature maps at multiple scales, but it can only handle a single task. Training the model separately to complete Task 1 and Task 2 not only requires setting the hyperparameters of the corresponding recognition tasks, but also takes longer to train. Therefore, this embodiment modifies its structure to adapt to multiple tasks. First, the single classification heads of the two auxiliary classifiers of GoogLeNet are modified into dual classification heads, namely the left and right foot classification heads and the integrity classification head, and then the single classification head structure of the main classifier is modified into a dual classification head. The rest of the model network is used as a hard shared parameter structure. Figure 6As shown, the entire model network consists of a basic convolution block (Basic convblock), an Inception-CBAM block (Inception-CBAM block), a main classifier (Mainclassifier), and an auxiliary classifier (Auxiliary classifier). The convolutional attention mechanism CBAM is introduced to the Inception block to extract more important spatial features. In terms of the implementation of each module, the basic convolution block obtains the basic feature map through alternating convolution and pooling operations. The Inception block uses multiple convolution kernels to extract features, and after CBAM, it pays more attention to some complex features. The multi-task learning model learns two types of task features through a shared layer (hard sharing) during the training phase, and finally the main classifier predicts the category of the test set samples. The auxiliary classifier enables the middle layer of the network to also have a classification effect, prevents the gradient from disappearing, and regularizes the network. The Inception-CBAM block is as follows Figure 7 , the basic structural units of the network (basic convolution block, main classifier, auxiliary classifier) ​​are as follows Figure 8 As shown. Although the Inception block has a strong ability to capture multi-scale features, the recognition task of this embodiment is relatively difficult. In order to further strengthen the important features, CBAM is introduced in the Inception block in the network. This attention mechanism includes a spatial attention module and a channel attention module. The channel attention module generates two descriptors for each channel through global average pooling and global maximum pooling, and then inputs the descriptors into a multi-layer shared perceptron to capture the relationship between channels. The outputs of the multi-layer perceptron are then merged and passed through an activation function to generate channel attention weights. Finally, these weights are multiplied with the original feature map to emphasize or suppress certain channels. The spatial attention module uses channel average pooling and channel maximum pooling to generate two two-dimensional feature maps across the channel dimension, merge the feature maps, and then use two-dimensional convolution to generate a spatial attention feature map. After the activation function, it is multiplied with the original feature to obtain the spatial position to be emphasized or suppressed.

[0130] Focal loss parameter settings:

[0131] However, in the left and right foot classification task, because some barefoot and shoe-wearing samples only have pressure on the toes or heels, and there are a large number of relatively complete left and right foot samples, it is easy for the network gradient to bias towards samples with large vectors and easy identification during training. In the integrity classification task, due to the structure of the sole, it is difficult to accurately identify the integrity of some shoe-wearing samples. Inspired by the ability of focal loss to weight the losses of two types of samples to improve model performance when the difficult and easy samples are unbalanced, focal loss was migrated to this field. In each round of iteration of the training set, the 128 samples input into the model each time are mostly easy-to-identify samples. It is easy to make mistakes when identifying samples with fewer pressure areas and large shape differences. Focal loss calculates the loss of each sample. When a sample is misidentified, a larger loss value is obtained. By weighting this loss to increase the contribution of the misidentified samples, the sample loss weights in the training process are adjusted to improve generalization. This will make the network pay more attention to the misidentified samples during training, while not ignoring simple samples. The formula of focal loss is as follows:

[0132] ;

[0133] Where γ is the focusing parameter, is the modulation factor.

[0134] ;

[0135] Among them, p∈[0,1] is the probability distribution of the sample category predicted by the multi-task model during the training phase, y∈{0,1}, when predicting left and right feet, the left foot is 1 and the right foot is 0. When predicting completeness, complete is 1 and incomplete is 0. When it tends to 1 and is a simple sample that is easy to identify, calculate , reducing the contribution of simple samples to the loss. When it approaches 0 and the sample is misclassified, Obtaining a larger weight for the difficult sample increases the loss value of the sample. And the focusing parameter γ calculates the loss weights of simple samples and difficult samples, and then multiplies the weights by their cross entropy loss, so that the model focuses on difficult samples during the training phase and avoids the network being biased towards a large number of simple samples.

[0136] This embodiment introduces focal loss as a loss function to balance the loss weights of difficult samples and simple samples in the multi-task model during the training phase, thereby improving model performance. After the focal loss function balances the loss weights of difficult samples and simple samples, a weighted sum is taken to obtain the total loss, which is expressed as:

[0137] ;

[0138] Among them, weight1 is the weight parameter of the left and right foot recognition task, weight2 is the weight parameter of the integrity recognition task, For the left and right foot recognition task loss, To ensure completeness of the identification task loss, both tasks use focal loss to calculate the corresponding losses. In this example, the initial values ​​of the gamma parameter and the two task weights are set to 0.5, 0.1, and 0.9, respectively. Gamma and task weights undergo 7 and 9 iterations, respectively. Gamma is incremented by a step of 0.5 in each iteration, and the two task weights are incremented and decremented by a step of 0.1 in each iteration. Task weights do not include 0 and 1 because the model needs to complete two recognition tasks simultaneously. When either weight is 0 or 1, the multi-task model degenerates into a single-task model.

[0139] When modeling barefoot data, the model performed best when the weight parameters for Task 1 and Task 2 were 0.9 and 0.1, respectively, and the gamma parameter of the focal loss was set to 3. The model achieved a 99.78% recognition rate for both left and right feet and a 97.79% complete recognition rate. When modeling shod data, the model performed best when the weight parameters for Task 1 and Task 2 were 0.5 and 0.5, respectively, and the gamma parameter of the focal loss was set to 1. The model achieved a 98.59% recognition rate for both left and right feet and a 94.35% complete recognition rate.

[0140] Comparison method:

[0141] This example compares the left and right foot classification methods, including comparisons based on plantar pressure centers, footprint-to-template similarity comparisons (COP End Points (EP), COP Dynamic Time Warping (DTW), P100 Template Matching (TM)), and deep learning-based methods (P100 Convolutional Neural Network (CNN), COP Temporal Convolutional Networks (TCN)). Furthermore, this example uses image classification models including AlexNet, DenseNet, and Vision Transformer (VIT) as baselines for comparison. The AlexNet network uses LRN normalization layers, which enhances the model's generalization and robustness. The dense connections of DenseNet alleviate the vanishing gradient problem, enhance feature propagation, promote feature reuse, and significantly reduce the number of parameters. Unlike convolution, VIT directly blocks the image and then encodes each block, incorporating positional information into the model through positional encoding.

[0142] Results and discussion:

[0143] Experimental setup:

[0144] This paper uses a subject-wise data splitting method, training the model with data from 48 participants and testing it with data from 12 participants, with a ratio of 8:2. Initially, the learning rate was set to 2e-5 with a learning rate decay factor of 1 / 10. The learning rate was adjusted every four iterations. Adam was used as the model's parameter optimizer, with a weight decay parameter of 1e-3, a batch size of 128, and a sample size of (128, 128). The loss function was the weighted sum of the losses for Task 1 and Task 2. The comparison methods based on deep learning all used a cross-entropy loss, Adam as the optimizer, a learning rate of 2e-5, a weight decay of 1e-3, a learning rate decay factor of 0.1, and a batch size of 128. Since COP EP, COP DTW, and P100 TM are neither machine learning nor deep learning methods, no hyperparameters were required.

[0145] Experimental results and analysis:

[0146] Comparative analysis with baseline methods:

[0147] Comparisons of the results of this example with several image classification models are shown in Tables 3 and 4. Compared to image classification methods, the proposed deep learning method achieved the highest accuracy under subject-wise segmentation. Under this segmentation, the recognition rates for left and right foot data for shod and barefoot subjects were 98.59% and 99.78%, respectively, and the recognition accuracy for completeness was 94.35% and 97.79%, respectively. Furthermore, under this segmentation, the recognition rate for barefoot data generally decreased after filtering, while the recognition rate for shod data generally increased after filtering.

[0148] Comparing the results of the barefoot image classification model before and after filtering revealed that, with the exception of VIT, AlexNet, DenseNet, and the model proposed in this embodiment all achieved left and right foot recognition accuracy rates exceeding 93%. Furthermore, the accuracy of these four models for completeness of recognition generally reached above 96%, demonstrating the reliability of deep learning methods for image recognition. Comparing the results of the shoe-wearing image classification model before and after filtering revealed that the four models performed worse than barefoot recognition for left and right foot recognition, indicating that shoed footprint patterns are more complex and difficult to recognize than barefoot patterns. VIT's left and right foot recognition performance was still lower than that of AlexNet, DenseNet, and the model proposed in this embodiment because after VIT slices the image, the continuity between adjacent pressure blocks is no longer preserved. The continuity of plantar pressure points is crucial for left and right foot recognition, a limitation of VIT itself: a lack of inductive bias compared to convolution. AlexNet, DenseNet, and the model proposed in this embodiment all use two-dimensional convolution to extract image features, leveraging the continuity between pixels and making them well suited for the task in this embodiment.

[0149] Another finding is that, except for VIT, the accuracy of most left-right foot recognition is higher than that of complete foot recognition, which may be because complete foot recognition is more complicated than left-right foot recognition.

[0150] The impact of filtering on recognition:

[0151] Table 3 Comparison of barefoot pressure data before and after filtering (unit: %)

[0152]

[0153] Table 4 Comparison of shoe pressure data before and after filtering (unit: %)

[0154]

[0155] As can be seen from Table 3, the performance of left and right foot identification and integrity judgment of barefoot plantar pressure data after filtering is reduced in most cases. In the model proposed in this embodiment, the recognition rate of left and right feet decreased by 0.22% after filtering. This may be because the original barefoot data has relatively little interference information, and filtering such samples will lead to a reduction in the details of the original image, thereby reducing the discrimination and causing misclassification. As can be seen from Table 4, the accuracy of left and right foot identification and integrity judgment of shod plantar pressure data after filtering is improved in most cases. In the model proposed in this embodiment, the integrity judgment improved by 0.23% after filtering. This shows that filtering shoe-mounted plantar pressure data is very necessary. This is because the structure of the sole pattern and the material of the sole have an impact on the plantar pressure pattern, which may bring more high-frequency information. Filtering can reduce this impact, thereby improving the recognition rate of the model.

[0156] Comparison with recent methods:

[0157] A comparison of the model proposed in this embodiment and the most recent classification method is shown in Table 5. Regardless of whether the data is collected barefoot or wearing shoes, the method of this embodiment has advantages in identifying the left and right feet and judging the integrity.

[0158] Table 5 Comparison with recent classification methods (unit: %)

[0159]

[0160] Recently, Macdonald et al. proposed COP EP, COP DTW, and COP TCN to distinguish left and right foot plantar pressure data. These three methods use the COP Anterior-Posterior (AP) and Medio-Lateral (ML) sequences to classify left and right feet. Comparing the results for barefoot and shod data shows that COP EP and COP DTW perform similarly on both data types. The former compares the size of the heel and toe in the ML direction to distinguish left and right, while the latter distinguishes left and right by calculating the similarity between the left and right DTW templates and the samples. Due to the incompleteness of collected samples, COP EP can easily misclassify incomplete footprints in the left-right recognition task because it only compares the values ​​at the end points of the sequence. COP DTW requires finding representative templates that contain incomplete samples. After averaging the COP samples in the training set, this can lead to significant differences between the COP time series templates and the COP sequences obtained from normal footprint data, which contributes to the poor recognition performance of COP EP and COP DTW.

[0161] COP TCN is a time series causal convolution, that is, the current inference result is only related to the time step before the input sequence, and does not depend on the later time step. TCN performs better than COP EP and COP DTW. This is because TCN introduces residual connections and causal convolutions. The residual connections give the network the effect of identity mapping and ensure that the learning effect will not deteriorate due to the deepening of the network layers during training. Compared with ordinary convolution layers, causal convolution is more suitable for processing time series data and can extract more representative features. However, the effect of this method is not as good as the method of this embodiment. It may be caused by the difference in length between the incomplete sequence and the complete sequence. The incomplete COP sequence needs to be padded with 0 before being input into the model training, which may make it difficult to learn the discriminative features from the incomplete footprint COP data. The pressure sequence representation method used in this embodiment avoids the problem of different sequence lengths.

[0162] The P100 CNN classifies left and right feet and their integrity by extracting image features. The P100 CNN directly uses pressure images as input, extracts features through two-dimensional convolution, and then uses fully connected layers for classification. However, its recognition performance is inferior to the model proposed in this example. This may be because the two convolutional layers only extract relatively shallow features and fail to learn higher-level image features, resulting in insufficient expressiveness of the extracted features. The P100 CNN performs better in identifying integrity because judging integrity through images is more accurate. However, due to the subjects' walking habits, some left and right footprints are similar, making it difficult to distinguish them using only extracted shallow image features. This contributes to the poor left and right foot recognition performance.

[0163] Template matching (P100™) performs poorly due to the presence of a large number of incomplete samples in the collected data. The algorithm must find the optimal alignment before calculating the SAD. The calculated orientation angles for incomplete samples differ significantly from those of the template. Template matching is sensitive to the sample orientation angles, so large discrepancies in orientation angles increase the SAD error, leading to recognition errors. Furthermore, template matching performance relies on constructing a representative template, which is impractical for large datasets.

[0164] Among existing classification algorithms, comparison-based methods (COP EP, COP DTW, and P100 TM) are inferior to deep learning-based methods (COP TCN and P100 CNN) in recognition accuracy. Existing deep learning methods are also inferior to the method proposed in this example. This is because existing comparison-based methods fail to account for incomplete footprint data. Competitive deep learning methods such as COP TCN and P100 CNN have limited expressive power in extracting temporal and spatial features, respectively, resulting in models with accuracy far lower than the method proposed in this example.

[0165] Cross-population testing:

[0166] In order to verify the performance of the plantar pressure extraction algorithm proposed in this embodiment on other data sets, this embodiment uses a data set collected by YI, which includes plantar pressure data of 29 healthy and 21 hemiplegic middle-aged and elderly people. Since middle-aged and elderly people have weak walking ability and are prone to falling, this data set only collects data when wearing shoes. The overall situation of the data set is shown in Table 6. When processing samples of healthy middle-aged and elderly people, the hyperparameters eps and min_samples of DBSCAN are set to 0.2 and 3 respectively, and the hyperparameters α and β of the distance matrix are 0.95 and 0.05 respectively. With these parameter settings, the samples are completely correctly segmented. When processing data of young and middle-aged people with hemiplegia, the hyperparameters eps and min_samples of DBSCAN are 0.12 and 3 respectively, and the hyperparameters of the distance matrix are consistent with those of healthy middle-aged and elderly people. The number of incorrectly segmented footprints obtained with such parameter settings is 9.

[0167] Table 6 Middle-aged and elderly subjects

[0168]

[0169] The segmentation method proposed in this embodiment is less effective on the data of hemiplegic subjects than on healthy subjects. This is because the stride of the hemiplegic subjects' footprints is smaller than that of healthy people, and there may even be a state where the two footprints overlap. The algorithm directly divides them together, resulting in abnormal segmented footprints. The distance between the footprints of healthy subjects is larger than that of hemiplegic subjects, making them easier to divide.

[0170] To verify the reliability of the proposed multi-task model in plantar pressure recognition, the authors conducted recognition task tests on correctly segmented samples from the aforementioned middle-aged and elderly dataset. The test results are shown in Table 7.

[0171] Table 7 Cross-population test (unit: %)

[0172]

[0173] Table 7 shows that the multi-task model trained on the shoe-wearing data of young students generalizes well on data from healthy middle-aged and elderly individuals, achieving 98.12% and 92.79% recognition rates for left and right feet and completeness, respectively. However, the recognition rates on the hemiplegic patient dataset were relatively poor, at 95.62% and 87.23%, respectively. This is because the students are healthy individuals, and their footprint patterns are more similar. The generalization on hemiplegic patients is weaker because the footprint patterns of hemiplegic patients differ significantly from those of students. Some hemiplegic patients, when walking normally, leave footprints that resemble the incomplete footprints of healthy individuals, leading to misjudgment by the model.

[0174] Multitasking vs. single-tasking:

[0175] This example modifies the multi-task model into a single-task model, and trains Model 1 and Model 2 on the student shoe-wearing data for the two tasks, respectively. The parameter settings during model training are the same as in 3.1. First, the optimal gamma parameter is searched for each model. The searched gamma parameter is 4. Model 1's left-right foot recognition accuracy is 1.65% lower than the multi-task model, while Model 2's completeness recognition accuracy is 0.47% lower than the multi-task model. To observe the generalization differences between the single-task and multi-task models on data collected by YI, the test results for Task 1 of Model 1 on data from healthy and hemiplegic elderly individuals are 97.49% and 92.34%, respectively. The test results for Task 2 of Model 2 on data from healthy and hemiplegic elderly individuals are 89.03% and 81.39%, respectively. Both are lower than the recognition rates of the multi-task model. Furthermore, given the same model size, the single-task model can only complete one recognition task. This example uses a multi-task learning model, with parameters hard-shared between tasks due to task correlation. The two tasks are weighted to reduce the possibility of parameter scaling. In addition, the two types of tasks can regularize each other through hard parameter sharing. By learning the parameters of the other task, the model is prevented from being biased towards a single task and the risk of overfitting is reduced.

[0176] Ablation experiment:

[0177] To quantify the impact of different layers in the Inception module on the overall model performance, we conducted ablation experiments. The original GoogleNet architecture consists of three Inception layers: shallow (Inception 3a, Inception 3b), middle (Inception 4a, Inception 4b, Inception 4c, Inception 4d, Inception 4e), and deep (Inception 5a, Inception 5b). By removing or retaining different layers, we analyzed the contribution of these three modules, as well as CBAM, to the model's classification performance. Because shoe-wearing data is more realistic, ablation experiments were performed on different layers of the network only.

[0178] When only shallow blocks were retained, the model performed poorly. When shallow and mid-level modules were combined, performance improved. This suggests that mid-level features, while complementing shallow features, can effectively enhance the model's ability to capture higher-level semantics. When shallow and deep modules were combined, the model's accuracy improved further, indicating that deep features are critical for extracting high-level semantic information. When only mid-level modules were retained, the recognition rate was second only to the combination of shallow, mid-level, and deep modules, demonstrating that mid-level features play a key role in capturing complex semantics. When mid-level and deep modules were combined, the model's performance was slightly lower than when only the mid-level was used, likely because mid-level features are sufficient to capture most of the important semantics. The model performed best when all shallow, mid-level, and deep modules were involved. The addition of CBAM improved the accuracy of the two tasks by 0.71% and 0.7%, respectively.

[0179] Through ablation experiments, we found that the role of shallow modules is limited. They can only capture low-level features and perform poorly when used alone. The middle-level modules contribute significantly to model performance and can independently capture discriminative features. Although deep modules are important in extracting high-order semantic information, their independent effect is not as good as the middle layer. The best performance is in the combination of shallow + middle + deep layers, indicating that features at different levels complement each other in the model to form the most effective feature expression.

[0180] This embodiment improves DBSCAN. The customized distance metric considers not only the spatial distance between force sensors but also the temporal relationship of the forces. Compared with existing methods for extracting and locating left and right feet from array sensors, this method is applicable not only to barefoot data but also to shoe-wearing data with complex pressure regions. Ablation studies show that the features extracted by combining shallow, mid-layer, and deep Inception blocks have the strongest expressive power. The convolutional attention module can improve model recognition performance. The focal loss adjusts the loss weights for simple samples and sparse, difficult samples, improving model generalization. Existing methods and deep learning methods demonstrate that barefoot data does not require filtering, while shoe-wearing data does. The left and right foot recognition accuracy rates for barefoot and shoe-wearing data reached 99.78% and 98.59%, respectively. Cross-population testing verified their reliability and demonstrated superior performance to other proposed methods. Plantar pressure data from more patients can be collected, and existing algorithms can be optimized and integrated into automated assessment software for further research.

[0181] The dataset in this example was created based on healthy individuals and a small number of hemiplegic patients. Therefore, the plantar pressure analysis of footprints, as well as the identification of left and right feet and foot integrity, for pathological gait can be validated on a larger dataset. Future work plans to collect plantar pressure data from more patients to optimize the existing algorithm and integrate it into automated assessment software.

[0182] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A method for automatic plantar pressure classification based on a multi-task learning framework, characterized in that: The method steps include: S1. Collect plantar pressure data; S2. Preprocessing of plantar pressure data: Plantar pressure extraction algorithm: The plantar pressure data is segmented and the plantar pressure sequence is extracted using the DBSCAN clustering algorithm; Footprint image representation: The extracted plantar pressure sequence is accumulated and summed to reduce the length and represented as a single frame image; Use Gaussian filtering to suppress noise in the image, and then perform data enhancement; S3. Introducing channel and spatial attention modules into the Inception block of GoogLeNet to build a multi-task learning model; the multi-task learning model is used to classify footprint images. S4. Obtain plantar pressure classification data; In step S3, a multi-task learning model is established: the single classification heads of the two auxiliary classifiers of GoogLeNet are modified into dual classification heads, which are defined as left and right foot classification heads and integrity classification heads, respectively for processing left and right foot prediction and integrity prediction; the single classification head structure of the main classifier is modified into a dual classification head, and the rest of GoogLeNet is used as a hard shared parameter structure.

2. The method for automatic plantar pressure classification based on a multi-task learning framework according to claim 1, characterized in that: In step S2, plantar pressure data is extracted using the DBSCAN algorithm and the contour merging algorithm. The measurement formula of the DBSCAN algorithm is: ； in, is the Euclidean distance matrix, is the Manhattan distance matrix, α and β are weight parameters.

3. The method for automatic plantar pressure classification based on a multi-task learning framework according to claim 1, characterized in that: In step S2, the footprint image is represented by: the extracted plantar pressure sequence is accumulated and summed to reduce the length, and the single footprint sequence s i Represented as a single-frame cumulative pressure image , expressed as: ； in, 、 、 and Then they are respectively represented as the pressure values ​​at their respective positions in the pressure matrix sequence; is the same in the pressure matrix sequence The cumulative sum of the pressure values ​​at a location is expressed as follows: ； in, is the maximum value of the valid sampling of the valid plantar pressure sequence, It is the pressure exerted on the sensor in the nth row and mth column of the sensor matrix at time t.

4. The method for automatic plantar pressure classification based on a multi-task learning framework according to claim 1, characterized in that: In step S2, the image is subjected to noise suppression processing by Gaussian filtering, which is expressed as: ； Among them, k is 1, σ is 1, i and j both range from -1 to 1. After calculation, the template needs to be processed in one step: all coefficients of the obtained Gaussian template are normalized.

5. The method for automatic plantar pressure classification based on a multi-task learning framework according to claim 1, characterized in that: In step S3, a focal loss function is added to the multi-task learning model training to adjust the sample loss weight during the model training process. The focal loss function formula is: ； ； in, is the focusing parameter, is the modulation factor, is the predicted probability, , when predicting the left and right feet, the left foot is 1 and the right foot is 0; when predicting the completeness, the complete one is 1 and the incomplete one is 0.

6. The method for automatic plantar pressure classification based on a multi-task learning framework according to claim 1, characterized in that: In step S3, the total loss of the task is calculated by the focal loss function, which is expressed as: ； in, is the weight parameter of the left and right foot recognition task, Identify task weight parameters for completeness, For the left and right foot recognition prediction task loss, To completely identify the prediction task loss, the tasks are all calculated using the focal loss function to calculate the corresponding loss.

7. An automatic plantar pressure classification system based on a multi-task learning framework, characterized by: The system is used to implement an automatic plantar pressure classification method based on a multi-task learning framework as described in any one of claims 1-6.

Images