An old cognitive impairment classification method based on multi-feature gait fusion

Abstract

The application discloses a kind of based on multi-feature gait fusion's senile cognitive impairment classification method. Including the following steps: first, gait contour from image data set is taken out and is preprocessed, obtain the skeleton map corresponding thereto;Then the two are input into multi-modal spatial feature extractor and the spatial feature of each frame image is extracted by fusion;The features extracted are cut horizontally, then down-sampling is carried out by horizontal pooling module;Next, the features obtained by down-sampling are input into multi-scale time feature fusioner, and the multi-scale fused time features are extracted;The model is trained by the joint loss function of Triplet loss and SoftMax loss, and the final classification result is obtained.The application aims to develop a kind of senile cognitive impairment classification method, without using contact sensor or index, and explore its potential as a convenient, objective, fast and non-contact cognitive impairment screening method.

Description

Technical Field

[0001] This invention relates to a classification method for cognitive impairment in the elderly based on multi-feature gait fusion, belonging to the field of computer vision and intelligent information technology. Background Technology

[0002] Cognitive impairment is a mental health disorder that primarily affects cognitive abilities, leading to alterations in performance on specific cognitive tasks, including learning, memory, perception, problem-solving, and reasoning. It can affect multiple cognitive domains simultaneously or sequentially, gradually or abruptly. Cognitive impairment and dementia are leading causes of disability in older adults, and promoting healthy brain aging is considered a key factor in reducing the burden of age-related disability. However, it is estimated that 45% of dementia cases can be prevented or delayed by modifying risk factors and improving daily living activities. However, routine non-cognitive assessments alone are insufficient for physicians to accurately predict a patient's cognitive function. Therefore, cognitive assessment is helpful in diagnosing impairments of thinking and in identifying potential interventions.

[0003] Slowed gait speed is a common physiological phenomenon with age, but it can also be closely associated with the onset of cognitive impairment, particularly in the early stages of mild cognitive impairment (MCI) and dementia. Previous research has shown that gait is a complex motor task involving multiple physiological systems, especially different anatomical regions of the brain. Slow gait speed often shows a significant correlation with cognitive decline, especially when problems arise in cognitive domains such as executive function and processing speed. By observing different aspects of gait, such as stride length and stride width, early signs of brain pathology can be revealed. These gait characteristics show clear associations with specific brain regions (such as the globus pallidus and prefrontal cortex), suggesting that cognitive impairment may have begun before gait changes appear. Therefore, gait, as a potential subclinical biomarker, is of significant value in the early identification of patients with cognitive impairment. Longitudinal assessment of gait changes may help identify individuals at risk of cognitive deterioration, thereby assisting clinicians in early detection of potential patients with cognitive impairment and in making referral and further diagnostic decisions. This comprehensive assessment method combining gait and cognitive function is more effective in identifying high-risk individuals for dementia than relying solely on cognitive measurements. Therefore, combining gait measurement with cognitive assessment can provide clinicians with more comprehensive information to intervene before cognitive decline occurs, which can help improve the early diagnosis rate of dementia and provide key clues for future treatment. Summary of the Invention

[0004] This invention proposes a classification method for age-related cognitive impairment based on multi-feature gait fusion. The aim is to develop a classification method for age-related cognitive impairment using deep learning, without the need for contact sensors or indicators, and to explore its potential as a convenient, objective, rapid, and non-contact screening method for cognitive impairment.

[0005] The CASIA-B gait dataset, released by the Institute of Automation, Chinese Academy of Sciences (CASIA), is a widely used gait recognition dataset aimed at promoting the research and development of gait recognition technology. The database contains images of 124 pedestrians, each captured from 0 to 180 degrees across 11 viewpoints, spaced in 18-degree intervals. Each viewpoint includes 10 walking sequences, encompassing three scenarios: wearing a coat (CL), carrying a backpack (BG), and walking normally (NM). There are 6 sequences for normal walking, and 2 sequences each for wearing a coat and carrying a backpack. A total of 1,118,373 images are included, averaging 82 images per gait sequence, with each image having a resolution of 320×240. The most commonly used testing protocol for CASIA-B is the subject-independent protocol, which uses data from the first 74 subjects for training and the remaining 50 subjects for testing. The test data was then split into Gallery and Probe. Gallery included the first four gait sequences in the NM walking scenario, while Probe consisted of the remaining sequences, namely the remaining two NM, two CL, and two BG sequences for each subject in each viewpoint.

[0006] Our model achieved good test results on the CASIA-B gait dataset, and the rationality and superiority of the proposed method were verified through comparative analysis.

[0007] The present invention achieves the above objectives through the following technical solutions:

[0008] Step 1: First, extract the gait contour map from the image dataset and preprocess it to obtain the corresponding gait skeleton map;

[0009] Step 2: Then, the gait contour map and gait skeleton map are input into the multimodal spatial feature extractor to extract and fuse the spatial features of each frame image;

[0010] Step 3: The extracted features are then horizontally segmented into 16 parts, and then downsampled using the horizontal pooling module to extract discriminative information features of the local human body.

[0011] Step 4: Next, input the downsampled features into the multi-scale temporal feature fusion unit to extract the multi-scale fused temporal features;

[0012] Step 5: Train the model using a joint loss function combining Triplet loss and SoftMax loss to obtain the final classification result. Attached Figure Description

[0013] Figure 1 This is a network structure diagram of the present invention.

[0014] Figure 2 This is a block diagram of the multimodal spatial feature extractor of the present invention.

[0015] Figure 3 This is a block diagram of the multi-scale temporal feature fusion module of the present invention. Detailed Implementation

[0016] The present invention will be further described below with reference to the accompanying drawings:

[0017] The overall structure of the proposed method is as follows: Figure 1 As shown. First, the network takes the preprocessed 128×128 gait contour sequence image and 128×128 gait skeleton sequence image as input to the model, and extracts the spatial features F of each frame image through the Multimodal Spatial Feature Extractor (MSFE). MSFE :

[0018] F MSFE =MSFE(f ske ,f sil (1)

[0019] Where f ske and f sil These represent the gait skeleton image and gait contour image obtained after preprocessing, respectively, denoted as... and MSFE consists of three modules and a multimodal spatial feature extraction module (MSFEB). The structure of the multimodal spatial feature extraction module is shown in the figure below. Figure 2 As shown, the other three modules consist of two 3×3 convolutional layers and one 2×2 max pooling layer, respectively. The specific network structure is shown in Table 1.

[0020] Table 1. Specific details of the multimodal spatial feature extractor.

[0021]

[0022] The multimodal spatial feature extractor extracts the input 128×128 gait contour sequence image f siland 128×128 gait skeletal sequence images f ske The gait contour features F are obtained by inputting them into the first convolutional neural module, block1. sil and gait skeletal features F ske The resulting contour features F sil and skeletal features F ske The input is fed into the multimodal spatial feature extraction module, where it passes through a 1×1 convolutional layer to obtain the corresponding feature maps:

[0023]

[0024] The obtained features are then fused and activated using global pooling and the Sigmoid activation function to obtain the feature vector:

[0025]

[0026] in, Here c2 = 32, symbol This refers to matrix multiplication, symbol This represents the element-wise addition of two matrices, and ε is used to avoid the case where the denominator is 0. Here, ε = 1e-5. AvgPool2d(·) represents global pooling, and Sigmoid(·) is a commonly used activation function that maps variables to the range [0,1].

[0027] After obtaining the eigenvector F MSFEB Then, it is combined with the contour feature F sil and skeletal features F ske The two processes will be merged separately, as detailed below:

[0028] F MSFEB_sil =F sil +F MSFEB *F ske (5)

[0029] F MSFEB_ske =F ske +F MSFEB *F sil (6)

[0030] Finally, F MSFEB_sil and F MSFEB_ske The data are sequentially input into two convolutional neural modules, and then fused through two consecutive 1×1 convolutional layers to obtain the final multimodal spatial features. Here, the value of c4 is c4 = 128.

[0031] Next, the features F of each frame obtained are... MSFE The horizontal segment is divided into t parts, and the feature of each part is represented by F.p ={F i |i=1,2,…,16}, where The features of each part are then fed into the horizontal pooling module HP to extract discriminative information features from the local human body. Here, the HP module processes the input features F... i Downsampling is performed using global horizontal pooling and global max pooling:

[0032] F HP =AvgPool2d(F i )+MaxPool2d(F i (7)

[0033] Then, the F obtained through downsampling HP The input is fed into a multi-scale temporal feature fusion (MTFF) and processed to obtain the multi-scale fused temporal feature F. MTFF Specifically, it can be expressed as:

[0034]

[0035] in and These represent the local temporal features, global temporal features, and multi-scale temporal features of the j-th level, respectively; the TransEncode(·) function represents the features. Global spatiotemporal features are extracted through the encoding part of the standard Transformer structure; the LeakyReLU(·) function is an improved version of the ReLU function, which has a small negative slope when the input is less than 0, so that the relevant negative information is preserved when the input is negative; the Concat(·) function represents the concatenation operation, which connects the extracted global spatiotemporal features. and local spatiotemporal features Merging and connecting channels is performed. The LayerNorm(·) function represents standard normalization of the data, normalizing the data to a mean of 0 and a standard deviation of 1. The Attention(·) function learns each channel of the input feature through the attention mechanism, so that the model pays more attention to the relatively important temporal information. The TP(·) function represents the temporal pooling operation, and here we choose TP(g) = max(g).

[0036] Finally, we choose a joint loss function, combining the Triplet loss and SoftMax loss functions, to train the proposed model. This joint loss function can be defined as:

[0037] L mul =λt L t +λ s L s (11)

[0038] Where L t and L s Let λ represent the ternary loss and the label smoothing cross-entropy loss, respectively. t and λ s These represent the weight coefficients of the Triplet loss function and the SoftMax loss function, respectively.

[0039] The triplet loss function takes a triple as input, consisting of a training sample a, a sample p (which is in the same class as the training sample), and a sample n (which is not in the same class as the training sample). Its specific formula is as follows:

[0040] L t =max(d(f) a ,f p )-d(f a ,f n (12) + margin, 0)

[0041] Among them, f a f represents the feature vector of the training samples p and f n Let represent the feature vectors of the same class and different classes as the training samples, respectively. The d(·) function is used to calculate the Euclidean distance between the two sets of feature vectors. The parameter margin takes a value greater than 0.

[0042] The SoftMax loss function combines the SoftMax function and the cross-entropy loss function. Its specific formula is shown below:

[0043]

[0044] Among them, y i,k This represents the true label of sample i belonging to the k-th category. The value is 1 if sample i belongs to category k, and 0 otherwise. i,k This represents the score of the gait image on category k; N is the total number of samples, and K is the number of categories.

[0045] On the CASIA-B dataset, the proposed method was compared with several state-of-the-art gait recognition methods, including GaitSet, GaitPart, GLN, 3DLocal, ST, and CSTL. The results show that the proposed method outperforms the comparison methods. For NM, BG, and CL, excluding the same viewpoint, the proposed method achieves higher average accuracy than the best comparison method, CSTL, reaching 98.6%, 95.1%, and 86.3%, respectively. This is because the proposed method better utilizes local pose details and multi-scale information of the sequence, thereby improving the robustness of feature representation.

[0046] Table 2. Accuracy (%) of the CAS IA-B dataset without identical views ("NM", "BG", and "CL" refer to normal walking, walking with a bag, and walking with a coat, respectively)

[0047]

[0048] To verify the effectiveness of the improved module in classifying cognitive impairment in the elderly, the improved model was applied to a self-built dataset. The dataset collected gait videos of six walking sequences from 159 subjects at 45, 90, and 135 minutes. After frame extraction, 300,000 images were obtained and preprocessed to convert them into 128×128 gait contour and gait skeleton maps. Gait images from 95 subjects were selected as the training set, and gait images from the remaining 64 subjects were used as the test set for training. Experiments were conducted to compare the model with two network models, GaitSet and GaitPart. The experimental results are shown in Table 3.

[0049] Table 3. Model recognition performance (%) on a self-built gait dataset for elderly people with cognitive impairment.

[0050]

[0051] To further verify the effect of the optimization module on improving the model's recognition performance, an ablation experiment was conducted to verify the module's effectiveness. The experimental results are shown in Table 4.

[0052] Table 4. Ablation experiments (%) validating the effectiveness of this method module

[0053]

Claims

1. A classification method for age-related cognitive impairment based on multi-feature gait fusion, characterized in that... Includes the following steps: Step 1: First, extract the gait contour map from the image dataset and preprocess it to obtain the corresponding gait skeleton map; Step 2: Then, the gait contour map and gait skeleton map are input into a multimodal spatial feature extractor to extract and fuse the spatial features of each frame of the image; Step 3: The extracted features are then horizontally segmented into 16 parts, and then downsampled using the horizontal pooling module to extract discriminative information features of the local human body. Step 4: Next, input the downsampled features into the multi-scale temporal feature fusion unit to extract the multi-scale fused temporal features; Step 5: Train the model using a joint loss function combining Triplet loss and SoftMax loss to obtain the final classification result.

2. The method described in step two of claim 1, characterized in that... A multimodal spatial feature extractor was constructed to extract spatial features. It consists of three convolutional modules and one multimodal spatial feature extraction module. The convolutional modules consist of two 3×3 convolutional layers and one 2×2 max-pooling layer. The multimodal spatial feature extraction module can be expressed as: F MSFEB_sil =F sil +F MSFEB *F ske (2) F MSFEB_ske =F ske +F MSFEB *F sil (3) In formula (1) and This represents the contour features F obtained through the first convolutional module. sil and skeletal features F ske The input is fed into the multimodal spatial feature extraction module, where it passes through a 1×1 convolutional layer to obtain the corresponding feature maps. Here c2 = 32, symbol This refers to matrix multiplication, symbol This represents the element-wise addition of two matrices, ε = 1e-5, AvgPool2d(·) represents global pooling, and Sigmoid(·) is a commonly used activation function that maps variables to the range [0,1]. In formulas (2) and (3), F MSFEB_sil and F MSFEB_ske For the eigenvector F MSFEB With contour feature F sil and skeletal features F ske They were obtained by fusing them separately.

3. The method described in step four of claim 1, characterized in that... A multi-scale temporal fusion engine was constructed to extract temporal features. The construction method is as follows: By inputting the downsampled features into a multi-scale temporal feature fusion (MTFF), the resulting multi-scale fused temporal features can be expressed as follows: in and These represent the local time features, global time features, and multi-scale time features of the j-th level part, respectively. The TransEncode(·) function represents the characteristics Global spatiotemporal features are extracted through the encoding part of the standard Transformer structure; the LeakyReLU(·) function is an improved version of the ReLU function; the Concat(·) function represents a concatenation operation, which combines the extracted global spatiotemporal features. and local spatiotemporal features Merge and connect based on the number of channels; LayerNorm(·) function represents standard normalization of the data; Attention(·) function represents attention mechanism; TP(·) function represents time pooling operation, here we choose TP(g) = max(g).

Images