A method for classifying cognitive impairment in the elderly based on contour and skeleton gait maps

Abstract

The application discloses a kind of based on contour and skeleton gait chart old cognitive impairment classification method.There is the following step: first, 120 frames of gait contour chart and skeleton chart are taken out from image data set, input to short-term time template generator, and template sequence is generated with every 4 frames as a group;Then the template sequence is input to depth spatial feature extractor, and depth spatial feature is extracted;Again, the feature is divided into 16 parts according to the level, and the horizontal pooling module is used for downsampling, and the horizontal feature vector is input to multi-scale time aggregator after being obtained, and multi-scale time feature is extracted;Finally, the model is trained by the joint loss function of ternary loss and label smoothing cross-entropy loss, and the final classification result is obtained.The application aims to develop a kind of old cognitive impairment classification method, without using contact sensor or index, and explore its potential as a kind of convenient, objective, fast and non-contact cognitive impairment screening method.

Description

Technical Field

[0001] This invention relates to a classification method for cognitive impairment in the elderly based on contour and skeletal gait maps, belonging to the field of computer vision and intelligent information technology. Background Technology

[0002] Cognitive impairment is characterized by alterations in performance on specific cognitive tasks, such as orientation, attention, comprehension, memory, reasoning, problem-solving, organizational skills, processing speed, perseverance, and motivation / activation. These alterations can affect multiple cognitive domains simultaneously or sequentially, gradually or abruptly. Cognitive impairment and dementia are leading causes of disability in older adults, and promoting healthy brain aging is considered a key factor in reducing the burden of age-related disability. However, it is estimated that 40% of dementia cases can be prevented or delayed by modifying risk factors and improving daily living activities. Routine non-cognitive assessments alone are insufficient for physicians to accurately predict a patient's cognitive function. Therefore, cognitive assessment is helpful in diagnosing impairments of thinking and in identifying potential interventions.

[0003] Bipedalism is a crucial foundation for the evolution of the human neocortex, as it represents one of the most complex and sophisticated forms of locomotion. The link between motor function and cognition can be understood to some extent from the perspective of human bipedal evolution. Gait patterns are no longer viewed as purely motor tasks but rather as complex sensorimotor behaviors significantly influenced by cognitive and emotional aspects. Previous research has reported that slower walking speeds and greater speed declines over time, independent of cognitive changes, are associated with a greater risk of dementia, supporting the role of gait speed as a potential subclinical biomarker of cognitive impairment. Furthermore, spatial, temporal, and spatiotemporal measurements of gait, along with greater variability in gait parameters, are associated with and predictable declines in global and domain-specific cognitive abilities. Previous cognitive studies have employed various sensors and machine learning analysis methods. Some have used portable gait analysis systems for walking, timing, and jumping tests. The collected gait parameters have been used in machine learning classification models based on support vector machines (SVM) and principal component analysis (PCA). Others have used 3D accelerometer data to collect 23 dynamic gait variables and used random forests (RF) and artificial neural networks (ANN) to classify cognitive impairment. Summary of the Invention

[0004] The aim of this project is to develop a machine vision-based gait recognition method for age-related diseases without the use of contact sensors or indicators, and to explore its potential as a convenient, objective, rapid, and non-contact screening tool for cognitive impairment. A series of hyperparameters used in the machine vision network for gait feature extraction and recognition were then thoroughly optimized and evaluated. ROCAUC will evaluate its performance in assessing dementia and mild cognitive impairment (MCI).

[0005] This invention proposes a classification method for age-related cognitive impairment based on contour and skeletal gait maps. The aim is to develop a classification method for age-related cognitive impairment using deep learning, without the need for contact sensors or indicators, and to explore its potential as a convenient, objective, rapid, and non-contact screening method for cognitive impairment.

[0006] The present invention achieves the above objectives through the following technical solutions:

[0007] Step 1: First, extract 30 frames of gait contour maps and corresponding skeleton maps from the image dataset, and connect them together to obtain a dual-channel gait sequence;

[0008] Step 2: Then, input the dual-channel gait sequence into the short-term time template generator to generate a template sequence;

[0009] Step 3: Input the template sequence into the depth spatial feature extractor to extract depth spatial features;

[0010] Step 4: Next, the depth space features are divided into 16 parts horizontally, and downsampled using the horizontal pooling module to obtain the horizontal feature vector;

[0011] Step 5: Input the horizontal feature vector into the multi-scale temporal aggregator to extract multi-scale temporal features;

[0012] Step 6: Train the model using a joint loss function consisting of ternary loss and label smoothing cross-entropy loss to obtain the final classification result. Attached Figure Description

[0013] Figure 1 This is a diagram of the gait recognition network structure of the present invention.

[0014] Figure 2 This is a block diagram of the depth spatial feature extractor of the present invention.

[0015] Figure 3 This is a block diagram of the multi-scale time aggregator of the present invention. Detailed Implementation

[0016] The present invention will be further described below with reference to the accompanying drawings:

[0017] The overall structure of the proposed method is as follows: Figure 1 As shown. First, the gait contour map and corresponding skeletal map of frame t are extracted from the image dataset, represented as follows. and Where t is the length of the gait sequence, which is taken as 120 in the experiment, and (h,w) is the image size of each frame. Concatenating them together yields the dual-channel gait sequence X. in It can be represented as:

[0018]

[0019] The concat(·) operator represents a join operation.

[0020] The Short-Term Temporal Template Generator (STTG) is designed to process input two-channel gait sequences. Perform image template generation, and use X in The frames are divided into groups of M frames, and then a short-term time template is generated using system random sampling. Its calculation process can be represented as:

[0021]

[0022] Where i represents the index of the frame in the gait sequence, M represents the number of selected frames (in the experiment, M = 4), and k represents the index of the frame in the gait aggregation sequence.

[0023] Next, the template sequence X STTG As input to the Deep-wise Part Feature Extractor (DSFE), spatial features F are extracted. DSFE :

[0024]

[0025] Where c1 represents the number of feature channels, DSFE consists of 3 modules. The first module contains a 2D convolutional layer and a 2D depth space convolutional layer, and the last two modules contain two 2D convolutional layers. The specific network structure is shown in Table 1.

[0026] Table 1. Specific details of the depth spatial feature extractor

[0027]

[0028] The structure of a two-dimensional depth-wise spatial convolutional layer (DS-Conv2d) is as follows: Figure 2 As shown on the right, it can be expressed as:

[0029]

[0030] Where Conv2d(·) represents two-dimensional convolution, and DW-Conv2d(·) represents two-dimensional depthwise convolution, such as Figure 2As shown in the convolution diagram on the left, depthwise convolution extracts local features from a single-channel spatial feature map. Each kernel performs convolution operations on only a single channel. DW-D-Conv2d(·) represents two-dimensional dilated convolution, a special type of depthwise convolution. By introducing dilated convolution, it increases the model's receptive field, enabling the extraction of long-range features from a single spatial feature map. The synthesis of these two methods considers local contextual information and a large receptive field, allowing for the extraction of richer spatial information from gait images.

[0031] Then, F DSFE Each frame in the image is horizontally cut into p parts, resulting in... The data is then input into the Horizontal Pooling (HP) module for spatial downsampling, and global average pooling and global max pooling are used to extract the horizontal feature vector of the human body.

[0032]

[0033] in And expressed as In the experiment, p was set to 16. This represents the temporal characteristics of the j-th level segment. Then F... HP The input is fed into a multi-scale temporal feature aggregation (MTA), such as... Figure 3 As shown in (a), the MTA consists of p parallel multi-scale temporal modules (MTMs). Each MTM is responsible for extracting features from the corresponding part of the gait sequence to obtain frame-level and short-term temporal features of the gait sequence.

[0034] Then F HP Input the data into MTM in parts, such as Figure 3 As shown in (b), frame-level temporal features are extracted. Long and short time characteristics Re-aggregated into multi-scale temporal features This can be expressed as:

[0035]

[0036]

[0037]

[0038] in and Let represent the frame-level temporal features, long-short-term temporal features, and multi-scale temporal features of the j-th horizontal part, respectively, and at this time... BatchNorm(·) normalizes the data by batch to a mean of 0 and a standard deviation of 1; BiLSTM(·) is a Bi-directional Long Short-Term Memory (BiLSTM) network that can extract past and future information from time series, thereby introducing more contextual dependencies, and performs excellently in extracting both short-term and long-term relationships; concat(·) represents a concatenation operation, which concatenates frame-level temporal features F f With long and short time characteristics F ls The connections are made along the channel dimension; Attention(·) is the attention mechanism used, which focuses on the relationships between channels and weights the features along the channel dimension. The larger the weight, the higher the correlation between the channel and key temporal information. TP(·) represents temporal pooling, where TP(·) = max(·).

[0039] By extracting frame-level and short-term temporal features, abstract features at different scales in gait sequences are captured, and then an attention mechanism is used to aggregate more distinctive temporal information.

[0040] Finally, we choose a joint loss function of ternary loss and label smoothing cross-entropy loss to train the proposed model. This joint loss function L mul It can be defined as:

[0041] L mul =λ tri L tri +λ cro L cro (9)

[0042] Where L tri and L cro Let λ represent the ternary loss and the label smoothing cross-entropy loss, respectively. tri and λ cro These represent the weight coefficients of the loss function.

[0043] Ternary loss function L tri The input is a triple consisting of an anchor sample A, a positive sample P with the same identity as the anchor sample, and a negative sample N with a different identity from the anchor sample. The classification problem is solved by reducing the distance between samples A and P with the same identity and increasing the distance between samples A and N with different identities. tri The calculation process can be represented as:

[0044] L tri =max(D(f(X) A ),f(X P ))-D(f(XA ),f(X N ))+α,0) (10)

[0045] Where D(·) represents the calculation of Euclidean distance; X A X P and X N These represent the feature vectors of the anchor sample, positive sample, and negative sample, respectively; the parameter α is the control value of the distance between the positive and negative samples, which is a constant greater than 0 and represents the margin of the ternary loss.

[0046] Label smoothing cross-entropy loss function L cr The input to o is the predicted probability of the anchor sample, and the goal is to make the model output as close as possible to the true output. The common cross-entropy loss function L... c It can be expressed as:

[0047]

[0048]

[0049] in, This represents the predicted probability that anchor sample A belongs to the true class c; in ordinary cross-entropy loss, It is a label parameter; when the true category of A is c... The value is 1; otherwise, it is 0.

[0050] This one-hot label calculation method can lead to an overemphasis on the correct class during training, neglecting information from other classes. To address this issue and improve the model's generalization ability, a label smoothing regularization method is introduced. The values ​​of the labels are smoothed to optimize the cross-entropy loss. The smoothed label function is then used. It can be represented as:

[0051]

[0052]

[0053] Where ε is a constant, and in the experiment, ε = 0.1 is taken.

[0054] Experiments were conducted using the CASIA-B gait database created by the Pattern Recognition Laboratory of the Institute of Automation, Chinese Academy of Sciences. This database contains images of 124 pedestrians, each captured from 11 perspectives ranging from 0 to 180 degrees, with each perspective spaced 18 degrees apart. Each perspective includes 10 walking sequences, encompassing three scenarios: wearing a coat (CL), carrying a backpack (BG), and walking normally (NM). There are 6 sets of gait sequences for normal walking, and 2 sets each for wearing a coat and carrying a backpack. A total of 1,118,373 images are included, with an average of 82 images per gait sequence, and each image has a resolution of 320×240.

[0055] The most commonly used testing protocol for CASIA-B is the subject-independent protocol, which is trained using data from the first 74 subjects and tested on the remaining 50 subjects. The test data is then split into Gallery and Probe. Gallery includes the first four gait sequences in the NM walking scenario, while Probe consists of the remaining sequences, namely the remaining two NM, two CL, and two BG sequences for each subject in each viewpoint.

[0056] On the CASIA-B dataset, the proposed method was compared with several state-of-the-art gait recognition methods, including GaitSet, GaitPart, MT3D, and 3D Local. The results show that the proposed method outperforms the comparison methods. For NM, BG, and CL, excluding the same viewpoint, the proposed method achieves higher average accuracy than the best comparison method, 3D Local, reaching 98.8%, 95.2%, and 84.5%, respectively. This is because the proposed method better utilizes local pose details and multi-scale information of the sequence, thereby improving the robustness of feature representation.

[0057] Table 2. Accuracy (%) of the CASIA-B dataset when the same view is not included. “NM”, “BG”, and “CL” refer to normal walking, walking with a bag, and walking with a coat, respectively.

[0058]

[0059] Table 3 shows the classification results of three network models on the gait dataset of elderly people with cognitive impairment. AUC (Area Under Curve) is defined as the area under the ROC curve and the coordinate axis. AUC is a performance metric for evaluating the quality of a learner, representing the probability that a predicted positive example is ranked before a negative example. As can be seen from the table, the ROC values ​​of all three models for elderly gait features are above 80%, and the network of this invention has higher recognition performance.

[0060] The results verified a significant correlation between gait characteristics and cognitive impairment levels in older adults, demonstrating that the method of classifying cognitive impairment levels by analyzing gait characteristics has the potential to serve as a convenient, objective, rapid, and non-contact screening method for cognitive impairment.

[0061] Table 3. Binary AUC intervals (0.95 confidence level) for the gait dataset of elderly people with cognitive impairment

[0062]

Claims

1. A method for classifying age-related cognitive impairment based on contour and skeletal gait maps, characterized in that... Includes the following steps: Step 1: Extract the gait contour map and corresponding skeleton map of frame t from the image dataset, represented as follows: and Where t is the length of the gait sequence, and in the experiment, t = 120 is used; (h, w) is the image size of each frame. The gait contour map is connected with the corresponding skeleton map to obtain the dual-channel gait sequence X. in It is represented as: Where concat(·) represents a join operation; The Short-term Temporal Template Generator (STTG) is designed to process input two-channel gait sequences. Perform image template generation, and use X in Dividing the data into groups of M frames, a short-term time template is generated using system random sampling. The calculation process is represented as follows: Where i represents the index of the frame in the gait sequence, M represents the number of selected frames (in the experiment, M = 4), and k represents the index of the frame in the gait aggregation sequence. Step 2: Convert the template sequence X STTG As input to the Deep-wise Part Feature Extractor (DSFE), spatial features F are extracted. DSFE : Where c1 represents the number of feature channels, DSFE consists of 3 modules. The first module contains one 2D convolutional layer and one 2D depth-space convolutional layer, and the latter two modules contain two 2D convolutional layers. The structure of the 2D depth-space convolutional layer (DS-Conv2d) is expressed as follows: Where Conv2d(·) represents two-dimensional convolution, and DW-Conv2d(·) represents two-dimensional depthwise convolution; Step 3: Place F DSFE Each frame in the image is horizontally cut into p parts, resulting in... The data is then input into the Horizontal Pooling (HP) module for spatial downsampling, and global average pooling and global max pooling are used to extract the horizontal feature vector of the human body. in And expressed as In the experiment, p was set to 16. Represent the temporal characteristics of the j-th level part; then F HP The input is fed into a multi-scale temporal feature aggregation (MTA), which consists of p parallel multi-scale temporal modules (MTMs). Each MTM is responsible for extracting features from the corresponding part of the gait sequence, obtaining frame-level and short- and long-term temporal features of the gait sequence, and then converting the F... HP The components are input into the MTM to extract frame-level temporal features. Long and short time characteristics Re-aggregated into multi-scale temporal features Expressed as: in and Let represent the frame-level temporal features, long-short-term temporal features, and multi-scale temporal features of the j-th horizontal part, respectively, and at this time... BatchNorm(·) normalizes the data to a mean of 0 and a standard deviation of 1 by batch; BiLSTM(·) is a Bi-directional Long Short-Term Memory (BiLSTM) network; concat(·) represents a concatenation operation, which combines the frame-level temporal features F f With long and short time characteristics F ls Connect along the channel dimension; Attention(·) is the attention mechanism used, which focuses on the relationship between channels and performs feature weighting on the channel dimension. TP(·) represents temporal pooling, where TP(·) = max(·). Step 4: Select a joint loss function of ternary loss and label smoothing cross-entropy loss to train the proposed model. This joint loss function L mul Defined as: L mul =λ tri L tri +λ cro L cro (9) Where L tri and L cro Let λ represent the ternary loss and the label smoothing cross-entropy loss, respectively. tri and λ cro These represent the weight coefficients of the loss function, and the final classification result is obtained based on the trained model.

2. The method according to step one of claim 1, characterized in that... The feature sequence was generated by using a combination of multimodal gait of contour and skeleton as input, and templates were generated by random sampling of the system.

Images