Old people cognitive impairment classification method based on contour and skeleton gait map

Abstract

The invention discloses an elderly cognitive impairment classification method based on a contour and skeleton gait map. Comprising the following steps of: firstly, extracting 120 frames of gait profile diagrams and skeleton diagrams from an image data set, inputting the gait profile diagrams and skeleton diagrams into a short-term time template generator, and generating a template sequence by taking every four frames as a group; inputting the template sequence into a depth space feature extractor, and extracting depth space features; horizontally dividing the features into 16 parts, performing down-sampling by using a horizontal pooling module to obtain horizontal feature vectors, inputting the horizontal feature vectors into a multi-scale time aggregator, and extracting multi-scale time features; and finally, training the model through a joint loss function of ternary loss and label smooth cross entropy loss to obtain a final classification result. The invention aims to develop a senile cognitive impairment classification method without using a contact sensor or index, and explore the potential of the senile cognitive impairment classification method as a convenient, objective, rapid and non-contact cognitive impairment screening method.

Description

Technical Field

[0001] The invention relates to a method for classifying elderly cognitive impairment based on contour and skeleton gait diagram, and belongs to the field of computer vision and intelligent information technology. Background Art

[0002] Cognitive impairment is characterized by changes in performance in specific cognitive tasks, such as orientation, attention, comprehension difficulties, memory, reasoning, problem solving, organizational skills, processing speed, perseverance, and motivation / initiation, which can affect multiple cognitive domains simultaneously or sequentially, gradually or suddenly. Cognitive impairment and dementia are the leading causes of disability in older adults, and promoting healthy brain aging is considered a key factor in reducing the burden of age-related disability. However, it is estimated that 40% of dementia can be prevented or delayed by modifying its risk factors and improving daily living activities. Routine non-cognitive assessments alone are insufficient for physicians to accurately predict a patient's cognitive function. Therefore, cognitive assessments can aid in the diagnosis and potential intervention of disorders that impair thinking.

[0003] Bipedalism is an important foundation for the evolution of the human neocortex, as it is one of the most complex and sophisticated of all movements. The connection between motor function and cognition can be understood in part from the perspective of human bipedal evolution. Gait patterns are no longer viewed as purely motor tasks, but rather as complex sensorimotor behaviors that are heavily influenced by cognitive and affective aspects. Previous studies have reported that slower walking speed and greater speed decline over time, independent of cognitive changes, are associated with a greater risk of dementia, supporting the role of gait speed as a possible subclinical marker of cognitive impairment. In addition, spatial, temporal, and spatiotemporal measures of gait and greater variability in gait parameters are associated with and predictive of global and domain-specific cognitive decline. Different sensors and machine learning analysis methods have been used in previous cognitive studies. There are portable gait analysis systems for walking, timing, and jumping tests. The collected gait parameters were used in machine learning classification models based on support vector machines (SVM) and principal component analysis (PCA). There are also 23 dynamic gait variables collected using 3D accelerometer data, and random forests (RF) and artificial neural networks (ANN) were used to classify cognitive impairment. Summary of the invention

[0004] The aim of this project is to develop a machine vision-based gait recognition method for geriatric diseases without the use of contact sensors or indicators, and to explore its potential as a convenient, objective, rapid and contactless screening tool for cognitive impairment. A range of hyperparameters for gait feature extraction and recognition in the machine vision network are then deeply optimized and evaluated. ROCAUC will evaluate the performance in the assessment of dementia and mild cognitive impairment (MCI).

[0005] This paper proposes a classification method for cognitive impairment in the elderly based on contour and skeletal gait graphs. The purpose is to develop a classification method for cognitive impairment in the elderly using deep learning methods without the use of contact sensors or indicators, and to explore its potential as a convenient, objective, rapid and non-contact method for screening cognitive impairment.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0007] Step 1: First, extract 30 frames of gait contour images and corresponding skeleton images from the image dataset and connect them together to obtain a dual-channel gait sequence;

[0008] Step 2: Then input the dual-channel gait sequence into the short-term template generator to generate a template sequence;

[0009] Step 3: Input the template sequence into the deep space feature extractor to extract the deep space features;

[0010] Step 4: Then divide the deep spatial features into 16 parts horizontally and downsample them using the horizontal pooling module to obtain the horizontal feature vector;

[0011] Step 5: Input the horizontal feature vector into the multi-scale time aggregator to extract multi-scale time features;

[0012] Step 6: Train the model through the joint loss function of ternary loss and label smoothed cross entropy loss to obtain the final classification result. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a diagram of the gait recognition network structure of the present invention.

[0014] Figure 2 This is a module diagram of the deep spatial feature extractor of the present invention.

[0015] Figure 3 This is a module diagram of the multi-scale time aggregator of the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further described below in conjunction with the accompanying drawings:

[0017] The overall structure of the proposed method is as follows Figure 1 First, we extract the gait profile and the corresponding skeleton image of frame t from the image dataset, which is represented as and Where t is the length of the gait sequence, t = 120 in the experiment, (h, w) is the image size of each frame. Connect them together to get the dual-channel gait sequence X in , which can be expressed as:

[0018]

[0019] Where concat(·) represents a concatenation operation.

[0020] The Short-term Temporal Template Generator (STTG) is designed to input a dual-channel gait sequence. Generate image template and set X in Divide each M frames into a group, and then use the system random sampling method to generate a short-term time template to obtain Its calculation process can be expressed as:

[0021]

[0022] Where i represents the index of the frame in the gait sequence, M represents the number of selected frames, and M=4 is taken in the experiment. k represents the index of the frame in the gait aggregation sequence.

[0023] Next, the template sequence X STTG As the input of the Depth-wise Part Feature Extractor (DSFE), the spatial feature F is extracted DSFE :

[0024]

[0025] where c 1 Indicates the number of feature channels. DSFE consists of three modules. The first module contains a two-dimensional convolutional layer and a two-dimensional deep spatial convolutional layer. The last two modules contain two two-dimensional convolutional layers. The specific network structure is shown in Table 1.

[0026] Table 1 Details of the deep spatial feature extractor

[0027]

[0028] The structure of the two-dimensional deep spatial convolution layer (Depth-wise Spatial Conv2d, DS-Conv2d) is as follows Figure 2 As shown on the right, it can be expressed as:

[0029]

[0030] Where Conv2d(·) represents a two-dimensional convolution, and DW-Conv2d(·) represents a two-dimensional depth convolution, such as Figure 2As shown in the convolution diagram on the left, deep convolution is the extraction of local features of a single-channel spatial feature map, and each of its convolution kernels only performs convolution operations on a single channel. DW-D-Conv2d(·) represents a two-dimensional deep hole convolution, which is a special deep convolution. By introducing hole convolution, the receptive field of the model is increased, and the long-range features of a single spatial feature map can be extracted. The synthesis of the two takes into account local context information and a large receptive field, and can extract richer spatial information from gait images.

[0031] Then, F DSFE Each frame in is cut horizontally into p parts, and we get Then it is input into the horizontal pooling module (HP) for downsampling of the spatial dimension, and the horizontal feature vector of the human body is extracted through global average pooling and global maximum pooling:

[0032]

[0033] in And expressed as In the experiment, p is set to 16. represents the time series characteristics of the jth horizontal part. Then F HP Input to the Multi-scale Temporal Feature Aggregation (MTA), such as Figure 3 As shown in (a), MTA consists of p parallel multi-scale temporal modules (MTMs). Each MTM is responsible for extracting features from the corresponding part of the gait sequence and obtaining the frame-level and long- and short-term temporal features of the gait sequence.

[0034] Then F HP Enter the parts into MTM, such as Figure 3 As shown in (b), frame-level temporal features are extracted Long- and short-term characteristics Reaggregate into multi-scale temporal features It can be expressed as:

[0035]

[0036]

[0037]

[0038] in and Respectively represent the frame-level temporal features, long-term and short-term temporal features, and multi-scale temporal features of the j-th horizontal part, and at this time BatchNorm(·) means that the data is standardized to a mean of 0 and a standard deviation of 1 by batch. BiLSTM(·) is a bidirectional long short-term memory network (BiLSTM), which can obtain past and future information in the time series, thereby introducing more contextual dependencies and excels in extracting both short-term and long-term relationships. concat(·) represents a concatenation operation, which combines the frame-level temporal features F f and long-term and short-term time characteristics F ls Connect along the channel dimension; Attention(·) is the attention mechanism used. By introducing the attention mechanism, we focus on the relationship between channels and perform feature weighting on the channel dimension. The larger the weight, the higher the correlation between the channel and the key time information. TP(·) represents time pooling, where TP(·) = max(·).

[0039] By extracting frame-level and long- and short-term temporal features, abstract features at different scales in the gait sequence are captured, and then the attention mechanism is used to aggregate more distinctive temporal information.

[0040] Finally, we choose the joint loss function of triplet loss and label smoothed cross entropy loss to train the proposed model. The joint loss function L mul Can be defined as:

[0041] L mul =λ tri L tri +λ cro L cro (9)

[0042] Where L tri and L cro denote the ternary loss and label smoothed cross entropy loss, respectively, tri and λ cro They represent the weight coefficients of the loss function respectively.

[0043] The ternary loss function L tri The input of L is a triplet, including the anchor sample A, the positive sample P with the same identity as the anchor sample, and the negative sample N with a different identity from the anchor sample. The classification problem is solved by reducing the distance between samples A and P with the same identity and expanding the distance between different samples A and N. tri The calculation process can be expressed as:

[0044] L tri =max(D(f(X A ),f(X P ))-D(f(XA ),f(X N ))+α,0) (10)

[0045] Where D(·) represents the calculation of Euclidean distance; X A , X P and X N Represent the feature vectors of anchor samples, positive samples, and negative samples respectively; the parameter α is the control value of the distance between positive and negative samples, which is a constant greater than 0 and represents the margin of the ternary loss.

[0046] Label smoothed cross entropy loss function L cr The input of o is the predicted probability of the anchor sample, and the model output is made as close to the true output as possible. The ordinary cross entropy loss function L c can be expressed as:

[0047]

[0048]

[0049] in, Represents the predicted probability that the anchor sample A belongs to the true category c; in the ordinary cross entropy loss, is a label parameter. When the true category of A is c, The value is 1, otherwise it will be 0.

[0050] This one-hot label calculation method will cause excessive focus on the correct category information during training, thereby ignoring the information of other categories. In order to solve the above problem and improve the generalization ability of the model, the label smoothing regularization method is introduced. The value of is smoothed to optimize the cross entropy loss. The smoothed label function can be expressed as:

[0051]

[0052]

[0053] Here, ε is a constant, and ε=0.1 is taken in the experiment.

[0054] The CASIA-B gait database created by the Pattern Recognition Laboratory of the Institute of Automation, Chinese Academy of Sciences was used for related experiments. The database contains 124 pedestrians, and each pedestrian was photographed from 11 viewing angles from 0 to 180 degrees, with an interval of 18 degrees. Each viewing angle includes 10 walking sequences, including three walking scenes: wearing a coat (CL), a backpack (BG), and normal walking (NM). There are 6 sets of gait data for normal walking, and 2 sets of gait sequences for wearing a coat and a backpack respectively. There are a total of 1,118,373 images, with an average of 82 images per gait sequence, and the resolution of each image is 320×240.

[0055] The most commonly used test protocol for CASIA-B is the subject-independent protocol, which uses the data of the first 74 subjects for training and the remaining 50 subjects for testing. The test data is then split into Gallery and Probe. The Gallery includes the first four gait sequences in the NM walking scene, and the Probe consists of the remaining sequences, i.e., the remaining 2 NM, 2 CL, and 2 BG sequences for each subject per view.

[0056] On the CASIA-B dataset, the proposed method is compared with several state-of-the-art gait recognition methods, including GaitSet, GaitPart, MT3D, and 3D Local. The results show that the proposed method outperforms the compared methods. For NM, BG, and CL, excluding the same view angle, the proposed method has higher average precision than the best method 3D Local in the compared methods, reaching 98.8%, 95.2%, and 84.5%, respectively. This is because the proposed method better utilizes the local posture details and the multi-scale information of the sequence, thereby improving the robustness of the feature representation.

[0057] Table 2 Accuracy (%) of CASIA-B dataset without excluding the same view. “NM”, “BG” and “CL” refer to normal walking, walking with bag and walking with coat, respectively.

[0058]

[0059] Table 3 shows the classification results of the three network models on the elderly cognitive impairment gait data set. AUC (Area Under Curve) is defined as the area under the ROC curve and the coordinate axis. AUC is a performance indicator to measure the quality of the learner, indicating the probability that the predicted positive example is ranked before the negative example. It can be seen from the table that the RUC values ​​of the three models for the elderly gait characteristics are all above 80%, and the network of the present invention has higher recognition performance.

[0060] The results verified that there was an important correlation between the gait characteristics of the elderly and their cognitive impairment level, proving that the method of classifying the level of cognitive impairment by analyzing gait characteristics has the potential to serve as a convenient, objective, rapid and non-contact method for screening cognitive impairment.

[0061] Table 3 Binary classification AUC interval of elderly cognitive impairment gait dataset (0.95 confidence level)

[0062]

Claims

1. A method for classifying cognitive impairment in the elderly based on contour and skeletal gait graphs, characterized in that The following steps are involved: Step 1: First, take out 120 frames of gait contour images and corresponding skeleton images from the image dataset, connect them together, and get a dual-channel gait sequence, which is then input into the short-term template generator to generate a template sequence in groups of 4 frames. Step 2: Input the template sequence into the deep spatial feature extractor to extract the deep spatial features; Step 3: Divide the deep spatial features into 16 parts horizontally and downsample them using the horizontal pooling module. After obtaining the horizontal feature vector, it is input into the multi-scale time aggregator to extract multi-scale time features. Step 4: Train the model through the joint loss function of ternary loss and label smoothed cross entropy loss to obtain the final classification result.

2. The method according to step 1 of claim 1, characterized in that A multimodal gait combination of silhouette and skeleton is used as the input feature sequence, and template generation is performed through systematic random sampling.

3. The method according to step 2 of claim 1, characterized in that A deep spatial feature extractor is constructed to extract spatial features: it consists of three modules. The first module contains a two-dimensional convolutional layer, a two-dimensional deep spatial convolutional layer, and the last two modules contain two two-dimensional convolutional layers; the two-dimensional deep spatial convolutional layer can be expressed as: Among them, Conv2d(·) represents two-dimensional convolution, DW-Conv2d(·) represents two-dimensional depth convolution, and DW-D-Conv2d(·) represents two-dimensional depth hole convolution.

4. The method according to step 3 of claim 1, characterized in that A multi-scale time aggregator is constructed to extract time features. The construction method is as follows: MTA consists of 16 parallel multi-scale temporal modules (MTMs). Each MTM is responsible for extracting features from the corresponding part of the gait sequence and extracting frame-level temporal features. Long-term and short-term characteristics Reaggregate into multi-scale temporal features It can be expressed as: in and denote the frame-level temporal features, long-term and short-term temporal features, and multi-scale temporal features of the j-th horizontal part, respectively; BatchNorm(·) denotes that the data is standardized to a mean of 0 and a standard deviation of 1 by batch; BiLSTM(·) is a bidirectional long short-term memory network (BiLSTM); concat(·) denotes a concatenation operation, which combines the frame-level temporal features F f and long-term and short-term time characteristics F ls Connect along the channel dimension; Attention(·) is the attention mechanism used; TP(·) represents temporal pooling, where TP(·) = max(·).

Images