A gait feature-based cognitive impairment classification method for the elderly

Abstract

The application discloses a gait feature-based cognitive impairment classification method for the elderly. The method comprises the following steps: first, a new instance segmentation network C-BlendMask is constructed by combining the BottleneckTransformer and CoordConv structures on the basis of the instance segmentation network BlendMask, and the new instance segmentation network C-BlendMask is applied to a self-built dataset. Second, a gait recognition network Res-GaitSet is constructed by combining a residual structure and a joint training loss function, and the gait recognition network Res-GaitSet is used to extract gait features of the self-built dataset. Finally, a classification network with shared filters is constructed for gait feature classification of the self-built dataset, so as to preliminarily evaluate the cognitive impairment level of the elderly patients. Compared with the prior art, the method can quickly identify the cognitive impairment level of the elderly while not affecting the patients, and has a wide application prospect in gait recognition, health monitoring of the elderly and the like.

Description

TECHNICAL FIELD

[0001] The application relates to a gait feature-based cognitive impairment grade classification method, and belongs to the fields of computer vision and intelligent information technology. BACKGROUND

[0002] Cognitive impairment is characterized by a decline in the ability to perceive and understand the world around us. In clinical medicine, patients with cognitive impairment are grouped according to specific symptoms and signs. Dementia, mental retardation, aphasia, amnesia and delirium syndromes can all be characterized by cognitive impairment. The dementia we refer to is the overall degradation of an individual's intellectual function in a state of clear consciousness. According to the degree of disease progression, the process of cognitive impairment is mainly divided into three stages: from normal cognitive function of aging to mild cognitive impairment (MCI), and then to severe cognitive impairment (early dementia). The diagnosis of cognitive impairment is easily limited by the environment and conditions, and requires comprehensive evaluation and analysis by a doctor. There is currently no specific treatment drug and effective diagnosis and treatment method for this disease. Therefore, timely intervention measures and precise prevention strategies for patients with early cognitive impairment can effectively slow down the symptoms and delay the onset of the disease. It can also help patients with cognitive impairment to be diagnosed in time and receive good treatment and appropriate care, thereby delaying the onset of early dementia and reducing the prevalence rate.

[0003] Instance segmentation is the inevitable result of the development of target detection and semantic segmentation from coarse-grained reasoning to fine-grained reasoning, and is a research hotspot in the field of computer vision. Target detection or positioning refers to the gradual development process of digital image reasoning (i.e., from coarse-grained reasoning to fine-grained reasoning). When detecting target objects, it provides the category and location information of the target object in the image, and the specific location information is provided in the form of center coordinates or boundary coordinates. The semantic segmentation task is to assign each pixel of all instance objects in the target image to the class to which it belongs, and objects of the same class have the same class label. With in-depth research on segmentation technology, it is found that instance segmentation can distinguish different individuals in the same class of objects in more detail and provide different labels for them. Therefore, instance segmentation technology is equivalent to the combination of target detection technology and semantic segmentation technology, and solves both problems. Instance segmentation methods can be divided into two categories: candidate box-based methods and non-candidate box-based methods. The former is to first coarsely position the candidate box in the image, and then perform pixel-level template segmentation to obtain a bounding box-based object instance. Faster-RCNN is a typical candidate box-based instance segmentation method. The non-candidate box-based segmentation method first uses a multi-layer neural network to perform pixel-level prediction on instance object information, and then obtains instance results through pixel clustering. However, how to obtain suitable instance object information has always been the focus and difficulty of research. In addition, on simple scene datasets, non-candidate box-based methods have higher segmentation accuracy than candidate box-based methods, but in scenarios with many object categories and complex datasets, non-candidate box-based methods have obvious shortcomings. Therefore, how to learn more robust image features in a limited space and complex target is a problem worth studying. In addition, although convolutional neural networks (CNN) can meet the requirements of most instance segmentation tasks, when the same class of instance objects are close in the scene, how to ensure the effectiveness of network training is a problem worth studying.

[0004] Gait recognition refers to the process of identifying the identity of a target person according to his or her unique walking style during walking. Gait recognition can be used for gender recognition, age classification, gait-based clinical disease diagnosis, video monitoring and other different applications. Gait recognition methods can be roughly divided into the following three categories: model-based methods, appearance-based methods, and pose estimation methods.

[0005] As a kind of feedforward neural network, CNN uses convolutional layer to extract sample features, and carries out data mapping through nonlinear activation function. Its input sample is in the form of two-dimensional vector, and the corresponding sample features are extracted through multiple convolution and superposition layers. Then, the extracted features are expanded into one-dimensional vector and enter the complete link layer. Finally, different output functions are selected according to the task requirements. The image classification model includes two main processes of forward process and backward process. Among them, the forward process completes two tasks of feature extraction and target classification. The backward process is the backward propagation of classification error and weight update. The difference between the predicted value and the true value is used to update the parameters, and the model is trained through the forward propagation and backward propagation process. SUMMARY

[0006] The application provides a cognitive impairment grade classification method based on gait features. The purpose is to use deep learning methods to classify different cognitive impairment grades of self-built gait data sets.

[0007] The application achieves the above-mentioned purposes through the following technical solutions:

[0008] (1) On the basis of the full convolution instance segmentation network BlendMask, the new instance segmentation network C-BlendMask is constructed by combining the BottleneckTransformer structure and the CoordConv convolution layer, and the self-built data set is subjected to image instance segmentation.

[0009] (2) On the basis of the gait recognition network GaitSet, the new gait recognition network Res-GaitSet is constructed by combining the residual unit and the multiple loss function training mode, and the gait features in the gait data set are extracted.

[0010] (3) On the basis of the full convolution classification network ConvNeXt, the new gait feature classification network TBC-ConvNeXt is designed and built by combining the shared filter structure, the gait data set is classified, and the final classification result is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 It is an instance segmentation network based on the Bottleneck Transformer structure and the CoodConv convolution layer of the application.

[0012] Figure 2 It is a gait recognition network structure based on the residual unit and the multiple loss function of the application.

[0013] Figure 3 It is a full convolution classification network structure based on the shared filter structure of the application. DETAILED DESCRIPTION

[0014] The application is further described below with reference to the accompanying drawings:

[0015] The BlendMask network adopts a similar structure to other instance segmentation networks based on candidate boxes, and adds a bottom module for predicting scores in the network. The shape of the bottom module is N x K x H / s x W / s, where N is the batch size, K is the number of bottom score maps, H x W is the size of the input, and s refers to the step length of the score map output. The network uses DeepLab V3+ as the decoder. Traditional semantic segmentation networks can be used as the backbone features, which are input into the bottom module. YOLACT and pyramid feature network (FPN) can also be used as the input of the bottom module. The attention information of the top layer is predicted through an independent convolution layer on each detection tower. The blender, as the core module of the BlendMask network, takes the score map output by the bottom module, the attention information of the top layer, and the candidate box in the image as input, and outputs the final prediction result through attention with position information.

[0016] First, the clipping module in MaskR-CNN is used to clip the score map P d for each candidate box to a fixed size R x R feature map r d . The definition of r d is as follows:

[0017]

[0018] After the ROIAlign operation on the top layer attention, the interpolation a d is obtained. Using the interpolation a d , the size of the attention map is changed from the original M x M to R x R, where R = {r d |d = 1,...,D}.

[0019]

[0020] Then, the softmax function is used to normalize a' d along the Kth dimension to make it a set of score maps s d .

[0021]

[0022] Finally, in each entity r d of the region R and the score S, s dThe operation of element-wise multiplication is used between them, and summation is performed along the Kth dimension, and finally the mask logit·m is obtained d .

[0023]

[0024] where k is the index of the bias.

[0025] The global information aggregation ability of the Transformer is introduced, and the multi-head self-attention module (MHSA) is used to replace the simple 3x3 spatial convolution layer in the original segmentation network backbone structure based on the convolutional neural network. MSHA treats the position encoding as spatial attention, and all attentions are performed on the two-dimensional feature map. The relative position encodings R h and R w are regarded as spatial attention in different dimensions, and the relative position encodings R h and R w are two learnable vectors. After being added and fused into the same dimension and size as the spatial vector q, they are multiplied by the vector q, and the obtained qr T and qk T are added to obtain the attention logic. This makes MHSA pay more attention to the area that needs attention, and is easier to converge. And MHSA only introduces a multi-head structure in the blue area of the graph.

[0026] The CoordConv layer can be implemented as a simple extension of standard convolution, where additional channels are instantiated and filled with constant, untrained coordinate information, which are then concatenated channel-wise to the input representation and a standard convolution layer is applied. Specifically, the i-coordinate channel is a matrix of size h x w with rank -1, whose first row is filled with 0, the second row is filled with 1, the third row is filled with 2, and so on. The j-coordinate channel is similar, but the columns are filled with constant values while the rows are not. In all experiments, the i and j coordinate values are finally linearly scaled so that their values fall within the range [-1, 1]. For two-dimensional convolution, two (i, j) coordinates are sufficient to completely specify an input pixel, but more channels can be added to make the model prefer to learn a specific solution if necessary. If there is a need for more experiments, a third usable channel can be provided for the r coordinate, which can be represented as:

[0027]

[0028] The CoordConv convolution layer is added to the input module of the segmentation network C-BlendMask to better accurately segment the gait image.

[0029] Because the self-built dataset CI-Gait lacks corresponding instance segmentation labels, the comparative experiments of this invention were conducted on the public dataset COCO. Table 1 shows the segmentation experimental results of each model on the COCO dataset. Among them, Mask-RCNN is a flexible and efficient classic instance segmentation model, BlendMask is the basic instance segmentation model used in this invention, and C-BlendMask is the improved model of this invention.

[0030] Table 1. COCO dataset segmentation results (%)

[0031]

[0032] As can be seen from Table 1, the improved network C-BlendMask of this invention, in addition to the AP index, 50 Compared to the basic network BlendMask, the performance decreased by 0.3%, with improvements in AP and AP. 75 AP S AP M AP L All three networks show varying degrees of improvement over Mask-RCNN and the base network BlendMask, especially in segmentation of large-area objects, where they significantly outperform Mask-RCNN and BlendMask. When using the BoT-50 backbone network, the AP (Average Performance) metric is [not specified]. L Compared to the basic network BlendMask, it improved by about 2%, while when using the BoT-101 backbone network, the AP metric was... L It offers a 3.6% improvement over the basic network BlendMask.

[0033] To quantitatively verify the segmentation performance of the improved network model C-BlendMask on the self-built dataset CI-Gait, this invention employs the method of calculating the structural similarity (SSIM) between images in the manually segmented dataset Manual-Gait and their corresponding neural network-segmented images to verify the accuracy of the network model's segmentation. Each image in the Manual-Gait dataset achieved a segmentation similarity of over 95% with its corresponding neural network-segmented image. This demonstrates that the improved model C-BlendMask also achieves excellent segmentation performance for gait images in the CI-Gait dataset, proving the effectiveness and good generalization ability of the improved network.

[0034] The GaitSet model takes a set of gait contours as input. First, a convolutional neural network is used to independently extract local information (i.e., frame-level features) from different contour maps. Second, an operation called SetPooling (SP) is used to aggregate the local information into global information (i.e., individual sequence-level features). This operation is not applicable to the original contour maps but only to high-dimensional feature maps. Based on this property, the temporal and spatial information of the image can be better preserved relative to the gait template. Third, a structure called HorizontalPyramid Mapping (HPM) is used to map the sequence-level features to a more discriminative space to obtain the final deep ensemble representation.

[0035] Consider gait as a set. Given a dataset containing N people, each with an identity y. i Let i∈1,2,...,N, and represent the gait profile distribution of a person as P. i And assume distribution P i The pattern is solely related to the person's identity. Therefore, all the contours in a person's sequence can be considered as n contours. The set of, where Under this assumption, gait recognition can be divided into three steps, expressed by the following formula:

[0036] f i =H(G(F(X) i ))) (6)

[0037] Here, F is a convolutional network that extracts local information (i.e., frame-level features) from the gait contour map. Function G is a permutation-invariant function implemented by the SP module, used to map a set of local information (i.e., frame-level features) to global information (i.e., sequence-level features). Function H is used to learn P from ensemble-level features. i The discriminant representation is implemented using the HPM structure. Input X i It is a tensor with four dimensions: set dimension, image channel dimension, image height dimension, and image width dimension.

[0038] The original GaitSet network uses a simple 3×3 convolution operation for frame-level feature extraction. With few convolutional layers and a shallow network, the extracted features are insufficient to capture deeper image features. However, directly deepening the network by stacking simple convolutional layers can lead to phenomena such as "gradient explosion" and "gradient vanishing," resulting in non-convergence, poor optimization performance, and reduced accuracy on training and testing data. To address this issue, Kaiming He et al. proposed a novel residual network, ResNet, which is constructed by stacking multiple residual blocks. By adding an identity mapping layer on top of a shallow network, it can deepen the network while avoiding network degradation. The improved network, Res-GaitSet, uses residual units, typically found in deeper networks like ResNet50, to replace the original simple convolutional operations.

[0039] The original network uses Triplet Loss as the loss function. Its advantage lies in detail discrimination; that is, when two inputs are similar, Triplet loss can better model details, essentially adding a measure of the difference between the two inputs, allowing for more effective learning of the input. The input is a triple <a, p, n>, where a is an anchor example, p is a positive sample point with the same label as the anchor, and n is a negative sample point with a different label than the anchor. The function maximizes the difference L between the anchor and the negative sample points. n Reduce the difference L between the anchor point and the positive sample point p And make L n <L p This is used to calculate the similarity between samples. Triplet Loss is generally used for calculating the similarity between samples with small differences, and its definition is as follows:

[0040]

[0041] Here, α is the control value for the distance between positive and negative samples. When the distance between the anchor point and the negative sample is less than the distance between the anchor point and the positive sample plus α, the value within [·] is greater than zero, resulting in a loss. When the distance between the anchor point and the negative sample is greater than or equal to the distance between the anchor point and the positive sample plus α, the loss function value is 0. During each training iteration, each batch contains P×K triples, where P is the number of targets randomly selected from the training set, and K is the number of gait sequence maps from different angles randomly selected from each target. For each anchor point, the positive sample with the smallest distance and the negative sample with the largest distance are selected.

[0042] Softmax Loss is a combination of the softmax function and cross-entropy loss. It is often used in classification tasks in deep learning. Since gait feature recognition is essentially a multi-classification problem, it optimizes the network model by continuously narrowing the distance between the predicted and true distributions. Therefore, Softmax Loss is also applied to gait feature recognition networks. The function is defined as:

[0043]

[0044] Where x i Let y be the i-th feature sample in the j-th feature class. j Let W represent the true label class of the target, W be the weight of the last dimension of the feature, and b represent the weight bias of the last dimension of the feature. While using Softmax Loss in conjunction with Triplet Loss can stabilize the training process and accelerate convergence, this approach only considers whether the classification is correct and does not significantly optimize inter-class distances. Therefore, we further consider combining it with the new loss function Circle Loss and two other loss functions.

[0045] Circle Loss, from the perspective of similarity optimization, unifies two basic deep feature learning paradigms: class-label data learning and sample-to-label data learning, and merges their definitions into a single formula. It is an improvement on the Triplet Loss loss function, aiming to maximize the intra-class similarity interval and minimize the inter-class similarity interval. Its improvement lies in the fact that Triplet Loss uses an averaging approach for positive and negative samples, which results in insufficient discrimination between positive and negative samples during model convergence. The Circle Loss loss function adds a weight to the positive and negative sample pairs, controlling the gradient contribution of each sample, ultimately leading to a more discriminative model. The function is defined as:

[0046]

[0047] s p For intra-class similarity, s n For inter-class similarity, Circle Loss increases the difficulty of training the model for each category, ultimately resulting in stronger discriminative power. Through further generalization, Circle Loss achieves more flexible optimization methods and clearer convergence targets, improving the discriminative ability of learned features and facilitating deep feature learning.

[0048] Finally, the weighted sum of the three loss functions—TripletLoss, SoftmaxLoss, and Circle Loss—is used as the loss function for the Res-GaitSet network. The joint loss function is defined as follows:

[0049] αL Y +βL s +λL uni (10)

[0050] In the above formula, α, β, and λ are the weights corresponding to the three loss functions. Through various comparative experiments, the optimal settings for the three values ​​can be obtained, which are 0.4, 0.4, and 0.2, respectively.

[0051] Experiments were conducted using the CASIA-B gait database created by the Pattern Recognition Laboratory of the Institute of Automation, Chinese Academy of Sciences. This database contains 124 pedestrians, each with 11 viewpoints ranging from 0 to 180 degrees, spaced in 18-degree intervals. It includes three walking scenarios: wearing a coat (CL), carrying a backpack (BG), and walking normally (NM). There are 6 gait sequences for normal walking, and 2 sequences each for wearing a coat and carrying a backpack. Each walking scenario has 11 angular variations, resulting in 124 × 10 × 11 × 80 = 1,091,200 images, each with a resolution of 320 × 240. Gait feature recognition requires the following data: a training image set, a gallery image set, and a probe image set. The gallery and probe sets are used for testing. The training set for this chapter's experiments consists of 74 individuals from the CASIA-B dataset, and the remaining 50 individuals are used for the test set. During testing, the reference images (Gallery) were set to the first four sets of images in the normal walking condition from all gait sequence images, and the test image set (Probe) was set to all images in the gait sequence images in the case of wearing a coat and backpack, as well as the last two sets of images in the case of normal walking.

[0052] The test results are divided into two categories: one is Include Identical-View, which includes cases where the Gallery and Probe views are the same when performing gait recognition tests; the other is Exclude Identical-View, which excludes cases where the Gallery and Probe views are the same when calculating accuracy.

[0053] Table 2. Recognition results from the CASIA-B dataset.

[0054]

[0055] The improved network Res-GaitSet in this chapter achieves the best recognition results in three different walking scenarios: normal walking, wearing a coat, and carrying a backpack. Compared to the basic network GaitSet, the improved network improves the recognition accuracy by about 2% in each scenario.

[0056] Table 3. Recognition results of the self-built dataset

[0057]

[0058] The Res-GaitSet network also achieved high recognition accuracy on the self-built dataset. Compared to the basic GaitSet network, the improved Res-GaitSet network showed an improvement of approximately 4.7% in recognition accuracy under the Include Identical-View test and approximately 6% in recognition accuracy under the Exclude Identical-View test. This verifies the effectiveness of the improved network.

[0059] ConvNeXt, built entirely from standard ConvNet modules, competes with Transformers in accuracy, scalability, and robustness across all major benchmarks. ConvNeXt maintains the efficiency of standard ConvNet, and the nature of the fully convolutional model used for training and testing makes its implementation extremely simple. The fully convolutional network ConvNeXt is based on the standard ResNet network, with a series of experimental improvements and comparisons made against the Swing Transformer network.

[0060] Standard CNNs often fail to accurately summarize information in the presence of occlusion, leading to redundancy in multiple target parts. TBC, on the other hand, can remove redundancy, reduce correlation, and effectively handle highly overlapping instances.

[0061] Assume the input features are Output features are Where c, h, and w are the number of channels, the height of the feature map, and the width, respectively. The kernel size is k×k, and the bias term is ignored. Then, the standard convolution (SC) is:

[0062]

[0063] in It is a standard convolutional kernel, and the number of parameters in SC is c. o ×c i ×k×k.

[0064] Suppose that the input feature X is divided into G groups of equal size, denoted as X1, X2, ..., X... G The corresponding size of each group is c. i / G×h i ×w i Each group shares the same convolutional filter W g Then the output of the group convolution GC can be expressed as:

[0065]

[0066] in W represents a cascaded operation along the channel dimension. g It is a convolutional filter of group g, where g∈(1,...,G), the number of parameters in GC is

[0067] Tied Block Convolution (TBC) reduces the effective number of filters by reusing filters across different feature groups. It can be expressed as the following formula:

[0068]

[0069] in It is a shared TBC filter among all groups, with c parameters. o / B×c i / B×k×k, where B represents the number of groups the channel is divided into, i.e., there are B equal blocks.

[0070] To eliminate the redundancy of convolutional layers caused by the repeated use of similar filters, and to enhance the learning ability of the network to improve the accuracy of model classification and improve the stability of the network without reducing the model capacity (i.e., the ability of the neural network model to fit complex functions), the bottleneck structure of shared filters and the fully convolutional integral class network ConvNeXt are fused to obtain the fully convolutional bottleneck structure of shared filters TBC-ConvNeXt.

[0071] The experiments in this chapter were conducted on the ImageNet dataset. The effectiveness of the improved network was verified by comparing its classification accuracy with that of other classification models. The classification accuracy results of each network model are shown in Table 4-1.

[0072] Table 4. Classification results (%) of each classification model on ImageNet-1K

[0073]

[0074] The improved model TBC-ConvNeXt achieves approximately 1% improvement in classification accuracy compared to the original ConvNeXt network. Compared to the architecture search network RegNetY-4G and the residual network ResNetXt-101, it shows improvements of 3.3% and 2.4% in classification accuracy, respectively. Furthermore, compared to the two most recent Transformer variants, DeiT-S and Swin Transformer, it also shows improvements of approximately 3.5% and 2% in classification accuracy, respectively. Overall, the improved network model demonstrates good performance on classification tasks.

[0075] Table 5. Accuracy of Cognitive Impairment Level Classification Based on Gait Features

[0076]

[0077] Table 5 shows the classification results of gait characteristics of the elderly using three network models. As can be seen from the table, the classification accuracy of all three models for gait characteristics of the elderly is above 80%. The results verify a significant correlation between gait characteristics and cognitive impairment levels in the elderly, indicating a substantial difference between the gait of normal elderly individuals and those with cognitive impairment. This also demonstrates that the method of classifying cognitive impairment levels by analyzing gait characteristics can be used as a preliminary clinical assessment method for professional physicians.

Claims

1. A method for classifying age-related cognitive impairment levels based on gait features, characterized in that... Includes the following steps: (1) Based on the instance segmentation network BlendMask, the global information aggregation capability of Transformer is introduced. On the basis of the convolutional neural network, the simple 3×3 spatial convolutional layer in the backbone structure of the original segmentation network is replaced by a multi-head self-attention (MHSA) module, and the CoordConv convolutional layer is added after the input module of the segmentation network. (2) Replace the 3×3 spatial convolutional layer in the frame-level feature extraction module of the GaitSet network with residual units, and jointly train the Triplet Loss, Softmax Loss, and Circle Loss loss functions. The weighted sum of the three loss functions is used as the loss function of the Res-GaitSet network. The joint loss function is defined as: aL Y +βL s +λL uni (1) In the above formula, α, β, and λ are the weights corresponding to the three loss functions, respectively; (3) The bottleneck structure of the shared filter and the fully convolutional integral network ConvNeXt are fused together. The 3×3 convolutional layers in the network are divided into 4 identical groups, each with its own TBC setting. The filters play different roles in different layers. For the 1×1 convolutions at the inlet and outlet of the bottleneck layer, the inlet convolution is replaced with a TBC of B=2 to allow filter sharing while maintaining the outlet convolution to achieve cross-channel information aggregation. The 3×3 convolutions in the network are split into 4 identical TBCs with a basic width of 32, i.e., 4S×32W. At the same time, a mixer is introduced, which merges the outputs of multiple splitters. The introduction of the mixer further improves the model performance. The input of the mixer is the sum of the elements of the cascaded output or the distributed output.

Images