Cognitive disorder disease detection method integrating human body posture and cognition

By collecting and integrating gait and speech data and using deep learning models for Alzheimer's disease detection, the high cost and invasiveness of existing diagnostic methods are solved, achieving low-cost and accurate early diagnosis.

CN119014822APending Publication Date: 2024-11-26THE CHINESE UNIV OF HONG KONG (SHENZHEN)
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411455877.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-18
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing methods for early diagnosis of Alzheimer's disease require specialized equipment and invasive examinations, resulting in high costs and low patient compliance, making them difficult to promote on a large scale.

Method used

By collecting video and audio data of subjects performing gait and cognitive tasks, 3D convolutional networks and wav2vec models are used to extract posture and speech features. Combined with self-attention and cross-attention modules, a neural network model is trained for disease detection.

Benefits of technology

It enables non-invasive, low-cost early diagnosis of Alzheimer's disease, improving the accuracy of detection and its potential for widespread application.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119014822A_ABST
    Figure CN119014822A_ABST
Patent Text Reader

Abstract

The invention discloses a cognitive disorder disease detection method integrating human body postures and cognition, and belongs to the technical field of auxiliary diagnosis and treatment. The problems that an existing method is high in early diagnosis cost for cognitive disorder diseases such as the Alzheimer's disease, low in patient compliance and difficult in large-scale disease screening due to the fact that the existing method depends on participation of professional medical staff can be solved. The method comprises the following steps: S1, according to video data and voice data when each subject in all subjects executes a gait task and a cognition task at the same time, determining posture cognition features of each subject; s2, training a neural network model according to the posture cognitive features of all the subjects and the disease diagnosis tag of each subject to obtain a disease detection model; and S3, inputting the posture cognitive features of the to-be-detected person into the disease detection model to obtain a disease detection result of the to-be-detected person. The method is used for detecting cognitive impairment diseases.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method for detecting cognitive impairment diseases by integrating human posture and cognition, and belongs to the technical field of auxiliary diagnosis and treatment. Background Art

[0002] Alzheimer's disease (AD) is the leading cause of dementia. Early detection of the disease and timely discovery and intervention at the mild cognitive impairment (MCI) stage are important gateways for the prevention and control of AD. Intervention at the MCI stage may be the most effective strategy to delay the onset of AD. Alzheimer's disease has an insidious onset and is difficult to diagnose in the early stages, making it a difficult problem that has received much attention in the academic and medical fields.

[0003] The diagnosis of the AD disease spectrum is through positron emission tomography (PET) or cerebrospinal fluid testing. PET examination is a technology that uses positron tracers to detect biochemical or metabolic information in the body in an anatomical way by using the tracer principle. It requires fasting for 6 hours before the examination and taking or injecting radioactive electron display agents one hour in advance. It will cost a lot of time and have certain side effects on health. Cerebrospinal fluid testing requires puncture and fluid collection from the patient's lumbar spine, which is an invasive examination in an important part. Both of the above diagnostic methods require patients to pay a greater economic and time cost than ordinary examination methods, and must be completed in qualified professional medical institutions by medical personnel who have been trained for a long time. Therefore, the above diagnostic framework is difficult to be launched and promoted on a large scale, which brings difficulties to the large-scale diagnosis and screening of Alzheimer's disease. Summary of the invention

[0004] The present invention provides a method for detecting cognitive impairment diseases by integrating human posture and cognition, which can solve the problems of high cost for early diagnosis of cognitive impairment diseases such as Alzheimer's disease, low patient compliance, and difficulty in large-scale disease screening caused by reliance on the participation of professional medical staff.

[0005] The present invention provides a method for detecting cognitive impairment diseases by integrating human body posture and cognition, the method comprising: S1, determining the posture recognition characteristics of each subject based on the video data and voice data of each subject in all subjects when the subject performs the gait task and the cognitive task at the same time; S2, training a neural network model based on the posture recognition features of all subjects and the disease diagnosis label of each subject to obtain a disease detection model; S3. Inputting the posture recognition characteristics of the subject to be tested into the disease detection model to obtain the disease detection result of the subject to be tested.

[0006] Optionally, the S1 specifically includes: Determine the posture feature and voice feature of each subject according to the video data and voice data of each subject among all the subjects when the subject performs the gait task and the cognitive task at the same time; wherein the dimensions of the posture feature and the voice feature are consistent; According to the posture characteristics and speech characteristics of each subject, the posture recognition characteristics of each subject are determined.

[0007] Optionally, determining the posture feature and voice feature of each subject according to the video data and voice data of each subject among all the subjects when the subject performs the gait task and the cognitive task at the same time specifically includes: The first pre-trained model is used to extract features from the video data of each subject among all the subjects when the subject performs the gait task and the cognitive task at the same time, so as to obtain the posture features of each subject, and the second pre-trained model is used to extract features from the speech data of each subject among all the subjects when the subject performs the gait task and the cognitive task at the same time, so as to obtain the speech features of each subject.

[0008] Optionally, the first pre-trained model is a 3D convolutional network; and the second pre-trained model is a wav2vec model.

[0009] Optionally, determining the posture recognition feature of each subject according to the posture feature and voice feature of each subject specifically includes: Performing feature enhancement and modal interaction on the posture features and speech features of each subject to obtain the posture optimized features and speech optimized features of each subject; The posture optimization features and speech optimization features of each subject are feature-connected to obtain the posture recognition features of each subject.

[0010] Optionally, the step of performing feature enhancement and modal interaction on the posture feature and voice feature of each subject to obtain the posture optimization feature and voice optimization feature of each subject specifically includes: The posture features and speech features of each subject are respectively input into the self-attention module for feature enhancement to obtain the posture enhanced features and speech enhanced features of each subject; The posture enhanced features and speech enhanced features of each subject are input into the cross-attention module for modal interaction to obtain the posture optimized features and speech optimized features of each subject.

[0011] Optionally, before S2, the method further includes: The posture recognition features of each subject are regularized to obtain the processed posture recognition features.

[0012] Optionally, the neural network model is trained using a 5-fold stratified cross-validation scheme.

[0013] Optionally, before S3, the method further includes: The video data and the voice data of the subject are collected when the subject performs the gait task and the cognitive task at the same time, and the posture cognitive characteristics of the subject are determined according to the video data and the voice data. The beneficial effects that the present invention can produce include: The method for detecting cognitive impairment diseases by integrating human posture and cognition provided by the present invention can comprehensively capture the individual's performance in cognitive and motor functions by collecting synchronous data of posture and voice, and use multiple modal data to determine the posture cognitive characteristics of each subject. The disease detection model trained in this way can more accurately reflect the cognitive and motor defects of patients with cognitive impairment diseases.

[0014] The method for detecting cognitive disorders by integrating human posture and cognition provided by the present invention first collects human posture video data and voice data and aligns the time frames, extracts the deep features of the two modalities, and increases the utilization rate of information; secondly, uses the self-attention module to fully extract the information inside the modality; finally, uses the cross-attention module to fuse the two modalities to obtain the posture cognitive features, and then inputs the posture cognitive features into the disease detection model to output the results of disease diagnosis. This invention can solve the problems existing in the current Alzheimer's disease diagnosis system, and can also be extended to the diagnosis schemes of other related diseases with cognitive decline. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flow chart of a method for detecting cognitive impairment diseases by integrating human posture and cognition provided by an embodiment of the present invention; Figure 2 A schematic diagram of the Alzheimer's disease detection process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The present invention is described in detail below in conjunction with embodiments, but the present invention is not limited to these embodiments.

[0017] The embodiment of the present invention provides a method for detecting cognitive impairment diseases by integrating human body posture and cognition. Figure 1 and Figure 2 As shown, the method includes: S1. Determine the posture recognition characteristics of each subject according to the video data and voice data of each subject among all the subjects when the subject performs the gait task and the cognitive task at the same time.

[0018] For the convenience of description, the present invention takes Alzheimer's disease, a cognitive impairment disease, as an example to explain the scheme of the present invention in detail.

[0019] S1 specifically includes: (1) Determine the posture features and speech features of each subject based on the video data and speech data of each subject among all the subjects when the subject performs the gait task and the cognitive task at the same time; wherein the dimensions of the posture features and the speech features are consistent.

[0020] Specifically, the first pre-trained model is used to extract features from the video data of each subject in all subjects when performing the gait task and the cognitive task at the same time, so as to obtain the posture features of each subject, and the second pre-trained model is used to extract features from the voice data of each subject in all subjects when performing the gait task and the cognitive task at the same time, so as to obtain the voice features of each subject. The first pre-trained model is a 3D convolutional network; the second pre-trained model is a wav2vec model.

[0021] In an embodiment of the present invention, before feature extraction, it is necessary to first collect information on the subject and annotate the disease condition. Specifically, the Mini-Mental State Examination (MMSE) and the Montreal Cognitive Assessment (MoCA) can be completed to obtain annotated data such as the subject's gender, age, years of education, and Alzheimer's disease diagnosis results.

[0022] Then, dual-modal data collection combining gait and voice tasks is performed. Specifically, the above gait task is that after receiving the "start" command, the subject needs to walk 3 meters in a straight line from the designated starting point, turn left and sit down when encountering a chair in the path, and then stand up and return to the starting point; the subject needs to perform a cognitive task at the same time during this process, that is, subtract 7 from 100, and continue to subtract until the gait task is completed, and say the result each time. The video data is a video shot by a smartphone, and the voice data is a voice file recorded by a clip-on microphone. The naming of the video (.Mp4 format) and the voice file (.wav format) corresponds to the information of the subject obtained above.

[0023] After the video data and voice data of the subject are collected, feature extraction is performed. Specifically, the video data can use the first pre-training model to extract deep features, and the voice data can use the second pre-training model to extract deep features.

[0024] For speech data, the present invention inputs it into the wav2vec model to extract a 768-dimensional feature vector, and then reduces it to 512 dimensions through a fully connected layer to obtain the corresponding speech features. For video data, the present invention uses a 3D convolutional network (Inflated 3D ConvNet, I3D) pre-trained with the Kinetics-400 dataset to output a 2048-dimensional feature vector per frame, and then reduces it to 512 dimensions through a 3D convolutional layer to obtain the corresponding posture features, so as to maintain consistency with the dimensions of speech features in subsequent multimodal data analysis.

[0025] (2) Determine the posture recognition characteristics of each subject based on the posture characteristics and speech characteristics of each subject.

[0026] Specifically, the method includes: performing feature enhancement and modal interaction on the posture features and speech features of each subject to obtain the posture optimization features and speech optimization features of each subject, and then performing feature connection on the posture optimization features and speech optimization features of each subject to obtain the posture recognition features of each subject.

[0027] In an embodiment of the present invention, after obtaining the posture features and speech features of each subject, the posture features and speech features of each subject are respectively input into the self-attention module for feature enhancement to obtain the posture enhancement features and speech enhancement features of each subject. Then, the posture enhancement features and speech enhancement features of each subject are input into the cross-attention module for modal interaction to obtain the posture optimization features and speech optimization features of each subject. Finally, the posture optimization features and speech optimization features are connected to obtain the posture recognition features of each subject.

[0028] refer to Figure 2 As shown in Figure 1, self-attention is applied independently to audio and video modality features to enhance the representation of deep features. For the audio modality, this module helps capture temporal dependencies, such as speech rhythm and variation, which are associated with cognitive impairment caused by Alzheimer's disease. Similarly, this module can highlight important temporal consistency in gait movement patterns, which is important for detecting movement defects. Optimizing features within the modality in this way ensures that all deep features of the modal data are fully utilized.

[0029] In order to integrate bimodal data, the embodiment of the present invention uses a cross-attention mechanism to exchange information between the two modalities (representing speech features and gesture features respectively). Each modality receives its self-attention features from the self-attention layer and then feeds them into the cross-attention module to promote cross-modal interaction and further improve performance.

[0030] S2. Train a neural network model based on the posture recognition features of all subjects and the disease diagnosis label of each subject to obtain a disease detection model.

[0031] In the embodiment of the present invention, the neural network model is trained using a 5-fold stratified cross-validation scheme. The present invention uses the AdamW optimizer with a learning rate of 1x10 -5 , weight decay is 1x10 -5 , the batch size is 8. To alleviate overfitting, the present invention uses a Dropout layer of 0.65 and uses Focal Loss to handle the class imbalance problem. These parameter selections are effective for small and unbalanced datasets.

[0032] Before S2, the method further includes: The posture recognition features of each subject are regularized to obtain the processed posture recognition features.

[0033] On small and unbalanced datasets, regularization can help alleviate overfitting and improve generalization. Using the processed posture recognition features of all subjects and the disease diagnosis labels of each subject to train the neural network model can obtain a disease detection model with better prediction effect.

[0034] S3. Input the posture recognition characteristics of the subject to be tested into the disease detection model to obtain the disease detection result of the subject to be tested.

[0035] Before S3, the method further includes: The video data and voice data of the subject are collected when the subject performs the gait task and the cognitive task at the same time, and the posture cognitive characteristics of the subject are determined based on the video data and the voice data.

[0036] The process of processing the video data and voice data of the subject to be tested to obtain the posture recognition features is consistent with the processing method of the video data and voice data of the above-mentioned subject, and will not be repeated here.

[0037] The method for detecting cognitive impairment diseases by integrating human posture and cognition provided by the present invention, firstly, can comprehensively capture the performance of individuals in cognitive and motor functions by collecting synchronous data of human posture and voice, and greatly improve the utilization rate of task information by using the deep features of multiple modalities, so that the trained disease detection model can more accurately reflect the cognitive and motor defects of patients with Alzheimer's disease. Secondly, the self-attention mechanism is used to fully explore and extract the time dependency and feature patterns within each modality, ensuring that the subtle information within the modality can be effectively captured, and further improving the disease detection model's ability to express modal features. Finally, the deep features of the two modalities are fused through the cross-attention mechanism, realizing the complementarity and synergy of information between modalities, effectively improving the recognition accuracy of the disease detection model for Alzheimer's disease-related symptoms, and enhancing the classification performance. In short, the innovative design of the present invention in multimodal feature extraction and fusion makes its application in the early diagnosis of Alzheimer's disease have significant advantages.

[0038] The above are only a few embodiments of the present application and do not constitute any form of limitation to the present application. Although the present application is disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technician familiar with the profession, without departing from the scope of the technical solution of the present application, using the technical content disclosed above to make slight changes or modifications are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A method for detecting cognitive impairment diseases by integrating human posture and cognition. It is characterized in that The method comprises: S1, determining the posture recognition characteristics of each subject based on the video data and voice data of each subject in all subjects when the subject performs the gait task and the cognitive task simultaneously; S2, training a neural network model based on the posture recognition features of all subjects and the disease diagnosis label of each subject to obtain a disease detection model; S3. Inputting the posture recognition characteristics of the subject to be tested into the disease detection model to obtain the disease detection result of the subject to be tested.

2. The method according to claim 1, It is characterized in that The S1 specifically includes: Determine the posture feature and voice feature of each subject according to the video data and voice data of each subject among all the subjects when the subject performs the gait task and the cognitive task at the same time; wherein the dimensions of the posture feature and the voice feature are consistent; According to the posture characteristics and speech characteristics of each subject, the posture recognition characteristics of each subject are determined.

3. The method according to claim 2, It is characterized in that The step of determining the posture features and speech features of each subject according to the video data and speech data of each subject among all the subjects when the subject performs the gait task and the cognitive task at the same time specifically includes: The first pre-trained model is used to extract features from the video data of each subject among all the subjects when the subject performs the gait task and the cognitive task at the same time, so as to obtain the posture features of each subject, and the second pre-trained model is used to extract features from the speech data of each subject among all the subjects when the subject performs the gait task and the cognitive task at the same time, so as to obtain the speech features of each subject.

4. The method according to claim 3, It is characterized in that The first pre-trained model is a 3D convolutional network; the second pre-trained model is a wav2vec model.

5. The method according to claim 2, It is characterized in that Determining the posture recognition feature of each subject according to the posture feature and voice feature of each subject specifically includes: Performing feature enhancement and modal interaction on the posture features and speech features of each subject to obtain the posture optimized features and speech optimized features of each subject; The posture optimization features and speech optimization features of each subject are feature-connected to obtain the posture recognition features of each subject.

6. The method according to claim 5, It is characterized in that The feature enhancement and modal interaction are performed on the posture features and voice features of each subject to obtain the posture optimization features and voice optimization features of each subject, specifically including: The posture features and speech features of each subject are respectively input into the self-attention module for feature enhancement to obtain the posture enhanced features and speech enhanced features of each subject; The posture enhanced features and speech enhanced features of each subject are input into the cross-attention module for modal interaction to obtain the posture optimized features and speech optimized features of each subject.

7. The method according to claim 1, It is characterized in that Before S2, the method further includes: The posture recognition features of each subject are regularized to obtain the processed posture recognition features.

8. The method according to claim 1, It is characterized in that The neural network model was trained using a 5-fold stratified cross-validation scheme.

9. The method according to claim 1, It is characterized in that Before S3, the method further includes: The video data and the voice data of the subject are collected when the subject performs the gait task and the cognitive task at the same time, and the posture cognitive characteristics of the subject are determined according to the video data and the voice data.

Citation Information

Cited By

  • Cognitive disorder behavior monitoring and early warning method for Alzheimer's disease

    CN120732370A