AD symptom auxiliary judgment method and system based on expression and action of patient

Abstract

The invention provides an AD symptom auxiliary judgment method and system based on expression and action of a patient, relates to the technical field of medical auxiliary diagnosis, and realizes early and accurate recognition of AD symptoms by combining FiT, a bimodal emotion recognition algorithm, a ViTAR model, an image super-resolution technology and a Qwen2-VL model. According to the method, video image data without limitation of resolution and aspect ratio can be processed, emotional state features and human body posture features of a patient can be extracted, feature analysis and classification are performed through an intelligent learning module, the consistency and accuracy of diagnosis are improved, the system comprises a plurality of modules, and data are received, processed and learnt from data receiving and processing to feature learning. The AD symptom diagnosis is comprehensively assisted, the method is suitable for electronic equipment and a computer readable storage medium, the screening efficiency and objectivity are remarkably improved, and technical support is provided for early intervention and treatment of AD.

Description

Technical Field

[0001] The present invention relates to the technical field of medical auxiliary diagnosis, and more specifically, to a method and system for auxiliary judgment of AD symptoms based on patient expressions and movements. Background Art

[0002] In today's medical field, the diagnosis and prevention of Alzheimer's disease (AD) have attracted much attention. However, existing AD symptom judgment technologies face many challenges. On the one hand, early identification is difficult. The early symptoms of AD are hidden. Traditional methods often fail to capture subtle changes in time, causing many patients to miss the best time for intervention. On the other hand, the screening process is inefficient and lacks objectivity. It relies more on the subjective observation and experience judgment of medical staff. The judgment standards of different personnel vary, and there is a lack of unified and efficient objective quantitative means. At the same time, the accuracy of diagnosis is often affected by the experience level of doctors. Different levels of experience will lead to uneven diagnostic results. Moreover, when processing ultra-long videos containing patients' long-term behavioral performance, existing technologies are prone to data overload and difficulty in feature extraction. These comprehensive data cannot be effectively utilized. Moreover, in terms of dual-modal emotion recognition, the ability to accurately integrate speech and visual information to accurately judge the patient's emotional state still needs to be improved. Human posture estimation often fails to achieve ideal accuracy, making it difficult to fully explore the relationship between posture features and AD symptoms. In addition, image super-resolution reconstruction is not effective, and low-quality images affect the accurate extraction of key features such as expressions.

[0003] Therefore, the problems existing in existing technologies mainly include difficulties in early identification, insufficient screening efficiency and objectivity, diagnostic accuracy affected by doctor's experience, limitations in processing very long videos, accuracy of dual-modal emotion recognition, precision of human posture estimation, and image super-resolution reconstruction problems. Summary of the invention

[0004] In order to overcome the above problems existing in the prior art, the present invention designs an AD symptom auxiliary judgment method and system based on the patient's expression and movement, which can effectively solve the above technical problems.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A method for assisting in judging AD symptoms based on patient expressions and movements comprises the following steps:

[0007] Receive video image data of the patient's expression and action and corresponding voice signals;

[0008] Processing the received video image data using FiT, wherein the FiT is used to process the video image data with unlimited resolution and aspect ratio;

[0009] Analyze the patient's emotional state characteristics using a bimodal emotion recognition algorithm;

[0010] The patient's posture is estimated through the ViTAR model to obtain the human body posture characteristics;

[0011] Applying an image super-resolution technique to enhance the clarity of the video image data, wherein the image super-resolution technique reconstructs a low-resolution image into a high-resolution image;

[0012] Analyzing the patient's movements using a Qwen2-VL model to identify subtle changes in movement, wherein the Qwen2-VL model is used to process the video image data of different resolutions;

[0013] Intelligent learning is used to learn and judge the extracted features to assist in the diagnosis of AD symptoms.

[0014] Preferably, the FiT comprises:

[0015] A Transformer architecture for encoding a dynamic image sequence in the video image data into a dynamically sized token sequence;

[0016] The encoded token sequence is transformed to generate video image features with unlimited resolution and aspect ratio.

[0017] Preferably, the bimodal emotion recognition algorithm comprises:

[0018] Convolutional neural networks and pre-trained facial expression models are used to extract corresponding sound features and visual features from speech signals and visual signals respectively;

[0019] Fuse the extracted sound features and visual features to mine relevant information between modalities;

[0020] The long short-term memory recurrent neural network is used to identify the emotional state of the fused sound and vision bimodal features.

[0021] Preferably, estimating the patient's posture by using the ViTAR model comprises:

[0022] Slice and embed the Token in the video image data;

[0023] Iteratively process the Token by an adaptive Token merging unit;

[0024] Spread the processed Tokens on the grid, and gradually merge the Tokens in each unit;

[0025] All the tokens are mapped onto a grid of a fixed shape to estimate the human body posture.

[0026] Preferably, the image super-resolution technology includes:

[0027] Train a low-resolution dictionary DL;

[0028] Compute high and low resolution neighbor sets;

[0029] Calculate the mapping matrix set {Fi} from low-resolution features to high-resolution features.

[0030] Preferably, the intelligent learning includes:

[0031] Use machine learning algorithms to learn and pattern-recognize features of patients’ expressions, movements, and emotional states;

[0032] A classification model was established to distinguish normal behavior from abnormal behavior related to AD symptoms to assist in the diagnosis of AD symptoms.

[0033] An AD symptom auxiliary judgment system based on patient expressions and movements, comprising:

[0034] A data receiving module, used to receive video image data of the patient's expression and action and corresponding voice signals;

[0035] A FiT processing module, used for processing the video image data, wherein FiT is used for processing video image data with unlimited resolution and aspect ratio;

[0036] A bimodal emotion recognition module to analyze the characteristics of the patient's emotional state;

[0037] The ViTAR model module is used to estimate the patient's posture and obtain human posture features;

[0038] An image super-resolution module, used to enhance the clarity of the video image data, wherein the image super-resolution is used to reconstruct a low-resolution image into a high-resolution image;

[0039] Qwen2-VL model module, used to analyze the patient's movements and identify subtle changes in movement, wherein the Qwen2-VL model module is used to process video image data of different resolutions;

[0040] The intelligent learning module is used to learn and judge the extracted features to assist in diagnosing AD symptoms.

[0041] An electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the method for auxiliary determination of AD symptoms based on patient expressions and movements is implemented.

[0042] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned method for assisting in determining AD symptoms based on patient expressions and movements.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention utilizes FiT technology to process videos of different resolutions to accurately capture early signs of AD; the dual-modal emotion recognition algorithm combines expression and speech analysis to improve the accuracy of emotion state recognition; the application of the ViTAR model and the Qwen2-VL model provides objective data-driven screening and enhances the accuracy of posture estimation and motion analysis; the intelligent learning module reduces dependence on doctor's experience through machine learning and improves diagnostic consistency and accuracy; the Transformer architecture processes dynamic images and supports long-term behavior pattern monitoring; image super-resolution technology improves image clarity and provides high-quality data for feature extraction. The present invention provides an efficient and objective technical means for the early diagnosis and continuous monitoring of AD, significantly improving screening efficiency and the accuracy of disease identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived based on the provided drawings without paying any creative work.

[0045] Figure 1 A step diagram of a method for assisting in judging AD symptoms based on the patient's facial expressions and movements;

[0046] Figure 2 This is a structural diagram of an AD symptom auxiliary judgment system based on patient expressions and movements. DETAILED DESCRIPTION

[0047] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0048] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0049] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0050] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0051] Example 1

[0052] A method for assisting in judging AD symptoms based on patient expressions and movements comprises the following steps:

[0053] Receive video image data of the patient's expression and action and corresponding voice signals;

[0054] Processing the received video image data using FiT, wherein the FiT is used to process the video image data with unlimited resolution and aspect ratio;

[0055] Flexible Vision Transformer (FiT) is a Transformer architecture designed for diffusion models that can process video image data with unlimited resolution and aspect ratio.

[0056] Analyze the patient's emotional state characteristics using a bimodal emotion recognition algorithm;

[0057] The patient's posture is estimated through the ViTAR model to obtain the human body posture characteristics;

[0058] The ViTAR (Vision Transformer with Any Resolution) model is a new type of visual Transformer model, specially designed to solve the problem of performance degradation of traditional Vision Transformers (ViTs) when processing images of different resolutions.

[0059] Applying an image super-resolution technique to enhance the clarity of the video image data, wherein the image super-resolution technique reconstructs a low-resolution image into a high-resolution image;

[0060] Analyzing the patient's movements using a Qwen2-VL model to identify subtle changes in movement, wherein the Qwen2-VL model is used to process the video image data of different resolutions;

[0061] The Qwen2-VL model is a multimodal visual language model.

[0062] Intelligent learning is used to learn and judge the extracted features to assist in the diagnosis of AD symptoms.

[0063] The FiT includes:

[0064] A Transformer architecture for encoding a dynamic image sequence in the video image data into a dynamically sized token sequence;

[0065] The encoded token sequence is transformed to generate video image features with unlimited resolution and aspect ratio.

[0066] The bimodal emotion recognition algorithm includes:

[0067] Convolutional neural networks and pre-trained facial expression models are used to extract corresponding sound features and visual features from speech signals and visual signals respectively;

[0068] Fuse the extracted sound features and visual features to mine relevant information between modalities;

[0069] The long short-term memory recurrent neural network is used to identify the emotional state of the fused sound and vision bimodal features.

[0070] The estimating the patient's posture by using the ViTAR model includes:

[0071] Slice and embed the Token in the video image data;

[0072] Iteratively process the Token by an adaptive Token merging unit;

[0073] Spread the processed Tokens on the grid, and gradually merge the Tokens in each unit;

[0074] All the tokens are mapped onto a grid of a fixed shape to estimate the human body posture.

[0075] The image super-resolution technology includes:

[0076] Train a low-resolution dictionary DL;

[0077] Compute high and low resolution neighbor sets;

[0078] Calculate the mapping matrix set {Fi} from low-resolution features to high-resolution features.

[0079] The intelligent learning includes:

[0080] Use machine learning algorithms to learn and pattern-recognize features of patients’ expressions, movements, and emotional states;

[0081] A classification model was established to distinguish normal behavior from abnormal behavior related to AD symptoms to assist in the diagnosis of AD symptoms.

[0082] An AD symptom auxiliary judgment system based on patient expressions and movements, comprising:

[0083] A data receiving module, used to receive video image data of the patient's expression and action and corresponding voice signals;

[0084] A FiT processing module, used for processing the video image data, wherein FiT is used for processing video image data with unlimited resolution and aspect ratio;

[0085] A bimodal emotion recognition module to analyze the characteristics of the patient's emotional state;

[0086] The ViTAR model module is used to estimate the patient's posture and obtain human posture features;

[0087] An image super-resolution module, used to enhance the clarity of the video image data, wherein the image super-resolution is used to reconstruct a low-resolution image into a high-resolution image;

[0088] Qwen2-VL model module, used to analyze the patient's movements and identify subtle changes in movement, wherein the Qwen2-VL model module is used to process video image data of different resolutions;

[0089] The intelligent learning module is used to learn and judge the extracted features to assist in diagnosing AD symptoms.

[0090] An electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the method for auxiliary determination of AD symptoms based on patient expressions and movements is implemented.

[0091] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned method for assisting in determining AD symptoms based on patient expressions and movements.

[0092] The diagnosis room is equipped with 5 high-definition cameras with wide dynamic function, which are distributed in different positions of the room and can capture patients in 360 degrees without blind spots. The camera resolution is as high as 4K, and the frame rate can be stabilized at 30 frames per second, ensuring clear recording of the dynamic changes of patients' expressions and movements. At the same time, 4 high-sensitivity and directional microphones are installed to form a pickup array to accurately collect the voices made by patients. The effective pickup range covers the entire diagnosis room, and the frequency response is flat, which can accurately restore the audio signal of 20Hz-20kHz. A server is used as the data processing core, equipped with two Intel Xeon Platinum 8380 processors, a total of 80 cores and 160 threads, with 256GB of high-speed DDR4 memory and 4 NVIDIA A100 GPUs, which provide powerful computing power for complex data calculations and model reasoning. The server is pre-installed with a specially customized AD diagnostic auxiliary software platform, which integrates the full set of technical components involved in the present invention.

[0093] See also Figure 1 and 2When a patient with mild signs of cognitive impairment enters the diagnosis room, the system automatically activates the data collection process, the camera quickly turns on, and synchronously records the patient's facial expressions and movements from all angles. The microphone array captures the corresponding voice signals in real time and transmits the data to the server via a Category 6 (Cat6) Ethernet cable at a transmission rate of 10Gbps. During simple communication and interaction with medical staff, the patient sometimes shows a slightly confused look, rubs his fingers slightly, pauses briefly when speaking, and repeats words. These subtle expressions are accurately captured by the camera and microphone and transmitted to the server in real time, forming the original material for subsequent analysis.

[0094] After receiving the video image data, the server immediately calls the FiT module. Based on its internal Transformer architecture, FiT first divides the input dynamic image sequence into many tiny image blocks of 8×8 pixels each, and then encodes them into a token sequence of dynamic size. For example, for a frame of video image with a resolution of 3840×2160, approximately 129,600 tokens can be generated. Each token carries multi-dimensional information such as the position, color, and texture of the corresponding image block. The encoded token sequence is deeply transformed through a 16-head self-attention mechanism, focusing on the subtle contraction and relaxation of the eye muscles, the subtle changes in the curvature of the corners of the mouth, and the changes in the gloss of the facial skin. After multiple layers of complex feature fusion and transformation, video image features that adapt to different resolutions and aspect ratios are successfully generated. Even if the patient turns his head and leans forward and backward during the communication process, causing the picture to stretch and deform, FiT can still accurately extract key expression features, such as recognizing the brief dullness in her eyes and the occasional unconscious downward turn of the corners of her mouth. These features may be closely related to early AD symptoms.

[0095] The dual-modal emotion recognition algorithm runs synchronously. The visual signals in the video image data are processed using a modified MobileNetV3 convolutional neural network. The network is pre-trained on a facial expression dataset and can quickly and accurately extract visual features from the patient's facial image, such as the degree of relaxation of the cheek muscles and the stretch of the eyebrows. For voice signals, a speech feature extractor based on a deep neural network (DNN) is used in combination with linear predictive coding (LPC) technology to mine sound features from the voice collected by the microphone, such as changes in speech speed and fluctuations in tone. , pause intervals between sentences, etc. For example, the patient speaks slower than his peers, his tone lacks normal ups and downs, and long pauses occur frequently. These sound features are efficiently captured, and the feature fusion module based on the multi-head attention mechanism is used to deeply fuse the visual features with the sound features to fully explore the potential correlation information between the modalities. Finally, the long short-term memory recurrent neural network (LSTM) is used to identify the emotional state of the fused bimodal features, and it is accurately judged that the patient is currently in a mildly confused and slightly anxious emotional state, which is highly consistent with the common early emotional representation of AD patients.

[0096] The ViTAR model intervenes quickly, first embedding the tokens in the video image data in blocks of 16×16 pixels, converting the image into a dense sequence of embedded vectors. Its adaptive token merging unit sets priority weights for tokens in different parts based on the principles of human anatomy and kinematics. When the patient stands up and walks, it prioritizes the tokens corresponding to the leg, hip and waist joints. Through iterative processing, it gradually merges closely related tokens, evenly spreads the processed tokens on a preset two-dimensional grid, and continues to merge the tokens in each unit until all tokens are mapped to a grid of a fixed shape. It successfully constructs an accurate human posture model of the patient. From the model output, it can be seen that the patient walks with small steps and slow cadence, insufficient knee bending, and a slightly forward leaning center of gravity. These posture features provide key clues for inferring AD symptoms.

[0097] Image super-resolution technology works its magic on the collected video image data. It adopts a cutting-edge architecture based on a combination of generative adversarial networks (GANs) and attention mechanisms. The generative network is composed of 12 carefully designed residual modules stacked together. Residual learning is used to efficiently accelerate network convergence. In the training phase, a large data set containing hundreds of thousands of low-resolution and high-resolution face and body image pairs of people of different ages, genders, and ethnicities is used to train a low-resolution dictionary DL. At the same time, the high- and low-resolution neighbor sets are accurately calculated, and the mapping matrix set {Fi} from low-resolution features to high-resolution features is inferred. When processing the patient's video images, the low-resolution images are fed into the generative network, and after adversarial training and attention guidance, high-resolution images are output. For example, in the original image, the skin wrinkles on the patient's hands and the subtle facial expressions are blurred. After super-resolution processing, the accuracy and reliability of subsequent feature extraction are improved.

[0098] The Qwen2-VL model is responsible for analyzing the patient's movement changes. It cleverly has a built-in dynamic time warping (DTW) module to effectively deal with situations where the patient's movement rate is unstable. For example, the patient's hand occasionally trembles, sometimes fast and sometimes slow. The DTW module accurately aligns these movement sequences at different rates in the time dimension, accurately captures key parameters such as the frequency, amplitude, and duration of the tremor, and combines it with a pre-built human kinematics feature library that integrates a large amount of medical knowledge and clinical experience. It compares the identified subtle movement changes such as hand tremors and footsteps dragging with the typical AD-related movement patterns in the feature library one by one, confirming that the patient's hand tremors are consistent with the common static tremor characteristics of early AD patients, and the footsteps dragging is also consistent with the reduced motor coordination caused by AD.

[0099] The intelligent learning module brings together the above-mentioned all-round extracted features, covering facial expression features, emotional state features, human posture features, subtle movement change features, etc. It adopts advanced stacked integrated learning algorithms, integrating multiple efficient machine learning models such as random forests, multi-layer perceptrons, and lightweight gradient boosting trees. It dynamically allocates weights based on the performance of each model on different feature subsets to achieve the optimal fusion of feature analysis results. In the model training stage, it relies on sample data of AD patients and healthy people covering different disease stages, regions, and lifestyles, and uses the adaptive moment estimation (Adam) optimization algorithm to repeatedly train the model, enabling it to deeply grasp the abnormalities related to normal behaviors and AD symptoms. After the patient's characteristic data is injected, the model quickly outputs a suspicion rating of AD symptoms as moderate suspicion, and estimates that the disease is in the early to mid-stage of development. It also gives a confidence interval of up to 75%-85%, and generates a detailed diagnosis report. The report lists in detail the key symptom characteristics, such as the facial expressions, body postures, and changes in movement details mentioned above, explains the inherent logic of the judgment process, and recommends blood biomarker testing, brain functional magnetic resonance imaging (fMRI) and other auxiliary diagnosis methods to fully assist medical staff in finalizing accurate diagnosis strategies and provide a scientific path for subsequent treatment of patients.

[0100] The same or similar reference numerals correspond to the same or similar components;

[0101] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0102] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made on the basis of the above description. It is not necessary and impossible to list all the implementation methods here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for assisting in judging AD symptoms based on patient expressions and movements, characterized in that: include: Receive video image data of the patient's expression and action and corresponding voice signals; Processing the received video image data using FiT, wherein the FiT is used to process the video image data with unlimited resolution and aspect ratio; Analyze the patient's emotional state characteristics using a bimodal emotion recognition algorithm; The patient's posture is estimated through the ViTAR model to obtain the human body posture characteristics; Applying an image super-resolution technique to enhance the clarity of the video image data, wherein the image super-resolution technique reconstructs a low-resolution image into a high-resolution image; Analyzing the patient's movements using a Qwen2-VL model to identify subtle changes in movement, wherein the Qwen2-VL model is used to process the video image data of different resolutions; Intelligent learning is used to learn and judge the extracted features to assist in the diagnosis of AD symptoms.

2. The method for assisting in determining AD symptoms based on patient expressions and movements according to claim 1, characterized in that: The FiT includes: A Transformer architecture for encoding a dynamic image sequence in the video image data into a dynamically sized token sequence; The encoded token sequence is transformed to generate video image features with unlimited resolution and aspect ratio.

3. The method for assisting in determining AD symptoms based on patient expressions and movements according to claim 1, characterized in that: The bimodal emotion recognition algorithm includes: Convolutional neural networks and pre-trained facial expression models are used to extract corresponding sound features and visual features from speech signals and visual signals respectively; Fuse the extracted sound features and visual features to mine relevant information between modalities; The long short-term memory recurrent neural network is used to identify the emotional state of the fused sound and vision bimodal features.

4. The method for assisting in determining AD symptoms based on patient expressions and movements according to claim 1, characterized in that: The estimating the patient's posture by using the ViTAR model includes: Slice and embed the Token in the video image data; Iteratively process the Token by an adaptive Token merging unit; Spread the processed Tokens on the grid, and gradually merge the Tokens in each unit; All the tokens are mapped onto a grid of a fixed shape to estimate the human body posture.

5. The method for assisting in determining AD symptoms based on patient expressions and movements according to claim 1, characterized in that: The image super-resolution technology includes: Train a low-resolution dictionary DL; Compute high and low resolution neighbor sets; Calculate the mapping matrix set {Fi} from low-resolution features to high-resolution features.

6. The method for assisting in determining AD symptoms based on patient expressions and movements according to claim 1, characterized in that: The intelligent learning includes: Use machine learning algorithms to learn and pattern-recognize features of patients’ expressions, movements, and emotional states; A classification model was established to distinguish normal behavior from abnormal behavior related to AD symptoms to assist in the diagnosis of AD symptoms.

7. An AD symptom auxiliary judgment system based on patient expression and action, implemented by an AD symptom auxiliary judgment method based on patient expression and action as claimed in any one of claims 1 to 6, characterized in that: include: A data receiving module, used to receive video image data of the patient's expression and action and corresponding voice signals; A FiT processing module, used for processing the video image data, wherein FiT is used for processing video image data with unlimited resolution and aspect ratio; A bimodal emotion recognition module to analyze the characteristics of the patient's emotional state; The ViTAR model module is used to estimate the patient's posture and obtain human posture features; An image super-resolution module, used to enhance the clarity of the video image data, wherein the image super-resolution is used to reconstruct a low-resolution image into a high-resolution image; Qwen2-VL model module, used to analyze the patient's movements and identify subtle movement changes, wherein the Qwen2-VL model module is used to process video image data of different resolutions; The intelligent learning module is used to learn and judge the extracted features to assist in diagnosing AD symptoms.

8. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements an AD symptom auxiliary judgment method based on patient expressions and movements as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for auxiliary determination of AD symptoms based on patient expressions and movements as described in any one of claims 1 to 6 is implemented.

Images