A multi-modal Parkinson's disease video auxiliary diagnosis method and system based on a Transformer and attention fusion

Abstract

This invention relates to the fields of medical auxiliary diagnosis and computer vision technology, specifically to a multimodal video-assisted diagnostic method and system for Parkinson's disease based on Transformer and attention fusion. First, video data of the subject's hand pinching motion and linear back-and-forth walking motion are acquired and a unified sample label is established. Then, video preprocessing is completed through preprocessing, sliding time window segmentation, and data augmentation. Next, the preprocessed temporal image sequence is input into a spatiotemporal modeling network, where a visual Transformer is used to extract frame-level spatial features. A multi-layer long short-term memory network is then combined to capture long-term temporal motion dynamic information. Finally, a modality-level spatiotemporal attention fusion method adaptively learns modality weights and achieves feature fusion. Finally, the fused features are input into a classification network, outputting prediction results for Parkinson's disease patients or healthy controls. This invention comprehensively characterizes the motion features of the disease through multimodal fusion and accurate spatiotemporal modeling, improving diagnostic accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the fields of medical auxiliary diagnosis and computer vision technology, specifically to a multimodal video-assisted diagnosis method and system for Parkinson's disease based on Transformer and attention fusion. Background Technology

[0002] Parkinson's disease (PD) is a common neurodegenerative disease. Its core motor symptoms mainly include resting tremor, bradykinesia, increased muscle tone, and postural balance disorders. Among these, fine motor abnormalities of the hands and gait disorders are the most representative overt motor features and have significant clinical indicative value in the early stages of the disease. They are key targets for early intervention and disease monitoring.

[0003] Currently, the clinical diagnosis of Parkinson's disease still mainly relies on doctors to make a comprehensive judgment based on the patient's medical history, standardized scale assessments, and personal clinical experience. This diagnostic model has inherent limitations: on the one hand, the diagnostic results are easily affected by the differences in doctors' subjective experience, and consistency is difficult to guarantee; on the other hand, the traditional diagnostic process is time-consuming and lacks sensitivity in identifying early mild motor abnormalities, causing a large number of patients to miss the best intervention opportunity, making it difficult to meet the clinical needs of large-scale early screening.

[0004] To address these issues, researchers have been exploring computer vision-based assisted diagnostic technologies. By modeling and analyzing patient motion video data, they aim to provide objective quantitative evidence for disease diagnosis. However, existing technologies still have significant shortcomings: First, some methods only model single-modal videos (e.g., analyzing only hand movements or focusing solely on gait features), failing to comprehensively capture the complex multidimensional motion abnormalities of Parkinson's disease, resulting in insufficient feature characterization. Second, the few methods that incorporate multimodal information often employ simple feature concatenation or fixed-weight fusion strategies, failing to fully consider the differences in discriminative abilities of different modalities across different patient samples, making it difficult to adapt to the feature expression needs arising from disease heterogeneity. Third, traditional convolutional neural networks (CNNs) have limited ability to model long-term temporal motion dynamics and global spatial relationships when processing video data, making it difficult to accurately extract deep spatiotemporal features with diagnostic value. Summary of the Invention

[0005] The purpose of this invention is to propose a multimodal video-assisted diagnostic method and system for Parkinson's disease based on Transformer and attention fusion, so as to achieve accurate binary classification and identification of Parkinson's disease patients and healthy controls.

[0006] According to a first aspect of the embodiments of this disclosure, a multimodal video-assisted diagnostic method for Parkinson's disease based on Transformer and attention fusion is provided, comprising the following steps: Acquired video data of specific hand and gait movements of the subjects, with both types of video data covering complete multiple standard test movements; Preprocessing operations are performed on the obtained hand and gait videos respectively; The preprocessed temporal image sequence is input into the Spatiotemporal Modeling Network (STAF-Net). First, the Visual Transformer (ViT) is used to segment each frame image into blocks, transforming them into local patch features and completing patch-level modeling. The frame-level spatial feature representation of the sequence is obtained through the patch aggregation mechanism. Then, the frame-level spatial features are input into a multi-layer Long Short-Term Memory (LSTM) network for temporal dimension modeling to capture long-term temporal motion dynamic information. After extracting the spatiotemporal features of the hand modality and gait modality respectively, a modality-level spatiotemporal attention fusion method (MSTA) is introduced. Based on the sample features, the weight coefficients of different modalities in the current sample are adaptively learned to achieve weighted fusion of multimodal features. The fused multimodal features are input into a classification network, which outputs prediction results corresponding to Parkinson's disease patients or healthy controls, thus achieving binary classification-assisted diagnosis.

[0007] In one embodiment, video data of the subject's hand and gait-specific movements are acquired, specifically: Acquire hand movement video data and gait movement video data of subjects under standard test tasks: hand movement video focuses on the subject's hand pinching action to reflect their fine motor control ability; gait movement video records the subject's back-and-forth walking action on a designated straight line to characterize their overall motor coordination and gait stability during walking. The above hand and gait video data were collected using a camera. During the collection process, each modality had to complete at least three full test actions. A unified sample identifier was established for hand videos and gait videos corresponding to the same subject, so as to realize the association between the two modal data.

[0008] In one embodiment, the preprocessed temporal image sequence is input into the spatiotemporal modeling network, specifically as follows: Standardize hand and gait video data in the temporal dimension, including video cropping, frame rate unification, and video length equalization. According to the preset frame interval, frame images are extracted from the data video sequence to construct a time-series image sample set; Spatial preprocessing is performed on the extracted frame images from the video, including cropping of the moving subject region, background interference suppression, and image scale normalization; among them, the SAM2 segmentation method is used to segment the gait data into subject and environment. A sliding time window approach is used to segment consecutive video frames, dividing the original video sequence into multiple fixed-length sub-segments, with each sub-segment serving as a temporal modeling sample. During the training phase, data augmentation strategies such as random flipping, color perturbation, and random erasure are introduced into the video frame data.

[0009] In one embodiment, a Visual Transformer (ViT) is used to segment each frame of the image, converting it into local patch features and completing patch-level modeling: The preprocessed video image is divided into several local patches of fixed size, and each patch is mapped to a high-dimensional feature vector. Let the video image be Divide it into The size is The image patches, each of which is expanded and linearly mapped to obtain the PatchEmbedding: in Let represent the nth patch, and be a learnable linear projection matrix. For position encoding; The ViTEncoder structure is introduced to model the global relationships between patch features. ViTEncoder includes Multi-Head Self-Attention (MHSA) and Feedforward Neural Network (FFN). The self-attention mechanism is defined as follows: in, (Query) (Key) and (Value) is a matrix obtained through a linear transformation of the input, representing the query matrix, key matrix, and value matrix, respectively. It is the scaling factor.

[0010] In one embodiment, frame-level spatial features are input into a multi-layer long short-term memory (LSTM) network for temporal dimension modeling. Specifically, the frame-level spatial features are aggregated in chronological order to form a feature sequence. The input is then fed into a multi-layer LSTM network; the input gate, forget gate, and output gate of the LSTM network are defined as follows: in, It uses the Sigmoid activation function, with an output range of [0,1]. The input features at the current time, This indicates the hidden state at the previous moment. and , and , and For learnable weight matrix, This is a bias term.

[0011] In one embodiment, the implementation process of the modal-level spatiotemporal attention fusion method is as follows: Let the spatiotemporal characteristics of the hand mode and gait mode outputs be: Where 𝑇 represents the time step length and 𝐷 represents the feature dimension; The features of the two modalities are shared and mapped to construct a modality description vector: in and For learnable parameters, Represents a non-linear activation function; Normalized attention weights are obtained in the modality dimension using the Softmax function: in This represents the contribution weight of the nth mode to the final decision at time step t. Weighted fusion of hand modal features and gait modal features is performed based on weighting coefficients.

[0012] In one embodiment, the classification network includes two fully connected layers with a GELU activation function and a Dropout operation in between, which map the fused features to the target category space and output classification prediction results corresponding to Parkinson's disease patients or healthy controls.

[0013] According to a second aspect of the present disclosure, a multimodal video-assisted diagnostic system for Parkinson's disease based on Transformer and attention fusion is provided, comprising: The multimodal video acquisition module acquires video data of specific hand and gait movements of the subject. Both types of video data cover complete multiple standard test actions. The video data preprocessing module performs preprocessing operations on the obtained hand and gait videos respectively; The spatiotemporal feature modeling and fusion module inputs the preprocessed temporal image sequence into the spatiotemporal modeling network (STAF-Net). First, it uses a visual Transformer (ViT) to segment each frame image into blocks, transforming them into local patch features and completing patch-level modeling. The frame-level spatial feature representation of the sequence is obtained through a patch aggregation mechanism. Then, the frame-level spatial features are input into a multi-layer long short-term memory network (LSTM) for temporal dimension modeling to capture long-term temporal motion dynamic information. After extracting the spatiotemporal features of the hand modality and gait modality respectively, a modality-level spatiotemporal attention fusion method (MSTA) is introduced. Based on the sample features, the weight coefficients of different modalities in the current sample are adaptively learned to achieve weighted fusion of multimodal features. The binary classification diagnostic output module inputs the fused multimodal features into the classification network and outputs the prediction results corresponding to Parkinson's disease patients or healthy controls, thereby achieving binary classification assisted diagnosis.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the memory, wherein the processor executes the program to implement the aforementioned multimodal video-assisted diagnostic method for Parkinson's disease based on Transformer and attention fusion.

[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the aforementioned multimodal video-assisted diagnostic method for Parkinson's disease based on Transformer and attention fusion.

[0016] The advantages of the above technical solutions adopted in this invention compared with the prior art are as follows: 1. This invention integrates two core overt motor modalities of Parkinson's disease: hand and gait. The hand modality focuses on fine motor control, while the gait modality reflects overall motor coordination and stability. The data from the two modalities are complementary and interconnected. Compared with single-modality analysis methods, this invention can capture disease-related motor abnormalities more comprehensively and three-dimensionally, providing richer quantitative evidence for diagnostic decisions and effectively making up for the deficiency of single-modality analysis in covering the multidimensional motor manifestations of the disease.

[0017] 2. A dual modeling strategy combining Visual Transformer (ViT) and Multilayer Long Short-Term Memory (LSTM) networks is employed. ViT efficiently captures global spatial relationships and local detail features within image frames through patch-level modeling and global attention mechanisms; while LSTM captures long-term temporal motion dynamics between frames through gating mechanisms, effectively mitigating the gradient vanishing problem in long sequence modeling. The synergistic effect of these two technologies enables in-depth mining of the spatial structure and temporal dynamics of video data, significantly improving the accuracy and effectiveness of feature representation.

[0018] 3. A modality-level spatiotemporal attention fusion mechanism (MSTA) is introduced, which adaptively learns and dynamically adjusts the weight coefficients of hand and gait modalities based on the differences in discriminative abilities of different modalities in different samples, explicitly optimizing the modality contribution ratio at each time step. This mechanism ensures the complementarity of multimodal information while effectively suppressing the interference of redundant modalities or noise information on diagnostic decisions, significantly improving the model's adaptability to disease heterogeneity and the reliability of diagnostic results.

[0019] 4. This invention utilizes video data for assisted diagnosis, employing a non-contact acquisition method throughout. It eliminates the need for subjects to wear additional sensing devices, making it convenient and non-invasive, significantly reducing the barriers to subject cooperation and acquisition costs. Furthermore, the acquisition process is highly standardized, adaptable to various indoor and outdoor open environments, and easily promoted and applied in primary healthcare institutions and community screening settings. It provides a feasible technical solution for large-scale early screening and daily monitoring of Parkinson's disease, demonstrating broad practical application prospects. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0021] Figure 1 This is a flowchart of the multimodal video-assisted diagnosis method for Parkinson's disease based on Transformer and attention fusion in this invention; Figure 2 A flowchart for data preprocessing; Figure 3 Network structure diagram for extracting spatiotemporal features of data; Figure 4 This is a schematic diagram of the MSTA attention fusion method; Figure 5 This is a diagram of the overall architecture of the STAF-Net network. Detailed Implementation

[0022] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0025] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0026] Example 1: like Figure 1 As shown, this embodiment provides a multimodal video-assisted diagnostic method for Parkinson's disease based on Transformer and attention fusion, including the following steps: S1. Acquire video data of specific hand and gait movements of the subject, with both types of video data covering complete multiple standard test movements; Video acquisition equipment was set up in open indoor and outdoor areas, and the acquisition area was arranged in a standardized manner to ensure the standardization and stability of the data acquisition process. For hand data acquisition, subjects were required to complete the pinching and separating action of the thumb and index finger at a designated location; for gait data acquisition, subjects were required to complete periodic back-and-forth walking actions within a designated straight line area according to the acquisition requirements.

[0027] The acquisition equipment used was a Panasonic HDC-SD60 color camera, which was connected to a laptop for data acquisition and storage via USB cable during the acquisition process. The camera was kept stationary throughout the process, with a consistent acquisition angle and height to minimize the interference of external factors on data quality. Video data was stored continuously, and resolution and frame rate parameters were uniformly set to ensure consistent video data format across different modalities and samples.

[0028] By configuring a unified sample identifier for hand and gait data collected from the same subject, the association and binding of video data from different modalities can be achieved, ensuring the pairing and analysis of the two types of modal data in the future.

[0029] S2. Perform preprocessing operations on the obtained hand and gait videos respectively; like Figure 2 As shown, after acquiring the video data, the first step is to standardize it in the time dimension, specifically including: 1) trimming the original video length and removing invalid start and end segments; 2) unifying the frame rate of different source videos to a preset standard frame rate; 3) performing equal-length processing on the video sequence. For videos with insufficient length, a loop filling method is used to fill the video. For videos with excessive length, the length is trimmed according to the actual situation to ensure that all samples have a consistent frame rate in the time dimension.

[0030] After completing the temporal alignment, discrete frame images are extracted from the standardized video sequence at equal intervals according to a pre-set fixed frame interval Δt, thus constructing a temporal image sample dataset. Let the number of frames in the original video sequence be *x*, and the target number of sampled frames be *y*, then the sampling index interval corresponding to frame *x* can be represented as: in, This involves rounding down, specifically using the values ​​𝑁=800 and 𝑇=53, meaning that 53 images can be extracted from each video sample as a temporal image sample dataset. A representative frame is selected within this interval as the 𝑗-th temporal node, thus mapping the original continuous video into a discrete frame image sequence with equal time intervals. This process ensures the integrity of temporal information while effectively reducing the impact of redundant frames on the computational complexity of the model, laying the foundation for subsequent temporal feature modeling.

[0031] Spatial preprocessing is performed on each sampled frame image, with the following steps: First, the main subject region is cropped based on motion information to remove irrelevant background and highlight the subject. Second, the SAM2 segmentation method is used to segment the gait data between the subject and the environment, replacing background differences caused by variations in the subject's environment with the same color to reduce the interference of environmental factors on network modeling. Finally, the image size is normalized to match the network input size requirements. SAM2, proposed by Meta in 2024, is a novel plug-and-play segmentation method. This method applies cue information such as points, bounding boxes, or masks to the input data, combining visual feature encoding and cross-frame memory mechanisms to achieve automatic segmentation and continuous tracking of target regions in video sequences, thereby obtaining temporally consistent foreground regions even under complex background conditions. Through the above preprocessing steps, redundant background interference can be effectively removed, highlighting key subject regions and providing high-quality, structurally consistent input data for subsequent feature extraction and model training.

[0032] To fully characterize the local temporal dynamics of long sequences, a sliding time window mechanism is used to segment consecutive video frames. For example... Figure 4 As shown, with a window length T of 31 and a step size φ of 11, the complete time series is divided into multiple overlapping subsequences such as w1, w2, and w3. This segmentation method can effectively expand the amount of training data while ensuring that each sequence segment contains complete action information, alleviate the overfitting problem during model training, and improve the model's generalization ability and robustness.

[0033] During the model training phase, various random data augmentation strategies are introduced into the video frame data, including random horizontal flipping, random color jitter, and random erasing, to simulate pose changes, lighting changes, and local occlusion in real-world scenes, effectively improving the model's robustness in complex environments.

[0034] S3. The preprocessed temporal image sequence is input into the spatiotemporal modeling network (STAF-Net). First, the visual Transformer (ViT) is used to segment each frame image into blocks, transforming it into local patch features and completing patch-level modeling. The frame-level spatial feature representation of the sequence is obtained through the patch aggregation mechanism. Then, the frame-level spatial features are input into a multi-layer long short-term memory network (LSTM) for temporal dimension modeling to capture long-term temporal motion dynamic information. After completing the spatiotemporal feature extraction of hand modality and gait modality respectively, a modality-level spatiotemporal attention fusion method (MSTA) is introduced. Based on the sample features, the weight coefficients of different modalities in the current sample are adaptively learned to achieve weighted fusion of multimodal features. like Figure 3As shown, the preprocessed temporal image sequence is used as the input to the STAF-Net network. First, spatial feature modeling is performed on each frame. Specifically, a spatial feature extraction module based on VisionTransformer (ViT) is used to divide the input image into several local patches of fixed size, and each patch is mapped to a high-dimensional feature vector. Let the input image be... Divide it into The size is The image patches, each of which is expanded and linearly mapped to obtain the PatchEmbedding: in Let represent the nth patch, and be a learnable linear projection matrix. For location encoding, spatial location information of the patch is preserved. Based on this, the ViTEncoder structure is introduced to model the global relationships between patch features. ViTEncoder includes Multi-Head Self-Attention (MHSA) and a Feed-Forward Network (FFN), and its self-attention mechanism is defined as follows: in, (Query) (Key) and (Value) is a matrix obtained through a linear transformation of the input, representing the query matrix, key matrix, and value matrix, respectively. This is the scaling factor. The Transformer itself does not include an explicit sequence order modeling mechanism, therefore positional encoding is needed to provide the model with the positional information of each element in the sequence. Positional encoding can be represented using sine / cosine functions: in, Indicates location, Indicates a dimension index. This refers to the dimension of the model. Position encoding is typically added to the input features, enabling the model to perceive the relative or absolute order of the sequence.

[0035] Structurally, the Transformer comprises multiple stacked encoder modules, each containing multi-head self-attention layers and feedforward neural network layers. Residual connections and layer normalization operations enhance training stability and convergence speed. This structure gives the Transformer excellent scalability and expressive power. Unlike traditional RNNs and LSTMs, the Transformer can process the entire sequence in parallel, significantly improving training efficiency. Its powerful global dependency modeling capability ensures that regardless of the distance between words, the Transformer can directly establish dependencies, solving the problem of long-distance dependencies.

[0036] Based on the temporal dependencies between frames in a video sequence, the obtained frame-level spatial features are aggregated in temporal order to form a feature sequence: The aforementioned feature sequences are then input into a multilayer Long Short-Term Memory (LSTM) network for temporal modeling. LSTM effectively alleviates the vanishing gradient problem in traditional recurrent neural networks during long sequence modeling by introducing a gating mechanism. Its input gate, forget gate, and output gate are defined as follows: in, It uses the Sigmoid activation function, with an output range of [0,1]. The input features at the current time, This indicates the hidden state at the previous moment. and , and , and For learnable weight matrix, , This is a bias term. , , These represent how much of the current input information is written into the memory unit, how much of the previous memory is retained, and how much of the current memory is used as output. These three are not responsible for generating content, but only for controlling the proportion of information flow.

[0037] By stacking multiple layers of LSTM networks, the model's ability to model complex temporal dynamics can be further enhanced, thereby extracting high-level temporal feature representations reflecting hand movement and gait changes. The introduction of a gating mechanism allows LSTM to selectively memorize key information and forget irrelevant content, giving it greater stability and expressive power when processing long-span sequences. Compared to traditional recurrent neural networks, LSTM is more stable in modeling long-term temporal dependencies and exhibits robustness to noise and input fluctuations.

[0038] like Figure 5 As shown, modal feature attention fusion is completed at the network endpoint. After completing the spatiotemporal feature modeling of the hand modality and gait modality, high-level spatiotemporal feature representations of the two modalities are obtained respectively. However, directly concatenating or weighting multimodal features often implicitly assumes that "each modality contributes equally," making it difficult to reflect the differences in the importance of different modalities in disease discrimination across different samples and time periods. This can easily introduce redundant information and even lead to noisy modalities interfering with the final decision.

[0039] To address the aforementioned issues, this invention introduces a modality-level spatiotemporal attention fusion mechanism (MSTA) at the end of the STAF-Net network. By explicitly modeling the contribution weights of different modalities in the temporal and semantic dimensions, it achieves adaptive selection and fusion of multimodal features. Let the spatiotemporal features output by the hand modality and gait modality be: Where 𝑇 represents the time step length and 𝐷 represents the feature dimension. Traditional fusion methods typically employ: Fusion1 is a feature concatenation fusion method, and fusion2 is a fixed-weight fusion method. These methods fail to consider the differences in discriminative ability between different modalities at the same time step, and also fail to account for the unbalanced dependence of PD pathological manifestations on different motion modalities. Therefore, this invention introduces a modality-level attention weight learning mechanism (MSTA) to dynamically model the contributions of different modalities at each time step.

[0040] First, the features of the two modalities are shared and mapped to construct the modality description vector: in and For learnable parameters, This represents a non-linear activation function. Subsequently, normalized attention weights are obtained along the modality dimension using the Softmax function: in This represents the contribution weight of the *i*th modality to the final decision at time step *i*. The attention weights in this design are calculated in the modality dimension rather than the channel or spatial dimension, explicitly characterizing the discriminative relationship of "which modality the current time step relies on more," and dynamically adjusting the contribution ratios of different modalities based on sample characteristics. This ensures information complementarity while suppressing the interference of noisy modalities on the final decision.

[0041] S4. Input the fused multimodal features into the classification network and output the prediction results corresponding to Parkinson's disease patients or healthy controls to achieve binary classification-assisted diagnosis.

[0042] The multimodal spatiotemporal fusion features obtained in the preceding steps are used as input and fed into a classification decision network for discrimination processing. This classification head consists of two fully connected layers, with the GELU activation function and Dropout regularization operation introduced in the middle of the network to improve the nonlinear expressive power of the model and effectively alleviate the overfitting problem. Finally, the output is the prediction result corresponding to the number of categories.

[0043] Example 2: This embodiment provides a multimodal video-assisted diagnostic system for Parkinson's disease based on Transformer and attention fusion, including: The multimodal video acquisition module acquires video data of specific hand and gait movements of the subject. Both types of video data cover complete multiple standard test actions. The video data preprocessing module performs preprocessing operations on the obtained hand and gait videos respectively; The spatiotemporal feature modeling and fusion module inputs the preprocessed temporal image sequence into the spatiotemporal modeling network (STAF-Net). First, it uses a visual Transformer (ViT) to segment each frame image into blocks, transforming them into local patch features and completing patch-level modeling. The frame-level spatial feature representation of the sequence is obtained through a patch aggregation mechanism. Then, the frame-level spatial features are input into a multi-layer long short-term memory network (LSTM) for temporal dimension modeling to capture long-term temporal motion dynamic information. After extracting the spatiotemporal features of the hand modality and gait modality respectively, a modality-level spatiotemporal attention fusion method (MSTA) is introduced. Based on the sample features, the weight coefficients of different modalities in the current sample are adaptively learned to achieve weighted fusion of multimodal features. The binary classification diagnostic output module inputs the fused multimodal features into the classification network and outputs the prediction results corresponding to Parkinson's disease patients or healthy controls, thereby achieving binary classification assisted diagnosis.

[0044] The above modules can be deployed on the same device or distributed devices; the division of modules is only a functional logic description and does not limit the specific physical boundaries or implementation order.

[0045] Example 3: An electronic device is provided for running the aforementioned "a multimodal video-assisted diagnostic method for Parkinson's disease based on Transformer and attention fusion". The electronic device includes a processor, a memory, and optional communication interfaces / display devices / input devices, etc.; the memory stores a computer program that can run on the processor, and when the processor executes the program, it implements steps S1 to S4 of the method described in Embodiment 1, specifically including but not limited to: S1. Acquire video data of specific hand and gait movements of the subject, with both types of video data covering complete multiple standard test movements; S2. Video data preprocessing module, which performs preprocessing operations on the obtained hand and gait videos respectively; S3. The spatiotemporal feature modeling and fusion module inputs the preprocessed temporal image sequence into the spatiotemporal modeling network (STAF-Net). First, it uses a visual Transformer (ViT) to segment each frame image into blocks, transforming them into local patch features and completing patch-level modeling. The frame-level spatial feature representation of the sequence is obtained through a patch aggregation mechanism. Then, the frame-level spatial features are input into a multi-layer long short-term memory network (LSTM) for temporal dimension modeling to capture long-term temporal motion dynamic information. After extracting the spatiotemporal features of the hand modality and gait modality respectively, a modality-level spatiotemporal attention fusion method (MSTA) is introduced. Based on the sample features, the weight coefficients of different modalities in the current sample are adaptively learned to achieve weighted fusion of multimodal features. S4. The binary classification diagnostic output module inputs the fused multimodal features into the classification network and outputs the prediction results corresponding to Parkinson's disease patients or healthy controls, thereby achieving binary classification assisted diagnosis.

[0046] The electronic device hardware can be one of a server, personal computer, workstation, industrial controller, edge computing device, or mobile terminal; the processor can be a general-purpose CPU, GPU, NPU, FPGA, or a combination thereof; the memory can be RAM, ROM, flash memory, or disk array. The device can interact with local / remote data storage (acquiring observation data and outputting inversion results) through a communication interface. The above hardware configuration does not constitute a limitation of the present invention.

[0047] Example 4: A computer-readable storage medium storing a computer program, which, when run on a processor of an electronic device, causes the program to perform the method steps S1 to S4 described in Embodiment 1; the storage medium may be a disk, optical disk, flash memory, solid-state drive, read-only memory, random access memory, or any combination of the above media.

[0048] Those skilled in the art will understand that the modules or steps described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, which can then be stored in a storage device for execution by a computer device. Alternatively, they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. This disclosure is not limited to any particular combination of hardware and software.

[0049] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0050] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A multimodal video-assisted diagnostic method for Parkinson's disease based on Transformer and attention fusion, characterized in that, Includes the following steps: Acquired video data of specific hand and gait movements of the subjects, with both types of video data covering complete multiple standard test movements; Preprocessing operations are performed on the obtained hand and gait videos respectively; The preprocessed temporal image sequence is input into the spatiotemporal modeling network. First, the visual Transformer is used to segment each frame image into blocks, transforming them into local patch features and completing patch-level modeling. The frame-level spatial feature representation of the sequence is obtained through the patch aggregation mechanism. Then, the frame-level spatial features are input into a multi-layer long short-term memory network for temporal dimension modeling to capture long-term motion dynamic information. After extracting the spatiotemporal features of the hand modality and gait modality respectively, a modality-level spatiotemporal attention fusion method is introduced. Based on the sample features, the weight coefficients of different modalities in the current sample are adaptively learned to achieve weighted fusion of multimodal features. The fused multimodal features are input into a classification network, which outputs prediction results corresponding to Parkinson's disease patients or healthy controls, thus achieving binary classification-assisted diagnosis.

2. The multimodal video-assisted diagnosis method for Parkinson's disease based on Transformer and attention fusion as described in claim 1, characterized in that, Acquire video data of specific hand and gait movements of the subjects, specifically: Acquire hand movement video data and gait movement video data of subjects under standard test tasks: hand movement video focuses on the subject's hand pinching action to reflect their fine motor control ability; gait movement video records the subject's back-and-forth walking action on a designated straight line to characterize their overall motor coordination and gait stability during walking. The above hand and gait video data were collected using a camera. During the collection process, each modality had to complete at least three full test actions. A unified sample identifier was established for hand videos and gait videos corresponding to the same subject, so as to realize the association between the two modal data.

3. The multimodal video-assisted diagnosis method for Parkinson's disease based on Transformer and attention fusion as described in claim 1, characterized in that, The preprocessed temporal image sequence is input into the spatiotemporal modeling network, specifically as follows: Standardize hand and gait video data in the temporal dimension, including video cropping, frame rate unification, and video length equalization. According to the preset frame interval, frame images are extracted from the data video sequence to construct a time-series image sample set; Spatial preprocessing is performed on the extracted frame images from the video, including cropping of the moving subject region, background interference suppression, and image scale normalization; among them, the SAM2 segmentation method is used to segment the gait data into subject and environment. A sliding time window approach is used to segment consecutive video frames, dividing the original video sequence into multiple fixed-length sub-segments, with each sub-segment serving as a temporal modeling sample. During the training phase, data augmentation strategies such as random flipping, color perturbation, and random erasure are introduced into the video frame data.

4. The multimodal video-assisted diagnosis method for Parkinson's disease based on Transformer and attention fusion as described in claim 1, characterized in that, The visual Transformer is used to segment each frame of the image, converting it into local patch features and completing patch-level modeling. The preprocessed video image is divided into several local patches of fixed size, and each patch is mapped to a high-dimensional feature vector. Let the video image be Divide it into The size is The image patches, each of which is expanded and linearly mapped to obtain the PatchEmbedding: in Let represent the nth patch, and be a learnable linear projection matrix. For position encoding; The ViTEncoder structure is introduced to model the global relationships between patch features. ViTEncoder includes multi-head self-attention and a feedforward neural network. The self-attention mechanism is defined as follows: in, (Query) (Key) and (Value) is a matrix obtained through a linear transformation of the input, representing the query matrix, key matrix, and value matrix, respectively. It is the scaling factor.

5. The multimodal video-assisted diagnosis method for Parkinson's disease based on Transformer and attention fusion as described in claim 1, characterized in that, Frame-level spatial features are input into a multi-layer long short-term memory network for temporal dimension modeling. Specifically, the frame-level spatial features are aggregated in chronological order to form a feature sequence. The input is then fed into a multi-layer LSTM network; the input gate, forget gate, and output gate of the LSTM network are defined as follows: in, It uses the Sigmoid activation function, with an output range of [0,1]. The input features at the current time, This indicates the hidden state at the previous moment. and , and , and For learnable weight matrix, This is a bias term.

6. The multimodal video-assisted diagnosis method for Parkinson's disease based on Transformer and attention fusion as described in claim 1, characterized in that, The implementation process of the modal-level spatiotemporal attention fusion method is as follows: Let the spatiotemporal characteristics of the hand modality and gait modality outputs be: Where 𝑇 represents the time step length and 𝐷 represents the feature dimension; The features of the two modalities are shared and mapped to construct a modality description vector: in and For learnable parameters, Represents a non-linear activation function; Normalized attention weights are obtained in the modality dimension using the Softmax function: in This represents the contribution weight of the i-th mode to the final decision at time step i. Weighted fusion of hand modal features and gait modal features is performed based on weighting coefficients.

7. The multimodal video-assisted diagnosis method for Parkinson's disease based on Transformer and attention fusion as described in claim 1, characterized in that, The classification network consists of two fully connected layers with a GELU activation function and Dropout operation in between. It maps the fused features to the target category space and outputs the classification prediction results corresponding to Parkinson's disease patients or healthy controls.

8. A multimodal video-assisted diagnostic system for Parkinson's disease based on Transformer and attention fusion, characterized in that, include: The multimodal video acquisition module acquires video data of specific hand and gait movements of the subject. Both types of video data cover complete multiple standard test actions. The video data preprocessing module performs preprocessing operations on the obtained hand and gait videos respectively; The spatiotemporal feature modeling and fusion module inputs the preprocessed temporal image sequence into the spatiotemporal modeling network. First, it uses a visual Transformer to segment each frame image into blocks, transforming them into local patch features and completing patch-level modeling. The frame-level spatial feature representation of the sequence is obtained through a patch aggregation mechanism. Then, the frame-level spatial features are input into a multi-layer long short-term memory network for temporal dimension modeling to capture long-term motion dynamic information. After extracting the spatiotemporal features of the hand modality and gait modality respectively, a modality-level spatiotemporal attention fusion method is introduced. Based on the sample features, the weight coefficients of different modalities in the current sample are adaptively learned to achieve weighted fusion of multimodal features. The binary classification diagnostic output module inputs the fused multimodal features into the classification network and outputs the prediction results corresponding to Parkinson's disease patients or healthy controls, thereby achieving binary classification assisted diagnosis.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and running thereon, characterized in that, When the processor executes the program, it implements the multimodal video-assisted diagnosis method for Parkinson's disease based on Transformer and attention fusion as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal video-assisted diagnostic method for Parkinson's disease based on Transformer and attention fusion as described in any one of claims 1-7.

Images