Shaking symptom evaluation model based on static handwriting image and dynamic multi-sensor fusion signal and device thereof

Abstract

The application provides a tremor symptom evaluation model based on a static handwriting image and a dynamic multi-sensor fusion signal and a device thereof, which comprises a static space branch, a dynamic time branch and a backbone network; the static space branch is used for learning global spatial features of the static handwriting image; the dynamic time branch is used for learning spectrogram features in a time-frequency domain of the dynamic multi-sensor fusion signal; and the backbone network is used for fusing the global spatial features, the spectrogram features in the time-frequency domain and EHR features, so as to evaluate the tremor symptom. In addition, the method is verified by ablation research to verify the effectiveness of an attention mechanism, a transfer learning strategy and introduction of prior knowledge based on population characteristics.

Description

Technical Field

[0001] This application relates to tremor symptom assessment technology, and more particularly to a tremor symptom assessment model and device based on static handwritten images and dynamic multi-sensor fusion signals. Background Technology

[0002] Essential tremor (ET) is one of the most common movement disorders, with a prevalence of 5.79% in people over 65 years of age. Prospective studies have shown that patients with ET have a more than fourfold increased risk of developing Parkinson's disease (PD). Motor characteristics are the primary manifestation of ET. The Archimedes spiral-based handwriting task, involving sensorimotor and cognitive functions, is generally considered an effective diagnostic tool. However, clinically, ET tremor is easily confused with diseases such as PD and dystonic tremor. Neurologists rely primarily on visual characteristics for diagnosis of handwriting tasks; for example, the patient's symptoms worsen with the number of times the handwriting overlaps with the template. However, the clinical gold standard—the Clinical Tremor Scoring Scale (CRST)—does not specify detailed quantitative numbers. Due to the ambiguity in the description of overlap, it is easy for physicians to be confused; therefore, clinical scoring is mostly subjective, especially for grades 2 and 3.

[0003] With the development of advanced sensing and computer technologies, handwriting-based assistive diagnostic tools can provide real-time, low-cost, and objective solutions to support standard clinical assessments by neurologists. Many multi-sensor acquisition systems based on digitizers have emerged. These dynamic information acquisition tools can capture various dynamic characteristics of the writing process, including temporal variations (such as changes in pressure applied to the paper surface) and spatial information, such as pen tip orientation and tilt. Furthermore, high-dimensional dynamic features of the writing process can also be extracted from these signals. For example, Drottár et al. calculated features based on entropy and spectral energy and extracted standard kinematic features. Impedovo et al. also designed features in the Maxwell-Boltzmann and Sigma-Lognormal transform domains. These high-dimensional features were fed into a machine learning classifier to distinguish between PD patients and healthy controls (HC). However, because these abstract high-dimensional features representing multi-sensor fusion signals are compressed into single-valued vectors, much information is lost, and this compressed information could potentially be helpful for a more refined assessment of tremor severity.

[0004] Inspired by clinicians' diagnosing tremor severity visually, some scholars believe that the global spatial information provided by visual features is equally important. Recently, Moetesum et al. converted raw signals captured by pen-tip sensors into handwritten images and then designed a raw image network, a median-residual network, and an edge image network to automatically extract useful visual features. Finally, they fed this into an SVM classifier to distinguish between PD and HC, achieving an accuracy of 76.08%. Furthermore, other studies have designed deep end-to-end networks that can directly input converted images into the network for prediction. However, handwriting images converted from signals do not contain important template information, which is a crucial reference for expert diagnosis. Directly feeding handwriting images without templates into the network may lead to non-convergence. Summary of the Invention

[0005] In view of the above problems, this application aims to propose a tremor symptom assessment model and device based on static handwritten images and dynamic multi-sensor fusion signals.

[0006] This application proposes a tremor symptom assessment model based on static handwritten images and dynamic multi-sensor fusion signals, which includes: a static spatial branch, a dynamic temporal branch, and a backbone network;

[0007] The static spatial branch is used to learn the global spatial features of static handwritten images;

[0008] Dynamic time branching is used to learn the time-frequency domain spectral characteristics of dynamic multi-sensor fusion signals;

[0009] The backbone network is used to fuse global spatial features, time-frequency spectral features, and EHR features to assess tremor symptoms.

[0010] This application proposes a tremor symptom assessment device based on static handwritten images and dynamic multi-sensor fusion signals, which includes: a computing unit; the computing unit runs a tremor symptom assessment model based on static handwritten images and dynamic multi-sensor fusion signals; the tremor symptom assessment model includes: a static spatial branch, a dynamic temporal branch, and a backbone network;

[0011] The static spatial branch is used to learn the global spatial features of static handwritten images;

[0012] Dynamic time branching is used to learn the time-frequency domain spectral characteristics of dynamic multi-sensor fusion signals;

[0013] The backbone network is used to fuse global spatial features, time-frequency spectral features, and EHR features to assess tremor symptoms.

[0014] Preferably, the static spatial branch is a fine-tuned DenseNet-121 network; the fine-tuned DenseNet-121 network retains the pre-trained weights other than the last two layers of the DenseNet-121 network.

[0015] Images of large spirals, small spirals, or straight lines in static handwritten images are input into the static space branch.

[0016] Preferably, the dynamic time branch includes a second Conv 7-32 layer; the time-frequency domain spectral features of the dynamic multi-sensor fusion signal transformation by a large spiral, a small spiral, or a straight line are input into the second Conv 7-32 layer of the dynamic time branch;

[0017] The backbone network is a CNN network incorporating the SE-Module attention mechanism; it includes the following sequentially connected inputs: the first Conv 7-32 layers, the first concatenation layer fused with dynamic temporal branch features, the first Max pool 3 layers, Conv 6-64 layers, the second Max pool 3 layers, Conv 5-128 layers, the first Conv 3-128 layers, the second Conv 3-128 layers, the third Max pool 3 layers, Conv 3-256 layers, the first SE-Module attention mechanism layer, the second concatenation layer fused with static spatial branch features, the Adapt pool 3 layers, the Flatten layer, the first FC layer, the third concatenation layer fused with EHR features, the second FC layer, and the Output layer; images of large spirals, small spirals, or straight lines in static handwritten images are simultaneously input into the first Conv 7-32 layers of the backbone network;

[0018] The output of the static spatial branch is spliced ​​in the second splicing layer of the backbone network after passing through the Conv 3-512 layer and the second SE-Module attention mechanism layer;

[0019] The output of the dynamic time branch is spliced ​​in the first splicing layer of the backbone network;

[0020] EHR features are spliced ​​in the third splicing layer of the backbone network;

[0021] The output layer of the backbone network outputs the tremor symptom assessment results.

[0022] Preferably, the time-frequency domain spectrum of the dynamic multi-sensor fusion signal is the time-frequency domain spectrum obtained from the dynamic features of the pen tip.

[0023] 1) To design effective features for fine-grained ET tremor quantization, this application combines static writing images with templates and dynamic pen tip sensing signals. To effectively fuse temporal-spatial information, the method of this invention designs an attention-based deep learning network, namely a multimodal fusion neural network (MMFNet).

[0024] 2) This application utilizes transfer learning strategies, time-spectrum graphs describing spatiotemporal changes, and prior knowledge describing population information; and designs a joint optimization strategy to help the model obtain effective features with complementary information during the learning process.

[0025] 3) Finally, this application discusses its utility in quantifying the severity of ET symptoms and in the differential diagnosis of PD. This application tests the proposed MMFNet on three publicly available datasets and compares the results with those of state-of-the-art (SOTA) methods. These experimental results demonstrate the effectiveness and efficiency of the proposed method. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the tremor symptom assessment model and device based on static handwritten images and dynamic multi-sensor fusion signals of this application.

[0027] Figure 2 This is a schematic diagram of the structure of the tremor symptom assessment model based on static handwritten images and dynamic multi-sensor fusion signals of this application;

[0028] Figure 3 This is a flowchart illustrating the diagnostic process of the tremor symptom assessment model based on static handwritten images and dynamic multi-sensor fusion signals proposed in this application.

[0029] Figure 4 Box plot (five-fold cross-validation) of the transfer learning model in the dataset collected by the method of this invention;

[0030] Figure 5 Attention heatmaps of the tremor symptom assessment model (MMFNet) based on static handwritten images and dynamic multi-sensor fusion signals in this application on multiple handwriting tasks;

[0031] Figure 6 ROC curves of the tremor symptom assessment model (MMFNet) based on static handwritten images and dynamic multi-sensor fusion signals in this application on multiple handwriting tasks;

[0032] Figure 7 The confusion matrices of the tremor symptom assessment model (MMFNet) based on static handwritten images and dynamic multi-sensor fusion signals in this application are shown on the (a) large spiral, (b) small spiral and (c) straight line tasks, respectively. Detailed Implementation

[0033] The present application will now be described in detail with reference to the accompanying drawings.

[0034] The specific steps and implementation principles of the model in this application are as follows:

[0035] (1) An electronic digitizing tablet is designed to simultaneously record the patient's paper and electronic handwriting. A high-precision sensing digitizing pen is used to transmit the automatically rendered electronic handwriting to the local computer, while the raw data stream is transmitted to the cloud to ensure data portability. Specifically, in this embodiment, the device used for data acquisition is the Intuos Pro Paper Edition (430×287×8mm), which has 8,192 pen pressure levels and supports drawing on test paper using a Wacom Finetip Pen. Simultaneously, the rendered electronic handwriting is saved on the computer, and the raw data stream from the cloud can be converted into image formats such as jpg and npg. The Wacom Ink Layer Language (WILL) format includes the rendering result, collected sensor data, and related metadata streams. The method of this invention designs software specifically for exporting sequence information, facilitating the extraction of relevant information describing tremor.

[0036] (2) Data collection. Patients can use a digital pen to simultaneously record handwriting information on paper and a digital tablet. Specifically, a high-precision digital tablet is used to collect hand tremor data of patients under evoked movements during multiple handwriting tremor tasks; at the same time, the CRST scale is used to score the completion of the designated movements by the patients; for later follow-up and assessment of changes in the condition and for blind review and scoring by multiple neurological experts, professional neurologists record the patient's movements throughout the laboratory examination.

[0037] Preferably, the entire laboratory examination is guided by a neurologist specializing in movement disorders, and video data (CMOS camera, 48MP, 1920*1080 HD, 60 frames / second) is recorded to support independent scoring by three neurologists (mutually blinded). During the laboratory examination, the patient completes drawing and writing tasks on a digital tablet according to prompts, and the digital signal is wirelessly transmitted to the host computer via Bluetooth. The host computer displays the rendering effect of the electronic handwriting in real time and stores it on the computer's hard drive as a rendered image. To ensure the acquisition of valid tremor signals, the sampling frequency of the digital pen is set to 100Hz.

[0038] Preferably, two experts jointly score the tremor. If there is a discrepancy in the scores, a third expert is invited to review the video recording and make the final determination. The specific scoring process is as follows: 1) Two neurologists score the severity of the patient's action tremor by watching the video footage and observing the remaining handwriting, obtaining two mutually blinded scoring sheets. 2) A data analysis engineer tallies the consistent scores from the two experts. For actions with inconsistent scores, another experienced neurologist makes the final decision based on the video footage and the remaining handwriting until a reliable score is obtained. This experimental design also avoids training errors caused by systematic bias, making the machine learning model more reliable.

[0039] Specifically, the data acquisition unit includes an electronic digitizing tablet, a high-precision digital pen, a computer, a camera, a scanner, a wireless local area network, and data cables. An electronic handwriting database can be established based on the data acquisition unit, including paper handwriting, electronic handwriting, video recordings, and patient information from doctor consultations and examinations (including demographic information, past medical history, and scale tests).

[0040] Specifically, the designated writing actions include Archimedean spirals of different scales and straight-line tasks. The tremor symptoms induced by the designated actions fall under the category of action tremor tasks. This writing task, based on Part A of the CRST standard scale, requires the patient to sit comfortably in a chair, maintain a proper posture while writing, and ensure that the pen tip does not leave the paper as much as possible, that the handwriting remains within the boundaries, and that the writing is continuous from the starting point to the ending point.

[0041] The use of the CRST scale to score the patient's completion of a designated action refers to a professional physician judging the severity of tremor in the patient's writing task based on the completion of the designated action. As shown in Table 1, the tremor is divided into 5 levels: 0, 1, 2, 3, and 4. The CRST scale indicates that the tremor symptoms of the 5 levels of postural tremor are as follows: 1) Normal handwriting; 2) Mild, the handwriting occasionally touches the boundary; 3) Moderate amplitude, the handwriting frequently touches the boundary; 4) Large amplitude, the task is very difficult to complete, and errors may occur; 5) Large amplitude, the writing task cannot be completed.

[0042] Table 1. Rules for scoring tremor severity using various handwriting tasks in CRST

[0043]

[0044]

[0045] (3) Figure 1This diagram illustrates the flowchart of the tremor symptom assessment model and device based on static handwritten images and dynamic multi-sensor fusion signals of this application. The original pen tip motion data can be obtained through a digitizer, from which multimodal digital ink features are derived. Simultaneously, the digitizer can store the paper material containing the handwritten text, and image data corresponding to the paper writing material can be obtained through scanning. Traditional subjective evaluation relies on observing images rendered using relative coordinates, while the system based on deep learning algorithms proposed in this application can obtain objective evaluation references through static handwritten images and dynamic multi-sensor fusion signals. The deep learning algorithm learns the visual features of static handwritten images and fuses time-spectrum features transformed from different pen tip dynamic features to efficiently assess tremor: 1) It combines a transfer learning strategy to obtain visual features from handwritten images through a deep neural network; 2) It fuses the original sensor data stream derived from electronic handwriting with the high-level sequence features proposed in this invention into multi-sensor channel sequence data, and transforms it into a time-spectrum for multimodal fusion learning. Demographic parameters from electronic medical records are fused before the decision layer of the deep learning algorithm to obtain a more comprehensive and efficient feature representation.

[0046] A graphics tablet can simultaneously acquire raw pen tip motion data and paper writing data, from which multimodal dynamic electronic handwriting temporal spectrum features and high-dimensional visual features are derived. Traditional subjective evaluations rely on observing images rendered with relative coordinates, while the deep learning-based system proposed in this application obtains objective evaluations by learning and fusing high-dimensional dynamic information. Specifically, the pipeline includes: 1) Data preprocessing: Processing handwritten images and matched dynamic multisensor signals into input formats suitable for the model. 2) Modeling: The proposed MMFNet model can simultaneously fuse prior knowledge and pre-trained model weights. 3) Performance evaluation: Using the consensus score derived by a neuroscience expert committee as the gold standard, the model's classification performance is evaluated on a test set. Furthermore, the model's heatmap can show its attention to tremor details.

[0047] (4) Figure 2This paper presents the overall architecture and structural details of the multi-modal fusion neural network (MMFNet) model for static handwritten images and dynamic multi-sensor signals proposed in this application. The framework proposed in this application consists of three main steps: data preprocessing, classification, and evaluation. The data preprocessing pipeline includes instance segmentation and contrast enhancement of static handwritten images, signal slicing, filtering, and noise reduction, as well as time-frequency transform spectrograms of dynamic multi-sensor fusion signals. Furthermore, the method of this invention applies several image enhancement algorithms to static handwritten images, including contrast enhancement, flipping, rotation, thresholding, and illumination, to improve the model's generalization ability. Finally, the method of this invention designs a multi-granularity feature fusion algorithm to extract multi-modal information.

[0048] Figure 2 In this study, we first fine-tune (a) the pre-trained DenseNet-121 with the ImageNet-1000 dataset, and then (b) perform a second fine-tuning step to adapt the pre-trained branches to the specific dataset. (c) is the backbone network, used to fuse the static spatial branch to learn global spatial features of electronic handwriting images; (d) is the dynamic temporal branch to learn spectral features in the time-frequency domain. (e) is a coarse-grained EHR feature embedding layer used to integrate prior knowledge.

[0049] Specifically, the method of this invention deploys a transfer learning strategy on the collected dataset to fine-tune pre-trained network branches. For example... Figure 3 As shown, we first fine-tune the pre-trained densely connected network (DenseNet-121) using an easily accessible dataset (1000 different object categories from the ImageNet dataset), and then perform a second fine-tuning step to adapt the pre-trained model to a specific domain (e.g., the ET handwritten dataset that we collected).

[0050] In summary, the MMFNet model proposed in this application consists of three main parts: 1) a multi-branch sub-network structure for learning dynamic time-frequency spectral features; 2) a pre-trained DenseNet-121 model that uses transfer learning to learn high-level static visual features; and 3) a coarse-grained feature embedding layer of electronic medical records (EHRs) to integrate prior knowledge and deep features learned by the sub-networks. These modules can further discover relevant complementary information from different modalities by leveraging squeeze and evoked (SE) attention. Finally, the high-level features fused from the multimodal deep features are input into a classification decision layer composed of linear layers, outputting the predicted probability of the corresponding tremor severity. The overall framework structure is as follows: Figure 3 As shown.

[0051] The main steps of the MMFNet model are described in detail below:

[0052] 1) Kinematic feature extraction

[0053] like Figure 2 As shown in the flowchart, the method of this invention designs a feature extraction and fusion algorithm for multimodal data to improve the quantitative performance of the model. The main features include: 1) time-frequency spectrograms in multiple time-frequency transform domains; 2) pen tip dynamic features; and 3) demographic information from electronic health records (EHRs).

[0054] In clinical diagnosis, tremors observed during handwriting tasks mainly include intention tremor, physiological tremor, and various influencing factors such as noise interference during sensor signal recording. These non-steady-state signals are characterized by finite duration and temporal variation. Transform domain analysis methods can describe the energy distribution and time-varying characteristics of dynamic handwriting in the time-frequency domain, including Short-Time Fourier Transform (STFT), Continuity Spectrum (PSP), Continuous Wavelet Transform (CWT), Wigner-Ville Distribution (WVD), Wavelet Synchronization Transform (WST), Constant Q Gabor Transform (CQT), and Hilbert-Huang Transform (HHT).

[0055] First, to gain a more intuitive understanding of the temporal changes in the patient's handwriting during the writing process, we calculated the statistical characteristics of the electronic handwriting signals from multiple sensor sources and summarized them in Table 2. Specifically, the electronic handwriting D... i (t) contains the raw sensing signals of five channels at time t, namely the position sequence of the x-axis and y-axis. and z-axis pressure signal The angle of the pen tip relative to the horizontal plane and the rotation angle during the movement of the pen tip The dynamic changes in the writing trajectory can be derived from these sensor signals. For example, the direction of the τth sample point in the pen tip direction. It can be calculated as follows:

[0056]

[0057] In addition, the velocity at the τth sampling point in the velocity sequence It can be calculated as

[0058]

[0059] Similarly, acceleration and jerk are the second and third derivatives of velocity, respectively, as shown in equation (3-4):

[0060]

[0061]

[0062] Furthermore, the changes in radius and angle can be calculated using formula (5-6):

[0063]

[0064]

[0065] After obtaining the time-varying information of the pen tip's dynamic characteristics, we can further calculate its time-frequency spectrum. Specifically, the STFT achieves a sliding window representation of the entire signal D. i Joint time-frequency analysis of (t):

[0066]

[0067] Where w * (t) is the conjugate operation of the energy-normalized sliding time window function. PSP overlaps the spectral lines of each frame, then discretizes the spectrum to see which parts overlap more. Another linear method is CWT, which achieves scaling and shifting of the wavelet basis function ψ(t) to optimize the signal D. i Multiscale analysis of (t):

[0068]

[0069] Where a>0 is the scaling factor and t is the translation factor. Linear time-frequency analysis mainly describes the local properties of a signal, such as instantaneous frequency, while nonlinear time-frequency analysis mainly describes the local energy distribution of a signal. For example, WVD can be used to measure the quadratic form of the local time-frequency energy of a signal, which is defined as:

[0070]

[0071] Where s(t) is the analytic signal of the original signal after HHT, which can be calculated by formula (10):

[0072]

[0073] Furthermore, CQT can be calculated as:

[0074]

[0075] Where b is the median of the window function.

[0076] Finally, we extracted demographic information related to ET symptoms from electronic health records (EHRs) as prior knowledge for the deep network, including information such as the patient's age, gender, height, weight, and medical history.

[0077] 2) Multimodal feature fusion

[0078] A significant contribution of this invention is its multimodal analysis of various handwriting tasks. Static handwriting images describe spatial information containing writing task templates, while dynamic multisensor fusion signals record rich temporal variation information. Furthermore, features from electronic medical records (EHRs) can incorporate prior knowledge into automated diagnostic systems. These multimodal features can provide rich complementary information for tremor quantification. However, multimodal features still require effective fusion algorithms to address heterogeneity and dimensionality mismatch. Deep learning will no longer be limited to reliance on existing knowledge vocabularies. It can effectively solve problems related to complex, dynamically evolving multimodal data knowledge extraction, nonlinear fusion, and generalization of reasoning capabilities. We designed multiple branches to acquire high-dimensional information from specific modalities and jointly optimized the fused features through multiple losses.

[0079] like Figure 2 As shown, we learn the temporal-spatial features of electronic handwriting data through two identical structural static spatial branches and dynamic temporal branches. Specifically, for the input static handwritten image S... i and dynamic multi-sensor fusion signal D i Time-frequency transform spectrum Branch output B SD It can be defined as:

[0080]

[0081] Among them, Conv b This represents the receptive field of a CNN layer and a subnetwork of a specific size. and These are the network parameters for the two branches. Concat(·) indicates a join operation.

[0082] Furthermore, inspired by Table 1, the method of this invention considers the template in static handwriting images as an important reference for clinicians to quantify tremor symptoms. For example, subjects with severe tremors frequently intersect their handwriting with the template. Moreover, the template is part of the task, and subjects consciously restrict their trajectory when executing the template. Therefore, the method of this invention designs a cross-domain transfer learning branch, utilizing a model pre-trained on the high-quality ImageNet-1000 dataset to obtain global features of handwritten images with templates. Similar to the previous method, the output feature map of this branch can be obtained by formula (13):

[0083]

[0084] in, and These represent the pre-trained CNN layer and its pre-trained parameters, respectively.

[0085] Finally, the backbone network of the MMFNet model fuses the aforementioned deep features with the EHR features to predict the final tremor severity. The method of this invention uses Softmax loss to train the branch model and calculates the posterior probability for each class:

[0086]

[0087] Wherein, p(B) j The model represents the severity of the tremor, y. i The probability ∈{0,1,...,4} is assigned to the input handwritten data. It is class y i The parameters. Therefore, for all observable instances, the cross-entropy loss function used is defined as:

[0088]

[0089] Where I{·} is an indicator function.

[0090] Then, we define the objective function of the globally fused multimodal features extracted by the skeleton network as:

[0091]

[0092] in This indicates that the branch feature B is connected. SD and B T The fused feature map is obtained. `emb(E)` is optional and represents the embedded EHR data. Therefore, the method of this invention trains the entire MMFNet by jointly optimizing the losses of multiple branches, which can learn potential complementary information across spatiotemporal scales.

[0093] Based on the above description, the final loss function designed by the method of this invention is as follows:

[0094]

[0095] γ1 and γ2 are the balance coefficients corresponding to each branch.

[0096] Table 2. Dynamic kinematic characteristics used in this study.

[0097]

[0098] (5) The workflow diagram proposed by the method of this invention is as follows: Figure 3 As shown. Figure 3The method includes: 1) Data preprocessing. Paired handwritten images and dynamic multi-sensor fusion signals are processed into an input format suitable for the model. 2) Modeling. The MMFNet proposed in this application can optionally fuse prior knowledge and pre-trained model weights. 3) Performance evaluation. The model's classification performance is evaluated on a test set using consensus scores derived by a neurology expert committee as labels for model training. The method of this invention aims to quantify the severity of tremor through multi-task classification tasks. Therefore, various experiments are designed to verify the performance of the proposed MMFNet. Specifically, the method of this invention employs five evaluation metrics, including accuracy (ACC), precision (PRE), sensitivity (SEN), specificity (SPEC), and F1 score, to evaluate the performance of the method. These metrics are defined as follows:

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively. Furthermore, receiver operating characteristic (ROC) curves can typically be plotted using SEN and SPE scores to illustrate the model's performance across all classification thresholds. Additionally, as... Figure 3 As shown, after reserving 10% of the data as an independent test set, stratified five-fold cross-validation was performed on the remaining data to ensure the robustness of the classification results. Therefore, the method of this invention displays the results of five-fold cross-validation in the form of mean ± standard deviation (SD). Furthermore, the method of this invention measures the computational complexity (CC) of the algorithm through the Giga multiply-accumulate (GMac) operator and the parameters of the model.

[0105] (6) To verify the effectiveness of the proposed system, this application conducted extensive experiments: 1) Proof of the effectiveness of multimodal features (proving the effectiveness of cross-domain transfer branches and dynamic time branches respectively); 2) Ablation studies (verifying the utility of attention mechanisms and prior knowledge respectively); 3) Robustness testing (verifying across multiple databases); and 4) Comparison with existing state-of-the-art methods.

[0106] The specific experimental results verifying the method of this invention are as follows:

[0107] (1) Proof of the effectiveness of multimodal features

[0108] 1) Contributions from cross-domain migration branches

[0109] like Figure 3 As shown, the cross-domain transfer branch proposed in this invention comprises two main parts: 1) utilizing pre-trained knowledge from the source domain (ImageNet-1000) and then sharing network structure parameters; 2) fine-tuning the weights of the output layer of the pre-trained model for ET diagnosis of handwriting samples. We analyze the algorithm's performance on each handwriting task and explore the utility of the most commonly used pre-trained architectures, namely Xception, ResNet-50, VGGNet-16, and DenseNet-121.

[0110] Regarding the cross-domain migration branch proposed in this application, such as Figure 3 As shown, following the shared DenseNet-121 architecture, we added a convolutional layer and an SE module to predict the severity of tremors. The performance of these transfer learning networks with five-fold cross-validation is shown in Table 3. Furthermore, Figure 4 Box plots showing these results are presented. Experimental results demonstrate that the MMFNet proposed in this invention achieves state-of-the-art performance across all handwriting tasks. Among the four pre-trained CNN models, ResNet-50 exhibits the worst tremor prediction performance on the LS task, despite having the fewest parameters, achieving an F1 score of 63.62 ± 3.55%. Using ResNet-50 as the baseline model, VGGNet-16 improves the F1 score by 25.39% on this task. However, VGGNet-16 achieves an F1 score of only 78.48 ± 2.30% on the SL task, making it the only model with lower prediction performance than the LS task. Figure 4 Based on the box plot results shown, we recommend using DenseNet-121, which offers more stable predictions, as the backbone structure for cross-domain transfer branches. It exhibits optimal overall performance and moderate computational complexity. Specifically for the SL task, the obtained ACC, SEN, PRE, SPE, and AUC are 85.14±1.05%, 85.95±1.97%, 87.49±1.88%, 95.36±0.28%, 97.17±0.30%, and 86.02±1.82%, respectively. Thanks to the strong fitting ability of DenseNet-121, our proposed method achieves higher evaluation scores after incorporating the attention mechanism and significantly improves the prediction stability for both the SL and SL tasks.

[0111] Table 3. Performance of the transfer learning model on our dataset using five-fold cross-validation.

[0112]

[0113]

[0114] Note: GMac means Giga multiply accumulate; LS refers to large spiral task; SS refers to small spiral task; SL refers to linear task.

[0115] 2) Contribution of dynamic time branches

[0116] Through comparative experiments, we validated the contribution of the time-frequency spectrograms describing dynamic temporal changes. Table 4 shows the experimental results by replacing various time-frequency spectrograms in the dynamic time branch. We highlight the top 1, top 2, and top 3 performances (ranked by F1 score) obtained using a single time-frequency spectrogram of the network using bold, underline, and double underline, respectively. The results show that, except for HHT, almost all time-frequency spectrograms have good predictive performance in specific tasks. In particular, WVD, CWT, and STFT achieved the best F1 scores of 92.07%, 91.88%, and 92.32% in LS, SS, and SL tasks, respectively. For these specific handwriting tasks, we also combined these high-performing time-frequency spectrograms into the model. However, compared to the best results using only a single spectrogram, combining the first two time-frequency spectrograms into the model reduced the F1 score for tremor classification in the three handwriting tasks by 2.83–6.94%. Although the F1 score of the three spectrogram combinations was slightly reduced (by 0.75–5.02%), it was still not the best classification performance. These results indicate that these distinct features do not provide meaningful supplementary information for quantifying tremor models. Therefore, we suggest selecting only a single time-frequency spectrogram for a specific writing task to provide dynamic temporal information.

[0117] Table 4. Performance comparison of using spectrograms as input at different times.

[0118]

[0119]

[0120] Note that bold, underline, and double underline indicate the top 1, top 2, and top 3 performances obtained by using a single time-frequency spectrum as input for the dynamic time branch, respectively.

[0121] (Ranked by F1 score).

[0122] 2) Ablation Research

[0123] The method of this invention evaluates some variants of the proposed MMFNet under different settings through ablation experiments in order to analyze the relative contributions of some modules in the model architecture proposed in this application.

[0124] 1) The utility of attention mechanisms

[0125] The experimental results in Table 3 show that adding the SE module to the cross-domain transfer branch improves the model's performance in predicting the severity of ET symptoms. To further evaluate the effectiveness of the spatial information fusion method using the SE module in the proposed MMFNet, we compared it with a model without the attention module. In this experiment, we removed the SE module before the connection and Adapool operations, keeping other training configurations unchanged. The experimental results are shown in Table 5. We can see that the PRE on the LS task increased by 4.49%, and the F1 score on the attention module increased by approximately 4.61%. These results indicate that the SE module, which fuses multimodal features, helps to mine discriminative features and improves the performance of tremor symptom prediction.

[0126] Furthermore, we chose Grad-CAM to visualize the activation weights of the model's feature maps, providing interpretability of the decision boundaries. Figure 5 As shown, the deeper the red, the more significant the activation value; that is, the highlighted red area is the primary basis for the network to predict the severity of the corresponding tremor. The results demonstrate that the attention mechanism can provide more discriminative weight parameters for template details in handwritten images, thus focusing more on parts highly correlated with tremor and suppressing other useless information. For example, when classifying instances of severity level 3 in the SL task, the model without the attention mechanism only focuses on the top left corner, failing to pay sufficient attention to handwriting that significantly extends beyond the template. In contrast, the model with the SE module can consider both global and local connections, giving more attention to all written parts. Therefore, the accuracy of symptom diagnosis is greatly improved.

[0127] Table 5. Ablation studies of attention fusion methods

[0128]

[0129] 2) The utility of prior knowledge

[0130] Introducing demographic-based prior knowledge can avoid the problem of predicting tremor levels that violate common sense due to insufficient or uneven data distribution, and can also provide additional explanations for the model. As shown in Table 6, we evaluated the performance of models with and without fused prior knowledge. By comparison, we can find that MMFNet with fused prior knowledge performs better in tremor prediction than the model without fused prior knowledge. In particular, for the SL task, PRE increased by approximately 9.63%. Furthermore, Figure 6 A visual comparison of the ROC curves of MMFNet and a model that does not incorporate prior knowledge is shown. Figure 7 The confusion matrix of the MMFNet model is shown. We can see that MMFNet has the best SPE and SEN balance performance across all three writing tasks. This indicates that prior knowledge helps improve the diagnostic accuracy of the model.

[0131] Table 6. Ablation Study of Prior Knowledge Fusion Methods

[0132]

[0133]

[0134] (3) Robustness test

[0135] To verify the effectiveness and generalization ability of the proposed method, we tested it on publicly available handwritten PD datasets. These open-source PD datasets include PaHaW, HandPD, and NewHandPD datasets. To match the MMFNet setup, we selected the Archimedean spiral task on the datasets and switched to a binary classification output layer to diagnose PD patients and HC. Specific data descriptions and experimental results are shown in Table 7. The results show that our proposed method achieves F1 scores between 92.69% and 95.57% on the three datasets, demonstrating high diagnostic accuracy. It is worth noting that since the HandPD dataset only provides scanned images, MMFNet does not incorporate dynamic temporal information, which may lead to a decrease in predictive performance. Compared to the NewHandPD dataset, which provides handwritten images with templates, the model's test results on the PaHaW dataset are slightly lower, which may be a potential advantage of templates. This is because the labels provided by human experts will primarily refer to the amount of overlap between the handwriting and the template. All these results demonstrate the effectiveness of the proposed method in the differential diagnosis of PD.

[0136] Table 7. Test results of the proposed PD diagnostic model on a public handwritten dataset.

[0137]

[0138] (4) Comparison with existing advanced methods

[0139] To our knowledge, no relevant technology or research currently focuses on the symptom quantification of ET tremor. However, many works have explored classifying PD and HC using handwritten data. Therefore, we summarize the state-of-the-art (SOTA) research on this topic in Table 8 to demonstrate the effectiveness of our proposed method on other available datasets. The results show that the proposed MMFNet is the first method to integrate the use of temporal-space dynamic handwriting features and effectively combines temporal spectrograms and demographic features. These test results on three publicly available datasets outperform all SOTA studies. Many studies capture sensor signals with various dynamic characteristics using handwriting acquisition tools. These works extract pen pressure variations, stroke speed, hang time, and some nonlinear features to build machine learning models, such as SVM, with accuracy in identifying PD patients ranging from 61.25% to 95.58%. Clinicians often identify PD patients by visually assessing the degree of tremor in their handwriting. Therefore, some studies have explored converting sensor signals into static images and relying on expert experience to extract visual features from the images, which are then input into machine learning models. Notably, most recent deep learning-based studies have achieved better classification performance due to their strong fitting capabilities. For example, Diaz et al. extracted kinematic features, such as stroke speed, from the raw signals acquired by the pen tip sensor and directly input them into a one-dimensional CNN-RNN cascade model, achieving PD recognition accuracies of 94.75% and 94.44%. Similarly, Li et al. transformed the positional information of the signal into an image without a template. They directly fed this into a CNN model, achieving an accuracy of 85.70% on the HandPD dataset, but failing to converge on the PaHaW dataset. Previous research designs demonstrate that utilizing only a single modality may lead to insufficient progress or model convergence issues. Therefore, our proposed method achieves optimal results in PD recognition by fusing spatial features of handwriting images with pen tip dynamic features describing time-varying information.

[0140] Table 8. Comparison of SOTA works on PD diagnosis on public handwritten datasets

[0141]

[0142]

[0143] Recent work on handwriting processing in deep networks has improved the accuracy and versatility of tremor recognition. Results show that methods fusing temporal-spatial information from multimodal data outperform models using only single-modal data. Instead of simply using static images as classification input, Diaz et al. proposed dynamically augmented images to address this issue. They preserved information about writing time or speed by plotting the pen tip orientation, described by the pen tip sensor signals, as discrete sample points (without connecting them). While this adds some time-related information, visual features still need to be manually extracted as input to the ensemble learning model, resulting in a PD recognition accuracy of only 86.67%.

[0144] Compared to the methods described above, the method of this invention incorporates spatial information from handwritten scan images with templates, which helps deep networks capture global visual features. Simultaneously, MMFNet can fuse time-frequency spectral features, providing detailed features of the variation domain for tremor quantification. Furthermore, a joint optimization strategy incorporating an attention mechanism optimizes various classification losses, while dynamic information fusion promotes the learning of different pattern features through adaptive weight allocation. Due to these novel designs, experimental results demonstrate that our proposed MMFNet can perform fine-grained quantification of tremor severity across various writing tasks and achieves the same state-of-the-art classification performance in PD differential diagnosis.

[0145] Computer-based electronic handwriting analysis holds broad application prospects; for example, handwriting measurement can capture an individual's identity and cognitive characteristics. Patients with movement disorders often experience difficulty writing, so electronic handwriting analysis holds promise for aiding early clinical diagnosis and even automating the quantification of symptom severity. Domain experts can access these easy-to-use, user-friendly tools in their routine checks without requiring any specific computer expertise. In this sense, automated quantification systems based on electronic handwriting represent an attractive option because they not only provide professionals with timely automated responses but also store patient-related metadata to support retrospective research and analysis. The decision-making system proposed in this invention can provide additional objective evidence to support the final clinical assessment, thereby improving the efficiency of disease management.

[0146] Unless otherwise defined, all technical and / or scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention relates. The materials, methods, and embodiments mentioned in this application are illustrative only and not restrictive.

[0147] Although the present invention has been described in conjunction with specific embodiments, those skilled in the art can make appropriate substitutions, modifications and changes within the inventive spirit of this application, and such substitutions, modifications and changes still fall within the protection scope of this application.

Claims

1. A tremor symptom assessment device based on static handwritten images and dynamic multi-sensor fusion signals, comprising: Computational unit; The computing unit runs a tremor symptom assessment model based on static handwritten images and dynamic multi-sensor fusion signals; The tremor symptom assessment model includes: a static spatial branch, a dynamic temporal branch, and a backbone network; The static spatial branch is used to learn the global spatial features of static handwritten images; The dynamic time branch is used to learn the time-frequency domain spectral features of the dynamic multi-sensor fusion signal; the time-frequency domain spectral features of the dynamic multi-sensor fusion signal are obtained from the time-frequency domain spectral features of the pen tip dynamic features. The backbone network is used to fuse global spatial features, time-frequency domain spectral features, and EHR features to assess tremor symptoms; the backbone network is a CNN network that incorporates the SE-Module attention mechanism.

2. The tremor symptom assessment device based on static handwritten images and dynamic multi-sensor fusion signals according to claim 1, characterized in that: The static spatial branch is a fine-tuned DenseNet-121 network; the fine-tuned DenseNet-121 network retains the pre-trained weights other than the last two layers of the DenseNet-121 network. Images of large spirals, small spirals, or straight lines in static handwritten images are input into the static space branch.

3. The tremor symptom assessment device based on static handwritten images and dynamic multi-sensor fusion signals according to claim 2, characterized in that: The dynamic time branch includes the second Conv 7-32 layers; the time-frequency domain spectral features of the dynamic multi-sensor fusion signal transformation, which consists of a large spiral, a small spiral, or a straight line, are input into the second Conv 7-32 layers of the dynamic time branch.

4. The tremor symptom assessment device based on static handwritten images and dynamic multi-sensor fusion signals according to claim 2, characterized in that: The backbone network consists of Conv layers 7-32 (input layers), a first concatenation layer fused with dynamic temporal branch features, a first Max pool 3 layer, Conv layers 6-64, a second Max pool 3 layer, Conv layers 5-128, a first Conv layer 3-128, a second Conv layer 3-128, a third Max pool 3 layer, Conv layers 3-256, a first SE-Module attention mechanism layer, a second concatenation layer fused with static spatial branches, an Adapt pool 3 layer, a Flatten layer, a first FC layer, a third concatenation layer fused with EHR features, a second FC layer, and an Output layer. Images of large spirals, small spirals, or straight lines in static handwritten images are simultaneously input into Conv layers 7-32 of the backbone network.

5. The tremor symptom assessment device based on static handwritten images and dynamic multi-sensor fusion signals according to claim 4, characterized in that: The output of the static spatial branch is spliced ​​in the second splicing layer of the backbone network after passing through the Conv 3-512 layer and the second SE-Module attention mechanism layer; The output of the dynamic time branch is spliced ​​in the first splicing layer of the backbone network; EHR features are spliced ​​in the third splicing layer of the backbone network; The output layer of the backbone network outputs the tremor symptom assessment results.

Images