A hand-drawn spiral classification method based on Transformer multimodal algorithm

Abstract

The present invention discloses a method for classifying hand-drawn spirals based on a Transformer multimodal algorithm. The method uses a paper drawing board and a signature pen as data acquisition devices. After obtaining the hand-drawn spiral coils of the subject, the data is transferred to a computer for processing. The collected spiral images are converted into binary images of uniform size to remove noise. The tremor features and turn spacing features are extracted through a multimodal spiral classification Transformer network. After encoding different features of the same image, the data of each modality is fused through a layered fusion strategy of the multimodal spiral classification Transformer network. The fusion is further optimized through a cross-modal attention mechanism. The evaluation and grading of the spirals drawn by the subject are output through a decoder and an MLP head. The present invention realizes the extraction of multiple modal feature data by extracting the spacing features and tremor feature algorithms of the hand-drawn spiral coils. Then, by fusing the multimodal data, the accurate rating of hand-drawn spiral coils with different tremor levels is finally achieved.

Description

Technical Field

[0001] The present invention belongs to the field of computer science and artificial intelligence technology, and in particular relates to a hand-drawn spiral classification method based on a Transformer multimodal algorithm. Background Art

[0002] First, with the rapid development of artificial intelligence, deep learning has experienced explosive growth across various fields. The two main application areas of deep learning are computer vision (CV) and natural language processing (NLP). In CV, the application and refinement of convolutional neural networks (CNNs) are the cornerstone of numerous outstanding algorithms. This is primarily due to the convolutional operation's ability to effectively capture complex features in images. However, convolution also has inherent limitations. For example, its output only considers the influence of the current input, without factoring in the influence of inputs at other times. Therefore, it is inadequate for certain sequential signals. This problem is particularly acute in natural language processing (NLP), where input is often in the form of sequences, such as sentences or articles. In such signals, the relative position of words or clauses is crucial; swapping words can lead to completely different meanings. Furthermore, long-range features within a sentence are crucial for understanding semantics. To address these issues, recurrent neural networks (RNNs) have been introduced into the NLP field. Furthermore, building upon the original RNN model, the LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) alleviated the vanishing gradient problem in the temporal dimension by adding intermediate state information, achieving excellent results and becoming the standard RNN model. In RNN models, all network parameters are shared at every time step. This structural characteristic of the model is a double-edged sword, contributing to its superior performance in the field of natural language processing (NLP). However, it also severely hindered its development. Because RNN computations at each time step must be based on the computations completed at the previous step, this sequential dependency makes RNNs difficult to accelerate through large-scale parallel computing. Despite its efficiency issues, RNNs remain the most effective model for NLP, and they dominated the state-of-the-art results in various NLP subfields until the emergence of the Transformer model in 2017.

[0003] Secondly, multimodal technology has become a hot topic in the field of artificial intelligence in recent years, particularly following the significant advances in deep learning. This section aims to provide a more comprehensive background on multimodal technology, including its development history, key challenges, technical approaches, and practical applications in various fields. The development of multimodal technology can be traced back to early explorations in fields such as computer vision, natural language processing, and speech recognition, where researchers sought to enable computers to understand and process diverse forms of human information. In the early 21st century, with the rise of machine learning, particularly deep learning, multimodal learning began to develop rapidly. Deep learning provides the ability to effectively process large datasets and learn complex data representations, providing the impetus for the advancement of multimodal technology. Over the past decade, with the availability of large datasets and significant increases in computing power, research in multimodal learning has made significant progress. In particular, the introduction of the Transformer model provides a powerful framework for processing data from different modalities, facilitating the development of cross-modal representation learning and modality fusion methods.

[0004] The huge success of the Transformer in the field of Natural Language Processing (NLP) has quickly attracted the attention of scholars in the field of Computer Vision (CV). The first model to apply the Transformer model to visual tasks was the Vit image classification model proposed in 2020.

[0005] With the widespread application of Transformers in deep learning, researchers have proposed numerous Transformer-based multimodal models. Data from different modalities has distinct characteristics and distributions, and designing effective methods to fuse these heterogeneous data is a key challenge. Processing data from different modalities through joint learning aims to discover shared representations across modalities.

[0006] Existing multimodal Transformer technologies demonstrate deep learning's advanced capabilities in processing and integrating diverse data types, such as text, images, and audio. These technologies are typically designed to understand and generate content containing information from multiple modalities, supporting applications such as cross-modal retrieval, automatic description generation, and improved human-computer interaction experiences. ViLBERT (Visual and Language BERT) is a multimodal model designed for vision and language tasks. It independently processes image and text inputs through a two-stream architecture and then learns cross-modal representations through shared Transformer layers. ViLBERT provides an effective framework for modeling the relationship between vision and language and has been widely used in tasks such as image captioning and visual question answering. LXMERT is specifically designed for visual question answering and visual reasoning tasks. By jointly pre-training image and text representations on large-scale datasets, it can understand complex visual content and natural language queries. The LXMERT model consists of three Transformer encoders for image, question, and cross-modality, enabling efficient multimodal information processing. While CLIP primarily uses large-scale contrastive learning to jointly train image and text representations, it employs a simple yet powerful approach to understanding the relationship between image and text. By performing contrastive learning across modalities, CLIP can handle a wide range of visual tasks with little or no task-specific data adaptation. VisualBERT is a method that applies BERT to multimodal data by embedding image features into text sequences and processing the fused information with a single Transformer model. This approach demonstrates how to apply pre-trained language models to multimodal tasks such as visual question answering and image description through fine-tuning. These multimodal Transformer technology solutions each have their own focus and demonstrate the ability to integrate and process multimodal data in different application scenarios. As research continues to advance, we can expect that more innovative multimodal processing methods will be proposed to tackle more complex and challenging AI tasks.

[0007] However, current spiral classification methods based on multimodal fusion have significant limitations. There are two mainstream approaches: obtaining multimodal information from the artist (pen tip pressure, drawing speed, hand-drawn spirals, etc.) through auxiliary devices, and collecting multimodal information by drawing spirals on templates. Obtaining multimodal information from the artist through auxiliary devices often requires expensive equipment, such as tablets, to capture details such as drawing speed and pen tip pressure. This not only increases reliance on professional operators but also significantly increases implementation costs. Furthermore, current hand-drawn note collection schemes often require drawing according to specific auxiliary templates. This fixed model not only limits the artist's freedom but also makes it difficult to accurately capture the artist's actual hand tremor. More importantly, although spiral tremor and turn spacing information are crucial for classification, there is currently a lack of a multimodal network specifically designed for spiral feature extraction, and most studies have failed to fully address the simultaneous importance of these two features. For example, general-purpose Transformer networks, such as ViT, while able to focus on the tremor characteristics of spirals based on the input image, often overlook the spacing characteristics between the turns of the spiral. Summary of the Invention

[0008] In view of this, the present invention proposes a hand-drawn spiral classification method based on the Transformer multimodal algorithm, comprising the following steps:

[0009] S10, using a paper drawing board and a signature pen as data collection equipment, after obtaining the subject's hand-drawn spiral coil, it is transferred to the computer for processing;

[0010] S20, converting the collected spiral image into a binary image of uniform size and removing noise;

[0011] S30, extracting chatter and turn spacing features through a multimodal spiral classification Transformer network;

[0012] S40, after encoding different features of the same image, the data of each modality are fused through the layered fusion strategy of the multimodal spiral classification Transformer network;

[0013] S50, further optimizes the fusion through the cross-modal attention mechanism;

[0014] S60, output the evaluation grade of the spiral drawn by the subject through the decoder and MLP head.

[0015] Preferably, the S30 includes:

[0016] S31, saving the image tremor feature as a tremor feature matrix through a tremor extraction encoder;

[0017] S32, converting the turn spacing features of the image into a time series with significant differences through a turn spacing feature extraction encoder.

[0018] Preferably, the S40 includes converting the input image into two modalities through a cross-modal Transformer encoder, which can process feature vectors from different modalities; the cross-modal encoder adopts a cross-modal attention mechanism based on the Transformer architecture, so that the model can consider the information of tremor and turn spacing at the same time when calculating self-attention; primary fusion is achieved at the low level of the encoder, and the turn spacing and image tremor features are directly mixed by weighted splicing at the feature level, and a cross-modal attention mechanism is adopted in the middle layer of the encoder so that each modality can adjust its representation according to the contextual information of the other modality, and more complex fusion operations are performed at the high level of the encoder, including in-depth learning of the relationship between modalities and a dynamic weight adjustment mechanism to ensure that the model fully considers all relevant information before finally outputting the rating result.

[0019] Preferably, the S50 includes a Transformer-based cross-modal enhanced attention mechanism for processing and integrating the binary image modality of tremor information and the time series modality of turn spacing information; this mechanism is based on the self-attention and position encoding functions of the Transformer, and learns the fine-grained features of the binary image and time series respectively through modality-specific encoder branches; in the feature learning process, a cross-modal interaction module is added, which includes a multi-head attention mechanism to capture and amplify the associations in the binary image and time series data. This module dynamically adjusts the contribution of each modality in the feature fusion process by calculating the mutual attention scores between different modalities.

[0020] Preferably, the S31 includes the following steps:

[0021] S311, processing the hand-drawn spiral image by binarization;

[0022] S312: The processed image is divided into several small patches through the embedding layer. Each patch contains a portion of the spiral image and carries tremor information. The weight corresponding to the area with dense tremor is greater than the area with smooth or no spiral lines, so that more attention is paid to these key areas during the model training process.

[0023] At step S313, each image patch is converted into a one-dimensional vector. To maintain the spatial order of sequence elements, position encoding is introduced, embedding the weight matrix corresponding to each image patch into the corresponding one-dimensional vector. By equipping each image patch with a position encoding, the model can parse the specific location of each part in the entire spiral, thereby identifying tremor characteristics.

[0024] At step S314, the image blocks are combined with the position encoding and fed into the Transformer encoder layer, which includes a self-attention mechanism and a feedforward neural network, so that the model can learn the relationship between blocks in different regions of the image.

[0025] Preferably, the S32 includes the following steps:

[0026] S321, using image processing algorithms, including edge detection and morphological operations, to pre-process the input hand-drawn spiral image to highlight the spiral and remove noise;

[0027] S322, using an image skeletonization method to determine the coordinates of the center point of the helix and perform turn spacing measurement;

[0028] S323: For a hand-drawn spiral, rays are emitted from the center of the spiral as the origin to the surrounding 12 angles of 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300°, 330°, and 360°. The rays intersect the spiral at each layer, and the length of the line segment between each ray and the focus of each spiral layer is saved. The turn spacing information at different angles is stored as a time series.

[0029] S324, using a turn spacing feature extraction encoder to encode the spatial distribution features of the turn spacing, and reducing the dimension through a pooling layer to extract features.

[0030] Beneficial Effects: This invention addresses the current challenges of multimodal Transformer fusion models by proposing a method for extracting information from different modalities in an image and performing multimodal fusion. This method can be used not only in the present invention's fusion of the tremor and turn spacing features of a helical wire, but also in pioneering research in multimodal data fusion. Furthermore, the proposed feature extraction algorithm can also be applied to scenarios such as handwriting recognition and human motion recognition.

[0031] This invention provides a systematic solution for researchers and professionals in similar fields through its unique cross-modal hierarchical fusion Transformer model, refined tremor feature extraction algorithm, and innovative turn spacing measurement method. The combination of these technologies not only optimizes the processing flow of multimodal data, but also improves the accuracy and efficiency of hand-drawn spiral classification. By disclosing the technical details and methodology of this invention, we aim to promote knowledge sharing in this field, encourage peers to conduct in-depth research and development, and further promote the advancement of related science and technology. The disclosure of this invention can provide like-minded researchers with the necessary technical foundation and inspiration so that they can innovate and expand on this basis, thereby realizing broader applications in multidisciplinary fields such as bioinformatics and artificial intelligence.

[0032] Also include at least:

[0033] (1) In response to the complexity of multimodal data processing, a cross-modal hierarchical fusion Transformer model combining time series and image modalities is proposed. The model adopts an innovative cross-modal attention mechanism to achieve weighted splicing of feature vectors at different neural network levels, thereby effectively fusing tremor and turn spacing information. The present invention can perform fusion operations of different complexities at lower, intermediate, and higher network levels, especially a mechanism for dynamic weight adjustment of cross-modal data;

[0034] (2) In response to the limitations of tremor feature analysis, the present invention proposes an advanced tremor feature extraction encoder suitable for processing images with severe tremor or inter-layer crossing of spiral lines. The algorithm not only extracts the severity and density information of the tremor, but also converts this information into a weighted matrix to emphasize the importance of key areas in model training. The present invention proposes a tremor feature extraction method that combines position encoding and image block processing, and the application of the weighted matrix generated by this method in the model;

[0035] (3) This paper designs an algorithm for accurately measuring the turn spacing of a hand-drawn spiral. The center point coordinates are determined through preprocessing and skeletonization, and then multi-angle ray measurement is used to capture information on the different angular spacings. The present invention uses a multi-angle ray measurement method based on the center of the spiral as the reference, as well as an encoding and pooling strategy for the measured turn spacing time series data.

[0036] (4) This paper proposes a Transformer-based cross-modal enhanced attention mechanism for processing and integrating the tremor and turn spacing information of binary image modalities and time series modalities. Based on multi-head attention, this mechanism accurately captures and amplifies the subtle correlations in binary image and time series data. This paper combines self-attention and position encoding functions to implement an attention mechanism for fine-grained feature learning and cross-modal interaction, especially for the application of this mechanism in feature fusion and model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flowchart of the steps of a hand-drawn spiral classification method based on the Transformer multimodal algorithm according to an embodiment of the present invention;

[0038] Figure 2 Schematic diagram of a multimodal spiral classification Transformer network of a hand-drawn spiral classification method based on a Transformer multimodal algorithm according to an embodiment of the present invention;

[0039] Figure 3 This is a flowchart of the tremor feature extraction of the hand-drawn spiral classification method based on the Transformer multimodal algorithm according to an embodiment of the present invention;

[0040] Figure 4 This is a flow chart of spiral turn spacing feature extraction in a hand-drawn spiral classification method based on the Transformer multimodal algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] On the contrary, the present invention covers any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention as defined by the claims. Furthermore, to facilitate a better understanding of the present invention, certain specific details are described in detail below in the detailed description of the present invention. Those skilled in the art will be able to fully understand the present invention without these details.

[0043] See also Figure 1 To achieve the above purpose, the technical solution of the present invention is as follows:

[0044] A hand-drawn spiral classification method based on the Transformer multimodal algorithm includes the following steps:

[0045] S10, using a paper drawing board and a signature pen as data collection equipment, after obtaining the subject's hand-drawn spiral coil, it is transferred to the computer for processing;

[0046] S20, converting the collected spiral image into a binary image of uniform size and removing noise;

[0047] S30, extracting chatter and turn spacing features through a multimodal spiral classification Transformer network;

[0048] S40, after encoding different features of the same image, the data of each modality are fused through the layered fusion strategy of the multimodal spiral classification Transformer network;

[0049] S50, further optimizes the fusion through the cross-modal attention mechanism;

[0050] S60: Output the evaluation grade of the spiral line drawn by the subject through the decoder and MLP head (multi-layer classification perceptron).

[0051] The S30 includes:

[0052] S31, saving the image tremor feature as a tremor feature matrix through a tremor extraction encoder;

[0053] S32, converting the turn spacing features of the image into a time series with significant differences through a turn spacing feature extraction encoder.

[0054] S40 proposed a Transformer model for cross-modal layered fusion. The main principle is shown in Figure 2 As shown. For the data after the input image is converted into two modalities, the present invention designs a cross-modal Transformer encoder, which can process feature vectors from different modalities (time series and images). Based on the Transformer architecture, the cross-modal encoder adopts a cross-modal attention mechanism so that the model can consider the information of tremor and turn spacing at the same time when calculating self-attention. Primary fusion is achieved at the lower level of the encoder, and the turn spacing and image tremor features are directly mixed by weighted splicing at the feature level. The cross-modal attention mechanism is adopted in the middle layer of the encoder, so that each modality can adjust its representation according to the contextual information of the other modality. More complex fusion operations are performed at the high level of the encoder, including in-depth learning of the relationship between modalities and a dynamic weight adjustment mechanism to ensure that the model fully considers all relevant information before finally outputting the rating results.

[0055] S50 proposes a Transformer-based cross-modal enhanced attention mechanism specifically designed to process and integrate tremor information (binary image modality) and turn spacing information (time series modality). Leveraging the Transformer's powerful self-attention and positional encoding capabilities, this mechanism learns fine-grained features for both binary images and time series separately through modality-specific encoder branches. During feature learning, an innovative cross-modal interaction module, comprising a multi-head attention mechanism, is introduced to capture and amplify subtle correlations between binary images and time series data. This module dynamically adjusts the contribution of each modality in the feature fusion process by calculating mutual attention scores between different modalities.

[0056] Ultimately, this enhanced attention layer significantly improves the accuracy and efficiency of fusing binary images with time series data, enhancing the classification of different tremor levels in hand-drawn spirals. By understanding and fusing information from both modalities at different levels, our mechanism not only fully exploits the potential of the Transformer architecture but also sets a new standard for cross-modal feature extraction. This approach's precise modality fusion is crucial to the accuracy of the results.

[0057] See also Figure 3 , the input image is divided into 9 small blocks of the same size, each of which has a corresponding position encoding information recorded as x1-x9 and a corresponding tremor weight matrix recorded as w1-w9. S31 includes the following steps:

[0058] S311, processing the hand-drawn spiral image by binarization;

[0059] S312: The processed image is divided into several small patches through the embedding layer. Each patch contains a portion of the spiral image and carries tremor information. The weight corresponding to the area with dense tremor is greater than the area with smooth or no spiral lines, so that more attention is paid to these key areas during the model training process.

[0060] At step S313, each image patch is converted into a one-dimensional vector. To maintain the spatial order of sequence elements, position encoding is introduced, embedding the weight matrix corresponding to each image patch into the corresponding one-dimensional vector. By equipping each image patch with a position encoding, the model can parse the specific location of each part in the entire spiral, thereby identifying tremor characteristics.

[0061] At step S314, the image blocks are combined with the position encoding and fed into the Transformer encoder layer, which includes a self-attention mechanism and a feedforward neural network, so that the model can learn the relationship between blocks in different regions of the image.

[0062] One of the key technical aspects of this invention is the design of a multimodal spiral classification Transformer algorithm for extracting tremor features. Traditional tremor feature analysis methods are often limited to identifying specific regions or simple patterns, which often proves ineffective when processing images with complex or severe tremors. For example, while traditional methods that expand Archimedean spirals using polar coordinates can reveal the tremor characteristics of the spiral, their effectiveness is significantly reduced when faced with severe tremors or when the spirals intersect.

[0063] To overcome the limitations of the prior art, the present invention proposes an advanced encoder for tremor feature extraction. First, by binarizing the hand-drawn spiral image, the method not only highlights the tremor features but also preserves the overall information of the image. Through this preprocessing step, the image is significantly enhanced to highlight its tremor properties while maintaining its global visual structure. Subsequently, the processed image is divided into multiple small patches (36 patches in this embodiment) through an embedding layer, each of which contains a portion of the spiral image and carries specific tremor information. This segmentation technique enables the network to effectively identify tremor features at global and local locations. The present invention uses a sophisticated algorithm to extract the severity and density of the tremor in each image patch and converts this information into a weight matrix with significant differences. This ensures that the weights corresponding to areas with dense tremors far exceed those of areas with relatively flat or no spirals, thereby giving more attention to these key areas during model training.

[0064] Each image patch is converted into a one-dimensional vector, and a positional encoding is introduced to maintain the spatial order of sequence elements. Positional encoding plays a crucial role in the network, ensuring the coherence of spatial relationships between image patches. More crucially, the weight matrix corresponding to each image patch is embedded into the corresponding one-dimensional vector. By equipping each image patch with a positional encoding, the model can precisely determine the specific location of each component within the entire spiral, enabling more accurate identification of tremor characteristics.

[0065] This encoding method, which combines image patches, positional information, and a tremor signature matrix, provides a clear path to easily capture tremor characteristics across the entire image. Within this framework, regions with severe tremor receive higher weights and extra attention during model training, thereby optimizing the identification and analysis of tremor features. Image patches, combined with the positional encoding, are fed into a series of Transformer encoder layers. These layers incorporate a self-attention mechanism and a feedforward neural network, enabling the model to learn the complex relationships between patches in different regions of the image.

[0066] See also Figure 4 , S32 includes the following steps:

[0067] S321, using image processing algorithms, including edge detection and morphological operations, to pre-process the input hand-drawn spiral image to highlight the spiral and remove noise;

[0068] S322, using an image skeletonization method to determine the coordinates of the center point of the helix and perform turn spacing measurement;

[0069] S323: For a hand-drawn spiral, rays are emitted from the center of the spiral as the origin to the surrounding 12 angles of 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300°, 330°, and 360°. The rays intersect the spiral at each layer, and the length of the line segment between each ray and the focus of each spiral layer is saved. The turn spacing information at different angles is stored as a time series.

[0070] S324, using a turn spacing feature extraction encoder to encode the spatial distribution features of the turn spacing, and reducing the dimension through a pooling layer to extract features.

[0071] This paper presents a multimodal spiral classification Transformer algorithm for extracting turn spacing features. To accurately capture and analyze the turn spacing variations in hand-drawn spirals, the method first pre-processes the input hand-drawn spiral image using image processing algorithms such as edge detection and morphological operations to highlight the spiral and remove noise. Subsequently, image skeletonization is used to determine the coordinates of the spiral center point for accurate turn spacing measurement. For hand-drawn spirals, the spacing at different angles contains important information. Therefore, the present invention uses the center position of the spiral as the origin and emits rays to 12 angles of 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300°, 330°, and 360° around it. The rays will intersect with each layer of spirals, and the length of the line segment between each ray and the focus of each layer of spirals is saved, and the turn spacing information at different angles is stored as a time series; then the turn spacing feature extraction encoder proposed in the present invention is used to encode the spatial distribution characteristics of these turn spacings, and the dimension is reduced through the pooling layer to extract the most critical features.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A hand-drawn spiral classification method based on Transformer multimodal algorithm, characterized by: The following steps are involved: S10, using a paper drawing board and a signature pen as data collection equipment, after obtaining the subject's hand-drawn spiral coil, it is transferred to the computer for processing; S20, converting the collected spiral image into a binary image of uniform size and removing noise; S30, extracting chatter and turn spacing features through a multimodal spiral classification Transformer network; S40, after encoding different features of the same image, the data of each modality are fused through the layered fusion strategy of the multimodal spiral classification Transformer network; S50, further optimizes the fusion through the cross-modal attention mechanism; S60, output the evaluation grade of the spiral drawn by the subject through the decoder and MLP head; The S30 includes: S31, saving the image tremor feature as a tremor feature matrix through a tremor extraction encoder; S32, converting the turn spacing features of the image into a time series with significant differences through a turn spacing feature extraction encoder; The S40 includes converting the input image into two modalities and passing it through a cross-modal Transformer encoder capable of processing feature vectors from different modalities. The cross-modal encoder, based on the Transformer architecture, adopts a cross-modal attention mechanism so that the model can simultaneously consider tremor and turn spacing information when calculating self-attention. Primary fusion is achieved at the low level of the encoder, directly mixing turn spacing and image tremor features through weighted splicing at the feature level. A cross-modal attention mechanism is adopted in the middle layer of the encoder, so that each modality can adjust its representation based on the contextual information of the other modality. More complex fusion operations are performed at the high level of the encoder, including in-depth learning of the relationship between modalities and a dynamic weight adjustment mechanism to ensure that the model fully considers all relevant information before finally outputting the rating result. The S31 includes the following steps: S311, processing the hand-drawn spiral image by binarization; S312: The processed image is divided into several small patches through the embedding layer. Each patch contains a portion of the spiral image and carries tremor information. The weight corresponding to the area with dense tremor is greater than the area with smooth or no spiral lines, so that more attention is paid to these key areas during the model training process. At step S313, each image patch is converted into a one-dimensional vector. To maintain the spatial order of sequence elements, position encoding is introduced, embedding the weight matrix corresponding to each image patch into the corresponding one-dimensional vector. By equipping each image patch with a position encoding, the model can parse the specific location of each part in the entire spiral, thereby identifying tremor characteristics. At step S314, the image blocks are combined with the position encoding and fed into the Transformer encoder layer, which includes a self-attention mechanism and a feedforward neural network, so that the model can learn the relationship between blocks in different regions of the image.

2. The hand-drawn spiral classification method based on the Transformer multimodal algorithm according to claim 1 is characterized in that: The S50 includes a Transformer-based cross-modal enhanced attention mechanism for processing and integrating the binary image modality of tremor information and the time series modality of turn spacing information. This mechanism is based on the self-attention and position encoding functions of the Transformer and learns fine-grained features of the binary image and time series respectively through modality-specific encoder branches. During the feature learning process, a cross-modal interaction module is added, which includes a multi-head attention mechanism to capture and amplify the associations between binary images and time series data. This module dynamically adjusts the contribution of each modality in the feature fusion process by calculating the mutual attention scores between different modalities.

3. The hand-drawn spiral classification method based on the Transformer multimodal algorithm according to claim 1 is characterized in that: The S32 includes the following steps: S321, using image processing algorithms, including edge detection and morphological operations, to pre-process the input hand-drawn spiral image to highlight the spiral and remove noise; S322, using an image skeletonization method to determine the coordinates of the center point of the helix and perform turn spacing measurement; S323: For a hand-drawn spiral, rays are emitted from the center of the spiral as the origin to 12 surrounding angles of 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300°, 330°, and 360°. The rays intersect the spiral at each layer, and the length of the line segment between each ray and the focus of each spiral layer is saved. The turn spacing information at different angles is stored as a time series. S324, using a turn spacing feature extraction encoder to encode the spatial distribution features of the turn spacing, and reducing the dimension through a pooling layer to extract features.

Images