Multi-modal neural network model based on gated multi-head attention and application thereof

Abstract

The invention relates to a multi-modal neural network model based on gated multi-head attention and application thereof, and relates to the technical field of bioinformatics. The multi-modal neural network model comprises a preprocessing module, a gating fusion module and a data output module; the preprocessing module is used for preprocessing the sample data and inputting feature data obtained after preprocessing into the gating fusion module; the gating fusion module is used for extracting and integrating the feature data to obtain fused feature data and inputting the fused feature data into the data output module; the preprocessing module comprises an enhancement block and a feature extraction block; the gating fusion module comprises a convolutional layer, a pooling layer, a gating block and a combined remolding block; and the data output module comprises a multi-head attention block, a linear transformation layer and a classification layer. According to the model, multi-modal data can be integrated, the classification of the sample data has relatively high accuracy, and relatively accurate risk assessment can be carried out on various respiratory system diseases by utilizing the model.

Description

Technical Field

[0001] The present invention relates to the technical field of bioinformatics, and in particular to a multimodal neural network model based on gated multi-head attention and its application. Background Art

[0002] Respiratory diseases, such as asthma, chronic obstructive pulmonary disease (COPD), pneumonia, and now widespread COVID-19, pose a major threat to human health with high morbidity and mortality. Compared with cardiovascular diseases and cancer, respiratory diseases may not receive the same level of attention in the allocation of medical resources and research funding. Because these diseases require expensive treatments and extensive preventive screening, healthcare systems and governments often prioritize resources to other areas. However, a consensus statement published in Respiratory Research emphasizes the need for greater attention and resource allocation to respiratory diseases, especially COPD. The study highlights the need for improved diagnostic methods, increased investment in pulmonary rehabilitation and palliative care, and the use of digital technologies to improve patient self-management and treatment outcomes. The Centers for Disease Control and Prevention (CDC) also emphasized the significant impact of asthma on patients' quality of life and called for more effective diagnostic and treatment methods.

[0003] Accurate diagnostic methods are critical to the treatment of respiratory diseases, enabling appropriate and timely interventions and improving patient outcomes. However, traditional diagnostic methods, such as chest X-rays, CT scans, and RT-PCR tests, while effective in some cases, have limitations including high cost, long processing times, and reliance on specialized equipment and personnel. In particular, RT-PCR tests, the primary diagnostic method for COVID-19, while highly accurate, can take several hours to process and are more costly in resource-limited settings. Therefore, there is an urgent need to increase worldwide attention to respiratory diseases and provide more support and resources for their research and treatment. This includes improving existing diagnostic technologies to reduce costs and increase efficiency, as well as developing new diagnostic methods to address growing medical needs and challenges.

[0004] In this context, artificial intelligence (AI) has made important contributions in the field of medicine, especially in diagnosing respiratory diseases such as COVID-19. Since the outbreak of COVID-19, AI-based models have been widely developed and used to process various data types to detect and diagnose the virus. Many AI models have demonstrated the ability to quickly and accurately diagnose COVID-19, overcoming some of the limitations of traditional methods. For example, AI technology has been used to analyze chest X-rays and CT scans, but these methods require well-equipped diagnostic centers and involve time-consuming processes with high risks of infection.

[0005] To mitigate these risks and speed up the diagnostic process, researchers have developed AI-based methods to analyze non-invasive data such as cough, breath sounds, and voice recordings. Studies have shown that analyzing breath sounds using machine learning (ML) or deep learning (DL) techniques can provide key insights, reduce the risk of patient exposure and virus transmission, and provide faster and more accessible diagnostic tools. For example, Pahar et al. used the Coswara dataset to verify the effectiveness of machine learning in COVID-19 cough detection. Aly et al. successfully distinguished COVID-19 from other audio samples by processing breath sounds and voice data with a deep learning model. Although mainstream research has focused on binary classification tasks for specific diseases (such as COVID-19 detection), audio analysis methods are also applicable to identifying other respiratory diseases. For example, Dash et al. used a gradient boosting-based machine learning method that combined multiple features (e.g., periodicity, spectrum, cepstrum, and spectral descriptors) to classify COVID-19 and asthma. Current methods focus mainly on unimodal data and lack the integration of multimodal information, which hinders the accurate diagnosis of complex diseases. However, the integration of multimodal data can help improve classification performance. Therefore, there is an urgent need for a model that can cope with multimodal data and can be used for the diagnosis of respiratory diseases. Summary of the invention

[0006] In response to the above problems, the present invention provides a multimodal neural network model based on gated multi-head attention, which can integrate multimodal data and has high accuracy in classifying sample data. The model can be used to perform more accurate risk assessment of various respiratory diseases.

[0007] In order to achieve the above-mentioned object, the present invention provides a multimodal neural network model based on gated multi-head attention, comprising a preprocessing module, a gated fusion module and a data output module; the preprocessing module is used to preprocess the sample data, and input the feature data obtained after the preprocessing into the gated fusion module; the gated fusion module is used to extract and integrate the feature data to obtain fused feature data, and input the fused feature data into the data output module;

[0008] The preprocessing module includes an enhancement block and a feature extraction block; the gated fusion module includes a convolution layer, a pooling layer, a gated block and a combined reshaping block; the data output module includes a multi-head attention block, a linear transformation layer and a classification layer.

[0009] Audio features have shown great potential in diagnosing respiratory diseases, especially in the context of COVID-19. Pahar et al. demonstrated the effectiveness of machine learning models in detecting COVID-19 coughs using the Coswara dataset, showing the utility of audio features in distinguishing healthy from infected individuals. Nayak et al. proposed a system for detecting COVID-19 from speech signals using machine learning classifiers such as random forests and K-nearest neighbors (KNN). The KNN method achieved an AUC score of 0.80 on the Coswara dataset. Orlandic et al. further improved the consistency on the COUGHVID crowdsourced dataset using a semi-supervised learning algorithm, and the resulting COVID-19 classifier achieved an AUC of 79.7%. Dash et al. further explored this area by using a gradient boosting-based machine learning method to classify cough audio features, highlighting the power of unimodal methods in respiratory disease classification. Zealouk et al. analyzed the cough sounds of COVID-19 patients in combination with MFCC features, identified significant differences in certain resonance peaks, and achieved an overall accuracy of 86.66% in COVID-19 cough analysis. Ulukaya et al. introduced MSCCov19Net, a multi-branch deep learning model that achieved an accuracy of 0.748 on unseen clinical and non-clinical datasets. In addition, Celik's CovidCoughNet method achieved high performance in predicting and detecting COVID-19 from audio signals, with an AUC of 98.44%. However, all of the above studies focused on the effectiveness of single-modal audio features in identifying and diagnosing respiratory diseases.

[0010] Various data modalities, such as coughs, breath sounds, and speech sounds, are valuable in the diagnosis of respiratory diseases, and the integration of multiple data modalities can enhance the robustness and interpretability of the diagnostic model, and previous studies have failed to fully explore the potential of multimodal data in multi-class classification of respiratory diseases. Therefore, the inventors propose the above-mentioned multimodal neural network model based on gated multi-head attention, and solve the above-mentioned technical problems by introducing a new deep learning model GAMM-Net. The above-mentioned multimodal neural network model (GAMM-Net model) effectively integrates multimodal data, and by introducing a gating mechanism, adaptive control of the influence of each modality on the classification results is achieved, significantly improving the accuracy of the model; at the same time, a multi-head attention mechanism and a linear transformation module are introduced to further improve the effectiveness of the model in multimodal data classification tasks, and by combining the gating mechanism, multi-head attention and linear transformation, the influence of each modality is dynamically adjusted, thereby improving the classification performance.

[0011] In one embodiment, the sample data is multimodal data.

[0012] In one of the embodiments, the enhancement block processes the sample data by data enhancement to obtain enhanced data;

[0013] The feature extraction block includes a transformation block, a filter group, a logarithmic conversion block, and a feature generation block. The transformation block is used to convert the enhanced data from the time domain to the frequency domain. The converted enhanced data is passed to the logarithmic conversion block through the filter group for logarithmic conversion to obtain logarithmic data. The feature generation block is used to process the logarithmic data to generate feature data.

[0014] In one embodiment, the data enhancement includes: at least one of time shift, time stretch, pitch adjustment, volume adjustment, noise addition, and Mixup;

[0015] The conversion method includes short-time Fourier transform, the filter bank includes a mel ratio filter bank, and the feature data generation method includes discrete cosine transform.

[0016] In one embodiment, the convolution layer is used to input the feature data, the pooling layer is used to input the output of the convolution layer and output the pooling features, the gating block is used to input the pooling features and assign weights to the pooling features and output weighted features, and the combination and reshaping block is used to input the weighted features and combine and reshape the weighted features to obtain fused feature data.

[0017] In one embodiment, the formula of the convolutional layer is as follows: i =Conv1D(X i ),i=1,2,...,C, where X i represents the i-th mode (or channel) of the input data; Conv1D(·) represents a 1D convolution operation to extract the temporal features of each mode; F i represents the feature representation of the i-th mode after convolution; C represents the total number of modes.

[0018] In one embodiment, the formula of the pooling layer is as follows: i =GAP1D(F i ).

[0019] In one embodiment, the formula of the gating block is as follows: Where ⊙ represents element-wise multiplication, σ(·) is the sigmoid activation function, and W i represents the learned gating weights, represents the gated i-th modal feature;

[0020] The formula of the combined reshape block is as follows: Among them, Concat(·) represents feature concatenation, combining the feature vectors of all modalities into a complete representation; Reshape(·) means adjusting the concatenated features to a shape suitable for the input of the multi-head attention mechanism; F fused Represents the fused multimodal features; represents the Cth modal feature after the combined reshaping block.

[0021] In one of the embodiments, the multi-head attention block is used to input the fused feature data, capture the interaction between the fused feature data, and unify and normalize the output of the attention head, and input it into the linear transformation layer; the linear transformation layer includes several dense layers, and the classification layer includes a flattening layer, a refined dense layer, and a probability dense layer; the output of the multi-head attention block is passed to the classification layer output through the linear transformation layer to obtain the category of the sample data.

[0022] In one embodiment, the formula of the multi-head attention block is as follows: h =Attention(Q h ,K h ,V h ),h=1,2,...,H, where H is the number of attention heads, Q h , K h and V h are the query, key, and value matrices of the h-th head, Z h represents the output of the h-th attention head;

[0023] The formula of the linear transformation layer is as follows: Linear =Dense(Dense(F attn ,units=128,activation=relu),units=128), where F attn represents the result of the feature after being processed by the multi-head attention mechanism; Dense(·) represents the fully connected layer, F Linear Represents the result of the feature after being processed by the linear transformation layer;

[0024] The formula of the flattening layer is as follows: z = Flatten (F Linear ), F Linear It represents the result of the feature after being processed by the linear transformation layer. Flatten(·) means flattening the high-dimensional tensor into a one-dimensional vector. z is the one-dimensional vector after flattening.

[0025] The formula for the refined dense layer is as follows: 1 =ReLU(W 1 z+b 1 ), where W 1and b 1 Represent the weight and bias of the dense layer, d 1 Represents the output after being processed by the fully connected layer;

[0026] The formula of the probability dense layer is as follows: y = Softmax (W 2 d 1 +b 2 ), where W 2 and b 2 are the weights and biases of the final dense layer, d 1 Represents the feature representation after a layer of nonlinear transformation.

[0027] In one embodiment, the unification is achieved by linear transformation, and the linear transformation formula is as follows: Z = Concat (Z 1 ,Z 2 ,...,Z H )W O , where Z H represents the output of the Hth attention head, W O represents the linear projection matrix; the normalization is achieved through layer normalization and residual connection, and the formula of layer normalization is as follows: F attn =LayerNorm(F fused +Z), where Z represents the output of the multi-head attention mechanism; + represents residual connection; LayerNorm(·) represents layer normalization; F fused Represents the fused multimodal features; F attn Represents the result of the features processed by the multi-head attention mechanism.

[0028] The present invention also provides a method for assessing disease risk, comprising the following steps: inputting sample data to be assessed into the multimodal neural network model, calculating the category of the sample data to be assessed, and analyzing the risk of the sample data to be assessed for the disease to be assessed according to the category.

[0029] This multimodal neural network model (GAMM-Net model) achieved excellent performance in performance indicators such as classification accuracy, AUC, recall rate and F1 score, surpassing the results of current state-of-the-art research, demonstrating its robustness and effectiveness in handling complex respiratory disease classification tasks.

[0030] In one embodiment, the sample data is multimodal data, and the sample data includes at least two of breathing sounds, coughing sounds, and speech sounds; and the disease to be evaluated is a respiratory system disease.

[0031] In one embodiment, the respiratory disease includes at least one of asthma, COVID-19, and cold.

[0032] The present invention also provides a system for assessing disease risk, comprising:

[0033] A data storage module, used for storing the multimodal neural network model and sample data to be evaluated;

[0034] A data analysis module, used to perform analysis according to the method; and

[0035] The data display module is used to output and display the risk of the disease to be evaluated.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention provides a multimodal neural network model based on gated multi-head attention and its application. The multimodal neural network model based on gated multi-head attention can integrate multimodal data and has high accuracy in classifying sample data. The model can be used to perform more accurate risk assessment of various respiratory diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the architecture diagram of the GAMM-Net model in Example 1;

[0039] Figure 2 is a workflow diagram of the multimodal neural network model in Example 2;

[0040] Figure 3 The sound wave images of different categories obtained by classification by the multimodal neural network model in Example 2;

[0041] Figure 4 This is the confusion matrix result diagram of the GAMM-Net model on the test data. DETAILED DESCRIPTION

[0042] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0044] source:

[0045] Unless otherwise specified, the reagents, materials, and equipment used in this example are all commercially available; the experimental methods are all conventional experimental methods in the art unless otherwise specified.

[0046] Example 1

[0047] A multimodal neural network model based on gated multi-head attention (abbreviated as: GAMM-Net model).

[0048] The model consists of a preprocessing module, a gated fusion module, and a data output module; the preprocessing module includes an enhancement block and a feature extraction block; the gated fusion module includes a convolutional layer, a pooling layer, a gated block, and a combined reshaping block; the data output module includes a multi-head attention block, a linear transformation layer, and a classification layer.

[0049] The enhancement block processes the sample data through data enhancement (at least one of time shift, time stretch, pitch adjustment, volume adjustment, noise addition, and Mixup) to obtain enhanced data, and inputs the enhanced data into the feature extraction block. The feature extraction block includes a transformation block, a filter group, a logarithmic conversion block, and a feature generation block; the transformation block converts the input enhanced data from the time domain to the frequency domain. In this embodiment, the conversion is performed specifically by the short-time Fourier transform method; the converted enhanced data is passed to the logarithmic conversion block through the filter group to obtain logarithmic data. In this embodiment, the filter group adopts a mel ratio filter group; the logarithmic data is input into the feature generation block, and the feature data is output. In this embodiment, the feature data is generated by the discrete cosine transform method.

[0050] The feature data is input into the convolution layer of the gated fusion module, and each channel is initially processed through a series of 1D convolution blocks, which aims to extract the most relevant and informative features from the MFCC representation of the audio signal. These convolution blocks are specifically designed to emphasize the key frequency components of each sound mode and effectively capture the key patterns in the audio data. After being processed by the convolution layer, it is input into the pooling layer to obtain the pooled features. The features of each mode pass through the global average pooling (GAP) layer, which reduces the spatial dimension by averaging the values ​​on each feature map. The pooled features are input into the gated block to assign weights and output weighted features. The gated fusion mechanism combines the processed features of each mode to adjust their contributions, and then input into the combined reshaping block to combine and reshape the weighted features to obtain fused feature data.

[0051] Subsequently, the fused features are then input to the multi-head attention block to capture global dependencies across modalities. A linear transformation layer (including 2 dense layers) is added to further refine the feature representation. Finally, the processed features are input to the classification layer, which includes a flattening layer, a refined dense layer, and a probability dense layer. In the classification layer, the data processed by the linear transformation layer passes through the flattening layer, the refined dense layer, and the probability dense layer in sequence. In this process, the processed features are flattened and passed through two fully connected layers (dense). The softmax function is applied in the final layer to produce the predicted category probability of disease classification.

[0052] exist Figure 1 In the paper, the architecture of the GAMM-Net model is presented, highlighting its ability to handle multimodal data. The model is specifically designed to handle multiple input data channels, including cough sounds, breath sounds, and speech sounds, each of which is processed independently before fusion. Each modality is passed through a batch normalization layer to ensure data standardization. The input feature shape structure is (3, 216, 40), reflecting three different sound modalities: cough, breath sounds, and speech, with 216 frames per frame and 40 MFCC coefficients per frame.

[0053] Example 2

[0054] The workflow of the multimodal neural network model based on gated multi-head attention in Example 1.

[0055] 1. Data preprocessing.

[0056] 1. Data augmentation.

[0057] Initially, the three modalities — cough sounds, breath sounds, and speech sounds — were combined, and then the lower quality samples were filtered out. To enhance the dataset and improve the model generalization ability, several data augmentation techniques were applied:

[0058] Time shifting: This technique involves shifting the audio signal along the time axis by a certain number of samples, expressed as:

[0059] x′(t)=x(t+Δt)

[0060] Where Δt is the time offset applied. This models the temporal variation of the audio, helping the model to be more robust to differences in timing and synchronization.

[0061] Time stretching: stretching or compressing audio without changing its pitch, calculated as:

[0062]

[0063] where α is the time stretching factor. This simulates changes in speaking or breathing rate, making the model more resilient to differences in audio speed.

[0064] Pitch adjustment: Change the pitch of audio by adjusting the frequency content, formulated as:

[0065] x′(t)=x(t)·e jωt

[0066] where ω is the frequency shift. This augmentation mimics the variation in intonation of the voice, helping the model generalize better across different speakers.

[0067] Volume adjustment (dB gain): The amplitude of the audio signal is adjusted by the gain factor, expressed as:

[0068] x′(t)=g·x(t)

[0069] Where g is the gain factor in decibels (dB). This helps the model to be invariant to differences in volume, ensuring it can effectively handle both soft and loud inputs.

[0070] Noise addition: Add background noise to the audio, expressed as:

[0071] x′(t)=x(t)+n(t)

[0072] Where n(t) is the noise signal. This enhancement helps the model to identify relevant features even in noisy environments, which is crucial for practical applications.

[0073] Mixup: This technique mixes two different audio samples to create a new sample:

[0074] x′=λx i +(1-λ)x j

[0075] where λ∈[0,1], x i and x jare two different audio samples. Mixup smoothes the decision boundary of the model and reduces overfitting by providing a richer set of training samples.

[0076] These enhancements increase the diversity of the training data, helping to mitigate overfitting and improve the model's robustness under different conditions and inputs.

[0077] After the sample data is input into the preprocessing module, the enhancement block processes the sample data through the above data enhancement technology to obtain enhanced data.

[0078] 2. MFCC feature extraction.

[0079] After data enhancement, the feature extraction block is input to extract Mel-frequency cepstral coefficients (MFCCs) from the audio signal output by the enhancement block (i.e., enhanced data). Each audio sample is represented by 216 frames, with 40 MFCC coefficients per frame, providing a detailed time-frequency representation of the signal. The MFCC extraction process includes several key steps designed to capture the most relevant features of the audio data, which is particularly useful for distinguishing different breathing conditions.

[0080] The first step in MFCC extraction is to convert the audio signal from the time domain to the frequency domain using a short-time Fourier transform (STFT) via a transform block. This transform allows the frequency content of the signal to be analyzed over short, overlapping windows, which is critical for identifying temporal patterns in breathing sounds.

[0081] Once in the frequency domain, the power spectrum of each frame is passed through a mel-scaled filter bank, which simulates the nonlinear perception of frequency by the human ear. The mel-scale emphasizes lower frequencies, which are more relevant to human auditory perception, while compressing higher frequencies. This step is critical to ensure that the extracted features are perceptually relevant.

[0082] After applying the mel-scaled filter bank, the data is fed into a logarithmic transformation block, which calculates the logarithm of the resulting mel-scaled power to produce logarithmic data. Logarithmic transformation helps compress the dynamic range of the spectrum, making the features more robust to changes in signal amplitude. It also helps approximate the logarithmic perception of loudness by the human ear.

[0083] Finally, the logarithmic data is fed into the feature generation block, which applies a discrete cosine transform (DCT) to the logarithmic mel-scale spectrum to generate MFCCs (i.e., feature data). DCT decorrelates the features, compressing the information into a smaller set of coefficients, typically retaining the first few coefficients (40 in this case) that capture the most significant information about the frequency content of the audio signal.

[0084] The detailed principle and formula of MFCC extraction are shown in equations (1) and (2).

[0085]

[0086] In these equations, Sm(k) represents the mel-ratio spectrum, H m (k) is the mel filter bank, X m (n) is the short-time Fourier transform (STFT) of the signal.

[0087] In this embodiment, MFCCs are extracted at a sampling rate of 22,050 Hz, using 40 mel filter banks, and each audio sample is represented as a matrix of 216 frames multiplied by 40 coefficients. This configuration ensures that the most relevant temporal and spectral features of the audio signal are captured. The extracted MFCC features are then used as input to the GAMM-Net model, ensuring that the most relevant frequency features of the audio signal are used for effective disease classification.

[0088] 2. Gated fusion mechanism.

[0089] The gated fusion mechanism is a key component for extracting and integrating multimodal features, i.e., cough, breath sounds, and speech signals in the context of respiratory disease classification. Each modality represents a different type of audio signal, and integrating these diverse sources of information requires careful processing to emphasize the most relevant features.

[0090] Given input data X∈RN×D×C, where N is the number of samples, D is the dimension of each sample, and C represents the number of channels (modalities), the gated fusion module processes each modality independently using a one-dimensional convolutional layer:

[0091] F i =Conv1D(X i ),i=1,2,...,C (3)

[0092] The meaning of each character in formula (3): X i represents the i-th mode (or channel) of the input data; Conv1D(·) represents a 1D convolution operation to extract the temporal features of each mode; F i represents the feature representation of the i-th mode after convolution; C represents the total number of modes; this model has three modes: cough sound, breath sound and speech.

[0093] These 1D convolutional layers are designed to capture temporal features within each modality independently. For example, a cough signal may exhibit unique patterns different from speech or breathing, and these layers help separate these patterns. In this setting, the input to the cough modality is more heavily regularized by applying L1 regularization with a coefficient value of 0.01. This is because the quality of cough data can be inconsistent, and L1 regularization prevents overfitting by penalizing the weights, ensuring that only the most informative features are retained.

[0094] The output of each convolutional layer then goes into a pooling layer, passing through a global average pooling (GAP) layer to reduce the spatial dimension:

[0095] G i =GAP1D(F i )(4)

[0096] The GAP layer compresses the feature map into a low-dimensional vector, capturing the essence of the features while reducing the complexity of subsequent layers. This step is particularly important for handling the variability of different audio inputs by extracting key information from each modality.

[0097] Then, the pooled features obtained by the pooling layer output are input into the gating block to obtain the gated features. In this process, the model applies the gating mechanism to control the contribution of each modality to the final fusion representation:

[0098]

[0099] Here, the meaning of each character in formula (5) is: ⊙ represents element-by-element multiplication, σ(·) is the sigmoid activation function, W i represents the learned gating weights, represents the gated i-th modality feature. The sigmoid function ensures that the gated value remains between 0 and 1, effectively adjusting the importance of each modality feature vector. This approach allows the model to dynamically prioritize modalities based on their relevance to the current task.

[0100] Finally, the gated features of all modalities are input into the combined reshape block, concatenated and reshaped to obtain fused feature data for further attention mechanism processing:

[0101]

[0102] The meaning of each character in formula (6): Concat(·) means feature concatenation, combining the feature vectors of all modalities into a complete representation; Reshape(·) means adjusting the concatenated features to a shape suitable for the input of the multi-head attention mechanism; F fused Represents the fused multimodal features; represents the Cth modal feature after the combined reshaping block.

[0103] By concatenating these gated features, the model forms a comprehensive representation that contains key information from all input modalities. This fusion strategy is particularly effective in scenarios where different audio signals carry complementary information, as it allows the model to exploit these synergistic effects.

[0104] 3. Multi-head attention mechanism.

[0105] The multi-head attention (MHA) mechanism in GAMM-Net plays a crucial role in refining the fused features, allowing the model to capture long-term dependencies across modalities. This is particularly important in the context of respiratory disease classification, as the relationship between cough sounds, breath sounds, and speech patterns can be subtle but significant. The fused feature data is fed into the multi-head attention block to capture global dependencies across modalities.

[0106] MHA mechanism is used to fuse the features F fused , and is formulated as follows:

[0107] Z h = Attention(Q h , K h , V h ), h = 1, 2, . . . , H (7)

[0108] The meaning of each character in formula (7): H is the number of attention heads, Q h , K h and V h are the query, key, and value matrices of the h-th head, Z h represents the output of the hth attention head. Each attention head operates independently, allowing the model to focus on different aspects of the input features at the same time. In the model of this embodiment, H is set to 4, which enables the model to capture diverse interactions between modalities. This multi-faceted attention helps capture complex interactions between different modalities.

[0109] Then, the attention outputs of each head are concatenated and linearly transformed:

[0110] Z = Concat(Z 1 , Z 2 , . . . , Z H )W O (8)

[0111] Z H represents the output of the Hth attention head, W O Represents the linear projection matrix, transforming the concatenated multi-head output into a unified representation. This transformation combines the insights of each attention head into a unified representation, which is then normalized using layer normalization and residual connections to stabilize the training process and ensure that the learned representation is robust:

[0112] F attn = LayerNorm(F fused + Z) (9)

[0113] The meaning of each character in formula (9): Z represents the output of the multi-head attention mechanism; + represents the residual connection, which adds the original fusion feature to the attention feature; LayerNorm (·) represents layer normalization, which is used to stabilize training and improve the convergence of the model; F fused Represents the fused multimodal features; F attn Represents the result of the features processed by the multi-head attention mechanism.

[0114] The inclusion of MHA enhances the model’s ability to identify subtle but critical patterns in audio data, contributing to more accurate and reliable disease classification.

[0115] 4. Linear transformation layer.

[0116] The linear transformation layers in the GAMM-Net model are used to further refine the features processed by the multi-head attention mechanism. These layers play a key role in enhancing the discriminative ability of the model, ensuring that the complex interactions within the integrated and attended features are effectively captured before preparing for classification. The transformation is done using two dense layers as shown in the following equation:

[0117] F Linear = Dense(Dense(F attn , units = 128, activation = relu), units = 128) (10)

[0118] The meaning of each character in formula (10): F attn represents the result of the feature after being processed by the multi-head attention mechanism; Dense(·) represents the fully connected layer. The first layer uses the ReLU activation function to extract nonlinear features. The second layer does not use the activation function, adds linear transformation, extracts features and adjusts the feature representation for subsequent classification; F Linear Represents the result of the feature after being processed by the linear transformation layer;

[0119] After the multi-head attention mechanism, the output is passed through two dense layers. The first dense layer consists of 128 units and uses the ReLU activation function to introduce nonlinearity and capture complex relationships within the features. Subsequent dense layers perform linear transformations to map the features to a space that is more conducive to subsequent classification tasks. This two-layer structure ensures that the focus is on the most relevant information when the features are ready for final processing.

[0120] 5. Final classifier.

[0121] The final classification layer in the GAMM-Net model is responsible for producing probabilistic outputs corresponding to the different respiratory diseases being classified.

[0122] The output from the linear transformation layer is passed through a flatten layer to convert the multi-dimensional tensor into a one-dimensional vector:

[0123] z = Flatten(F Linear ) (11)

[0124] The meaning of each character in formula (11): F Linear It represents the result of the feature after being processed by the linear transformation layer; Flatten(·) means flattening the high-dimensional tensor into a one-dimensional vector; z is the one-dimensional vector after flattening.

[0125] Then, the vector z is fed into a dense layer with 512 units and ReLU activation function (i.e., refinement dense layer) to further refine the feature representation:

[0126] d 1 = ReLU(W 1 z + b 1 ) (12)

[0127] The meaning of each character in formula (12): W 1 and b 1 Represent the weight and bias of the dense layer, d 1 It represents the output after being processed by the fully connected layer, and the feature representation after a layer of nonlinear transformation, which is used for the subsequent classification operation.

[0128] The final output layer (i.e., probability dense layer) is a dense layer with 4 units, corresponding to the number of categories: asthma, cold, healthy, and COVID-19. This layer uses a softmax activation function to convert the output into a probability (the final output result is obtained through the softmax activation function, which generates a probability value for each category and selects the category with the highest probability value as the prediction result):

[0129] y = Softmax(W 2 d 1 + b 2 ) (13)

[0130] The meaning of each character in formula (13): W 2 and b 2 are the weights and biases of the final dense layer, d 1 Represents the feature representation after a layer of nonlinear transformation. The softmax function ensures that the sum of the probabilities across categories is equal to 1, providing a clear probabilistic explanation for the model's predictions:

[0131]

[0132] The model is optimized using the categorical cross entropy loss function, defined as:

[0133]

[0134] where y i is the true label and is the category The predicted probability of . This loss function is standard for multi-class classification tasks.

[0135] Use a learning rate of 1×10 -3 The Adam optimizer is used, which provides the best trade-off between convergence speed and model accuracy. In addition, the performance of the model on various classifications is comprehensively evaluated by tracking metrics such as accuracy, area under the curve (AUC), and recall.

[0136] This setting ensures that the model can handle the nuances of respiratory disease classification well and provide reliable and accurate predictions that are critical for medical diagnosis. The workflow of the multimodal neural network model of the present invention is as follows: Figure 2 As shown in the figure, the different types of sound waves classified by the model are as follows Figure 3 shown.

[0137] Experimental example

[0138] 1. Dataset and Experimental Settings.

[0139] The experiments were conducted on a server with the following configurations: (1) GPU: GA102GL[A10] with 25.4GB memory; (2) CPU: 12 Intel Xeon Silver 4210R cores; (3) RAM: 61.9GB; (4) Storage: 4TB. The experiments were implemented using CUDAv11.4, TensorFlow-GPU v2.10.0, and other related Python libraries.

[0140] The inventors used the Coswara crowdsourcing dataset, which includes multimodal data (cough sounds, breath sounds, speech) and respiratory disease data (colds, asthma, COVID-19). To solve the problem of dataset imbalance, the inventors supplemented it with COUGHVID data and Virufy's clinical data. The dataset is divided into a training set (6494 samples), a validation set (800 samples), and a test set (800 samples), where each category in the validation set and the test set contains 200 samples.

[0141] To enhance the dataset and improve the generalization of the model, the inventors applied several data augmentation techniques. These included shifting the audio signal by 20,000 samples in time, time stretching by a factor of 1.2 to simulate changes in speech or breathing rate, and increasing the amplitude by 100 dB to simulate different loudness levels. In addition, the pitch was adjusted by lowering 3 semitones, and white noise with an amplitude of 200 was added to simulate background noise. The inventors also used a mixup technique with a mixing ratio of α = 0.5 to combine audio sample pairs to provide mixed samples for the model and smooth decision boundaries.

[0142] For all experiments, the initial learning rate of the Adam optimizer was set to 0.001 and decreased by a factor of 0.1 every 20 epochs to enhance convergence. The batch size was configured to be 64. The categorical cross entropy loss function was used, and the classification performance was evaluated using the accuracy, AUC, and F1 score metrics.

[0143] 2. Ablation study.

[0144] 1. Multimodal ablation.

[0145] In the multimodal ablation study, as shown in Table 1, the inventors evaluated the impact of different modality combinations (cough sounds, breath sounds, speech) on disease classification performance. The results clearly show that using all three modalities simultaneously can provide the best performance, achieving an accuracy of 0.951 and an AUC of 0.992. The breakdown of the results shows that the absence of any modality will lead to a decrease in accuracy and AUC, emphasizing the importance of using a multimodal approach for effective disease classification. These findings highlight that each modality provides unique and valuable information, improves the overall classification performance, and their combination produces the most reliable diagnostic results.

[0146] Table 1 Multimodal data ablation study

[0147]

[0148] 2. Module ablation.

[0149] Table 2 shows the results of the ablation study on the multi-head attention mechanism (MHA) and additional linear transformations in the model. The addition of MHA and linear layers significantly enhances the classification performance, especially in terms of F1 score and AUC. The key observations of the ablation study include:

[0150] Table 2 Ablation study of attention mechanism and linear transformation

[0151]

[0152] (1) Without MHA: The AUC of the model drops to 0.950 and the accuracy drops to 0.925, indicating that the attention mechanism plays a crucial role in focusing on the most relevant features in different modalities.

[0153] (2) Without linear layer: The accuracy and AUC are reduced to 0.940 and 0.955 respectively. This shows that the linear transformation further refines the feature representation and helps improve the classification performance.

[0154] The module ablation results highlight the importance of each component in the model architecture and confirm that the integration of attention mechanism and linear transformation is crucial to achieve high classification accuracy and robustness.

[0155] 3. Multimodal disease classification.

[0156] Table 3 provides a comprehensive overview of the model’s diagnostic performance on different disease categories. The model performs very well in COVID-19 detection, achieving an accuracy of 0.990 and an AUC of 0.991. This highlights the model’s strong ability to correctly identify COVID-19 cases. The model’s performance on other disease categories is equally remarkable; for example, the model achieves an accuracy of 0.975 and an AUC of 0.997 for colds, and an accuracy of 0.950 and an AUC of 0.997 for asthma.

[0157] The overall performance indicators, accuracy of 0.951 and AUC of 0.992, reflect the reliable performance of the model on all categories. These indicators show that the model maintains a high level of diagnostic accuracy and robustness in distinguishing different disease states.

[0158] In order to get a clearer picture of the performance of the model on different categories, the inventors have drawn a confusion matrix (see Figure 4 ). The confusion matrix shows that the classification performance of the healthy class is relatively poor compared to other classes. Specifically, out of 200 samples classified as healthy, 15 were incorrectly identified as COVID-19. This misclassification may be attributed to the inherent similarity in acoustic features between the healthy and COVID-19 classes, which may cause confusion during the model classification process. Further analysis and possible additional feature engineering may be required to improve the accuracy of the healthy class and reduce this misclassification.

[0159] Table 3 Classification effect of GAMM-NET model on test data

[0160]

[0161] Finally, the performance of the model of the present invention is compared with other COVID-19 detection models in Table 4. The model of the present invention performs best among most other methods, especially in terms of accuracy and AUC, demonstrating the effectiveness of integrating multimodal data.

[0162] Table 4 Comparison with other COVID-19 detection models

[0163]

[0164]

[0165] 4. Conclusion

[0166] In the present invention, the inventors developed a system for multi-class respiratory disease classification using multimodal data, including cough, breathing, and speech. The inventors implemented a multimodal classification model based on a gating mechanism. The Coswara dataset was utilized, as well as crowdsourced data from COUGHVID and clinical data from the Virufy research team in India. To address the challenges posed by imbalanced and low-quality samples, the inventors adopted a data augmentation process including mixup, pitch adjustment, time shifting, silence removal, and segmentation. Mel-frequency cepstral coefficients (MFCCs) were extracted from the audio data as input features, forming an optimized dataset for training deep learning models for respiratory disease classification, with diagnostic categories including healthy, COVID-19 positive, asthma, and cold.

[0167] The inventors trained and tested the custom-designed GAMM-Net model of the present invention, which integrates a gated fusion mechanism, multi-head attention, and enhanced linear transformation for effective feature extraction and classification of multimodal audio data. Experimental results show that the model of the present invention surpasses most existing models on multiple evaluation indicators. Notably, the model performs well in COVID-19 detection, with an accuracy of 0.990 and an AUC of 0.991. The model also shows strong performance on other disease categories, with an accuracy of 0.975 and an AUC of 0.997 for cold detection and an accuracy of 0.950 and an AUC of 0.997 for asthma detection. Although the classification of healthy samples is relatively weak with an accuracy of 0.890, the accuracy reaches 0.983 due to the inherent similarity between healthy and COVID-19 samples in MFCC features. Overall, the model demonstrates strong diagnostic accuracy and reliability, with an accuracy of 0.951 and an AUC of 0.992, verifying the feasibility of the proposed model and improving the diagnostic performance of various respiratory diseases.

[0168] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multimodal neural network model based on gated multi-head attention, characterized in that: It includes a preprocessing module, a gated fusion module and a data output module; the preprocessing module is used to preprocess the sample data, and input the feature data obtained after the preprocessing into the gated fusion module; the gated fusion module is used to extract and integrate the feature data to obtain fused feature data, and input the fused feature data into the data output module; The preprocessing module includes an enhancement block and a feature extraction block; the gated fusion module includes a convolution layer, a pooling layer, a gated block and a combined reshaping block; the data output module includes a multi-head attention block, a linear transformation layer and a classification layer.

2. The multimodal neural network model according to claim 1, characterized in that: The enhancement block processes the sample data through data enhancement to obtain enhanced data; The feature extraction block includes a transformation block, a filter group, a logarithmic conversion block, and a feature generation block. The transformation block is used to convert the enhanced data from the time domain to the frequency domain. The converted enhanced data is passed to the logarithmic conversion block through the filter group for logarithmic conversion to obtain logarithmic data. The feature generation block is used to process the logarithmic data to generate feature data.

3. The multimodal neural network model according to claim 2, characterized in that: The data enhancement includes: at least one of time shift, time stretch, pitch adjustment, volume adjustment, noise addition, and Mixup; The conversion method includes short-time Fourier transform, the filter bank includes a mel ratio filter bank, and the feature data generation method includes discrete cosine transform.

4. The multimodal neural network model according to claim 1, characterized in that: The convolution layer is used to input the feature data, the pooling layer is used to input the output of the convolution layer and output the pooling features, the gating block is used to input the pooling features and assign weights to the pooling features and output weighted features, and the combination and reshaping block is used to input the weighted features and combine and reshape the weighted features to obtain fused feature data.

5. The multimodal neural network model according to claim 4, characterized in that: The formula for the gating block is as follows: Where ⊙ represents element-wise multiplication, σ(·) is the sigmoid activation function, and W i represents the learned gating weights, represents the gated i-th modal feature; The formula of the combined reshape block is as follows: Among them, Concat(·) represents feature concatenation, combining the feature vectors of all modalities into a complete representation; Reshape(·) means adjusting the concatenated features to a shape suitable for the input of the multi-head attention mechanism; F fused Represents the fused multimodal features; represents the Cth modal feature after the combined reshaping block.

6. The multimodal neural network model according to claim 1, characterized in that: The multi-head attention block is used to input the fused feature data, capture the interaction between the fused feature data, and unify and normalize the output of the attention head, and input it into the linear transformation layer; the linear transformation layer includes several dense layers, and the classification layer includes a flattening layer, a refined dense layer, and a probability dense layer; the output of the multi-head attention block is passed to the classification layer output through the linear transformation layer to obtain the category of the sample data.

7. The multimodal neural network model according to claim 6, characterized in that: The formula of the multi-head attention block is as follows: h =Attention(Q h ,K h ,V h ),h=1,2,...,H, where H is the number of attention heads, Q h , K h and V h are the query, key, and value matrices of the h-th head, Z h represents the output of the h-th attention head; The formula of the linear transformation layer is as follows: Linear =Dense(Dense(F attn ,units=128,activation=relu),units=128), where F attn represents the result of the feature after being processed by the multi-head attention mechanism; Dense(·) represents the fully connected layer, F Linear Represents the result of the feature after being processed by the linear transformation layer; The formula of the flattening layer is as follows: z = Flatten (F Linear ), F Linear It represents the result of the feature after being processed by the linear transformation layer. Flatten(·) means flattening the high-dimensional tensor into a one-dimensional vector. z is the one-dimensional vector after flattening. The formula of the refined dense layer is as follows: d1=ReLU(W1z+b1), where W1 and b1 represent the weight and bias of the dense layer respectively, and d1 represents the output after being processed by the fully connected layer; The formula of the probability dense layer is as follows: y=Softmax(W2d1+b2), where W2 and b2 are the weight and bias of the final dense layer, and d1 represents the feature representation after a layer of nonlinear transformation.

8. A method for assessing disease risk, characterized in that: The following steps are involved: The sample data to be evaluated is input into the multimodal neural network model described in any one of claims 1 to 7, the category of the sample data to be evaluated is calculated, and the risk of the sample data to be evaluated for the disease to be evaluated is analyzed according to the category.

9. The method according to claim 8, characterized in that The sample data is multimodal data, and the sample data includes at least two of breathing sounds, coughing sounds, and speech sounds; and the disease to be evaluated is a respiratory system disease.

10. A system for assessing disease risk, characterized in that: include: A data storage module, used to store the multimodal neural network model according to any one of claims 1 to 7 and sample data to be evaluated; A data analysis module, configured to perform analysis according to the method of any one of claims 8 to 9; and The data display module is used to output and display the risk of the disease to be evaluated.

Images