Lung function assessment method and system based on bidirectional cross attention

Abstract

The application discloses a lung function evaluation method and system based on bidirectional cross attention. The breath sound of a subject is collected, and the original audio is preprocessed to obtain standardized audio frames. Multi-scale time-frequency feature extraction is performed on the standardized audio frames to obtain a breath sound time-frequency feature sequence. The time-frequency feature sequence and a learnable hidden vector are input into a BiXT encoder for bidirectional cross attention encoding to obtain a breath sound time sequence position feature and a pathological semantic global feature, respectively. Spatial attention and channel attention enhancement of position information and semantic information are performed through a double attention module to realize feature fusion. Frame-level abnormal event detection, sequence-level breath sound classification and lung function state evaluation are performed based on the fused features. Model training is completed. The application considers global context modeling, feature fine extraction and multi-task joint output, which helps to improve the accuracy of breath abnormality detection, breath sound classification and lung function evaluation, and enhances the result interpretability.

Description

Technical Field

[0001] This invention belongs to the field of respiratory sound detection, classification, perception and recognition technology, and in particular relates to a method and system for detecting and classifying abnormal lung function based on deep learning and respiratory sounds. Background Technology

[0002] Lung sounds are the sound signals produced by the human body during respiration, containing rich physiological and pathological information. Therefore, lung diseases can be diagnosed through lung sound signals. Compared with techniques such as chest X-rays, pulmonary angiography, and arterial blood gas analysis, lung auscultation is more economical and non-invasive. However, because the frequency range of lung sounds exceeds the limits of human hearing, it can lead to missed diagnoses to some extent. Furthermore, lung auscultation is significantly influenced by subjective factors such as the physician's personal experience and professional knowledge. Therefore, misdiagnosis is possible. Computer-aided diagnosis based on lung sounds can, to some extent, alleviate the burden on the medical system and reduce missed and misdiagnosed cases.

[0003] It is worth noting that there is a significant lack of pulmonary function assessment in the current diagnosis and monitoring of lung diseases, which further exacerbates the burden of medical diagnosis and treatment bias. As an important means of diagnosing, assessing, and predicting the prognosis of lung diseases, pulmonary function assessment requires the accurate acquisition of physiological parameters such as lung ventilation, gas exchange, and airway responsiveness to assist in diagnosis and treatment. However, its clinical application and research currently have many shortcomings: First, traditional pulmonary function tests are greatly affected by patient cooperation. Elderly people, those with severe lung diseases, and other high-risk groups are unable to complete the standardized procedures, making assessment difficult. Second, traditional tests are not sensitive enough to early airway abnormalities and have high operational thresholds, resulting in low coverage at the grassroots level. Third, most existing research and deep learning frameworks based on lung sounds do not further acquire pulmonary function status assessment through breath sound signals, ignoring the indicative role of pulmonary function status in disease type and severity. This leads to diagnostic models that cannot fully reflect the lung status, making it difficult to achieve accurate hierarchical diagnosis and treatment, and failing to meet the assessment needs of dynamic changes in pulmonary function during long-term monitoring, thus limiting the clinical value of computer-aided diagnostic systems.

[0004] With the advancement of deep learning, deep learning models have gradually become an important tool for lung sound recognition. To achieve long-term monitoring of recordings of a single patient or comparisons between different patients, there is a need to meet and realize an automated framework for lung sound event detection. In this task, continuous lung sound recordings can be segmented into discrete events to better segment and locate the time points and features of breath sounds. Despite this need, only a few studies have explored the use of deep learning techniques in lung sound event detection; most literature focuses only on distinguishing healthy participants from abnormal patients. Furthermore, existing deep learning frameworks are primarily cascaded, where features extracted by the previous network are used as input to the next network. This can lead to the loss of some features that the subsequent network might focus on. For example, when the dimensions of two networks are mismatched, dimensionality transformation may cause the discarding or loss of some features. In addition, feature extraction and transmission may introduce noise or interference, leading to a certain degree of information loss.

[0005] In 2022, Tharindu et al. (Robust and Interpretable Temporal Convolution Network for Event Detection in Lung Sound Recordings) proposed using a spatiotemporal convolutional network (TCN) with different branches to identify and detect abnormal sounds at different time locations. However, this paper has the following drawbacks: it only uses the Mel-frequency cepstral coefficients (MFCCs) of breath sounds as input features, missing fine-grained features of different breath sounds; it uses a multi-branch TCN structure, but the branches are independent of each other, making it impossible to effectively utilize the interaction information between features; and it does not further obtain lung function status assessment through breath sound signals, making it difficult to comprehensively reflect the pathophysiological state of the lungs, thus reducing the accuracy of diagnosis and clinical applicability.

[0006] In 2025, Gaoyang Dong et al. proposed a two-branch network based on time-frequency information to automatically classify four types of respiratory sounds (Respiratory sounds classification by fusing the time-domain and 2D spectral features). However, this paper has the following drawbacks: Wav2vec and SSAST, as Transformer networks, suffer from increased complexity and high training costs; the proposed method can only handle semantic tasks and cannot effectively perceive the location information of abnormal sounds or assess lung function, thus failing to meet the actual needs of clinical practice for grading the severity of lung diseases and dynamically monitoring lung function.

[0007] Zhang Minghui et al. (CN116246654A) used an improved Swing-Transformer to achieve automatic classification of breath sounds. However, this invention has the following drawbacks: the use of Swing-Transformer in this invention increases complexity and training costs, making it suitable only for complex and high-precision visual tasks; the method proposed in this invention cannot synchronously model "semantics (what)" and "location (where)" during the processing, and the two cannot interact and be optimized simultaneously, which limits dense prediction tasks.

[0008] Chen Riqing et al. (CN118072968A) divided lung sounds into multiple temporal segments and predicted boundary windows and predicted values, thus achieving both segmentation and localization, as well as classification and recognition. However, this invention has the following drawbacks: the use of a concatenated network leads to the loss of important features and makes it impossible to focus on certain important features; although it achieves both localization and classification, it does not perform importance screening for features at different scales during use, which results in insensitivity to changes in the scale and position of the bounding box; and it does not achieve dynamic assessment of lung function through breath sound signals, making it impossible to infer lung function status based on the changing trends of breath sound features.

[0009] In summary, the current research status has the following problems: the self-attention layer of the Transformer-based model has excessively high complexity; existing models suffer from performance degradation, modality limitations, and insufficient generality; "semantics (what)" and "location (where)" cannot be modeled synchronously and are inefficient; as an important source of information, lung function assessment is not effectively obtained through breath sound signals, failing to meet the actual clinical needs for accurate grading of lung diseases and dynamic monitoring of lung function. Summary of the Invention

[0010] The purpose of this invention is to propose a lung function assessment method and system based on bidirectional cross-attention to address the challenges posed by increased lung diseases due to air pollution and an aging population. By introducing a novel deep learning architecture, this method overcomes problems such as feature loss, insufficient information utilization, and low operational efficiency in existing methods, achieving accurate detection and localization of abnormal sound events in temporal lung sound data. This system can not only effectively classify different types of breath sounds but also accurately pinpoint the start and end times of abnormal sounds, thereby improving the diagnostic capabilities of medical systems for lung diseases, reducing the burden on medical staff, and lowering the risk of missed diagnoses and misdiagnoses.

[0011] To achieve the objective of this invention, a lung function assessment method based on bidirectional cross-attention is disclosed, comprising the following steps:

[0012] Step 1: Collect the subject's breath sounds, and then preprocess the raw audio to obtain standardized audio frames;

[0013] Step 2: Extract multi-scale time-frequency features from the standardized audio frames to obtain the time-frequency feature sequence of breath sounds;

[0014] Step 3: Input the time-frequency feature sequence of breath sounds and the learnable latent vector into the BiXT encoder and perform BiXT bidirectional cross-attention encoding to obtain the temporal position features of breath sounds and the global features of pathological semantics, respectively.

[0015] Step 4: Feature Fusion; Introduce a dual attention module to enhance spatial and channel attention for location and semantic information respectively, thereby achieving refined feature extraction;

[0016] Step 5: Multi-task detection and classification output, including frame-level abnormal event detection, sequence-level breath sound classification, and lung function status assessment;

[0017] Step 6: Model training implementation.

[0018] Furthermore, in step 1, the subject's chest or back breath sounds are collected using a microphone or electronic stethoscope; the original audio is then subjected to 75–2000 Hz bandpass filtering, adaptive removal of heart sounds, respiratory cycle segmentation, and amplitude normalization to obtain standardized audio frames.

[0019] Furthermore, in step 2, Mel spectrograms, Mel cepstral coefficients (MFCC), and constant Q transform (CQT) features are extracted from the preprocessed audio frames. After being concatenated along the channels, the token sequence is obtained through linear projection. Where N is the number of time frames, For feature dimensions.

[0020] Furthermore, in step 3, the token sequence... With learnable latent vectors The data is fed into the BiXT encoder, and the following process is executed for each layer:

[0021] Step 3-1: Initialize the implicit vectors;

[0022]

[0023] These are learnable latent vectors responsible for global semantics, and M is the number of latent vectors.

[0024] Step 3-2: Linear projection yields reference features and value features;

[0025] Latent vector reference features: ;

[0026] Latent vector value features: ;

[0027] Breathing sound frame reference features: ;

[0028] Breath sound frame value characteristics: ;

[0029] Represents the reference projection matrix. Denotes the value projection matrix; where X lat This represents the learnable pathological semantic latent vector matrix, where the subscript lat indicates the latent (latent) branch; X tok This represents the temporal token feature matrix of breath sounds, where the subscript tok indicates the token (breath sound frame) branch; R lat and R tok These represent latent vector reference features and breath sound frame reference features, respectively; R stands for the first letter of Reference; V lat and V tok These represent latent vector value features and breath sound frame value features, respectively; V stands for the first letter of Value; W... R W represents the reference projection matrix. V The values ​​represent the projection matrix, both of which are learnable parameters; M represents the number of latent vectors, N represents the number of breath sound time frames, and D represents the feature dimension; case is consistent, and X, R, V, and W all represent matrices.

[0030] Step 3-3: Calculate the symmetric similarity matrix;

[0031]

[0032] Representing implicit vectors The breathing sound frame similarity matrix, A is the Attention similarity matrix; the subscript lat denotes the latent vector branch, and the subscript tok denotes the breathing sound frame token branch, A tok ,lat represents the reverse breathing sound frame to latent vector similarity matrix; R lat Represents the latent vector reference feature, R tok This represents the reference feature of the breathing sound frame; the superscript T indicates matrix transpose; √D indicates that the square root of the feature dimension D is used as a scaling factor to suppress excessively large dot product values.

[0033] This indicates the scaling factor to prevent the value from becoming too large.

[0034] Satisfies symmetry:

[0035]

[0036] Steps 3-4: Two-way attention update;

[0037]

[0038]

[0039] This represents the amount of latent vector attention update. The attention update value for the breathing sound frame; Softmax is the normalization function that converts similarity into attention weights; V tok V represents the frame value characteristics of respiratory sounds. lat This represents the latent vector value feature.

[0040] Steps 3-5: Residual connectivity and layer normalization;

[0041]

[0042]

[0043] Layer normalization for stable training;

[0044] Steps 3-6: Feedforward Network (FFN) and Secondary Normalization;

[0045]

[0046]

[0047] 'Represents the updated latent vector output features of the current layer. This represents the output features of the breathing sound frame after the current layer is updated; Represents a feedforward network and performs nonlinear feature transformation; This represents the latent vector output by this layer. ; This indicates the characteristics of the breath sounds output from this layer. Where N is the number of time frames, M is the pathological semantic latent vector, and D is the feature dimension.

[0048] Furthermore, step 4 specifically includes the following steps:

[0049] Step 4-1: Feature splitting and input definition;

[0050] Temporal location features Feed it into the position attention branch to model the long-range temporal dependency between breathing sound frames, and improve the accuracy of abnormal location;

[0051] Pathological semantic features The input channel attention branch is used to model the global correlation between pathological feature channels and enhance the semantic expression of abnormality types.

[0052] Step 4-2, Location Attention Encoding;

[0053] Location features Perform dimensional reshaping to generate a query matrix. Key matrix Value matrix ;

[0054] Calculate the similarity matrix between Q and K, and obtain the temporal positional attention weights after Softmax normalization. W Q W K W V These represent the learnable projection matrices for the query, key, and value, respectively.

[0055] V is weighted and aggregated using attention weights, and then residually connected and layer normalized with the original location features;

[0056] Output enhanced temporal location features .

[0057] Step 4-3, Channel Attention Encoding;

[0058] semantic features Construct the autocorrelation matrix directly at the channel dimension Dependencies between learning channels;

[0059] Channel attention weights are obtained through Softmax, and semantic features are then weighted and enhanced.

[0060] By incorporating residual connections and layer normalization, the enhanced pathological semantic features are output. .

[0061] Step 4-4: Dual attention feature fusion;

[0062] Enhanced temporal location features With pathological semantic features Element-wise weighted fusion is performed to obtain joint features that simultaneously possess high-precision location awareness and strong semantic representation capabilities. This provides high-quality feature support for subsequent multi-task output; due to F pos ∈R N×D F cha ∈R M×D Due to the different dimensions, F_cha is first converted to R through pooling, broadcasting, or linear mapping before fusion. N×D And then with F pos Element-wise summation followed by LayerNorm normalization.

[0063] Furthermore, in step 5, the frame-level abnormal event detection steps are as follows:

[0064] Perform a linear transformation and sigmoid function on the features of each frame to output the anomaly probability.

[0065]

[0066] This represents the probability of an anomaly in frame t. The sigmoid function outputs a probability of 0-1, and linear indicates a linear transformation operation. This represents the feature vector of the t-th frame in the joint features;

[0067] Set an anomaly probability threshold, when If the value exceeds the threshold, it is determined to be an abnormal breathing sound frame. The timestamp is output, and finally the start time, end time and abnormal probability of all abnormal breathing sound frames are output.

[0068] Furthermore, in step 5, the specific steps for classifying sequential breath sounds are as follows:

[0069] Perform global average pooling on all frames, then classify them:

[0070]

[0071] The probability of breath sounds is represented by N, where N is the number of time frames for the joint features. The pathological type with the highest probability is selected as the final classification result for the breath sound sequence, and the classification label and corresponding probability are output. This supports the accurate identification of common abnormal sounds and provides a basis for preliminary screening of lung diseases.

[0072] Furthermore, in step 5, the lung function status assessment steps are as follows:

[0073] Based on sequence-level breath sound classification results and frame-level abnormal features, combined with preset lung function assessment mapping rules, the lung function status is inferred through breath sound features; based on the pathological types obtained from the classification, combined with the duration of abnormal breath sounds and the mean abnormal probability features, the assessment level of lung ventilation and gas exchange function is mapped, and the assessment basis is output.

[0074] Furthermore, in step 6, the joint loss function implemented during model training is...

[0075]

[0076] Indicates the total loss. Classification cross-entropy loss, Frame-level binary classification loss, Focus loss; α, β, and γ represent the weight coefficients of the above three losses, which are used to adjust the relative contributions of the classification task, frame-level detection task, and hard sample learning task in joint training.

[0077] The optimizer uses AdamW with an initial learning rate of With cosine annealing learning rate scheduling; batch size 32; on the ICBHI 2017 dataset;

[0078] Data augmentation includes: SpecAugment method, random velocity perturbation, additive white Gaussian noise, and random gain perturbation.

[0079] To achieve the objectives of this invention, a lung function assessment system based on bidirectional cross-attention is disclosed, comprising the following modules:

[0080] Audio acquisition and preprocessing module: Acquires the subject's breathing sounds, then preprocesses the raw audio to obtain standardized audio frames;

[0081] Multi-scale feature extraction module: Performs multi-scale time-frequency feature extraction on standardized audio frames to obtain a time-frequency feature sequence of breath sounds;

[0082] Bidirectional cross-attention module: The time-frequency feature sequence of breath sounds and the learnable latent vectors are fed into the BiXT encoder to perform BiXT bidirectional cross-attention encoding, which yields the temporal position features of breath sounds and the global features of pathological semantics, respectively.

[0083] Feature fusion module; introduces a dual attention submodule, which enhances the spatial attention and channel attention of location information and semantic information respectively, so as to achieve refined feature extraction;

[0084] Multi-task detection and classification output module; including frame-level abnormal event detection, sequence-level breath sound classification, and lung function status assessment;

[0085] The model training implementation module is used to implement bidirectional cross-attention lung function assessment.

[0086] Compared with existing technologies, the significant advancements of this invention are: 1) Computational efficiency: The complexity of the bidirectional cross-attention mechanism is O(T²d / 4), which is 75% lower than that of standard self-attention; 2) Global modeling: By achieving global contextual interaction of signals through bidirectional (forward + backward) cross-attention, it retains richer long-term temporal correlations than local attention methods, improving the recall rate of low-frequency, long-term abnormal sounds, such as wheezing; 3) Dual attention network: By modeling global dependencies and feature correlations from both spatial and channel dimensions, it can fully capture multi-scale features of respiratory sounds, vocal tract response correlations, and long-distance semantic connections, significantly improving the accuracy of respiratory sound pathological feature extraction and lung disease segmentation and recognition; 4) Multi-task output: The joint training of frame-level detection and sequence-level classification enables the model to take into account both event time localization and overall sound pattern recognition, assisting clinicians in providing accurate auscultation timestamps; 5) Clinical interpretability: The attention weight matrix visualization reveals the key respiratory time periods that the model focuses on, enhancing clinicians' trust in AI decision-making.

[0087] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0088] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0089] Figure 1 This is a schematic diagram of the overall system framework of the present invention;

[0090] Figure 2 This is a schematic diagram illustrating the principle of the bidirectional cross-attention mechanism.

[0091] Figure 3 This is a schematic diagram of the multi-scale feature extraction module;

[0092] Figure 4 This is a schematic diagram of a dual attention module;

[0093] Figure 5a A diagram showing the classification results of respiratory sounds in the HFlung dataset;

[0094] Figure 5b The image shows the classification results of breath sounds in the ICHBI2017 dataset;

[0095] Figure 5c A diagram showing the classification results of respiratory sounds in the RespiratoryDatabase@TR dataset;

[0096] Figure 6 The localization effect is shown on a public dataset;

[0097] Figure 7 An interpretability graph on a public dataset. Detailed Implementation

[0098] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0099] A deep learning-based method and system for detecting and classifying abnormal respiratory sounds, which preprocesses, transforms, and augments the acquired respiratory sound signals, can improve the accuracy of feature extraction, the accuracy of classification results, and the precision of segmentation and localization for small sample data.

[0100] The overall system framework of this invention is as follows: Figure 1 As shown, the system sequentially includes audio acquisition and preprocessing, multi-scale feature extraction, feature fusion, lung function status assessment, output category and time period, and feature map visualization.

[0101] For the breath sounds in the HF-Lung-Sound dataset, the breath sounds are first downsampled to 4000Hz, then fast wavelet decomposition is performed to remove redundant noise, followed by normalization. Then, the lung sound time-series data is divided into multiple lung sound time-series segments, and short-time Fourier transform (STFT) is performed on the lung sound segments. Mel filter banks are used to generate multi-scale features such as Mel time-frequency plots, Mel cepstral coefficients (MFCC), and constant Q transform (CQT) as inputs.

[0102] Subsequently, a bidirectional Transformer architecture (BiXT) is used to achieve feature fusion through synchronous bidirectional attention interaction between the pathological semantic latent vector and the temporal position token. This allows for feature fusion with only one computation of the attention matrix, significantly improving efficiency while preserving complete global dependencies and local temporal features. A detection head is then used to simultaneously model the "semantic features" and "temporal position" of breath sounds. The bidirectional cross-attention mechanism is as follows: Figure 2 As shown, the latent vector branch and the respiratory sound frame token branch share the similarity matrix and perform attention updates respectively, thereby synchronously modeling pathological semantic information and temporal location information.

[0103] Step 1: Audio Acquisition and Preprocessing

[0104] (1) Acquisition: A medical electronic stethoscope or electret microphone with a working frequency of 75–2000 Hz was used to acquire monophonic breath sound signals at a sampling rate of 16kHz and in 16-bit PCM format.

[0105] (2) Bandpass filtering: A fourth-order Butterworth bandpass filter (passband 75–2000 Hz) is used to filter out body noise and low-frequency baseline drift.

[0106] (3) Adaptive removal of heart sounds: An adaptive notch filter is used to suppress the heart sound frequency components (mainly 20–150 Hz) of S1 / S2.

[0107] (4) Respiratory cycle segmentation: Based on envelope detection and threshold decision, continuous recordings are segmented into individual inspiratory-expiratory cycle segments, with frame length uniformly filled or truncated to 2 or 4 seconds.

[0108] (5) Normalization: Perform z-score amplitude normalization on each frame of audio.

[0109] Step 2: Multi-scale time-frequency feature extraction

[0110] The following three types of features are extracted from each preprocessed audio frame:

[0111] (a) Mel-Spectrogram;

[0112] (b) Mel-frequency cepstral coefficients (MFCC);

[0113] (c) Constant Q-transform (CQT);

[0114] The three types of features mentioned above are concatenated along the channel dimension and then linearly projected onto a unified dimension d through pointwise convolution (1×1 Conv) to obtain the time-frequency feature sequence of breath sounds. Where: N is the number of time frames (around 400). The feature dimension is D=256. The multi-scale feature extraction module is as follows: Figure 3 As shown, the Mel spectrogram, Mel cepstral coefficients, and constant Q transform features are spliced ​​and linearly projected to form a unified-dimensional time-frequency feature sequence of breath sounds.

[0115] Step 3: BiXT bidirectional cross-attention encoding

[0116] Will With learnable latent vectors The data is fed into the BiXT encoder, and the following process is executed for each layer:

[0117] (1) Initialize the implicit vectors:

[0118]

[0119] For learnable latent vectors, responsible for global semantics ("what sound is it"), M=64: number of latent vectors.

[0120] (2) Linear projection yields reference features and value features

[0121] Latent vector reference features:

[0122] Latent vector value features:

[0123] Breathing sound frame reference features:

[0124] Breath sound frame value characteristics:

[0125] Represents the reference projection matrix (learnable parameter). Represents the projection matrix of values ​​(learnable parameters).

[0126] Among them, X lat This represents the learnable pathological semantic latent vector matrix, where the subscript lat indicates the latent (latent) branch; X tok This represents the temporal token feature matrix of breath sounds, where the subscript tok indicates the token (breath sound frame) branch; R lat and R tok These represent latent vector reference features and breath sound frame reference features, respectively; R stands for the first letter of Reference; V lat and V tok These represent latent vector value features and breath sound frame value features, respectively; V stands for the first letter of Value; W... R W represents the reference projection matrix. V The values ​​represent the projection matrix, both of which are learnable parameters; M represents the number of latent vectors, N represents the number of breath sound time frames, and D represents the feature dimension; case is consistent, and X, R, V, and W all represent matrices.

[0127] (3) Calculate the symmetric similarity matrix

[0128]

[0129] Representing implicit vectors The breathing sound frame similarity matrix, A is the Attention similarity matrix; the subscript lat denotes the latent vector branch, and the subscript tok denotes the breathing sound frame token branch, A tok ,lat represents the reverse breathing sound frame to latent vector similarity matrix; Rlat Represents the latent vector reference feature, R tok This represents the reference feature of the breathing sound frame; the superscript T indicates matrix transpose; √D indicates that the square root of the feature dimension D is used as a scaling factor to suppress excessively large dot product values.

[0130] Scaling factor to prevent values ​​from becoming too large

[0131] Satisfies symmetry:

[0132]

[0133] (4) Two-way attention update

[0134]

[0135]

[0136] Latent vector attention update amount Attention update rate of breathing sound frames

[0137] V tok V represents the frame value characteristics of respiratory sounds. lat This represents the latent vector value feature.

[0138] Softmax: A normalization function that converts similarity into attention weights.

[0139] (5) Residual connectivity and layer normalization

[0140]

[0141]

[0142] Layer normalization for stable training.

[0143] (6) Feedforward Network FFN and Quadratic Normalization

[0144]

[0145]

[0146] 'Represents the updated latent vector output features of the current layer. This represents the output features of the breathing sound frame after the current layer is updated; Feedforward network performs nonlinear feature transformation; This layer outputs latent vectors; Breath sound characteristics of this layer

[0147] Step 4: Feature Fusion Module

[0148] Dual attention modules, such as Figure 4 As shown, it includes a position attention branch and a channel attention branch, which are used to enhance the temporal position dependence of breath sounds and the channel dependence of pathological semantics, respectively.

[0149] After completing step three, the temporal location features of breath sounds were obtained. Global features of pathological semantics Where N is the number of time frames, M is the pathological semantic latent vector, and D is the feature dimension. To further enhance the joint modeling capability of location dependence and semantic channel dependence, this step introduces a dual attention module to enhance spatial attention and channel attention on location information and semantic information respectively, thereby achieving refined feature extraction.

[0150] (1) Feature splitting and input definition

[0151] Temporal location features Feed it into the position attention branch to model the long-range temporal dependency between breathing sound frames, and improve the accuracy of abnormal location;

[0152] Pathological semantic features The input channel attention branch is used to model the global correlation between pathological feature channels and enhance the semantic expression of abnormality types.

[0153] (2) Location attention encoding (temporal location feature enhancement)

[0154] Location features Perform dimensional reshaping to generate a query matrix. Key matrix Value matrix ;

[0155] Calculate the similarity matrix between Q and K, and obtain the temporal positional attention weights after Softmax normalization. W Q W K W V These represent the learnable projection matrices for the query, key, and value, respectively.

[0156] V is weighted and aggregated using attention weights, and then residually connected and layer normalized with the original location features;

[0157] Output enhanced temporal location features .

[0158] (3) Channel attention encoding (pathological semantic feature enhancement)

[0159] semantic features Construct the autocorrelation matrix directly at the channel dimension Dependencies between learning channels;

[0160] Channel attention weights are obtained through Softmax, and semantic features are then weighted and enhanced.

[0161] By incorporating residual connections and layer normalization, the enhanced pathological semantic features are output. .

[0162] (4) Dual attention feature fusion

[0163] Enhanced temporal location features With pathological semantic features Element-wise weighted fusion is performed to obtain joint features that simultaneously possess high-precision location awareness and strong semantic representation capabilities. This provides high-quality feature support for subsequent multi-task output; due to F pos ∈R N×D F cha ∈R M×D Due to the different dimensions, F_cha is first converted to R through pooling, broadcasting, or linear mapping before fusion. N×D And then with F pos Element-wise summation followed by LayerNorm normalization.

[0164] Step 5: Multi-task detection and classification output

[0165] (1) Frame-level abnormal event detection

[0166] Perform a linear transformation and sigmoid function on the features of each frame to output the anomaly probability.

[0167]

[0168] This represents the probability of an anomaly in frame t. The sigmoid function outputs a probability of 0-1, and linear indicates a linear transformation operation. Let represent the feature vector of the t-th frame in the joint features.

[0169] Set an abnormal probability threshold (the default threshold is 0.5, which can be adaptively adjusted according to clinical needs). If the sound exceeds the threshold, it is determined to be an abnormal respiratory sound frame. The timestamp is output, and finally the start time, end time and abnormal probability of all abnormal respiratory sound frames are output, so as to achieve accurate localization of abnormal sounds and provide clinicians with clear key auscultation periods.

[0170] (2) Classification of sequential breath sounds

[0171] Perform global average pooling on all frames, then classify them:

[0172]

[0173] The probability of breath sounds is represented by N, where N is the number of time frames for the joint features. The pathological type with the highest probability is selected as the final classification result for the breath sound sequence, and the classification label and corresponding probability are output. This supports the accurate identification of common abnormal sounds and provides a basis for preliminary screening of lung diseases.

[0174] (3) Assessment of pulmonary function status

[0175] Based on sequence-level breath sound classification results and frame-level abnormal features, combined with pre-defined lung function assessment mapping rules, lung function status is inferred through breath sound features. Specifically, based on the pathological types obtained from the classification, combined with features such as the duration of abnormal breath sounds and the mean abnormal probability, the assessment level of lung ventilation and gas exchange function (normal, mildly abnormal, moderately abnormal, and severely abnormal) is mapped, and the assessment criteria are output (e.g., "Asthma-type breath sounds, abnormal frames account for 35% of the total, indicating mild abnormality in lung ventilation function").

[0176] This assessment can help clinicians quickly grasp the patient's lung function status, and is especially suitable for the elderly, critically ill patients and other people who cannot cooperate with traditional lung function tests, so as to achieve non-invasive and convenient assessment of lung function and meet the needs of long-term clinical monitoring.

[0177] Step Six: Model Training Implementation

[0178] Joint loss function

[0179]

[0180] Indicates the total loss. Classification cross-entropy loss, Frame-level binary classification loss, Focus loss; α, β, and γ represent the weight coefficients of the above three losses, which are used to adjust the relative contributions of the classification task, frame-level detection task, and hard sample learning task in joint training.

[0181] The optimizer uses AdamW with an initial learning rate of It uses cosine annealing for learning rate scheduling; batch size is 32; and it is jointly trained on the ICBHI 2017 dataset (containing 920 breath sound recordings and 6898 respiratory cycles) and a private clinical dataset.

[0182] Data augmentations include: SpecAugment (20 bins of frequency mask, 50 frames of time mask), random velocity perturbation (±10%), additive white Gaussian noise (SNR = 5–30 dB), and random gain perturbation (±6 dB).

[0183] like Figure 5a , Figure 5b and Figure 5c As shown, the present invention exhibits high diagonal concentration in the respiratory sound classification confusion matrix on three public datasets, indicating that the model can accurately distinguish different types of respiratory sounds.

[0184] like Figure 6 As shown, the model can locate the time region where abnormal events occur in the breath sound time-frequency map; for example... Figure 7 As shown, the attention interpretability results can reveal the key respiratory sound segments that the model focuses on.

[0185] This invention proposes a bidirectional cross-attention mechanism, BiXT, which introduces a learnable pathological semantic latent vector and a breath sound temporal token for bidirectional interaction. The two share the same symmetric similarity matrix, and semantic information and temporal position information can be updated synchronously by calculating attention only once, thus overcoming the core defect of existing methods that "semantics" and "position" cannot be modeled synchronously.

[0186] This invention proposes a dual-attention branch module at the feature fusion level. It models global position dependence in the spatial dimension and learns inter-channel correlation in the channel dimension. After weighted fusion, it strengthens the expression of abnormal features and improves the accuracy of breath sound localization and classification. At the same time, based on the breath sound features enhanced by this module, it can further realize the assessment of lung function status. There is no need to fuse additional lung function indicators. The physiological and pathological state of the lungs can be inferred from the breath sounds alone.

[0187] This invention integrates three types of multi-scale time-frequency features—Mel spectrogram, MFCC, and constant Q-transform—at the input end, overcoming the problem of information loss caused by relying on a single feature in existing methods. At the output end, it jointly trains two tasks: frame-level abnormal event detection and sequence-level breath sound classification, enabling the model to simultaneously possess accurate time localization and type recognition capabilities, filling the gap in existing methods that "only classify, not localize." Furthermore, the output end adds lung function-related auxiliary assessment results based on breath sound features, achieving the integration of abnormal lung sound detection, classification, and lung function assessment. Lung function status can be inferred solely through breath sounds, meeting the practical needs of clinical diagnosis and long-term monitoring.

[0188] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0189] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A lung function assessment method based on bidirectional cross-attention, characterized in that, Includes the following steps: Step 1: Collect the subject's breath sounds, and then preprocess the raw audio to obtain standardized audio frames; Step 2: Extract multi-scale time-frequency features from the standardized audio frames to obtain the time-frequency feature sequence of breath sounds; Step 3: Input the time-frequency feature sequence of breath sounds and the learnable latent vector into the BiXT encoder and perform BiXT bidirectional cross-attention encoding to obtain the temporal position features of breath sounds and the global features of pathological semantics, respectively. Step 4: Feature Fusion; Introduce a dual attention module to enhance spatial and channel attention for location and semantic information respectively, thereby achieving refined feature extraction; Step 5: Multi-task detection and classification output, including frame-level abnormal event detection, sequence-level breath sound classification, and lung function status assessment; Step 6: Model training implementation.

2. The lung function assessment method based on bidirectional cross-attention according to claim 1, characterized in that, In step 1, chest or back breath sounds of the subject are collected using a microphone or electronic stethoscope; the raw audio is then subjected to 75–2000Hz bandpass filtering, adaptive removal of heart sounds, respiratory cycle segmentation, and amplitude normalization to obtain standardized audio frames.

3. The lung function assessment method based on bidirectional cross-attention according to claim 1, characterized in that, In step 2, Mel spectrograms, Mel cepstral coefficients (MFCC), and constant Q transform (CQT) features are extracted from the preprocessed audio frames. After being concatenated along the channels, the token sequence is obtained by linear projection. Where N is the number of time frames, For feature dimensions.

4. The lung function assessment method based on bidirectional cross-attention according to claim 1, characterized in that, In step 3, the token sequence With learnable latent vectors The data is fed into the BiXT encoder, and the following process is executed for each layer: Step 3-1: Initialize the implicit vectors; These are learnable latent vectors responsible for global semantics, and M is the number of latent vectors. Step 3-2: Linear projection yields reference features and value features; Latent vector reference features: ; Latent vector value features: ; Breathing sound frame reference features: ; Breath sound frame value characteristics: ; Represents the reference projection matrix. Represents the projection matrix of values; Among them, X lat This represents the learnable pathological semantic latent vector matrix, where the subscript lat indicates the latent branch of the latent vector; X tok This represents the respiratory sound temporal token feature matrix, where the subscript tok indicates the respiratory sound frame token branch; R lat and R tok V represents the latent vector reference feature and the breath sound frame reference feature, respectively; lat and V tok W represents the latent vector value feature and the breath sound frame value feature, respectively; R W represents the reference projection matrix. V The values ​​represent the projection matrix, both of which are learnable parameters; M represents the number of latent vectors, N represents the number of breath sound time frames, and D represents the feature dimension; case is consistent, and X, R, V, and W all represent matrices; Step 3-3: Calculate the symmetric similarity matrix; Representing implicit vectors The breathing sound frame similarity matrix, A is the Attention similarity matrix; the subscript lat denotes the latent vector branch, and the subscript tok denotes the breathing sound frame token branch, A tok ,lat represents the reverse breathing sound frame to latent vector similarity matrix; R lat Represents the latent vector reference feature, R tok This represents the reference feature of the breathing sound frame; the superscript T indicates matrix transpose; √D indicates that the square root of the feature dimension D is used as a scaling factor to suppress excessively large dot product values. This indicates the scaling factor to prevent the value from becoming too large. Satisfies symmetry: Steps 3-4: Two-way attention update; This represents the amount of latent vector attention update. The attention update value for the breathing sound frame; Softmax is the normalization function that converts similarity into attention weights; V tok V represents the frame value characteristics of respiratory sounds. lat Represents latent vector value features; Steps 3-5: Residual connectivity and layer normalization; Layer normalization for stable training; Steps 3-6: Feedforward Network (FFN) and Secondary Normalization; 'Represents the updated latent vector output features of the current layer. This represents the output features of the breathing sound frame after the current layer is updated; Represents a feedforward network and performs nonlinear feature transformation; This represents the latent vector output by this layer. ; This indicates the characteristics of the breath sounds output from this layer. Where N is the number of time frames, M is the pathological semantic latent vector, and D is the feature dimension.

5. The lung function assessment method based on bidirectional cross-attention according to claim 1, characterized in that, Step 4 specifically includes the following steps: Step 4-1: Feature splitting and input definition; Temporal location features Feed it into the position attention branch to model the long-range temporal dependency between breathing sound frames, and improve the accuracy of abnormal location; Pathological semantic features The input channel attention branch is used to model the global correlation between pathological feature channels and enhance the semantic expression of abnormality types. Step 4-2, Location Attention Encoding; Location features Perform dimensional reshaping to generate a query matrix. Key matrix Value matrix ; Calculate the similarity matrix between Q and K, and obtain the temporal positional attention weights after Softmax normalization. W Q W K W V These represent the learnable projection matrices for the query, key, and value, respectively. V is weighted and aggregated using attention weights, and then residually connected and layer normalized with the original location features; Output enhanced temporal location features ; Step 4-3, Channel Attention Encoding; semantic features Construct the autocorrelation matrix directly at the channel dimension Dependencies between learning channels; Channel attention weights are obtained through Softmax, and semantic features are then weighted and enhanced. By incorporating residual connections and layer normalization, the enhanced pathological semantic features are output. ; Step 4-4: Dual attention feature fusion; Enhanced temporal location features With pathological semantic features Element-wise weighted fusion is performed to obtain joint features that simultaneously possess high-precision location awareness and strong semantic representation capabilities. This provides high-quality feature support for subsequent multi-task output; due to F pos ∈R N×D F cha ∈R M×D Due to the different dimensions, F_cha is first converted to R through pooling, broadcasting, or linear mapping before fusion. N×D And then with F pos Element-wise summation followed by LayerNorm normalization.

6. The lung function assessment method based on bidirectional cross-attention according to claim 1, characterized in that, In step 5, the frame-level abnormal event detection steps are as follows: Perform a linear transformation and sigmoid function on the features of each frame to output the anomaly probability. This represents the probability of an anomaly in frame t. The sigmoid function outputs a probability of 0-1, and linear indicates a linear transformation operation. This represents the feature vector of the t-th frame in the joint features; Set an anomaly probability threshold, when If the value exceeds the threshold, it is determined to be an abnormal breathing sound frame. The timestamp is output, and finally the start time, end time and abnormal probability of all abnormal breathing sound frames are output.

7. The lung function assessment method based on bidirectional cross-attention according to claim 1, characterized in that, In step 5, the specific steps for classifying sequential breath sounds are as follows: Perform global average pooling on all frames, then classify them: This represents the probability of breath sounds, where N is the number of time frames for the joint feature; The pathological type with the highest probability is selected as the final classification result of the breath sound sequence, and the classification label and corresponding probability are output to support the accurate identification of common abnormal sounds and provide a basis for the preliminary screening of lung diseases.

8. The lung function assessment method based on bidirectional cross-attention according to claim 1, characterized in that, In step 5, the lung function status assessment steps are as follows: Based on sequence-level breath sound classification results and frame-level abnormal features, combined with preset lung function assessment mapping rules, the lung function status is inferred through breath sound features; based on the pathological types obtained from the classification, combined with the duration of abnormal breath sounds and the mean abnormal probability features, the assessment level of lung ventilation and gas exchange function is mapped, and the assessment basis is output.

9. A lung function assessment method based on bidirectional cross-attention according to claim 1, characterized in that, In step 6, the joint loss function for model training is: Indicates the total loss. Classification cross-entropy loss, Frame-level binary classification loss, Focus loss; α, β, and γ represent the weight coefficients of the above three losses, which are used to adjust the relative contributions of the classification task, frame-level detection task, and hard sample learning task in joint training; The optimizer uses AdamW with an initial learning rate of This is coordinated with cosine annealing learning rate scheduling. Batch size 32; In the ICBHI 2017 dataset; Data augmentation includes: SpecAugment method, random velocity perturbation, additive white Gaussian noise, and random gain perturbation.

10. A lung function assessment system based on bidirectional cross-attention, said system being based on the method of any one of claims 1-9, characterized in that, Includes the following modules: Audio acquisition and preprocessing module: Acquires the subject's breathing sounds, then preprocesses the raw audio to obtain standardized audio frames; Multi-scale feature extraction module: Performs multi-scale time-frequency feature extraction on standardized audio frames to obtain a time-frequency feature sequence of breath sounds; Bidirectional cross-attention module: The time-frequency feature sequence of breath sounds and the learnable latent vectors are fed into the BiXT encoder to perform BiXT bidirectional cross-attention encoding, which yields the temporal position features of breath sounds and the global features of pathological semantics, respectively. Feature fusion module; A dual attention submodule is introduced to enhance spatial and channel attention for location and semantic information respectively, thereby achieving refined feature extraction. Multi-task detection and classification output module; This includes frame-level abnormal event detection, sequence-level breath sound classification, and lung function status assessment; The model training implementation module is used to implement bidirectional cross-attention lung function assessment.

Images