Breathing sound recognition method and system based on multi-modal data fusion

Abstract

The invention discloses a breath sound recognition method and system based on multi-modal data fusion, and relates to the field of breath sound intelligent recognition, firstly, a case text is subjected to structured processing through key entity extraction and natural language recombination, and in combination with audio data enhancement, redundant information is removed, and data quality is improved. Then, a two-channel encoder is used for extracting deep embedded vectors of the text and the audio, and through multi-modal feature splicing and self-encoder joint modeling, deep interaction and feature remodeling between modals are achieved in a hidden layer space; on the basis, a mixed loss function containing reconstruction loss and classification loss is constructed, and model parameters are optimized cooperatively through back propagation. The effectiveness of feature fusion is ensured, the difference between unsupervised feature learning and supervised classification tasks is balanced, and the breath sound recognition precision and robustness of the model in a complex scene are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of intelligent respiratory sound recognition, and more specifically, to a respiratory sound recognition method and system based on multimodal data fusion. Background Technology

[0002] In recent years, with the development of artificial intelligence technology, people have increasingly valued the use of computer technology for the auxiliary diagnosis of respiratory diseases, which has improved diagnostic efficiency to some extent. Traditional auxiliary diagnosis of respiratory diseases mainly relies on the processing and analysis of auscultatory audio signals of breath sounds. However, traditional single-modal methods often exhibit limitations when faced with real-world scenarios where the quality of auscultatory audio data is poor or the amount of data is scarce, especially when processing complex abnormal breath sound audio, making it difficult to fully extract the rich features contained within, thus limiting the accuracy of diagnosis. To overcome the limitations of single audio modalities and improve the robustness of breath sound recognition, combining multimodal data such as case texts with deep learning algorithms has become an important research direction.

[0003] While introducing multimodal data offers a new approach to improving recognition performance, existing breath sound recognition schemes based on multimodal data fusion still have several shortcomings. First, in terms of feature extraction, the original medical record text data often contains a lot of redundant information, and existing processing methods often fail to perform effective structured cleaning and deep semantic mining, making it difficult for the model to capture key semantic features. Second, in the feature fusion stage, existing methods handle the fusion of features from different modalities rather arbitrarily, often simply splicing them together without deep and effective interaction between multimodal data, thus affecting the model's efficiency in utilizing comprehensive information. More importantly, during model training, the unsupervised learning objective aimed at enhancing the interaction of feature information between modalities often conflicts with the supervised learning objective oriented towards classification accuracy. This conflict can lead to oscillations or unstable results in the model training process, ultimately affecting the final analysis results.

[0004] Therefore, how to construct a breathing sound recognition method that can effectively remove text redundancy, achieve deep intermodal interaction, and balance the conflict between different learning objectives is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, according to one aspect of this application, a method for respiratory sound recognition based on multimodal data fusion is provided, comprising: a training phase and an inference phase; the training phase includes: acquiring training case text data and training respiratory sound audio data; performing structuring and augmentation preprocessing on the training case text data and training respiratory sound audio data to obtain structured case text data and data-augmented Mel spectrograms; performing dual-channel feature encoding on the structured case text data and data-augmented Mel spectrograms to obtain text embedding vectors and audio embedding vectors; inputting the text embedding vectors and audio embedding vectors into a multimodal respiratory sound classification model based on the true class probability distribution to obtain a mixture loss function; updating the parameters of the multimodal respiratory sound classification model based on the mixture loss function and utilizing backpropagation of gradient descent to obtain a trained multimodal respiratory sound classification model; The inference phase includes: structuring the acquired test case text and test patient audio to obtain the test structured text and test Mel spectrogram; and inputting the test structured text and test Mel spectrogram into a trained multimodal respiratory sound classification model to obtain the final respiratory sound category.

[0006] According to another aspect of this application, a respiratory sound recognition system based on multimodal data fusion is provided, comprising: a training module and an inference module; the training module comprising: a training data acquisition unit for acquiring training case text data and training respiratory sound audio data; a training data structuring unit for performing structuring and enhancement preprocessing on the training case text data and training respiratory sound audio data to obtain structured case text data and data-enhanced Mel spectrograms; a dual-channel feature encoding unit for performing dual-channel feature encoding on the structured case text data and data-enhanced Mel spectrograms to obtain text embedding vectors and audio embedding vectors; a hybrid loss function generation unit for inputting the text embedding vectors and audio embedding vectors into a multimodal respiratory sound classification model based on the true class probability distribution to obtain a hybrid loss function; and a respiratory sound classification model generation unit for updating the parameters of the multimodal respiratory sound classification model based on the hybrid loss function and utilizing backpropagation of gradient descent to obtain a trained multimodal respiratory sound classification model. The inference module includes: a data structuring unit, used to structure the acquired test case text and test patient audio to obtain the test structured text and the test Mel spectrogram; and a respiratory sound category generation unit, used to input the test structured text and the test Mel spectrogram into a trained multimodal respiratory sound classification model to obtain the final respiratory sound category.

[0007] Compared with existing technologies, this application provides a respiratory sound recognition method and system based on multimodal data fusion, addressing the problems of redundant case text, insufficient modal interaction, and conflicting training objectives in existing technologies. First, the case text is structured through key entity extraction and natural language reconstruction, combined with audio data augmentation to remove redundant information and improve data quality. Then, a dual-channel encoder is used to extract deep embedding vectors from the text and audio respectively. Through multimodal feature concatenation and joint modeling with an autoencoder, deep interaction and feature reshaping between modalities are achieved in the hidden space, effectively capturing potential cross-modal associations. Based on this, a hybrid loss function including reconstruction loss and classification loss is constructed, and model parameters are collaboratively optimized through backpropagation. This ensures the effectiveness of feature fusion and balances the differences between unsupervised feature learning and supervised classification tasks, significantly improving the model's respiratory sound recognition accuracy and robustness in complex scenarios. Attached Figure Description

[0008] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0009] Figure 1 This is a flowchart of a breathing sound recognition method based on multimodal data fusion according to an embodiment of this application.

[0010] Figure 2 This is a schematic diagram of the architecture of a breathing sound recognition method based on multimodal data fusion according to an embodiment of this application.

[0011] Figure 3 The diagram below shows the specific structure of the text encoder in the breathing sound recognition method based on multimodal data fusion according to an embodiment of this application.

[0012] Figure 4 This is a block diagram of the audio encoder in the breathing sound recognition method based on multimodal data fusion according to an embodiment of this application.

[0013] Figure 5 This is a flowchart of step S14 in the breathing sound recognition method based on multimodal data fusion according to an embodiment of this application.

[0014] Figure 6 This is a block diagram of a breathing sound recognition system based on multimodal data fusion according to an embodiment of this application. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] In view of the problems in the prior art, this application proposes a breathing sound recognition method based on multimodal data fusion. Figure 1 This is a flowchart of a breathing sound recognition method based on multimodal data fusion according to an embodiment of this application. Figure 2 This is a schematic diagram of the architecture of a breathing sound recognition method based on multimodal data fusion according to an embodiment of this application. Figure 1 and Figure 2 As shown, the breath sound recognition method based on multimodal data fusion according to an embodiment of this application includes: a training phase S1 and an inference phase S2; the training phase S1 includes: S11, acquiring training case text data and training breath sound audio data; S12, performing structuring and enhancement preprocessing on the training case text data and training breath sound audio data to obtain structured case text data and data-enhanced Mel spectrograms; S13, performing dual-channel feature encoding on the structured case text data and data-enhanced Mel spectrograms to obtain text embedding vectors and audio embedding vectors; S14, based on real class... The probability distribution is determined by inputting the text embedding vector and audio embedding vector into the multimodal respiratory sound classification model to obtain a mixture loss function; S15, the parameters of the multimodal respiratory sound classification model are updated based on the mixture loss function and using backpropagation of gradient descent to obtain a trained multimodal respiratory sound classification model; the inference stage S2 includes: S21, structuring the acquired test case text and test patient audio to obtain test structured text and test Mel spectrogram; S22, inputting the test structured text and test Mel spectrogram into the trained multimodal respiratory sound classification model to obtain the final respiratory sound category.

[0017] It is worth mentioning that the breath sound recognition method provided in this application is logically divided into a training phase and an inference phase, with the training phase being the prerequisite and core foundation for building a high-precision recognition model. Due to the non-stationarity of breath sound audio signals and the semantic complexity of case texts, there is a highly complex nonlinear mapping relationship between the two and the final pathological category, making accurate recognition difficult to achieve through traditional rule pre-setting. Therefore, a training phase is needed to drive the deep learning model to autonomously learn the latent features and modal correlation patterns in the data. This phase aims to utilize a large amount of labeled multimodal data to calculate the prediction results through forward propagation, and to evaluate the current error of the model using a hybrid loss function. Then, the backpropagation algorithm of gradient descent is used to iteratively update the weights and bias parameters within the model. Through this process, the model can gradually converge to the optimal state while balancing unsupervised feature interactions and supervised classification tasks, ultimately generating a well-trained multimodal breath sound classification model with strong generalization ability, providing an algorithmic carrier for subsequent accurate inference of the test patients. The training phase is described in detail below.

[0018] In step S11, training case text data and training respiratory sound audio data are acquired. It should be understood that in the field of intelligent medical diagnosis, the complexity of respiratory diseases makes it difficult to achieve ideal diagnostic accuracy by relying solely on data from a single modality. While traditional auscultation audio contains pathological features, it is highly susceptible to environmental noise interference and fails to reflect individual physiological differences in patients; while case text data provides crucial background information, its unstructured form makes it difficult for computers to directly utilize. To construct a highly robust model capable of simultaneously understanding acoustic patterns and semantic context, it is first necessary to acquire comprehensive and complementary multimodal information from the source. This step, as the cornerstone of the entire method, aims to establish a raw dataset containing rich acoustic details and precise clinical background, providing the necessary material basis for subsequent structured cleaning, feature encoding, and deep modal fusion, thereby ensuring that the model can learn potential cross-modal correlation features during the training phase.

[0019] One possible implementation of step S11 is as follows: First, the source and composition of the data need to be clearly defined during the implementation process. Training data is acquired based on ethically reviewed clinical databases or publicly available breath sound standard datasets (such as the ICBHI dataset). In this stage, data acquisition is divided into two parallel channels: an audio data acquisition channel and a text data acquisition channel. For training breath sound audio data, this refers to the raw digital signals collected at specific locations on the patient's chest wall using an electronic stethoscope or high-fidelity recording device. This audio data is not subjected to complex filtering during acquisition, preserving its original acoustic characteristics, including any possible ambient background noise, heartbeat sounds, and skin friction sounds. From a physical perspective, this audio data exists in the form of a one-dimensional time series, stored in WAV or MP3 format. For example, the sampling rate is set to 44.1kHz or 4kHz, the bit depth to 16 bits, and the duration of each audio sample is set to between 10 and 20 seconds. For example, when a specific audio sample is acquired… It manifests as a series of amplitude values ​​that vary over time. ,in This represents the total number of sampling points. The audio data includes not only normal breath sounds (such as bronchial breath sounds and vesicular breath sounds), but also pathological abnormal breath sounds, specifically covering wheezes (with continuous high-frequency characteristics), crackles (with discontinuous popping characteristics), and complex sounds that combine the two.

[0020] Meanwhile, acquiring training case text data is another core step in this process. This training case text data comes from the patient's electronic medical record (EHR) or clinical record sheet, which corresponds one-to-one with the aforementioned audio samples. Initially, this data is unstructured raw records, primarily covering four key dimensions of information: patient gender, age, respiratory system condition, and auscultation site. This text information exists in the form of natural language strings. For example, a specific text sample might be recorded as: Male, 68 years old, suffering from chronic obstructive pulmonary disease, auscultation site in the posterior left lower lung. Here, gender provides context for physiological differences; age reflects the physiological changes in the respiratory system with aging, such as decreased lung elastic recoil; respiratory system condition describes past medical history or current diagnosis, such as healthy, upper respiratory tract infection, COPD, bronchiectasis, etc.; and the auscultation site indicates the anatomical location of sound acquisition, such as the trachea, anterior chest, back, lateral chest, etc., which has a significant impact on sound conduction characteristics. During implementation, these text fields are extracted using database query languages ​​(such as SQL) or data interfaces, and stored in JSON or CSV format in association with the corresponding audio file IDs to form the original text-audio pairs.

[0021] Based on this, to meet the needs of supervised learning, the corresponding true class probability distribution needs to be obtained simultaneously with the data acquisition. Each audio-text data pair is associated with a specific diagnostic label. According to the technical solution, breath sounds are mainly divided into four categories: normal breath sounds, wheezing sounds, popping sounds, and a mixture of wheezing and popping sounds. The true class label is represented using one-hot encoding. For example, if the true class of the current sample is popping sound, and the category index order is [normal, wheezing, popping, mixed], then the true class probability distribution of this sample is... This is represented as a vector [0,0,1,0]. This probability distribution will serve as the benchmark for calculating the loss function in subsequent training phases.

[0022] In addition, the data acquisition phase also involves preliminary data screening and organization. To ensure the effectiveness of training, invalid samples with sampling times that are too short (less than 5 seconds) or with missing signals need to be removed. The final training dataset D consists of M samples, denoted as a set. .in, The original case text data representing the i-th sample includes the aforementioned descriptions such as age and gender. Represents the original respiratory sound audio data of the i-th sample; This represents the corresponding true category label. For example, the 100th sample in the dataset might contain: an audio file "100_Audio.wav" containing a 20-second recording with intermittent popping sounds, the corresponding text record "Male, 68 years old, COPD, lower left," and the label popping sound.

[0023] In step S12, the training case text data and training respiratory sound audio data undergo structuring and enhancement preprocessing to obtain structured case text data and data-enhanced Mel spectrograms. Correspondingly, the raw data obtained in step S11 often contains a large amount of noise and non-standardized information. Case text data suffers from inconsistent writing formats and excessive redundant descriptions; directly inputting it into the model can lead to distraction and difficulty in capturing key pathological features. The raw respiratory sound audio data is a one-dimensional time-series signal, containing not only environmental background noise and heart sound interference, but also pathological features primarily manifested in the frequency domain energy distribution. Traditional time-dimensional analysis is insufficient to effectively characterize the spectral characteristics of wheezing or popping sounds. Therefore, to enable the deep learning model to efficiently extract features from heterogeneous data, the raw data needs to be cleaned and its dimensions transformed. This step aims to transform the messy text into a semantically clear standardized sequence through structuring processing, and simultaneously transform the one-dimensional audio into two-dimensional image features containing rich time-frequency information through signal processing techniques. This provides high-quality, high signal-to-noise ratio, and semantically aligned input data for subsequent dual-channel feature encoding, ensuring that the multimodal model can focus on the most diagnostically valuable features.

[0024] In one possible implementation of step S12, the training case text data and training respiratory sound audio data are subjected to structuring and enhancement preprocessing to obtain structured case text data and data-enhanced Mel spectrograms, including: step S121, performing key entity extraction and natural language reconstruction on the training case text data to obtain coherent structured natural language text; step S122, performing word segmentation on the coherent structured natural language text to obtain structured case text data; step S123, performing bandpass filtering and short-time Fourier transform on the training respiratory sound audio data to obtain Mel spectrograms; and step S124, performing data enhancement on the Mel spectrograms to obtain data-enhanced Mel spectrograms.

[0025] The implementation of step S12 above is as follows: In the text processing branch, step S121 is executed first. The original unstructured case text obtained in S11, for example, a record may be displayed as: Patient condition: Male, 68 years old, chief complaint: COPD, abnormal sound heard on auscultation of the posterior left lower lung. Since the original record contains redundant words such as patient condition, chief complaint, and abnormal sound, which have low contribution to the classification task, and different doctors have different recording habits, it is necessary to clean it through key entity extraction technology. In specific implementation, a pre-built medical entity dictionary and a rule-based text parsing algorithm are used to scan and match the text, accurately filter and retain key information in four core dimensions: Sex, Age, Respiratory system condition, and Location. For example, for the above sample, the entity extracted by the algorithm is {Sex: "Male", Age: "68", Condition: "COPD", Location: "posterior left lower lung"}. In order to solve the problem of different expressions of the same concept, such as COPD, a standardization mapping is also performed at this stage to unify the terminology. After the extraction is completed, the natural language reconstruction stage begins. To ensure the retained key information has a coherent grammatical structure, enabling the subsequent Transformer model to capture contextual dependencies, this step uses a pre-defined natural language template to concatenate discrete entities into fluent sentences. The pre-defined template can be set as: the patient is [Age] years old [Sex], suffers from [Condition], and the auscultation location is [Location]. Filling the extracted entities into the template generates the coherent structured natural language text: The patient is a 68-year-old male with COPD, and the auscultation location is the posterior left lower lung. This process effectively removes noise from the original text while preserving semantic information crucial for diagnosis, ensuring a highly consistent syntactic structure across all training samples.

[0026] Subsequently, step S122 is executed to convert human-readable natural language into a computer-processable numerical sequence. During implementation, a mature natural language processing (NLP) tokenizer is invoked, such as a pre-trained tokenizer based on the BERT or RoBERTa model. This tokenizer receives the coherent text output from step S121 and segments it into independent tokens according to a pre-defined vocabulary. The vocabulary is essentially a collection containing a large number of predefined characters, words, or sub-word units. Its construction is based on the statistical features of a large-scale corpus or statistically established according to the total number of different words appearing in all training case text data in this application, thus covering various medical terms and common expressions. For Chinese text, the tokenizer will segment "patient 68 years old" into ['patient', 'for', '68', 'years old']; for English text, a sub-word segmentation strategy may be used, such as breaking down complex medical terms into smaller semantic units. After segmentation, the system adds special control characters, such as [CLS] at the beginning of the sentence and [SEP] at the end of the sentence, to mark the beginning and end of the sequence. Ultimately, to transform the text into a machine-readable vector sequence, the word segmentation sequence is traversed, and a unique integer index (ID) corresponding to each word is retrieved from the vocabulary. Rare words not included in the vocabulary are mapped to specific unknown tags. Through this lookup and mapping mechanism, the original text is transformed into a fixed-length integer vector; for example, the text above might be converted into the vector sequence [101,2345,1023,5678,...,102]. This structured case text data not only preserves the semantics of key entities but also implicitly reveals the logical relationships between entities through word order, preparing the data for subsequent embedding operations by the text encoder.

[0027] Meanwhile, in the audio processing branch, step S123 is executed. First, bandpass filtering is performed. Since the original breath sound signal often contains heart sounds (usually below 100Hz), muscle friction sounds, and environmental power frequency interference or high-frequency electronic noise (usually above 2000Hz), these noises can mask the characteristics of lung sounds. Therefore, a Butterworth filter is used for preprocessing the signal. In practice, the passband frequency range of the filter is set to 100Hz to 2000Hz, a classic range covering the energy distribution of most normal breath sounds, wheezing (high frequency), and popping sounds (transient). The Butterworth filter is chosen because it has the flattest frequency response characteristics within its passband, minimizing distortion of the amplitude-frequency characteristics of the effective signal. The filtered audio signal... The signal-to-noise ratio was significantly improved, primarily preserving the sound components generated by lung ventilation. Next, a short-time Fourier transform (STFT) was performed on the denoised audio signal. Since breath sounds are non-stationary signals, their frequency components change over time; for example, the pitch of wheezing may rise and fall with exhalation. A simple Fourier transform cannot reflect this time-varying characteristic. Therefore, STFT is needed to convert the one-dimensional time-domain signal into a time-frequency domain signal. In practice, firstly, a signal with a duration of... The audio signal is divided into frames, with frame length and frame shift set. For example, for a signal with a sampling rate of 4kHz, the frame length can be set to 25ms (corresponding to L sampling points), and the frame shift to 10ms. To reduce spectral leakage caused by framing, each frame of the signal is multiplied by a Hamming window function. Then, the discrete Fourier transform of each frame of signal is calculated, and the calculation formula is as follows: In this formula, This represents the complex spectral value of the nth time frame at the kth frequency point. This represents the value of the m-th sample point of the input audio signal within the current frame. This is a Hamming window function used to smoothly truncate edges. The number of points in the Fourier transform is usually a power of 2, such as 1024. It is the rotation factor, representing the unit circular base in the complex plane. Indicates the time index within the window. This is the imaginary unit. This formula converts the time-domain amplitude of each frame into complex coefficients in the frequency domain. After obtaining the complex spectrum, the power spectrum is calculated. Since the human ear and deep learning models primarily focus on the energy distribution of sound rather than phase information, it is necessary to calculate the square of the modulus of the spectrum. Furthermore, considering the extremely wide range of sound intensity perception, using a logarithmic scale can better compress the dynamic range. The calculation formula is: in, This is the logarithmic power spectrum, which reflects the energy intensity of audio at specific times and frequencies. These are the Fourier coefficients obtained above. However, the linear frequency scale cannot accurately simulate the human ear's frequency perception characteristics; the human ear is more sensitive to changes in low frequencies than high frequencies. To better align with auditory mechanisms and reduce feature dimensionality, a Mel filter bank mapping is needed. First, a Mel filter bank is constructed, containing H triangular filters, such as 128, whose center frequencies are uniformly distributed on the Mel scale. The conversion relationship between linear frequency f and Mel frequency s is defined by the following formula: , This represents the center frequency of the h-th Mel filter. The formula represents the Mel scale value, indicating that the Mel frequency has a logarithmic relationship with the linear frequency. In the low-frequency range, the Mel filters are denser and have narrower bandwidths; in the high-frequency range, the filters are sparser and have wider bandwidths. This allows the model to perform multi-resolution feature extraction of the low-frequency fundamental breath sound and the high-frequency overtones of wheezing. Finally, the operation of generating the Mel spectrogram is performed. The logarithmic power spectrum obtained above is then used... With Mel filter bank Matrix multiplication is performed, which involves weighted summation of the energy in the linear frequency band and mapping it to the Mel frequency band. The calculation formula is: ,in, This represents the energy value output by the Mel filter in the h-th time frame of the n-th time frame. The frequency response weights of the h-th Mel filter at frequency k are obtained by constructing a filter bank based on the Mel frequency scale. The calculation results of all time frames are stacked in chronological order to finally generate a filter with dimension [missing value]. A two-dimensional matrix, known as a Mel spectrogram, is used to visually represent the energy distribution of respiratory sounds on a Mel frequency scale over time: the horizontal axis represents time, the vertical axis represents Mel frequency, and the brightness or color intensity of pixels represents energy intensity. For example, wheezing sounds appear as continuous horizontal bright lines on the spectrogram, while popping sounds appear as short vertical stripes. This two-dimensional feature map not only significantly compresses the dimensionality of the original data but also highlights the pathologically significant time-frequency structure.

[0028] After generating the Mel spectrogram in the preceding step S123, the processing flow enters the crucial data augmentation stage, namely step S124. This step aims to generate training samples with greater diversity through transformations in the feature space and time dimension. One possible implementation of step S14 involves data augmentation of the Mel spectrogram to obtain a data-augmented Mel spectrogram, including: step S124-1, performing a Mixup enhancement process based on linear interpolation on the Mel spectrogram; and / or step S124-2, performing a Time-Warping enhancement process based on the time dimension on the Mel spectrogram.

[0029] First, step S124-1 is implemented. The core logic of this step is no longer simply flipping or cropping a single image, but rather breaking the absolute boundary between training samples and their labels in traditional classification tasks through a sample pairing and linear superposition strategy, constructing a continuous feature transition space. The implementation process begins with the sample pairing and coefficient generation stage. In each training batch, the algorithm randomly selects two different samples from the Mel-spectrum set output in step S123 as input objects. Let these two samples be... and ,in The Mel spectrogram matrix representing the i-th sample has dimensions of . For example, (1024, 128); The Mel spectrogram matrix representing the j-th sample, with dimensions equal to... Completely identical. These two samples could come from the same pathological category or completely different categories, such as one being normal breath sounds and the other wheezing. Meanwhile, the algorithm uses a Beta distribution to generate a normalized weighted coefficient. The Beta distribution consists of two shape parameters. and Control, setting = (For example, a value of 0.2 or 0.4), so that the generated The values ​​are mainly distributed around 0 or 1, but cover the entire [0,1] interval. Specifically, during generation, a probability density function is constructed based on preset shape parameters. Then, each time samples are paired, a real value falling within the closed interval of 0 to 1 is independently sampled from this distribution using a pseudo-random number generation algorithm. This is a scalar value used to quantify the weighting of the two samples in the mixed new sample. After obtaining the input sample pairs and weighting coefficients, the process proceeds to the feature space mixing stage. This stage performs pixel-level linear interpolation, which involves weighted summation of the corresponding elements in the two two-dimensional image matrices at their spatial locations. The feature mixing calculation formula is as follows: In this formula, This represents the generated new sample with mixed features, which is also a dimensional sample. The matrix. As a mixture factor, it determines the new sample Which original sample is it visually and numerically closer to? For example, =0.7, It is a thyrso audio spectrum characterized by continuous bright lines at high frequencies. If it is a normal respiratory audio spectrum showing a stable low-frequency distribution, then the generated... While preserving the main morphological features of wheezing, a background texture of normal breathing sounds is overlaid, with the overall energy intensity being a linear combination of the two. This processing method simulates complex acoustic scenarios where multiple pathological features may coexist or be subject to background noise interference, forcing the model to identify key features beneath blurred boundaries. Next, a label space mixing stage is performed. Because the input features have already been mixed, the original discrete category labels can no longer accurately describe the new samples. Therefore, the tag needs to undergo synchronous linear interpolation due to its attributes. For example... and These are samples and The corresponding original true category labels exist in the form of one-hot encoded vectors. For example, if the category is defined as [normal, wheezing, popping, mixed], and If it is a wheezing sound, then =[0,1,0,0] If it is a normal breath sound, then =[1,0,0,0]. The formula for calculating mixed tags is as follows: In this formula, This indicates the newly generated soft tag. Continuing from the above... =0.7, the calculation result is =0.7×[0,1,0,0]+0.3×[1,0,0,0]=[0.3,0.7,0,0]. This new label vector is no longer a binary distribution of either 0 or 1, but rather represents a 30% probability that the sample belongs to normal breathing sounds and a 70% probability that it belongs to wheezing sounds. This process transforms the classification task from a simple logical judgment to probability distribution fitting, greatly reducing the model's sensitivity to mislabeled data and smoothing the inter-class decision boundaries. After completing the above calculations, the data output stage will generate new samples with mixed features. and hybrid new labels Packaged as valid input for subsequent training steps.

[0030] After processing in S124-1, or in parallel, step S124-2 is performed. The input for this step can be the original generated Mel spectrogram or a sample after Mixup. Given that breath sounds are a physiological signal, and a patient's respiratory rate is not constant but influenced by age, lung capacity, and pathological condition, the duration of the same breath sound event (such as a single inhalation) may vary. Time-Warping is designed to simulate this elastic variation over time. The implementation process begins with interval selection. On the time axis of the input Mel spectrogram, i.e., the horizontal axis, the algorithm randomly generates a time point as the starting anchor point. And set a fixed time interval length. The fixed time interval length is determined by randomly selecting a duration of [duration] on the time dimension of the respiratory sound Mel-ray spectrogram. The interval is determined by the range. For example, for a spectrogram with a total duration of 10 seconds, corresponding to 1000 frames, the interval is set. The time interval is 100 frames, or approximately 1 second. If the starting point is randomly selected as frame 300, the selected operation range is from frame 300 to frame 400. This range covers a local segment of the breath sound signal. The algorithm then proceeds to the translation amount generation stage. To determine the direction and degree of time warp, the algorithm generates a random time translation amount. The translation amount follows a specific uniform distribution, and its distribution formula is as follows: ,in, Represents a uniform distribution function. The lower realm The preset maximum translation distance, for example, set to Half of that, i.e., 50 frames. (Generated) This is an integer value representing the number of frames to be shifted on the timeline. Although the formula uses a positive interval, in practical applications it includes directionality, meaning it can shift left or right. Next, the core time warp operation is performed. The physical meaning of this operation is to non-linearly stretch or compress the spectral characteristics within the selected interval. In practice, the selected interval is maintained... Without changing the boundary points, shift the center point of the interval or a specific anchor point along the time axis by a distance. If the shift is to the right, the time segment to the left of the anchor point is stretched (simulating slower breathing) and the time segment to the right is compressed (simulating faster breathing); the reverse is also true. At the matrix level, this involves resampling the spectrogram pixels. For stretched regions, since the original pixels are insufficient to fill the enlarged space, the algorithm uses bilinear or bicubic interpolation to calculate the energy value of the newly added pixels, ensuring the continuity of the spectral texture; for compressed regions, downsampling or pixel merging is performed. For example, in the selected interval of frames 300 to 400, if the feature of frame 350 is shifted to the right to frame 370, the data between frames 300 and 370 will be sparsified (stretched), while the data between frames 370 and 400 will be denser (compressed). This processing does not change the total number of frames in the spectrogram (the total duration remains unchanged), but rather changes the distribution density of local features on the time axis. Finally, the spectrogram after the above distortion processing is labeled as a data-enhanced Mel spectrogram in the final output stage. Through this series of complex mathematical transformations and signal processing, the original single breath sound sample was expanded into a rich dataset containing mixed acoustic features, soft probability labels, and multiple time rate variants. This enhanced data not only preserves the original pathological semantics but also introduces physically consistent random perturbations, providing highly challenging yet informative training material for the subsequent dual-channel feature encoder.

[0031] In step S13, dual-channel feature encoding is performed on the structured case text data and the data-enhanced Mel spectrogram to obtain text embedding vectors and audio embedding vectors. It is understandable that after the structuring and enhancement preprocessing of the original multimodal data, the resulting structured case text data and data-enhanced Mel spectrogram, while improved in information purity and diversity, still exhibit significant differences in data form and internal structure: the former is a sequence of lexical units carrying discrete semantic information, while the latter is a two-dimensional image matrix representing continuous time-frequency energy distribution. To achieve efficient fusion of the two modalities for joint diagnosis, they need to be transformed into a unified and comparable feature space. Therefore, a dual-channel feature encoding scheme is adopted, aiming to utilize state-of-the-art deep learning models in their respective fields to extract deep, abstract, and dimensionally consistent feature representations, i.e., embedding vectors, from these two heterogeneous data sets, respectively.

[0032] In one possible implementation of step S13, dual-channel feature encoding is performed on the structured case text data and the data-enhanced Mel spectrogram to obtain a text embedding vector and an audio embedding vector, including: step S131, inputting the structured case text data into a text encoder to obtain a text embedding vector; step S132, inputting the data-enhanced Mel spectrogram into an audio encoder to obtain an audio embedding vector. Specifically, the text encoder includes a Word2vec module and a text encoding module composed of multiple Transformer modules; the audio encoder is an audio spectrum Transformer network.

[0033] The implementation of step S13 above is as follows: First, step S131 is executed, in which the structured case text data is input into the text encoder to obtain the text embedding vector. Figure 3 This is a structural block diagram of the text encoder in the breathing sound recognition method based on multimodal data fusion according to an embodiment of this application. Figure 3 As shown, the input for this step originates from the structured case text sequence output in step S12. Recalling the example from the previous step, the current input sample is a sequence of integer indices after word segmentation. [101,2345,1023,5678,...,102], where 101 represents the sentence beginning marker [CLS], 2345 represents the patient, 1023 represents 68 years old, etc. This sequence is first fed into the first level of the text encoder, the Word2vec module. The main function of the Word2vec module is to map discrete word indices into dense real-valued vectors. The core of this module contains a pre-trained or randomly initialized word embedding matrix. ,in This represents the size of the vocabulary list, such as setting it to 5000 commonly used medical and general terms. The dimension representing the word vector, for example, set to 768 dimensions. Let the set be the real number set. In practice, for each integer index in the input sequence... The module retrieves data from the matrix via table lookup operations. Take the corresponding first one from the middle Row vector. For example, index 2345 is replaced with a 768-dimensional vector [0.12, -0.56, ..., 0.09]. After this step, the length is... The integer sequence is converted to dimension The word embedding matrix is ​​initially captured, but it does not yet include contextual information. This matrix is ​​then fed into a text encoding module consisting of multiple Transformer modules. This module is composed of N, such as 12, identical Transformer encoder layers stacked together. Each Transformer module mainly contains two sub-layers: a multi-head self-attention mechanism and a feedforward neural network. Before the data enters the first Transformer module, positional encoding is superimposed to give the model the ability to recognize word order. After entering the Transformer layer, multi-head self-attention computation is performed first. For the input feature matrix, i.e., the output or initial word embedding of the previous layer, the model linearly maps it into a query vector, a key vector, and a value vector using three learnable weight matrices. This process allows the model to dynamically adjust the representation of each word according to the context. For example, when the model processes the word "abnormal sound," it can focus on the location description of the relatively distant lower left lung through the attention mechanism, thus incorporating location information into the features of the abnormal sound. The multi-head mechanism means that the model performs the above calculations multiple times in parallel, each focusing on a different semantic subspace (e.g., some heads focus on syntactic structure, while others focus on entity reference). Finally, the outputs of all heads are concatenated and linearly transformed to obtain the output of the current layer. After the attention layer, the data flows through layer normalization and residual connections before entering the feedforward neural network. The FFN consists of two fully connected layers and a non-linear activation function (such as ReLU or GELU) to further extract deep non-linear features. After 12 layers of this iterative processing, the originally static word embeddings are transformed into a deep semantic feature matrix containing rich contextual dependencies, while its dimension remains [missing information]. To obtain the final one-dimensional text embedding vector, the vector at the first position in the output sequence of the last Transformer module is extracted, corresponding to the classification terminator of the [CLS] tag. This terminator is designed during training to aggregate global semantic information of the entire sentence. For example, the extracted [CLS] vector is... Finally, the vector is fed into a fully connected layer to map its dimensions to the target dimension (e.g., 128 dimensions), resulting in the final text embedding vector. This vector encapsulates global deep semantic information such as 68-year-old male, COPD, and posterior left lower lung.

[0034] In parallel with text encoding, step S132 is performed. The input to this step is the Mel spectrogram output from step S12, enhanced by Mixup or Time-Warping, with dimensions of... For example, (1024, 128). Figure 4 This is a block diagram showing the specific structure of the audio encoder in the breathing sound recognition method based on multimodal data fusion according to an embodiment of this application. Figure 4 As shown, the audio encoder uses an Audio Spectrogram Transformer (AST) network. Unlike traditional convolutional neural networks, AST directly applies the Transformer architecture to image patches. First, image patching is performed. The 1024×128 spectrogram is divided into non-overlapping rectangular blocks of size 16×16. 128 / 16=8 blocks are cut in the frequency dimension, and 1024 / 16=64 blocks are cut in the time dimension, resulting in a total of 8×64=512 image patches. Next, each 16×16 two-dimensional image patch is flattened and mapped to a one-dimensional vector through a linear projection layer, with the dimension set to 1. For example, a 768-dimensional spectrogram. At this point, the input spectrogram is transformed into a vector sequence of length 512. To preserve the time-frequency positional information of the spectrogram, a learnable positional embedding is added to the vector at each position. This step is crucial because it tells the model which blocks belong to the low-frequency region, which belong to the high-frequency region, and their order on the time axis. The processed sequence is input into the core Transformer encoder stack layer of the AST. Its architecture is almost identical to the Transformer in the text encoder, also including a multi-head self-attention mechanism and a feedforward neural network. In the audio context, the self-attention mechanism plays a role in capturing long-distance time-frequency dependencies. For example, for a howling sound sample, its features are represented by a horizontal line that persists over time in the high-frequency band (corresponding to the top of the spectrogram). The Transformer's attention mechanism allows image blocks belonging to different time points of this horizontal line to pay attention to each other, thereby recognizing this continuous acoustic pattern; at the same time, it can also distinguish the vertical texture features corresponding to transient popping sounds. Through layer-by-layer propagation, the model gradually abstracts high-level acoustic event features from local spectral textures. After processing through multiple Transformer layers, the output matrix of the final layer is obtained, with dimensions (512, 768), representing the depth feature representation of 512 image patches. Instead of directly extracting the [CLS] marker, a special pooling operation is performed: average pooling along the frequency dimension. Since the original image patches retain their spatial structure logic when converted into a sequence, the output matrix can logically be restored to a (64, 8, 768) structure, i.e., 64 time steps and 8 frequency bands. Average pooling along the frequency dimension means averaging the feature vectors of the 8 frequency patches corresponding to each time step, thereby eliminating specific positional differences in the frequency dimension and preserving the global spectral overview of that time step. This operation compresses the data dimension to (64, 768), corresponding to a feature sequence of 64 time frames. Subsequently, these 64 vectors are subjected to global average pooling again or specific aggregated features are extracted to obtain a global feature vector that can represent the entire audio segment. Finally, this global feature vector is input into a fully connected layer, mapping its dimension from 768 to the same dimension as the text embedding vector, i.e., 128 dimensions. The output is an audio embedding vector. This vector not only encodes the physical characteristics of the breath sound, such as timbre, pitch, and duration, but also, through data augmentation and deep network learning, it is robust to noise and variations.

[0035] In step S14, based on the true class probability distribution, the text embedding vector and audio embedding vector are input into the multimodal respiratory sound classification model to obtain a hybrid loss function. It should be understood that after the processing in the preceding steps, the heterogeneous case text and respiratory sound audio have been mapped into independent single-modal high-dimensional feature vectors. However, the diagnosis of respiratory diseases often relies on the complex nonlinear relationship between these two; for example, there is a strong pathological correspondence between the acoustic features of wheezing and the textual description of chronic obstructive pulmonary disease. Simple feature juxtaposition cannot reveal this deep-seated underlying logic. To enable computers to comprehensively consider multiple pieces of information like clinicians, this application constructs a unified fusion architecture that maps features of different dimensions to the same latent space for interaction, and supervises this interaction process through a mechanism to ensure that the fused features retain the key structure of the original information while maximizing their service to the final classification task.

[0036] Figure 5 This is a flowchart of step S14 in the breath sound recognition method based on multimodal data fusion according to an embodiment of this application. Figure 5 As shown, in one possible implementation of step S14, based on the true class probability distribution, the text embedding vector and the audio embedding vector are input into a multimodal breathing sound classification model to obtain a hybrid loss function, including: step S141, performing multimodal feature concatenation and autoencoder joint modeling on the text embedding vector and the audio embedding vector to obtain connection features, hidden layer features and reconstructed connection features; step S142, based on the true class probability distribution, performing classification inference and parallel calculation of dual losses on the connection features, hidden layer features and reconstructed connection features to obtain reconstruction loss and classification loss; step S143, constructing the hybrid loss function based on the reconstruction loss and classification loss.

[0037] The implementation of step S14 above is as follows: Step S141 no longer treats text and audio as separate entities, but fuses them into one through mathematical transformation. First, a feature concatenation operation is performed. The input of this operation is directly from the output of step S13, namely a one-dimensional text embedding vector and a one-dimensional audio embedding vector. In the previous embodiment, the dimensions of these two vectors were both set to 128. During implementation, the feature fusion module receives these two vectors and performs a concatenation operation along the feature dimensions. This means that the numerical sequence of the audio vector is directly appended to the end of the text vector, thereby forming a flattened one-dimensional connection feature containing information from both modalities. In this specific example, the dimension of the generated connection feature is 128 + 128 = 256. This 256-dimensional vector physically carries both the patient's clinical background and the current physiological auscultation signal. Subsequently, the core encoding process begins. This process relies on the autoencoder module in the multimodal respiratory sound classification model. This autoencoder is embedded in the overall architecture as a feature fusion network. Its architecture includes an encoder part and a decoder part. During the encoding phase, the generated 256-dimensional connected features are input into the encoder. The encoder is essentially a fully connected layer, or dense layer, containing a set of learnable weight matrices and bias vectors. To force the model to uncover potential commonalities between text and audio, such as discovering the correlation between COPD text and reduced low-frequency breath sounds in audio features, the encoder is designed to perform dimensionality reduction mapping. The hidden layer features are set to, for example, 64 dimensions. At this point, the shape of the weight matrix is ​​initialized to (64, 256), using... The multiplication form, if it is This results in a value of 256×64. Here, the formula logic is followed, and the bias vector has a shape of 64. These parameters are generated as random small values ​​conforming to a normal distribution using the Xavier or He initialization method during the initial training phase and are continuously updated during subsequent training. The specific encoding calculation follows the formula: In this formula, The 256-dimensional connection features representing the input. The weight matrix represents the encoder, which determines how the 256 features of the input are weighted and combined. This is a bias term used to adjust the activation threshold. (Operation) The process involves matrix-vector multiplication, compressing and projecting 256-dimensional information into a 64-dimensional space. This process forces the model to discard redundant information (such as the influence of function words in text or background noise in audio), retaining only the features most essential to describing the patient's condition. Representing a nonlinear activation function, the Sigmoid function is chosen in this embodiment. It maps any real value after linear transformation to the interval (0,1). After this series of operations, the output is the hidden layer feature. This 64-dimensional vector is a flattened multimodal fusion feature. It no longer distinguishes between text and audio, but becomes a shared latent representation after their interaction. For example, a value in the hidden layer feature may be simultaneously influenced by two factors: an age greater than 60 and spectral energy concentrated in low frequencies, thus characterizing a specific aging respiratory feature. Finally, the decoding process is performed. To verify whether the hidden layer feature truly and completely preserves the key information of the original data, it is necessary to attempt to restore it. This is achieved through the decoder part of the autoencoder. The decoder receives the 64-dimensional hidden layer feature and maps it back to the original 256-dimensional space through another fully connected layer. In this application, to reduce the number of model parameters and prevent overfitting, a special folded autoencoder structure is adopted. This means that the weight parameters of the decoder are not independently defined, but share the same set of parameters with the encoder. Specifically, the weight matrix of the decoder is forced to be the transpose of the encoder weight matrix. Therefore, the weight matrix of the decoder has a shape of (256, 64). This not only compresses the model size but also acts as a regularization mechanism, ensuring that the encoding and decoding processes are approximate inverse transformations. The decoding process also includes an independent bias vector with a shape of (256,). The goal of the decoding computation is to generate reconstructed connectivity features. The computation logic involves first performing a linear transformation, followed by applying an activation function (such as Sigmoid or Linear) to match the distribution range of the input data. The final generated reconstructed connectivity feature is a 256-dimensional vector that represents the original input that the model fills in based on the compressed memory, i.e., the hidden layer features.

[0038] In one possible implementation of step S142, based on the true class probability distribution, classification inference and dual loss are performed in parallel on the connection features, hidden layer features, and reconstructed connection features to obtain reconstruction loss and classification loss, including: step S142-1, performing classification inference on the hidden layer features to obtain the predicted classification result; step S142-2, calculating the cross-entropy loss between the predicted classification result and the true class probability distribution to obtain the classification loss; step S142-3, calculating the reconstruction loss between the connection features and the reconstructed connection features.

[0039] First, step S142-1 is executed, where the hidden layer feature is set as a 64-dimensional real-valued vector, for example, hidden layer feature = [0.82, -0.15, ..., 0.33]. This vector is considered to be a multimodal fusion feature that has undergone deep interaction between text and audio, removing redundant information and highly condensing pathological semantics. This feature is directly input into the model's classification layer. The classification layer is essentially a fully connected layer, whose core contains a weight matrix and a bias vector. Considering that the scenario in this application is a four-class classification task of respiratory sounds (normal, wheezing, popping, and mixed), the weight matrix... The shape is set to (4, 64), and the bias vector is... The length is 4. These parameters are generated randomly during the initial training phase and updated via gradient descent in each iteration. During computation, a linear transformation is first performed, calculated using the formula: .here, Hidden layer features, It is a vector of length 4, called the logistic value, where each value represents the model's raw score for the corresponding category. For example, the calculated... The possible values ​​are [2.1, 4.5, -1.2, 0.5], indicating that the model initially considers the second category, specifically wheezing, to be the most likely. However, the Logits values ​​are unbounded real numbers and cannot be directly used as probabilities. Therefore, a Softmax activation function is applied next... This is transformed into a probability distribution. The formula amplifies the differences between scores through exponential operations and ensures that the sum of the output values ​​for all categories is 1. Continuing the example above, after Softmax calculation, the predicted classification result might be [0.08, 0.85, 0.02, 0.05]. This vector represents the predicted probability distribution for each breath sound category, explicitly indicating that the model considers the sample to have an 85% probability of belonging to wheezing, while the probability of belonging to other categories is lower.

[0040] Next, step S142-2 is executed. This step aims to quantify the difference between the model's predictions and objective facts. First, the source of the true class probability distribution needs to be clarified. In the data acquisition phase of step S11, each training sample was accompanied by a diagnostic label annotated by a professional physician. This label was converted into a vector in one-hot encoded format during the preprocessing phase. If the true diagnosis of the current sample is indeed wheezing, and the category index order is [normal, wheezing, popping, mixed], then =[0,1,0,0]. Based on this, the cross-entropy loss function is used for calculation. Cross-entropy is a classic metric in information theory for measuring the difference between two probability distributions, and it performs excellently in classification tasks. Its calculation formula is as follows: In this formula, Represents the classification loss value. The number of categories is 4. It is the true class probability distribution vector The k-th element in It is the k-th element in the predicted classification result. This refers to the natural logarithm. Let's analyze this using the numerical examples above: Because... Only the second element is 1, and the rest are 0. Therefore, the formula simplifies to calculating the negative logarithm of the predicted probability corresponding to the true class. That is... This value of 0.1625 intuitively reflects the accuracy of the classification. If the model's predictions are very accurate, for example... If the value is close to 1.0, the loss approaches 0; conversely, if the model makes an incorrect prediction, for example, it assumes... If the value is only 0.1, the loss becomes very large (-log(0.1)≈2.3). This is the classification loss. This will serve as the supervised learning component in the subsequent hybrid loss function, specifically driving the model to optimize parameters to improve diagnostic accuracy.

[0041] Simultaneously, step S142-3 is executed. This step is the core embodiment of introducing an autoencoder for unsupervised learning in this technical solution. Its purpose is to supervise the effectiveness of feature information interaction and prevent the loss of crucial original information, such as age details in text or faint background noise in audio, during extreme compression of hidden features. The input data consists of the original 256-dimensional connectivity features generated in step S141 and the reconstructed 256-dimensional connectivity features restored by the decoder. This step uses Mean Squared Error (MSE) as the metric. MSE sensitively reflects the numerical Euclidean distance between two vectors. Its calculation formula is as follows: Or, in vector norm form: In this formula, Indicates the reconstruction loss. To flatten the feature dimension, it is 256 here; and These are the numerical values ​​of the connectivity feature and the reshaping feature in the k-th dimension, respectively. Specifically, for example, in the first dimension... =0.5, and the restored =0.45, difference is 0.05, square is 0.0025; on the second dimension, =-0.1, =0.1, difference is 0, squared is 0. The algorithm accumulates the squared differences across all 256 dimensions, then averages the sum by dividing by the total number of dimensions. For example, if the accumulated total squared error is 2.56, then the averaged reconstruction loss... =2.56 / 256=0.01. The smaller this loss value, the more perfectly the decoder can reconstruct the input, which proves that the hidden layer features generated by the encoder completely preserve the original information structure of the text and audio. Conversely, if the reconstruction loss is too large, it indicates that the model has lost information. In this case, the loss will generate a gradient signal, forcing the encoder and decoder to adjust their weights in order to better capture the latent distribution of the data. Finally, step S142 outputs two scalar values ​​in parallel: classification loss. Such as 0.1625 and reconstruction loss For example, 0.01.

[0042] Step S143 involves fusing the two independent error metrics calculated in the previous steps into a single scalar value using a weighted strategy to drive end-to-end optimization of the model. The process of constructing the hybrid loss function follows a specific weighted summation formula: In this formula, This represents the final mixed loss value used for backpropagation. and These are hyperparameters that control the weights of the supervised and unsupervised learning processes, respectively. The settings of these two hyperparameters directly determine whether the model prioritizes improving classification accuracy or preserving feature integrity during training. In a preferred implementation configuration, to balance the contributions of both to gradient updates, [the following is a list of hyperparameters]. and All values ​​are set to 0.5. Based on the above values, the formula is used for calculation, and the value of the mixed loss function is: =0.5×0.1625+0.5×0.01=0.08625. The final scalar value of 0.08625 obtained from this calculation is the objective function value of the current training iteration step.

[0043] It is worth noting that in the multimodal respiratory sound recognition mechanism of the previous embodiments, the unsupervised autoencoder constrained the feature reconstruction process by minimizing the mean square error. However, this method has inherent limitations when dealing with data that is inherently heterogeneous. Specifically, the text embedding vector formed after encoding the case text data contains the core semantic information of the vector direction in a high-dimensional space; while the Mel spectrogram features converted from respiratory sound audio data, as a direct mapping of physical signals, have information intensity closely related to the numerical amplitude of the feature vector. If a uniform mean square error is directly used to constrain the concatenated connection features, the optimization process will inevitably be dominated by audio features with numerical amplitudes much larger than those of text features. This leads the model to prioritize the restoration of audio information in order to reduce the overall error, which may sacrifice the smaller but crucial text semantic information for diagnosis, thus causing the modality dominance problem. Furthermore, this reconstruction mechanism only focuses on the independent restoration of values ​​in each dimension, failing to explicitly constrain the intrinsic correlation between features of different modalities. For example, the strong correlation between the textual description of wheezing and a specific high-frequency continuous spectral band pattern in the Mel spectrogram, i.e., the covariance relationship, may be weakened or lost during feature compression and reconstruction. To address these issues, this application introduces an optimized reconstruction strategy based on heterogeneous metrics and covariance consistency. By imposing metric criteria that match the geometric characteristics of different modal data and introducing consistency constraints for cross-modal second-order statistics, a more precise and robust reconstruction loss function is constructed.

[0044] Specifically, in a possible preferred embodiment of step S142-3, calculating the reconstruction loss between the connectivity features and the reconstructed connectivity features includes: Modal decoupling and heterogeneous manifold distance calculation are performed on the connected features and reconstructed connected features to obtain the heterogeneous reconstruction loss. This step aims to calculate the reconstruction error separately using different distance metrics based on the distinct data characteristics of text and audio features, ensuring that information from both modalities is fairly and effectively preserved. In practice, the 256-dimensional connected features and their corresponding 256-dimensional reconstructed connected features are logically decomposed back into their original text and audio components. That is, the connected features are split into a 128-dimensional text component and a 128-dimensional audio component; similarly, the reconstructed connected features are also split accordingly. For the text component, since its semantics are mainly determined by vector direction, cosine distance, which is sensitive to direction but insensitive to modality, is used to measure the reconstruction error. For the audio component, whose information is closely related to numerical amplitude, root mean square error is used for measurement, and normalization is achieved by dividing by the variance of all audio features in the current training batch, thereby eliminating the numerical advantage of audio features due to their physical dimensions. By introducing independent balancing hyperparameters, the two losses are finally weighted and summed to obtain the heterogeneous reconstruction loss. Its calculation process is defined by the following formula: In this formula, Represents the heterogeneous reconstruction loss. and These represent the text portions in the connection features and the reconstructed connection features, respectively. and This represents the audio portion. and These are preset balance coefficients, determined through systematic hyperparameter tuning experiments such as grid search on independent validation sets. The aim is to find the numerical combination that optimizes the model's final classification performance. For example, they can all be set to 0.5 to indicate equal importance. Let represent the variance of the audio features in the current batch. For example, if the cosine similarity between the original text vector and the reconstructed text vector of a sample is 0.998, then its text reconstruction loss is 0.002. Meanwhile, the mean squared error between its original audio vector and the reconstructed audio vector is 0.08, and the variance of the audio features in the current batch is 4, then the audio reconstruction loss is 0.08 / 4 = 0.02. Therefore, the final heterogeneous reconstruction loss for this sample is 0.5 × 0.002 + 0.5 × 0.02 = 0.011.

[0045] Cross-modal covariance consistency constraints are calculated for both the connectivity features and the reconstructed connectivity features to obtain the cross-covariance consistency loss. This step aims to ensure that the deep interaction relationships between the text and audio modal information remain unchanged after feature reconstruction. Simply ensuring the fidelity of the information within each modality is insufficient; maintaining their inherent logical connections is crucial for accurate diagnosis. In practice, the cross-modal covariance matrices for the 128-dimensional text and 128-dimensional audio parts of the original connectivity features are calculated, along with the covariance matrix for the corresponding part in the reconstructed connectivity features. Each element in the covariance matrix quantifies the linear correlation between a certain dimension of the text feature and a certain dimension of the audio feature. Finally, the overall difference between these two covariance matrices is measured by calculating the squared Frobenius norm between them; this difference value is the cross-covariance consistency loss. The smaller this loss, the better the reconstruction process maintains the second-order statistical dependencies between modalities. The calculation formula is as follows: in, Represents cross-covariance consistency loss. This represents the transpose of the text feature vector. This represents the square of the Frobenius norm. For example, if the calculated original covariance matrix and the reconstructed covariance matrix are very close in value, their Frobenius norm difference may be a small value, such as 0.05. This indicates that the model has successfully preserved, for example, the strong correlation between pneumonia text information and the spectral features of popping sounds in audio.

[0046] The heterogeneous reconstruction loss and the cross-covariance consistency loss are weighted and aggregated to obtain the reconstruction loss. To construct a comprehensive unsupervised learning objective that ensures both the fidelity of information across modalities and the integrity of cross-modal structures, this step involves a weighted sum of the two calculated loss terms. A hyperparameter is introduced to balance their relative importance, thus forming the final reconstruction loss signal driving the autoencoder optimization. This aggregation process is defined by the following formula: In this formula, This is the final optimized reconstruction loss. and These are the heterogeneous reconstruction loss and cross-covariance consistency loss calculated above, respectively. This is a hyperparameter used to adjust the strength of the covariance constraint. Its value can be set empirically, for example, to 0.2, to ensure that the model, while focusing on the reconstruction of each mode itself, also imposes sufficient constraints on the correlation between modes. Continuing the example above, if... =0.011, =0.05, and If the mean squared error is 0.2, then the final optimized reconstruction loss is 0.011 + 0.2 × 0.05 = 0.021. This carefully designed reconstruction loss, compared to the original mean squared error loss, can work together with the classification loss in the mixed loss function to guide the entire model training process.

[0047] By employing this composite optimization and reconstruction loss, the inherent defects of traditional mean squared error in processing heterogeneous data are fundamentally overcome. On one hand, this strategy utilizes heterogeneity metrics, designing independent error measurement criteria for the directional sensitivity of text features and the amplitude sensitivity of audio features, effectively solving the modality dominance problem caused by differences in numerical amplitude and ensuring the balanced preservation and expression of textual semantic information and audio physical information during the fusion process. On the other hand, the introduced cross-modal covariance consistency constraint, as a high-order regularization method, ensures that the autoencoder must maintain the inherent second-order correlation structure between text and audio when compressing and reconstructing features. This makes the final generated hidden features not only a simple memorization of the original information but also a deep understanding of the cross-modal internal logic, thus achieving stronger structured representation capabilities and noise suppression effects. In summary, this dual constraint mechanism, which integrates first-order information fidelity and second-order structural correlation, significantly improves the discriminative power and robustness of multimodal fusion features, laying a solid foundation for achieving high accuracy and high generalization in breath sound recognition and ultimately providing reliable auxiliary diagnostic support for clinical applications.

[0048] In step S15, the parameters of the multimodal breath sound classification model are updated based on the hybrid loss function and using backpropagation with gradient descent to obtain the trained multimodal breath sound classification model. Correspondingly, in the construction of a deep learning model, simply calculating the loss value characterizing the model's performance through forward propagation is insufficient to endow the model with true intelligent diagnostic capabilities. The model parameters in the initial state are usually randomly initialized or pre-trained based on a general dataset, meaning that its internal weights and biases have not yet been adapted for the specific task of breath sound classification, and cannot accurately resolve the complex nonlinear mapping relationship between case text and breath sound spectrum. Although the preceding steps quantify the difference between the model's prediction results and the true labels (i.e., the hybrid loss function value), without some mechanism to feed this error signal back into the network, the model cannot recognize and correct its mistakes. Therefore, this step aims to establish a feedback loop based on mathematical optimization, using the calculated mixed loss as a guiding signal to drive the gradient descent algorithm to search for the optimal solution in the huge parameter space. Through the backpropagation mechanism, the overall error is precisely distributed to each neuron of the network, thereby iteratively fine-tuning the model parameters, and finally forging an initialized mathematical architecture into a well-trained multimodal respiratory sound classification model with high robustness and high accuracy.

[0049] One possible implementation of step S15 is as follows: First, perform the gradient calculation and backpropagation phase. In this phase, the algorithm considers the mixed loss function... The objective function that the entire neural network needs to minimize. The set of learnable parameters in the network. The algorithm is extremely complex, encompassing fine-tuning parameters of the Word2vec embedding matrix in the text encoder, multi-head attention weight matrices and feedforward network weights in the Transformer module, position embeddings and layer normalization parameters in the AST network of the audio encoder, encoding weight matrices and biases in the autoencoder, and fully connected weights and biases in the final classification layer. To determine how each parameter should be adjusted to further reduce the total loss from the values ​​obtained above, the algorithm uses the chain rule from calculus for backpropagation. Specifically, the error signal is first generated at the output, i.e., the classification loss. The generated gradient signal is backpropagated from the classification layer to reconstruct the loss. The generated gradient signals are fed back from the decoder. These two gradient streams converge at the hidden layer interface of the autoencoder and propagate further back to the dual-channel encoder before feature fusion. For any weight parameter in the network... The algorithm calculates the partial derivative of the loss function with respect to that parameter. This partial derivative (i.e., gradient) is a vector that indicates the direction in which the loss function grows fastest in the current high-dimensional space for that parameter. For example, if the gradient of a certain layer's weight in the text encoder is positive, it means that reducing that weight can reduce the total loss. It is particularly important to note that because the autoencoder in this scheme employs a folded structure—that is, the decoder weights are constrained to be the transpose of the encoder weights—the algorithm accumulates the gradient contributions from both the encoding and decoding paths to the same set of weights when calculating the gradient, ensuring that parameter updates simultaneously satisfy the requirements of feature compression and feature reconstruction.

[0050] The parameter update phase then begins. After obtaining the gradient information for all parameters, the optimizer is invoked to perform the specific numerical updates. In this implementation, the Adam (Adaptive Moment Estimation) optimizer is chosen because it combines the advantages of the momentum method and the RMSprop algorithm, enabling it to adaptively adjust the learning rate for each parameter. Before implementation, a series of hyperparameters need to be set, and the initial learning rate is determined based on the preferred experimental configuration. It is set to 0.001. The Adam optimizer maintains two internal state variables based on the gradient calculated in the current step: the first moment estimate (the exponential moving average of the gradient) and the second moment estimate (the exponential moving average of the squared gradient). Parameter updates follow a specific mathematical formula: In this formula, This represents the old parameters before the update (such as the current classifier weights). This represents the updated parameters. and These are the moment estimates after bias correction. This is a very small constant added to prevent the denominator from being zero. Using this formula, the model applies a smaller update step size to parameters with large gradients or frequent updates, and a larger step size to sparsely updated parameters, thus achieving smooth and efficient convergence.

[0051] Finally, the iterative training phase is executed. The gradient calculation and parameter update process described above is not completed all at once, but needs to be repeated on the entire dataset. In practice, the training data is divided into multiple batches; in this application, the batch size is chosen to be 8. This means that the model calculates the average mixture loss and performs a parameter update every time it reads 8 text-audio sample pairs. When the model has traversed all samples in the training set (e.g., 6898 samples), it is said to have completed one training epoch. To ensure sufficient model convergence, the number of training iterations is preset to 100. At the end of each epoch, the model performance is evaluated using an independent validation set. As the number of iterations increases, the mixture loss function value will gradually decrease from the initial value and approach zero, while the model's classification sensitivity and specificity on the validation set will gradually improve. The training process terminates when the preset 100 epochs are reached or the loss value no longer decreases significantly. At this point, all weights and bias parameters in the network have been adjusted from the initial random state to the optimal state, enabling the maximum extraction of multimodal features and accurate differentiation of normal breathing sounds, wheezing sounds, popping sounds, and mixed sounds. This set of fixed optimal parameters, along with its network structure definition, constitutes the final trained multimodal respiratory sound classification model, which can be directly deployed for inference tasks on the patients to be tested.

[0052] After completing the model training phase, the multimodal breath sound classification model has solidified its optimal weights and bias parameters through backpropagation, enabling it to identify pathological features from complex heterogeneous data. The inference phase, as the practical application of this method in clinical scenarios, aims to transform this static mathematical model into a dynamic diagnostic tool for processing novel, unlabeled patient data. Because deep learning models are extremely sensitive to the distribution characteristics of input features, and the feature space mapping rules established during training are based on data generated in a specific format, strict standardization and forward inference operations are performed on the test data to ensure the model can correctly transfer its learned cross-modal association knowledge to new samples. This phase reuses the preprocessing logic from training to transform the original medical records and auscultation recordings into model-resolvable feature tensors. Then, without changing the model parameters, it calculates the breath sound category that best matches the current physiological state, thereby achieving efficient and objective auxiliary diagnosis. The inference phase is described below.

[0053] In step S21, the acquired case text and patient audio are structured to obtain the structured text and Mel spectrogram. It should be understood that during the training phase, the multimodal respiratory sound classification model has learned a deep mapping relationship between the semantics of the case text and the time-frequency features of respiratory sounds based on a large amount of sample data, and has solidified tens of thousands of model parameters. However, in actual clinical applications, the acquired case text is usually a non-standardized record handwritten by doctors, and the patient audio is a raw digital signal containing environmental noise. These raw data differ significantly from the input format required by the model in terms of physical form and feature distribution; direct input will lead to the model's inability to recognize or to produce incorrect inference results. To ensure that the trained model can correctly understand the current patient's physiological information, a data processing procedure strictly consistent with the training phase needs to be performed on the test data. Step S21 aims to build this standardized data bridge by transforming the heterogeneous raw test data into structured text and Mel spectrograms aligned with the training space, eliminating bias at the data source level, and providing a feature carrier that conforms to the model input specifications for subsequent accurate inference.

[0054] One possible implementation of step S21 is as follows: First, receive raw input from the clinical front end. For example, obtain the case text data of a patient to be tested, which is female, 70 years old, diagnosed with pneumonia, with auscultation of the right lower lung, and a raw respiratory sound audio file Patient_Test_001.wav with a duration of 10 seconds and a sampling rate of 4kHz.

[0055] For the test case text, the processing flow first initiates key entity extraction. Using the same text parsing algorithm as in the training phase, four key information dimensions are extracted from the original string: {gender: "female", age: "70", condition: "pneumonia", location: "right lower lung"}. Then, natural language reconstruction is performed, filling these entities into a pre-defined standardized template to generate a coherent structured natural language text for testing: The patient is a 70-year-old female with pneumonia; the auscultation location is the right lower lung. Next, an NLP tokenizer sharing the same vocabulary as in the training phase is invoked to encode the statement. The tokenizer segments the text into tokens and maps them to an integer index sequence, for example, generating the structured text for testing, such as [101,3456,1024,5678,…,102]. This sequence is completely consistent with the format of the training data in terms of length and numerical definition, ensuring that the text encoder can correctly extract its semantic embeddings.

[0056] In parallel, for the audio of the patient under test, the processing flow first applies a Butterworth filter for bandpass filtering. The passband frequency is set to 100Hz to 2000Hz to filter out low-frequency heart sound interference and high-frequency electronic noise from the original recording, retaining the effective frequency band for pulmonary ventilation. Subsequently, a short-time Fourier transform (STFT) is performed on the denoised audio signal. The Hamming window function, frame length, and frame shift are set consistent with those used in the training phase, and the spectral coefficients for each frame are calculated. Next, the logarithmic power spectrum is calculated, converting the complex domain to the energy domain. To match the characteristics of human hearing and adapt to the input requirements of the audio encoder, a Mel filter bank is used to map linear frequencies to a Mel frequency scale. Finally, a weighted summation of the filter bank is used to generate the Mel spectrogram of the patient under test. This spectrogram is a two-dimensional matrix, such as 1024×128, where the horizontal axis represents time, the vertical axis represents Mel frequency, and the pixel value represents energy intensity. Specifically, this structuring process is the same as the process in the training phase, and therefore will not be elaborated further.

[0057] In step S22, the structured text and Mel spectrogram to be tested are input into the trained multimodal respiratory sound classification model to obtain the final respiratory sound category. That is, after the data standardization process in the preceding steps, the original clinical information of the patient has been transformed into a computer-processable structured text sequence and Mel spectrogram image. However, these high-dimensional feature data are merely mathematical expressions of physical signals or symbols, and have not yet established a direct logical connection with specific pathological conclusions. To achieve intelligent assisted diagnosis, it is necessary to utilize the deep neural network model obtained through iterative optimization with massive amounts of data during the training phase to perform forward propagation calculations on the features of the test samples. This step aims to invoke the fixed weights and bias parameters in the trained multimodal respiratory sound classification model, map the test samples into the trained feature space, and analyze their implicit pathological states by fusing text semantics and acoustic patterns, ultimately outputting the diagnostic label with the highest confidence, thereby providing clinicians with objective decision-making basis.

[0058] One possible implementation of step S22 is as follows: First, the feature encoding stage is performed. The input to this stage directly follows the structured text and Mel spectrogram output from step S21. Continuing with the scenario in the previous embodiment, the structured text is an encoded integer sequence representing a 70-year-old woman with pneumonia and a right lower lung, and the Mel spectrogram is a two-dimensional matrix reflecting the sound energy distribution in the patient's lungs. During processing, a trained text encoder (containing Word2vec and Transformer modules) is used to process the structured text. The model calls the word embedding matrix and attention weights optimized during training to map and encode the discrete integer sequence into a one-dimensional real vector, i.e., the text vector to be tested. According to the aforementioned settings, this vector has a dimension of 128, numerically condensing semantic information such as pneumonia and old age. Simultaneously, a trained audio encoder (AST network) is used to process the Mel spectrogram to be tested. The AST network uses fixed position embedding parameters and Transformer layer weights to capture high-frequency burst textures in the spectrogram and maps them into a 128-dimensional audio vector to be tested. These two vectors are aligned with the distribution of the training data in the numerical space, which forms the basis for subsequent inference.

[0059] The algorithm then proceeds to the feature fusion stage. To simulate the process of a doctor making a comprehensive judgment based on medical records and auscultation results, the algorithm concatenates the generated text vector and audio vector along the feature dimension to form a 256-dimensional flattened vector. This concatenated vector is then input into the encoder part of the trained autoencoder. It is important to note that the autoencoder no longer performs backpropagation updates at this stage; it is used solely as a feature extractor, and only its encoding path is utilized. The encoder weight matrix determined during the training phase is then used... (Shape is 64×256) and bias vector The length is 64. A linear transformation and nonlinear activation are applied to the spliced ​​vector to be tested. The calculation formula is: In this formula, It is the Sigmoid activation function. This is the output hidden layer feature to be tested. Through this calculation, the original 256-dimensional concatenated feature is compressed and mapped into a 64-dimensional low-dimensional dense vector. This hidden layer feature to be tested... It is a product of multimodal information interaction. For example, it may express the comprehensive pathological feature of a specific popping sound pattern (from audio) that appears in the context of pneumonia (from text) through a specific combination of numerical values.

[0060] Finally, the classification decision is executed. The 64-dimensional hidden features containing interaction information are input into the model's classification layer. The classification layer also uses the pre-trained weight matrix. The shape is 4×64 and the bias vector First, calculate the logical value vector. The algorithm obtains the raw scores of the test sample in four categories (normal, wheezing, crackles, and mixed). For example, the calculated logistic value vector is [-1.2, -0.5, 3.8, 0.2]. To obtain an intuitive probabilistic interpretation, the Softmax function is applied to normalize the logistic values, and the result is the probability distribution of each breath sound category, for example [0.005, 0.012, 0.956, 0.027]. This result indicates that the model believes that the test sample has a 95.6% probability of belonging to the third category (crackles). The algorithm selects the label corresponding to the index with the highest probability as the output through the Argmax operation. Combined with the preceding clinical background (pneumonia patients are often accompanied by crackles), this inference result is consistent with the actual pathology. Finally, the label crackles is output as the final breath sound category, completing the complete inference process from raw multimodal data to a specific diagnostic conclusion.

[0061] In summary, a breath sound recognition method based on multimodal data fusion, as described in this application, addresses the problems of redundant case text, insufficient modal interaction, and conflicting training objectives in existing technologies. First, the case text is structured through key entity extraction and natural language reconstruction, combined with audio data enhancement to remove redundant information and improve data quality. Then, a dual-channel encoder is used to extract deep embedding vectors from the text and audio respectively. Multimodal feature concatenation and autoencoder are used for joint modeling to achieve deep interaction and feature reshaping between modalities in the hidden space, effectively capturing potential cross-modal associations. Based on this, a hybrid loss function including reconstruction loss and classification loss is constructed, and model parameters are collaboratively optimized through backpropagation. This ensures the effectiveness of feature fusion and balances the differences between unsupervised feature learning and supervised classification tasks, significantly improving the accuracy and robustness of the model in breath sound recognition in complex scenarios.

[0062] Figure 6 This is a block diagram of a respiratory sound recognition system based on multimodal data fusion according to an embodiment of this application. Figure 6As shown, the respiratory sound recognition system 100 based on multimodal data fusion according to an embodiment of this application includes: a training module 110 and an inference module 120; the training module 110 includes: a training data acquisition unit 111, used to acquire training case text data and training respiratory sound audio data; a training data structuring unit 112, used to perform structuring and enhancement preprocessing on the training case text data and training respiratory sound audio data to obtain structured case text data and data-enhanced Mel spectrograms; a dual-channel feature encoding unit 113, used to perform dual-channel feature encoding on the structured case text data and data-enhanced Mel spectrograms to obtain text embedding vectors and audio embedding vectors; and a hybrid loss function generation unit 11. 4. A mixed loss function is obtained by inputting text embedding vectors and audio embedding vectors into a multimodal respiratory sound classification model based on the true class probability distribution; a respiratory sound classification model generation unit 115 is used to update the parameters of the multimodal respiratory sound classification model based on the mixed loss function and by using backpropagation of gradient descent to obtain a trained multimodal respiratory sound classification model; the inference module 120 includes: a data structuring unit 121, used to structure the acquired test case text and test patient audio to obtain test structured text and test Mel spectrogram; a respiratory sound category generation unit 122, used to input the test structured text and test Mel spectrogram into the trained multimodal respiratory sound classification model to obtain the final respiratory sound category.

[0063] Here, those skilled in the art will understand that the specific operations of each step in the above-described multimodal data fusion-based respiratory sound recognition system have been referenced above. Figures 1 to 5 The description of the breathing sound recognition method based on multimodal data fusion has been detailed, and therefore, its repeated description will be omitted.

Claims

1. A method for recognizing respiratory sounds based on multimodal data fusion, characterized in that, include: Training phase and inference phase; The training phase includes: Acquire training case text data and training breath sound audio data; The training case text data and training breath sound audio data were preprocessed with structure and enhancement to obtain structured case text data and data-enhanced Mel spectrograms. Dual-channel feature encoding was performed on structured case text data and data-enhanced Mel spectrograms to obtain text embedding vectors and audio embedding vectors; Based on the true category probability distribution, text embedding vectors and audio embedding vectors are input into a multimodal breath sound classification model to obtain a hybrid loss function; The parameters of the multimodal respiratory sound classification model are updated based on a hybrid loss function and by using backpropagation of gradient descent to obtain a well-trained multimodal respiratory sound classification model. The inference phase includes: The acquired case text and patient audio were structured to obtain the structured text and Mel spectrogram of the test. The structured text to be tested and the Mel spectrogram to be tested are input into the trained multimodal respiratory sound classification model to obtain the final respiratory sound category.

2. The respiratory sound recognition method based on multimodal data fusion according to claim 1, characterized in that, The training case text data and training breath sound audio data underwent structuring and augmentation preprocessing to obtain structured case text data and data-augmented Mel spectrograms, including: Key entity extraction and natural language reorganization are performed on the training case text data to obtain coherent structured natural language text; Tokenization is performed on coherent structured natural language text to obtain structured case text data; Bandpass filtering and short-time Fourier transform were performed on the training respiratory sound audio data to obtain the Mel spectrogram; Data augmentation is performed on the Mel spectrogram to obtain a data-augmented Mel spectrogram.

3. The respiratory sound recognition method based on multimodal data fusion according to claim 2, characterized in that, Data augmentation is performed on the Mel spectrogram to obtain a data-enhanced Mel spectrogram, including: Perform Mixup enhancement processing on the Mel spectrogram based on linear interpolation; and / or The Mel spectrogram is enhanced by time-dimensional warping.

4. The respiratory sound recognition method based on multimodal data fusion according to claim 1, characterized in that, Dual-channel feature encoding was performed on structured case text data and data-enhanced Mel spectrograms to obtain text embedding vectors and audio embedding vectors, including; The structured case text data is input into a text encoder to obtain text embedding vectors; The data-enhanced Mel spectrogram is input into the audio encoder to obtain the audio embedding vector.

5. The respiratory sound recognition method based on multimodal data fusion according to claim 4, characterized in that, The text encoder includes a Word2vec module and a text encoding module composed of multiple Transformer modules; the audio encoder is an audio spectrum Transformer network.

6. The respiratory sound recognition method based on multimodal data fusion according to claim 1, characterized in that, Based on the true class probability distribution, text embedding vectors and audio embedding vectors are input into a multimodal breath sound classification model to obtain a hybrid loss function, including: Multimodal feature concatenation and autoencoder joint modeling of text embedding vectors and audio embedding vectors are performed to obtain connectivity features, hidden layer features, and reconstructed connectivity features; Based on the true class probability distribution, classification inference and dual loss are performed in parallel to obtain reconstruction loss and classification loss for connection features, hidden layer features and reconstructed connection features; The hybrid loss function is constructed based on reconstruction loss and classification loss.

7. The respiratory sound recognition method based on multimodal data fusion according to claim 6, characterized in that, Based on the true class probability distribution, classification inference and dual loss calculation are performed in parallel on connectivity features, hidden layer features, and reconstructed connectivity features to obtain reconstruction loss and classification loss, including: The hidden layer features are used to perform classification inference to obtain the predicted classification result; The classification loss is obtained by calculating the cross-entropy loss between the predicted classification result and the true class probability distribution. Calculate the reconstruction loss between the connectivity features and the reconstructed connectivity features.

8. The respiratory sound recognition method based on multimodal data fusion according to claim 7, characterized in that, Calculate the reconstruction loss between connectivity features and reconstructed connectivity features, including: Modal decoupling and heterogeneous manifold distance calculation are performed on the connectivity features and reconstructed connectivity features to obtain the heterogeneous reconstruction loss; Cross-modal covariance consistency constraints are calculated for connectivity features and reshaped connectivity features to obtain cross-covariance consistency loss. The reconstruction loss is obtained by weighted aggregation of heterogeneous reconstruction loss and cross-covariance consistency loss.

9. A respiratory sound recognition system based on multimodal data fusion, characterized in that, include: Training module and inference module; The training module includes: The training data acquisition unit is used to acquire training case text data and training breath sound audio data; The training data structuring unit is used to perform structuring and augmentation preprocessing on training case text data and training respiratory sound audio data to obtain structured case text data and data-augmented Mel spectrograms. A dual-channel feature encoding unit is used to perform dual-channel feature encoding on structured case text data and data-enhanced Mel spectrograms to obtain text embedding vectors and audio embedding vectors; The hybrid loss function generation unit is used to input text embedding vectors and audio embedding vectors into a multimodal breathing sound classification model based on the true class probability distribution to obtain a hybrid loss function; A breath sound classification model generation unit is used to update the parameters of a multimodal breath sound classification model based on a mixture loss function and backpropagation using gradient descent to obtain a trained multimodal breath sound classification model. The inference module includes: The data structuring unit is used to structure the acquired test case text and test patient audio to obtain the test structured text and the test Mel spectrogram; The breath sound category generation unit is used to input the structured text to be tested and the Mel spectrogram to be tested into the trained multimodal breath sound classification model to obtain the final breath sound category.

Images