CA-LMF multi-mode fusion detection method and system based on respiratory heterogeneity signals

Abstract

The application provides a CA-LMF multi-mode fusion detection method and system based on respiratory heterogeneous signals, and belongs to the technical field of medical equipment, and comprises the following steps: (1) setting parameters for signal acquisition; (2) signal preprocessing; (3) extracting effective characteristic values; (4) cross attention feature CA fusion; (5) low-rank multi-modal LMF fusion; (6) information detection and information classification processing. The application collects respiratory heterogeneous signals based on microphones and optical fiber sensors, utilizes a feature engineering and multi-modal fusion method, realizes rapid, accurate and non-invasive information detection classification of child pneumonia without radiation damage. The system collects child respiratory signals, adopts a cross attention and low-rank multi-modal fusion method (CA-LMF) to obtain fusion features after preprocessing and feature extraction, and combines an integrated learning method to classify and process child pneumonia.

Description

Technical Field

[0001] This invention provides a CA-LMF multimodal fusion detection method and system based on respiratory heterogeneous signals, belonging to the field of medical device technology. Background Technology

[0002] An investigation revealed the following two problems with current methods for pneumonia detection and information classification:

[0003] (1) It poses a radiation risk and has a low level of intelligence:

[0004] Imaging examinations require patients to travel to the hospital and undergo a relatively cumbersome process, lacking convenience. Although traditional imaging examinations are non-invasive to some extent, radiation exposure cannot be avoided, which remains an objective drawback. Imaging techniques such as chest X-rays and CT scans involve radiation exposure, especially for children, whose risk of radiation-induced cancer is far higher than that of adults. This makes traditional methods inherently safe, particularly among children, a vulnerable group. This method, heavily reliant on the doctor's subjective judgment and lacking intelligent and objective standards for testing and information classification, may lead to missed diagnoses or misdiagnoses.

[0005] (2) High cost and long time consumption

[0006] Traditional imaging examinations currently used in clinical practice are relatively expensive, costing between 60 and 120 yuan per examination, increasing the financial burden on patients. Furthermore, traditional imaging examinations require multiple steps, including appointment scheduling, queuing, imaging, and reporting, which is time-consuming. These factors present certain problems in terms of cost and convenience for traditional methods. Summary of the Invention

[0007] This invention addresses the aforementioned technical problems by using the CA-LMF multimodal feature fusion method and the SVS integrated detection and information classification model.

[0008] The specific technical solution is as follows:

[0009] The CA-LMF multimodal fusion detection method based on respiratory heterogeneous signals includes the following steps:

[0010] (1) Set parameters for signal acquisition;

[0011] A sensing fiber optic cable is placed between the patient's first and second ribs to collect the patient's respiratory vibration signals; a microphone is worn under the patient's collar and positioned 3-5 cm above the patient's mouth and nose to collect the patient's respiratory audio signals. Simultaneously, the patient's respiratory sound signals and the vibration signals from the respiratory fiber optic cable are collected.

[0012] (2) Signal preprocessing;

[0013] During the fiber optic signal acquisition process, the acquired data is first compressed and preprocessed, then the signal is converted into a frequency domain signal by Discrete Cosine Transform (DCT) to filter out high-frequency noise, and the low-frequency part is subjected to inverse DCT to obtain the reconstructed low-frequency physiological signal.

[0014] During the audio signal acquisition process, noise reduction is performed to obtain a clean breathing audio signal. Subsequently, using the short-time energy and short-time zero-crossing rate calculation results, the audio signal endpoints are detected using a dual-threshold method, and the effective portion is segmented and reconstructed to obtain an effective signal that is conducive to algorithm analysis.

[0015] (3) Extract effective feature values;

[0016] For fiber optic signals, a total of 80 dimensions are extracted from three categories: time-domain features, frequency-domain features, and wavelet transform features. Specifically, time-domain features comprise 13 dimensions; frequency-domain features comprise 39 dimensions; and wavelet transform features comprise 28 dimensions.

[0017] For audio signals, the extracted audio signal features are roughly divided into three categories: frequency domain features, sound quality features, and rhythm features, totaling 523 dimensions. Among them, frequency domain features have a total of 273 dimensions; sound quality features have a total of 130 dimensions; and rhythm features have a total of 120 dimensions.

[0018] (4) Cross-attention feature CA fusion;

[0019] After signal feature extraction, frequency domain features of two homogeneous heterogeneous signals are selected. Since both features describe the characteristics of the signal in the frequency domain and can reflect the inherent structure and characteristics of the signal, cross-attention feature (CA) fusion is performed to obtain similar fused features.

[0020] Audio frequency domain features, serving as the primary sequence, guide the model to focus more on and utilize the important information of audio features in the attention mechanism; vibration frequency domain features, as auxiliary sequences, provide additional information related to audio features. The relationship between these two types of features is captured through CA feature fusion.

[0021] (5) Low-rank multimodal (LMF) fusion;

[0022] The timbre and rhythm features in the respiratory audio signal are taken as special audio features, and the time domain and wavelet transform domain features in the respiratory vibration signal are taken as special fiber features. Combined with similarity fusion features, low-rank multimodal (LMF) fusion is performed on the three types of feature vectors. The similarity information of the signals is combined with their unique information to obtain the global fusion feature, that is, the complete signal information of the respiratory heterogeneous signal.

[0023] (6) Information detection and information classification processing;

[0024] The global fusion features are fed into the SVS ensemble learning model for information detection and classification.

[0025] Five base classification models—RF, SVM, KNN, LogiR, and XGBoost—were used for classification. The models were automatically sorted based on their accuracy and then split into three groups of classifiers in descending order of accuracy. The optimal hyperparameter combination was found through parameter optimization. Weights were then assigned to each base classifier according to the weight formula. Based on these weights, the three groups of classifiers were used to construct Voting soft voting models α, β, and γ, respectively. The five base classifiers were then combined to form a Voting soft voting model δ. Finally, using α, β, and γ as primary classifier models and δ as a secondary classifier model, an SVS fusion model was constructed.

[0026] This invention employs a CA-LMF multimode fusion detection system based on heterogeneous respiratory signals, comprising a narrowband laser source, a photoelectric converter, a reference fiber, two optical couplers, and an optical attenuator. These components are integrated into a system-supported integrated terminal. The sensing fiber and wireless microphone are externally located on the integrated terminal, simultaneously transmitting the collected signals to the terminal for processing. The narrowband laser source is connected to the first optical coupler for beam splitting before transmission to the sensing and reference fibers. When the subject experiences vibration signals due to lung respiration, the sensing fiber deforms, causing changes in the internal optical path and resulting in a phase difference between the sensing and reference fibers. The two beams of light with this phase difference are then coupled through a second optical coupler and sent to the photoelectric conversion module to convert the optical signals into electrical signals, which are then transmitted to a PC via a data acquisition card.

[0027] This invention utilizes microphones and fiber optic sensors to collect heterogeneous respiratory signals. By employing feature engineering and multimodal fusion methods, it achieves rapid, accurate, and radiation-free non-invasive information detection and classification of childhood pneumonia. The system collects children's respiratory signals, preprocesses and extracts features, then uses a cross-attention and low-rank multimodal fusion method (CA-LMF) to obtain fused features, and combines this with ensemble learning methods to classify childhood pneumonia.

[0028] During the system's signal acquisition and preprocessing, the respiratory audio and vibration signals of children are acquired through microphones and fiber optic sensors. Each signal segment is recorded for approximately 20 seconds and includes at least four complete breathing processes. Noise reduction and endpoint detection are performed on the audio signals, while compression preprocessing and noise filtering are applied to the fiber optic signals.

[0029] During the feature extraction and fusion process of the system, frequency domain, sound quality, and rhythm features of the audio signal are extracted, while time domain, frequency domain, and wavelet features of the fiber optic signal are extracted. A cross-attention mechanism is used to fuse the similarity features of the frequency domain features of the two signals, and then a low-rank multimodal fusion method is combined to perform global feature fusion of the three types of features.

[0030] During the information detection and classification process of the system, the constructed Shote-Voting-Stacking (SVS) ensemble learning model is used to perform binary classification of pneumonia and non-pneumonia using global fusion features.

[0031] Compared to traditional detection and information classification methods, this system avoids radiation exposure, making it particularly suitable for children; it reduces reliance on doctors' professional skills, making it easier to use in resource-limited environments; the cost per acquisition is lower than that of traditional imaging examinations; the acquisition time is short, the accuracy is high, and it can quickly provide detection and information classification results. Attached Figure Description

[0032] Figure 1 This is a flowchart of the present invention;

[0033] Figure 2 This invention relates to the SVS ensemble learning model;

[0034] Figure 3 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0035] The specific technical solution of the present invention will be described in conjunction with the accompanying drawings.

[0036] like Figure 1 As shown, the CA-LMF multimodal fusion detection method based on respiratory heterogeneous signals includes the following steps:

[0037] (1) Set parameters for signal acquisition;

[0038] First, after the system is set up and powered on, the user interacts with the GUI interface to set parameters such as the recording time, number of times, and frequency of respiratory signals. Once signal acquisition begins, the sensing fiber optic cable is placed between the child's first and second ribs to collect the child's respiratory vibration signals. A microphone is worn under the child's collar, positioned 3-5 cm from the child's mouth and nose to collect the child's respiratory audio signals. Simultaneously, the patient's respiratory sounds and the vibration signals from the respiratory fiber optic cable are collected, with the acquisition time fixed at approximately 20 seconds, including at least four complete respiratory cycles.

[0039] (2) Signal preprocessing;

[0040] Internally, during fiber optic signal acquisition, the acquired data is first compressed and preprocessed to reduce computational load. Since noise from electromagnetic interference and unexpected physical movements can significantly interfere with the signal, this invention performs Discrete Cosine Transform (DCT) processing on the signal to convert it into a frequency domain signal, filtering out high-frequency noise, and then performs an inverse DCT on the low-frequency component to obtain the reconstructed low-frequency physiological signal.

[0041] During the audio signal acquisition process, noise reduction is performed to obtain a clean breathing audio signal. Subsequently, using calculations such as short-time energy and short-time zero-crossing rate, the audio signal endpoints are detected using a dual-threshold method, and the effective portion is segmented and reconstructed to obtain an effective signal that is conducive to algorithm analysis.

[0042] (3) Extract effective feature values;

[0043] After completing the preprocessing steps for both signals, effective feature values ​​are extracted for each. For fiber optic signals, the system extracts a total of 80 dimensions across three categories: time-domain features, frequency-domain features, and wavelet transform features. Specifically, time-domain features comprise 13 dimensions; frequency-domain features comprise 39 dimensions; and wavelet transform features comprise 28 dimensions. For audio signals, the system extracts 523 dimensions across three categories: frequency-domain features, timbre features, and rhythm features. Specifically, frequency-domain features comprise 273 dimensions; timbre features comprise 130 dimensions; and rhythm features comprise 120 dimensions.

[0044] (4) CA feature fusion;

[0045] After signal feature extraction, frequency domain features of the two homologous but heterogeneous signals are selected. Both features describe the signal's characteristics in the frequency domain, reflecting its intrinsic structure and properties. Cross-Attention (CA) feature fusion is then performed. Audio frequency domain features serve as the primary sequence, guiding the model to focus more on and utilize important information from audio features within the attention mechanism; vibration frequency domain features serve as an auxiliary sequence, providing additional information related to the audio features. Through CA feature fusion, the interrelationship between these two types of features is captured, providing a more comprehensive representation of signal characteristics, resulting in Similar Fusion (SF) Features.

[0046] (5) Low-rank multimodal (LMF) fusion;

[0047] The structural characteristics of audio and vibration signals make them both similar and unique in the one-dimensional representation of respiratory signals. After fusing the similar features of the two through CA feature fusion, the system utilizes the complementarity of heterogeneous data and combines their unique features. The system takes the timbre and rhythm features in the respiratory audio signal as audio special (AS) features, and the time domain and wavelet transform domain features in the respiratory vibration signal as fiber optical special (FOS) features. Combined with similar fusion (SF) features, the system performs low-rank multimodal (LMF) fusion on the three types of feature vectors, combining the similar information of the signals with their unique information to obtain global fusion (GF) features, that is, the complete signal information of the heterogeneous respiratory signal.

[0048] (6) Information detection and information classification processing;

[0049] The system feeds the globally fused features into the Smote-Voting-Stacking (SVS) ensemble learning model for information detection and classification. The model structure flowchart is as follows: Figure 2 As shown, five base classification models—RF, SVM, KNN, LogiR, and XGBoost—were used for classification. The models were automatically sorted based on classification accuracy and then split into three groups of classifiers in descending order of accuracy. The optimal hyperparameter combination was found through parameter optimization. Weights were then assigned to each base classifier according to the following formula. Based on these weights, the three groups of classifiers were used to construct Voting soft voting models α, β, and γ, respectively. The five base classifiers were then combined to form a Voting soft voting model δ. Finally, using α, β, and γ as primary classifier models and δ as a secondary classifier model, an SVS fusion model was constructed.

[0050] like Figure 3 The CA-LMF multimode fusion detection system based on respiratory heterogeneous signals used in this invention includes a narrowband laser source, a photoelectric converter, a reference fiber, two optical couplers, and an optical attenuator. All of these devices are integrated in the system support integrated terminal. The sensing fiber and the wireless microphone are placed outside the integrated terminal, and the collected signals are sent to the terminal for processing.

[0051] A narrow-bandwidth laser source is split by an optical coupler and transmitted to the sensing fiber and the reference fiber. When the subject's breathing generates vibration signals, the sensing fiber deforms, causing changes in the internal optical path and resulting in a phase difference between the sensing and reference fibers. These two beams with phase difference are then coupled by a second optical coupler and sent to a photoelectric conversion module to convert the optical signals into electrical signals, which are then transmitted to a PC via a data acquisition card.

[0052] Addressing the shortcomings of existing information detection and classification technologies, such as radiation risks, low intelligence, high costs, and long processing times, this invention represents a significant leap forward in the flexibility and intelligence of pneumonia detection and information classification methods.

[0053] Based on the technical characteristics of fiber optic vibration sensing and audio sensing recording, this invention enables the system to avoid the potential carcinogenic risks associated with radiation exposure in the detection and classification of pneumonia in children, which are inherent to traditional imaging techniques such as chest X-rays and CT scans.

[0054] In terms of functional flexibility and versatility, this invention supports users to interact with subjects through the system's integrated terminal. The latter will analyze and detect the pneumonia status of different patients based on the user's operation through a respiratory signal analysis model.

[0055] In the respiratory signal analysis process, time-domain, frequency-domain, and wavelet-domain features of audio and fiber optic signals are extracted. A cross-attention mechanism is used to fuse similar features, and then a low-rank multimodal fusion method is employed to fuse global features. This CA-LMF multimodal feature fusion method, leveraging the complementarity of heterogeneous respiratory signals, provides reliable information completeness and pathological interpretation for the detection and classification of pneumonia, giving this invention high theoretical credibility in scenarios requiring multimodal fusion of heterogeneous signals.

[0056] In addition, the SVS ensemble learning model constructed in this invention adopts the Stacking ensemble learning framework, which integrates multiple traditional machine learning algorithms, thereby improving the model's robustness. Users can adjust the SVS model to meet the detection and information classification needs of different diseases, and it can be further used for the analysis and research of detection and information classification of other respiratory or lung diseases, demonstrating its generalization and scalability.

[0057] Meanwhile, due to the system's independent operating mode, this invention is free from dependence on brands, transmission interfaces, and data formats compared to similar products, possessing strong cross-brand versatility. This invention features fast detection and information classification speed, high classification accuracy, specificity, and sensitivity, which to some extent alleviates the workload and intensity of outpatient procedures for medical personnel during peak pneumonia seasons.

Claims

1. A CA-LMF multimodal fusion detection method based on respiratory heterogeneous signals, characterized in that, Includes the following steps: (1) Set parameters for signal acquisition; The sensing fiber is placed between the patient's first and second ribs to collect the patient's respiratory vibration signal; the microphone is worn under the patient's collar and positioned 3-5 cm from the patient's mouth and nose to collect the patient's respiratory audio signal; at the same time, the patient's respiratory sound signal and the respiratory fiber vibration signal are collected. (2) Signal preprocessing; During the fiber optic signal acquisition process, the acquired data is first compressed and preprocessed, then the signal is converted into a frequency domain signal by Discrete Cosine Transform (DCT) to filter out high-frequency noise, and the low-frequency part is subjected to inverse DCT to obtain the reconstructed low-frequency physiological signal. During the acquisition of audio signals, noise reduction is performed on the audio signals to obtain pure breathing audio signals; Subsequently, using the calculation results of short-time energy and short-time zero-crossing rate, the audio signal endpoints are detected by the dual-threshold method, and the effective part of the reconstruction is cut and reconstructed to obtain an effective signal that is conducive to algorithm analysis. (3) Extract effective feature values; For fiber optic signals, extract time-domain features, frequency-domain features, and wavelet transform features; For audio signals, the extracted audio signal features are divided into frequency domain features, sound quality features, and rhythm features; (4) Cross-attention feature CA fusion; After signal feature extraction, frequency domain features of two homogeneous heterogeneous signals are selected. Since both features describe the characteristics of the signal in the frequency domain and can reflect the inherent structure and characteristics of the signal, cross-attention feature (CA) fusion is performed to obtain similar fused features. (5) Low-rank multimodal LMF fusion; The timbre and rhythm features in the respiratory audio signal are taken as special audio features, and the time domain and wavelet transform domain features in the respiratory vibration signal are taken as special fiber features. Combined with similarity fusion features, low-rank multimodal (LMF) fusion is performed on the three types of feature vectors. The similarity information of the signals is combined with their unique information to obtain global fusion features, that is, the complete signal information of the respiratory heterogeneous signal. (6) Information detection and information classification processing; The global fusion features are fed into the SVS ensemble learning model for information detection and classification.

2. The CA-LMF multimodal fusion detection method based on respiratory heterogeneous signals according to claim 1, characterized in that, In step (3): For fiber optic signals, a total of 80 dimensions are extracted from three categories: time-domain features, frequency-domain features, and wavelet transform features; among them, time-domain features total 13 dimensions; frequency-domain features total 39 dimensions; and wavelet transform features total 28 dimensions. For audio signals, the extracted audio signal features are divided into three categories: frequency domain features, sound quality features, and rhythm features, totaling 523 dimensions; among them, frequency domain features total 273 dimensions; sound quality features total 130 dimensions; and rhythm features total 120 dimensions.

3. The CA-LMF multimodal fusion detection method based on respiratory heterogeneous signals according to claim 1, characterized in that, In step (4) Cross-attention feature (CA) fusion, audio frequency domain features serve as the main sequence to guide the model to pay more attention to and utilize the important information of audio features in the attention mechanism; vibration frequency domain features serve as the auxiliary sequence to provide additional information related to audio features; and the relationship between these two features is captured through CA feature fusion.

4. The CA-LMF multimodal fusion detection method based on respiratory heterogeneous signals according to claim 1, characterized in that, In step (6) information detection and information classification processing, five base classification models, namely RF, SVM, KNN, LogiR and XGBoost, are used to classify and judge the data. The data is automatically sorted according to the classification accuracy and split into three groups of classifiers in descending order. The hyperparameter combination that makes the model achieve the best effect is found by the parameter optimization method. Then, the weights of each base classifier are calculated according to the weight formula. Based on the weights, the three groups of classifiers are constructed into Voting soft voting models α, β and γ respectively. The five base classifiers are constructed into Voting soft voting model δ. Then, with α, β and γ as primary classifier models and δ as secondary classifier models, the SVS fusion model is constructed.

5. A CA-LMF multimodal fusion detection system based on respiratory heterogeneous signals, used in the detection method according to any one of claims 1 to 4, characterized in that, The system includes a narrowband laser source, a photoelectric converter, a reference fiber, two optical couplers, and an optical attenuator. All of these devices are integrated into the system support integrated terminal. The sensing fiber and the wireless microphone are located outside the integrated terminal, and the collected signals are sent to the terminal for processing. A narrow-bandwidth laser source is connected to an optical coupler to split the light and transmit it to the sensing fiber and the reference fiber. When the subject generates a vibration signal due to breathing, the sensing fiber deforms, causing a change in the optical path inside the fiber, which in turn leads to a phase change in the optical path, resulting in a phase difference between the sensing fiber and the reference fiber. The two light signals with the phase difference are then coupled through an optical coupler and sent to the photoelectric conversion module to convert the light signal into an electrical signal, which is then transmitted to the PC via a data acquisition card.

Images