Children congenital heart disease information processing method based on incomplete multiple modes

Abstract

The invention discloses a children congenital heart disease information processing method based on incomplete multi-modality, comprising the following steps: S1, acquiring signal data, and performing feature extraction and modal interaction by using LSTM and a multi-modality heterogeneous dynamic fusion network to obtain interaction information between each modality; s2, inputting interaction information among the modals into a teacher network and a student network, and updating the interaction information; s3, updating the edge deployment network by using the updated teacher network; and S4, deploying the network by using the updated edge, and outputting a result. According to the invention, an incomplete modal data-oriented edge perception distillation mechanism is designed, the self-adaption and generalization ability of the model to partial modal deficiency in a resource-limited scene is enhanced, and an intelligent screening system with high accuracy and strong robustness is constructed.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and specifically to a method for processing information on congenital heart disease in children based on incomplete multimodal approaches. Background Technology

[0002] Current research largely focuses on single physiological signals or single examination results obtained through technical means. In clinical practice, individual physiological signals, symptoms, and examination results only provide localized information about the cardiovascular system's functional status, offering limited information and failing to reflect global changes in the heart. This significantly reduces the detection rate of congenital heart disease. While some studies utilize multimodal data, they often employ early feature-level fusion. This simplistic approach can lead to an excessive increase in the feature space dimensionality with the addition of modalities, resulting in performance degradation. To address these issues, this invention aims to fully utilize information from various modalities by synchronizing the collected multimodal data and constructing a heterogeneous fusion model.

[0003] On the other hand, historical data on congenital heart disease suffers from incomplete multimodal collection and inconsistent data labeling, making it difficult to support large-scale training of traditional deep learning algorithms. Therefore, overcoming the fundamental theoretical and algorithmic bottlenecks of small-sample learning in multimodal, small-sample historical data on congenital heart disease and constructing a highly reliable screening model for children with congenital heart disease has become an important research direction. Knowledge distillation aims to use a high-performance teacher model with a large number of parameters to guide the training of a lightweight student model, thereby significantly reducing the computational overhead and latency during model deployment while maintaining high inference performance. Meta-learning, on the other hand, focuses on "learning how to learn," endowing the model with the ability to adaptively adjust hyperparameters or network structure, enabling it to quickly and efficiently adapt and learn when facing new tasks based on experience accumulated from previous tasks. Therefore, this invention employs a knowledge distillation model based on meta-learning to address the problems of multimodal and small-sample data.

[0004] Currently, the main methods for screening for congenital heart disease are as follows:

[0005] 1) Screening based on traditional medical diagnostic methods Initial diagnosis relies primarily on medical history taking, physical examination, electrocardiogram (ECG), and chest X-ray. Those testing positive in screening require referral for echocardiography for definitive diagnosis. While traditional methods are readily available, they have limitations such as expensive equipment, the potential for missing small defects, and reliance on physician experience.

[0006] 2) Intelligent screening based on deep learning This method primarily relies on existing pre-trained deep learning models for analysis and screening, offering advantages over traditional methods in terms of intelligence and effectiveness, and supporting telemedicine applications. However, existing pre-trained deep learning models all require large amounts of high-quality labeled image data, and in particular, they cannot meet the practical needs of intelligent screening for congenital heart disease in remote areas. Therefore, it is necessary to consider the incomplete modalities of the actual edge. Summary of the Invention

[0007] To address the above problems, this invention proposes a method for processing information on congenital heart disease in children based on incomplete multimodal approaches.

[0008] The technical solution of this invention is: a method for processing information on congenital heart disease in children based on incomplete multimodal methods, comprising the following steps: S1. Acquire signal data and use LSTM and multimodal heterogeneous dynamic fusion network to perform feature extraction and modal interaction to obtain the interaction information between various modes; S2. Input the interaction information between the various modalities into the teacher network and student network for updating; S3. Use the updated teacher network to update the edge deployment network; S4. Utilize the updated edge deployment network and output the results.

[0009] Furthermore, S1 includes the following sub-steps: S11. Acquire signal data, specifically including electrocardiogram data, heart sound data, and echocardiogram data; S12. Use LSTM to extract feature data for each modality; S13. Based on the semantic space corresponding to the feature data of each modality, a shared semantic space is generated using a multimodal heterogeneous dynamic fusion network; S14. Using a multilayer perceptron, the shared semantic space is mapped to a low-dimensional space to obtain the interaction information between various modalities.

[0010] Furthermore, S2 includes the following sub-steps: S21. Copy the student network and update the copied student network using the training set; S22. Calculate the loss of the updated student network using the test set, use the loss as a feedback signal to update the teacher network, and discard the updated student network.

[0011] Furthermore, S3 includes the following sub-steps: S31. Transmit the interaction information between modalities from the updated teacher network to the edge deployment network; S32. Reshape the fusion characteristics of the teacher network and the fusion characteristics of the deployment network to obtain the fusion characteristics of the reshaped teacher network and the fusion characteristics of the reshaped deployment network. S33. Based on the fusion characteristics of the reshaped teacher network and the reshaped deployment network, calculate the edge-aware loss and update the edge deployment network.

[0012] Furthermore, S33 includes the following sub-steps: S331. Based on the fusion characteristics of the reshaped teacher network and the fusion characteristics of the reshaped deployment network, determine the relationship between sample representations in the updated teacher network and the relationship between sample representations in the edge deployment network. S332. Calculate the relationship difference matrix based on the relationship between the sample representations in the updated teacher network and the relationship between the sample representations in the edge deployment network. S333, Calculate the distance from the sample representation to the decision edge; S334. Calculate the edge-aware loss based on the relational difference matrix and the distance from the sample representation to the decision edge, and update the edge deployment network.

[0013] Furthermore, in S331, the relationship between sample representations in the updated teacher network... The expression is: ; in, Indicating the first in the teacher network i The feature vector of each sample Indicating the first in the teacher network j Each sample feature vector Indicates the calculation of the first i The first sample and the first j A function relating the relationships between samples. This indicates that the vector is transposed.

[0014] Furthermore, in S331, the relationship between sample representations in the edge-deployed network. The expression is: ; in, Indicates the corresponding number after reshaping i The feature vector of each sample Indicates the reshaped first j The feature vector of each sample Indicates the calculation of the first i The first sample and the first j A function relating the relationships between samples. This indicates that the vector is transposed.

[0015] Furthermore, in S332, the relational difference matrix The expression is: ; in, Indicating the first in the teacher network i The first sample and the first j The relationship between individual samples Indicates the first in the network deployment i The first sample and the first j The relationship between individual samples Indicates the batch size.

[0016] Furthermore, in S333, the distance from the sample representation to the decision edge... The expression is: ; in, Indicating the first in the teacher network i The information entropy corresponding to each sample Indicating the first in the teacher network i The output vector corresponding to each sample. This represents the softmax function used for normalization. Represents a logarithmic function.

[0017] Furthermore, in S334, edge-aware loss The expression is as follows: In this embodiment of the invention, S2 includes the following sub-steps: ; in, Indicates batch size. This represents the distance from the sample representation to the decision edge. This represents the softmax function used for normalization. Indicates the first i Differences in the relationships between individual samples.

[0018] The beneficial effects of this invention are: (1) This invention proposes a self-supervised heterogeneous fusion network based on multimodal data such as heart sounds, electrocardiograms and echocardiograms, which breaks through the limitations of traditional homogeneous fusion and realizes cross-modal deep complementarity of cardiac electrical activity, mechanical activity and structural function to obtain global characterization; (2) The present invention constructs a meta-learning knowledge distillation model to overcome the dependence of traditional deep learning on large-scale labeled data and achieve high-precision model training under small sample conditions; (3) The present invention designs an edge-aware distillation mechanism for incomplete modal data, enhances the model’s adaptability and generalization ability to partial modal missing in resource-constrained scenarios, and constructs an intelligent screening system with high accuracy and strong robustness. Attached Figure Description

[0019] Figure 1 A flowchart of a method for processing information on congenital heart disease in children based on incomplete multimodal approaches; Figure 2 This is the overall technology roadmap; Figure 3 This is a diagram of the architecture of the Multimodal Heterogeneous Dynamic Fusion Network (HDFN) module. Figure 4 A multimodal semantic relation representation graph; Figure 5 This is a schematic diagram of the feedback from a knowledge distillation teaching experiment based on meta-learning. Figure 6 This is a schematic diagram of edge-sensing distillation. Detailed Implementation

[0020] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0021] like Figure 1 As shown, this invention provides a method for processing information on congenital heart disease in children based on incomplete multimodal approaches, comprising the following steps: S1. Acquire signal data and use LSTM and multimodal heterogeneous dynamic fusion network to perform feature extraction and modal interaction to obtain the interaction information between various modes; S2. Input the interaction information between the various modalities into the teacher network and student network for updating; S3. Use the updated teacher network to update the edge deployment network; S4. Utilize the updated edge deployment network and output the results.

[0022] This invention focuses on common medical examination data related to congenital heart disease, such as heart sounds, electrocardiograms, and echocardiograms. It explores a novel screening method for congenital heart disease based on small sample size, multimodality, and strong perception from the perspectives of knowledge distillation and edge perception. The invention establishes a comprehensive processing mechanism for multimodal data such as heart sounds, electrocardiograms, and echocardiograms to achieve intelligent screening for congenital heart disease.

[0023] First, the data needs to be preprocessed. Then, the input modal data is encoded to extract the features of each modality. The model is pre-trained through self-supervised learning. The output of the pre-trained model is then used as the input to the meta-learning and edge-aware distillation model. The edge-aware module processes the missing views and fuses the features of the existing data.

[0024] The overall technical route is roughly as follows: Figure 2 As shown in the diagram. X1, X2, and X3 represent the input data, LSTM represents the corresponding encoder network, and F1, F2, and F3 represent the corresponding encoded feature vectors. m This indicates that by reshaping the encoder network Linear, Let Y represent the reshaped features, Y represent the output of the teacher network, and y represent the output of the student network.

[0025] MLP fusion This represents the multilayer perceptron fusion module, used to reconstruct the features after passing through the HDFN encoder. HDFN (Hierarchical Deep Fusion Network) is a hierarchical fusion network used for different modalities, specifically as follows: Figure 3 As shown.

[0026] In this embodiment of the invention, S1 includes the following sub-steps: S11. Acquire signal data, specifically including electrocardiogram data, heart sound data, and echocardiogram data; S12. Use LSTM to extract feature data for each modality; S13. Based on the semantic space corresponding to the feature data of each modality, a shared semantic space is generated using a multimodal heterogeneous dynamic fusion network; S14. Using a multilayer perceptron, the shared semantic space is mapped to a low-dimensional space to obtain the interaction information between various modalities.

[0027] This invention first preprocesses the input signal data using methods such as low-pass filtering to remove high-frequency noise, and then extracts heart sound feature data X using Python code. a and the corresponding audio and video modal data features are used as input data (X) t ,X v Then, a Long-Shot-Term Memory (LSTM) network is used to extract the temporal features, thereby obtaining the feature vector F input into the model network. a F v Finally, the feature vector (F_ECG) will be obtained and input into the model network. a ), heart sounds (F) t), echocardiography (F) v Finally, the features of each modality are simultaneously used as input for dynamic heterogeneous fusion. The specific fusion process is as follows: Figure 3 As shown.

[0028] F t F represents the extracted ECG feature vector. v F represents the extracted feature vector from the echocardiogram. a This represents the extracted heart sound feature vector, where T represents electrocardiogram features, A represents heart sound features, V represents echocardiogram features, and F represents heart sound features. uni F represents the feature vector after encoding T, A, and V. bi denoted as TA, TV, and AV respectively, TAV represents the feature vectors fused from the three levels, and MLP represents a multilayer perceptron network that further fuses the features after the three are fused.

[0029] First, hierarchical heterogeneous recombination is required here. For data with M modalities, a total of M levels of heterogeneous fusion are needed to enable the model to learn a relatively ideal multimodal representation. Thus, three levels of modal fusion representations are obtained. Since single-modality data does not require fusion, it is directly used as the single-modality level representation (F...). uni Dual-modal representation (F) bi ) is a pairwise combination of single-modal representations, while the three-modal level representation (F) tri () is a combination of representations of three modalities. According to the theory of multimodal representation learning, the feature representation of each modality... Each of the features They all belong to their own semantic space S m That is From a mathematical perspective, the feature space of a multimodal system can be mapped to a space S containing all modal features. M Furthermore, the feature space of a multimodal system must necessarily share a semantic space S. share Then, to better learn the joint distribution of multimodal representations, an MLP neural network is used to map the multimodal representations from a high-dimensional space to a low-dimensional space, thereby obtaining the interaction information between modalities. The specific relationships are represented as follows: Figure 4 As shown, S t F represents the semantic space of heart sounds t S represents the heart sound. a F represents the semantic space of electrocardiogram. a S represents the eigenvector of electrocardiogram. v F represents the semantic space of echocardiography. v Indicates echocardiography, S share S represents a shared semantic space.M The semantic space that represents itself.

[0030] In this embodiment of the invention, S2 includes the following sub-steps: S21. Copy the student network and update the copied student network using the training set; S22. Calculate the loss of the updated student network using the test set, use the loss as a feedback signal to update the teacher network, and discard the updated student network.

[0031] Meta-learning employs the core idea of ​​enabling models to adjust hyperparameters, thereby quickly learning new tasks based on existing knowledge. Meta-learning generally consists of two parts: a meta-learner f... m One is the internal learner f i The former learns how to update the latter, enabling the latter to effectively learn new tasks. In f m With the help of f i The process of training on the task is called the inner loop. After the loop updates, f i Let f i '(f m For meta-learning, its optimization goal is to make the updated inner learner, f, more efficient. i '(f m This approach achieves better performance by using a meta-loop to perform gradient descent iterations on specific datasets to complete optimization. The meta-learning-based knowledge distillation model allows the teacher model to adjust its parameters during distillation, changing the output based on the student's performance on an independent test set (Quiz Set). For each iteration, a copy of the student network S, denoted as S', is first made. Then, training data is sampled from the training set and updated using the regular distillation loss. Next, the loss of S' on these samples is calculated from independent test sets, and this loss is used as feedback to update the teacher model. Finally, S' is discarded, and the updated teacher model is used to distill S on the same batch of training data, thus enabling the teacher model to receive feedback from the student model in a differentiable manner and adjust its parameters. The specific process is as follows: Figure 5 As shown.

[0032] In this embodiment of the invention, S3 includes the following sub-steps: S31. Transmit the interaction information between modalities from the updated teacher network to the edge deployment network; S32. Reshape the fusion characteristics of the teacher network and the fusion characteristics of the deployment network to obtain the fusion characteristics of the reshaped teacher network and the fusion characteristics of the reshaped deployment network. S33. Based on the fusion characteristics of the reshaped teacher network and the reshaped deployment network, calculate the edge-aware loss and update the edge deployment network.

[0033] Edge-aware loss helps deploy networks to focus more on samples near the decision boundary.

[0034] like Figure 2 As shown, the Margin-aware Distillation (MAD) model transmits integrated multimodal information from the teacher network to the edge deployment network. It assumes a mini-batch multimodal input x = {x1, ..., x...} m}, x m ∈R b This represents the data of the m-th modality, where b is the batch size and E is the value of E. t m and E d m Let represent the encoder of the m-th modality in the teacher network and the deployment network, respectively. and This represents the integration characteristics of teachers and the deployment network. Where b is the batch size. It refers to the number of output channels. and It is a spatial dimension. t P r and P d Let represent the task predictor for the teacher, regularization, and deployment network, respectively. t y r and y d R represents the teacher, regularization, and deployment network, respectively. b k The prediction matrix, k is the number of modal categories. While proposing to learn the teacher's knowledge through relation consistency, MAD also proposed to measure class boundaries and guide the unified model to pay more attention to nearby samples. In order to solve the problem of difficult detection caused by the irregularity of the boundary, MAD introduced the classification uncertainty of each sample to reweight its contribution to the total loss. Specifically, it is divided into three steps. (1) First, calculate the relation difference vector g. td ∈R b , where g td By z t and z d Calculations show that MAD first will and Remodeling and Then MAD uses the same correlation function Calculate the relation matrix separately and And in the teacher network and Relationship between sample representations The relationship can be represented as follows: as well as At this point, MAD calculates the difference matrix between the two, and sums each one to obtain... This is used here to represent the teacher network and the deployment network from the first... The difference in the relationship between a sample and other samples in the same batch. (2) MAD is based on calculate He outputs the information entropy of the logit for each sample as its classification uncertainty. Since samples with higher classification uncertainty are closer to the decision boundary, they are more likely to be misclassified. Therefore, It can also be used to indicate from The sample represents the distance to the decision edge. (3) MAD will As the corresponding component Calculate by weight It can also calculate losses. This allows the network deployment to focus more on samples at the decision boundary. The specific implementation is as follows: Figure 6 As shown.

[0035] In this embodiment of the invention, S331 describes the relationship between sample representations in the updated teacher network. The expression is: ; in, Indicating the first in the teacher network i The feature vector of each sample Indicating the first in the teacher network j Each sample feature vector Indicates the calculation of the first i The first sample and the first j A function relating the relationships between samples. This indicates that the vector is transposed.

[0036] In this embodiment of the invention, S331 describes the relationship between sample representations in the edge deployment network. The expression is: ; in, Indicates the corresponding number after reshaping i The feature vector of each sample Indicates the reshaped first j The feature vector of each sample Indicates the calculation of the first i The first sample and the firstj A function relating the relationships between samples. This indicates that the vector is transposed.

[0037] In this embodiment of the invention, in S332, the relational difference matrix The expression is: ; in, Indicating the first in the teacher network i The first sample and the first j The relationship between individual samples Indicates the first in the network deployment i The first sample and the first j The relationship between individual samples Indicates the batch size.

[0038] In this embodiment of the invention, S333 is the distance from the sample representation to the decision edge. The expression is: ; in, Indicating the first in the teacher network i The information entropy corresponding to each sample Indicating the first in the teacher network i The output vector corresponding to each sample. This represents the softmax function used for normalization. Represents a logarithmic function.

[0039] In this embodiment of the invention, in S334, edge-aware loss... The expression is: ; in, Indicates batch size. This represents the distance from the sample representation to the decision edge. This represents the softmax function used for normalization. Indicates the first i Differences in the relationships between individual samples.

[0040] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for processing information on congenital heart disease in children based on incomplete multimodal approaches, characterized in that, Includes the following steps: S1. Acquire signal data and use LSTM and multimodal heterogeneous dynamic fusion network to perform feature extraction and modal interaction to obtain the interaction information between various modes; S2. Input the interaction information between the various modalities into the teacher network and student network for updating; S3. Use the updated teacher network to update the edge deployment network; S4. Utilize the updated edge deployment network and output the results.

2. The method for processing information on congenital heart disease in children based on incomplete multimodal processing according to claim 1, characterized in that, S1 includes the following sub-steps: S11. Acquire signal data, specifically including electrocardiogram data, heart sound data, and echocardiogram data; S12. Use LSTM to extract feature data for each modality; S13. Based on the semantic space corresponding to the feature data of each modality, a shared semantic space is generated using a multimodal heterogeneous dynamic fusion network; S14. Using a multilayer perceptron, the shared semantic space is mapped to a low-dimensional space to obtain the interaction information between various modalities.

3. The method for processing information on congenital heart disease in children based on incomplete multimodal processing according to claim 1, characterized in that, S2 includes the following sub-steps: S21. Copy the student network and update the copied student network using the training set; S22. Calculate the loss of the updated student network using the test set, use the loss as a feedback signal to update the teacher network, and discard the updated student network.

4. The method for processing information on congenital heart disease in children based on incomplete multimodal processing according to claim 1, characterized in that, S3 includes the following sub-steps: S31. Transmit the interaction information between modalities from the updated teacher network to the edge deployment network; S32. Reshape the fusion characteristics of the teacher network and the fusion characteristics of the deployment network to obtain the fusion characteristics of the reshaped teacher network and the fusion characteristics of the reshaped deployment network. S33. Based on the fusion characteristics of the reshaped teacher network and the reshaped deployment network, calculate the edge-aware loss and update the edge deployment network.

5. The method for processing information on congenital heart disease in children based on incomplete multimodal processing according to claim 4, characterized in that, S33 includes the following sub-steps: S331. Based on the fusion characteristics of the reshaped teacher network and the fusion characteristics of the reshaped deployment network, determine the relationship between sample representations in the updated teacher network and the relationship between sample representations in the edge deployment network. S332. Calculate the relationship difference matrix based on the relationship between the sample representations in the updated teacher network and the relationship between the sample representations in the edge deployment network. S333, Calculate the distance from the sample representation to the decision edge; S334. Calculate the edge-aware loss based on the relational difference matrix and the distance from the sample representation to the decision edge, and update the edge deployment network.

6. The method for processing information on congenital heart disease in children based on incomplete multimodal processing according to claim 5, characterized in that, In step S331, the relationship between sample representations in the updated teacher network is described. The expression is: ； in, Indicating the first in the teacher network i The feature vector of each sample Indicating the first in the teacher network j Each sample feature vector Indicates the calculation of the first i The first sample and the first j A function relating the relationships between samples. This indicates that the vector is transposed.

7. The method for processing information on congenital heart disease in children based on incomplete multimodal processing according to claim 5, characterized in that, In S331, the relationship between sample representations in the edge deployment network The expression is: ； in, Indicates the corresponding number after reshaping i The feature vector of each sample Indicates the reshaped first j The feature vector of each sample Indicates the calculation of the first i The first sample and the first j A function relating the relationships between samples. This indicates that the vector is transposed.

8. The method for processing information on congenital heart disease in children based on incomplete multimodal processing according to claim 5, characterized in that, In S332, the relational difference matrix The expression is: ； in, Indicating the first in the teacher network i The first sample and the first j The relationship between individual samples Indicates the first in the network deployment i The first sample and the first j The relationship between individual samples Indicates the batch size.

9. The method for processing information on congenital heart disease in children based on incomplete multimodal processing according to claim 5, characterized in that, In S333, the distance from the sample representation to the decision edge The expression is: ； in, Indicating the first in the teacher network i The information entropy corresponding to each sample Indicating the first in the teacher network i The output vector corresponding to each sample. This represents the softmax function used for normalization. Represents a logarithmic function.

10. The method for processing information on congenital heart disease in children based on incomplete multimodal processing according to claim 4, characterized in that, In S334, edge sensing loss The expression is: ； in, Indicates batch size. This represents the distance from the sample representation to the decision edge. This represents the softmax function used for normalization. Indicates the first i Differences in the relationships between individual samples.

Images