A multi-view two-modal congenital heart disease classification system and a training method thereof
By employing modal adaptive coding and probabilistic completion networks to perform differential coding and alignment fusion of B-mode and Color Doppler modes, combined with dynamic topological coupling and confidence-weighted pooling, the problem of missing view modalities in ultrasound classification of congenital heart disease is solved, thereby improving classification accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTHWEST PETROLEUM UNIV
- Filing Date
- 2026-05-15
- Publication Date
- 2026-06-12
AI Technical Summary
Existing technologies for ultrasound classification of congenital heart disease suffer from issues such as missing views and modalities. They lack a unified missing perception mechanism, cannot effectively identify missing states, lack reliable cross-modal completion methods, cannot distinguish between valid and invalid views during fusion, are susceptible to noise interference, and cannot adapt to individual differences in cardiac malformations, resulting in low fusion accuracy.
A modality adaptive coding network, a probabilistic completion network, a view-modality alignment fusion network, and a dynamic topology coupling network are employed, combined with a confidence-weighted pooling module, to achieve differentiated encoding, probabilistic completion, and dynamic alignment fusion of B-mode and Color Doppler modalities. Confidence-weighted pooling is performed using validity masks and completion uncertainties to improve classification accuracy.
It effectively improves the discriminative power and classification accuracy of congenital heart disease features, stably restores missing information, corrects feature-level spatial offset, adapts to individualized anatomical variations, enhances the accuracy and robustness of view fusion under pathological conditions, suppresses interference from invalid views, and improves the reliability of classification decisions.
Smart Images
Figure CN122200201A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image processing technology, specifically to a multi-view bimodal classification system for congenital heart disease and its training method. Background Technology
[0002] B-mode and Color Doppler are two modalities of cardiac ultrasound. B-mode ultrasound data is suitable for interpreting anatomical structures, while Color Doppler ultrasound data is suitable for interpreting blood flow distribution. Each can only reflect the cardiovascular status locally. However, current ultrasound classification studies of congenital heart disease are mostly limited to a single view or a single modality, resulting in insufficient information utilization and limited classification accuracy. Although some studies use multi-view and multi-modal data, they only use simple feature stitching, which can easily lead to a surge in feature dimensions and a decline in model performance.
[0003] Clinical data acquisition commonly suffers from view and modality loss issues. Existing technologies lack a unified loss detection mechanism, making it difficult to effectively identify missing states. Reliable cross-modal completion methods are also lacking, hindering the recovery of missing features. Furthermore, the inability to distinguish between valid and invalid views during fusion makes them susceptible to noise interference. Additionally, existing methods rely on static anatomical topological priors, failing to adapt to individual differences in cardiac malformations, resulting in low fusion accuracy. Deterministic completion methods do not consider completion uncertainties, lacking confidence-based decision-making. Spatial misalignment between B-mode and Color Doppler deep features remains uncorrected, severely impacting fusion performance. Summary of the Invention
[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a multi-view, two-modal classification system for congenital heart disease and its training method, which solves the problem of low classification accuracy in clinical scenarios such as incomplete view acquisition and modality loss.
[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: Firstly, a multi-view, bimodal classification system for congenital heart disease includes: A modal adaptive coding network is used to encode the ultrasound data of two modes in each view to obtain the features of the two modes in each view; A probabilistic completion network is used to combine a validity mask to probabilistically complete the missing modalities of each view, thereby obtaining the completion features of the two modalities of each view and the completion uncertainty of each view. The view-modal alignment and fusion network is used to align and fuse the completion features of two modalities of the same view to obtain the fused features of each view; Dynamic topology coupling network is used to couple the fusion features of each view to obtain the coupling features of each multi-view; The confidence-weighted pooling module is used to perform confidence-weighted pooling on the coupled features of each multi-view based on the validity mask and the completion uncertainty of each view, so as to obtain the global fusion features. A classifier is used to obtain the classification result of congenital heart disease based on global fusion features.
[0006] Furthermore, the two modes are: B-mode and Color Doppler; The validity mask is preset by the user based on the presence or absence of the two modalities in each view; For views , For the view sequence number, its validity mask includes: view B-mode validity mask If the view The value is 1 if the B-mode exists, and 0 otherwise; view Color Doppler validity mask If the view The value is 1 if the Color Doppler exists, and 0 otherwise. view Validity mask , ,in For logical OR operation.
[0007] Furthermore, the modality adaptive coding network includes: A shared encoding unit is used to extract cross-modal underlying texture information for each view based on B-mode ultrasound data and Color Doppler ultrasound data; The dynamic feature extraction unit is used to extract dynamic representations of blood flow changes based on cross-modal basic texture information to obtain the Color Doppler features before alignment. The structural feature extraction unit is used to extract anatomical structural features based on cross-modal basic texture information to obtain the B-mode features before alignment; The modality alignment unit is used to map the unaligned Color Doppler features and B-mode features to a unified semantic space and output the aligned Color Doppler features and B-mode features.
[0008] Furthermore, the probabilistic completion network includes a first probabilistic completion subnetwork that takes B-mode features as input and outputs Color Doppler completion features, and a second probabilistic completion subnetwork that takes Color Doppler features as input and outputs B-mode completion features. The first probabilistic completion subnetwork and the second probabilistic completion subnetwork have the same structure, both including: a first encoder module, a first decoder module, a second encoder module, and a second decoder module connected in series.
[0009] Furthermore, both the first encoder module and the second encoder module calculate the mean vector and standard deviation vector of their respective input features, and both calculate their respective output latent sampling vectors using the following formula: ; Where z is the potential sampling vector output by the first encoder module or the second encoder module. The mean vector of features is input to either the first encoder module or the second encoder module. ϵ is the standard deviation vector of the input features for the first encoder module or the second encoder module, ϵ is a random noise vector that follows a standard normal distribution with a mean of 0 and a covariance of the identity matrix, and ⨀ is element-wise multiplication.
[0010] Furthermore, the expressions for both the first decoder module and the second decoder module are: ; in, The features output by the first decoder module or the second decoder module. For the potential sample vector of the first decoder module or the second decoder module, For the validity mask of the input first decoder module or second decoder module, This is the decoding mapping function.
[0011] Furthermore, using the probabilistic completion network, combined with the view... Validity mask , for view The missing modalities are probabilistically completed to obtain the view. Two-modal completion features and views The completion of uncertainty includes the following steps: A1. If the view If both modes are complete, proceed to step A2; otherwise, proceed to step A3.
[0012] A2, View Color Doppler features as views The Color Doppler completion feature will change the view B-mode features as views The B-mode completion feature will change the view. The completion uncertainty is set to a constant close to 0 (e.g., 0 or 10).-6 In this embodiment, is set to 0, and the process ends.
[0013] A3, if the view If the B-mode is present but the Color Doppler mode is missing, proceed to step A4; otherwise, proceed to step A5.
[0014] A4. View B-mode features as views The B-mode completion features are then used to complete the subnetwork with the first probability, and the view is then... The completion uncertainty of the first candidate is used as a view. Complete the uncertainty, and then end: View The B-mode features are input to its first encoder module to convert the view... Validity mask Simultaneously serving as a validity mask for its first and second decoder modules, the features output by its first decoder module are used as the view. The Color Doppler completion features are used to calculate the view according to the following formula. Uncertainty regarding the completion of the first candidate: ; in, For view The uncertainty of completing the first candidate For view The standard deviation vector obtained when the B-mode features are input into the first encoder module of the first probability completion subnetwork. for No. One element, for Dimensions.
[0015] A5. View Color Doppler features as views The Color Doppler feature is completed, and the second probability is used to complete the subnetwork as follows, and the view is then... The completion uncertainty of the second candidate is used as a view. Complete the uncertainty, and then end: View The Color Doppler features are input into its first encoder module to convert the view... Validity mask Simultaneously serving as a validity mask for its first and second decoder modules, the features output by its first decoder module are used as the view. The B-mode completion features are calculated according to the following formula. Uncertainty of completing the second candidate: ; in, For view The uncertainty of the completion of the second candidate, For view The standard deviation vector obtained when the ColorDoppler features are input into the first encoder module of the second probability completion subnetwork. for No. Each element.
[0016] Furthermore, the view-modal alignment and fusion network aligns and fuses the views. The method for completing features for each modality includes the following steps: B1. The view is determined by the following formula. The Color Doppler completion features are used for spatial remapping to align with the view. B-mode completion feature alignment: ; in, For view Color Doppler completion features aligned with B-mode completion features. Remapping the front view for space Color Doppler completion features, For view Spatial remapping function; B2. Merge views using the following formula. The B-mode completion feature and the ColorDoppler completion feature aligned with the B-mode completion feature are used to obtain the view. Fusion characteristics: ; in, For view The fusion characteristics For view B-mode completion features, For view The fusion coefficient.
[0017] Furthermore, the expression for the dynamic topological coupling network is: ; ; ; in, For the first A multi-view coupling feature, The second sequence number of the view. For the total number of views, This is the fusion attention weight matrix after dynamic topological coupling. Based on the attention weight matrix, For consistency scoring function, For view The fusion characteristics For view The fusion characteristics For the validity mask matrix, to For vision Figure 1 to Color Doppler modal validity mask, to For vision Figure 1 to B-mode validity mask.
[0018] Furthermore, the confidence-weighted pooling module performs confidence-weighted pooling on each multi-view coupled feature based on the validity mask and the completion uncertainty of each view to obtain the global fused feature. The method includes the following steps: C1. Calculate the score of each original view based on the multi-view coupling characteristics using the following formula: ; in, For the first Original view score, Assign a score to the view level. For the first A multi-view coupling feature; C2. Calculate the score for each corrected view using the following formula, based on the validity mask and the completion uncertainty of each view: ; in, For the first Correct view score, For view The completion uncertainty, For view Validity mask, The penalty coefficient is... This is the mask suppression constant; C3. Calculate the weights of each normalized view based on the scores of each corrected view using the following formula: ; in, For the first Normalize view weights, For the first Correct view score, The second sequence number of the view. For the total number of views, It is a natural exponential function; C4. Using the following formula, based on the weights of each normalized view, perform confidence-weighted pooling on each multi-view coupled feature to obtain the global fused feature: ; in, This is a global fusion feature.
[0019] Secondly, a training method for a multi-view bimodal congenital heart disease classification system, used to train the aforementioned multi-view bimodal congenital heart disease classification system, includes the following steps: S1. Construct a training sample set, where each training sample has: a label for the congenital heart disease classification result, a complete scene, and... A missing scenario It is a positive integer greater than or equal to 2; The complete scene has at least two views, each of which contains B-mode ultrasound data and Color Doppler ultrasound data; Each of the missing scenes is formed by randomly occluding certain views of the complete scene using B-mode ultrasound data or ColorDoppler ultrasound data. S2. Construct and combine the reconstruction loss function, cross-modal consistency loss function and cyclic consistency loss function to form a joint loss function. Train the probability completion network based on the complete scene of each training sample in the training sample set. S3. Construct the multi-scenario missing consistency loss function using the following formula: ; ; in, For multiple scenarios where consistency is lacking, For missing scene numbers, For the first The weight of the loss for each missing scenario. For KL divergence calculation, The classifier provides the distribution of congenital heart disease classifications for the complete scene. For the classifier for the first The distribution of congenital heart disease classification obtained from missing scenarios, For the first The mean of the uncertainty of view completion for each missing scene; S4. Using the following formula, combine the multi-scene missing consistency loss function, the cross-entropy loss function, and the joint loss function from S2 to form the total loss function. Then, using the gradient descent algorithm, jointly train the multi-view bimodal congenital heart disease classification system based on the training sample set: ; in, For the total loss function, Let cross-entropy be the loss function. Let S2 be the joint loss function.
[0020] Further, S2 includes the following steps: S21. For the first probability completion subnetwork: The B-mode features of each view in each complete scene, obtained by the modality adaptive coding network, are input into its first encoder module and used as known modality features. The latent sampling vector output by its first encoder module is used as the first latent sampling vector. The features output by its first decoder module are used as cross-modal completion features. The latent sampling vector output by its second encoder module is used as the second latent sampling vector. The features output by its second decoder module are used as the known modal features for cyclic reconstruction. Using the same view's Color Doppler features as cross-modal target features. ; For the second probability completion subnetwork: The Color Doppler features of each view in each complete scene, obtained by the modality adaptive coding network, are input into its first encoder module and used as known modality features. The latent sampling vector output by its first encoder module is used as the first latent sampling vector. The features output by its first decoder module are used as cross-modal completion features. The latent sampling vector output by its second encoder module is used as the second latent sampling vector. The features output by its second decoder module are used as the known modal features for cyclic reconstruction. Using B-mode features from the same view as cross-modal target features ; S22. Construct and combine the reconstruction loss function, cross-modal consistency loss function, and cyclic consistency loss function using the following formulas to form a joint loss function: ; ; ; ; in, For the joint loss function, To reconstruct the loss function, For cross-modal consistency loss function, Let the cycle consistency loss function be... For cross-modal consistency loss weights, For cycle consistency loss weights, For L1 norm operations, Square of the 2-norm operation; S23. Train the probability completion network based on the complete scene and joint loss function of each training sample in the training sample set.
[0021] The beneficial effects of this invention are as follows: (1) This invention performs differentiated encoding on the B-mode that provides anatomical structure and the Color Doppler mode that provides blood flow information, making full use of the complementary information of structure and blood flow, breaking through the limitations of single view and single modality information, and effectively improving the discriminativeness and classification accuracy of congenital heart disease features; (2) The probabilistic completion network is based on the probabilistic principle of conditional variational inference and the reparameterization mechanism to complete the missing modal features. Combined with reconstruction loss, cross-modal consistency loss and cyclic consistency loss for joint optimization, it can stably recover missing information. (3) The present invention performs alignment on B-mode and Color Doppler mode multiple times, effectively corrects feature-level spatial offset generated during imaging and feature processing, improves the accuracy of same-view fusion, and avoids the attenuation of discrimination information caused by feature misalignment; (4) The present invention adopts a dynamic topological coupling mechanism to combine the views in a coupled manner, and adaptively updates the cross-view fusion weights in a data-driven manner, thus getting rid of the limitations of static anatomical topological priors. It can adapt to individualized anatomical variations such as cardiac transposition and chamber ratio imbalance, and improve the accuracy and robustness of view fusion under pathological conditions. (5) In the confidence weighted pooling stage, the present invention utilizes the completion uncertainty of the probability completion stage and combines it with the validity mask to realize confidence weighted pooling, which can effectively suppress the interference of invalid views and low confidence completion features, and improve the reliability and stability of classification decision. Attached Figure Description
[0022] Figure 1 This is a structural diagram of a multi-view, two-modal classification system for congenital heart disease according to an embodiment of the present invention; Figure 2 This is a structural diagram of the modality adaptive coding network according to an embodiment of the present invention; Figure 3 This is a structural diagram of the first or second probability completion subnetwork in an embodiment of the present invention; Figure 4 This is a flowchart illustrating a training method for a multi-view, two-modal congenital heart disease classification system according to an embodiment of the present invention. Detailed Implementation
[0023] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0024] like Figure 1 As shown, in one embodiment of the present invention, a multi-view bimodal classification system for congenital heart disease includes: A modal adaptive coding network is used to encode the ultrasound data of two modes in each view to obtain the features of the two modes in each view; A probabilistic completion network is used to combine a validity mask to probabilistically complete the missing modalities of each view, thereby obtaining the completion features of the two modalities of each view and the completion uncertainty of each view. The view-modal alignment and fusion network is used to align and fuse the completion features of two modalities of the same view to obtain the fused features of each view; Dynamic topology coupling network is used to couple the fusion features of each view to obtain the coupling features of each multi-view; The confidence-weighted pooling module is used to perform confidence-weighted pooling on the coupled features of each multi-view based on the validity mask and the completion uncertainty of each view, so as to obtain the global fusion features. A classifier is used to obtain the classification result of congenital heart disease based on global fusion features.
[0025] The two modes described in this invention are: B-mode and Color Doppler.
[0026] B-mode is used for interpreting anatomical structures, and Color Doppler is used for interpreting blood flow distribution.
[0027] The validity mask is preset by the user based on the presence or absence of the two modalities in each view; for each view... , For the view sequence number, its validity mask includes: view B-mode validity mask If the view The value is 1 if the B-mode exists, and 0 otherwise; view Color Doppler validity mask If the view The value is 1 if the Color Doppler exists, and 0 otherwise. view Validity mask , ,in For logical OR operation.
[0028] In this embodiment, the multiple views include at least two of the following: parasternal long-axis view, aortic arch long-axis view, great artery short-axis view, inferior vena cava-aortic short-axis view, inferior vena cava-aortic long-axis view, biatrial view, four-chamber view, and left ventricular short-axis view.
[0029] Modality adaptive coding networks, such as Figure 2 As shown, it includes: A shared encoding unit is used to extract cross-modal underlying texture information for each view based on B-mode ultrasound data and Color Doppler ultrasound data; The dynamic feature extraction unit is used to extract dynamic representations of blood flow changes based on cross-modal basic texture information to obtain the Color Doppler features before alignment. The structural feature extraction unit is used to extract anatomical structural features based on cross-modal basic texture information to obtain the B-mode features before alignment; The modality alignment unit is used to map the unaligned Color Doppler features and B-mode features to a unified semantic space and output the aligned Color Doppler features and B-mode features.
[0030] The shared encoding unit of this invention adopts a visual encoder architecture commonly used in the field of medical image processing technology, including but not limited to convolutional neural network encoders, self-attention-based Transformer encoders, or hybrid encoders of convolution and Transformer; the dynamic feature extraction unit and the structural feature extraction unit are constructed by a lightweight task head, initialized with publicly available pre-trained parameters, and retrained or fine-tuned in combination with ultrasound data to improve training convergence stability and clinical scenario adaptability; the modality alignment unit can also be implemented by a spatial mapping network in this field, which will not be elaborated further.
[0031] The design of the modality adaptive coding network enables the present invention to perform differential coding on the B-mode that provides anatomical structure and the Color Doppler mode that provides blood flow information. It makes full use of the complementary information of structure and blood flow, breaks through the limitations of single view and single modality information, and effectively improves the discriminativeness and classification accuracy of congenital heart disease features.
[0032] The probabilistic completion network is based on the probabilistic principles of conditional variational inference and the reparameterization mechanism. It includes a first probabilistic completion subnetwork that takes B-mode features as input and outputs Color Doppler completion features, and a second probabilistic completion subnetwork that takes Color Doppler features as input and outputs B-mode completion features.
[0033] The first probability completion subnetwork and the second probability completion subnetwork have the same structure, such as Figure 3 As shown, each module includes: a first encoder module, a first decoder module, a second encoder module, and a second decoder module connected in series.
[0034] Both the first encoder module and the second encoder module calculate the mean vector and standard deviation vector of their respective input features, and both calculate their respective output latent sampling vectors using the following formula: ; Where z is the potential sampling vector output by the first encoder module or the second encoder module. The mean vector of features is input to either the first encoder module or the second encoder module. ϵ is the standard deviation vector of the input features for the first encoder module or the second encoder module, ϵ is a random noise vector that follows a standard normal distribution with a mean of 0 and a covariance of the identity matrix, and ⨀ is element-wise multiplication.
[0035] The expressions for both the first decoder module and the second decoder module are: ; in, The features output by the first decoder module or the second decoder module. For the potential sample vector of the first decoder module or the second decoder module, For the validity mask of the input first decoder module or second decoder module, This is the decoding mapping function.
[0036] The first decoder module and the second decoder module of this invention mainly perform statistical calculations of mean and standard deviation, which are readily implemented by those skilled in the art; and the functions of the first decoder module and the second decoder module are determined by the decoding mapping function. To achieve this, the corresponding physical structure uses MLP (Multilayer Perceptron) + CNN (Convolutional Neural Network), which requires training and learning.
[0037] This invention uses a probabilistic completion network method to view For example: combining views Validity mask , for view The missing modalities are probabilistically completed to obtain the view. Two-modal completion features and views The completion of uncertainty includes the following steps: A1. If the view If both modes are complete, proceed to step A2; otherwise, proceed to step A3.
[0038] A2, View Color Doppler features as views The Color Doppler completion feature will change the view B-mode features as views The B-mode completion feature will change the view. The completion uncertainty is set to a constant close to 0 (e.g., 0 or 10). -6 In this embodiment, is set to 0, and the process ends.
[0039] A3, if the view If the B-mode is present but the Color Doppler mode is missing, proceed to step A4; otherwise, proceed to step A5.
[0040] A4. View B-mode features as views The B-mode completion features are then used to complete the subnetwork with the first probability, and the view is then... The completion uncertainty of the first candidate is used as a view. Complete the uncertainty, and then end: View The B-mode features are input to its first encoder module to convert the view... Validity mask Simultaneously serving as a validity mask for its first and second decoder modules, the features output by its first decoder module are used as the view. The Color Doppler completion features are used to calculate the view according to the following formula. Uncertainty regarding the completion of the first candidate: ; in, For view The uncertainty of completing the first candidate For view The standard deviation vector obtained when the B-mode features are input into the first encoder module of the first probability completion subnetwork. For the first One element, for Dimensions.
[0041] A5. View Color Doppler features as views The Color Doppler feature is completed, and the second probability is used to complete the subnetwork as follows, and the view is then... The completion uncertainty of the second candidate is used as a view. Complete the uncertainty, and then end: View The Color Doppler features are input into its first encoder module to convert the view... Validity mask Simultaneously serving as a validity mask for its first and second decoder modules, the features output by its first decoder module are used as the view. The B-mode completion features are calculated according to the following formula. Uncertainty of completing the second candidate: ; in, For view The uncertainty of the completion of the second candidate, For view The standard deviation vector obtained when the ColorDoppler features are input into the first encoder module of the second probability completion subnetwork. for No. Each element.
[0042] View-Modal Alignment and Fusion Network Aligns and Fusions Views The method for completing features for each modality includes the following steps: B1. The view is determined by the following formula. The Color Doppler completion features are used for spatial remapping to align with the view. B-mode completion feature alignment: ; in, For view Color Doppler completion features aligned with B-mode completion features. Remapping the front view for space Color Doppler completion features, For view The space remapping function.
[0043] Spatial remapping functions are used for feature-level micro-correction to compensate for spatial misalignment caused by breathing, heartbeat, and probe micro-movements; the implementation methods are not limited to downsampling interpolation, spatial remapping, etc.
[0044] This invention performs multiple alignments on the B-mode and Color Doppler modes, effectively correcting feature-level spatial offsets generated during imaging and feature processing, improving the accuracy of in-view fusion, and avoiding the attenuation of discrimination information caused by feature misalignment.
[0045] B2. Merge views using the following formula. The B-mode completion feature and the ColorDoppler completion feature aligned with the B-mode completion feature are used to obtain the view. Fusion characteristics: ; in, For view The fusion characteristics For view B-mode completion features, For view The fusion coefficient.
[0046] The fusion coefficient is a learnable parameter with an initial value of 1.
[0047] Furthermore, the expression for the dynamic topological coupling network is: ; ; ; in, For the first A multi-view coupling feature, The second sequence number of the view. For the total number of views, This is the fusion attention weight matrix after dynamic topological coupling. Based on the attention weight matrix, For consistency scoring function, For view The fusion characteristics For view The fusion characteristics For the validity mask matrix, to For vision Figure 1 to Color Doppler modal validity mask, to For vision Figure 1 to B-mode validity mask.
[0048] Consistency scoring function It can be implemented using a neural network with learnable vector inputs and scalar outputs, and is not limited to any type of neural network.
[0049] This invention employs a dynamic topological coupling mechanism to couple the views together, and adaptively updates the cross-view fusion weights in a data-driven manner. This overcomes the limitations of static anatomical topological priors and can adapt to individualized anatomical variations such as cardiac transposition and chamber disproportion, thereby improving the accuracy and robustness of view fusion under pathological conditions.
[0050] The confidence-weighted pooling module performs confidence-weighted pooling on each multi-view coupled feature based on the validity mask and the completion uncertainty of each view to obtain the global fused feature. The method includes the following steps: C1. Calculate the score of each original view based on the multi-view coupling characteristics using the following formula: ; in, For the first Original view score, Assign a score to the view level. For the first A multi-view coupling feature.
[0051] View-level scoring head It can be implemented using a neural network with learnable vector inputs and scalar outputs, and is not limited to any type of neural network.
[0052] C2. Calculate the score for each corrected view using the following formula, based on the validity mask and the completion uncertainty of each view: ; in, For the first Correct view score, For view The completion uncertainty, For view Validity mask, The penalty coefficient is... This is the mask suppression constant.
[0053] The mask suppression constant is used to suppress the score of invalid views to a minimum. To prevent it from entering the efficient competition set, a sufficiently large positive number needs to be chosen. The exemplary experimental value is... .
[0054] Penalty coefficient It can be based on the overall performance in the range Select from within.
[0055] C3. Calculate the weights of each normalized view based on the scores of each corrected view using the following formula: ; in, For the first Normalize view weights, For the first Correct view score, The second sequence number of the view. For the total number of views, It is a natural exponential function.
[0056] C4. Using the following formula, based on the weights of each normalized view, perform confidence-weighted pooling on each multi-view coupled feature to obtain the global fused feature: ; in, This is a global fusion feature.
[0057] In the confidence-weighted pooling stage, this invention utilizes the completion uncertainty from the probability completion stage and combines it with a validity mask to achieve confidence-weighted pooling. This effectively suppresses interference from invalid views and low-confidence completion features, thereby improving the reliability and stability of classification decisions.
[0058] The classifier in this embodiment is not limited to a specific category; a Classifier classifier commonly used in this field can be used.
[0059] This embodiment also provides a training method for a multi-view bimodal congenital heart disease classification system, used to train the aforementioned multi-view bimodal congenital heart disease classification system, such as... Figure 4 As shown, it includes the following steps: S1. Construct a training sample set, where each training sample has: a label for the congenital heart disease classification result (corresponding to the classifier's output), a complete scene, and... A missing scenario It is a positive integer greater than or equal to 2; The complete scene has at least two views, each of which contains B-mode ultrasound data and Color Doppler ultrasound data; Each of the missing scenes is formed by randomly occluding certain views of the complete scene using B-mode ultrasound data or ColorDoppler ultrasound data. S2. Construct and combine the reconstruction loss function, cross-modal consistency loss function, and cyclic consistency loss function to form a joint loss function. Based on the complete scene of each training sample in the training sample set, train the probability completion network, including the following steps: S21. For the first probability completion subnetwork: The B-mode features of each view in each complete scene, obtained by the modality adaptive coding network, are input into its first encoder module and used as known modality features. The latent sampling vector output by its first encoder module is used as the first latent sampling vector. The features output by its first decoder module are used as cross-modal completion features. The latent sampling vector output by its second encoder module is used as the second latent sampling vector. The features output by its second decoder module are used as the known modal features for cyclic reconstruction. Using the same view's Color Doppler features as cross-modal target features. ; For the second probability completion subnetwork: The Color Doppler features of each view in each complete scene, obtained by the modality adaptive coding network, are input into its first encoder module and used as known modality features. The latent sampling vector output by its first encoder module is used as the first latent sampling vector. The features output by its first decoder module are used as cross-modal completion features. The latent sampling vector output by its second encoder module is used as the second latent sampling vector. The features output by its second decoder module are used as the known modal features for cyclic reconstruction. Using B-mode features from the same view as cross-modal target features .
[0060] S22. Construct and combine the reconstruction loss function, cross-modal consistency loss function, and cyclic consistency loss function using the following formulas to form a joint loss function: ; ; ; ; in, For the joint loss function, To reconstruct the loss function, For cross-modal consistency loss function, Let the cycle consistency loss function be... For cross-modal consistency loss weights, For cycle consistency loss weights, For L1 norm operations, Squared for the 2-norm operation.
[0061] and Users can freely adjust the settings based on their level of emphasis on cross-modal consistency and cyclic consistency.
[0062] S23. Train the probability completion network based on the complete scene and joint loss function of each training sample in the training sample set.
[0063] The probabilistic completion network is based on the probabilistic principles of conditional variational inference and the reparameterization mechanism to complete missing modal features. By combining reconstruction loss, cross-modal consistency loss and cyclic consistency loss for joint optimization, it can stably recover missing information.
[0064] S3. Construct the multi-scenario missing consistency loss function using the following formula: ; ; in, For multiple scenarios where consistency is lacking, For missing scene numbers, For the first The weight of the loss for each missing scenario. For KL divergence calculation, The classifier provides the distribution of congenital heart disease classifications for the complete scene. For the classifier for the first The distribution of congenital heart disease classification obtained from missing scenarios, For the first The mean of the uncertainty of view completion for each missing scene.
[0065] S4. Using the following formula, combine the multi-scene missing consistency loss function, the cross-entropy loss function, and the joint loss function from S2 to form the total loss function. Then, using the gradient descent algorithm, jointly train the multi-view bimodal congenital heart disease classification system based on the training sample set: ; in, For the total loss function, Let cross-entropy be the loss function. Let S2 be the joint loss function.
[0066] Cross-entropy loss function The gradient descent algorithm is well-known in this field and will not be elaborated further.
[0067] During training, the probability completion network and classifier can be stabilized in the early stage. In the middle and late stages of training, the parameters of the probability completion network, view-modal alignment fusion network, dynamic topology coupling network, confidence weighted pooling module and classifier are jointly updated. In the formal use stage, only the forward computation process is retained and no additional iterative optimization is introduced to meet the real-time requirements of clinical practice.
[0068] In summary, this invention can effectively solve problems such as missing view modalities, low feature fusion accuracy, and poor model generalization in clinical practice. It maintains high classification accuracy and robustness even in scenarios with incomplete data, adapts to individual differences in cardiac anatomical malformations, and provides a robust technical solution for automated ultrasound diagnosis of congenital heart disease.
[0069] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A multi-view, bimodal classification system for congenital heart disease, characterized in that, include: A modal adaptive coding network is used to encode the ultrasound data of two modes in each view to obtain the features of the two modes in each view; A probabilistic completion network is used to combine a validity mask to probabilistically complete the missing modalities of each view, thereby obtaining the completion features of the two modalities of each view and the completion uncertainty of each view. The view-modal alignment and fusion network is used to align and fuse the completion features of two modalities of the same view to obtain the fused features of each view; Dynamic topology coupling network is used to couple the fusion features of each view to obtain the coupling features of each multi-view; The confidence-weighted pooling module is used to perform confidence-weighted pooling on the coupled features of each multi-view based on the validity mask and the completion uncertainty of each view, so as to obtain the global fusion features. A classifier is used to obtain the classification result of congenital heart disease based on global fusion features.
2. The multi-view bimodal classification system for congenital heart disease according to claim 1, characterized in that, The two modes are: B-mode and Color Doppler; The validity mask is preset by the user based on the presence or absence of the two modalities in each view; For views , For the view sequence number, its validity mask includes: view B-mode validity mask If the view The value is 1 if the B-mode exists, and 0 otherwise; view Color Doppler validity mask If the view The value is 1 if the Color Doppler exists, and 0 otherwise. view Validity mask , ,in For logical OR operation.
3. The multi-view bimodal congenital heart disease classification system according to claim 2, characterized in that, The modality adaptive coding network includes: A shared encoding unit is used to extract cross-modal underlying texture information for each view based on B-mode ultrasound data and Color Doppler ultrasound data; The dynamic feature extraction unit is used to extract dynamic representations of blood flow changes based on cross-modal basic texture information to obtain the Color Doppler features before alignment. The structural feature extraction unit is used to extract anatomical structural features based on cross-modal basic texture information to obtain the B-mode features before alignment; The modality alignment unit is used to map the unaligned Color Doppler features and B-mode features to a unified semantic space and output the aligned Color Doppler features and B-mode features.
4. The multi-view bimodal congenital heart disease classification system according to claim 2, characterized in that, The probabilistic completion network includes a first probabilistic completion subnetwork that takes B-mode features as input and outputs Color Doppler completion features, and a second probabilistic completion subnetwork that takes Color Doppler features as input and outputs B-mode completion features. The first probabilistic completion subnetwork and the second probabilistic completion subnetwork have the same structure, both including: a first encoder module, a first decoder module, a second encoder module, and a second decoder module connected in series.
5. The multi-view bimodal classification system for congenital heart disease according to claim 4, characterized in that, Both the first encoder module and the second encoder module calculate the mean vector and standard deviation vector of their respective input features, and both calculate their respective output latent sampling vectors using the following formula: ； Where z is the potential sampling vector output by the first encoder module or the second encoder module. The mean vector of features is input to either the first encoder module or the second encoder module. ϵ is the standard deviation vector of the input features for the first encoder module or the second encoder module, ϵ is a random noise vector that follows a standard normal distribution with a mean of 0 and a covariance of the identity matrix, and ⨀ is element-wise multiplication.
6. The multi-view bimodal classification system for congenital heart disease according to claim 5, characterized in that, The expressions for both the first decoder module and the second decoder module are: ； in, The features output by the first decoder module or the second decoder module. For the potential sample vector of the first decoder module or the second decoder module, For the validity mask of the input first decoder module or second decoder module, This is the decoding mapping function.
7. The multi-view bimodal classification system for congenital heart disease according to claim 2, characterized in that, The view-modal alignment and fusion network aligns and fuses the views. The method for completing features for each modality includes the following steps: B1. The view is determined by the following formula. The Color Doppler completion features are used for spatial remapping to align with the view. B-mode completion feature alignment: ； in, For view Color Doppler completion features aligned with B-mode completion features. Remapping the front view for space Color Doppler completion features, For view Spatial remapping function; B2. Merge views using the following formula. The B-mode completion feature and the ColorDoppler completion feature aligned with the B-mode completion feature are used to obtain the view. Fusion characteristics: ； in, For view The fusion characteristics For view B-mode completion features, For view The fusion coefficient.
8. The multi-view bimodal classification system for congenital heart disease according to claim 2, characterized in that, The expression for the dynamic topology coupling network is: ； ； in, For the first A multi-view coupling feature, The second sequence number of the view. For the total number of views, This is the fusion attention weight matrix after dynamic topological coupling. Based on the attention weight matrix, For consistency scoring function, For view The fusion characteristics For view The fusion characteristics This is the validity mask matrix.
9. A training method for a multi-view bimodal classification system for congenital heart disease, characterized in that, Training a multi-view bimodal congenital heart disease classification system as described in any one of claims 1 to 8 includes the following steps: S1. Construct a training sample set, in which each training sample has: a label for the classification result of congenital heart disease, a complete scene, and no less than 2 missing scenes; S2. Construct and combine the reconstruction loss function, cross-modal consistency loss function and cyclic consistency loss function to form a joint loss function. Train the probability completion network based on the complete scene of each training sample in the training sample set. S3. Construct a loss function for inconsistent loss across multiple scenarios; S4. Combine the multi-scene missing consistency loss function, the cross-entropy loss function, and the joint loss function of S2 to form the total loss function. Use the gradient descent algorithm to jointly train the multi-view bimodal congenital heart disease classification system based on the training sample set.