Multi-prototype cross-modal contrast learning fetal heart ultrasound image intelligent diagnosis system

Abstract

The invention relates to the technical field of medical image artificial intelligence, in particular to a multi-prototype cross-modal contrast learning fetal heart ultrasound image intelligent diagnosis system which comprises the steps that a to-be-detected fetal heart ultrasound image is obtained through an obtaining module, and a processing module inputs the to-be-detected fetal heart ultrasound image into a pre-trained image diagnosis model; the method comprises the following steps: performing feature extraction on a to-be-detected fetal heart ultrasound image, performing cross-modal multi-prototype comparison according to a feature extraction result to obtain a diagnosis category and a most matched image-text prototype pair, outputting the diagnosis category and the most matched image-text prototype pair by an output module, and finally performing CHD classification and screening according to the diagnosis category and the most matched image-text prototype pair. The problem that the model generalization ability is insufficient due to the fact that a single prototype center is mainly relied on to represent each category and the significant difference of the same category under different conditions is neglected is solved, and multi-category CHD intelligent screening is achieved by utilizing unlabeled images and a small amount of text priori on the premise that a large number of graph-text pairs are not needed.

Description

Technical Field

[0001] This application relates to the field of medical image artificial intelligence technology, and in particular to a multi-prototype cross-modal comparative learning intelligent diagnostic system for fetal cardiac ultrasound images. Background Technology

[0002] The prevalence of fetal CHD (Congenital Heart Disease) and the current status of ultrasound screening: CHD is one of the most common serious malformations in newborns, posing a threat to perinatal life safety and long-term quality of life. Prenatal ultrasound examination is currently the clinically recognized first-line screening method. Standard views such as 4CH (Four-Chamber Heart) and 3VT (Three Vessel Trachea) are routinely obtained at 20–24 weeks of gestation. However, early screening (11–13+6 weeks) can further improve the timing of intervention and the effectiveness of pregnancy management.

[0003] In related technologies, multimodal learning methods attempt to combine ultrasound images with text report information to improve the accuracy and interpretability of CHD diagnosis, but the following technical shortcomings still exist:

[0004] (1) Because high-quality one-to-one correspondence image-text annotation is very expensive and difficult to share, current models mostly rely on pure visual features or strict one-to-one image-text matching annotation (i.e. each image needs to be matched with a detailed text report). Moreover, medical data annotation requires the participation of senior ultrasound physicians, which is time-consuming and expensive. As a result, high-quality annotation data is scarce, model training samples are insufficient, and it is difficult to capture complex semantics and intra-class diversity.

[0005] (2) Ultrasound images of the same CHD category (such as “ventricular septal defect”) may show significant differences due to factors such as gestational age, scanning equipment, and fetal position. However, existing methods usually only set a single prototype center for each category, ignoring the significant differences of the same diagnostic category under different gestational ages, scanning equipment, or fetal position conditions.

[0006] (3) A large number of unlabeled images in clinical practice are not effectively utilized, and traditional supervised learning cannot learn from them, making it difficult to promote in real screening scenarios. Summary of the Invention

[0007] This application provides a multi-prototype cross-modal contrastive learning intelligent diagnostic system for fetal cardiac ultrasound images to solve the problem of insufficient model generalization ability caused by relying mainly on a single prototype center to represent each category and ignoring the significant differences of the same category under different conditions.

[0008] The first aspect of this application provides a multi-prototype cross-modal contrastive learning intelligent diagnostic system for fetal echocardiography images, comprising:

[0009] The acquisition module is used to acquire the fetal echocardiogram image of the fetus to be tested;

[0010] The processing module is used to input the fetal echocardiogram image to be tested into a pre-trained image diagnostic model, extract features from the fetal echocardiogram image through the pre-trained image diagnostic model, and perform cross-modal multi-prototype comparison based on the feature extraction results to obtain the diagnostic category and the best matching image-text prototype pair.

[0011] The output module is used to output the diagnostic category and the most matching image-text prototype pair for CHD classification and screening based on the diagnostic category and the most matching image-text prototype pair.

[0012] According to one embodiment of this application, the above-described multi-prototype cross-modal contrastive learning intelligent diagnostic system for fetal echocardiography images further includes: a training module, wherein the training module is specifically used for:

[0013] Acquire multi-view fetal echocardiogram image datasets and physician report texts;

[0014] Data augmentation processing is performed on each image in the multi-view fetal echocardiography image dataset to obtain a mixed labeled-unlabeled training set, and the doctor's report text is standardized into class-level text descriptions covering each diagnostic category;

[0015] Using a pre-trained Vision Transformer as the visual branch, a multi-dimensional visual feature vector is output for each image in the labeled-unlabeled mixed training set; and using frozen BERT encoding as the text branch, a semantic embedding vector of the same dimension is output for the class-level text description.

[0016] Based on the multidimensional visual feature vector, K-Means clustering is performed on the feature vectors of images of the same type to obtain multiple image prototypes. Based on the semantic embedding vector of the same dimension, multiple text descriptions of the same category are embedded as text prototypes. Based on a preset update strategy, the multiple image prototypes and the text prototypes are updated with preset coefficients to obtain updated multiple image prototypes and updated text prototypes.

[0017] Based on the updated multiple image prototypes and the updated text prototypes, a cross-modal multi-prototype contrastive learning framework is jointly constructed using bidirectional InfoNCE loss and triplet loss. The cross-modal multi-prototype contrastive learning framework guides samples to simultaneously approach the corresponding image prototypes and text prototypes and move away from heterogeneous prototypes, thereby obtaining many-to-many positive and negative sample pairs.

[0018] Based on a preset joint optimization total loss function, the pre-trained image diagnosis model is obtained by training the many-to-many positive and negative sample pairs.

[0019] According to one embodiment of this application, the preset update strategy is as follows:

[0020]

[0021] Among them, h T h is the semantic embedding vector of the current batch of the same type. l For the feature vectors of the current batch of similar images, This represents the mean of semantic embedding vectors of the same type in the current batch. This represents the average feature vector of the current batch of similar images. Let k be the prototype vector of the image of category c. Let be the m-th text prototype vector of category c, cm be the m-th text prototype of category c, c be the category, m be the index number of the text prototype, k be the index number of the image prototype, T(t) be the current state of the text prototype set, T(t+1) be the updated text set, ck be the k-th image prototype of category c, I(t+1) be the updated image prototype set, and I(t) be the current image prototype set.

[0022] According to one embodiment of this application, the bidirectional InfoNCE is:

[0023]

[0024] Where τ is the temperature coefficient, P T P is the set of all text prototypes. I p is a set of image prototypes. T+ p is the text prototype corresponding to the image feature vector of the current sample. I+ z is the image prototype corresponding to the text feature vector of the current sample, and z is the prototype of the candidate negative sample to be compared.

[0025] According to one embodiment of this application, the preset joint optimization total loss function is:

[0026] L=λL sup +(1-λ)(L cmc +L tri +L cons );

[0027] Among them, L sup L is the cross-entropy supervision term. cmc For cross-modal contrast loss, L tri For triplet loss, L cons λ represents the unlabeled consistency loss, and λ is the loss weight parameter.

[0028] The multi-prototype cross-modal contrastive learning fetal echocardiography image intelligent diagnostic system according to embodiments of this application acquires fetal echocardiography images of the fetus to be tested through an acquisition module, and inputs the fetal echocardiography images of the fetus to be tested into a pre-trained image diagnostic model through a processing module. Feature extraction is performed on the fetal echocardiography images of the fetus to be tested, and cross-modal multi-prototype comparison is performed based on the feature extraction results to obtain the diagnostic category and the best matching image-text prototype pair, which is output by the output module. Finally, CHD classification and screening are performed based on the diagnostic category and the best matching image-text prototype pair. In this way, the problems of insufficient model generalization ability caused by mainly relying on a single prototype center to represent each category and ignoring the significant differences of the same category under different conditions are solved. It achieves multi-category CHD intelligent screening without the need for a large number of image-text pairs, using unlabeled images and a small amount of text prior.

[0029] A second aspect of this application provides a multi-prototype cross-modal comparative learning method for intelligent diagnosis of fetal cardiac ultrasound images, comprising the following steps:

[0030] Acquire echocardiogram images of the fetus to be tested;

[0031] The fetal echocardiogram image to be tested is input into a pre-trained image diagnostic model. The pre-trained image diagnostic model extracts features from the fetal echocardiogram image to be tested. Based on the feature extraction results, cross-modal multi-prototype comparison is performed to obtain the diagnostic category and the best matching image-text prototype pair.

[0032] Output the diagnostic category and the best matching image-text prototype pair to perform CHD classification and screening based on the diagnostic category and the best matching image-text prototype pair.

[0033] According to one embodiment of this application, before inputting the fetal echocardiogram image to be tested into a pre-trained image diagnostic model, the method further includes:

[0034] Acquire multi-view fetal echocardiogram image datasets and physician report texts;

[0035] Data augmentation processing is performed on each image in the multi-view fetal echocardiography image dataset to obtain a mixed labeled-unlabeled training set, and the doctor's report text is standardized into class-level text descriptions covering each diagnostic category;

[0036] Using a pre-trained Vision Transformer as the visual branch, a multi-dimensional visual feature vector is output for each image in the labeled-unlabeled mixed training set; and using frozen BERT encoding as the text branch, a semantic embedding vector of the same dimension is output for the class-level text description.

[0037] Based on the multidimensional visual feature vector, K-Means clustering is performed on the feature vectors of images of the same type to obtain multiple image prototypes. Based on the semantic embedding vector of the same dimension, multiple text descriptions of the same category are embedded as text prototypes. Based on a preset update strategy, the multiple image prototypes and the text prototypes are updated with preset coefficients to obtain updated multiple image prototypes and updated text prototypes.

[0038] Based on the updated multiple image prototypes and the updated text prototypes, a cross-modal multi-prototype contrastive learning framework is jointly constructed using bidirectional InfoNCE loss and triplet loss. The cross-modal multi-prototype contrastive learning framework guides samples to simultaneously approach the corresponding image prototypes and text prototypes and move away from heterogeneous prototypes, thereby obtaining many-to-many positive and negative sample pairs.

[0039] Based on a preset joint optimization total loss function, the pre-trained image diagnosis model is obtained by training the many-to-many positive and negative sample pairs.

[0040] According to one embodiment of this application, the preset update strategy is as follows:

[0041]

[0042] Among them, h T h is the semantic embedding vector of the current batch of the same type. I For the feature vectors of the current batch of similar images, This represents the mean of semantic embedding vectors of the same type in the current batch. This represents the average feature vector of the current batch of similar images. Let k be the prototype vector of the image of category c. Let be the m-th text prototype vector of category c, cm be the m-th text prototype of category c, c be the category, m be the index number of the text prototype, k be the index number of the image prototype, T(t) be the current state of the text prototype set, T(t+1) be the updated text set, ck be the k-th image prototype of category c, I(t+1) be the updated image prototype set, and I(t) be the current image prototype set.

[0043] According to one embodiment of this application, the bidirectional InfoNCE is:

[0044]

[0045] Where τ is the temperature coefficient, P T P is the set of all text prototypes. I p is a set of image prototypes. T+ p is the text prototype corresponding to the image feature vector of the current sample. I+z is the image prototype corresponding to the text feature vector of the current sample, and z is the prototype of the candidate negative sample to be compared.

[0046] According to one embodiment of this application, the preset joint optimization total loss function is:

[0047] L=λL sup +(1-λ)(L cmc +L tri +L cons );

[0048] Among them, L sup L is the cross-entropy supervision term. cmc For cross-modal contrast loss, L tri For triplet loss, L cons λ represents the unlabeled consistency loss, and λ is the loss weight parameter.

[0049] The multi-prototype cross-modal contrastive learning method for intelligent diagnosis of fetal echocardiography images according to embodiments of this application acquires fetal echocardiography images of the fetus to be tested through an acquisition module, and inputs the fetal echocardiography images of the fetus to be tested into a pre-trained image diagnosis model through a processing module. Feature extraction is performed on the fetal echocardiography images of the fetus to be tested, and cross-modal multi-prototype comparison is performed based on the feature extraction results to obtain the diagnostic category and the best matching image-text prototype pair, which is output by the output module. Finally, CHD classification and screening are performed based on the diagnostic category and the best matching image-text prototype pair. This solves the problem that the model's generalization ability is insufficient because it mainly relies on a single prototype center to represent each category and ignores the significant differences of the same category under different conditions. It achieves intelligent screening of multiple categories of CHD by using unlabeled images and a small amount of text prior without the need for a large number of image-text pairs.

[0050] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. Attached Figure Description

[0051] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0052] Figure 1 This is a block diagram of a multi-prototype cross-modal contrastive learning intelligent diagnostic system for fetal cardiac ultrasound images provided according to an embodiment of this application;

[0053] Figure 2 This is a schematic diagram of the overall framework of CMPCN (Class-level Multi-Prototype Cross-modal Network) according to an embodiment of this application;

[0054] Figure 3 This is a flowchart illustrating a multi-prototype dynamic update and splitting mechanism according to one embodiment of this application;

[0055] Figure 4 This is a flowchart of a multi-prototype cross-modal comparative learning method for intelligent diagnosis of fetal cardiac ultrasound images according to embodiments of this application. Detailed Implementation

[0056] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0057] The following describes an embodiment of the multi-prototype cross-modal contrastive learning intelligent diagnostic system for fetal echocardiography images according to the accompanying drawings. Addressing the problem mentioned in the background art, where the system primarily relies on a single prototype center to represent each category, neglecting significant differences within the same category under different conditions and thus resulting in insufficient model generalization ability, this application provides a multi-prototype cross-modal contrastive learning intelligent diagnostic system for fetal echocardiography images. In this system, an acquisition module acquires fetal echocardiography images of the target fetus, and a processing module inputs these images into a pre-trained image diagnostic model. Feature extraction is performed on the fetal echocardiography images, and cross-modal multi-prototype comparison is conducted based on the feature extraction results to obtain the diagnostic category and the best-matching image-text prototype pair, which is then output by the output module. Finally, CHD classification and screening are performed based on the diagnostic category and the best-matching image-text prototype pair. This solves the problem of insufficient model generalization ability caused by primarily relying on a single prototype center to represent each category and neglecting significant differences within the same category under different conditions. It achieves intelligent screening of multiple CHD categories using unlabeled images and a small amount of text priors without requiring a large number of image-text pairs.

[0058] Specifically, Figure 1 This is a block diagram illustrating a multi-prototype cross-modal contrastive learning intelligent diagnostic system for fetal cardiac ultrasound images provided in an embodiment of this application.

[0059] like Figure 1 As shown, the multi-prototype cross-modal contrastive learning fetal cardiac ultrasound image intelligent diagnostic system 10 includes: an acquisition module 100, a processing module 200, and an output module 300.

[0060] The module includes an acquisition module 100 for acquiring fetal echocardiogram images; a processing module 200 for inputting the fetal echocardiogram images into a pre-trained image diagnostic model, extracting features from the images, and performing cross-modal multi-prototype comparisons based on the feature extraction results to obtain diagnostic categories and the best-matching image-text prototype pairs; and an output module 300 for outputting diagnostic categories and the best-matching image-text prototype pairs for CHD classification and screening based on the diagnostic categories and the best-matching image-text prototype pairs.

[0061] Specifically, in this application embodiment, in order to effectively address the key challenges in fetal echocardiography image analysis, such as high annotation costs and intra-class diversity, this application embodiment introduces multi-prototype representation and semi-supervised learning strategies. The aim is to fully utilize unannotated images and a small amount of text priors without requiring a large number of image-text pairs, thereby achieving highly robust, multi-category CHD intelligent screening.

[0062] Specifically, firstly, multiple perspectives (such as three-vessel trachea 3VT and four-chamber heart 4CH) of fetal echocardiography images are acquired, and these images are input into a pre-trained image diagnosis model (e.g., the ViT (VisionTransformer) model) for network processing, i.e., feature extraction is performed on the fetal echocardiography images using the ViT model. Secondly, cross-modal multi-prototype comparison is performed based on the feature extraction results to obtain the diagnostic category and the best matching image-text prototype pair. Then, the diagnostic category and the best matching image-text prototype pair are output through the output module 300 to perform CHD classification and screening based on the diagnostic category and the best matching image-text prototype pair, ultimately achieving accurate and reliable diagnosis of fetal heart disease.

[0063] According to one embodiment of this application, the aforementioned multi-prototype cross-modal contrastive learning fetal echocardiography image intelligent diagnostic system 10 further includes: a training module, wherein the training module is specifically used for: acquiring a multi-view fetal echocardiography image dataset and doctor's report text; performing data augmentation processing on each image in the multi-view fetal echocardiography image dataset to obtain a labeled-unlabeled mixed training set, and standardizing the doctor's report text into class-level text descriptions covering various diagnostic categories; and using pre-trained Vision... The Transformer, acting as the visual branch, outputs a multi-dimensional visual feature vector for each image in the labeled-unlabeled mixed training set. A frozen BERT encoding, acting as the text branch, outputs a semantic embedding vector of the same dimension for class-level text descriptions. Based on the multi-dimensional visual feature vectors, K-Means clustering is performed on the feature vectors of images of the same class to obtain multiple image prototypes. Based on the semantic embedding vectors of the same dimension, multiple text descriptions of the same class are embedded as text prototypes. Furthermore, based on a preset update strategy, multiple image prototypes and text prototypes are updated with preset coefficients, resulting in updated image prototypes and text prototypes. Based on the updated image prototypes and text prototypes, a cross-modal multi-prototype contrastive learning framework is jointly constructed using bidirectional InfoNCE loss and triplet loss. This framework guides samples to simultaneously approach their corresponding image prototypes and text prototypes while moving away from heterogeneous prototypes, resulting in many-to-many positive and negative sample pairs. Based on a preset joint optimization total loss function, a pre-trained image diagnostic model is trained using these many-to-many positive and negative sample pairs.

[0064] Specifically, before performing actual diagnosis using a pre-trained image diagnostic model based on the acquired fetal echocardiogram images, the model first needs to be trained so that it can be used to diagnose the fetal echocardiogram images and output a reliable diagnosis of fetal heart disease.

[0065] Specifically, if Figure 2 As shown, firstly, a fetal echocardiogram dataset containing multiple perspectives, including three-vessel trachea (3VT) and four-chamber heart (4CH), and physician report texts were acquired. The fetal echocardiogram dataset mainly includes EP-CHDT8000 (9321 images, 4 categories) and SYF-CHD30000 (31749 images, 13 categories). The physician report texts were standardized using GPT-4o to be class-level text descriptions covering each diagnostic category. Then, data augmentation was performed on each image in the multi-perspective fetal echocardiogram dataset, i.e., the fetal echocardiogram images were uniformly adjusted to 512×512 pixels, and the dataset was divided into training / testing in an 8:2 ratio to retain a large number of unlabeled images for semi-supervised learning. Augmentation was performed randomly, including cropping, flipping, and noise injection, to construct a mixed labeled-unlabeled training set.

[0066] Secondly, using a pre-trained Vision Transformer as the visual branch, it outputs a multi-dimensional (e.g., d-dimensional) visual feature vector for each image in the labeled-unlabeled mixed training set. And using frozen BERT (Bidirectional Encoder Representations from Transformers) encoding as the text branch, it outputs a semantic embedding vector h of the same dimension for the class-level text description. T The parameter spaces of the two branches are aligned through linear mapping, and multi-head cross attention is introduced in the intermediate layer to achieve deep fusion of image and text.

[0067] Again, such as Figure 3 As shown, based on multidimensional visual feature vectors, multiple prototype vectors are maintained for each diagnostic category on both the image and text sides, mainly including: (1) for the same type of image feature vector h I K-Means clustering is performed to initialize image prototypes. During training, multiple image prototypes and text prototypes are updated sequentially with a preset coefficient α based on a preset update strategy, resulting in updated image prototypes and text prototypes. For example, the updated image prototypes and text prototypes are obtained. Image prototype (2) If a sub-clustering structure is detected in the samples covered by the prototype, a new prototype is automatically generated by splitting them; (3) Multiple description embeddings are directly retained on the text side, and after linear transformation to align the scale, multiple text description embedding vectors of the same category are used as text prototypes based on semantic embedding vectors of the same dimension. This is to achieve text prototype initialization; during training, multiple image prototypes and text prototypes are updated in a rolling manner with a preset coefficient of 0.9. The expression corresponding to the preset update strategy is:

[0068]

[0069] Among them, h T h is the semantic embedding vector of the current batch of the same type. I For the feature vectors of the current batch of similar images, This represents the mean of semantic embedding vectors of the same type in the current batch. The feature vectors of the current batch of similar images are used as the average. When a sub-clustering structure appears in the prototype-covered samples, a new prototype is generated by adaptively splitting the samples according to density. Let k be the prototype vector of the image of category c. Let be the m-th text prototype vector of category c, cm be the m-th text prototype of category c, c be the category, m be the index number of the text prototype, k be the index number of the image prototype, T(t) be the current state of the text prototype set, T(t+1) be the updated text set, ck be the k-th image prototype of category c, I(t+1) be the updated image prototype set, and I(t) be the current image prototype set.

[0070] Then, based on the updated multiple image prototypes and updated text prototypes, a cross-modal multi-prototype contrastive learning framework is jointly constructed using bidirectional InfoNCE (NoiseContrastive Estimation of Information) loss and triplet loss. This framework guides samples to simultaneously approach their corresponding image and text prototypes while moving away from outlier prototypes, resulting in many pairs of positive and negative samples. This improves the precision and stability of the discrimination boundary. The positive-negative sample construction process is as follows: for any image feature h... I Based on the categories and corresponding image and text prototypes, positive pairs are selected, while other category prototypes are negative pairs. The triplet loss constrains samples to move away from out-of-class prototypes, enhancing the discrimination boundary. The bidirectional InfoNCE loss can be expressed as:

[0071]

[0072] Where τ is the temperature coefficient, P T P is the set of all text prototypes. I p is a set of image prototypes. T+ p is the text prototype corresponding to the image feature vector of the current sample. I+ Let z be the image prototype corresponding to the text feature vector of the current sample, and z be the prototype of the candidate negative sample to be compared, from P T or P I It was sampled from the middle.

[0073] Finally, semi-supervised pseudo-label generation and consistency constraints are performed. For unlabeled images, pseudo-labels are generated by retrieving the most similar text prototypes using shared class-level text descriptions. Positive contrast pairs are only included when the image-text similarity exceeds a threshold. For example, positive contrast pairs are included when the image-text similarity is greater than the threshold δ. The original-strongly enhanced image pairs are used as consistency samples, and KL (Kullback-Leibler Divergence) divergence is used to constrain the consistency of the two predictions to solidify the pseudo-labels and expand the training samples. Its expression is:

[0074]

[0075] Where P(x) is the model prediction probability distribution of the original sample x. Enhanced version of the original sample x The model predicts the probability distribution.

[0076] Then, based on the preset joint optimization total loss function, including cross-entropy supervision term, cross-modal contrast loss, triplet loss, and unlabeled consistency loss, the SGD (Stochastic Gradient Descent) optimizer (initial learning rate 0.02, batch size 64) is used to optimize the entire network and prototype library to achieve end-to-end network training. Finally, a pre-trained image diagnosis model is obtained by training on many-to-many positive and negative sample pairs. The preset joint optimization total loss function is:

[0077] L=λL sup +(1-λ)(L cmc +L tri +L cons );

[0078] Among them, L sup L is the cross-entropy supervision term. cmc For cross-modal contrast loss, L tri For triplet loss, L cons For unlabeled consistency loss, SGD optimization is used with an initial learning rate of 0.02, a batch size of 64, and λ as the loss weight parameter.

[0079] Therefore, the image diagnostic model obtained from the above training processes the fetal echocardiogram images to be tested, outputs the corresponding diagnostic category and its best matching image-text prototype pair, and completes automated and interpretable CHD classification and screening.

[0080] In summary, the embodiments of this application can achieve the following beneficial effects:

[0081] (1) Text priors can be fully utilized without one-to-one image-text annotation, reducing data preparation costs; Multi-prototype design characterizes intra-class differences, improving diagnostic robustness in complex scenarios;

[0082] (2) The semi-supervised mechanism effectively introduces unlabeled image information and improves the model's generalization ability;

[0083] (3) It has no dependency on encoder architecture and is compatible with visual backbones such as ViT-Base and Swin-Tiny or any BERT series text models. It is flexible in deployment and has low computational overhead.

[0084] The multi-prototype cross-modal contrastive learning fetal echocardiography image intelligent diagnostic system according to embodiments of this application acquires fetal echocardiography images of the fetus to be tested through an acquisition module, and inputs the fetal echocardiography images of the fetus to be tested into a pre-trained image diagnostic model through a processing module. Feature extraction is performed on the fetal echocardiography images of the fetus to be tested, and cross-modal multi-prototype comparison is performed based on the feature extraction results to obtain the diagnostic category and the best matching image-text prototype pair, which is output by the output module. Finally, CHD classification and screening are performed based on the diagnostic category and the best matching image-text prototype pair. In this way, the problems of insufficient model generalization ability caused by mainly relying on a single prototype center to represent each category and ignoring the significant differences of the same category under different conditions are solved. It achieves multi-category CHD intelligent screening without the need for a large number of image-text pairs, using unlabeled images and a small amount of text prior.

[0085] Next, referring to the accompanying drawings, a multi-prototype cross-modal comparative learning method for intelligent diagnosis of fetal cardiac ultrasound images, based on embodiments of this application, is described.

[0086] Figure 4 This is a flowchart of the multi-prototype cross-modal comparative learning method for intelligent diagnosis of fetal cardiac ultrasound images, according to an embodiment of this application.

[0087] like Figure 4 As shown, this multi-prototype cross-modal contrastive learning method for intelligent diagnosis of fetal echocardiography images includes the following steps:

[0088] In step S401, an ultrasound image of the fetal heart is acquired;

[0089] In step S402, the fetal echocardiogram image to be tested is input into a pre-trained image diagnostic model. The pre-trained image diagnostic model extracts features from the fetal echocardiogram image to be tested, and performs cross-modal multi-prototype comparison based on the feature extraction results to obtain the diagnostic category and the best matching image-text prototype pair.

[0090] In step S403, the diagnostic category and the best matching image-text prototype pair are output to perform CHD classification and screening based on the diagnostic category and the best matching image-text prototype pair.

[0091] According to one embodiment of this application, before inputting the fetal echocardiogram image to be tested into a pre-trained image diagnostic model, the method further includes:

[0092] Acquire multi-view fetal echocardiogram image datasets and physician report texts;

[0093] Data augmentation was performed on each image in the multi-view fetal echocardiography dataset to obtain a mixed labeled-unlabeled training set, and the doctor's report text was normalized into class-level text descriptions covering each diagnostic category.

[0094] Using a pre-trained Vision Transformer as the visual branch, a multi-dimensional visual feature vector is output for each image in the labeled-unlabeled mixed training set; and using frozen BERT encoding as the text branch, a semantic embedding vector of the same dimension is output for class-level text descriptions.

[0095] Based on multidimensional visual feature vectors, K-Means clustering is performed on feature vectors of similar images to obtain multiple image prototypes. Based on semantic embedding vectors of the same dimension, multiple text descriptions of the same category are embedded as text prototypes. Based on a preset update strategy, multiple image prototypes and text prototypes are updated in a rolling manner with preset coefficients to obtain multiple updated image prototypes and updated text prototypes.

[0096] Based on updated multiple image prototypes and updated text prototypes, a cross-modal multi-prototype contrastive learning framework is jointly constructed using bidirectional InfoNCE loss and triplet loss. The cross-modal multi-prototype contrastive learning framework guides samples to simultaneously approach the corresponding image prototypes and text prototypes and move away from heterogeneous prototypes, resulting in many-to-many positive and negative sample pairs.

[0097] Based on a pre-defined joint optimization total loss function, a pre-trained image diagnosis model is obtained by training on many-to-many positive and negative sample pairs.

[0098] According to one embodiment of this application, the preset update strategy is as follows:

[0099]

[0100] Among them, h T h is the semantic embedding vector of the current batch of the same type. I For the feature vectors of the current batch of similar images, This represents the mean of semantic embedding vectors of the same type in the current batch. This represents the average feature vector of the current batch of similar images. Let k be the prototype vector of the image of category c. Let be the m-th text prototype vector of category c, cm be the m-th text prototype of category c, c be the category, m be the index number of the text prototype, k be the index number of the image prototype, T(t) be the current state of the text prototype set, T(t+1) be the updated text set, ck be the k-th image prototype of category c, I(t+1) be the updated image prototype set, and I(t) be the current image prototype set.

[0101] According to one embodiment of this application, bidirectional InfoNCE is:

[0102]

[0103] Where τ is the temperature coefficient, PT P is the set of all text prototypes. I p is a set of image prototypes. T+ p is the text prototype corresponding to the image feature vector of the current sample. I+ z is the image prototype corresponding to the text feature vector of the current sample, and z is the prototype of the candidate negative sample to be compared.

[0104] According to one embodiment of this application, the preset joint optimization total loss function is:

[0105] L=λL sup +(1-λ)(L cmc +L tri +L cons );

[0106] Among them, L sup L is the cross-entropy supervision term. cmc For cross-modal contrast loss, L tri For triplet loss, L cons λ represents the unlabeled consistency loss, and λ is the loss weight parameter.

[0107] The multi-prototype cross-modal contrastive learning method for intelligent diagnosis of fetal echocardiography images according to embodiments of this application acquires fetal echocardiography images of the fetus to be tested through an acquisition module, and inputs the fetal echocardiography images of the fetus to be tested into a pre-trained image diagnosis model through a processing module. Feature extraction is performed on the fetal echocardiography images of the fetus to be tested, and cross-modal multi-prototype comparison is performed based on the feature extraction results to obtain the diagnostic category and the best matching image-text prototype pair, which is output by the output module. Finally, CHD classification and screening are performed based on the diagnostic category and the best matching image-text prototype pair. This solves the problem that the model's generalization ability is insufficient because it mainly relies on a single prototype center to represent each category and ignores the significant differences of the same category under different conditions. It achieves intelligent screening of multiple categories of CHD by using unlabeled images and a small amount of text prior without the need for a large number of image-text pairs.

[0108] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0109] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0110] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0111] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0112] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0113] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0114] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0115] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A multi-prototype cross-modal contrastive learning intelligent diagnostic system for fetal cardiac ultrasound images, characterized in that, include: The acquisition module is used to acquire the fetal echocardiogram image of the fetus to be tested; The processing module is used to input the fetal echocardiogram image to be tested into a pre-trained image diagnostic model, extract features from the fetal echocardiogram image through the pre-trained image diagnostic model, and perform cross-modal multi-prototype comparison based on the feature extraction results to obtain the diagnostic category and the best matching image-text prototype pair. The output module is used to output the diagnostic category and the most matching image-text prototype pair for CHD classification and screening based on the diagnostic category and the most matching image-text prototype pair.

2. The system according to claim 1, characterized in that, Also includes: The training module, specifically, is used for: Acquire multi-view fetal echocardiogram image datasets and physician report texts; Data augmentation processing is performed on each image in the multi-view fetal echocardiography image dataset to obtain a mixed labeled-unlabeled training set, and the doctor's report text is standardized into class-level text descriptions covering each diagnostic category; Using a pre-trained Vision Transformer as the visual branch, a multi-dimensional visual feature vector is output for each image in the labeled-unlabeled mixed training set; and using frozen BERT encoding as the text branch, a semantic embedding vector of the same dimension is output for the class-level text description. Based on the multidimensional visual feature vector, K-Means clustering is performed on the feature vectors of images of the same type to obtain multiple image prototypes. Based on the semantic embedding vector of the same dimension, multiple text descriptions of the same category are embedded as text prototypes. Based on a preset update strategy, the multiple image prototypes and the text prototypes are updated with preset coefficients to obtain updated multiple image prototypes and updated text prototypes. Based on the updated multiple image prototypes and the updated text prototypes, a cross-modal multi-prototype contrastive learning framework is jointly constructed using bidirectional InfoNCE loss and triplet loss. The cross-modal multi-prototype contrastive learning framework guides samples to simultaneously approach the corresponding image prototypes and text prototypes and move away from heterogeneous prototypes, thereby obtaining many-to-many positive and negative sample pairs. Based on a preset joint optimization total loss function, the pre-trained image diagnosis model is obtained by training the many-to-many positive and negative sample pairs.

3. The system according to claim 2, characterized in that, The preset update strategy is as follows: Among them, h T h is the semantic embedding vector of the current batch of the same type. I For the feature vectors of the current batch of similar images, This represents the mean of semantic embedding vectors of the same type in the current batch. This represents the average feature vector of the current batch of similar images. Let k be the prototype vector of the image of category c. Let be the m-th text prototype vector of category c, cm be the m-th text prototype of category c, c be the category, m be the index number of the text prototype, k be the index number of the image prototype, T(t) be the current state of the text prototype set, T(t+1) be the updated text set, ck be the k-th image prototype of category c, I(t+1) be the updated image prototype set, and I(t) be the current image prototype set.

4. The system according to claim 2, characterized in that, The bidirectional InfoNCE is: Where τ is the temperature coefficient, P T P is the set of all text prototypes. I p is a set of image prototypes. T+ p is the text prototype corresponding to the image feature vector of the current sample. I+ z is the image prototype corresponding to the text feature vector of the current sample, and z is the prototype of the candidate negative sample to be compared.

5. The system according to claim 2, characterized in that, The preset joint optimization total loss function is: L=λL sup +(1-λ)(L cmc +L tri +L cons ); Among them, L sup L is the cross-entropy supervision term. cmc For cross-modal contrast loss, L tri For triplet loss, L cons λ represents the unlabeled consistency loss, and λ is the loss weight parameter.

6. A multi-prototype cross-modal contrastive learning intelligent diagnostic method for fetal cardiac ultrasound images, characterized in that, Includes the following steps: Acquire echocardiogram images of the fetus to be tested; The fetal echocardiogram image to be tested is input into a pre-trained image diagnostic model. The pre-trained image diagnostic model extracts features from the fetal echocardiogram image to be tested. Based on the feature extraction results, cross-modal multi-prototype comparison is performed to obtain the diagnostic category and the best matching image-text prototype pair. Output the diagnostic category and the best matching image-text prototype pair to perform CHD classification and screening based on the diagnostic category and the best matching image-text prototype pair.

7. The method according to claim 6, characterized in that, Before inputting the fetal echocardiogram image to be tested into the pre-trained image diagnostic model, the method further includes: Acquire multi-view fetal echocardiogram image datasets and physician report texts; Data augmentation processing is performed on each image in the multi-view fetal echocardiography image dataset to obtain a mixed labeled-unlabeled training set, and the doctor's report text is standardized into class-level text descriptions covering each diagnostic category; Using a pre-trained Vision Transformer as the visual branch, a multi-dimensional visual feature vector is output for each image in the labeled-unlabeled mixed training set; and using frozen BERT encoding as the text branch, a semantic embedding vector of the same dimension is output for the class-level text description. Based on the multidimensional visual feature vector, K-Means clustering is performed on the feature vectors of images of the same type to obtain multiple image prototypes. Based on the semantic embedding vector of the same dimension, multiple text descriptions of the same category are embedded as text prototypes. Based on a preset update strategy, the multiple image prototypes and the text prototypes are updated with preset coefficients to obtain updated multiple image prototypes and updated text prototypes. Based on the updated multiple image prototypes and the updated text prototypes, a cross-modal multi-prototype contrastive learning framework is jointly constructed using bidirectional InfoNCE loss and triplet loss. The cross-modal multi-prototype contrastive learning framework guides samples to simultaneously approach the corresponding image prototypes and text prototypes and move away from heterogeneous prototypes, thereby obtaining many-to-many positive and negative sample pairs. Based on a preset joint optimization total loss function, the pre-trained image diagnosis model is obtained by training the many-to-many positive and negative sample pairs.

8. The method according to claim 6, characterized in that, The preset update strategy is as follows: Among them, h T h is the semantic embedding vector of the current batch of the same type. I For the feature vectors of the current batch of similar images, This represents the mean of semantic embedding vectors of the same type in the current batch. This represents the average feature vector of the current batch of similar images. Let k be the prototype vector of the image of category c. Let be the m-th text prototype vector of category c, cm be the m-th text prototype of category c, c be the category, m be the index number of the text prototype, k be the index number of the image prototype, T(t) be the current state of the text prototype set, T(t+1) be the updated text set, ck be the k-th image prototype of category c, I(t+1) be the updated image prototype set, and I(t) be the current image prototype set.

9. The method according to claim 7, characterized in that, The bidirectional InfoNCE is: Where τ is the temperature coefficient, P T P is the set of all text prototypes. I p is a set of image prototypes. T+ p is the text prototype corresponding to the image feature vector of the current sample. I+ z is the image prototype corresponding to the text feature vector of the current sample, and z is the prototype of the candidate negative sample to be compared.

10. The method according to claim 7, characterized in that, The preset joint optimization total loss function is: L=λL sup +(1-λ)(L cmc +L tri +L cons ); Among them, L sup L is the cross-entropy supervision term. cmc For cross-modal contrast loss, L tri For triplet loss, L cons λ represents the unlabeled consistency loss, and λ is the loss weight parameter.

Images