An ultrasonic image-oriented progressive text-image matching contrast learning method

Abstract

The application provides a progressive text-image matching contrast learning method for ultrasound images, and relates to the technical field of ultrasound image processing, and comprises the following steps: constructing a multi-modal training data set, and sorting a candidate image set according to clinical importance weights; setting multi-stage training, and increasing the size of the candidate set with training; for each report, selecting an image with the highest comprehensive score from the activated candidate set as a positive sample; mining difficult negative samples; dynamically adjusting the positive and negative sample weights according to the matching quality; constructing a bidirectional contrast learning loss, and integrating the clinical importance weights and the dynamic sample weights; and updating the encoder parameters through back propagation. The application utilizes the priority knowledge of the standard scanning sequence of ultrasound and an adaptive sample balancing mechanism to realize efficient cross-modal semantic alignment, and improves the matching precision and the training efficiency.

Description

Technical Field

[0001] This invention relates to the field of ultrasound image processing technology, and in particular to a progressive image-text matching and comparison learning method for ultrasound images. Background Technology

[0002] In the field of medical imaging, constructing image-text multimodal models is a crucial technological foundation for achieving intelligent diagnosis. In recent years, visual-language pre-trained models have made significant progress in cross-modal learning of natural images and text, and are gradually being applied to medical image processing. Researchers have developed specialized medical visual-language pre-trained models, such as BiomedCLIP, which is pre-trained based on large-scale medical image-text pairs; BioViL, which focuses on multimodal learning of radiological images; and MedCLIP, which is optimized for medical terminology and diagnostic expressions. However, these models are primarily designed for static images such as CT, MRI, or pathological slides, and are typically trained on data with a strict one-to-one pairing of images and text. Their adaptability to medical ultrasound imaging scenarios, which have unique data structure characteristics, is limited.

[0003] Medical ultrasound data presents unique characteristics and challenges. In clinical ultrasound examinations, a single examination typically generates multiple images but corresponds to a single comprehensive report. The same patient may undergo multiple examinations, resulting in multiple reports (initial and follow-up), forming a patient-level multi-image, multi-text data structure, unlike the strict one-to-one pairing relationships in natural image datasets. Ultrasound examinations follow standardized scanning procedures, and images from different slice types and scanning locations exhibit significantly different importance in clinical diagnosis, but existing methods rarely consider this hierarchical clinical importance information. Furthermore, in multi-image-to-one-text data structures, the truly key images that match the diagnostic report often constitute only a minority; a large number of images have low correlation with the report. A systematic solution remains lacking for accurately identifying key matching images from the candidate image set and establishing an effective contrastive learning mechanism.

[0004] Chinese invention patent CN112992308A discloses a training method for a medical image report generation model. This method extracts features from medical images using a visual feature extraction network, concatenates image category labels and self-learned labels based on the visual feature sequence, encodes these features using an encoding network to obtain a visual encoded feature vector, and then generates an image report using a decoding network. This method employs a multi-task learning strategy, performing auxiliary tasks such as image classification, self-supervised rotation angle prediction, and model distillation in parallel with the main task (generating image reports). It improves the model's image recognition ability and image-to-text conversion performance by jointly optimizing multiple loss functions. However, this patented method uses a single-image input architecture and does not consider the actual data structure of multiple image candidate sets in medical examinations. Summary of the Invention

[0005] In view of this, the present invention provides a progressive image-text matching and comparison learning method for ultrasound images. By establishing an image grading system based on clinical importance, combined with a progressive training strategy and a dynamic sample selection mechanism, it achieves accurate matching between ultrasound images and diagnostic reports.

[0006] The technical solution of this invention is implemented as follows:

[0007] This invention provides a progressive image-text matching and comparison learning method for ultrasound images, including:

[0008] S1. Construct a multimodal training dataset containing patient ultrasound examination data, with each patient corresponding to multiple ultrasound images and at least one ultrasound diagnostic report; using the report as the training sample unit, all images of the patient to which the report belongs constitute the candidate image set for that report;

[0009] S2. Identify the section type of each image in the candidate image set, assign a clinical importance weight to each image according to the preset ultrasound section clinical importance mapping table, and sort the candidate image set in descending order according to the clinical importance weight.

[0010] S3. Define multiple stages in the training process, and set the candidate set size limit for each stage. The weighting increases as training progresses; in the current training phase, the top clinically important images are selected from the sorted candidate image set. The image is used to construct the activation candidate set for the current stage;

[0011] S4. For each report in the current training batch, use a visual encoder and a text encoder to encode the features of the image in its corresponding activation candidate set and the report, respectively. Calculate the cross-modal similarity between the image and the report. Combine the clinical importance weight of the image, select the image with the highest comprehensive score from the activation candidate set as the best matching image of the report, and use the best matching image as a positive sample.

[0012] S5. For each report in the current training batch, exclude positive samples from its corresponding activation candidate set, calculate the cross-modal similarity between the remaining images and the report as the difficulty score, and select a preset number of images as negative samples according to the difficulty score.

[0013] S6. Dynamically adjust the weights of positive and negative samples based on the matching quality of the current training batch;

[0014] S7. Construct a bidirectional contrastive learning loss. In the image-to-text direction, positive samples are compared with all reports in the batch. In the text-to-image direction, reports are compared with positive and negative samples. The weights of positive and negative samples are dynamically adjusted and clinical importance weights are incorporated.

[0015] S8. Update the parameters of the visual encoder and text encoder through backpropagation based on bidirectional contrastive learning loss;

[0016] S9. According to the set training rounds and stages, repeat steps S3 to S8, update the size of the activation candidate set at different training stages, and complete the model training.

[0017] Preferably, in step S2, the preset ultrasound section clinical importance mapping table is constructed based on the mandatory section system specified in the ultrasound standard scanning guidelines, and the weight allocation is determined by combining the frequency of different sections appearing in the diagnostic report in clinical practice.

[0018] For echocardiography, the parasternal long-axis view is assigned the highest clinical importance weight, the apical four-chamber view is assigned the second highest weight, the parasternal short-axis view and the apical two-chamber view are assigned medium weight, and the weight of other auxiliary views decreases in that order.

[0019] For abdominal ultrasound, the right intercostal scan is assigned the highest weight, while the subxiphoid scan and lateral decubitus scan are assigned medium weight.

[0020] The clinical importance weights are adjusted based on the base weights and combined with the image quality score, which is automatically calculated by a pre-trained image quality assessment network.

[0021] Preferably, in step S3, the training process is divided into ten stages, with a limit on the size of the candidate set for each stage. The numbers are 5, 8, 12, 18, 25, 35, 50, 70, 100, and 150, with each stage lasting 20 training rounds.

[0022] When the total number of images in the candidate image set corresponding to the i-th report is less than the candidate set size limit for the current stage. When this is the case, all candidate images corresponding to that report are used as the activation candidate set.

[0023] Preferably, in step S4, the formula for selecting the optimal matching image is:

[0024] ;

[0025] in, For the first Report The corresponding optimal matching image; To activate images in the candidate set; For the current training phase The activation candidate set represents the candidate image set. Clinical importance weighting ranking A subset of images; for The clinical importance weight; For image and report Cross-modal similarity; The historical matching score from the previous training step; This indicates the current training step count, using batch-level iteration counting; This represents the historical weighting coefficient.

[0026] Preferably, the historical matching scores are updated at the end of each training step using an exponential moving average, updating only the images in the current training phase's activation candidate set:

[0027] ;

[0028] in For smoothing coefficients; The historical matching score at the end of the step-th training step; The weights are the information for the current training step.

[0029] Preferably, in step S5, positive samples are excluded from the activation candidate set, and the remaining images are used as negative sample candidate images. The difficulty score is defined as the cross-modal similarity between the negative sample candidate image and the report. The selection range of negative samples is limited to the activation candidate set in the current training phase.

[0030] Select the image with the highest difficulty score from the negative sample candidate images as a preset number of negative samples. The preset number is automatically adjusted according to the size of the activation candidate set, and the baseline value for the number of negative samples is set to 3.

[0031] Preferably, in step S6, the matching quality is evaluated by the average of the cross-modal similarities between all positive samples in the current training batch and their corresponding reports; the weights of positive samples are adjusted inversely according to the matching quality; and the weights of negative samples are adjusted in the positive direction according to the matching quality.

[0032] Preferably, the bidirectional contrastive learning loss includes contrast loss in the image-to-text direction and contrast loss in the text-to-image direction;

[0033] In the image-to-text direction, positive samples are compared with all reports in the batch. The matching probability of positive samples and their corresponding reports is calculated by softmax normalization. Negative log-likelihood is used as loss and dynamic positive sample weights and clinical importance weights are incorporated.

[0034] In the text-to-image direction, the report is compared with positive sample images and negative samples. The denominator includes positive sample items and negative sample weights. The negative sample items are weighted using dynamic negative sample weights, which also incorporate positive sample weights and clinical importance weights.

[0035] The bidirectional contrastive learning loss is obtained by averaging the losses in both directions for all samples in the batch.

[0036] Preferably, the visual encoder uses an ultrasound image encoder pre-trained based on the Teacher-Student framework, with a Vision Transformer architecture. The input is an RGB image with dimensions of 224×224×3, and the output is a 768-dimensional image feature vector. During fine-tuning, only the last 8 layers are fine-tuned, while the parameters of the first 16 layers are frozen.

[0037] The text encoder uses ClinicalBERT, which is pre-trained in the medical field. Based on the BERT architecture, it is pre-trained on medical literature and clinical reports, and outputs a 768-dimensional text feature vector. It is fine-tuned with all parameters.

[0038] Preferably, in step S4, the image features and text features are projected through a multilayer perceptron, and the projection network contains two fully connected layers and the hidden layer dimension is 1024.

[0039] The projected features are L2 normalized to be normalized into unit vectors.

[0040] Cross-modal similarity is calculated using the cosine similarity of normalized features, which is the inner product of the normalized image feature vector and the normalized text feature vector, with a value range of [-1, 1].

[0041] The present invention has the following advantages over the prior art:

[0042] (1) This invention systematically solves the problem of image-text matching under a multi-image-to-text data structure by establishing a progressive matching priority strategy based on standard ultrasound scanning sequences and an adaptive positive-negative sample balancing contrastive learning mechanism. This method introduces the standardized process of ultrasound clinical examination and the hierarchical structure of cross-sectional importance into the contrastive learning framework. From the dynamic adjustment of the activation candidate set to the adaptive balancing of positive and negative sample weights, it realizes progressive feature learning from simple core cross-sections to complex auxiliary cross-sections, enabling the model to accurately identify key images that truly match the diagnostic report in the candidate image set, thereby improving the cross-modal semantic alignment accuracy of ultrasound images and diagnostic reports.

[0043] (2) The progressive candidate set scheduling strategy in this invention gradually expands the size of the activation candidate set in stages and prioritizes images with high clinical importance for training. This strategy enables the model to focus on learning the image-text correspondence of standard sections in the early stages of training, establishing stable core feature representations, and then gradually expanding to auxiliary sections, which conforms to the cognitive process of clinical diagnosis from the main section to the auxiliary section. Compared with the training method that uses all candidate images at once, this strategy avoids the slow convergence and noise interference caused by a large candidate set in the early stages of training, speeds up the model convergence speed, and improves the matching accuracy of clinically important sections;

[0044] (3) The optimal matching selection mechanism based on clinical importance weight in this invention introduces the hierarchical structure of section importance in the ultrasound standard scanning guidelines into the positive sample selection process. This mechanism selects the optimal matching image as a positive sample by comprehensively considering cross-modal similarity, clinical importance weight, and historical matching information, ensuring that the training process prioritizes standard sections with high diagnostic value, making the learned image-text correspondence more consistent with clinical diagnostic logic. The exponential moving average update strategy of historical matching scores maintains the stability of matching selection and avoids the adverse effects of single-step prediction fluctuations on the training process;

[0045] (4) The hard negative sample mining strategy in this invention selects images with the highest semantic similarity to the reported images but not actual matches from the activation candidate set as negative samples. This strategy makes the model face more challenging discrimination tasks during training, improving its ability to distinguish easily confused samples. The selection range of hard negative samples is consistent with the progressive strategy, and selection is only made from the candidate set activated at the current stage, ensuring that the difficulty of the negative samples matches the training stage and avoiding training instability caused by introducing overly difficult samples in the early stage of training;

[0046] (5) The adaptive positive and negative sample weight adjustment mechanism in this invention dynamically balances the contribution of positive and negative samples to the loss function based on the matching quality of the current training batch. When the matching quality is low, the weight of positive samples is increased to strengthen the learning of correct matching. When the matching quality is high, the weight of negative samples is increased to increase the discrimination difficulty. This mechanism effectively solves the problem of severe imbalance in the number of positive and negative samples in the scenario of multiple images to one document, so that the contrastive learning maintains an appropriate learning focus in different training stages, thereby improving training efficiency and final model performance. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart of the method of the present invention;

[0049] Figure 2 This is a diagram illustrating the technical implementation of the present invention;

[0050] Figure 3 This is a schematic diagram illustrating the training phase division of the present invention;

[0051] Figure 4 This is a schematic diagram of the comparative learning process of the present invention. Detailed Implementation

[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0053] like Figure 1 and Figure 2 As shown, this invention provides a progressive image-text matching and comparison learning method for ultrasound images, including:

[0054] S1. Construct a multimodal training dataset containing patient ultrasound examination data, with each patient corresponding to multiple ultrasound images and at least one ultrasound diagnostic report; using the report as the training sample unit, all images of the patient to which the report belongs constitute the candidate image set for that report;

[0055] S2. Identify the section type of each image in the candidate image set, assign a clinical importance weight to each image according to the preset ultrasound section clinical importance mapping table, and sort the candidate image set in descending order according to the clinical importance weight.

[0056] S3. Define multiple stages in the training process, and set the candidate set size limit for each stage. The weighting increases as training progresses; in the current training phase, the top clinically important images are selected from the sorted candidate image set. The image is used to construct the activation candidate set for the current stage;

[0057] S4. For each report in the current training batch, use a visual encoder and a text encoder to encode the features of the image in its corresponding activation candidate set and the report, respectively. Calculate the cross-modal similarity between the image and the report. Combine the clinical importance weight of the image, select the image with the highest comprehensive score from the activation candidate set as the best matching image of the report, and use the best matching image as a positive sample.

[0058] S5. For each report in the current training batch, exclude positive samples from its corresponding activation candidate set, calculate the cross-modal similarity between the remaining images and the report as the difficulty score, and select a preset number of images as negative samples according to the difficulty score.

[0059] S6. Dynamically adjust the weights of positive and negative samples based on the matching quality of the current training batch;

[0060] S7. Construct a bidirectional contrastive learning loss. In the image-to-text direction, positive samples are compared with all reports in the batch. In the text-to-image direction, reports are compared with positive and negative samples. The weights of positive and negative samples are dynamically adjusted and clinical importance weights are incorporated.

[0061] S8. Update the parameters of the visual encoder and text encoder through backpropagation based on bidirectional contrastive learning loss;

[0062] S9. According to the set training rounds and stages, repeat steps S3 to S8, update the size of the activation candidate set at different training stages, and complete the model training.

[0063] In one embodiment of the present invention, step S1 includes:

[0064] A multimodal dataset containing 854,093 patients was constructed, comprising 245,385,096 ultrasound images and 1,916,179 ultrasound diagnostic reports. The data covers various ultrasound examination types, including cardiac ultrasound, abdominal ultrasound, and vascular ultrasound.

[0065] A coarse-grained pairing relationship is used at the patient level, where all images of a patient are associated with all reports of that patient. The same patient may have multiple examinations corresponding to multiple reports. On average, each patient contains approximately 2.24 reports and 287 images.

[0066] During training, training samples are constructed using reports as the basic unit. The report index is denoted by i, and the i-th report is denoted as . The candidate image set consists of all images of the patient to whom the report belongs. The size of the candidate set is denoted as That is, the total number of images of the patient;

[0067] Image preprocessing includes resizing to 224×224 pixels and normalizing to the [-1, 1] interval. The scan location and cross-section type label for each image are automatically obtained through a pre-trained cross-section classifier. After training on large-scale labeled data, this classifier achieves an accuracy of over 92% in recognizing standard cross-sections. For data with a high degree of DICOM metadata normalization, cross-section type information can be directly extracted from the metadata as a supplement. When the confidence of the cross-section type label is low, default weights are used to ensure the robustness of the method.

[0068] Text preprocessing uses a medical-grade tokenizer for tokenization, with a maximum length of 512 tokens. Regarding data augmentation strategies, the visual encoder of this invention employs a pre-trained ultrasound image encoder. During the pre-training phase, this encoder utilizes data augmentation strategies suitable for the characteristics of ultrasound images, including random cropping sizes from 96×96 pixels to 336×336 pixels, adaptive sampling with random occlusion ratios between 0.1 and 0.5, color jittering including random adjustments to brightness, contrast, and saturation, and geometric transformations including random rotation angles ranging from -15° to 15°, horizontal flipping probability of 0.5, and scaling ratios ranging from 0.8 to 1.2. These augmentation strategies function during the encoder's pre-training phase. During the image-text matching and comparison learning phase of this invention, considering the anatomical orientation specificity of ultrasound images and the accuracy requirements of cross-modal semantic alignment, the data augmentation strategy is adjusted to use only random cropping ranges [0.8, 1.0] and color jittering, including brightness ranges [0.8, 1.2] and contrast ranges [0.8, 1.2]. Geometric transformations that alter the anatomical orientation, such as horizontal flipping and rotation, are no longer used to ensure consistency between the image and the anatomical positional relationship described in the text, thus avoiding semantic conflicts.

[0069] In one embodiment of the present invention, step S2 employs a progressive matching priority strategy based on standard ultrasound scanning sequences. This strategy designs a priority ranking based on the importance of standard ultrasound scanning sequences and sections in clinical ultrasound examinations, and the process is as follows:

[0070] Based on the mandatory scan section system specified in the standard scanning guidelines issued by the American Society of Echocardiography (ASE) and the European Association for Cardiovascular Imaging (EACVI), combined with the frequency statistics of different sections in diagnostic reports in clinical practice and literature consensus, and after evaluation by field experts, a clinical importance mapping table was determined.

[0071] For echocardiography, the parasternal long-axis view has an importance weight of 1.0, serving as the most important basic view for observing the aortic root, left ventricular outflow tract, mitral valve, and left atrium; the apical four-chamber view has a weight of 0.9, used to assess atrial and ventricular structure and overall cardiac morphology; the parasternal short-axis view has a weight of 0.8, used to assess left ventricular function and segmental wall motion; the apical two-chamber view has a weight of 0.7, used to assess the left atrium and left ventricle; the weights of other auxiliary views decrease sequentially to 0.5.

[0072] For abdominal ultrasound, the right intercostal scan has a weight of 1.0 and is used as the main section for liver examination; the subxiphoid scan has a weight of 0.8 and is used for observation of the gallbladder and lower edge of the liver; the lateral decubitus scan has a weight of 0.6 and is used for kidney examination.

[0073] The formula for calculating clinical importance weight is:

[0074] ;

[0075] in For the first Zhang Image For image Clinical importance weighting The basic importance weights are based on the image section type. For image The aspect type label, For image The image quality score, ranging from 0 to 1, is automatically calculated by a pre-trained image quality assessment network, which evaluates the image's sharpness, contrast, and anatomical structure discernibility. This is a quality adjustment coefficient used to control the degree of influence of image quality on importance weights; it is set to 0.2.

[0076] For each text report The corresponding candidate image set is all images of the patient to whom the report belongs, denoted as . ,in, For report indexing, For the first The number of images belonging to the patient in each report Indicates candidate set The first in Zhang images, among which ;

[0077] The candidate set is sorted according to clinical importance weight:

[0078] ;

[0079] in, The function returns a sequence of indices sorted in descending order of their weight values. This indicates that the items are arranged in descending order, from most important to least important. This is the sorted candidate set index sequence.

[0080] like Figure 3 As shown, in one embodiment of the present invention, step S3 includes:

[0081] The training process is divided into ten stages, with a limit on the size of the candidate set for each stage. The training epochs are arranged in sequence as follows: 5, 8, 12, 18, 25, 35, 50, 70, 100, and 150, with each stage lasting 20 training rounds. The training epoch range corresponding to the scheduling sequence is {[0, 20], [21, 40], [41, 60], [61, 80], [81, 100], [101, 120], [121, 140], [141, 160], [161, 180], [181, 200]}. This sequence adopts an approximately exponential growth pattern to balance focused learning in the early stages of training with comprehensive coverage in the later stages. The stage division and candidate set size are determined through experimental optimization to achieve the best balance between training stability and computational efficiency.

[0082] In each training phase The candidate set selection strategy is to select candidates ranked by clinical importance weight. The image, in which For the first The candidate set size for each stage, with specific values ​​for each stage: 5, 8, 12, 18, 25, 35, 50, 70, 100, 150. When the... Number of candidate images corresponding to each report Smaller than the current stage candidate set size When, use all Zhang images were used as a candidate set;

[0083] Define the candidate set for activation in the current stage as , indicating candidate set Clinical importance weighting ranking A subset of images, when hour This design is based on a priority learning mechanism for the clinical diagnostic process. In the early stages of training, by limiting the candidate set size and prioritizing clinically important sections, the model focuses on establishing cross-modal semantic alignments of core anatomical structures. These core features serve as the foundational representations for subsequent learning of auxiliary sections. As training progresses and the model's representations of clinically important sections gradually stabilize, the candidate set is gradually expanded to cover more auxiliary sections, achieving a progressive feature learning from core to comprehensive features. This ensures that the model first learns the most diagnostically valuable image-text correspondences, aligning with the cognitive process in clinical diagnosis from primary to auxiliary sections. This mechanism accelerates model convergence and improves the matching accuracy for clinically important sections.

[0084] like Figure 4 As shown, in one embodiment of the present invention, step S4 includes:

[0085] An ultrasound image encoder pre-trained based on the Teacher-Student framework is used as a visual encoder. In the first stage, the encoder obtains feature representation capabilities suitable for the characteristics of ultrasound images through self-supervised learning. The architecture selected is Vision Transformer. The input is an RGB image with a size of 224×224×3, and the output is a 768-dimensional image feature vector. The fine-tuning strategy only fine-tunes the last 8 layers to maintain the generalization ability of the low-level features, and the parameters of the first 16 layers are frozen and do not participate in training.

[0086] The text encoder uses ClinicalBERT, which is pre-trained in the medical field. The model is based on the BERT architecture of ClinicalBERT. The pre-training data includes large-scale medical literature and clinical reports. The encoding capability is specifically optimized for understanding medical terms and clinical expressions. The output is a 768-dimensional text feature vector. The text encoder adopts a full parameter fine-tuning strategy to adapt to the specific expression habits of ultrasound reports.

[0087] Image feature projection using a multilayer perceptron and text feature projection ,in The image feature projection network consists of two fully connected layers with a hidden layer dimension of 1024. For visual encoders, For the input image, The text feature projection network also contains two fully connected layers with a hidden layer dimension of 1024. For text encoders, Input text, then perform L2 normalization. and ,in The normalized image features, These are the normalized text features. It is an L2 norm;

[0088] Cross-modal similarity is calculated using the cosine similarity of normalized features. , which is the inner product of the normalized image feature vector and the normalized text feature vector, with a value range of [-1, 1];

[0089] For the This report details the candidate set activated from its current stage. The optimal matching image is selected from the following, and the matching selection formula is:

[0090] ;

[0091] in For the first Report The corresponding optimal matching image, For images in the candidate set, For image and report Cross-modal similarity is calculated using the cosine similarity of normalized features. , The historical matching score from the previous training step. This indicates the current training step number, using batch-level iteration counting. This is the historical weighting coefficient, used to control the impact of historical information on the current match, set to 0.15. Let... This indicates that the best matching image is in the candidate set. The index in, i.e. ,in ;

[0092] Historical matching scores are updated using an exponential moving average at the end of each training step, but only images in the current phase's activation set are updated:

[0093] ;

[0094] in This is the smoothing coefficient, with a value of 0.8. The historical matching score at the end of the step-th training step. The weights of the information for the current training step are used in the initial training phase. The historical matching score is initialized to Initial values ​​are given based on clinical importance weights. The historical scores of inactive images remain unchanged, and when an image is first activated in a subsequent stage, the initial values ​​are used as its historical score. This design ensures that matching selection considers both semantic similarity and prioritizes clinically important aspects, while maintaining training stability.

[0095] In one embodiment of the present invention, step S5 employs an adaptive positive and negative sample balancing strategy for negative sample selection:

[0096] Traditional contrastive learning methods face a severe imbalance in the number of positive and negative samples when processing candidate sets. In each report's candidate set, there is only one true matching positive sample, i.e., the optimal matching image. However, the presence of a large number of negative samples leads to an imbalance that makes the model prone to biased learning. This invention proposes an adaptive positive-negative sample balancing strategy to address this problem. The difficult negative sample mining strategy dynamically selects the most challenging negative samples based on the current model's prediction results, and the selection range of difficult negative samples remains consistent with the progressive strategy, selecting only from the candidate set activated at the current stage;

[0097] The formula for calculating difficulty is:

[0098] ;

[0099] in negative sample images Relative to text The level of difficulty This indicates that the best matching sample is excluded. This indicates that the best matching image corresponding to the i-th report is in the candidate set. The index in the text is defined as the cross-modal similarity between the negative sample and the text. The higher the similarity, the easier it is for the negative sample to be misclassified as a positive sample, and the more valuable it is for training.

[0100] The formula for selecting difficult negative samples is:

[0101] ;

[0102] in For the selected set of indices of difficult negative samples, The function returns the one with the highest difficulty. Index of each sample, The number of effective hard negative samples, where, This indicates the size of the activation set; subtracting 1 is to exclude positive samples, and the number of difficult negative samples is the baseline value. Setting it to 3, experiments verified that this value achieves the optimal balance between training efficiency and discriminative ability. It automatically adjusts to the maximum number of available negative samples when the activation set is small. This strategy allows the model to face more challenging discrimination tasks during training, improving its ability to distinguish easily confused samples.

[0103] In one embodiment of the present invention, step S6 dynamically adjusts the positive sample weights and negative sample weights based on the training progress and matching quality using an adaptive weight adjustment strategy. The process includes:

[0104] The training progress assessment formula is:

[0105] ;

[0106] in For the first Average matching quality over training steps Let be the batch size, and let represent the number of report samples in each sub-batch. The summation is performed on the matching similarity of all report samples in the batch.

[0107] The formula for adjusting the weights of positive samples is:

[0108] ;

[0109] in For positive sample weights, The adjustment coefficient is set to 0.5. When the matching quality is low, the weight of positive samples is increased to strengthen the learning of correct matches.

[0110] The formula for adjusting the weights of negative samples is:

[0111] ;

[0112] in For negative sample weights, The adjustment coefficient is set to 0.3. When the matching quality is high, the weight of negative samples is increased to increase the learning difficulty and improve the discrimination ability.

[0113] This mechanism effectively solves the problem of severe imbalance in the number of positive and negative samples in the scenario of multiple images to one text, and enables contrastive learning to maintain an appropriate learning focus at different training stages.

[0114] In one embodiment of the present invention, in step S7, the bidirectional contrast learning loss includes contrast loss in the image-to-text direction and contrast loss in the text-to-image direction.

[0115] In the image-to-text direction, positive samples are compared with all reports in the batch. The matching probability of positive samples and their corresponding reports is calculated by softmax normalization. Negative log-likelihood is used as loss and dynamic positive sample weights and clinical importance weights are incorporated.

[0116] In the text-to-image direction, the report is compared with positive sample images and negative samples. The denominator includes positive sample items and negative sample weights. The negative sample items are weighted using dynamic negative sample weights, which also incorporate positive sample weights and clinical importance weights.

[0117] The bidirectional contrastive learning loss is obtained by averaging the losses in both directions for all samples in the batch.

[0118] In this embodiment, the contrast loss in the image-to-text direction is:

[0119] ;

[0120] in For the first Image-to-text contrast loss term for each report sample. The temperature parameter is set to 0.07. Clinical importance weights are assigned to the optimally matched images to emphasize the learning of important cross-sections. For the report sample index in the batch, from 1 to B, The first in the batch For each report, the denominator calculates the similarity of all texts in the batch as a normalization term;

[0121] The contrast loss in the text-to-image direction is:

[0122] ;

[0123] in For the first The text-to-image contrast loss term for each report sample includes positive sample terms and weighted terms for hard negative samples in the denominator. For indexing hard negative samples in the candidate set, For candidate set The first in Zhang image, here This represents the index value retrieved from the set of hard negative sample indices, according to the aforementioned definition. This corresponds to the difficult negative sample image;

[0124] The overall contrastive learning loss function is:

[0125] ;

[0126] The loss function is an average of the bidirectional losses of all samples in the batch. This loss function incorporates clinical importance weights and an adaptive sample balancing mechanism, making the image-text correspondence learned by the model more consistent with clinical diagnostic logic.

[0127] In one embodiment of the present invention, step S8 includes:

[0128] Model parameter updates employ the backpropagation algorithm. First, the bidirectional contrastive learning loss function is calculated. Then, the gradient of the loss function with respect to all trainable parameters is calculated through an automatic differentiation mechanism, including the parameters of the last 8 layers of the visual encoder, all parameters of the text encoder, and the parameters of the image and text projection networks.

[0129] The optimizer uses AdamW, with parameters set as follows: The weight decay is set to 0.01, and L2 regularization is applied to the parameters to prevent overfitting. The gradient clipping threshold is set to 1.0; when the global norm of the gradient exceeds this threshold, all gradients are scaled proportionally to prevent gradient explosion. Regarding the learning rate settings, the initial learning rate for the visual encoder is... Cosine scheduling is used to gradually decay to Initial learning rate for the text encoder Using linear scheduling to gradually decay to A mixed-precision training technique is employed, using automatic mixed-precision AMP. Half-precision floating-point numbers (FP16) are used in forward propagation and gradient calculation to improve computational efficiency, while single-precision floating-point numbers (FP32) are used during parameter updates to ensure numerical stability. The gradient accumulation step is set to 4, meaning that a parameter update is performed after accumulating gradients in every 4 mini-batches, achieving a training effect equivalent to a batch size of 256. Distributed training utilizes 8 NVIDIA V100 GPUs, employing a data parallelism strategy. Each GPU processes data in batches of 8, and gradient synchronization ensures consistent parameter updates.

[0130] In one embodiment of the present invention, step S9 includes:

[0131] The training process is iteratively optimized according to the preset training rounds and phase divisions. The total number of training rounds is set to 200 epochs, with each training phase consisting of 20 epochs, for a total of 10 phases. The candidate set size limit for each phase increases in sequence as follows: 5, 8, 12, 18, 25, 35, 50, 70, 100, 150.

[0132] At the beginning of each training phase, the candidate set size limit for that phase is determined based on the current phase number s. Then, step S3 is executed to update the activation candidate set of all reports;

[0133] In each training epoch of this phase, the entire training dataset is traversed, and steps S4 to S8 are performed for each mini-batch, including feature encoding and cross-modal similarity calculation, optimal matching image selection, hard negative sample mining, dynamic weight adjustment, loss function calculation, and parameter update.

[0134] After each training step, the historical matching score is updated to provide stable reference information for matching selection in the next training step;

[0135] At the end of each epoch, evaluate model performance on the validation set, calculate image-text matching accuracy and cross-modal retrieval metrics, and monitor training progress.

[0136] After completing 20 epochs in the current stage, proceed to the next stage, expand the size of the activation candidate set, and repeat the above training process.

[0137] The training cutoff condition is to complete all 200 epochs of training, or to terminate training early when the validation set performance does not show a significant improvement within 3 consecutive phases (60 epochs) to avoid overfitting.

[0138] During training, checkpoints representing the best-performing model at the end of each stage are saved, and the model that performs best on the validation set is ultimately selected as the training result. The entire training process achieves progressive learning from core aspects to comprehensive aspects, ensuring that the model first masters the textual and graphical correspondences of clinically important aspects before gradually expanding to auxiliary aspects. This aligns with the cognitive logic of clinical diagnosis and effectively improves training efficiency and the final model performance.

[0139] The present invention is illustrated by a specific embodiment:

[0140] In this embodiment, data from 90,000 patients undergoing echocardiography were used for training. This subset, selected from the total dataset, contained 1,620,000 echocardiogram images, averaging 18 images per patient per report. Due to the standardized scanning process, the number of echocardiogram images is relatively concentrated in standard sections, covering eight standard sections including the parasternal long axis, short axis, four-chamber view, and two-chamber view. Clinical importance weights were set as follows: parasternal long axis 1.0, four-chamber view 0.9, short axis 0.8, two-chamber view 0.7, aortic short axis 0.6, subxiphoid 0.6, parasternal right ventricle 0.5, and other sections 0.5. Key parameter settings included a quality adjustment coefficient. Historical weighting coefficient Smoothing coefficient Positive sample adjustment coefficient Negative sample adjustment coefficient The model was validated on a cardiac disease diagnostic report generation task, showing significant improvements over the traditional CLIP model in BLEU-4, ROUGE-L, and cross-modal retrieval Recall@1 metrics. During training, the matching confidence of clinically important sections significantly improved from the beginning to the end of training, and the matching confidence of ordinary sections was also effectively improved, demonstrating the effectiveness of priority learning based on clinical importance. The difficult negative sample mining strategy significantly improved the model's ability to distinguish easily confused samples, exhibiting better accuracy in matching tasks with complex cases. Finally, the model achieved high accuracy in the cardiac ultrasound image matching task, with particularly outstanding accuracy in matching clinically important sections, validating the effectiveness of the progressive matching priority strategy and the adaptive positive and negative sample balancing mechanism.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A progressive image-text matching and comparison learning method for ultrasound images, characterized in that, include: S1. Construct a multimodal training dataset containing patient ultrasound examination data, with each patient corresponding to multiple ultrasound images and at least one ultrasound diagnostic report; Using reports as training sample units, all images of the patient to whom the report belongs constitute the candidate image set for that report; S2. Identify the section type of each image in the candidate image set, assign a clinical importance weight to each image according to the preset ultrasound section clinical importance mapping table, and sort the candidate image set in descending order according to the clinical importance weight. S3. Define multiple stages in the training process, and set the candidate set size limit for each stage. The weighting increases as training progresses; in the current training phase, the top clinically important images are selected from the sorted candidate image set. The image is used to construct the activation candidate set for the current stage; S4. For each report in the current training batch, use a visual encoder and a text encoder to encode the features of the image in its corresponding activation candidate set and the report, respectively. Calculate the cross-modal similarity between the image and the report. Combine the clinical importance weight of the image, select the image with the highest comprehensive score from the activation candidate set as the best matching image of the report, and use the best matching image as a positive sample. S5. For each report in the current training batch, exclude positive samples from its corresponding activation candidate set, calculate the cross-modal similarity between the remaining images and the report as the difficulty score, and select a preset number of images as negative samples according to the difficulty score. S6. Dynamically adjust the weights of positive and negative samples based on the matching quality of the current training batch; S7. Construct a bidirectional contrastive learning loss. In the image-to-text direction, positive samples are compared with all reports in the batch. In the text-to-image direction, reports are compared with positive and negative samples. The weights of positive and negative samples are dynamically adjusted and clinical importance weights are incorporated. S8. Update the parameters of the visual encoder and text encoder through backpropagation based on bidirectional contrastive learning loss; S9. According to the set training rounds and stages, repeat steps S3 to S8, update the size of the activation candidate set at different training stages, and complete the model training.

2. The progressive image-text matching and comparison learning method for ultrasound images according to claim 1, characterized in that, In step S2, the preset ultrasound section clinical importance mapping table is constructed based on the mandatory section system specified in the ultrasound standard scanning guidelines, and the weight allocation is determined by combining the frequency of different sections appearing in the diagnostic report in clinical practice. For echocardiography, the parasternal long-axis view is assigned the highest clinical importance weight, the apical four-chamber view is assigned the second highest weight, the parasternal short-axis view and the apical two-chamber view are assigned medium weight, and the weight of other auxiliary views decreases in that order. For abdominal ultrasound, the right intercostal scan is assigned the highest weight, while the subxiphoid scan and lateral decubitus scan are assigned medium weight. The clinical importance weights are adjusted based on the base weights and combined with the image quality score, which is automatically calculated by a pre-trained image quality assessment network.

3. The progressive image-text matching and comparison learning method for ultrasound images according to claim 1, characterized in that, In step S3, the training process is divided into ten stages, with a limit on the candidate set size for each stage. The numbers are 5, 8, 12, 18, 25, 35, 50, 70, 100, and 150, with each stage lasting 20 training rounds. When the total number of images in the candidate image set corresponding to the i-th report is less than the candidate set size limit for the current stage. When this is the case, all candidate images corresponding to that report are used as the activation candidate set.

4. The progressive image-text matching and comparison learning method for ultrasound images according to claim 1, characterized in that, In step S4, the formula for selecting the optimal matching image is: ； in, For the first Report The corresponding optimal matching image; To activate images in the candidate set; For the current training phase The activation candidate set represents the candidate image set. Clinical importance weighting ranking A subset of images; for The clinical importance weight; For image and report Cross-modal similarity; The historical matching score from the previous training step; This indicates the current training step count, using batch-level iteration counting; This represents the historical weighting coefficient.

5. The progressive image-text matching and comparison learning method for ultrasound images according to claim 4, characterized in that, Historical matching scores are updated using an exponential moving average at the end of each training step, updating only the images in the current training phase's activation candidate set: ； in For smoothing coefficients; The historical matching score at the end of the step-th training step; The weights are the information for the current training step.

6. The progressive image-text matching and comparison learning method for ultrasound images according to claim 1, characterized in that, In step S5, positive samples are excluded from the activation candidate set, and the remaining images are used as negative sample candidate images. The difficulty score is defined as the cross-modal similarity between the negative sample candidate image and the report. The selection range of negative samples is limited to the activation candidate set in the current training phase. Select the images with the highest difficulty scores from the negative sample candidate images as a preset number of negative samples. The preset number is automatically adjusted according to the size of the activation candidate set, and the baseline value for the number of negative samples is set to 3.

7. The progressive image-text matching and comparison learning method for ultrasound images according to claim 1, characterized in that, In step S6, the matching quality is evaluated by the average of the cross-modal similarities between all positive samples in the current training batch and their corresponding reports; the weights of positive samples are adjusted inversely according to the matching quality; and the weights of negative samples are adjusted in the positive direction according to the matching quality.

8. The progressive image-text matching and comparison learning method for ultrasound images according to claim 1, characterized in that, Bidirectional contrastive learning loss includes contrast loss in the image-to-text direction and contrast loss in the text-to-image direction; In the image-to-text direction, positive samples are compared with all reports in the batch. The matching probability of positive samples and their corresponding reports is calculated by softmax normalization. Negative log-likelihood is used as loss and dynamic positive sample weights and clinical importance weights are incorporated. In the text-to-image direction, the report is compared with positive sample images and negative samples. The denominator includes positive sample items and negative sample weights. The negative sample items are weighted using dynamic negative sample weights, which also incorporate positive sample weights and clinical importance weights. The bidirectional contrastive learning loss is obtained by averaging the losses in both directions for all samples in the batch.

9. A progressive image-text matching and comparison learning method for ultrasound images according to claim 1, characterized in that, The visual encoder is an ultrasound image encoder pre-trained based on the Teacher-Student framework. The architecture is VisionTransformer. The input is an RGB image with a size of 224×224×3, and the output is a 768-dimensional image feature vector. During fine-tuning, only the last 8 layers are fine-tuned, and the parameters of the first 16 layers are frozen. The text encoder uses ClinicalBERT, which is pre-trained in the medical field. Based on the BERT architecture, it is pre-trained on medical literature and clinical reports, and outputs a 768-dimensional text feature vector. It is fine-tuned with all parameters.

10. A progressive image-text matching and comparison learning method for ultrasound images according to claim 1, characterized in that, In step S4, image features and text features are projected through a multilayer perceptron. The projection network contains two fully connected layers and the hidden layer dimension is 1024. The projected features are L2 normalized to be normalized into unit vectors. Cross-modal similarity is calculated using the cosine similarity of normalized features, which is the inner product of the normalized image feature vector and the normalized text feature vector, with a value range of [-1, 1].

Images