Echocardiographic image quality assessment method based on image and text feature fusion

Abstract

The present invention belongs to the field of medical imaging technology, and provides an echocardiogram image quality assessment method based on the fusion of image and text features, including: using a target detection model to extract and classify features, using a visual-language feature alignment module to align image features with text features, and using a large language model to select sentence templates, linear projection, and merging. The present invention generates an accurate and comprehensive echocardiogram image quality assessment report through cross-domain migration of image and text features, and improves the interpretability and granularity of the assessment results; by introducing a target detection model, the system can identify and understand the positional relationship of different anatomical structures in medical images, providing rich medical field prior knowledge for subsequent quality assessment, and realizing effective cross-domain migration from natural images to echocardiograms; by introducing sentence templates, the consistency of quality control texts is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and in particular to an ultrasound cardiogram image quality assessment method based on image and text feature fusion. Background Art

[0002] As a non-invasive, real-time imaging advanced medical imaging technology, echocardiography is widely used in the diagnosis of heart disease, monitoring of treatment progress, and evaluation of cardiac function. However, affected by multiple factors such as imaging conditions, doctors' operating skills and experience, and movement of human organs and tissues, the image quality of echocardiography varies. Low-quality images may have problems such as missing structural information and blurred display boundaries, which affect the doctor's diagnostic accuracy and decision-making efficiency. Therefore, accurate evaluation of echocardiography quality plays an indispensable role in ensuring the effectiveness and reliability of diagnosis. In clinical practice, echocardiography image quality control mainly relies on the doctor's experience and expertise, and usually diagnoses by manually screening high-quality images from a large number of images. This method is not only inefficient, but also highly subjective and inconsistent, and the evaluation results between different doctors may vary significantly.

[0003] Invention name: An echocardiogram quality evaluation method, system and terminal device, application number: CN202210991916.2, the invention discloses an echocardiogram quality evaluation method, system and terminal device, the method includes constructing an echocardiogram quality evaluation network model, collecting echocardiograms to be detected, inputting the echocardiograms to be detected into the echocardiogram quality evaluation network model, the echocardiogram quality evaluation network model outputs a number of echocardiogram quality control data, and weighted calculation is performed on the said several echocardiogram quality control data to obtain the quality assessment score of the echocardiogram. However, its training labels only select high-quality echocardiogram videos based on the subjective evaluation of relevant personnel with a medical background, and there are still large observer differences, and there is a lack of objective indicators to evaluate image quality. In addition, this method only considers image features, ignores the fusion of clinical text information, and the evaluation results lack interpretability. Invention name: Ultrasonic cardiogram section recognition method and ultrasonic cardiogram video section quality control method, application number: CN202310248785.3, the invention discloses an ultrasonic cardiogram section recognition method, an ultrasonic cardiogram video section quality control method, an electronic device and a readable storage medium. However, its image quality control is only based on the calculation of the predicted probability value of the section classification, lacks objective indicators for evaluating the image quality of each structure in the ultrasonic cardiogram, and lacks the experience of doctors in evaluating the image quality. Invention name: A fetal ultrasound image quality control method and system, application number: CN 201610991842.7, the invention discloses a fetal ultrasound image quality control method and system. The method realizes fetal ultrasound image quality control by judging whether the area of ​​the region of interest in the section occupies more than 1 / 2 of the scanning sector, judging whether the gastric bubble is full and has clear boundaries, judging whether the umbilical vein is hooked and displayed continuously and uninterruptedly. The method uses objective indicators for each structure in the fetal abdominal ultrasound image to evaluate the image quality and incorporates the experience of doctors. Fetal abdominal ultrasound images focus mainly on the morphology and development of organs, while echocardiography evaluates the structure and function of the heart. The difference in imaging targets determines that the focus and technical means of quality control evaluation need to be different. Therefore, the quality control method for the fetal abdomen cannot meet the evaluation needs of cardiac ultrasound images. In addition, this method does not incorporate text information and cannot output image quality evaluation results in the form of descriptive text. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide an echocardiography image quality assessment method based on the fusion of image and text features, and to generate an accurate and comprehensive echocardiography image quality assessment report through cross-domain migration of image and text features, so as to improve the interpretability and granularity of the assessment results; by introducing a target detection model, the system can identify and understand the positional relationship of different anatomical structures in medical images, and provide rich prior knowledge in the medical field for subsequent quality assessment, so as to achieve effective cross-domain migration from natural images to echocardiography; by introducing sentence templates, the consistency of quality control text is guaranteed.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] An echocardiographic image quality assessment method based on image and text feature fusion, comprising:

[0007] A pre-trained target detection model is used to extract and classify the cardiac anatomical structure in the target section image to obtain category label data and position data; the target detection model includes: a feature extraction network, a region candidate network, a first ROI pooling submodule and a classification and regression module; the feature extraction network is used to extract feature information of the cardiac anatomical structure in the target section image to obtain a first feature map; the region candidate network is used to calculate the foreground confidence score and vertex coordinate value of each candidate box according to the first feature map to obtain ROI region information; the first ROI pooling submodule is used to perform a maximum pooling operation on the ROI region information and the first feature map to obtain a feature vector of fixed size; the classification and regression module is used to classify and correct the position of the feature vector using a Softmax function and a border regression method to obtain the category label data and the position data;

[0008] According to the category label data and the position data, the image features of the cardiac anatomical structure and the quality control text features are aligned using a visual-linguistic feature alignment module based on a frozen target detection network to obtain query embedding and prompts; the visual-linguistic feature alignment module includes: a second ROI pooling submodule and a Q-Former submodule; the second ROI pooling submodule and the first ROI pooling submodule have the same structure and are synchronized with each other; the second ROI pooling submodule is used to perform a pooling operation on the output of the target detection model to obtain a second feature map; the Q-Former submodule is used to use a substructured input strategy to implement an alignment operation between image features and text features according to the second feature map to obtain the query embedding and the prompt;

[0009] The query embedding and the prompt are subjected to sentence template selection, linear projection and merging using a frozen large language model to obtain a quality-controlled text description.

[0010] Preferably, the image features of the cardiac anatomical structure and the quality control text features are aligned using a visual-linguistic feature alignment module based on a frozen object detection network according to the category label data and the position data to obtain query embedding and prompts, and further comprising:

[0011] When the target detection model cannot identify the target structure in the target section image, the elements in the second feature map corresponding to the target section image are replaced with -1.

[0012] Preferably, the query embedding and the prompt are subjected to sentence template selection, linear projection and merging using a frozen large language model to obtain a quality control text description, including:

[0013] Determining the sentence template of the large language model according to a masked language modeling strategy;

[0014] performing a linear projection on the query embedding and the hint;

[0015] The projected query embedding and the prompt are merged according to the sentence template to obtain the quality control text description.

[0016] Preferably, the base model of the large language model is BLIP-2.

[0017] Preferably, the first total loss function during training of the target detection model is:

[0018] ;

[0019] in, ; ; is the total loss function; is the cross entropy loss function; is the bounding box regression loss function; A hyperparameter for balancing the cross entropy loss function and the bounding box regression loss function; is the overall true label of the i-th sample; The probability of predicting the i-th overall true label for the target detection model; The difference between the bounding box predicted by the object detection model and the true bounding box.

[0020] Preferably, the second total loss function during the training of the large language model is:

[0021] ;

[0022] in, is the second total loss function; is the sample size; is the loss value of the i-th sample; is the number of categories; The quality assessment structure ground truth label representing the cth anatomical structure of the ith sample; The probability that the c-th anatomical structure quality assessment prediction result of the ith sample is the true label.

[0023] The present invention discloses the following technical effects:

[0024] The present invention provides an echocardiogram image quality assessment method based on the fusion of image and text features. Through the cross-domain migration of image and text features, the shortcomings of the prior art that the evaluation relies heavily on image features and ignores the fusion of text information are solved, and accurate and comprehensive assessment of the echocardiogram image quality is achieved; by introducing a target detection model, the defect of the prior art that the detailed assessment of each anatomical structure is lacking in image quality assessment is solved, and the recognition and understanding of the positional relationship of different anatomical structures and the effective cross-domain migration from natural images to echocardiograms are achieved; by introducing a sentence template, the defects of the conventional text generation method in outputting the text format and the poor word generation accuracy are solved, and the generation of consistent quality control text is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 A schematic diagram of an echocardiographic image quality assessment process based on image and text feature fusion provided by an embodiment of the present invention;

[0027] Figure 2 An implementation flow chart provided for an embodiment of the present invention;

[0028] Figure 3 A flow chart of a target detection model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The purpose of the present invention is to provide an echocardiography image quality assessment method based on the fusion of image and text features. Through the cross-domain migration of image and text features, an accurate and comprehensive echocardiography image quality assessment report is generated to improve the interpretability and granularity of the assessment results; by introducing a target detection model, the system is able to identify and understand the positional relationship of different anatomical structures in medical images, providing rich prior knowledge in the medical field for subsequent quality assessment, so as to achieve effective cross-domain migration from natural images to echocardiography; by introducing sentence templates, the consistency of quality control text is guaranteed.

[0031] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Figure 1 A schematic diagram of an echocardiographic image quality assessment process based on image and text feature fusion provided by an embodiment of the present invention, Figure 2 The implementation flow chart provided for the embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the present invention provides an echocardiographic image quality assessment method based on image and text feature fusion, comprising:

[0033] Step 100: Use the pre-trained target detection model to extract and classify the cardiac anatomical structure in the target section image to obtain category label data and position data; target detection model (refer to Figure 3 ) includes: a feature extraction network, a region candidate network, a first ROI pooling submodule and a classification and regression module; the feature extraction network is used to extract feature information of the cardiac anatomical structure in the target section image to obtain a first feature map; the region candidate network is used to calculate the foreground confidence score and vertex coordinate value of each candidate box according to the first feature map to obtain ROI region information; the first ROI pooling submodule is used to perform a maximum pooling operation on the ROI region information and the first feature map to obtain a feature vector of fixed size; the classification and regression module is used to classify and correct the position of the feature vector using the Softmax function and the bounding box regression method to obtain category label data and position data;

[0034] Step 200: Aligning the image features of the cardiac anatomical structure with the quality control text features using a visual-linguistic feature alignment module based on a frozen target detection network according to the category label data and the position data, and obtaining query embedding and prompts; the visual-linguistic feature alignment module includes: a second ROI pooling submodule and a Q-Former submodule; the second ROI pooling submodule and the first ROI pooling submodule have the same structure and are synchronized with each other; the second ROI pooling submodule is used to perform a pooling operation on the output of the target detection model to obtain a second feature map; the Q-Former submodule is used to use a substructured input strategy to achieve an alignment operation of the image features with the text features according to the second feature map, and obtain query embedding and prompts;

[0035] Step 300: Linearly project and merge the query embedding and prompt using the frozen large language model to obtain a quality-controlled text description.

[0036] Preferably, the image features of the cardiac anatomical structure are aligned with the quality control text features using a visual-linguistic feature alignment module based on a frozen object detection network according to the category label data and the position data to obtain query embedding and prompts, and further comprising:

[0037] When the target detection model cannot recognize the target structure in the target section image, the elements in the second feature map corresponding to the target section image are replaced with -1.

[0038] Specifically, the frozen large language model is used to select sentence templates, linearly project and merge query embeddings and prompts to obtain quality-controlled text descriptions, including:

[0039] Determine the sentence template of the large language model according to the masked language modeling strategy;

[0040] Linearly project query embeddings and hints;

[0041] The projected query embedding and prompt are merged according to the sentence template to obtain the quality control text description.

[0042] Preferably, the base model of the large language model is BLIP-2.

[0043] Preferably, the first total loss function during target detection model training is:

[0044] ;

[0045] in, ; ; is the total loss function; is the cross entropy loss function; is the bounding box regression loss function;

[0046] To balance the hyperparameters of the cross entropy loss function and the bounding box regression loss function, it is usually set to 1; is the overall true label of the i-th sample; The probability of predicting the i-th overall true label for the target detection model; The difference between the predicted bounding box and the true bounding box for the object detection model.

[0047] Preferably, the second total loss function during large language model training is:

[0048] ;

[0049] in, is the second total loss function; is the sample size; is the loss value of the i-th sample; is the number of categories; The quality assessment structure ground truth label representing the cth anatomical structure of the ith sample; The probability that the quality assessment prediction result of the c-th anatomical structure of the ith sample is the true label of the structure.

[0050] Optionally, compared with conventional one-stage and two-stage target detection models, TridentNet performs better in dealing with multi-scale target detection tasks. When detecting multiple cardiac ultrasound anatomical structures, TridentNet is used as the benchmark model of the target detection module because the valve structure area is relatively small, while the atrium and ventricle structure area is large.

[0051] Specifically, the pre-trained target detection model is constructed: TridentNet is used as the benchmark model of the target detection model. It is mainly composed of the following modules: Feature extraction network: Through a set of convolutional layers, ReLU activation functions and pooling operations, rich feature information is extracted from the input image to obtain a feature map for the subsequent region candidate network and pooling network. Region candidate network: Input the acquired feature map and output the information of multiple ROI regions, including the confidence score of each candidate box as the foreground, and the coordinate values ​​of the vertices of the candidate box, providing key information for subsequent processing. ROI pooling: Take the ROI region output by the region candidate network and the feature map output by the feature extraction network as input, perform a maximum pooling operation on each detection box, so as to process the structures of different sizes on the original feature map into a fixed-size feature vector, which is convenient for the classification and regression of the subsequent detection box. Classification and regression module: Receive the feature vector from ROI pooling as input, classify the image through the Softmax function, and use the border regression to correct the precise position of the object. Finally, this module outputs the category label of each ROI and its precise position in the image.

[0052] Furthermore, the BLIP-2 multimodal model is used as the benchmark model for the visual-linguistic echocardiography quality control model (the base model of the large language model). The visual representation is the image features of different anatomical structures in the echocardiogram, and the language representation is the text description of the clarity and completeness of different anatomical structures. The learnable Query Transformer (Q-Former) is used to achieve cross-domain feature migration between the two modalities of vision and language, and is the key module for achieving cross-domain migration of visual and language features in the BLIP-2 model. In order to achieve accurate alignment of the image features of cardiac anatomical structures with the quality control text features, a frozen pre-trained object detection module is first introduced; then, a ROI pooling module is cascaded after the object detection network, and its structure is the same as the ROI pooling module in the detection network, and the same pre-trained weights are shared, so that the feature vector input to the Q-Former has a fixed size; secondly, for the cardiac anatomical structures that cannot be recognized by the object detection module, a special preprocessing method is used, that is, a vector with the same size as the feature vector but all element values ​​are -1 is used to replace them. This preprocessing method is based on the following considerations: Since the target detection network is pre-trained on the preset ImageNet dataset and fine-tuned on high-quality echocardiography data, if some structures are not recognized during the detection process, it often means that the quality of these structures in the input image is poor. Therefore, the vectors with all element values ​​of -1 are input into Q-Former as the feature representation of these structures, which is both in line with the actual situation and helps the model to reasonably infer and predict these structures during the learning process; finally, the training strategy of structured input is adopted, and Q-Former receives each structural feature map as an independent input, that is, for a complete image, Q-Former needs to receive and process N feature maps (number of samples), and through Q-Former's powerful cross-domain migration capability, the image features of different anatomical structures of the heart and the corresponding quality control text features are accurately aligned.

[0053] Furthermore, evaluation text generation based on a large language model: First, the Query Embedding output by the Q-Former submodule is linearly projected through a fully connected layer to keep it consistent with the text embedding dimension of the Large Language Model (LLM) in BLIP-2. The projection result is used as a soft visual prompt to adjust the LLM according to the visual representation extracted by Q-Former; then, the projected Query Embedding is merged with the Prompt, and Q-Former acts as an information bottleneck in the process, passing only the most valuable visual information to the LLM, while filtering out irrelevant visual details to achieve effective fusion of visual information and text information; secondly, for the visual language model, Prompt is the key to guiding the model to generate text output. On the one hand, it can specify the model to perform different tasks, thereby generating different output answers; on the other hand, it can also provide certain reference knowledge for the model. In the present invention, Prompt is constructed in the form of "image introduction" + "structure introduction" + "quality control results", specifically:

[0054] “This is a <view>showing the <structure>, its clarity and integrity are evaluated as follows:"

[0055] This enables the large language model to better understand the various structural morphological features of the Q-Former output, making the input feature map of Prompt consistent with that of Q-Former, further improving the model's performance in text generation and image understanding. Finally, a sentence template is designed for LLM based on the masked language modeling strategy, which replaces the text describing the image quality with a specific <mask>The model completes the template into a complete description based on the remaining content of the sentence, combined with the feature information extracted from the image and the prompt, and merges these descriptions to form a comprehensive echocardiography quality assessment report.

[0056] The beneficial effects of the present invention are as follows:

[0057] (1) Introducing visual-language model for echocardiography quality assessment: The present invention innovatively applies visual-language model to echocardiography quality assessment. Through cross-domain migration of image and text features, an accurate and comprehensive echocardiography image quality assessment report is generated. The clarity and completeness of different anatomical structures in multiple ultrasound sections can be quantitatively evaluated, significantly improving the interpretability and clinical application value of the assessment results.

[0058] (2) Injecting medical prior knowledge into the model: The present invention introduces the target detection task in the first stage of model training, so that the model can recognize and understand the positional relationship of different anatomical structures in medical images, provide rich medical field prior knowledge for subsequent quality assessment, and realize effective cross-domain migration from natural images to echocardiograms.

[0059] (3) Using mask modeling strategy to ensure the consistency of quality control text: The present invention adopts mask language modeling strategy and provides predefined text generation templates. In the template, key content is <mask>Marking is used as a placeholder. On the one hand, it fixes the generated format and ensures the consistency of the output text format; on the other hand, it fills the model <mask>, guiding the model to generate appropriate words or phrases more accurately in combination with the context. This strategy effectively ensures that the description of each anatomical structure meets the specifications and consistency requirements of the quality control report, so that the output text not only maintains the fixed format, but also has the accuracy of the description and standardization that is easy to interpret.

[0060] The present invention generates an accurate and comprehensive echocardiography image quality assessment report through cross-domain migration of image and text features, thereby improving the interpretability and granularity of the assessment results. By introducing a target detection model, the system is able to identify and understand the positional relationship of different anatomical structures in medical images, providing rich prior knowledge in the medical field for subsequent quality assessment, and realizing effective cross-domain migration from natural images to echocardiograms. By introducing sentence templates, the consistency of quality control texts is ensured.

[0061] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0062] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.< / mask> < / mask> < / mask> < / structure> < / view>

Claims

1. A method for evaluating the quality of an echocardiographic image based on the fusion of image and text features, characterized in that: include: The pre-trained target detection model is used to extract and classify the cardiac anatomical structure in the target section image to obtain category label data and location data; The target detection model includes: a feature extraction network, a region candidate network, a first ROI pooling submodule and a classification and regression module; the feature extraction network is used to extract feature information of the cardiac anatomical structure in the target section image to obtain a first feature map; the region candidate network is used to calculate the foreground confidence score and vertex coordinate value of each candidate box according to the first feature map to obtain ROI region information; the first ROI pooling submodule is used to perform a maximum pooling operation on the ROI region information and the first feature map to obtain a feature vector of fixed size; the classification and regression module is used to classify and position correct the feature vector using a Softmax function and a frame regression method to obtain the category label data and the position data; According to the category label data and the position data, the image features of the cardiac anatomical structure and the quality control text features are aligned using a visual-linguistic feature alignment module based on a frozen target detection network to obtain query embedding and prompts; the visual-linguistic feature alignment module includes: a second ROI pooling submodule and a Q-Former submodule; the second ROI pooling submodule and the first ROI pooling submodule have the same structure and are synchronized with each other; the second ROI pooling submodule is used to perform a pooling operation on the output of the target detection model to obtain a second feature map; the Q-Former submodule is used to use a substructured input strategy to implement an alignment operation between image features and text features according to the second feature map to obtain the query embedding and the prompt; The query embedding and the prompt are subjected to sentence template selection, linear projection and merging using a frozen large language model to obtain a quality-controlled text description.

2. The method for evaluating the quality of an echocardiographic image based on image and text feature fusion according to claim 1, characterized in that: According to the category label data and the position data, a visual-language feature alignment module based on a frozen object detection network is used to align the image features of the cardiac anatomical structure with the quality control text features to obtain query embedding and prompts, further comprising: When the target detection model cannot identify the target structure in the target section image, the elements in the second feature map corresponding to the target section image are replaced with -1.

3. The method for evaluating the quality of an echocardiographic image based on image and text feature fusion according to claim 1, characterized in that: The query embedding and the prompt are subjected to sentence template selection, linear projection, and merging using a frozen large language model to obtain a quality control text description, including: Determining the sentence template of the large language model according to a masked language modeling strategy; performing a linear projection on the query embedding and the hint; The projected query embedding and the prompt are merged according to the sentence template to obtain the quality control text description.

4. The method for evaluating the quality of an echocardiographic image based on image and text feature fusion according to claim 1, characterized in that: The basic model of the large language model is BLIP-2.

5. The method for evaluating the quality of an echocardiographic image based on image and text feature fusion according to claim 1, characterized in that: The first total loss function during the training of the target detection model is: ； in, ; ; is the total loss function; is the cross entropy loss function; is the bounding box regression loss function; A hyperparameter for balancing the cross entropy loss function and the bounding box regression loss function; is the overall true label of the i-th sample; The probability of predicting the i-th overall true label for the target detection model; The difference between the bounding box predicted by the object detection model and the true bounding box.

6. The method for evaluating the quality of an echocardiographic image based on image and text feature fusion according to claim 1, characterized in that: The second total loss function during the training of the large language model is: ； in, is the second total loss function; is the sample size; is the loss value of the i-th sample; is the number of categories; The quality assessment structure ground truth label representing the cth anatomical structure of the ith sample; The probability that the quality assessment prediction result of the c-th anatomical structure of the ith sample is the true label of the structure.

Images